* silent semantic changes with reiser4 @ 2004-08-24 20:25 Christoph Hellwig 2004-08-24 20:35 ` Lee Revell ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-24 20:25 UTC (permalink / raw) To: akpm, reiser; +Cc: linux-fsdevel, linux-kernel After looking trough the code and mailinglists I'm quite unhappy with a bunch of user-visible changes that Hans sneaked in and make reiser4 incompatible with other filesystems and have a slight potential to break even in the kernel. o files as directories - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless - meaning of the -x permission. This one has different meanings on directories vs files on UNIX systems. If we want to support directories as files we'll probably have to find a way to work around this. - dentry aliasing. I can't find a formal guarantee in the code this can't happen o metafiles - ..metas as a magic name that's just taken out of the namespace doesn't sound like a good idea. If we want this it should be a VFS-level option and there should be a translation-layer to xattrs. Not doing this will again confuse applications greatly that expect uniform filesystem behaviour. Given these problems I request that these interfaces are removed from reiser4 for the kernel merge, and if added later at the proper VFS level after discussion on linux-kernel and linux-fsdevel, like we did for xattrs. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:25 silent semantic changes with reiser4 Christoph Hellwig @ 2004-08-24 20:35 ` Lee Revell 2004-08-24 20:38 ` Christoph Hellwig 2004-08-24 20:38 ` Jeff Garzik 2004-08-25 19:53 ` Hans Reiser 2 siblings, 1 reply; 1835+ messages in thread From: Lee Revell @ 2004-08-24 20:35 UTC (permalink / raw) To: Christoph Hellwig; +Cc: Andrew Morton, reiser, linux-fsdevel, linux-kernel On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote: > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > .htaccess handling in apache and glibc compilation this also renders > this flag entirely useless and opens up the races it tries to > prevent against cmpletely useless So `find -type d' would list every file on the system? Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:35 ` Lee Revell @ 2004-08-24 20:38 ` Christoph Hellwig 2004-08-24 20:42 ` Lee Revell 2004-08-24 21:18 ` Jamie Lokier 0 siblings, 2 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-24 20:38 UTC (permalink / raw) To: Lee Revell Cc: Christoph Hellwig, Andrew Morton, reiser, linux-fsdevel, linux-kernel On Tue, Aug 24, 2004 at 04:35:18PM -0400, Lee Revell wrote: > On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote: > > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > > .htaccess handling in apache and glibc compilation this also renders > > this flag entirely useless and opens up the races it tries to > > prevent against cmpletely useless > > So `find -type d' would list every file on the system? the find I have here is using lstat and not open with O_DIRECTORY, so no. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:38 ` Christoph Hellwig @ 2004-08-24 20:42 ` Lee Revell 2004-08-24 21:18 ` Jamie Lokier 1 sibling, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-24 20:42 UTC (permalink / raw) To: Christoph Hellwig; +Cc: Andrew Morton, reiser, linux-fsdevel, linux-kernel On Tue, 2004-08-24 at 16:38, Christoph Hellwig wrote: > On Tue, Aug 24, 2004 at 04:35:18PM -0400, Lee Revell wrote: > > On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote: > > > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > > > .htaccess handling in apache and glibc compilation this also renders > > > this flag entirely useless and opens up the races it tries to > > > prevent against cmpletely useless > > > > So `find -type d' would list every file on the system? > > the find I have here is using lstat and not open with O_DIRECTORY, so > no. > Ugh, how embarrassing, I completely forgot about stat(). Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:38 ` Christoph Hellwig 2004-08-24 20:42 ` Lee Revell @ 2004-08-24 21:18 ` Jamie Lokier 1 sibling, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-24 21:18 UTC (permalink / raw) To: Christoph Hellwig, Lee Revell, Andrew Morton, reiser, linux-fsdevel, linux-kernel Christoph Hellwig wrote: > > > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > > > .htaccess handling in apache and glibc compilation this also renders > > > this flag entirely useless and opens up the races it tries to > > > prevent against cmpletely useless > > > > So `find -type d' would list every file on the system? > > the find I have here is using lstat and not open with O_DIRECTORY, so > no. The find-like program I use (called treescan) uses O_DIRECTORY as an optimisation. It assumes that O_DIRECTORY will only open objects which are directories and can be read using readdir(). However, if reiser4 returns d_type values, then it won't even attempt an open on non-DT_DIR objects, and that's a better optimisation. (reiserfs doesn't return d_type values, unfortunately). So the list of files that treescan finds depends on whether reiser4 implements d_type. This is nothing like a POSIX filesystem. You untar a tree, and then listing it recursively shows extra things created by reiser4. I quite like the principle, but because it's not like POSIX and doesn't match some program's expectations, it's a problem in its present form. xattrs aren't a complete solution as you can't store structured data in an xattr. For example, with reiser4's model, you can cd into a .tar, .zip, .mp3 or .xml file and list the internal structure along with the file's metadata. You can't do that with xattrs. Programs exist which quite reasonably assume that when you create a file, you can't opendir() the file, and recursive listings (like find, ls -R et al.) won't automatically traverse into every file. On the other hand, being able to enter a file in a directory-like way allows structured representations of the contents to be accessed in the very useful "everything's a file" way -- i.e. ordinary tools. So here's a semantic proposal: 1. O_DIRECTORY won't open an ordinary file. Corollary: opendir("file") won't open an ordinary file. 2. An ordinary file path followed by "/" won't open an ordinary file. Corollary: opendir("file/") won't open an ordinary file. This is because appending a trailing slash is an alternate way for userspace to get the same results as O_DIRECTORY. 3. An ordinary file path followed by "/" _and_ one or more path components will open the file as a directory and enter it. Corollary: opendir("file/.") will open an ordinary file. 4. The type of "file/." shall be S_IFDIR, _not_ S_IFREG. Corollary: stat("file/.") will return that it's a directory. The intention here is that explicit requests to examine the metadata or alternate structure representations of a file will create such a view, but the view is only available if requested explicitly. When such a view is created, the results of stat(), O_DIRECTORY and opendir() are absolutely consistent. This will minimise confusion. Programs which recurse over a directory tree won't look inside any of the files. However they can be explicitly asked to recurse starting from a path inside a file: then they'll recurse over a single file's metadata and structured data. Regarding the problems of safe locking in the VFS. The VFS assumes that directories are not hard linked: i.e. that they cannot appear at more than one path in a filesystem. Files-as-directories breaks that. However, VFS does support directories on multiple paths, using bind mounts. So it wouldn't be out of the question if entering a file (as described above) effectively auto-mounted a bind mount at that point. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:25 silent semantic changes with reiser4 Christoph Hellwig 2004-08-24 20:35 ` Lee Revell @ 2004-08-24 20:38 ` Jeff Garzik 2004-08-24 20:53 ` viro 2004-08-25 19:53 ` Hans Reiser 2 siblings, 1 reply; 1835+ messages in thread From: Jeff Garzik @ 2004-08-24 20:38 UTC (permalink / raw) To: Christoph Hellwig; +Cc: akpm, reiser, linux-fsdevel, linux-kernel Christoph Hellwig wrote: > o files as directories > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > .htaccess handling in apache and glibc compilation this also renders > this flag entirely useless and opens up the races it tries to > prevent against cmpletely useless Ouch. I would definitely classify this as a security hole, since userland definitely uses O_DIRECTORY to avoid races. Jeff ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:38 ` Jeff Garzik @ 2004-08-24 20:53 ` viro 2004-08-24 21:22 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-24 20:53 UTC (permalink / raw) To: Jeff Garzik; +Cc: Christoph Hellwig, akpm, reiser, linux-fsdevel, linux-kernel On Tue, Aug 24, 2004 at 04:38:25PM -0400, Jeff Garzik wrote: > Christoph Hellwig wrote: > > o files as directories > > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > > .htaccess handling in apache and glibc compilation this also renders > > this flag entirely useless and opens up the races it tries to > > prevent against cmpletely useless > > > Ouch. > > I would definitely classify this as a security hole, since userland > definitely uses O_DIRECTORY to avoid races. Feh. That's far from the worst parts of the mess introduced by "hybrid" crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit higher on the suckitude scale, IMO. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:53 ` viro @ 2004-08-24 21:22 ` viro 2004-08-25 18:28 ` Hans Reiser 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-24 21:22 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, akpm, reiser, linux-fsdevel, linux-kernel, Jeff Garzik On Tue, Aug 24, 2004 at 09:53:44PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > Feh. That's far from the worst parts of the mess introduced by "hybrid" > crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit > higher on the suckitude scale, IMO. While we are at it - consider these hybrids vetoed until a) sys_link()/sys_link() deadlock is fixed b) sys_link()/sys_rename() deadlock is fixed c) correctness proof of the locking scheme (in Documentation/filesystems/directory-locking) is updated to match the presense of the file/directory hybrids. Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince us that there's no more crap of that kind". IMO a reasonable request, seeing that the first look at the patches in -mm4 had turned up two exploits in that area, despite the *YEARS* of warnings about potential trouble and need to be careful there (actually, I've given Hans too much credit and assumed that link/link never happens since nobody would be dumb enough to provide ->link() method for non-directory inodes; turns out that somebody is dumb enough and link/link is as exploitable as link/rename). ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 21:22 ` viro @ 2004-08-25 18:28 ` Hans Reiser 2004-08-25 18:45 ` Christoph Hellwig 0 siblings, 1 reply; 1835+ messages in thread From: Hans Reiser @ 2004-08-25 18:28 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list I allowed myself to get talked out of a final top to bottom code audit, and obviously that was a mistake. It will probably take about 6 weeks. Apologies for wasting your time before that was done. Hans viro@parcelfarce.linux.theplanet.co.uk wrote: >On Tue, Aug 24, 2004 at 09:53:44PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > > >>Feh. That's far from the worst parts of the mess introduced by "hybrid" >>crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit >>higher on the suckitude scale, IMO. >> >> > >While we are at it - consider these hybrids vetoed until > a) sys_link()/sys_link() deadlock is fixed > b) sys_link()/sys_rename() deadlock is fixed > c) correctness proof of the locking scheme (in >Documentation/filesystems/directory-locking) is updated to match the >presense of the file/directory hybrids. > >Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince >us that there's no more crap of that kind". IMO a reasonable request, seeing >that the first look at the patches in -mm4 had turned up two exploits in >that area, despite the *YEARS* of warnings about potential trouble and need >to be careful there (actually, I've given Hans too much credit and assumed >that link/link never happens since nobody would be dumb enough to provide >->link() method for non-directory inodes; turns out that somebody is dumb >enough and link/link is as exploitable as link/rename). >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 18:28 ` Hans Reiser @ 2004-08-25 18:45 ` Christoph Hellwig 2004-08-26 9:02 ` Hans Reiser 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-25 18:45 UTC (permalink / raw) To: Hans Reiser Cc: viro, Linus Torvalds, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list On Wed, Aug 25, 2004 at 11:28:56AM -0700, Hans Reiser wrote: > I allowed myself to get talked out of a final top to bottom code audit, > and obviously that was a mistake. > > It will probably take about 6 weeks. Apologies for wasting your time > before that was done. I don't think you'll get anywhere with auditing. We need to write down the semantics you want, define them at the VFS level and make sure they're not conflicting with defined userspace semantics or kernel assumptions. I think you need to learn the basic distinction between the VFS layer and a lowlevel filesystem driver. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 18:45 ` Christoph Hellwig @ 2004-08-26 9:02 ` Hans Reiser 2004-08-26 18:38 ` William Lee Irwin III 0 siblings, 1 reply; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 9:02 UTC (permalink / raw) To: Christoph Hellwig Cc: viro, Linus Torvalds, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list Christoph Hellwig wrote: > > >I don't think you'll get anywhere with auditing. We need to write down >the semantics you want, define them at the VFS level and make sure >they're not conflicting with defined userspace semantics or kernel >assumptions. > >I think you need to learn the basic distinction between the VFS layer >and a lowlevel filesystem driver. > > > > > How old are you? I thought you were the guy at Linux Tag with fashion oriented hair who gave a talk on his XFS work? Did I confuse you with someone else? Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:02 ` Hans Reiser @ 2004-08-26 18:38 ` William Lee Irwin III 2004-08-26 18:49 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: William Lee Irwin III @ 2004-08-26 18:38 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, viro, Linus Torvalds, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list On Thu, Aug 26, 2004 at 02:02:29AM -0700, Hans Reiser wrote: > How old are you? I thought you were the guy at Linux Tag with fashion > oriented hair who gave a talk on his XFS work? Did I confuse you with > someone else? Index: mm1-2.6.9-rc1/CREDITS =================================================================== --- mm1-2.6.9-rc1.orig/CREDITS 2004-08-26 10:37:57.424567967 -0700 +++ mm1-2.6.9-rc1/CREDITS 2004-08-26 11:35:24.977260109 -0700 @@ -2677,6 +2677,14 @@ S: 79098 Freiburg S: Germany +N: Hans Reiser +E: reiser@namesys.com +W: http://www.namesys.com/ +D: official Linux kernel hairstyle critic +S: 6979 Exeter Dr +S: Oakland, CA 94611 +S: USA + N: Joerg Reuter E: jreuter@yaina.de W: http://yaina.de/jreuter/ ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:38 ` William Lee Irwin III @ 2004-08-26 18:49 ` Linus Torvalds 2004-08-26 18:56 ` William Lee Irwin III 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 18:49 UTC (permalink / raw) To: William Lee Irwin III Cc: Hans Reiser, Christoph Hellwig, viro, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list On Thu, 26 Aug 2004, William Lee Irwin III wrote: > > +N: Hans Reiser > +E: reiser@namesys.com > +W: http://www.namesys.com/ > +D: official Linux kernel hairstyle critic > +S: 6979 Exeter Dr > +S: Oakland, CA 94611 > +S: USA > + LOL. In all fairness, how _would_ you have described it? Besides, I don't think this should go in the CREDITS file, since hair styling criticism is clearly an ongoing MAINTAINERS issue, no? Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:49 ` Linus Torvalds @ 2004-08-26 18:56 ` William Lee Irwin III 2004-08-27 8:14 ` Hans Reiser 2004-08-27 10:10 ` Giuseppe Bilotta 0 siblings, 2 replies; 1835+ messages in thread From: William Lee Irwin III @ 2004-08-26 18:56 UTC (permalink / raw) To: Linus Torvalds Cc: Hans Reiser, Christoph Hellwig, viro, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list On Thu, 26 Aug 2004, William Lee Irwin III wrote: >> +N: Hans Reiser >> +E: reiser@namesys.com >> +W: http://www.namesys.com/ >> +D: official Linux kernel hairstyle critic >> +S: 6979 Exeter Dr >> +S: Oakland, CA 94611 >> +S: USA On Thu, Aug 26, 2004 at 11:49:19AM -0700, Linus Torvalds wrote: > LOL. > In all fairness, how _would_ you have described it? > Besides, I don't think this should go in the CREDITS file, since hair > styling criticism is clearly an ongoing MAINTAINERS issue, no? As far as maintenance goes, absolutely. As far as describing it goes, I probably would have said it was short, red, and spiky. Index: mm1-2.6.9-rc1/MAINTAINERS =================================================================== --- mm1-2.6.9-rc1.orig/MAINTAINERS 2004-08-26 10:38:09.867927190 -0700 +++ mm1-2.6.9-rc1/MAINTAINERS 2004-08-26 11:58:27.759469733 -0700 @@ -1352,6 +1352,12 @@ L: linuxppc64-dev@lists.linuxppc.org S: Supported +LINUX KERNEL HAIRSTYLE CRITICISM +P: Hans Reiser +M: reiser@namesys.com +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: chrisw@osdl.org ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:56 ` William Lee Irwin III @ 2004-08-27 8:14 ` Hans Reiser 2004-08-27 10:10 ` Giuseppe Bilotta 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:14 UTC (permalink / raw) To: William Lee Irwin III Cc: Linus Torvalds, Christoph Hellwig, viro, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list William Lee Irwin III wrote: > >As far as maintenance goes, absolutely. As far as describing it goes, >I probably would have said it was short, red, and spiky. > > > Ah, so it was him, is he disowning XFS these days or? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:56 ` William Lee Irwin III 2004-08-27 8:14 ` Hans Reiser @ 2004-08-27 10:10 ` Giuseppe Bilotta 2004-08-27 15:25 ` William Lee Irwin III 1 sibling, 1 reply; 1835+ messages in thread From: Giuseppe Bilotta @ 2004-08-27 10:10 UTC (permalink / raw) To: linux-fsdevel; +Cc: linux-kernel William Lee Irwin III wrote: > On Thu, 26 Aug 2004, William Lee Irwin III wrote: > >> +N: Hans Reiser > >> +E: reiser@namesys.com > >> +W: http://www.namesys.com/ > >> +D: official Linux kernel hairstyle critic > >> +S: 6979 Exeter Dr > >> +S: Oakland, CA 94611 > >> +S: USA > > On Thu, Aug 26, 2004 at 11:49:19AM -0700, Linus Torvalds wrote: > > LOL. > > In all fairness, how _would_ you have described it? > > Besides, I don't think this should go in the CREDITS file, since hair > > styling criticism is clearly an ongoing MAINTAINERS issue, no? > > As far as maintenance goes, absolutely. As far as describing it goes, > I probably would have said it was short, red, and spiky. > > > Index: mm1-2.6.9-rc1/MAINTAINERS > =================================================================== > --- mm1-2.6.9-rc1.orig/MAINTAINERS 2004-08-26 10:38:09.867927190 -0700 > +++ mm1-2.6.9-rc1/MAINTAINERS 2004-08-26 11:58:27.759469733 -0700 > @@ -1352,6 +1352,12 @@ > L: linuxppc64-dev@lists.linuxppc.org > S: Supported > > +LINUX KERNEL HAIRSTYLE CRITICISM Hm. He doesn't critic the Linux Kernel hairstyle, but the Linux Kernel Developers' ... -- Giuseppe "Oblomov" Bilotta Can't you see It all makes perfect sense Expressed in dollar and cents Pounds shillings and pence (Roger Waters) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 10:10 ` Giuseppe Bilotta @ 2004-08-27 15:25 ` William Lee Irwin III [not found] ` <20040827161539.GC28876@khan.acc.umu.se> 0 siblings, 1 reply; 1835+ messages in thread From: William Lee Irwin III @ 2004-08-27 15:25 UTC (permalink / raw) To: Linus Torvalds; +Cc: Giuseppe Bilotta, linux-fsdevel, linux-kernel William Lee Irwin III wrote: >> +LINUX KERNEL HAIRSTYLE CRITICISM On Fri, Aug 27, 2004 at 12:10:12PM +0200, Giuseppe Bilotta wrote: > Hm. He doesn't critic the Linux Kernel hairstyle, but the Linux > Kernel Developers' ... Absolutely right. Index: mm1-2.6.9-rc1/MAINTAINERS =================================================================== --- mm1-2.6.9-rc1.orig/MAINTAINERS 2004-08-26 10:38:09.000000000 -0700 +++ mm1-2.6.9-rc1/MAINTAINERS 2004-08-27 08:27:42.738139199 -0700 @@ -1352,6 +1352,12 @@ L: linuxppc64-dev@lists.linuxppc.org S: Supported +LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM +P: Hans Reiser +M: reiser@namesys.com +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: chrisw@osdl.org ^ permalink raw reply [flat|nested] 1835+ messages in thread
[parent not found: <20040827161539.GC28876@khan.acc.umu.se>]
* Re: silent semantic changes with reiser4 [not found] ` <20040827161539.GC28876@khan.acc.umu.se> @ 2004-08-27 16:18 ` William Lee Irwin III 0 siblings, 0 replies; 1835+ messages in thread From: William Lee Irwin III @ 2004-08-27 16:18 UTC (permalink / raw) To: David Weinehall; +Cc: akpm, torvalds, linux-fsdevel, linux-kernel On Fri, Aug 27, 2004 at 08:25:39AM -0700, William Lee Irwin III wrote: >> +LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM > Aren't there more than one Linux Kernel Developer? Or is this to > indicate that he indeed only criticised the hairstyle of one developer? Best I repair that, then. Index: mm1-2.6.9-rc1/MAINTAINERS =================================================================== --- mm1-2.6.9-rc1.orig/MAINTAINERS 2004-08-26 10:38:09.000000000 -0700 +++ mm1-2.6.9-rc1/MAINTAINERS 2004-08-27 09:18:59.234195262 -0700 @@ -1352,6 +1352,12 @@ L: linuxppc64-dev@lists.linuxppc.org S: Supported +LINUX KERNEL DEVELOPERS' HAIRSTYLE CRITICISM +P: Hans Reiser +M: reiser@namesys.com +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: chrisw@osdl.org ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:25 silent semantic changes with reiser4 Christoph Hellwig @ 2004-08-25 19:53 ` Hans Reiser 2004-08-24 20:38 ` Jeff Garzik 2004-08-25 19:53 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-25 19:53 UTC (permalink / raw) To: Christoph Hellwig Cc: akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus. Sigh. Dear Christoph, Let me see if I can summarize what you and your contingent are saying, and if I misconstrue anything, let me know.;-) You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles. When I said we should just add some nice optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it. Making files into directories caused only two applications out of the entire OS to notice the change, and that was because of a bug in what error code we returned that we are going to fix. You think that was a disaster; I think it was a triumph. Now a cleanly architected filesystem with no attributes and just files and directories that can do everything attributes are used for exists. You don't want it to have the competitive advantage. Instead, you want it to have its clean design excised until you have something that duplicates it ready to go, and only then should it be allowed that users will use the features of your competitor's filesystem which you disdained implementing for so long. Since you never studied or understood namespace design principles (or you would not have created and supported xattrs), you want to rename it to be called VFS, rewrite what we have done, and take over as the maintainer, mangling its design in a committee clusterfuck as you go. We have just implemented very trivial semantic enhancements of the FS namespace, nothing like as ambitious as www.namesys.com/whitepaper.html or WinFS, and you are already pissing your pants. Is that a fair summary? Eat my dust. Hans PS I should of course qualify what I have said. The use of files and directories in place of attributes is not a finished work. It has bugs, sys_reiser4() does not yet work, and there are little features still missing like having files readdir ignores. Still, except for the bugs, what we have is usable, and there are a lot of happy reiser4 users right now even with the bugs. It will need a little bit more time, and then all the pieces will be in place. PPS If you implement your filesystems as reiser4 plugins, and rename reiser4's plugin code to be called "vfs", your filesystems will go faster. Not as fast as reiser4 though, because it has a better layout and that affects performance a lot, but faster is faster.... See www.namesys.com/benchmarks.html for details. PPPS Since we have such a performance lead, Namesys is about to change its focus from the storage layer to semantics, look at www.namesys.com/whitepaper.html for details. Semantic enhancements are the important stuff, and finally Namesys is where we have all the storage layer prerequisites done right, and the real work can begin. The gap between us is about to widen further. Christoph Hellwig wrote: >After looking trough the code and mailinglists I'm quite unhappy with >a bunch of user-visible changes that Hans sneaked in and make reiser4 >incompatible with other filesystems > if we leave you in the dust, run faster.... not my problem.... >Given these problems I request that these interfaces are removed from >reiser4 for the kernel merge, and if added later at the proper VFS level >after discussion on linux-kernel and linux-fsdevel, like we did for >xattrs. > > > > If you can't help fight WinFS, then get out of the way. Namesys is on the march. Read www.namesys.com/whitepaper.html. Or, be smart, recognize that reiser4 is faster and more flexible than your storage layers because we are older and wiser and worked harder at it, join the team, and start contributing plugins that tap into the higher performance it offers. Microsoft tried to build a storage layer that could handle small objects without losing performance, failed, and gave up at considerable cost to their architecture and pocketbook. We just broke a hole in the enemy line. You could come swarming through it with us, but it sounds like you prefer complaining to HQ that we are getting too far in front of you. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-25 19:53 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-25 19:53 UTC (permalink / raw) To: Christoph Hellwig Cc: akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus. Sigh. Dear Christoph, Let me see if I can summarize what you and your contingent are saying, and if I misconstrue anything, let me know.;-) You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles. When I said we should just add some nice optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it. Making files into directories caused only two applications out of the entire OS to notice the change, and that was because of a bug in what error code we returned that we are going to fix. You think that was a disaster; I think it was a triumph. Now a cleanly architected filesystem with no attributes and just files and directories that can do everything attributes are used for exists. You don't want it to have the competitive advantage. Instead, you want it to have its clean design excised until you have something that duplicates it ready to go, and only then should it be allowed that users will use the features of your competitor's filesystem which you disdained implementing for so long. Since you never studied or understood namespace design principles (or you would not have created and supported xattrs), you want to rename it to be called VFS, rewrite what we have done, and take over as the maintainer, mangling its design in a committee clusterfuck as you go. We have just implemented very trivial semantic enhancements of the FS namespace, nothing like as ambitious as www.namesys.com/whitepaper.html or WinFS, and you are already pissing your pants. Is that a fair summary? Eat my dust. Hans PS I should of course qualify what I have said. The use of files and directories in place of attributes is not a finished work. It has bugs, sys_reiser4() does not yet work, and there are little features still missing like having files readdir ignores. Still, except for the bugs, what we have is usable, and there are a lot of happy reiser4 users right now even with the bugs. It will need a little bit more time, and then all the pieces will be in place. PPS If you implement your filesystems as reiser4 plugins, and rename reiser4's plugin code to be called "vfs", your filesystems will go faster. Not as fast as reiser4 though, because it has a better layout and that affects performance a lot, but faster is faster.... See www.namesys.com/benchmarks.html for details. PPPS Since we have such a performance lead, Namesys is about to change its focus from the storage layer to semantics, look at www.namesys.com/whitepaper.html for details. Semantic enhancements are the important stuff, and finally Namesys is where we have all the storage layer prerequisites done right, and the real work can begin. The gap between us is about to widen further. Christoph Hellwig wrote: >After looking trough the code and mailinglists I'm quite unhappy with >a bunch of user-visible changes that Hans sneaked in and make reiser4 >incompatible with other filesystems > if we leave you in the dust, run faster.... not my problem.... >Given these problems I request that these interfaces are removed from >reiser4 for the kernel merge, and if added later at the proper VFS level >after discussion on linux-kernel and linux-fsdevel, like we did for >xattrs. > > > > If you can't help fight WinFS, then get out of the way. Namesys is on the march. Read www.namesys.com/whitepaper.html. Or, be smart, recognize that reiser4 is faster and more flexible than your storage layers because we are older and wiser and worked harder at it, join the team, and start contributing plugins that tap into the higher performance it offers. Microsoft tried to build a storage layer that could handle small objects without losing performance, failed, and gave up at considerable cost to their architecture and pocketbook. We just broke a hole in the enemy line. You could come swarming through it with us, but it sounds like you prefer complaining to HQ that we are getting too far in front of you. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser (?) @ 2004-08-25 20:06 ` Matthew Wilcox 2004-08-26 8:41 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Matthew Wilcox @ 2004-08-25 20:06 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List That's a nice marketing talk. Get back to us when you have some technical contribution to make. -- "Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:06 ` Matthew Wilcox @ 2004-08-26 8:41 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:41 UTC (permalink / raw) To: Matthew Wilcox Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Matthew Wilcox wrote: >That's a nice marketing talk. Get back to us when you have some technical >contribution to make. > > > www.namesys.com/download.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:41 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:41 UTC (permalink / raw) To: Matthew Wilcox Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Matthew Wilcox wrote: >That's a nice marketing talk. Get back to us when you have some technical >contribution to make. > > > www.namesys.com/download.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser (?) (?) @ 2004-08-25 20:08 ` Christoph Hellwig 2004-08-25 20:19 ` Christoph Hellwig ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:08 UTC (permalink / raw) To: Hans Reiser Cc: linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote: > You ignored everything I said during the discussion of xattrs about how > there is no need to have attributes when you can just have files and > directories, and that xattrs reflected a complete ignorance of name > space design principles. Actually in most of the discussion you simply didn't participate. While xattrs might not be the nicest interface they have the advantag of not breaking the SuS assumption of what directories vs files are, and they do not break the Linux O_DIRECTORY semantics that are defined and need to solve real-world races either. > When I said we should just add some nice > optional features to files and directories so that they can do > everything that attributes can do if they are used that way, you just > didn't get it. You instead went for the quick ugly hack called xattrs. > You then got that ugly hack done first, because quick hacks are, well, > quick. I then went about doing it the right way for Reiser4, and got > DARPA to fund doing it. I was never silent about it. For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories. > Now a cleanly architected filesystem with no attributes and just files > and directories that can do everything attributes are used for exists. > You don't want it to have the competitive advantage. Instead, you want > it to have its clean design excised until you have something that > duplicates it ready to go, and only then should it be allowed that users > will use the features of your competitor's filesystem which you > disdained implementing for so long. My competitors filesystem? If you look at MAINTAINERS I maintain only vxfs and sysvfs, neither of which I'd suggest anyone to run their system on. > Since you never studied or understood namespace design principles (or > you would not have created and supported xattrs), you want to rename it > to be called VFS, rewrite what we have done, and take over as the > maintainer, mangling its design in a committee clusterfuck as you go. Hans, please stop the personal crap or the black helicopters will kidnap you. When was the last time you actually worked on kernel namespace code instead of talking marketing bullshit and ignoring all real world problems. > If you implement your filesystems as reiser4 plugins, and rename > reiser4's plugin code to be called "vfs", your filesystems will go > faster. Not as fast as reiser4 though, because it has a better layout > and that affects performance a lot, but faster is faster.... See > www.namesys.com/benchmarks.html for details. Could you pass on that crack pipe please? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:08 ` Christoph Hellwig @ 2004-08-25 20:19 ` Christoph Hellwig 2004-08-25 20:24 ` Linus Torvalds 2004-08-26 8:42 ` Hans Reiser 2004-08-25 20:22 ` Linus Torvalds 2004-08-25 20:35 ` Alex Zarochentsev 2 siblings, 2 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:19 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Btw, I just got reminded you might take my saying as an "piss of you're idea stinks" or similar things. So let me clarify again the actual technical and project managment issues another time before we start getting really personal :) Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux. So now go on and try to work together with the other peope doing VFS level work instead of hiding, or if you think you can't work together with us search a nice research OS where you can take over the VFS layer, if your ideas prove to be good I'm sure Linux will pick them up sooner or later. Christoph ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:19 ` Christoph Hellwig @ 2004-08-25 20:24 ` Linus Torvalds 2004-08-25 20:25 ` Christoph Hellwig ` (2 more replies) 2004-08-26 8:42 ` Hans Reiser 1 sibling, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-25 20:24 UTC (permalink / raw) To: Christoph Hellwig Cc: Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > Over the last at least five years we've taken as much as possible > semantics out of the filesystems and into the VFS layer, thus having > a separation between the semantical layer (VFS) and the low level > filesystem. Your attributes are absoultely a VFS thing and as such > should not happen at the filesystem layer, and no, that doesn't mean > they're bad per se, I just think they are a rather bad fit for Linux. Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer. On the other hand, I think doing it inside the filesystem with ugly hacks is an acceptable way to prototype the idea before it's been proven to really be workable. Maybe it has more problems with legacy apps than we'd expect.. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:24 ` Linus Torvalds @ 2004-08-25 20:25 ` Christoph Hellwig 2004-08-25 20:59 ` viro 2004-08-26 8:43 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:25 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:24:36PM -0700, Linus Torvalds wrote: > Now this I agree with, in the sense that I think that if we want to > support this, it should be supported at a VFS layer. > > On the other hand, I think doing it inside the filesystem with ugly hacks > is an acceptable way to prototype the idea before it's been proven to > really be workable. Maybe it has more problems with legacy apps than we'd > expect.. Oh, I'm the last person to tell anyone how to prototype things. I just don't want such inconsistancies in the mainline kernel. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:24 ` Linus Torvalds 2004-08-25 20:25 ` Christoph Hellwig @ 2004-08-25 20:59 ` viro 2004-08-26 8:43 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: viro @ 2004-08-25 20:59 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:24:36PM -0700, Linus Torvalds wrote: > > > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > Over the last at least five years we've taken as much as possible > > semantics out of the filesystems and into the VFS layer, thus having > > a separation between the semantical layer (VFS) and the low level > > filesystem. Your attributes are absoultely a VFS thing and as such > > should not happen at the filesystem layer, and no, that doesn't mean > > they're bad per se, I just think they are a rather bad fit for Linux. > > Now this I agree with, in the sense that I think that if we want to > support this, it should be supported at a VFS layer. ACK. However, I'm still not seeing *ANYTHING* that would look like a workable scheme in presense of hardlinks. Show me how to make that deadlock- and race-free and we might very well do it in VFS. _That_ is what's missing and it's needed no matter where it's implemented. You want hybrid objects - you want to solve that one. So far I've seen nothing workable. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:24 ` Linus Torvalds @ 2004-08-26 8:43 ` Hans Reiser 2004-08-25 20:59 ` viro 2004-08-26 8:43 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > >>Over the last at least five years we've taken as much as possible >>semantics out of the filesystems and into the VFS layer, thus having >>a separation between the semantical layer (VFS) and the low level >>filesystem. Your attributes are absoultely a VFS thing and as such >>should not happen at the filesystem layer, and no, that doesn't mean >>they're bad per se, I just think they are a rather bad fit for Linux. >> >> > >Now this I agree with, in the sense that I think that if we want to >support this, it should be supported at a VFS layer. > >On the other hand, I think doing it inside the filesystem with ugly hacks > > what is ugly? ;-/ >is an acceptable way to prototype the idea before it's been proven to >really be workable. Maybe it has more problems with legacy apps than we'd >expect.. > > Linus > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:43 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > >>Over the last at least five years we've taken as much as possible >>semantics out of the filesystems and into the VFS layer, thus having >>a separation between the semantical layer (VFS) and the low level >>filesystem. Your attributes are absoultely a VFS thing and as such >>should not happen at the filesystem layer, and no, that doesn't mean >>they're bad per se, I just think they are a rather bad fit for Linux. >> >> > >Now this I agree with, in the sense that I think that if we want to >support this, it should be supported at a VFS layer. > >On the other hand, I think doing it inside the filesystem with ugly hacks > > what is ugly? ;-/ >is an acceptable way to prototype the idea before it's been proven to >really be workable. Maybe it has more problems with legacy apps than we'd >expect.. > > Linus > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:19 ` Christoph Hellwig @ 2004-08-26 8:42 ` Hans Reiser 2004-08-26 8:42 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:42 UTC (permalink / raw) To: Christoph Hellwig Cc: linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > >Over the last at least five years we've taken as much as possible >semantics out of the filesystems and into the VFS layer, thus having >a separation between the semantical layer (VFS) and the low level >filesystem. > VFS is the common filesystem layer. The only reason you think semantics belong in the common filesystem layer is that you are not innovating in your semantics, and feel content with stasis. I don't. I expect that semantics will get radically changed over the next few years as we compete with Giampaolo and whatever lesser lights are working at Microsoft. > Your attributes are absoultely a VFS thing and as such >should not happen at the filesystem layer, and no, that doesn't mean >they're bad per se, I just think they are a rather bad fit for Linux. > >So now go on and try to work together with the other peope doing VFS >level work instead of hiding, or if you think you can't work together >with us search a nice research OS where you can take over the VFS layer, >if your ideas prove to be good I'm sure Linux will pick them up sooner >or later. > > Christoph > > > > I tell you what, use xattrs for all the half speed filesystems, and the users and I will use metafiles. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:42 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:42 UTC (permalink / raw) To: Christoph Hellwig Cc: linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > >Over the last at least five years we've taken as much as possible >semantics out of the filesystems and into the VFS layer, thus having >a separation between the semantical layer (VFS) and the low level >filesystem. > VFS is the common filesystem layer. The only reason you think semantics belong in the common filesystem layer is that you are not innovating in your semantics, and feel content with stasis. I don't. I expect that semantics will get radically changed over the next few years as we compete with Giampaolo and whatever lesser lights are working at Microsoft. > Your attributes are absoultely a VFS thing and as such >should not happen at the filesystem layer, and no, that doesn't mean >they're bad per se, I just think they are a rather bad fit for Linux. > >So now go on and try to work together with the other peope doing VFS >level work instead of hiding, or if you think you can't work together >with us search a nice research OS where you can take over the VFS layer, >if your ideas prove to be good I'm sure Linux will pick them up sooner >or later. > > Christoph > > > > I tell you what, use xattrs for all the half speed filesystems, and the users and I will use metafiles. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:42 ` Hans Reiser (?) @ 2004-08-26 9:24 ` Christoph Hellwig 2004-08-26 18:45 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 9:24 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 01:42:06AM -0700, Hans Reiser wrote: > Christoph Hellwig wrote: > > > > >Over the last at least five years we've taken as much as possible > >semantics out of the filesystems and into the VFS layer, thus having > >a separation between the semantical layer (VFS) and the low level > >filesystem. > > > VFS is the common filesystem layer. The only reason you think semantics > belong in the common filesystem layer is that you are not innovating in > your semantics, and feel content with stasis. > > I don't. I expect that semantics will get radically changed over the > next few years as we compete with Giampaolo and whatever lesser lights > are working at Microsoft. Hans, please stop the gooddamn personal attack bullshit. How do you for example suggestion exporting your semantics over the network if they're not done at the VFS level? How do you want some clusterfilesystem support them or tmpfs? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:24 ` Christoph Hellwig @ 2004-08-26 18:45 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 18:45 UTC (permalink / raw) To: Christoph Hellwig Cc: linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > > >How do you for example suggestion exporting your semantics over the >network if they're not done at the VFS level? How do you want some >clusterfilesystem support them or tmpfs? > > > > by accesses to filename/metas..... That's the beauty of simplicity, not much is needed. xattrs on the other hand would be a lot of work to integrate into a cluster filesystem. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 18:45 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 18:45 UTC (permalink / raw) To: Christoph Hellwig Cc: linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > > >How do you for example suggestion exporting your semantics over the >network if they're not done at the VFS level? How do you want some >clusterfilesystem support them or tmpfs? > > > > by accesses to filename/metas..... That's the beauty of simplicity, not much is needed. xattrs on the other hand would be a lot of work to integrate into a cluster filesystem. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:42 ` Hans Reiser (?) (?) @ 2004-08-26 13:38 ` Chris Mason 2004-08-26 23:53 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Chris Mason @ 2004-08-26 13:38 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, 2004-08-26 at 04:42, Hans Reiser wrote: > Christoph Hellwig wrote: > > > > >Over the last at least five years we've taken as much as possible > >semantics out of the filesystems and into the VFS layer, thus having > >a separation between the semantical layer (VFS) and the low level > >filesystem. > > > VFS is the common filesystem layer. The only reason you think semantics > belong in the common filesystem layer is that you are not innovating in > your semantics, and feel content with stasis. This misunderstanding is the heart of the problem really. In many ways, the VFS is the semantic layer, it defines the picture of the directory tree and the locking rules the applications see. There's no rule that says you can't innovate in the VFS, in fact it is where the semantic innovations belong. Putting them there is what allows other filesystems to implement your semantics, and this is exactly what linux needs if these new semantics are going to be standardized. In other words, writing a reiser4 plugin is one way to get to the new semantics, but it must not be the only way. Keeping them entirely within reiser4 is good for the research stage, but for actual deployment they need to be finalized and pushed up. If the early linux filesystems had taken the same attitude you have (don't write new filesystems, only write plugins), there would be no framework allowing the wealth of filesystems we do have, including reiser4. -chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:38 ` Chris Mason @ 2004-08-26 23:53 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:53 UTC (permalink / raw) To: Chris Mason Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Chris Mason wrote: > > >If the early linux filesystems had taken the same attitude you have >(don't write new filesystems, only write plugins), there would be no >framework allowing the wealth of filesystems we do have, including >reiser4. > > > Au contraire, with my approach, there would be 10x as many filesystems, because people would be easily able to mix-and-match different methods and plugins from different filesystems together, and if you wanted the MySQL semantics, you would mount mysql, and if you wanted the zope semantics you would mount zope. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:53 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:53 UTC (permalink / raw) To: Chris Mason Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Chris Mason wrote: > > >If the early linux filesystems had taken the same attitude you have >(don't write new filesystems, only write plugins), there would be no >framework allowing the wealth of filesystems we do have, including >reiser4. > > > Au contraire, with my approach, there would be 10x as many filesystems, because people would be easily able to mix-and-match different methods and plugins from different filesystems together, and if you wanted the MySQL semantics, you would mount mysql, and if you wanted the zope semantics you would mount zope. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:53 ` Hans Reiser (?) @ 2004-08-27 9:16 ` Christoph Hellwig -1 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-27 9:16 UTC (permalink / raw) To: Hans Reiser Cc: Chris Mason, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 04:53:51PM -0700, Hans Reiser wrote: > >If the early linux filesystems had taken the same attitude you have > >(don't write new filesystems, only write plugins), there would be no > >framework allowing the wealth of filesystems we do have, including > >reiser4. > > > > > > > Au contraire, with my approach, there would be 10x as many filesystems, > because people would be easily able to mix-and-match different methods > and plugins from different filesystems together, and if you wanted the > MySQL semantics, you would mount mysql, and if you wanted the zope > semantics you would mount zope. And if you wanted to make free love you'd make free love you'd make free love and there would be piece on all earth.. You seem to have lost contact to reality a little. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:08 ` Christoph Hellwig 2004-08-25 20:19 ` Christoph Hellwig @ 2004-08-25 20:22 ` Linus Torvalds 2004-08-25 20:35 ` Christoph Hellwig ` (7 more replies) 2004-08-25 20:35 ` Alex Zarochentsev 2 siblings, 8 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-25 20:22 UTC (permalink / raw) To: Christoph Hellwig Cc: Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > For one thing _I_ didn't decide about xattrs anyway. And I still > haven't seen a design from you on -fsdevel how you try to solve the > problems with files as directories. Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes. It's the UNIX way. And yes, the semantics can _easily_ be solved in very unixy ways. One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is: - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) Problem solved. Very user-friendly, and very intuitive. Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless. The argument that applications like "tar" won't understand the file-as-directory thing is _flawed_, since legacy apps won't understand xattrs either. Oh, add a O_NOXATTRS flag to force a path lookup to only use regular directories, the same way we have O_NOFOLLOW and friends. That allows people to see the difference, if they care (ie a file server might decide that it doesn't want to expose things like this). I never liked the xattr stuff. It makes little sense, and is totally useless for 99.9999% of everything. I still don't see the point of it, except for samba. Ugly. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds @ 2004-08-25 20:35 ` Christoph Hellwig 2004-08-25 20:41 ` Hans Reiser ` (6 subsequent siblings) 7 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:35 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > And yes, the semantics can _easily_ be solved in very unixy ways. > > One way to solve it is to just realize that a final slash at the end > implies pretty strongly that you want to treat it as a directory. So what > you do is: > > - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) > - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) > > Problem solved. Very user-friendly, and very intuitive. That would solve the O_DIRECTORY issue, the dentry aliasing still needs work though with the semantics for link/unlink/rename. Maybe Hans & you should start 2.7 to work this out? :) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds @ 2004-08-25 20:41 ` Hans Reiser 2004-08-25 20:41 ` Hans Reiser ` (6 subsequent siblings) 7 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-25 20:41 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov I just want to add that I AM capable of working with the other filesystem developers in a team-player way, and I am happy to cooperate with making portions more reusable where there is serious interest from other filesystems in that, but Christoph is a puppy who has never written or designed a major filesystem from scratch, and Nikita is a big dog who has written stuff very few projects are lucky enough to see the likes of, and when Christoph insults Nikita's code, or my design guidance for that code, it is not going to bring out my good side. The plugin and metafiles code needs many improvements, but Christoph does not have the expertise to understand what those needed improvements are because he hasn't invested the work into understanding the code. Christoph is a bright and clever young fellow, who just hasn't had the years of study of the field yet. I wish him well, and away.;-) Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-25 20:41 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-25 20:41 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov I just want to add that I AM capable of working with the other filesystem developers in a team-player way, and I am happy to cooperate with making portions more reusable where there is serious interest from other filesystems in that, but Christoph is a puppy who has never written or designed a major filesystem from scratch, and Nikita is a big dog who has written stuff very few projects are lucky enough to see the likes of, and when Christoph insults Nikita's code, or my design guidance for that code, it is not going to bring out my good side. The plugin and metafiles code needs many improvements, but Christoph does not have the expertise to understand what those needed improvements are because he hasn't invested the work into understanding the code. Christoph is a bright and clever young fellow, who just hasn't had the years of study of the field yet. I wish him well, and away.;-) Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:41 ` Hans Reiser (?) @ 2004-08-25 20:51 ` Chris Mason -1 siblings, 0 replies; 1835+ messages in thread From: Chris Mason @ 2004-08-25 20:51 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov On Wed, 2004-08-25 at 16:41, Hans Reiser wrote: > I just want to add that I AM capable of working with the other > filesystem developers in a team-player way, and I am happy to cooperate > with making portions more reusable where there is serious interest from > other filesystems in that, Prove it. Stop replying for today and come back tomorrow with some useful discussions. Christoph suggested that some of the v4 semantics belong in the VFS and therefore linux as a whole. He's helping you to make sure the semantics and fit nicely with the rest of kernel interfaces and are race free. Take him up on the offer. -chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:41 ` Hans Reiser (?) @ 2004-08-25 20:58 ` Markus Törnqvist -1 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-25 20:58 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov On Wed, Aug 25, 2004 at 01:41:14PM -0700, Hans Reiser wrote: >Christoph is a bright and clever young fellow, who just hasn't had the >years of study of the field yet. I wish him well, and away.;-) I see this is as an opportunity where you can share some of your experience to Cristoph and many others and work to get the semantics into VFS. Please make this work :) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-25 20:58 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: mjt @ 2004-08-25 20:58 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov On Wed, Aug 25, 2004 at 01:41:14PM -0700, Hans Reiser wrote: >Christoph is a bright and clever young fellow, who just hasn't had the >years of study of the field yet. I wish him well, and away.;-) I see this is as an opportunity where you can share some of your experience to Cristoph and many others and work to get the semantics into VFS. Please make this work :) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-25 20:58 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-25 20:58 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov On Wed, Aug 25, 2004 at 01:41:14PM -0700, Hans Reiser wrote: >Christoph is a bright and clever young fellow, who just hasn't had the >years of study of the field yet. I wish him well, and away.;-) I see this is as an opportunity where you can share some of your experience to Cristoph and many others and work to get the semantics into VFS. Please make this work :) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:41 ` Hans Reiser @ 2004-08-25 21:03 ` Rik van Riel -1 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-25 21:03 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov On Wed, 25 Aug 2004, Hans Reiser wrote: > other filesystems in that, but Christoph is a puppy who has never It's not Christoph who's shown more bark than bite in this thread. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-25 21:03 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-25 21:03 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov On Wed, 25 Aug 2004, Hans Reiser wrote: > other filesystems in that, but Christoph is a puppy who has never It's not Christoph who's shown more bark than bite in this thread. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:03 ` Rik van Riel @ 2004-08-26 9:00 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 9:00 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov Rik van Riel wrote: >On Wed, 25 Aug 2004, Hans Reiser wrote: > > > >>other filesystems in that, but Christoph is a puppy who has never >> >> > >It's not Christoph who's shown more bark than bite in this thread. > > > The bite is at www.namesys.com/download.html Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 9:00 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 9:00 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov Rik van Riel wrote: >On Wed, 25 Aug 2004, Hans Reiser wrote: > > > >>other filesystems in that, but Christoph is a puppy who has never >> >> > >It's not Christoph who's shown more bark than bite in this thread. > > > The bite is at www.namesys.com/download.html Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds 2004-08-25 20:35 ` Christoph Hellwig 2004-08-25 20:41 ` Hans Reiser @ 2004-08-25 20:42 ` viro 2004-08-25 21:00 ` Christophe Saout ` (2 more replies) 2004-08-25 21:52 ` Matt Mackall ` (4 subsequent siblings) 7 siblings, 3 replies; 1835+ messages in thread From: viro @ 2004-08-25 20:42 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > > > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > haven't seen a design from you on -fsdevel how you try to solve the > > problems with files as directories. > > Hey, files-as-directories are one of my pet things, so I have to side with > Hans on this one. I think it just makes sense. A hell of a lot more sense > than xattrs, anyway, since it allows scripts etc standard tools to touch > the attributes. > > It's the UNIX way. Not if you allow link(2) on them. And not if you design and market your stuff as a general-purpose backdoor into kernel. Note how *EVERY* *DAMN* *OPERATION* is made possible to override by "plugins". Which is the reason for deadlocks in question, BTW. Don't fool yourself - that's what Hans is selling. Target market: ISV. Marketed product: a set of hooks, the wider the better, no matter how little sense it makes. The reason for doing that outside of core kernel: bypassing any review and being able to control the product being sold (see above). Shame that it got an actual filesystem mixed in with the marketing plans and general insanity... ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:42 ` viro @ 2004-08-25 21:00 ` Christophe Saout 2004-08-25 21:00 ` Linus Torvalds 2004-08-26 8:43 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-25 21:00 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 2418 bytes --] Am Mittwoch, den 25.08.2004, 21:42 +0100 schrieb : > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > > haven't seen a design from you on -fsdevel how you try to solve the > > > problems with files as directories. > > > > Hey, files-as-directories are one of my pet things, so I have to side with > > Hans on this one. I think it just makes sense. A hell of a lot more sense > > than xattrs, anyway, since it allows scripts etc standard tools to touch > > the attributes. > > > > It's the UNIX way. > > Not if you allow link(2) on them. That doesn't make sense anyway. (actually, I tried what happens and the result was an Oops ;)) It should be completely forbidden to link into a meta-directory or out of such a directory. You could think of those meta-directory as a sysfs for that inode. Of course it's not an own filesystem and that means that there need to be a lot of security precautions in the VFS layer. Where something like that belongs anyway, if done correctly. > And not if you design and market your > stuff as a general-purpose backdoor into kernel. Note how *EVERY* *DAMN* > *OPERATION* is made possible to override by "plugins". Which is the reason > for deadlocks in question, BTW. What do you mean? If you tell that file that you want it to be compressed or encrypted or modify some attributes (like ACLs) this isn't necessarily a backdoor. > Don't fool yourself - that's what Hans is selling. Target market: ISV. > Marketed product: a set of hooks, the wider the better, no matter how > little sense it makes. The reason for doing that outside of core kernel: > bypassing any review and being able to control the product being sold (see > above). Yes, I don't think it was a good idea either. Probably someone should remove these features and make it a "normal" filesystem. The people who need it now can turn it on again and a real solution could be worked out in Linux 2.7. I wouldn't use it on a public server anyway now because I'm not convinced some malicious guy could find a way to exploit that. What if you changed into a meta directory using ftp and some manage to break things? This might be very dangerous. I personally think that the idea of doing something like this (I'm not speaking of the current implementation which I think is really bad) is the right way to go in the long term. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-25 21:00 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-25 21:00 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 2418 bytes --] Am Mittwoch, den 25.08.2004, 21:42 +0100 schrieb : > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > > haven't seen a design from you on -fsdevel how you try to solve the > > > problems with files as directories. > > > > Hey, files-as-directories are one of my pet things, so I have to side with > > Hans on this one. I think it just makes sense. A hell of a lot more sense > > than xattrs, anyway, since it allows scripts etc standard tools to touch > > the attributes. > > > > It's the UNIX way. > > Not if you allow link(2) on them. That doesn't make sense anyway. (actually, I tried what happens and the result was an Oops ;)) It should be completely forbidden to link into a meta-directory or out of such a directory. You could think of those meta-directory as a sysfs for that inode. Of course it's not an own filesystem and that means that there need to be a lot of security precautions in the VFS layer. Where something like that belongs anyway, if done correctly. > And not if you design and market your > stuff as a general-purpose backdoor into kernel. Note how *EVERY* *DAMN* > *OPERATION* is made possible to override by "plugins". Which is the reason > for deadlocks in question, BTW. What do you mean? If you tell that file that you want it to be compressed or encrypted or modify some attributes (like ACLs) this isn't necessarily a backdoor. > Don't fool yourself - that's what Hans is selling. Target market: ISV. > Marketed product: a set of hooks, the wider the better, no matter how > little sense it makes. The reason for doing that outside of core kernel: > bypassing any review and being able to control the product being sold (see > above). Yes, I don't think it was a good idea either. Probably someone should remove these features and make it a "normal" filesystem. The people who need it now can turn it on again and a real solution could be worked out in Linux 2.7. I wouldn't use it on a public server anyway now because I'm not convinced some malicious guy could find a way to exploit that. What if you changed into a meta directory using ftp and some manage to break things? This might be very dangerous. I personally think that the idea of doing something like this (I'm not speaking of the current implementation which I think is really bad) is the right way to go in the long term. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:00 ` Christophe Saout (?) @ 2004-08-25 22:59 ` Andrea Arcangeli 2004-08-26 8:35 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Andrea Arcangeli @ 2004-08-25 22:59 UTC (permalink / raw) To: Christophe Saout Cc: viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote: > It should be completely forbidden to link into a meta-directory or out > of such a directory. [..] agreed. > Yes, I don't think it was a good idea either. Probably someone should I personally would like plugins only if the API they use wouldn't allow them to corrupt the underlying fs, I'm not sure if this is the case with reiserfs4. About the backwards compatibility, another option is to add a O_FILEDIR and have bash learn about it when you cd into a file. No magic with the slashes then. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:59 ` Andrea Arcangeli @ 2004-08-26 8:35 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:35 UTC (permalink / raw) To: Andrea Arcangeli Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote: > > >>It should be completely forbidden to link into a meta-directory or out >>of such a directory. [..] >> >> > >agreed. > > > >>Yes, I don't think it was a good idea either. Probably someone should >> >> > >I personally would like plugins only if the API they use wouldn't allow >them to corrupt the underlying fs, I'm not sure if this is the case with >reiserfs4. > > You compile crap into the kernel, you are screwed. We have not and will not change that. Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes being less ambitious works better, and dynamically loaded plugins would have made the infrastructure too bulky to be fun and fast. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:35 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:35 UTC (permalink / raw) To: Andrea Arcangeli Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote: > > >>It should be completely forbidden to link into a meta-directory or out >>of such a directory. [..] >> >> > >agreed. > > > >>Yes, I don't think it was a good idea either. Probably someone should >> >> > >I personally would like plugins only if the API they use wouldn't allow >them to corrupt the underlying fs, I'm not sure if this is the case with >reiserfs4. > > You compile crap into the kernel, you are screwed. We have not and will not change that. Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes being less ambitious works better, and dynamically loaded plugins would have made the infrastructure too bulky to be fun and fast. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:35 ` Hans Reiser (?) @ 2004-08-26 11:28 ` Andrea Arcangeli 2004-08-26 11:44 ` Christophe Saout ` (3 more replies) -1 siblings, 4 replies; 1835+ messages in thread From: Andrea Arcangeli @ 2004-08-26 11:28 UTC (permalink / raw) To: Hans Reiser Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > Reiser4 plugins are not for end users to download from amazon.com, they > are for weekend hackers to send me a cool plugin for me to review, > assign a plugin id to, and send to Linus in the next release. Sometimes then what's the difference in having the plugin fixed in stone into reiserfs? That's my whole point. Get the patch from the weekend hacker, check it, send the patch to Linus to add the new feature to reiser4, just call it "feature" not plugin. That's how it works normally for everything. Many fs have many features, many of them optional. you wouldn't need to build any hook infrastructure either that way. Hooks would be needed if this wasn't open source me thinks. Or if you want people to fetch the module from amazon without your review. Another reason I could see the modularization/hooking useful is if those feature would take lots of kernel space, but this sure isn't the case for reiserfs, infact having modules would waste _more_ ram due the half wasted page allocation for the module text. The only single reason we use modules is to avoid wasting tons of ram by loading every possible device driver on earth, it's not that we use modules because they're more flexible, if something they're more fragile. I never use modules in my test kernel just to go fast (because I self compile them). The last good thing of the modules is during development you don't need to reboot the machine to test a kernel change in a driver, but you don't need that with reiserfs since you can make the fs itself a module (just change the name of the fs, AFIK you do that all the time already). ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:28 ` Andrea Arcangeli @ 2004-08-26 11:44 ` Christophe Saout 2004-08-26 23:53 ` Hans Reiser ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 11:44 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 640 bytes --] Am Donnerstag, den 26.08.2004, 13:28 +0200 schrieb Andrea Arcangeli: > then what's the difference in having the plugin fixed in stone into > reiserfs? That's my whole point. I think you're not exactly understanding what a reiser4 "plugin" is. Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins". In fact, reiser4 plugins are - users of the reiser4 core API - some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS) [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 11:44 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 11:44 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 640 bytes --] Am Donnerstag, den 26.08.2004, 13:28 +0200 schrieb Andrea Arcangeli: > then what's the difference in having the plugin fixed in stone into > reiserfs? That's my whole point. I think you're not exactly understanding what a reiser4 "plugin" is. Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins". In fact, reiser4 plugins are - users of the reiser4 core API - some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS) [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:44 ` Christophe Saout (?) @ 2004-08-26 12:16 ` Andrea Arcangeli 2004-08-26 12:23 ` Christophe Saout 2004-08-27 18:38 ` Alex Zarochentsev -1 siblings, 2 replies; 1835+ messages in thread From: Andrea Arcangeli @ 2004-08-26 12:16 UTC (permalink / raw) To: Christophe Saout Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:44:59PM +0200, Christophe Saout wrote: > I think you're not exactly understanding what a reiser4 "plugin" is. > > Currently plugins are not modules. It just means that there's a defined > interface between the reiser4 core and the plugins. Just like you could > see filesystems as "VFS plugins". but filesystems are exactly always modules (in precompiled kernels at least). Anyways I don't see any visible EXPORT_SYMBOL in reiser4. > > In fact, reiser4 plugins are > - users of the reiser4 core API > - some of them are implementing Linux VFS methods (thus being some sort > of glue code between the Reiser4 storage tree and the Linux VFS) then it seems a bit misleading to call those things plugins. As you wrote those are "users of the reiser4 core API", with plugins I always meant dynamically loadable things, like plugins for web browers (hence kernel modules in kernel space). While here apparently you can't plugin anything at runtime, it's just code that uses the reiser4 core API that can be modified with a kernel patch as usual. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:16 ` Andrea Arcangeli @ 2004-08-26 12:23 ` Christophe Saout 2004-08-27 18:38 ` Alex Zarochentsev 1 sibling, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 12:23 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1469 bytes --] Am Donnerstag, den 26.08.2004, 14:16 +0200 schrieb Andrea Arcangeli: > > Currently plugins are not modules. It just means that there's a defined > > interface between the reiser4 core and the plugins. Just like you could > > see filesystems as "VFS plugins". > > but filesystems are exactly always modules (in precompiled kernels at > least). Yes, of cousre. > Anyways I don't see any visible EXPORT_SYMBOL in reiser4. Yes, there's no module infrastructure there. These "plugins" need to be linked at compile time. Some plugins need to be anyway since the the filesystem is useless without them. Others whould be optional features. > > In fact, reiser4 plugins are > > - users of the reiser4 core API > > - some of them are implementing Linux VFS methods (thus being some sort > > of glue code between the Reiser4 storage tree and the Linux VFS) > > then it seems a bit misleading to call those things plugins. > > As you wrote those are "users of the reiser4 core API", with plugins I > always meant dynamically loadable things, like plugins for web browers > (hence kernel modules in kernel space). While here apparently you can't > plugin anything at runtime, it's just code that uses the reiser4 core > API that can be modified with a kernel patch as usual. Right. The name is probably at bit misleading. But assuming the compression "plugin" whould be dynamically loadable at some later point it would be a plugin then? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 12:23 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 12:23 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1469 bytes --] Am Donnerstag, den 26.08.2004, 14:16 +0200 schrieb Andrea Arcangeli: > > Currently plugins are not modules. It just means that there's a defined > > interface between the reiser4 core and the plugins. Just like you could > > see filesystems as "VFS plugins". > > but filesystems are exactly always modules (in precompiled kernels at > least). Yes, of cousre. > Anyways I don't see any visible EXPORT_SYMBOL in reiser4. Yes, there's no module infrastructure there. These "plugins" need to be linked at compile time. Some plugins need to be anyway since the the filesystem is useless without them. Others whould be optional features. > > In fact, reiser4 plugins are > > - users of the reiser4 core API > > - some of them are implementing Linux VFS methods (thus being some sort > > of glue code between the Reiser4 storage tree and the Linux VFS) > > then it seems a bit misleading to call those things plugins. > > As you wrote those are "users of the reiser4 core API", with plugins I > always meant dynamically loadable things, like plugins for web browers > (hence kernel modules in kernel space). While here apparently you can't > plugin anything at runtime, it's just code that uses the reiser4 core > API that can be modified with a kernel patch as usual. Right. The name is probably at bit misleading. But assuming the compression "plugin" whould be dynamically loadable at some later point it would be a plugin then? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:16 ` Andrea Arcangeli 2004-08-26 12:23 ` Christophe Saout @ 2004-08-27 18:38 ` Alex Zarochentsev 2004-08-28 6:17 ` Hans Reiser 1 sibling, 1 reply; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-27 18:38 UTC (permalink / raw) To: Andrea Arcangeli Cc: Christophe Saout, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 02:16:30PM +0200, Andrea Arcangeli wrote: > On Thu, Aug 26, 2004 at 01:44:59PM +0200, Christophe Saout wrote: > > I think you're not exactly understanding what a reiser4 "plugin" is. > > > > Currently plugins are not modules. It just means that there's a defined > > interface between the reiser4 core and the plugins. Just like you could > > see filesystems as "VFS plugins". > > but filesystems are exactly always modules (in precompiled kernels at > least). > > Anyways I don't see any visible EXPORT_SYMBOL in reiser4. they would be there, I think. > > > > > In fact, reiser4 plugins are > > - users of the reiser4 core API > > - some of them are implementing Linux VFS methods (thus being some sort > > of glue code between the Reiser4 storage tree and the Linux VFS) > > then it seems a bit misleading to call those things plugins. There should be infracture to change plugin on fly, and choosing object's plugins at object's creation time. Currently only certain object's plugins can be changed through <object>/metas/plugin interface. > > As you wrote those are "users of the reiser4 core API", with plugins I > always meant dynamically loadable things, like plugins for web browers > (hence kernel modules in kernel space). While here apparently you can't > plugin anything at runtime, it's just code that uses the reiser4 core > API that can be modified with a kernel patch as usual. -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:38 ` Alex Zarochentsev @ 2004-08-28 6:17 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 6:17 UTC (permalink / raw) To: Alex Zarochentsev Cc: Andrea Arcangeli, Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Alex Zarochentsev wrote: > > >There should be infracture to change plugin on fly, and choosing object's >plugins at object's creation time. Currently only certain object's plugins >can be changed through <object>/metas/plugin interface. > > To be more clear for those not familiar with our project, there is much more work to be done, and one of the areas that (extremely) needs more work is changing plugins on the fly and clean interfaces for specifying them at creation time. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:28 ` Andrea Arcangeli @ 2004-08-26 23:53 ` Hans Reiser 2004-08-26 23:53 ` Hans Reiser ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:53 UTC (permalink / raw) To: Andrea Arcangeli Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > > >>Reiser4 plugins are not for end users to download from amazon.com, they >>are for weekend hackers to send me a cool plugin for me to review, >>assign a plugin id to, and send to Linus in the next release. Sometimes >> >> > >then what's the difference in having the plugin fixed in stone into >reiserfs? That's my whole point. Get the patch from the weekend hacker, >check it, send the patch to Linus to add the new feature to reiser4, >just call it "feature" not plugin. > Think of it as being like VFS, where you can plugin new filesystems. Only in this case, you can plugin new kinds of files, and everything you need to implement those new kinds of files (items, nodes, keys, processing before flushing to disk, etc.) also gets implemented as a plugin. It is an Uber-VFS. > The only single reason we >use modules is to avoid wasting tons of ram by loading every possible >device driver on earth, > I am not concerned about ram in this design, I want nifty new kinds of files easy crafted over a weekend by sysadmins working in Canada and Guatemala. Software engineering cost is what matters to me ( I turned 40, so I think different now....;-) ) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:53 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:53 UTC (permalink / raw) To: Andrea Arcangeli Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > > >>Reiser4 plugins are not for end users to download from amazon.com, they >>are for weekend hackers to send me a cool plugin for me to review, >>assign a plugin id to, and send to Linus in the next release. Sometimes >> >> > >then what's the difference in having the plugin fixed in stone into >reiserfs? That's my whole point. Get the patch from the weekend hacker, >check it, send the patch to Linus to add the new feature to reiser4, >just call it "feature" not plugin. > Think of it as being like VFS, where you can plugin new filesystems. Only in this case, you can plugin new kinds of files, and everything you need to implement those new kinds of files (items, nodes, keys, processing before flushing to disk, etc.) also gets implemented as a plugin. It is an Uber-VFS. > The only single reason we >use modules is to avoid wasting tons of ram by loading every possible >device driver on earth, > I am not concerned about ram in this design, I want nifty new kinds of files easy crafted over a weekend by sysadmins working in Canada and Guatemala. Software engineering cost is what matters to me ( I turned 40, so I think different now....;-) ) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:53 ` Hans Reiser (?) @ 2004-08-27 0:18 ` Andrea Arcangeli -1 siblings, 0 replies; 1835+ messages in thread From: Andrea Arcangeli @ 2004-08-27 0:18 UTC (permalink / raw) To: Hans Reiser Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:53:23PM -0700, Hans Reiser wrote: > Think of it as being like VFS, where you can plugin new filesystems. > Only in this case, you can plugin new kinds of files, and everything you > need to implement those new kinds of files (items, nodes, keys, > processing before flushing to disk, etc.) also gets implemented as a > plugin. It is an Uber-VFS. I'm just confused by the naming, following your comparison with it being like VFS, I don't call reiser4 a "plugin" of the VFS. The vfs offers an interface for the fs to register into it. Like the scsi highlevel code also offers an interface for the lowlevel driver to register into it, but we don't call scsi lowlevel drivers "plugins". the ext3 directory code calling ext3_getblk would be also a plugin in your vocabulary, and the ext3_getblk call would be plugin-hook. > I am not concerned about ram in this design, I want nifty new kinds of > files easy crafted over a weekend by sysadmins working in Canada and > Guatemala. Software engineering cost is what matters to me ( I turned > 40, so I think different now....;-) ) Adding proper layering and abstractions is sure a good thing, if you can turn the street into a crowd of reiser4 kernel hackers because your API is so clean that any sysadmin can craft new kind of files over the weekend you're welcome to go ahead ;). (I just doubt the "plugin" terminology can make a difference by the time people start scratching their head into the code, the design and the clean abstraction of the code will make a difference instead) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:28 ` Andrea Arcangeli 2004-08-26 11:44 ` Christophe Saout 2004-08-26 23:53 ` Hans Reiser @ 2004-08-28 10:49 ` Yury Umanets 2004-08-28 11:11 ` Yury Umanets 3 siblings, 0 replies; 1835+ messages in thread From: Yury Umanets @ 2004-08-28 10:49 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > > >>Reiser4 plugins are not for end users to download from amazon.com, they >>are for weekend hackers to send me a cool plugin for me to review, >>assign a plugin id to, and send to Linus in the next release. Sometimes >> >> > >then what's the difference in having the plugin fixed in stone into >reiserfs? That's my whole point. Get the patch from the weekend hacker, >check it, send the patch to Linus to add the new feature to reiser4, >just call it "feature" not plugin. That's how it works normally for >everything. Many fs have many features, many of them optional. you >wouldn't need to build any hook infrastructure either that way. Hooks >would be needed if this wasn't open source me thinks. Or if you want >people to fetch the module from amazon without your review. > >Another reason I could see the modularization/hooking useful is if those >feature would take lots of kernel space, but this sure isn't the case >for reiserfs, infact having modules would waste _more_ ram due the half >wasted page allocation for the module text. The only single reason we >use modules is to avoid wasting tons of ram by loading every possible >device driver on earth, it's not that we use modules because they're >more flexible, if something they're more fragile. I never use modules in >my test kernel just to go fast (because I self compile them). The last >good thing of the modules is during development you don't need to reboot >the machine to test a kernel change in a driver, but you don't need that >with reiserfs since you can make the fs itself a module (just change the >name of the fs, AFIK you do that all the time already). > > There lots of good things about plugins as they are in reiser4. And probably the best one (as for me) is that, reiser4 core code (quite complex thing) does not need to be changed after adding new plugins (read new features). It is written once and provides some basic functionality (balancing, atoms, etc.), which works well with all kinds of plugins if they implement required functions from plugin interface. As for me, this looks like Linux's VFS. Sure, if I want to add some very new filesystem with features which were not foreseen, and VFS does not provide needed interface, it will need some changes. But in simple case, when I just want to develop once more disk filesystem, which say will be different from another disk filesystem in only that respect it manages its metadata in different manner and aims to achieve better performance this way. In this case I will not need to chnage VFS at all. The same about reiser4 core and new plugins. >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > -- umka ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:28 ` Andrea Arcangeli @ 2004-08-28 11:11 ` Yury Umanets 2004-08-26 23:53 ` Hans Reiser ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: Yury Umanets @ 2004-08-28 11:11 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > > >>Reiser4 plugins are not for end users to download from amazon.com, they >>are for weekend hackers to send me a cool plugin for me to review, >>assign a plugin id to, and send to Linus in the next release. Sometimes >> >> > >then what's the difference in having the plugin fixed in stone into >reiserfs? That's my whole point. Get the patch from the weekend hacker, >check it, send the patch to Linus to add the new feature to reiser4, >just call it "feature" not plugin. That's how it works normally for >everything. Many fs have many features, many of them optional. you >wouldn't need to build any hook infrastructure either that way. Hooks >would be needed if this wasn't open source me thinks. Or if you want >people to fetch the module from amazon without your review. > >Another reason I could see the modularization/hooking useful is if those >feature would take lots of kernel space, but this sure isn't the case >for reiserfs, infact having modules would waste _more_ ram due the half >wasted page allocation for the module text. The only single reason we >use modules is to avoid wasting tons of ram by loading every possible >device driver on earth, it's not that we use modules because they're >more flexible, if something they're more fragile. I never use modules in >my test kernel just to go fast (because I self compile them). The last >good thing of the modules is during development you don't need to reboot >the machine to test a kernel change in a driver, but you don't need that >with reiserfs since you can make the fs itself a module (just change the >name of the fs, AFIK you do that all the time already). > > There lots of good things about plugins as they are in reiser4. And probably the best one (as for me) is that, reiser4 core code (quite complex thing) does not need to be changed after adding new plugins (read new features). It is written once and provides some basic functionality (balancing, atoms, etc.), which works well with all kinds of plugins if they implement required functions from plugin interface. As for me, this looks like Linux's VFS. Sure, if I want to add some very new filesystem with features which were not foreseen, and VFS does not provide needed interface, it will need some changes. But in simple case, when I just want to develop once more disk filesystem, which say will be different from another disk filesystem in only that respect it manages its metadata in different manner and aims to achieve better performance this way. In this case I will not need to chnage VFS at all. The same about reiser4 core and new plugins. >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > -- umka ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 11:11 ` Yury Umanets 0 siblings, 0 replies; 1835+ messages in thread From: Yury Umanets @ 2004-08-28 11:11 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > > >>Reiser4 plugins are not for end users to download from amazon.com, they >>are for weekend hackers to send me a cool plugin for me to review, >>assign a plugin id to, and send to Linus in the next release. Sometimes >> >> > >then what's the difference in having the plugin fixed in stone into >reiserfs? That's my whole point. Get the patch from the weekend hacker, >check it, send the patch to Linus to add the new feature to reiser4, >just call it "feature" not plugin. That's how it works normally for >everything. Many fs have many features, many of them optional. you >wouldn't need to build any hook infrastructure either that way. Hooks >would be needed if this wasn't open source me thinks. Or if you want >people to fetch the module from amazon without your review. > >Another reason I could see the modularization/hooking useful is if those >feature would take lots of kernel space, but this sure isn't the case >for reiserfs, infact having modules would waste _more_ ram due the half >wasted page allocation for the module text. The only single reason we >use modules is to avoid wasting tons of ram by loading every possible >device driver on earth, it's not that we use modules because they're >more flexible, if something they're more fragile. I never use modules in >my test kernel just to go fast (because I self compile them). The last >good thing of the modules is during development you don't need to reboot >the machine to test a kernel change in a driver, but you don't need that >with reiserfs since you can make the fs itself a module (just change the >name of the fs, AFIK you do that all the time already). > > There lots of good things about plugins as they are in reiser4. And probably the best one (as for me) is that, reiser4 core code (quite complex thing) does not need to be changed after adding new plugins (read new features). It is written once and provides some basic functionality (balancing, atoms, etc.), which works well with all kinds of plugins if they implement required functions from plugin interface. As for me, this looks like Linux's VFS. Sure, if I want to add some very new filesystem with features which were not foreseen, and VFS does not provide needed interface, it will need some changes. But in simple case, when I just want to develop once more disk filesystem, which say will be different from another disk filesystem in only that respect it manages its metadata in different manner and aims to achieve better performance this way. In this case I will not need to chnage VFS at all. The same about reiser4 core and new plugins. >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > -- umka ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:42 ` viro 2004-08-25 21:00 ` Christophe Saout @ 2004-08-25 21:00 ` Linus Torvalds 2004-08-25 21:25 ` viro 2004-08-26 8:43 ` Hans Reiser 2 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-25 21:00 UTC (permalink / raw) To: viro Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > > > > It's the UNIX way. > > Not if you allow link(2) on them. Heh. I don't think that's a very strong argument against being "unixy", considering how traditional unix _used_ to handle directories. mkdir/rmdir/rename only came later. Now, obviously they did come later for a good reason, but still.. The interesting part is that thanks to the dcache, we should be perfectly able to actually _see_ circular links etc, so some of the problems with linking directories should actually be quite solvable - something that is _not_ true for a traditional UNIX VFS layer. Of course, the dcache introduces some new problems of its own wrt directory aliasing, but I don't actually think that should be fundamental either. Treating them more as a "static mountpoint" from an FS angle and less as a traditional Unix hardlink should be doable, I'd have thought. (Also, it's entirely possible that the filesystem may not support some of the more esoteric linking/renaming operations. For example, in a traditional xattrs setup where the xattr is linked on-disk with the file it is associated with, you simply _can't_ link it somewhere else, or rename it to any other directory. That's not a VFS layer issue, obviously, but I thought I'd bring up the point that file-as-dir cases may have limitations that normal files don't have). > And not if you design and market your stuff as a general-purpose > backdoor into kernel. Now that's a separate argument, and not one I'm personally interested in arguing at least right now. I haven't actually looked at the reiser4 code, so I'm really _only_ arguing against special-case attributes. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:00 ` Linus Torvalds @ 2004-08-25 21:25 ` viro 2004-08-26 0:11 ` Jamie Lokier 2004-08-29 9:55 ` Mikaël Cluseau 0 siblings, 2 replies; 1835+ messages in thread From: viro @ 2004-08-25 21:25 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 02:00:01PM -0700, Linus Torvalds wrote: > Of course, the dcache introduces some new problems of its own wrt > directory aliasing, but I don't actually think that should be fundamental > either. Treating them more as a "static mountpoint" from an FS angle and > less as a traditional Unix hardlink should be doable, I'd have thought. Yeah, if we ditch the "mountpoints are busy and untouchable" stuff. Which I'd love to, but it's a hell of a visible (and admin-visible) change. FWIW, current deadlocks are unrelated to actual operation succeeding. Look: we have sys_link() making sure that parent of target is a directory (PATH_LOOKUP, in a "it has ->lookup()" sense), then locking target's parent, then checking that it has ->link() (everyone on reiser4 does) and then checking that source (old link to file) is *not* a directory (in S_ISDIR sense). Then we lock source. Note that currently it's OK - we get "all non-directories are always locked after all directories". With filesystem that provides hybrid objects with non-NULL ->link() it's not true and we are in deadlock country. Before we get anywhere near fs code. I'm not saying that this particular instance is hard to fix, but it wasn't even looked at. All it would take is checking the description of current locking scheme and looking through the proof of correctness (present in the tree). That's the first point where said proof breaks if we have hybrids. And it's what, about 4 screenfuls of text? I have no problems with discussing such stuff and no problems with having it merged if it actually works. But let's start with something better than "let's hope nothing breaks if we just add such objects and do nothing else, 'cause hybridi files/directories are good, mmmkay?" ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:25 ` viro @ 2004-08-26 0:11 ` Jamie Lokier 2004-08-26 0:30 ` viro 2004-08-29 9:55 ` Mikaël Cluseau 1 sibling, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 0:11 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List This message suggests a way to extend the VFS safe locking rules to include files-as-directories. viro@parcelfarce.linux.theplanet.co.uk wrote: > Note that currently it's OK - we get "all non-directories are always locked > after all directories". With filesystem that provides hybrid objects with > non-NULL ->link() it's not true and we are in deadlock country. Before > we get anywhere near fs code. Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)? Simply doing a path walk would lock the file and then cross the mount point to a directory. A way to ensure that preserves the lock order is to require that the metadata is in a different filesystem to its file (i.e. not crossing a bind mount to the same filesystem). That has the side effect of preventing hard links between metadata files and non-metadata, which in my opinion is fine. Path walking will lock the file, and then lock the directory on a different filesystem. Lock order is still safe, provided a strict order is maintained between the two filesystems. The strict order is ensured by preventing bind mounts which create a path cycle containing a file->metadata edge. One way to ensure that is to prevent mounts on the metadata filesystems, but the rule doesn't have to be that strict. This condition only needs to be checked in the mount() syscall. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:11 ` Jamie Lokier @ 2004-08-26 0:30 ` viro 2004-08-26 1:00 ` Jamie Lokier 2004-08-26 1:13 ` Chris Wright 0 siblings, 2 replies; 1835+ messages in thread From: viro @ 2004-08-26 0:30 UTC (permalink / raw) To: Jamie Lokier Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:11:52AM +0100, Jamie Lokier wrote: > Is this a problem if we treat entering a file-as-directory as crossing > a mount point (i.e. like auto-mounting)? Yes - mountpoints can't be e.g. unlinked. Moreover, having directory mounted on non-directory is also an interesting situation. > Simply doing a path walk would lock the file and then cross the mount > point to a directory. *Ugh* What would happen if you open that directory or chdir there? If it's "underlying file stays locked" - we are in even more obvious deadlocks. > A way to ensure that preserves the lock order is to require that the > metadata is in a different filesystem to its file (i.e. not crossing a > bind mount to the same filesystem). > > That has the side effect of preventing hard links between metadata > files and non-metadata, which in my opinion is fine. We don't actually need a different fs - different vfsmount will do just fine. > The strict order is ensured by preventing bind mounts which create a > path cycle containing a file->metadata edge. One way to ensure that > is to prevent mounts on the metadata filesystems, but the rule doesn't > have to be that strict. This condition only needs to be checked in > the mount() syscall. You really don't want to lock mountpoint on path lookup, so I don't see how that would be relevant - it's a hell to clean up, for one thing (I've crossed ten mountpoints on the way, when do I unlock them and how do I prevent deadlocks from that?) Besides, different namespaces can have completely different mount trees, so tracking down all that stuff would be hell in its own right. The main issue I see with all schemes in that direction (and something like that could be made workable) is the semantics of unlink() on mountpoints. *Especially* with users being able to see attributes of files they do not own (e.g. reiser4 mode/uid/gid stuff). Ability to pin down any damn file on the system and make it impossible to replace is not something you want to give to any user. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:30 ` viro @ 2004-08-26 1:00 ` Jamie Lokier 2004-08-26 3:13 ` viro ` (2 more replies) 2004-08-26 1:13 ` Chris Wright 1 sibling, 3 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 1:00 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List viro@parcelfarce.linux.theplanet.co.uk wrote: > > Is this a problem if we treat entering a file-as-directory as crossing > > a mount point (i.e. like auto-mounting)? > > Yes - mountpoints can't be e.g. unlinked. Moreover, having directory > mounted on non-directory is also an interesting situation. Ok, so can we make it so mountpoints can be unlinked? :) The mount would continue to exist, but with no name, until its last user disappears. > > Simply doing a path walk would lock the file and then cross the mount > > point to a directory. > > *Ugh* > > What would happen if you open that directory or chdir there? If it's > "underlying file stays locked" - we are in even more obvious deadlocks. I think the underlying file does not stay locked, and once you've entered it as a directory, it can be unlinked. If you have the directory open or chdir into it, then it _may_ have the effect of keeping the file's storage allocated when you unlink it -- just like when a file is unlinked while opened. As that is not a user-visible property, it's a filesystem-specific implementation detail as to whether it keeps the file's data in existence while the metadata directories and/or files are open. > > A way to ensure that preserves the lock order is to require that the > > metadata is in a different filesystem to its file (i.e. not crossing a > > bind mount to the same filesystem). > > > > That has the side effect of preventing hard links between metadata > > files and non-metadata, which in my opinion is fine. > > We don't actually need a different fs - different vfsmount will do just fine. > > The strict order is ensured by preventing bind mounts which create a > > path cycle containing a file->metadata edge. One way to ensure that > > is to prevent mounts on the metadata filesystems, but the rule doesn't > > have to be that strict. This condition only needs to be checked in > > the mount() syscall. > > You really don't want to lock mountpoint on path lookup, so I don't see > how that would be relevant - it's a hell to clean up, for one thing > (I've crossed ten mountpoints on the way, when do I unlock them and > how do I prevent deadlocks from that?) Besides, different namespaces > can have completely different mount trees, so tracking down all that > stuff would be hell in its own right. I didn't mean locking a chain of mountpoints, I meant the temporary state where two dentries and/or inodes are locked, parent and child, during a path walk. However I'm not very familiar with that part of the VFS and I see that the current RCU dcache might not lock that much during a path walk. > The main issue I see with all schemes in that direction (and something > like that could be made workable) is the semantics of unlink() on > mountpoints. *Especially* with users being able to see attributes of > files they do not own (e.g. reiser4 mode/uid/gid stuff). Ability to > pin down any damn file on the system and make it impossible to replace > is not something you want to give to any user. I agree, users shouldn't be able to pin down a file. I think unlink() should succeed on a file while something is visiting inside its metadata directory. It's a filesystem quality-of-implementation feature whether that actually releases the file's data. It's a desirable feature because one user shouldn't be able to pin another user's quota'd data if they don't have permission to open the file, but if it's not implemented by a filesystem then it doesn't break anything fundamental. It's a semantics question whether unlinking a file makes the metadata (i.e. "uid", "mode", "content-type" etc.) disappear at the same time, or if the metadata stays around until the last visitor leaves it. A filesystem might be able to keep the metadata in existence even if it deletes the file's storage on unlink(), but it would be nice for the VFS to declare which semantic is preferred. One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way. In those cases, if you open one of the virtually generated "archive content" files, then you might expect the data to continue to exist after the underlying file is unlinked. I think that's reasonable: being inside an archive is very similar to having it open. There is no quota pinning problem with this, because "archive content" files should inherit permission restrictions from the underlying file. If you can't read the file, then you can't read it's unpacked contents. (reiser4 doesn't offer that last feature, but any of the myriad userspace filesystem hooks could offer it if the VFS has approprate auto-mounting file-as-directory hooks). -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:00 ` Jamie Lokier @ 2004-08-26 3:13 ` viro 2004-08-26 8:49 ` Hans Reiser 2004-08-26 10:05 ` Chris Wedgwood 2 siblings, 0 replies; 1835+ messages in thread From: viro @ 2004-08-26 3:13 UTC (permalink / raw) To: Jamie Lokier Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote: > viro@parcelfarce.linux.theplanet.co.uk wrote: > > > Is this a problem if we treat entering a file-as-directory as crossing > > > a mount point (i.e. like auto-mounting)? > > > > Yes - mountpoints can't be e.g. unlinked. Moreover, having directory > > mounted on non-directory is also an interesting situation. > > Ok, so can we make it so mountpoints can be unlinked? :) User-visible change of behaviour and IIRC a SuS violation on top of that. > I think the underlying file does not stay locked, and once you've > entered it as a directory, it can be unlinked. So why lock it at all in that case? > I didn't mean locking a chain of mountpoints, I meant the temporary > state where two dentries and/or inodes are locked, parent and child, > during a path walk. However I'm not very familiar with that part of > the VFS and I see that the current RCU dcache might not lock that much > during a path walk. Never had been needed on crossing mountpoints, actually. > I agree, users shouldn't be able to pin down a file. > > I think unlink() should succeed on a file while something is visiting > inside its metadata directory. See above. Again, the fundamental problem with that is allowing unlink and friends on a mountpoint. I would love to do that, but it always generated -EBUSY on all Unices. Linux got a bit more users and userland code than Plan 9 - they can afford such changes, but... And yes, from the kernel POV it's trivial to do - witness the MNT_DETACH codepath in umount - it's much simpler than "normal" umount exactly because it doesn't try to emulate old "it's busy, can't umount" behaviour. With umount we could introduce "don't bother with that shit" flag. With unlink() we would have to make that default behaviour to be useful. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:00 ` Jamie Lokier @ 2004-08-26 8:49 ` Hans Reiser 2004-08-26 8:49 ` Hans Reiser 2004-08-26 10:05 ` Chris Wedgwood 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:49 UTC (permalink / raw) To: Jamie Lokier Cc: viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier wrote: > > >One of the big potential uses for file-as-directory is to go inside >archive files, ELF files, .iso files and so on in a convenient way. > > Yes, this was part of the plan, tar file-directory plugins would be cute. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:49 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:49 UTC (permalink / raw) To: Jamie Lokier Cc: viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier wrote: > > >One of the big potential uses for file-as-directory is to go inside >archive files, ELF files, .iso files and so on in a convenient way. > > Yes, this was part of the plan, tar file-directory plugins would be cute. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:49 ` Hans Reiser (?) @ 2004-08-26 14:05 ` Adrian Bunk 2004-08-26 14:25 ` Christophe Saout 2004-08-26 15:02 ` Jamie Lokier -1 siblings, 2 replies; 1835+ messages in thread From: Adrian Bunk @ 2004-08-26 14:05 UTC (permalink / raw) To: Hans Reiser Cc: Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote: > Jamie Lokier wrote: > > >One of the big potential uses for file-as-directory is to go inside > >archive files, ELF files, .iso files and so on in a convenient way. > > > > > Yes, this was part of the plan, tar file-directory plugins would be cute. Silly question: GNU Midnight Commander allows for ages to go into e.g. tar files, so I know the benefits of this. Additionally, in GNU Midnight Commander, this works no matter which file system I use (e.g. it works on iso9660), and it even works the same way on other OS's like e.g. Solaris and NetBSD. What is the technical reason why a tar plugin should be reiser4 specific, instead of a generic VFS or userspace solution that would allow the same also on other fs like e.g. iso9660? cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:05 ` Adrian Bunk @ 2004-08-26 14:25 ` Christophe Saout 2004-08-26 14:45 ` Matthew Wilcox 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:02 ` Jamie Lokier 1 sibling, 2 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 14:25 UTC (permalink / raw) To: Adrian Bunk Cc: Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1316 bytes --] Am Donnerstag, den 26.08.2004, 16:05 +0200 schrieb Adrian Bunk: > > Yes, this was part of the plan, tar file-directory plugins would be cute. > > Silly question: > > GNU Midnight Commander allows for ages to go into e.g. tar files, so I > know the benefits of this. Additionally, in GNU Midnight Commander, this > works no matter which file system I use (e.g. it works on iso9660), and > it even works the same way on other OS's like e.g. Solaris and NetBSD. > > What is the technical reason why a tar plugin should be reiser4 > specific, instead of a generic VFS or userspace solution that would > allow the same also on other fs like e.g. iso9660? I don't think that this should be done inside the kernel. tar is a perfectly valid userspace application. What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough). But below page-cache and thus invisible from the VFS or the applications it is extremely useful for the compression plugin. When changing some data in the middle of the file the compressed data might grow in size. reiser4 can handle this inside the storage layer and doesn't mess with the rest of the filesystem. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:25 ` Christophe Saout @ 2004-08-26 14:45 ` Matthew Wilcox 2004-08-26 15:04 ` Jamie Lokier 1 sibling, 0 replies; 1835+ messages in thread From: Matthew Wilcox @ 2004-08-26 14:45 UTC (permalink / raw) To: Christophe Saout Cc: Adrian Bunk, Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:25:13PM +0200, Christophe Saout wrote: > What reiser4 can do, but the VFS can't is to insert or remove data in > the middle of a file. Adding this above the page cache would probably be > almost impossible (truncate seems already complicated enough). > > But below page-cache and thus invisible from the VFS or the applications > it is extremely useful for the compression plugin. When changing some > data in the middle of the file the compressed data might grow in size. > reiser4 can handle this inside the storage layer and doesn't mess with > the rest of the filesystem. Never say never ... regular readers of -fsdevel already know what I'm going to say so can skip this post. When compressing data on an ext2-like filesystem, you can use the property of "holes" to deallocate disc blocks in the middle of a file. It works like this: Split the file into 64k chunks Compress each chunk individually, then write them to disc on 64k boundaries, leaving gaps. Now you can quickly seek to the appropriate 64k block for the data you're looking for and decompress just that 64k block. This is how RISCiX managed to fit an 80MB distro onto a 50MB disc. Someone has written an equivalent filesystem for Linux called SquashFS. -- "Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:25 ` Christophe Saout 2004-08-26 14:45 ` Matthew Wilcox @ 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:12 ` Christophe Saout ` (2 more replies) 1 sibling, 3 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 15:04 UTC (permalink / raw) To: Christophe Saout Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Christophe Saout wrote: > What reiser4 can do, but the VFS can't is to insert or remove data in > the middle of a file. Adding this above the page cache would probably be > almost impossible (truncate seems already complicated enough). That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems. By the way, can reiser4 share parts of files between different files? -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:04 ` Jamie Lokier @ 2004-08-26 15:12 ` Christophe Saout 2004-08-26 16:00 ` Jamie Lokier 2004-08-26 15:41 ` Nikita Danilov 2004-08-29 12:34 ` Alex Zarochentsev 2 siblings, 1 reply; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 15:12 UTC (permalink / raw) To: Jamie Lokier Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1479 bytes --] Am Donnerstag, den 26.08.2004, 16:04 +0100 schrieb Jamie Lokier: > Christophe Saout wrote: > > What reiser4 can do, but the VFS can't is to insert or remove data in > > the middle of a file. Adding this above the page cache would probably be > > almost impossible (truncate seems already complicated enough). > > That would be one of those "special features" that a > VFS-plus-userspace implementation of archive views could take > advantage of on reiser4, while using a slower method (sometimes much > slower) on all other filesystems. I'm just thinking about something. While you can't cut bytes out of unix iles a lot of filesystems can do this (holes). Most of them only on a block boundary, reiser4 on a byte boundary. If the filesystems could export this functionality using an enhanced API we could implement a compression plugin and other things on the VFS level that works with every filesystem supporting the required mechanisms, not only reiser4. And those features would take advantage of reiser4's storage mechanisms. I think Hans made the plugins reiser4-only because only reiser4 has a similar API at that time (and obviously because he didn't even think about doing it otherwise). > By the way, can reiser4 share parts of files between different files? At the moment a file is exactly one object. But someone could write another file plugin that spans a file across multiple objects, then yes, multiple files could share parts. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:12 ` Christophe Saout @ 2004-08-26 16:00 ` Jamie Lokier 2004-08-27 2:17 ` Hans Reiser 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 16:00 UTC (permalink / raw) To: Christophe Saout Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Christophe Saout wrote: > I'm just thinking about something. While you can't cut bytes out of unix > iles a lot of filesystems can do this (holes). Most of them only on a > block boundary, reiser4 on a byte boundary. If the filesystems could > export this functionality using an enhanced API we could implement a > compression plugin and other things on the VFS level that works with > every filesystem supporting the required mechanisms, not only reiser4. > And those features would take advantage of reiser4's storage mechanisms. > I think Hans made the plugins reiser4-only because only reiser4 has a > similar API at that time (and obviously because he didn't even think > about doing it otherwise). Yes. In some cases the compression or re-archiving algorithm would choose different parameters, depending on the alignment capabilities of the underlying filesystem. In some cases, that alignment is "offset 0 is the only one you can have", which is a severe special case where truncate and rewrite is required for changes. > > By the way, can reiser4 share parts of files between different files? > > At the moment a file is exactly one object. But someone could write > another file plugin that spans a file across multiple objects, then yes, > multiple files could share parts. Nice. That's useful for files which appear as themselves and also as part of a container. And also for revision managed filesystems :) -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:00 ` Jamie Lokier @ 2004-08-27 2:17 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 2:17 UTC (permalink / raw) To: Jamie Lokier Cc: Christophe Saout, Adrian Bunk, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Vitaly Fertman Jamie Lokier wrote: > > > > >>>By the way, can reiser4 share parts of files between different files? >>> >>> >>At the moment a file is exactly one object. But someone could write >>another file plugin that spans a file across multiple objects, then yes, >>multiple files could share parts. >> >> > >Nice. That's useful for files which appear as themselves and also as >part of a container. > >And also for revision managed filesystems :) > >-- Jamie > > > > > Inheritance is one of the unimplemented reiser4 features that Vitaly is working on. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:12 ` Christophe Saout @ 2004-08-26 15:41 ` Nikita Danilov 2004-08-29 12:34 ` Alex Zarochentsev 2 siblings, 0 replies; 1835+ messages in thread From: Nikita Danilov @ 2004-08-26 15:41 UTC (permalink / raw) To: Jamie Lokier Cc: Christophe Saout, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier writes: > Christophe Saout wrote: > > What reiser4 can do, but the VFS can't is to insert or remove data in > > the middle of a file. Adding this above the page cache would probably be > > almost impossible (truncate seems already complicated enough). > > That would be one of those "special features" that a > VFS-plus-userspace implementation of archive views could take > advantage of on reiser4, while using a slower method (sometimes much > slower) on all other filesystems. > > By the way, can reiser4 share parts of files between different files? It can, but fsck is supposed to cope with this. :) > > -- Jamie Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:12 ` Christophe Saout 2004-08-26 15:41 ` Nikita Danilov @ 2004-08-29 12:34 ` Alex Zarochentsev 2004-08-29 15:52 ` Shaya Potter 2004-09-01 22:05 ` Jamie Lokier 2 siblings, 2 replies; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-29 12:34 UTC (permalink / raw) To: Jamie Lokier Cc: Christophe Saout, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:04:34PM +0100, Jamie Lokier wrote: > Christophe Saout wrote: > > What reiser4 can do, but the VFS can't is to insert or remove data in > > the middle of a file. Adding this above the page cache would probably be > > almost impossible (truncate seems already complicated enough). > > That would be one of those "special features" that a > VFS-plus-userspace implementation of archive views could take > advantage of on reiser4, while using a slower method (sometimes much > slower) on all other filesystems. > > By the way, can reiser4 share parts of files between different files? no, those file plugins are not written yet :) Do you mean COW files or some another thing? For COW files, the reiser4 support is not enough, we need to teach cp(1) or another utility to inform the fs layer that copying can be done by creation of COW files. > -- Jamie -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 12:34 ` Alex Zarochentsev @ 2004-08-29 15:52 ` Shaya Potter 2004-09-01 22:05 ` Jamie Lokier 1 sibling, 0 replies; 1835+ messages in thread From: Shaya Potter @ 2004-08-29 15:52 UTC (permalink / raw) To: Alex Zarochentsev Cc: Jamie Lokier, Christophe Saout, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Sun, 2004-08-29 at 16:34 +0400, Alex Zarochentsev wrote: > Do you mean COW files or some another thing? For COW files, the reiser4 > support is not enough, we need to teach cp(1) or another utility to inform the > fs layer that copying can be done by creation of COW files. it's easy enough to add that via an fs ioctl(), I did the equivalent last semester for some research I'm doing while using ext3cow (which unfortunately was put on hold for the summer), it's just a take off of sys_link(), a sys_cowpy() of sorts. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 12:34 ` Alex Zarochentsev 2004-08-29 15:52 ` Shaya Potter @ 2004-09-01 22:05 ` Jamie Lokier 1 sibling, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 22:05 UTC (permalink / raw) To: Alex Zarochentsev Cc: Christophe Saout, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Alex Zarochentsev wrote: > > > What reiser4 can do, but the VFS can't is to insert or remove data in > > > the middle of a file. Adding this above the page cache would probably be > > > almost impossible (truncate seems already complicated enough). > > > > That would be one of those "special features" that a > > VFS-plus-userspace implementation of archive views could take > > advantage of on reiser4, while using a slower method (sometimes much > > slower) on all other filesystems. > > > > By the way, can reiser4 share parts of files between different files? > > no, those file plugins are not written yet :) > > Do you mean COW files or some another thing? COW files would be nice as well, but I meant another thing: for files which are parts of an uncompressed archive, it makes sense, if the fs offers it efficiently, to share the storage. Doing it properly requires that memory is also shared, for example when COW files are used to isolate trees in virtual server jails, you still want the mapped executables to share memory. You can't do that just by writing a plugin. > For COW files, the reiser4 support is not enough, we need to teach > cp(1) or another utility to inform the fs layer that copying can be > done by creation of COW files. Actually we don't. The VM and VFS layers could deduce it with no changes to cp(1), by tracking which pages are identical due to read(3) and write(3). However, changing cp(1) is easier. reiser4 support is obviously the first prerequisite... -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:05 ` Adrian Bunk 2004-08-26 14:25 ` Christophe Saout @ 2004-08-26 15:02 ` Jamie Lokier 2004-08-28 23:14 ` Horst von Brand 1 sibling, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 15:02 UTC (permalink / raw) To: Adrian Bunk Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Adrian Bunk wrote: > What is the technical reason why a tar plugin should be reiser4 > specific, instead of a generic VFS or userspace solution that would > allow the same also on other fs like e.g. iso9660? It should be a generic VFS plugin, not reiser4 or userspace. The VFS plugin should call out to userspace for most actions (except handling cached data), and it should take advantage of special reiser4 features for storage and performance optimisations. But it should still work over a standard filesystem, when those special features aren't available. I guess FUSE and many earlier projects are heading in this direction. A generic userspace solution doesn't let you "cd" into a tar file from all programs like you can inside Midnight Commander. Gnome and KDE take the approach that every userspace system call should be intercepted and filtered, to create the illusion of virtual data. As a result, different programs see different virtual data: you can't just cut and paste a path from Gnome or KDE into any other program. It's not just a "social problem of libraries" thing: sometimes I have programs which don't link to libc. Sometimes I have programs which mustn't link with anything that calls malloc(). It'd be silly for them to have a different view of the filesystem just because they can't link with some userspace library. The Gnome/KDE/Midnight Commander pure userspace solution is silly: if _every_ program in the system should get the same view, it makes much more sense for the kernel to filter the system calls and redirect the virtual accesses to a userspace daemon, while keeping the real accesses at full speed. Furthermore is makes much more sense for the kernel's page cache to hold those uncompressed pages, than for every userspace application to try and cooperatively manage a cache of uncompressed fragments in the most inefficient way. There's another problem with the Midnight Commander approach. If I "cd" into a tar file, and then a program writes to the tar file, I don't always see the changes straight away. The two views aren't coherent. This isn't an easy problem to solve, but it should be solved. When a simple "cd" into .tar.gz or .iso is implemented properly, it will have _no_ performance penalty after you have first looked in the file, so long as it remains in the on-disk cache. And, the filesystem will manage that cache intelligently. Imagine: for looking at source files and such, you probably won't bother untarring in future, and you won't bother keeping untarred source trees in your home directory for easy access to things you look at often. Why waste the space? You could install whole applications as a .tar and run them from within it, with no performance penalty. Similarly, the filesystem will be able to archive directories automatically that haven't been touched in a long time, with no visible change except increased storage space. "grep" will be a bit slower, but you'll have a useful search tool by then (using coherent indexes) which will be more useful than grep, and much faster. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:02 ` Jamie Lokier @ 2004-08-28 23:14 ` Horst von Brand 2004-09-01 20:08 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Jamie Lokier 0 siblings, 1 reply; 1835+ messages in thread From: Horst von Brand @ 2004-08-28 23:14 UTC (permalink / raw) To: Jamie Lokier Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier <jamie@shareable.org> said: [...] > When a simple "cd" into .tar.gz or .iso is implemented properly, it > will have _no_ performance penalty after you have first looked in the > file, so long as it remains in the on-disk cache. And, the filesystem > will manage that cache intelligently. Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or un-tarred in memory (or on disk cache) for this to be true, and that takes quite a long time. Each time you want to peek anew at linux/Makefile, the whole tarfile will have to be read and stored somewhere, and that is just too slow for my taste. The .tar format is optimized for compact storage, the on-disk format of a filesystem is optimized for fast access and modifiability. Now go ahead and enlarge a file on your .iso/.tar a bit...it will take ages to rebuild the whole thing. There is a _reason_ why there are filesystems and archives, and they use different formats. If it weren't so, everybody and Aunt Tillie would just carry .ext3's around, and would wonder what the heck all this fuss is about. > Imagine: for looking at source files and such, you probably won't > bother untarring in future, and you won't bother keeping untarred > source trees in your home directory for easy access to things you look > at often. Why waste the space? You could install whole applications > as a .tar and run them from within it, with no performance penalty. Perpetuum mobile is a nice idea too. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-08-28 23:14 ` Horst von Brand @ 2004-09-01 20:08 ` Jamie Lokier 2004-09-01 20:50 ` Linus Torvalds 2004-09-02 14:06 ` Horst von Brand 0 siblings, 2 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 20:08 UTC (permalink / raw) To: Horst von Brand Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List I'm going to explain why filesystem support for .tar.gz or other "document container"-like formats is useful. This does _not_ mean tar in the kernel (I know someone who can't read will think that if I don't say it isn't); it does mean hooks in the kernel for maintaining coherency between different views and filesystem support for cacheing. The vision I'm going for here is: 1. OpenOffice and similar programs store compound documents in standard file formats, such as .tar.gz, compressed XML and such. Fs support can reduce CPU time handling these sorts of files, as I explain below, while still working with the standard file formats. With appropriate userspace support, programs can be written which have access to the capabilities on all platforms, but reduced CPU time occurs only on platforms with the fs support. 2. Real-time indexing and local search engine tools. This isn't just things like local Google; it's also your MP3 player scanning for titles & artists, your email program scanning for subject lines to display the summary fast, your blog server caching built pages, your programming environment scanning for tags, and your file transfer program scanning for shared deltas to reduce bandwidth. I won't explain how these work as it would make this mail too long. It should be clear to anyone who thinks about it why the coherency mechanism is essential for real-time, and a consistent interface to container internals helps with performance. Horst von Brand wrote: > Jamie Lokier <jamie@shareable.org> said: > > When a simple "cd" into .tar.gz or .iso is implemented properly, it > > will have _no_ performance penalty after you have first looked in the > > file, so long as it remains in the on-disk cache. And, the filesystem > > will manage that cache intelligently. > > Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or > un-tarred in memory (or on disk cache) for this to be true, and that takes > quite a long time. Each time you want to peek anew at linux/Makefile, the > whole tarfile will have to be read and stored somewhere, Wrong. "So long as it remains in the on-disk cache" means each time you peek at linux/Makefile, the tarfile is _not_ read. For a tarfile it's slow the first time, and when it falls out of the on-disk cache, otherwise, for component files you are using regularly (even over a long time) it's as fast as reading a plain file. You obviously know this, as you mentioned on-disk cache in the reply, so I infer from the rest of your mail that what you're trying to say is more about modifications than reading archives. That it would be silly to keep working data in .tar.gz files, because working inside them regularly would be slow. Which means you must be assuming, incorrectly, that these .tar.gz files are really kept up to date on disk with every component file modification. Which is silly. .tar.gz files are suitable for *transport* and *archival*, not regular random access; it's almost rude of you to suggest I didn't know that. The proposal is that .tar.gz files (and others) are analysed on demand and content cached on disk as it is read. Then subsequent reads will be as fast as if you had unpacked the archives by hand, manually using the tar command. This is obviously exactly the same as you do now, with a small bit of added convenience. The other part of the proposal is that when you modify a component, the modifications are stored on disk in the same way as ordinary files, using the regular high performance random access disk structures. Nothing is done to recreate the archive at this point; I think this is where you misunderstood and thus flamed. _If_ after modifying components, you then read the .tar.gz as a file, then (and only then) is it recreated, taking in the worst case the same time as running the tar command. The _only_ times when that occurs are precisely those times when you would have run the tar command manually: because you only read the .tar.gz file when you need the flat file for some purpose, such as attaching it to an email, transferring by FTP or HTTP, or reading it into a program that needs it in that format. If there is anything about that strategy that doesn't make sense, then I suggest I have failed to explain it properly, and you're welcome to demand a clearer explanation. > the .tar format is optimized for compact storage, the on-disk format > of a filesystem is optimized for fast access and modifiability. Actually no, .tar is not compact at all. It's also not optimised for random read access, but after an index is build it is very fast for that. .tar.gz is compact. Although that is not especially fast for random read access, you can build a "compression dictionary index" which optimises random read access even in a .tar.gz, without ever unpacking the whole thing. Some formats like .iso, .zip and .jar are optimised for compact storage _and_ fast random access. They come with an index, and don't need one to be built and cached. These are not filesystem-like formats, obviously, but they are the formats you need to pack and unpack when exchanging data with other people. That's the _only_ reason they're on your disk (virtually; they may not really exist some of the time). > Now go ahead and enlarge a file on your .iso/.tar a bit...it > will take ages to rebuild the whole thing. There is a _reason_ why there > are filesystems and archives, and they use different formats. If it weren't > so, everybody and Aunt Tillie would just carry .ext3's around, and would > wonder what the heck all this fuss is about. If you enlarge a file in your .iso/.tar subdirectory a bit... nothing happens. Why would a smart programmer do anything so silly as rebuild the archive at that point? _If_ you subsequently read the .iso/.tar _file_, then and only then does it rebuild. Once, after lots of component writes. The only time you would ever do that is if you are specifically reading the archive file, which means you actually want to use the repacked file at that point, for example to FTP it somewhere or use as an email attachment. If the filesystem does not do that on demand, then you would have run the tar command manually at that point, precisely because that's a point where you need the repacked archive. So in case where the filesystem repacks the archive, it takes exactly the same time as you would have taken anyway; it's just automatic instead of manual. (As a handy side effect, the automatic method offers lower latency for transmissions). Now, why would we bother with all this? I see three reasons: convenience, time efficiency, and storage efficiency. Convenience is simply that it is handy to be able to look inside archive files, in those situations where we _currently_ use them, without having to manually untar when we need to, and without having to remember to clean up old directories when we discover we aren't using those often any more. This is _not_ an argument for using .tar.gz files in place of ordinary directories! Convenience applies to doing the things you do now. Time efficiency has two angles. A simple one is that accessing .tar.gz contents through any kind of filesystem interface, even pure userspace, can be faster than unpacking whole files, simply because there are ways to decode parts without unpacking it all. However, the main time efficiency that I see comes from the increasing number of applications where the "Open" and "Save" operations store data in *.gz files (e.g. OpenOffice compressed XML documents), or *.tar.gz files (some compound document formats), or other things like that. (If you think about it, quite a lot of things are like that). With these, every "Open" currently has to decompress and maybe unpack an archive format. Every "Save" currently has to pack and then compress. This is done so the user sees a single flat file containing a complex document, but it is a waste of CPU time until the user actually transports the flat file. The lazy proposal, as described earlier in this mail, _removes_ these decompression, unpacking, packing and compression CPU-intensive steps when they are unnecessary. The experience of a single file containing a complex document is maintained, but the CPU time is reduced in many typical operations. "Open" gets faster after you first look at a file, "Save" gets a lot faster for large documents, and the equivalent of grep (or later, real-time local search engine) gets a lot faster too. There is no operation where CPU time is overall increased. This is what I've meant throughout this thread, when I say containers: document files of the kind used to hold text, figures, etc. that are typically transported as a unit, and edited as a unit, but nonetheless at the moment they're stored in somewhat CPU intensive formats, for compactness. That's fine for a 1 page letter, but think of the OpenOffice 500-page book containing a large number of diagrams. However, even simple programs that read & write compressed XML benefit. The proposal allows that sort of thing to be handled more time efficiently that it is today, and in a way that is very practical to use. (It's unthinkable that OpenOffice and similar programs would have a lot of code which stored data in a special way just for Linux, just for these performance benefits which are otherwise user-invisible, but it's thinkable that a general purpose userspace library which is portable to all platforms could be written, which takes advantage of the facility when it's available and does the equivalent of today's "compress on save" when the filesystem facility isn't available). Finally, storage efficiency comes from simply allowing the filesystem and supporting tools to decide when it is best to store data in unpacked, packed & compressed, both at the same time, or another archival form. The filesystem has comparativaly good knowledge of which data to archive and when, but it can only maintain the illusion if there's a mechanism to make archived forms and unpacked forms coherent. Now, I'm sure there is a way to implement this on top of a neat and simple kernel feature involving weird bind mounts, leases, dnotifies and FUSE. But those kernel offering are quite a mess at the moment and don't fit together in a way which can usefully create this effect. Auto-mounting uservfs directories over file-as-directory, using moveable bind mounts _nearly_ offers the kernel primitives we need to build this in userspace and get the all the efficiencies. But not quite. (We could obviously do it all in userspace by putting _everything_ in a userspace filesystem, but that would be silly as it would throw away all of the performance of having a threaded filesystem in the kernel. It might do as a proof of concept though). -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-01 20:08 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Jamie Lokier @ 2004-09-01 20:50 ` Linus Torvalds 2004-09-02 0:24 ` Jamie Lokier 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 14:06 ` Horst von Brand 1 sibling, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-01 20:50 UTC (permalink / raw) To: Jamie Lokier Cc: Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 1 Sep 2004, Jamie Lokier wrote: > > I'm going to explain why filesystem support for .tar.gz or other > "document container"-like formats is useful. This does _not_ mean tar > in the kernel (I know someone who can't read will think that if I > don't say it isn't); it does mean hooks in the kernel for maintaining > coherency between different views and filesystem support for cacheing. I think that's a valid thing, but there are some fundamental problems with it if you expect it to work on a normal filesystem (ie something that isn't fundamentally designed as a database). For example, _what_ kind of coherency do you think is acceptable? Quite frankly, using standard UNIX interfaces, absolute coherency just isn't an option, because it's just not possible to try to atomically update a view at the same time somebody else is writing to the "main file". "mmap()" is the most obvious example of this, but even the _basic_ notion of multiple "read" calls is not atomic without locking that is _way_ too expensive. A "read()" on a file is not atomic even on the _plain_ file: if somebody does a concurrent "write()", the reader may see a partial update. This becomes a million times more confusing if the reader is seeing a structured view of the file the writer is modifying. Also, it's likely impossible to write() to the view-file, again unless you expect all the underlying filesystems to be something really special. So from a _practical_ standpoint, I suspect that the best you can really do pretty cheaply (and which gets you 90% of what you probably want) is: - open-close consistency: the "validity" of the cache is checked at _open_ time, and no guarantees are given about the cache being updated afterwards. - read-only access to the cache (ie you can only read the view, not write to it). and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible. Suggested interface: int open_cached_view(int base_fd, char *type, char *subname); where "type" would be the type of the view (ie "tar" for a tar-file view, "idtag" for a mp3 ID tag, or NULL for "autodetect default view") and "subname" would be the cache entry name (ie the tar-file filename, or the tag type to open). I bet you could write a small library to test this out for a few types. See if it's useful to you. And only if it's useful (and would make a huge performance difference) would it be worth putting in the kernel. Implementation of the _user_space_ library would be something like this: #define MAXNAME 1024 int open_cached_view(int base_fd, char *type, char *subname) { struct stat st; char filename[PATH_MAX]; char name[MAXNAME]; int len, cachefd; if (fstat(base_fd, &st) < 0) return -1; sprintf(name, "/proc/self/fd/%d", base_fd); len = readlink(name, filename, sizeof(filename)-1); if (len < 0) return -1; filename[len] = 0; /* FIXME! Replace '/' with '#' in "type" and "subname" */ len = snprintf(name, sizeof(name), "%04llx/%04llx/%s/%s/%s", (unsigned long long) st.st_dev, (unsigned long long) st.st_ino, type ? : "default", subname, filename); errno = ENAMETOOLONG; if (len >= sizeof(name)) return -1; cachefd = open(name, O_RDONLY); if (cachefd >= 0) { /* Check mtime here - maybe we could have kernel support */ return cachefd; } if (errno != ENOENT) return -1; /* .. try to generate cache file here .. */ see what I'm aiming at? You start out with a generic "attribute cache" library that does some hacky things (like depending on "mtime" for coherency) and then if that works out you can see if it's useful. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-01 20:50 ` Linus Torvalds @ 2004-09-02 0:24 ` Jamie Lokier 2004-09-02 1:07 ` Linus Torvalds 2004-09-02 3:35 ` David Masover 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 1 sibling, 2 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-02 0:24 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > > I'm going to explain why filesystem support for .tar.gz or other > > "document container"-like formats is useful. This does _not_ mean tar > > in the kernel (I know someone who can't read will think that if I > > don't say it isn't); it does mean hooks in the kernel for maintaining > > coherency between different views and filesystem support for cacheing. Thanks for your reply. > I think that's a valid thing, but there are some fundamental problems with > it if you expect it to work on a normal filesystem (ie something that > isn't fundamentally designed as a database). > > For example, _what_ kind of coherency do you think is acceptable? Quite > frankly, using standard UNIX interfaces, absolute coherency just isn't an > option, because it's just not possible to try to atomically update a view > at the same time somebody else is writing to the "main file". "mmap()" is > the most obvious example of this, but even the _basic_ notion of multiple > "read" calls is not atomic without locking that is _way_ too expensive. > > A "read()" on a file is not atomic even on the _plain_ file: if somebody > does a concurrent "write()", the reader may see a partial update. This > becomes a million times more confusing if the reader is seeing a > structured view of the file the writer is modifying. I agree. Coherency should be about equivalent to what writing flat files offers. (For now. Eventually I think we'll see transactional I/O in Linux, to solve the sort of updater-to-server glitches which are currently solved by putting even trivial data behind clunky database servers). The coherency I'd like would also ensure that you'll never see an invalid file structure in the long term, even if during writing operations you might see temporarily invalid structure. That is easy if you only have read-only views. > Also, it's likely impossible to write() to the view-file, again unless you > expect all the underlying filesystems to be something really special. Right. I wonder if you read the part of my message which deals with lazy updates of container files. The idea is that write() to a view-file doesn't repack the container file until the container file is read. Practically, that means the view-file's write handler, which is probably in userspace, grabs a kind of mandatory lock (similar to F_SETLEASE) on the container file and then truncates it. After a time, or when a program tries to read the container (whichever comes first), the view-file's handler is notified, it regenerates the container file, and releases the lock. This is the form of coherency needed to make writes work properly. (You can take it more fine grained, locking at the page level, but that's just to improve performance some more. It's not fundamental.) Notice that it doesn't need a special filesystem. View-file writes will work with any ordinary filesystem. A special filesystem would make it perform better (much better), by allowing the truncated state to persist across reboots, with an xattr to say what's needed to recreate the data -- but it's not fundamentally necessary. It does need kernel support for that kind of lock, which F_SETLEASE isn't, and notifications of mount events, and notifications of file writes. > So from a _practical_ standpoint, I suspect that the best you can really > do pretty cheaply (and which gets you 90% of what you probably want) is: > > - open-close consistency: the "validity" of the cache is checked at > _open_ time, and no guarantees are given about the cache being > updated afterwards. > - read-only access to the cache (ie you can only read the view, not write > to it). I think that gets 90% of the useful features, but leaves out some applications which would raise eyebrows when they actually work -- i.e. the component editing applications. > and quite frankly, I think you can do the above pretty much totally in > user space with a small library and a daemon (in fact, ignoring security > issues you probably don't even need the daemon). And if you can prototype > it like that, and people actually find it useful, I suspect kernel support > for better performance might be possible. Right now it's stupidly impossible for a daemon to monitor a file for changes and not be tricked by mtime faults, because someone can link to the file and modify it on another path. That's obviously a bug in dnotify which I keep meaning to fix. There are other dnotify problems, some of which are fundamental when it's over a network. I do care about security issues, for things like an "md5" or "compiled" attribute especially, but for a prototype that's just something to live with. > Suggested interface: > > int open_cached_view(int base_fd, char *type, char *subname); Something userspace is useful anyway, so that a fully userspace alternative is available, so that people writing apps will take up the approach and take advantage of fs-level support where available while still being portable elsewhere. > see what I'm aiming at? You start out with a generic "attribute cache" > library that does some hacky things (like depending on "mtime" for > coherency) and then if that works out you can see if it's useful. Ok but I don't think that form of it is useful! It's the sort of interface that specific attribute-using programs will have to use if they're to be portable, but it doesn't provide any special advantages to any other programs. Gnome VFS, KDE etc. provide much of that kind of interface already. Evidently some people find that useful. But I don't, precisely because it's so tied into one set of programs or another. This is where the file-as-directory metadata stuff is so potentially interesting. It's actually a nice interface that every program can see. It's the nice interface when running on Linux that'll make it deliciously worthwhile for portable applications, such as MP3 retaggers and PDF image extractors, to be written in the form of a tool plus a plugin for a common view-file library, instead of all in the tool as they are now. Without the feature of a nice interface on Linux, there's no reason why application writers would bother to learn and use a view-file library. To be fair, the whole thing could be prototyped (with glitches, but enough to demonstrate) in userspace, running everything through something like uservfs or even NFS, especially with file-as-directory being blessed at the VFS level. That is exactly the right way to go about it. I started some work on that back in '99, around the same time Pavel was exerimenting with podfuk (which become uservfs), but haven't and don't have the time or money(*) to take it further. -- Jamie (*) That's a hint to potential employers who want to support this work, by the way. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 0:24 ` Jamie Lokier @ 2004-09-02 1:07 ` Linus Torvalds 2004-09-02 2:30 ` Hubert Chan 2004-09-02 3:46 ` The argument for fs assistance in handling archives David Masover 2004-09-02 3:35 ` David Masover 1 sibling, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 1:07 UTC (permalink / raw) To: Jamie Lokier Cc: Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Jamie Lokier wrote: > > This is where the file-as-directory metadata stuff is so potentially > interesting. It's actually a nice interface that every program can see. .. but you can _still_ do exactly this in a user-space library and prototype it. Just take over "open" (and opendir/readdir), and you can do so. > To be fair, the whole thing could be prototyped (with glitches, but > enough to demonstrate) in userspace, running everything through > something like uservfs or even NFS, especially with file-as-directory > being blessed at the VFS level. I do believe that you can prototype it even without any kernel support. This is why I want to have stronger reasons for real VFS support. I see the Samba argument, and I buy that one, if only because it has such a high consistency requirement (let's face it, people expect more guarantees of their file servers than of most other things). But on the other hand, the Samba guys apparently _are_ happy with "openat()", so they don't necessarily need any normal namespace stuff. And I suspect that a user-level prototype would show that what you _really_ want may be something else altogether.. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Why do we need openat? (was: The argument for fs assistance in handling archives) 2004-09-02 1:07 ` Linus Torvalds @ 2004-09-02 2:30 ` Hubert Chan 2004-09-02 3:46 ` The argument for fs assistance in handling archives David Masover 1 sibling, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-02 2:30 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes: [...] Linus> This is why I want to have stronger reasons for real VFS Linus> support. I see the Samba argument, and I buy that one, if only Linus> because it has such a high consistency requirement (let's face Linus> it, people expect more guarantees of their file servers than of Linus> most other things). But on the other hand, the Samba guys Linus> apparently _are_ happy with "openat()", so they don't necessarily Linus> need any normal namespace stuff. Maybe things have got lost in this thread (or these many threads), so bear with me, but what exactly does openat() gain us that we can't get from exporting things from the normal file namespace? The only issue that I've seen raised is that you need to be able to distinguish between the contents of a directory and its substreams. And I've responded that streams should be stuck in a specially-named subdirectory (like ..streams) (which should be done even if only files could contain substreams), to which I have seen no objections raised (not even as much as a "you're stupid"), and so can only assume nobody (who cares) has any objections. So as far as I can see, openat() has no advantages, and exporting things through the normal namespace has several advantages, such as: - can manipulate substreams using legacy programs -- no modifications needed. And don't tell me "runat" until someone shows me how to deal with my GIMP/emacs problem. - useful for exporting different things (streams, acl's or other access control schemes, xattrs, version control, automatic untar, ...) - only need to modify tar (and backup programs) once -- any new filesystem extensions, no matter how complex, can be exported through the filesystem interface, and tar will be able to catch them. - referring to the substreams is simpler ("foo.txt/streams/bar" instead of "the bar stream of foo.txt"), which leads to simpler user interface. This is even more marked if we, for some strange reason, discover that it's a good idea for streams to be allowed to have substreams of their own ("foo.txt/streams/bar/streams/baz" vs. "the baz stream of the bar stream of foo.txt") - I think I'm forgetting some... If this is indeed the case, then it seems to me that the logical choice is to export streams through the normal file namespace. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Why do we need openat? (was: The argument for fs assistance in handling archives) @ 2004-09-02 2:30 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-02 2:30 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes: [...] Linus> This is why I want to have stronger reasons for real VFS Linus> support. I see the Samba argument, and I buy that one, if only Linus> because it has such a high consistency requirement (let's face Linus> it, people expect more guarantees of their file servers than of Linus> most other things). But on the other hand, the Samba guys Linus> apparently _are_ happy with "openat()", so they don't necessarily Linus> need any normal namespace stuff. Maybe things have got lost in this thread (or these many threads), so bear with me, but what exactly does openat() gain us that we can't get from exporting things from the normal file namespace? The only issue that I've seen raised is that you need to be able to distinguish between the contents of a directory and its substreams. And I've responded that streams should be stuck in a specially-named subdirectory (like ..streams) (which should be done even if only files could contain substreams), to which I have seen no objections raised (not even as much as a "you're stupid"), and so can only assume nobody (who cares) has any objections. So as far as I can see, openat() has no advantages, and exporting things through the normal namespace has several advantages, such as: - can manipulate substreams using legacy programs -- no modifications needed. And don't tell me "runat" until someone shows me how to deal with my GIMP/emacs problem. - useful for exporting different things (streams, acl's or other access control schemes, xattrs, version control, automatic untar, ...) - only need to modify tar (and backup programs) once -- any new filesystem extensions, no matter how complex, can be exported through the filesystem interface, and tar will be able to catch them. - referring to the substreams is simpler ("foo.txt/streams/bar" instead of "the bar stream of foo.txt"), which leads to simpler user interface. This is even more marked if we, for some strange reason, discover that it's a good idea for streams to be allowed to have substreams of their own ("foo.txt/streams/bar/streams/baz" vs. "the baz stream of the bar stream of foo.txt") - I think I'm forgetting some... If this is indeed the case, then it seems to me that the logical choice is to export streams through the normal file namespace. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 1:07 ` Linus Torvalds 2004-09-02 2:30 ` Hubert Chan @ 2004-09-02 3:46 ` David Masover 1 sibling, 0 replies; 1835+ messages in thread From: David Masover @ 2004-09-02 3:46 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Linus Torvalds wrote: [...] | This is why I want to have stronger reasons for real VFS support. I see | the Samba argument, and I buy that one, if only because it has such a high | consistency requirement (let's face it, people expect more guarantees of | their file servers than of most other things). But on the other hand, the | Samba guys apparently _are_ happy with "openat()", so they don't | necessarily need any normal namespace stuff. I see your Samba and I raise you an NFS. For that matter, how about every distributed FS to come. All it needs to do is support transactions (which most try to anyway) and files-as-directories to support _all_ future FS features transparently. It might ask the fs for a non-recursive serialization/dump of a file rather than just reading it directly -- this makes things like named pipes and device nodes sane. This would be optional and only supported when the client and server both have support for the same fs -- other, more hack-ish things would be needed to support Samba. It would also use the best caching mechanism discussed earlier. (I'm not yet arrogant enough to insist that mine is the best.) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTaXs3gHNmZLgCUhAQIG5g/+MEHPsNK41goKI8yJPwKdigbRbrfGf3sj xpFGR15/DAW4b4qxr1BIt5hyAQoFLq13POM/lCV12hKp1RSo9atuKim6DyUhOX1c 0xUK6efY0ew+mvYHKaJUgwrD8BV7XYaTS+5y/4wbpgdOlPHsx06lHVh90KtAUAE7 vdsFWRAgHT/g34rSRBJfHAaom/7VtDce25roLrB3CNI2v+9tERYee+xkBPoy2+Y1 5tLgoInDi05z7bUu0lCJ+Rq2LCHuKGmgInI86gul5BfHlO3j2zYnJ5VYT8QR6onM iThiPzFrrpoXEjQRRo7E4WHQCD/2BGXzQdS78JzwTt1B8oNrIkZR9/tcv6keNzr6 fyUhtQef5TZJtwpON1gZPqraBUiTDUdWHJSXGLB5+78vMy+bS7gHqXN5DR049+7b urQFtnZBhi3zY8cpT0hTPnW2PcGnDuGMt9qUiFBn8Hx9Tkti6y+7Jl42a40jgBAW z4hSHPKVzu8d/qeBapVHtqIkaLTdim1JnmFY+SCiiuCPEMOELDxw+o/rcwTIls9s 3TBJ5lqL/+84W8JU6t1CpzEgF+dRhPSBNyrG5msVixT7cmB2GO87rLBfchM1cotq lBMyuDwmwO6RanwkEvOV//3wsG8POMw5f2Gx8eljqU9uzSpYcKry+Is2ZVmHKJ/1 24eqEXIB24Y= =GSMf -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 0:24 ` Jamie Lokier 2004-09-02 1:07 ` Linus Torvalds @ 2004-09-02 3:35 ` David Masover 2004-09-02 3:38 ` Linus Torvalds 1 sibling, 1 reply; 1835+ messages in thread From: David Masover @ 2004-09-02 3:35 UTC (permalink / raw) To: Jamie Lokier Cc: Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jamie Lokier wrote: [...] |>A "read()" on a file is not atomic even on the _plain_ file: if somebody |>does a concurrent "write()", the reader may see a partial update. This |>becomes a million times more confusing if the reader is seeing a |>structured view of the file the writer is modifying. | | | I agree. Coherency should be about equivalent to what writing flat | files offers. | | (For now. Eventually I think we'll see transactional I/O in Linux, to Eventually? This was supposed to be in reiser4, maybe it's been put off until 4.1... And that is the right solution. Not the only one, but the right one. Caching isn't the only thing sorely in need of transaction support right now. Actually, I find it hard to think of anything on Linux which shouldn't have transactions -- why should /etc/fstab or /home/david/homework be more fragile than /var/lib/mysql? |>Also, it's likely impossible to write() to the view-file, again unless you |>expect all the underlying filesystems to be something really special. | | | Right. I wonder if you read the part of my message which deals with | lazy updates of container files. | | The idea is that write() to a view-file doesn't repack the container | file until the container file is read. | | Practically, that means the view-file's write handler, which is | probably in userspace, grabs a kind of mandatory lock (similar to In userspace? I don't buy that -- this is exactly what a reiser4 plugin is for, just in OOP technology. Instead of "write handler", it's probably called "write method". | F_SETLEASE) on the container file and then truncates it. After a | time, or when a program tries to read the container (whichever comes | first), the view-file's handler is notified, it regenerates the | container file, and releases the lock. Why "after a time"? Should be "when a program tries to read the container" -- and with transactions or even some clever locking, it can be just as consistant as the flatfile write. [...] | (You can take it more fine grained, locking at the page level, but | that's just to improve performance some more. It's not fundamental.) Actually, "locking at the page level" rarely makes more sense than "use two files". And with transactions, I think it works like this: Process A opens file.tar.gz/contents/foo.c and starts writing. When it's maybe halfway through, process B opens file.tar.gz for reading. So it gets the version of foo.c that existed before A started writing. B finishes before A does, and then process C opens file.tar.gz for writing - -- it has to wait for A to complete. | Notice that it doesn't need a special filesystem. View-file writes | will work with any ordinary filesystem. A special filesystem would | make it perform better (much better), by allowing the truncated state | to persist across reboots, with an xattr to say what's needed to | recreate the data -- but it's not fundamentally necessary. I am hoping that someone will implement a VFS-based storage layer plugin for reiser4. This means that one can use reiser4 interface features (such as reading file.tar.gz as a directory) without needing to reformat. It also means that one could conceivably port reiser4 to a userland library, even. I think. I wonder if it's as much of a performance hit as uservfs. [...] |>Suggested interface: |> |> int open_cached_view(int base_fd, char *type, char *subname); Oh, God no. Alright, it's more useful than no caching at all, but still... Alright, let's say I have /usr/src/linux as a hardlink farm to /usr/src/linux.tar.bz2/contents/. With kernel support, I can actually do a make from inside /usr/src/linux and have it behave sanely. Do you want to patch make? If yes, you may as well patch everything. In fact, you may as well patch glibc to replace open with your open_cached_file. Not to mention handling things like readdir, stat, and so on... Not to mention the overhead of checking if a file is part of a cache for every single open, ever... Putting it in the filesystem simplifies things. There's already some sort of read/write method for the file -- overload that for files which need to be fetched and cached. Files which are already in the cache, or have nothing to do with the cache, remain exactly as fast as they are now. I'm not sure about the best way to make this portable between filesystems. I'm not sure if it's really that useful to do so. I'm beginning to see why Hans would like reiser4 to be the new VFS. | Something userspace is useful anyway, so that a fully userspace | alternative is available, so that people writing apps will take up the | approach and take advantage of fs-level support where available while | still being portable elsewhere. Maybe. But here, I'd suggest that the kernel implementation is fine, too. AFAIK, Reiser4 was designed to be portable -- specifically because certain apps -- not OSes, but apps -- might embed it for a fee. |>see what I'm aiming at? You start out with a generic "attribute cache" |>library that does some hacky things (like depending on "mtime" for |>coherency) and then if that works out you can see if it's useful. | | | Ok but I don't think that form of it is useful! It's the sort of | interface that specific attribute-using programs will have to use if | they're to be portable, but it doesn't provide any special advantages | to any other programs. | | Gnome VFS, KDE etc. provide much of that kind of interface already. | Evidently some people find that useful. But I don't, precisely | because it's so tied into one set of programs or another. | | This is where the file-as-directory metadata stuff is so potentially | interesting. It's actually a nice interface that every program can see. Exactly! Every program should be able to see the interface. The interface should also be transparent sometimes. Example -- reimplementing InterMezzo/Lustre. Files that are cached are exactly as fast to write as local files. To invalidate the cache, you delete the file. | Without the feature of a nice interface on Linux, there's no reason | why application writers would bother to learn and use a view-file library. That's not as compelling a reason. Application writers don't care at all whether it's in the kernel or userland. All putting it in the kernel does for adoption is make everyone sit up and notice. [...] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTaU5XgHNmZLgCUhAQJsBQ//WBWU28IrQc8mFokLyeFtIAIErqUE65oj TDcx0/Dg+2JD4zAMht6idhMDfYU5kvBj5C9srMvdklprgPJ/dIQSS/MS8o6qnZl4 kRvjUwI/vBV9a2gwxzt32NrLK+ZFuWwOiRJ1ia6zOyxqOE/jx8z1VDR7R17lbakh EbFG/2taoHyRke4O+rxig93OxTUBJ72KBKZ+LGrDMwzm7uNP069UupWO0ARpBlMM 6T0o8bm4QK7dgqXgowDGx22hFo5ZPLc3undsHepnu+i8cZMPpOMigbgwpiG/VNnu t00WOrJGigNkeOrCB4y4LO6LQFL7DjLRRI3T9Y3t3YUFegYDdG6A1L2ZQcrxqtlj alSx6YYYigH2+MBaN+ONIPlZVe9ybdTpAWl+hkS1nvKaNVc5xYUTFBhBAzIr/4Ol Zuu5sE8/Q7+5oOCqp9zW51g4oBIlDE0CY3EoZ1P5UrQXT5KphVWr8AUXcP+9CL5c vsIbYWTmbU/XgxIb+lyB2anTGnMzy1TxUHVxAR4ijXPSDuAbojJhtg1VuDMGhahq NfW34jRbDJSPDoar/eN2jxF8a4bG7xt8w10LO0lnnjznNJdsurk4HmsHFhPZ0DJj OJTaChsEr+KUldLyNila8A+6Qg4LvSvNSPArXxCTkGizN+jeyFlNSgqFOQajS/Pk fptJFhA1x3Q= =dUCH -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 3:35 ` David Masover @ 2004-09-02 3:38 ` Linus Torvalds 2004-09-02 4:27 ` David Masover 2004-09-02 5:53 ` Hans Reiser 0 siblings, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 3:38 UTC (permalink / raw) To: David Masover Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 1 Sep 2004, David Masover wrote: > > And that is the right solution. Not the only one, but the right one. > Caching isn't the only thing sorely in need of transaction support right > now. Actually, I find it hard to think of anything on Linux which > shouldn't have transactions -- why should /etc/fstab or > /home/david/homework be more fragile than /var/lib/mysql? It's easy to talk big. It's damn hard to _implement_ a complex system, and make it stable and bug-free, and support legacy applications. There is a reason why we do only what _must_ be done in kernel space. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 3:38 ` Linus Torvalds @ 2004-09-02 4:27 ` David Masover 2004-09-02 6:04 ` Linus Torvalds 2004-09-02 5:53 ` Hans Reiser 1 sibling, 1 reply; 1835+ messages in thread From: David Masover @ 2004-09-02 4:27 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Linus Torvalds wrote: | | On Wed, 1 Sep 2004, David Masover wrote: | |>And that is the right solution. Not the only one, but the right one. |>Caching isn't the only thing sorely in need of transaction support right |>now. Actually, I find it hard to think of anything on Linux which |>shouldn't have transactions -- why should /etc/fstab or |>/home/david/homework be more fragile than /var/lib/mysql? | | | It's easy to talk big. | | It's damn hard to _implement_ a complex system, and make it stable and | bug-free, and support legacy applications. | | There is a reason why we do only what _must_ be done in kernel space. | | Linus I think transactions should be done in kernel space. It's the only way to "enforce" them sanely. Besides, I'm not the one who first suggested this transaction idea. That was Hans, and it was in the whitepaper. I don't know if it's done yet -- I think it got put off for v4.1. But if it is going to be done anyway, doesn't it solve the consistency issue? I can understand wanting caching to work now, but which is the priority - -- caching or transactions? I vote transactions. The consistancy of fstab is far more important to me than the consistancy of foo.tgz/bar. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTahTXgHNmZLgCUhAQIumQ//SqEViCUbBcFJ3xsiQfEEUSe2gU+6dyJZ 7i3OeexnLRKrnxIsij5SmYTmGdYXhpaQWMrrjie8uFX0ocA0HTMsA1BzB4ZhtKnO sYc/jAZusGpAjnyy0t0Q/hwjoMRIsCRVwqnD17Qw8rcuwYT5yaABMboXPiOPjQqu JTkCeb+ATq5hF9VYL9TbcqaOycV7BMh+N7VXGPS/PIgeyYXyNwfIBWHrN8sjVQDA 3h9a9IEjEyReMdqlNbv73/wggGNqsmprDLIMFhMSPP4LXS5/xt1AhXW1R+mNDXjA ckNq1lLjD1iUIQ7gIPHjR1Vi5op5GaXqVtdJLXtTyvNuc3nM/yFzM0T7+hgO4rSj F0VkFFV8FbKRSVv1MsPv1O8EpFNpAs1uX+1g/vZ4ZHwPuOaMeXKT6rVTsr9WesfU NdPMi0bXF6fAHbpjw3k6v+Jg95Mg+w8bgYOEDptz6zRuex8xC+LPrXE9ZDrYX+nm +xG2wtMZyMr2y8wMp71RLz6Sj+Vq0cp4HNqccsMLfZ1OEWu2IzneBAmCSGQjLrxL 9coj0G3WpFwspTClh1DGBfuBJPS5O33iEsJ2w80bMhy/wKK5kZG/NyMAGQKazxAr +7Se6t2RGHIxX5/EEoPI/e0AKLxFuvPi9ilq4bJNNDBMUO2wOkB8V0GLGzFA/0F2 9ALDAaO/dg8= =pvxM -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 4:27 ` David Masover @ 2004-09-02 6:04 ` Linus Torvalds 2004-09-02 7:15 ` Hans Reiser 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 6:04 UTC (permalink / raw) To: David Masover Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 1 Sep 2004, David Masover wrote: > > I think transactions should be done in kernel space. It's the only way > to "enforce" them sanely. Well, they clearly can't, since it depends on the filesystem. Also, transactions have a tendency to be expensive, and people have a tendency to ask for more than you give them. Doing transactions on one file is only the beginning - you'll find people who want transactions across file boundaries etc. That's a basic fact when it comes to pretty much anything: you can always find people who want something better. You can't be all things to all people, so what you actually _want_ to do is to expose the _minimal_ set of capabilities that people can build on. So I definitely don't want this discussion to degenerate (any more than it lng since has ;) into what people _wish_ for. No, it should be a "what is the _least_ we can absolutely do" that solves real problems. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 6:04 ` Linus Torvalds @ 2004-09-02 7:15 ` Hans Reiser 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 7:55 ` Robin Rosenberg 0 siblings, 2 replies; 1835+ messages in thread From: Hans Reiser @ 2004-09-02 7:15 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > >. Doing transactions on one file is >only the beginning - you'll find people who want transactions across file >boundaries etc. > > Yup, that's what reiser4 is aiming at and where things get exciting for version control systems and the like. We are just missing the api code for it at the moment, all the infrastructure is there, and the api is in progress in sys_reiser4. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:15 ` Hans Reiser @ 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 8:58 ` Hans Reiser ` (2 more replies) 2004-09-02 7:55 ` Robin Rosenberg 1 sibling, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 7:36 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Hans Reiser wrote: > Linus Torvalds wrote: > > > >. Doing transactions on one file is > >only the beginning - you'll find people who want transactions across file > >boundaries etc. > > Yup, that's what reiser4 is aiming at and where things get exciting for > version control systems and the like. We are just missing the api code > for it at the moment, all the infrastructure is there, and the api is in > progress in sys_reiser4. But _my_ point is, no user program is going to take _advantage_ of anything that only one filesystem on one system offers. So there's no point. It's much saner (from _any_ app standpoint) to roll their own database in user space - that way it just works. In other words, nobody is really ever going to take advantage of this. This is _not_ how technical advanncement happens. The way you get people to take advantage of something is to have a nice gradual ramp-up, not a sudden new feature that they can't realistically use. So before you do transactions in the filesystem, you have to be able to do them in user space - and then make the filesystem version be _compatible_ with that user space library. That way you can get people to use the library ("hey, it works anywhere") and then you can get them to use your filesystem ("hey, if you use our super-duper filesystem, you can do what you are already doing five times faster"). See? You're starting at the wrong end. Give me a portable interface to use _first_. Then do transactions in the filesystem. (Now, as an academic exercise and as a learnign experience and prototyping work it migt make sense to do the filesystem first. But then you have to really _consider_ it a prototype, and realize that to make it useful you'd still have to do the user thing before you can "sell" people on the idea). Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:36 ` Linus Torvalds @ 2004-09-02 8:58 ` Hans Reiser 2004-09-02 9:12 ` Oliver Hunt ` (6 more replies) 2004-09-02 9:04 ` Hans Reiser 2004-09-02 12:50 ` James Bruce 2 siblings, 7 replies; 1835+ messages in thread From: Hans Reiser @ 2004-09-02 8:58 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > But _my_ point is, no user program is going to take _advantage_ of > >anything that only one filesystem on one system offers. > > Apple does not have this problem.... and yes, the apps will take advantage of it, which is different from depending on it. If you use the wrong fs you will lose some of the features of the app. For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice (excepting Clearcase, which priced itself into a niche market). It is 25 years past time for someone to change things. That someone will have first mover advantage, and the more little semantic features possessed the more lure there will be to use it which will increase market share which will lure more apps into depending on it and in a few years the other filesystems will (deservedly) have only a small market share because the apps won't all work on them. Besides, there are enhancements which are simply compelling. You can write a dramatically better performance version control system with a much simpler design if the FS is atomic. Our transaction manager first draft was written by a version control guy, and he would probably be happy to tell you how lack of atomicity other than rename makes version control software design hideous. We have the performance lead. By next year we will be stable enough for mission critical servers, and then we start the serious semantic enhancements. If you don't embrace progress, then you doom Linux to following behind, because the guys at Apple are pretty aggressive now that Jobs is back, and they WILL change the semantics, and they will do so in compelling ways, and Linux will be reduced to aping them when it should be leading them. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser @ 2004-09-02 9:12 ` Oliver Hunt 2004-09-02 9:26 ` Helge Hafting 2004-09-02 9:52 ` Spam ` (5 subsequent siblings) 6 siblings, 1 reply; 1835+ messages in thread From: Oliver Hunt @ 2004-09-02 9:12 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List How would we go about finding out how many data forks were in a file? Because in order to be able to retrieve data from a fork we would need to know that the fork were there. Currently this would imply that we go looking through mtab or some such to find out what fs we're running on, which seems ugly. Alternatively we go through the _exciting_ task of making every other fs (with the exceptions of ntfs, and whatever it is that macs use, which would need there own custom code) and add code that effectively goes getNumForks(fileref){ return 1;} or add a new open call that can take a fork number... either way we have to add a new syscall, that doesn't conform to any real standard(though i suppose it would be possible to use macOS/windows style fork iopening interface) I personally like the concept of having multiple forks in a file, but in this case I'm inclined towards usermode first, then if it takes off add kernel level support. --Oliver Hunt On Thu, 02 Sep 2004 01:58:30 -0700, Hans Reiser <reiser@namesys.com> wrote: > Linus Torvalds wrote: > > > But _my_ point is, no user program is going to take _advantage_ of > > > >anything that only one filesystem on one system offers. > > > > > Apple does not have this problem.... > > and yes, the apps will take advantage of it, which is different from > depending on it. If you use the wrong fs you will lose some of the > features of the app. > > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice (excepting Clearcase, which priced itself > into a niche market). It is 25 years past time for someone to change > things. That someone will have first mover advantage, and the more > little semantic features possessed the more lure there will be to use it > which will increase market share which will lure more apps into > depending on it and in a few years the other filesystems will > (deservedly) have only a small market share because the apps won't all > work on them. > > Besides, there are enhancements which are simply compelling. You can > write a dramatically better performance version control system with a > much simpler design if the FS is atomic. Our transaction manager > first draft was written by a version control guy, and he would probably > be happy to tell you how lack of atomicity other than rename makes > version control software design hideous. > > We have the performance lead. By next year we will be stable enough for > mission critical servers, and then we start the serious semantic > enhancements. > > If you don't embrace progress, then you doom Linux to following behind, > because the guys at Apple are pretty aggressive now that Jobs is back, > and they WILL change the semantics, and they will do so in compelling > ways, and Linux will be reduced to aping them when it should be leading > them. > > Hans > > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:12 ` Oliver Hunt @ 2004-09-02 9:26 ` Helge Hafting 0 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-09-02 9:26 UTC (permalink / raw) To: Oliver Hunt Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Oliver Hunt wrote: >How would we go about finding out how many data forks were in a file? >Because in order to be able to retrieve data from a fork we would need >to know that the fork were there. Currently this would imply that we >go looking through mtab or some such to find out what fs we're running >on, which seems ugly. > > > Depends on how the forks eventually get implemented. With the file-as-directory concept, all you need is to look at the file's directory part to see what is there. (The forks, implemented as files in a subdirectory.) It is done the same way as for an ordinary directory, so nothing "new". >Alternatively we go through the _exciting_ task of making every other >fs (with the exceptions of ntfs, and whatever it is that macs use, >which would need there own custom code) and add code that effectively >goes > >getNumForks(fileref){ return 1;} > > > Necessary if some other mechanism is used, sure. Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives @ 2004-09-02 9:26 ` Helge Hafting 0 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-09-02 9:26 UTC (permalink / raw) To: Oliver Hunt Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Oliver Hunt wrote: >How would we go about finding out how many data forks were in a file? >Because in order to be able to retrieve data from a fork we would need >to know that the fork were there. Currently this would imply that we >go looking through mtab or some such to find out what fs we're running >on, which seems ugly. > > > Depends on how the forks eventually get implemented. With the file-as-directory concept, all you need is to look at the file's directory part to see what is there. (The forks, implemented as files in a subdirectory.) It is done the same way as for an ordinary directory, so nothing "new". >Alternatively we go through the _exciting_ task of making every other >fs (with the exceptions of ntfs, and whatever it is that macs use, >which would need there own custom code) and add code that effectively >goes > >getNumForks(fileref){ return 1;} > > > Necessary if some other mechanism is used, sure. Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:26 ` Helge Hafting (?) @ 2004-09-02 9:45 ` Oliver Hunt 2004-09-02 10:02 ` Spam -1 siblings, 1 reply; 1835+ messages in thread From: Oliver Hunt @ 2004-09-02 9:45 UTC (permalink / raw) To: Helge Hafting Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Depends on how the forks eventually get implemented. > With the file-as-directory concept, all you need is to > look at the file's directory part to see what is there. (The forks, > implemented as files in a subdirectory.) It is done the same way > as for an ordinary directory, so nothing "new". > This still doesn't solve the problem of how we distinguish between a multi-fork file and a directory, for those old programs(ie. almost all that currently exist), to be able to access meaningful data we would need to either return just the primary fork, a serialized version of all forks in the file, or make the file look more or less identical to a directory. The first option could cause problems when transfering files between different filesystems, the second results in programs getting metadata they can't handle, and the third option results in few of the advantages over, well, directories... And even those applications that could handle the fork information nicely would need to fs type to find out whether they were handling a directory or a multi-forked file... As I say I like the idea, but I can't see anyway of implementing it in a way that is useful without first putting considerable effort into at least the VFS if not all the actual fs drivers. --Oliver Hunt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:45 ` Oliver Hunt @ 2004-09-02 10:02 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 10:02 UTC (permalink / raw) To: Oliver Hunt Cc: Helge Hafting, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List >> Depends on how the forks eventually get implemented. >> With the file-as-directory concept, all you need is to >> look at the file's directory part to see what is there. (The forks, >> implemented as files in a subdirectory.) It is done the same way >> as for an ordinary directory, so nothing "new". >> > This still doesn't solve the problem of how we distinguish between a > multi-fork file > and a directory, for those old programs(ie. almost all that currently > exist), to be able to access meaningful data we would need to either > return just the primary fork, a serialized version of all forks in the > file, or make the file look more or less identical to a directory. Well. wasn't the idea that unless programs specifically tried to open the file-as-dir as a directory it would look like a file? ls -F would show it as file. Or have I understood wrong? > The first option could cause problems when transfering files between > different filesystems, The meta-data should be deleted if it the file is copied or moved to a medium that doesn't support it. However a _warning_ may be shown to the user if there is risk to loose data. I do not think we should not implement something like this because most other filesystems in Linux doesn't support it. Think of other limitations there is in various ones. Large file support. Limited amount of files, length of file names, etc. > the second results in programs getting metadata they can't handle, and > the third option results in few of the advantages over, well, > directories... And even those applications that could handle the fork > information nicely would need to fs type to find out whether they were > handling a directory or a multi-forked file... No, if an application supported the file-as-dir then it could simply try to open the file as a directory. It would fail on systems that didn't support it. The meta-data and file streams would be seen as ordinary files. If applications support the contents or not isn't really relevant, is it?. > As I say I like the idea, but I can't see anyway of implementing it in > a way that is useful without first putting considerable effort into at > least the VFS if not all the actual fs drivers. Indeed. It is important that something like this gets implemented as a transparent way as possible. If it could be done in a general way so other filesystems like ext3/4 can eventually support it then that would be wonderful. I do not, however, think that we should block it in reiser4 because no other filesystems support it. ~S > --Oliver Hunt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives @ 2004-09-02 10:02 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 10:02 UTC (permalink / raw) To: Oliver Hunt Cc: Helge Hafting, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List >> Depends on how the forks eventually get implemented. >> With the file-as-directory concept, all you need is to >> look at the file's directory part to see what is there. (The forks, >> implemented as files in a subdirectory.) It is done the same way >> as for an ordinary directory, so nothing "new". >> > This still doesn't solve the problem of how we distinguish between a > multi-fork file > and a directory, for those old programs(ie. almost all that currently > exist), to be able to access meaningful data we would need to either > return just the primary fork, a serialized version of all forks in the > file, or make the file look more or less identical to a directory. Well. wasn't the idea that unless programs specifically tried to open the file-as-dir as a directory it would look like a file? ls -F would show it as file. Or have I understood wrong? > The first option could cause problems when transfering files between > different filesystems, The meta-data should be deleted if it the file is copied or moved to a medium that doesn't support it. However a _warning_ may be shown to the user if there is risk to loose data. I do not think we should not implement something like this because most other filesystems in Linux doesn't support it. Think of other limitations there is in various ones. Large file support. Limited amount of files, length of file names, etc. > the second results in programs getting metadata they can't handle, and > the third option results in few of the advantages over, well, > directories... And even those applications that could handle the fork > information nicely would need to fs type to find out whether they were > handling a directory or a multi-forked file... No, if an application supported the file-as-dir then it could simply try to open the file as a directory. It would fail on systems that didn't support it. The meta-data and file streams would be seen as ordinary files. If applications support the contents or not isn't really relevant, is it?. > As I say I like the idea, but I can't see anyway of implementing it in > a way that is useful without first putting considerable effort into at > least the VFS if not all the actual fs drivers. Indeed. It is important that something like this gets implemented as a transparent way as possible. If it could be done in a general way so other filesystems like ext3/4 can eventually support it then that would be wonderful. I do not, however, think that we should block it in reiser4 because no other filesystems support it. ~S > --Oliver Hunt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 10:02 ` Spam (?) @ 2004-09-02 11:00 ` Giuliano Pochini -1 siblings, 0 replies; 1835+ messages in thread From: Giuliano Pochini @ 2004-09-02 11:00 UTC (permalink / raw) To: Spam; +Cc: linux-kernel On Thu, 2 Sep 2004, Spam wrote: > Well. wasn't the idea that unless programs specifically tried to > open the file-as-dir as a directory it would look like a file? > > ls -F would show it as file. Or have I understood wrong? Yes, otherwise apps that do if (S_ISDIR()) { .. } else if (S_ISREG()) { .. } would behave differently from apps that check the file in different order, and they would probably break because in a regular fs that order is not important. The goal is to keep the semantic changes as hidden as possible for apps that don't know about the new features. -- Giuliano. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 10:02 ` Spam (?) (?) @ 2004-09-02 14:38 ` Christian Mayrhuber -1 siblings, 0 replies; 1835+ messages in thread From: Christian Mayrhuber @ 2004-09-02 14:38 UTC (permalink / raw) To: reiserfs-list, Spam; +Cc: linux-fsdevel, linux-kernel On Thursday 02 September 2004 12:02, Spam wrote: > > As I say I like the idea, but I can't see anyway of implementing it in > > a way that is useful without first putting considerable effort into at > > least the VFS if not all the actual fs drivers. > > Indeed. It is important that something like this gets implemented as > a transparent way as possible. If it could be done in a general way > so other filesystems like ext3/4 can eventually support it then that > would be wonderful. I do not, however, think that we should block it > in reiser4 because no other filesystems support it. What about extending the namespace with leading ".." and "...". In Unix names starting with a "." already have the meaning of being a hidden/config entry. A name starting ".." means streams/metainformation, etc. something belonging to the file/directory that should get backed up, copied, etc. There could be a "..streams", "..metas" or a "..acl" entry. If it starts with "..." it means some system specific information, like the name of the hash algorithm used for the current directory. It's basically information that is not portable and not required by applications, like "...fsplugins". There is still the big problem of how to copy files with their associated streams or meta information onto a standard unix filesystem as the file/directory duality cannot be expressed. (It's forbidden to have a directory + file with the same name in the same directory) Maybe copy could create something like a ".#filename" directory for this kind of information if the advanced features are not supported on the target filesystem. This is neither nice nor clean, but at least you don't loose information. I do not suggest that the kernel should simulate the advanced features with ".#filename" directories, it's more a backup/restore thing that could work if it's written down properly. -- lg, Chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser @ 2004-09-02 9:52 ` Spam 2004-09-02 9:52 ` Spam ` (5 subsequent siblings) 6 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 9:52 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Linus Torvalds wrote: >> But _my_ point is, no user program is going to take _advantage_ of >> >>anything that only one filesystem on one system offers. >> >> > Apple does not have this problem.... > and yes, the apps will take advantage of it, which is different from > depending on it. If you use the wrong fs you will lose some of the > features of the app. > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice (excepting Clearcase, which priced itself > into a niche market). It is 25 years past time for someone to change > things. That someone will have first mover advantage, and the more > little semantic features possessed the more lure there will be to use it > which will increase market share which will lure more apps into > depending on it and in a few years the other filesystems will > (deservedly) have only a small market share because the apps won't all > work on them. > Besides, there are enhancements which are simply compelling. You can > write a dramatically better performance version control system with a > much simpler design if the FS is atomic. Our transaction manager > first draft was written by a version control guy, and he would probably > be happy to tell you how lack of atomicity other than rename makes > version control software design hideous. Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :) It allows you to restore previous versions of the file even if you delete or overwrite it. Features like this do make a good point and helps protecting data - something that is important IMHO. http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp > We have the performance lead. By next year we will be stable enough for > mission critical servers, and then we start the serious semantic > enhancements. > If you don't embrace progress, then you doom Linux to following behind, > because the guys at Apple are pretty aggressive now that Jobs is back, > and they WILL change the semantics, and they will do so in compelling > ways, and Linux will be reduced to aping them when it should be leading > them. > Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives @ 2004-09-02 9:52 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 9:52 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Linus Torvalds wrote: >> But _my_ point is, no user program is going to take _advantage_ of >> >>anything that only one filesystem on one system offers. >> >> > Apple does not have this problem.... > and yes, the apps will take advantage of it, which is different from > depending on it. If you use the wrong fs you will lose some of the > features of the app. > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice (excepting Clearcase, which priced itself > into a niche market). It is 25 years past time for someone to change > things. That someone will have first mover advantage, and the more > little semantic features possessed the more lure there will be to use it > which will increase market share which will lure more apps into > depending on it and in a few years the other filesystems will > (deservedly) have only a small market share because the apps won't all > work on them. > Besides, there are enhancements which are simply compelling. You can > write a dramatically better performance version control system with a > much simpler design if the FS is atomic. Our transaction manager > first draft was written by a version control guy, and he would probably > be happy to tell you how lack of atomicity other than rename makes > version control software design hideous. Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :) It allows you to restore previous versions of the file even if you delete or overwrite it. Features like this do make a good point and helps protecting data - something that is important IMHO. http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp > We have the performance lead. By next year we will be stable enough for > mission critical servers, and then we start the serious semantic > enhancements. > If you don't embrace progress, then you doom Linux to following behind, > because the guys at Apple are pretty aggressive now that Jobs is back, > and they WILL change the semantics, and they will do so in compelling > ways, and Linux will be reduced to aping them when it should be leading > them. > Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam (?) @ 2004-09-02 9:32 ` Alan Cox -1 siblings, 0 replies; 1835+ messages in thread From: Alan Cox @ 2004-09-02 9:32 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 10:52, Spam wrote: > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) Netapps have done this for years with NFS, Subversion does it with DAV, clearcase does it with NFS. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam (?) (?) @ 2004-09-02 10:27 ` V13 2004-09-02 14:35 ` John Stoffel -1 siblings, 1 reply; 1835+ messages in thread From: V13 @ 2004-09-02 10:27 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thursday 02 September 2004 12:52, Spam wrote: > > Linus Torvalds wrote: > > Besides, there are enhancements which are simply compelling. You can > > write a dramatically better performance version control system with a > > much simpler design if the FS is atomic. Our transaction manager > > first draft was written by a version control guy, and he would probably > > be happy to tell you how lack of atomicity other than rename makes > > version control software design hideous. > > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) > > It allows you to restore previous versions of the file even if you > delete or overwrite it. > > Features like this do make a good point and helps protecting data - > something that is important IMHO. I believe you mean something simillar to: file1.txt;1 file1.txt;2 file1.txt;3 (yeap, it's VMS) where you'll have to cleanup old versions when you don't need them any more... AFAIK that this is older than HDDs <<V13>> ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 10:27 ` V13 @ 2004-09-02 14:35 ` John Stoffel 2004-09-02 18:42 ` Hubert Chan 0 siblings, 1 reply; 1835+ messages in thread From: John Stoffel @ 2004-09-02 14:35 UTC (permalink / raw) To: V13 Cc: Spam, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List V13> I believe you mean something simillar to: V13> file1.txt;1 V13> file1.txt;2 V13> file1.txt;3 (yeap, it's VMS) Or TOPS-20, a precursor to VMS in some ways. It was a nice feature. V13> where you'll have to cleanup old versions when you don't need V13> them any more... AFAIK that this is older than HDDs It was usually an automatic cleanup past a certain point or if you went over disk quota. In any case, while I do like this feature, I'm not sure how we would cleanly implement this inside the unix namespace, or if inside a new namespace, how that new namespace would be joined with a standard Unix one. John ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 14:35 ` John Stoffel @ 2004-09-02 18:42 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-02 18:42 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "John" == John Stoffel <stoffel@lucent.com> writes: V13> I believe you mean something simillar to: V13> file1.txt;1 V13> file1.txt;2 V13> file1.txt;3 (yeap, it's VMS) [...] John> In any case, while I do like this feature, I'm not sure how we John> would cleanly implement this inside the unix namespace, or if John> inside a new namespace, how that new namespace would be joined John> with a standard Unix one. file1.txt/..version/1 or, for those version control systems like svn and arch where the version number refers to the version of the whole project tree: project/..version/1/file1.txt You could even export strange things like: project/..by-date/2004-08-24/file1.txt to see what the file looked like on August 24. Or project/..diff/3-1/file1.txt to see a diff between version 3 and 1. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives @ 2004-09-02 18:42 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-02 18:42 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "John" == John Stoffel <stoffel@lucent.com> writes: V13> I believe you mean something simillar to: V13> file1.txt;1 V13> file1.txt;2 V13> file1.txt;3 (yeap, it's VMS) [...] John> In any case, while I do like this feature, I'm not sure how we John> would cleanly implement this inside the unix namespace, or if John> inside a new namespace, how that new namespace would be joined John> with a standard Unix one. file1.txt/..version/1 or, for those version control systems like svn and arch where the version number refers to the version of the whole project tree: project/..version/1/file1.txt You could even export strange things like: project/..by-date/2004-08-24/file1.txt to see what the file looked like on August 24. Or project/..diff/3-1/file1.txt to see a diff between version 3 and 1. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam @ 2004-09-02 11:09 ` Oliver Neukum -1 siblings, 0 replies; 1835+ messages in thread From: Oliver Neukum @ 2004-09-02 11:09 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Am Donnerstag, 2. September 2004 11:52 schrieb Spam: > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) > > It allows you to restore previous versions of the file even if you > delete or overwrite it. There's no need to do that in kernel, unless you want to be able to force it unto users. Regards Oliver ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives @ 2004-09-02 11:09 ` Oliver Neukum 0 siblings, 0 replies; 1835+ messages in thread From: Oliver Neukum @ 2004-09-02 11:09 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Am Donnerstag, 2. September 2004 11:52 schrieb Spam: > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) > > It allows you to restore previous versions of the file even if you > delete or overwrite it. There's no need to do that in kernel, unless you want to be able to force it unto users. Regards Oliver ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 11:09 ` Oliver Neukum @ 2004-09-02 11:11 ` Spam -1 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 11:11 UTC (permalink / raw) To: Oliver Neukum Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Am Donnerstag, 2. September 2004 11:52 schrieb Spam: >> Btw, version control for ordinary files would be a great feature. I >> think something like it is available through Windows 2000/3 server. >> Isn't it called "Shadow Copies". It works over network shares. :) >> >> It allows you to restore previous versions of the file even if you >> delete or overwrite it. > There's no need to do that in kernel, unless you want to be able > to force it unto users. Exactly ;) Difference with having it in just certain applications like Gnome-VFS etc is that it would work with all applications, over SMB shares etc. This is a great advantage and benefit for many desktop users but also in a corporate environment. Even though there are backups of users documents it is a pain to restore them every time they loose the original file. Besides, the backup is never as fresh either. > Regards > Oliver ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives @ 2004-09-02 11:11 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 11:11 UTC (permalink / raw) To: Oliver Neukum Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Am Donnerstag, 2. September 2004 11:52 schrieb Spam: >> Btw, version control for ordinary files would be a great feature. I >> think something like it is available through Windows 2000/3 server. >> Isn't it called "Shadow Copies". It works over network shares. :) >> >> It allows you to restore previous versions of the file even if you >> delete or overwrite it. > There's no need to do that in kernel, unless you want to be able > to force it unto users. Exactly ;) Difference with having it in just certain applications like Gnome-VFS etc is that it would work with all applications, over SMB shares etc. This is a great advantage and benefit for many desktop users but also in a corporate environment. Even though there are backups of users documents it is a pain to restore them every time they loose the original file. Besides, the backup is never as fresh either. > Regards > Oliver ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam ` (3 preceding siblings ...) (?) @ 2004-09-02 15:40 ` Horst von Brand -1 siblings, 0 replies; 1835+ messages in thread From: Horst von Brand @ 2004-09-02 15:40 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: [...] > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) Right. And when space runs low, the old copies get the axe. And then you need the old version... and are royally screwed because you trusted the system to save it for you while willy-nilly fooling around. I never liked the MS-DOS "undelete" feature, because I saw too many people burned. It is _less_ data security than "deleted file is gone for good", in the end sum. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam @ 2004-09-02 18:21 ` Lee Revell -1 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 18:21 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 05:52, Spam wrote: > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) > > It allows you to restore previous versions of the file even if you > delete or overwrite it. > > Features like this do make a good point and helps protecting data - > something that is important IMHO. > > http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp > Netapp's WAFL has had this for ages, they are called snapshots, and they are one of the most useful file system features ever. But Microsoft certainly did not invent them. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives @ 2004-09-02 18:21 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 18:21 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 05:52, Spam wrote: > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) > > It allows you to restore previous versions of the file even if you > delete or overwrite it. > > Features like this do make a good point and helps protecting data - > something that is important IMHO. > > http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp > Netapp's WAFL has had this for ages, they are called snapshots, and they are one of the most useful file system features ever. But Microsoft certainly did not invent them. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser 2004-09-02 9:12 ` Oliver Hunt 2004-09-02 9:52 ` Spam @ 2004-09-02 10:05 ` Bostjan Skufca @ domenca.com 2004-09-02 14:24 ` John Stoffel ` (3 subsequent siblings) 6 siblings, 0 replies; 1835+ messages in thread From: Bostjan Skufca @ domenca.com @ 2004-09-02 10:05 UTC (permalink / raw) To: reiserfs-list I think Hans' idea about further evolution is right. (2.7 which will lead into 3.0 maybe?) Regards, Bostjan On Thursday 02 of September 2004 10:58, Hans Reiser wrote: > Linus Torvalds wrote: > > But _my_ point is, no user program is going to take _advantage_ of > > > >anything that only one filesystem on one system offers. > > Apple does not have this problem.... > > and yes, the apps will take advantage of it, which is different from > depending on it. If you use the wrong fs you will lose some of the > features of the app. > > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice (excepting Clearcase, which priced itself > into a niche market). It is 25 years past time for someone to change > things. That someone will have first mover advantage, and the more > little semantic features possessed the more lure there will be to use it > which will increase market share which will lure more apps into > depending on it and in a few years the other filesystems will > (deservedly) have only a small market share because the apps won't all > work on them. > > Besides, there are enhancements which are simply compelling. You can > write a dramatically better performance version control system with a > much simpler design if the FS is atomic. Our transaction manager > first draft was written by a version control guy, and he would probably > be happy to tell you how lack of atomicity other than rename makes > version control software design hideous. > > We have the performance lead. By next year we will be stable enough for > mission critical servers, and then we start the serious semantic > enhancements. > > If you don't embrace progress, then you doom Linux to following behind, > because the guys at Apple are pretty aggressive now that Jobs is back, > and they WILL change the semantics, and they will do so in compelling > ways, and Linux will be reduced to aping them when it should be leading > them. > > Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser ` (2 preceding siblings ...) 2004-09-02 10:05 ` Bostjan Skufca @ domenca.com @ 2004-09-02 14:24 ` John Stoffel 2004-09-02 15:36 ` Alan Cox 2004-09-02 15:32 ` Horst von Brand ` (2 subsequent siblings) 6 siblings, 1 reply; 1835+ messages in thread From: John Stoffel @ 2004-09-02 14:24 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans> For 30 years nothing much has happened in Unix filesystem Hans> semantics because of sheer cowardice (excepting Clearcase, which Hans> priced itself into a niche market). Keeping a consistent namespace is cowardice? And ClearCase is an example of an application which sets up it's own namespace and grafts it onto the standard Unix one. Most standard Unix tools work just fine inside clearcase Views, but to manage the metadata (xattrs, or whatever you want to call them), you need to use the 'cleartool' command. Hmmm... sounds like 'runat' to me. Hans> It is 25 years past time for someone to change things. That Hans> someone will have first mover advantage, and the more little Hans> semantic features possessed the more lure there will be to use Hans> it which will increase market share which will lure more apps Hans> into depending on it and in a few years the other filesystems Hans> will (deservedly) have only a small market share because the Hans> apps won't all work on them. This is all pure marketing speak and economic theory. Show us the *technical* advantages, not just wishful thinking. Hans> Besides, there are enhancements which are simply compelling. Hans> You can write a dramatically better performance version control Hans> system with a much simpler design if the FS is atomic. Define atomic please, with state diagrams and clear examples. Hans> We have the performance lead. By next year we will be stable Hans> enough for mission critical servers, and then we start the Hans> serious semantic enhancements. I think you've got it backwards. Make your serious semantic enhancements first, then make it stable, then make it fast. Because when you change the semantics, you break all kinds of things and then it doesn't matter how fast you are. Hans> If you don't embrace progress, then you doom Linux to following Hans> behind, because the guys at Apple are pretty aggressive now that Hans> Jobs is back, and they WILL change the semantics, and they will Hans> do so in compelling ways, and Linux will be reduced to aping Hans> them when it should be leading them. Monkey see, Monkey do then. I'd like to point out another successful company which has extended the standard Unix namespace and that's 'Network Appliance' with it's .snapshot directory structure. It's a great idea and allows my users to restore files from snapshots without me having to think about it. But it still causes problems since when I use rsync to move data, I need to put in stuff like: rsync -az --exclude ".snapshot" --delete --delete-excluded <src> <dest> to make sure it doesn't descend into that directory of previous versions and try to copy them over as well. And of course now now other apps can use the .snapshot name. But what if other vendors aped (ook ook) this decided to use their own names? Who decides which gets priority? I think you need to go back and re-read Pike's paper on namespaces that you pointed to before and mull it over. And look at how simplicity is inherently powerful. If the design is too complicated, you're probably doing it wrong. John ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 14:24 ` John Stoffel @ 2004-09-02 15:36 ` Alan Cox 0 siblings, 0 replies; 1835+ messages in thread From: Alan Cox @ 2004-09-02 15:36 UTC (permalink / raw) To: John Stoffel Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 15:24, John Stoffel wrote: > This is all pure marketing speak and economic theory. I wouldn't grace it with the label economic theory. Lets be honest about this.. its pure -research-. Research is good but don't screw around with Linux semantics until you've done the research, figured out how to also make it useful to real users and proved its a good idea despite the past evidence from things like MacOS Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser ` (3 preceding siblings ...) 2004-09-02 14:24 ` John Stoffel @ 2004-09-02 15:32 ` Horst von Brand 2004-09-02 17:43 ` Linus Torvalds 2004-09-02 18:22 ` Martin J. Bligh 6 siblings, 0 replies; 1835+ messages in thread From: Horst von Brand @ 2004-09-02 15:32 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser <reiser@namesys.com> said: > Linus Torvalds wrote: > > But _my_ point is, no user program is going to take _advantage_ of > >anything that only one filesystem on one system offers. > Apple does not have this problem.... ... because Apple customers are a captive lot... > and yes, the apps will take advantage of it, which is different from > depending on it. If you use the wrong fs you will lose some of the > features of the app. ... which is undesirable to say the very least for cross-platform Unix (even worse, Unix + Windows) applications, so applications won't ever use it (until Linux + super-duper filesystem is more than, say, 80% of the market). I.e., it won't happen soon enough for this to make sense right now. Perhaps in 13 more years... > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice Careful threading there... calling respected people names without any proof isn't exactly great PR. > (excepting Clearcase, which priced itself > into a niche market). It is 25 years past time for someone to change > things. Your opinion. Some people disagree. > That someone will have first mover advantage, and the more > little semantic features possessed the more lure there will be to use it > which will increase market share which will lure more apps into depending > on it and in a few years the other filesystems will (deservedly) have > only a small market share because the apps won't all work on them. You seem to be blinded by the "success effect": You only ever see the ones who succeed, you don't see the masses who tried to be "first movers" and failed utterly. > Besides, there are enhancements which are simply compelling. You can > write a dramatically better performance version control system with a > much simpler design if the FS is atomic. Iff the performance of the FS is better _at doing what the system wants than the system itself_. That the CVS is simpler/faster is no use whatsoever if the underlying OS is more complex/slower. Besides, what is ideal for CVS isn't necesarily good for word processing or databases, and probably irrelevant or at least overkill for a lot of other applications, which all end up paying the costs. Concentrate on giving what _most_ applications want, make it possible for applications with weird requirements to do their job. > Our transaction manager > first draft was written by a version control guy, and he would probably > be happy to tell you how lack of atomicity other than rename makes > version control software design hideous. Right. But it doesn't make the design of text editing, or word processing, or mail handling, or... hideous, does it? Having a strong background in an area does put blinds on one's eyes. > We have the performance lead. I recently saw (preliminary) numbers directly contradicting this somewhere. > By next year we will be stable enough for > mission critical servers, and then we start the serious semantic > enhancements. By which the performance lead (if any) will do down the drain. There ain't no free lunch. > If you don't embrace progress, then you doom Linux to following behind, > because the guys at Apple are pretty aggressive now that Jobs is back, > and they WILL change the semantics, and they will do so in compelling > ways, and Linux will be reduced to aping them when it should be leading > them. Why on earth should Linux copy a niche system? By now Linux's marketshare is _much_ larger than Apple's... I'd say it is the other way around: Apple tries desperately to differentiate itself from MSFT (and traditional Unix + Linux) in order not to dissapear (and keep their grip on their users). If they are doing it right, only time will tell. But that is no reason to screw up Linux' strengths. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser ` (4 preceding siblings ...) 2004-09-02 15:32 ` Horst von Brand @ 2004-09-02 17:43 ` Linus Torvalds 2004-09-02 18:22 ` Martin J. Bligh 6 siblings, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 17:43 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Hans Reiser wrote: > > Apple does not have this problem.... Deep, Hans. Deep. You're ignoring the issue. You're flaming about not "embracing progress", when I was talking about how progress happens. You don't seem to understand _how_ people embrace progress, Hans. Railing at windmills isn't real smart. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser ` (5 preceding siblings ...) 2004-09-02 17:43 ` Linus Torvalds @ 2004-09-02 18:22 ` Martin J. Bligh 2004-09-02 18:38 ` Christer Weinigel 2004-09-02 19:52 ` Steve Bergman 6 siblings, 2 replies; 1835+ messages in thread From: Martin J. Bligh @ 2004-09-02 18:22 UTC (permalink / raw) To: Hans Reiser, Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice Or because it works fine, and isn't broken. M. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 18:22 ` Martin J. Bligh @ 2004-09-02 18:38 ` Christer Weinigel 2004-09-02 19:02 ` Christian Iversen 2004-09-02 19:52 ` Steve Bergman 1 sibling, 1 reply; 1835+ messages in thread From: Christer Weinigel @ 2004-09-02 18:38 UTC (permalink / raw) To: Martin J. Bligh Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List "Martin J. Bligh" <mbligh@aracnet.com> writes: > > For 30 years nothing much has happened in Unix filesystem semantics > > because of sheer cowardice > > Or because it works fine, and isn't broken. I've heard the same argument a lot of times. People complaining that Unix is so seventies because it sticks to the old boring philosophy of everything is a file and that a file is a stream of bytes, nothing more. Modern operating systems such as VMS with basic database handling in the OS itself, or MacOS or NT with named streams is so much more modern. Why don't we get with the times? It may be because just because of the simplicity it's fairly easy to use, harder to break och does one thing well. If you want structured storage, use a database, on top of the low level primitives, or use multiple files in a directory. Why complicate things? /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 18:38 ` Christer Weinigel @ 2004-09-02 19:02 ` Christian Iversen 0 siblings, 0 replies; 1835+ messages in thread From: Christian Iversen @ 2004-09-02 19:02 UTC (permalink / raw) To: reiserfs-list On Thursday 02 September 2004 20:38, Christer Weinigel wrote: > "Martin J. Bligh" <mbligh@aracnet.com> writes: > > > For 30 years nothing much has happened in Unix filesystem semantics > > > because of sheer cowardice > > > > Or because it works fine, and isn't broken. > > I've heard the same argument a lot of times. People complaining that > Unix is so seventies because it sticks to the old boring philosophy of > everything is a file and that a file is a stream of bytes, nothing > more. Modern operating systems such as VMS with basic database > handling in the OS itself, or MacOS or NT with named streams is so > much more modern. Why don't we get with the times? > > It may be because just because of the simplicity it's fairly easy to > use, harder to break och does one thing well. If you want structured > storage, use a database, on top of the low level primitives, or use > multiple files in a directory. Why complicate things? Sounds very much like the reason for using only triangle primitives on modern 3D-graphics hardware. And that works :) -- Regards, Christian Iversen ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 18:22 ` Martin J. Bligh @ 2004-09-02 19:52 ` Steve Bergman 2004-09-02 19:52 ` Steve Bergman 1 sibling, 0 replies; 1835+ messages in thread From: Steve Bergman @ 2004-09-02 19:52 UTC (permalink / raw) To: Martin J. Bligh Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote: > > For 30 years nothing much has happened in Unix filesystem semantics > > because of sheer cowardice > > Or because it works fine, and isn't broken. OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a designer of filesystems. I'm just a guy that puts together and supports Linux systems for my customers. In following this thread, I may be missing huge chunks of concept. However, a few things are becoming clear to me: 1. The file as directory thing adds complexity that the administrator has to deal with. Symlinks are useful, but it's still aggravating to tar off a directory structure, take it somewhere, and then realize that all you have is links to something not in the archive because you didn't get your tar switches just right. Now we're talking about adding another set of "files which are not really files" to the semantics. More complexity. I'll take simplicity over some ivory tower ideal of "unified name space" any day. 2. The use of multiple streams within files by Linux apps would make Linux as cross-platform unfriendly as MS is trying to be. Say these features start getting used and you copy an OO.org document from a Linux box to a BSD box. It's broken. Of course, OO.org wouldn't use the streams in the first place because it would destroy their cross platform portability. So what's the point? No one who cares about cross platform portability can use it. Everyone who doesn't care about cross platform portability please raise your hand. 3. MS does require attributes and multiple streams, which makes these features important (even essential) to Samba, and Samba alone. Samba is important to Linux, so this can't be ignored. (Here I am implicitly assuming that Samba will need kernel support for this to do it right.) So it seems to me that the only real consideration is giving Samba what it needs without making the semantics one bit more complex than absolutely necessary. It might even be wise to discourage use of these ambiguous new objects by the casual application programmer. Then again, maybe I just have tunnel vision... -Steve Bergman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives @ 2004-09-02 19:52 ` Steve Bergman 0 siblings, 0 replies; 1835+ messages in thread From: Steve Bergman @ 2004-09-02 19:52 UTC (permalink / raw) To: Martin J. Bligh Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote: > > For 30 years nothing much has happened in Unix filesystem semantics > > because of sheer cowardice > > Or because it works fine, and isn't broken. OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a designer of filesystems. I'm just a guy that puts together and supports Linux systems for my customers. In following this thread, I may be missing huge chunks of concept. However, a few things are becoming clear to me: 1. The file as directory thing adds complexity that the administrator has to deal with. Symlinks are useful, but it's still aggravating to tar off a directory structure, take it somewhere, and then realize that all you have is links to something not in the archive because you didn't get your tar switches just right. Now we're talking about adding another set of "files which are not really files" to the semantics. More complexity. I'll take simplicity over some ivory tower ideal of "unified name space" any day. 2. The use of multiple streams within files by Linux apps would make Linux as cross-platform unfriendly as MS is trying to be. Say these features start getting used and you copy an OO.org document from a Linux box to a BSD box. It's broken. Of course, OO.org wouldn't use the streams in the first place because it would destroy their cross platform portability. So what's the point? No one who cares about cross platform portability can use it. Everyone who doesn't care about cross platform portability please raise your hand. 3. MS does require attributes and multiple streams, which makes these features important (even essential) to Samba, and Samba alone. Samba is important to Linux, so this can't be ignored. (Here I am implicitly assuming that Samba will need kernel support for this to do it right.) So it seems to me that the only real consideration is giving Samba what it needs without making the semantics one bit more complex than absolutely necessary. It might even be wise to discourage use of these ambiguous new objects by the casual application programmer. Then again, maybe I just have tunnel vision... -Steve Bergman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 19:52 ` Steve Bergman @ 2004-09-02 20:06 ` Spam -1 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 20:06 UTC (permalink / raw) To: Steve Bergman Cc: Martin J. Bligh, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs > On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote: >> > For 30 years nothing much has happened in Unix filesystem semantics >> > because of sheer cowardice >> >> Or because it works fine, and isn't broken. > OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a > designer of filesystems. I'm just a guy that puts together and supports > Linux systems for my customers. > In following this thread, I may be missing huge chunks of concept. > However, a few things are becoming clear to me: > 1. The file as directory thing adds complexity that the administrator > has to deal with. Symlinks are useful, but it's still aggravating to > tar off a directory structure, take it somewhere, and then realize that > all you have is links to something not in the archive because you didn't > get your tar switches just right. Now we're talking about adding > another set of "files which are not really files" to the semantics. > More complexity. I'll take simplicity over some ivory tower ideal of > "unified name space" any day. Are you afraid to learn something new? ;) Just joking. But really, it doesn't have to be very difficult. The extra streams etc would just be saved as files. If tar is patched then it would be no problem and no extra stuff but perhaps a switch --save-metas. > 2. The use of multiple streams within files by Linux apps would make > Linux as cross-platform unfriendly as MS is trying to be. Say these > features start getting used and you copy an OO.org document from a Linux > box to a BSD box. It's broken. Of course, OO.org wouldn't use the > streams in the first place because it would destroy their cross platform > portability. So what's the point? No one who cares about cross > platform portability can use it. Everyone who doesn't care about cross > platform portability please raise your hand. Actually MacOS and Windows support multiple streams, only Linux doesn't. But of course there are BSD's etc too. I'd say to leave them behind. The file streams would make my day a lot easier. The idea to split up contents of OO.org files into streams is bad. But that doesn't make the file streams bad. I see many uses that would make my every day life easier. It isn't about cross plat form compatibility, but to add features that are useful and meta-data and file streams are. Also. No one forces you to use either meta-data or streams, just as no one forces you to use ACLs or other things. > 3. MS does require attributes and multiple streams, which makes these > features important (even essential) to Samba, and Samba alone. Samba is > important to Linux, so this can't be ignored. (Here I am implicitly > assuming that Samba will need kernel support for this to do it right.) I do not think the Samba would really require the streams support but it would certainly make life easier for Samba. Not to mention that these files would also be natively viewed on the Linux host. > So it seems to me that the only real consideration is giving Samba what > it needs without making the semantics one bit more complex than > absolutely necessary. It might even be wise to discourage use of these > ambiguous new objects by the casual application programmer. > Then again, maybe I just have tunnel vision... I would agree with tunnel vision. The kernel should provide the tools and options. Users and developers can then invent new things to use them. :) ~S > -Steve Bergman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives @ 2004-09-02 20:06 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 20:06 UTC (permalink / raw) To: Steve Bergman Cc: Martin J. Bligh, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs > On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote: >> > For 30 years nothing much has happened in Unix filesystem semantics >> > because of sheer cowardice >> >> Or because it works fine, and isn't broken. > OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a > designer of filesystems. I'm just a guy that puts together and supports > Linux systems for my customers. > In following this thread, I may be missing huge chunks of concept. > However, a few things are becoming clear to me: > 1. The file as directory thing adds complexity that the administrator > has to deal with. Symlinks are useful, but it's still aggravating to > tar off a directory structure, take it somewhere, and then realize that > all you have is links to something not in the archive because you didn't > get your tar switches just right. Now we're talking about adding > another set of "files which are not really files" to the semantics. > More complexity. I'll take simplicity over some ivory tower ideal of > "unified name space" any day. Are you afraid to learn something new? ;) Just joking. But really, it doesn't have to be very difficult. The extra streams etc would just be saved as files. If tar is patched then it would be no problem and no extra stuff but perhaps a switch --save-metas. > 2. The use of multiple streams within files by Linux apps would make > Linux as cross-platform unfriendly as MS is trying to be. Say these > features start getting used and you copy an OO.org document from a Linux > box to a BSD box. It's broken. Of course, OO.org wouldn't use the > streams in the first place because it would destroy their cross platform > portability. So what's the point? No one who cares about cross > platform portability can use it. Everyone who doesn't care about cross > platform portability please raise your hand. Actually MacOS and Windows support multiple streams, only Linux doesn't. But of course there are BSD's etc too. I'd say to leave them behind. The file streams would make my day a lot easier. The idea to split up contents of OO.org files into streams is bad. But that doesn't make the file streams bad. I see many uses that would make my every day life easier. It isn't about cross plat form compatibility, but to add features that are useful and meta-data and file streams are. Also. No one forces you to use either meta-data or streams, just as no one forces you to use ACLs or other things. > 3. MS does require attributes and multiple streams, which makes these > features important (even essential) to Samba, and Samba alone. Samba is > important to Linux, so this can't be ignored. (Here I am implicitly > assuming that Samba will need kernel support for this to do it right.) I do not think the Samba would really require the streams support but it would certainly make life easier for Samba. Not to mention that these files would also be natively viewed on the Linux host. > So it seems to me that the only real consideration is giving Samba what > it needs without making the semantics one bit more complex than > absolutely necessary. It might even be wise to discourage use of these > ambiguous new objects by the casual application programmer. > Then again, maybe I just have tunnel vision... I would agree with tunnel vision. The kernel should provide the tools and options. Users and developers can then invent new things to use them. :) ~S > -Steve Bergman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 20:06 ` Spam (?) @ 2004-09-02 20:16 ` Martin J. Bligh -1 siblings, 0 replies; 1835+ messages in thread From: Martin J. Bligh @ 2004-09-02 20:16 UTC (permalink / raw) To: Spam, Steve Bergman Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs >> 1. The file as directory thing adds complexity that the administrator >> has to deal with. Symlinks are useful, but it's still aggravating to >> tar off a directory structure, take it somewhere, and then realize that >> all you have is links to something not in the archive because you didn't >> get your tar switches just right. Now we're talking about adding >> another set of "files which are not really files" to the semantics. >> More complexity. I'll take simplicity over some ivory tower ideal of >> "unified name space" any day. > > Are you afraid to learn something new? ;) Just joking. But really, > it doesn't have to be very difficult. The extra streams etc would > just be saved as files. If tar is patched then it would be no > problem and no extra stuff but perhaps a switch --save-metas. If they're saved as files, and the app has to be changed to use them anyway, then what's the point? Just change the app to use new files instead. > I would agree with tunnel vision. The kernel should provide the > tools and options. Users and developers can then invent new things > to use them. :) Ugh. Change for changes sake is not a good thing. There's enough real problems in the world without inventing random features. More complexity without gain is a Bad Thing (tm). I'm not saying streams is bad ... just that there don't seem to have been very many convincing (to me) arguments raised for it yet. The versioning stuff would be nice, IMHO, because the stuff mainly using it wouldn't need to be modified in many instances ... "vi /etc/configfile" would keep old copies for you (only the recovery tool would need to understand it). Saving icons seems easy enough to do with "foo.icon" as another file. M. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 19:52 ` Steve Bergman (?) (?) @ 2004-09-02 20:35 ` Erik Hensema -1 siblings, 0 replies; 1835+ messages in thread From: Erik Hensema @ 2004-09-02 20:35 UTC (permalink / raw) To: linux-kernel Steve Bergman (steve@rueb.com) wrote: > However, a few things are becoming clear to me: > > 1. The file as directory thing adds complexity that the administrator > has to deal with. Symlinks are useful, but it's still aggravating to > tar off a directory structure, take it somewhere, and then realize that > all you have is links to something not in the archive because you didn't > get your tar switches just right. Now we're talking about adding > another set of "files which are not really files" to the semantics. > More complexity. I'll take simplicity over some ivory tower ideal of > "unified name space" any day. As with any addition of complexity we have to ask ourselves: do the advantages outweigh the disadvantages? Adding complexity into the kernel may simplify userspace. Adding the complexity of an IP layer into the kernel simplifies networking applications, for example (they won't have to talk IP to the kernel). In this case it's hard to imagine what advantages a multistream file will bring, simply because we aren't used to working with them. Or maybe there aren't any advantages at all. Point is: we don't know yet. Maybe we have to toy with the idea for some years before we really learn why multistream files are so great. I do realize that this isn't how Linux development is done. First we need a problem, then somebody sits down and codes a solution. With reiser4 we've got a solution but we haven't figured out what the problem is yet. > 2. The use of multiple streams within files by Linux apps would make > Linux as cross-platform unfriendly as MS is trying to be. Say these > features start getting used and you copy an OO.org document from a Linux > box to a BSD box. It's broken. Of course, OO.org wouldn't use the > streams in the first place because it would destroy their cross platform > portability. So what's the point? No one who cares about cross > platform portability can use it. Everyone who doesn't care about cross > platform portability please raise your hand. But do we really want to be the lowest common denominator? If you want to lead, you can't be compatible (in some cases). Also, if the reiser4 ideas catch on, they may spark the development of other, similar, filesystems. Maybe we'll end up with a ext4 in the 2.8 kernel... > 3. MS does require attributes and multiple streams, which makes these > features important (even essential) to Samba, and Samba alone. Samba is > important to Linux, so this can't be ignored. (Here I am implicitly > assuming that Samba will need kernel support for this to do it right.) > > So it seems to me that the only real consideration is giving Samba what > it needs without making the semantics one bit more complex than > absolutely necessary. It might even be wise to discourage use of these > ambiguous new objects by the casual application programmer. > > Then again, maybe I just have tunnel vision... Not tunnel vision. You're just looking at what we need *now*, and that happens to be exactly what we *have* now. One possible use for multistream files that I see, is storage of 'not so important' data alongside of the real data of a file. Think: the mimetype of a file; some metadata like author, keywords, etc. When you copy the file and lose all metadata except the file itself, no real harm is done. This could be used by the new mime system Gnome uses, for example. On regular filesystems, the mimetype of a file is determined by its extension. On reiser4(-like filesystems), the mimetype is determined by the 'mimetype' stream. This would clearly simplify the gnome userspace, as there is no need anymore for a seperate extension<->mimetype conversion anymore. It would also be compatible with other tools or desktop environements, we only have to agree on what stream we're going to use for what metadata. IMHO reiser4 is a toy we should give the Linux users to play with. Let's see what they can come up with. With the talented hackers in the community, it should be great. -- Erik Hensema <erik@hensema.net> (attempting to write english while drinking beer) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 8:58 ` Hans Reiser @ 2004-09-02 9:04 ` Hans Reiser 2004-09-02 12:50 ` James Bruce 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-09-02 9:04 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > >But _my_ point is, no user program is going to take _advantage_ of >anything that only one filesystem on one system offers. > >So there's no point. > >I > Filesystem diversity will only be crippling for Linux if you choose for it to be so. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 8:58 ` Hans Reiser 2004-09-02 9:04 ` Hans Reiser @ 2004-09-02 12:50 ` James Bruce 2004-09-02 17:50 ` Linus Torvalds 2 siblings, 1 reply; 1835+ messages in thread From: James Bruce @ 2004-09-02 12:50 UTC (permalink / raw) To: Linus Torvalds Cc: Hans Reiser, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >... >But _my_ point is, no user program is going to take _advantage_ of >anything that only one filesystem on one system offers. > >So there's no point. > >It's much saner (from _any_ app standpoint) to roll their own database in >user space - that way it just works. > >In other words, nobody is really ever going to take advantage of this. >This is _not_ how technical advanncement happens. The way you get people >to take advantage of something is to have a nice gradual ramp-up, not a >sudden new feature that they can't realistically use. >... > Sure, but there are plenty of existing interfaces that you could emulate. One could make a small library to use a transactional filesystem to implement the Berkely DB interface (libdb) for example. So on filesystems without such support your app could use the regular userspace database, but on a transactional filesystem it'd just use regular files, which would simplify database management and likely increase performance over the userspace-only version (libdb is pretty slow). In terms of functionality it'd just be a drop-in replacement, just like math libraries that use MMX/SSE when available. - Jim Bruce ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 12:50 ` James Bruce @ 2004-09-02 17:50 ` Linus Torvalds 0 siblings, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 17:50 UTC (permalink / raw) To: James Bruce Cc: Hans Reiser, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, James Bruce wrote: > Linus Torvalds wrote: > > >In other words, nobody is really ever going to take advantage of this. > >This is _not_ how technical advanncement happens. The way you get people > >to take advantage of something is to have a nice gradual ramp-up, not a > >sudden new feature that they can't realistically use. > > Sure, but there are plenty of existing interfaces that you could > emulate. Absolutely. We should look at them, and whether they solve any issues. I'm not saying that Hans would have to make up a new interface. Quite the reverse. One interface I suggested to use for attributes (and everybody hates ;) was the existing one from Solaris. Similarly, there are probably perfectly fine interfaces for the issue of transactions.. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:15 ` Hans Reiser 2004-09-02 7:36 ` Linus Torvalds @ 2004-09-02 7:55 ` Robin Rosenberg 1 sibling, 0 replies; 1835+ messages in thread From: Robin Rosenberg @ 2004-09-02 7:55 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thursday 02 September 2004 09.15, Hans Reiser wrote: > Linus Torvalds wrote: > >. Doing transactions on one file is > >only the beginning - you'll find people who want transactions across file > >boundaries etc. > > Yup, that's what reiser4 is aiming at and where things get exciting for > version control systems and the like. We are just missing the api code > for it at the moment, all the infrastructure is there, and the api is in > progress in sys_reiser4. Hope that is the same as or implied XA-support. Having to syncrhonized file updates with database transactions is not an uncommon problem. -- robin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 3:38 ` Linus Torvalds 2004-09-02 4:27 ` David Masover @ 2004-09-02 5:53 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-09-02 5:53 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >On Wed, 1 Sep 2004, David Masover wrote: > > >>And that is the right solution. Not the only one, but the right one. >>Caching isn't the only thing sorely in need of transaction support right >>now. Actually, I find it hard to think of anything on Linux which >>shouldn't have transactions -- why should /etc/fstab or >>/home/david/homework be more fragile than /var/lib/mysql? >> >> > >It's easy to talk big. > >It's damn hard to _implement_ a complex system, and make it stable and >bug-free, and support legacy applications. > >There is a reason why we do only what _must_ be done in kernel space. > > Linus > > > > Filesystems need transactions. Given that they need it, it was very tempting to build an infrastructure offering some of the functionality to user space, and so we did in reiser4. What we have is very rudimentary, but I think I am prepared to say that important aspects of high performance transactions are better done in the kernel than in user space. It will be interesting to see what aspects of transaction functionality belong in user space as our transactions API comes into existence. Clearly there are some things about isolation that are better implemented by user space because user space can make optimizations the kernel doesn't know enough to make. Still, it will be interesting to see how low we can make the overhead of isolation. Right now reiser4 supports atomicity but not isolation, which is to say we offer to guarantee that a set of operations will either all survive a crash or none of them will, but we don't let you roll them back without a crash. (Originally I called this a transcrash rather than atomicity, but nobody liked the term.) Being modest in our ambition was important to performance, but maybe in time we will find a high performance way to be more ambitious. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-01 20:50 ` Linus Torvalds 2004-09-02 0:24 ` Jamie Lokier @ 2004-09-02 9:46 ` Alan Cox 2004-09-02 16:11 ` Jamie Lokier ` (2 more replies) 1 sibling, 3 replies; 1835+ messages in thread From: Alan Cox @ 2004-09-02 9:46 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > and quite frankly, I think you can do the above pretty much totally in > user space with a small library and a daemon (in fact, ignoring security > issues you probably don't even need the daemon). And if you can prototype > it like that, and people actually find it useful, I suspect kernel support > for better performance might be possible. Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox @ 2004-09-02 16:11 ` Jamie Lokier 2004-09-02 17:41 ` Dave Kleikamp 2004-09-02 17:46 ` Linus Torvalds 2004-09-02 20:38 ` Frank van Maarseveen 2 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-09-02 16:11 UTC (permalink / raw) To: Alan Cox Cc: Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Alan Cox wrote: > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > > and quite frankly, I think you can do the above pretty much totally in > > user space with a small library and a daemon (in fact, ignoring security > > issues you probably don't even need the daemon). And if you can prototype > > it like that, and people actually find it useful, I suspect kernel support > > for better performance might be possible. > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > it since the late 1990's. Firstly, if I have to do it from a Gnome program, about the only program where looking in a tar file is visibly useful is Nautilus. Ironically, clicking on a tar file in Nautilus doesn't work, despite having a dependency on gnome-vfs2. :/ Secondly, no, Gnome and MC don't support entering a container file, letting you make changes in it, and remembering those changes to _lazily_ regenerate the container file when you need it linearized, possibly months later or never, by some unrelated program. Thirdly, you must be referring to the Gnome versions of Bash, Make, GCC, coreutils and Perl which I haven't found. Perhaps we have a different idea of what "supports this" means :) uservfs, which is based on gnome-vfs and getting a bit rusty due to disuse, does try to solve the last problem. Unfortunately it needs further work to have a nicer interface, and the second problem is still not solved by gnome-vfs. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 16:11 ` Jamie Lokier @ 2004-09-02 17:41 ` Dave Kleikamp 0 siblings, 0 replies; 1835+ messages in thread From: Dave Kleikamp @ 2004-09-02 17:41 UTC (permalink / raw) To: Jamie Lokier Cc: Alan Cox, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 11:11, Jamie Lokier wrote: > Firstly, if I have to do it from a Gnome program, about the only > program where looking in a tar file is visibly useful is Nautilus. > Ironically, clicking on a tar file in Nautilus doesn't work, despite > having a dependency on gnome-vfs2. :/ This should be fixed in Nautilus, not the kernel. > Secondly, no, Gnome and MC don't support entering a container file, > letting you make changes in it, and remembering those changes to > _lazily_ regenerate the container file when you need it linearized, > possibly months later or never, by some unrelated program. Why do this in a tar file? tar = "tape archive". It isn't designed to be a file system. Sure, it's nice to have tools that make it easier to access files in a tar file, but to this isn't a job for the kernel. > Thirdly, you must be referring to the Gnome versions of Bash, Make, > GCC, coreutils and Perl which I haven't found. Perhaps we have a > different idea of what "supports this" means :) Please don't tell me that we have expectations to run make from within a tar file. This is getting silly. tar does a pretty good job of extracting files into real directories, and putting them back into an archive. I don't see a need to teach the kernel how to deal with compound files when user space can do it very easily. Shaggy ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 16:11 ` Jamie Lokier @ 2004-09-02 17:46 ` Linus Torvalds 2004-09-02 17:50 ` Christoph Hellwig ` (2 more replies) 2004-09-02 20:38 ` Frank van Maarseveen 2 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 17:46 UTC (permalink / raw) To: Alan Cox Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Alan Cox wrote: > > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > > and quite frankly, I think you can do the above pretty much totally in > > user space with a small library and a daemon (in fact, ignoring security > > issues you probably don't even need the daemon). And if you can prototype > > it like that, and people actually find it useful, I suspect kernel support > > for better performance might be possible. > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > it since the late 1990's. And nobody has asked for kernel support that I know of. So either "it just works" in user space, or people haven't figured out the kernel could help them. Or decided it's not worth it, exactly because they'd still have to support systems/filesystems that can't be converted. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:46 ` Linus Torvalds @ 2004-09-02 17:50 ` Christoph Hellwig 2004-09-02 18:03 ` Linus Torvalds 2004-09-02 18:46 ` Alan Cox 2004-09-02 19:50 ` Spam 2 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-09-02 17:50 UTC (permalink / raw) To: Linus Torvalds Cc: Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 10:46:32AM -0700, Linus Torvalds wrote: > > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > > > and quite frankly, I think you can do the above pretty much totally in > > > user space with a small library and a daemon (in fact, ignoring security > > > issues you probably don't even need the daemon). And if you can prototype > > > it like that, and people actually find it useful, I suspect kernel support > > > for better performance might be possible. > > > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > > it since the late 1990's. > > And nobody has asked for kernel support that I know of. > > So either "it just works" in user space, or people haven't figured out the > kernel could help them. Or decided it's not worth it, exactly because > they'd still have to support systems/filesystems that can't be converted. http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto a kernel filesystem. The code is horrible, but it shows that it can be done. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:50 ` Christoph Hellwig @ 2004-09-02 18:03 ` Linus Torvalds 2004-09-02 19:04 ` Charles P. Wright 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 18:03 UTC (permalink / raw) To: Christoph Hellwig Cc: Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Christoph Hellwig wrote: > > http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto > a kernel filesystem. The code is horrible, but it shows that it can > be done. I do like the setup where the extended features are done as a "view" on top of some other filesystem, so that you can choose to _either_ access the raw (and supposedly stable, simply by virtue of simplicity) or the "fancy" interface. Without having to reformat the disk to a filesystem you don't trust, or you have other reasons you can't use (disk sharing with other systems, whatever). It doesn't have to be "user", btw, in the sense that a lot of the normal code could be in kernel mode. Same way as Tux handling all the regular static requests entirely in kernel mode, but having the ability for calling down to apache.. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 18:03 ` Linus Torvalds @ 2004-09-02 19:04 ` Charles P. Wright 2004-09-02 19:42 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: Charles P. Wright @ 2004-09-02 19:04 UTC (permalink / raw) To: Linus Torvalds; +Cc: Christoph Hellwig, Jamie Lokier, linux-fsdevel On Thu, 2004-09-02 at 14:03, Linus Torvalds wrote: > On Thu, 2 Sep 2004, Christoph Hellwig wrote: > > > > http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto > > a kernel filesystem. The code is horrible, but it shows that it can > > be done. > > I do like the setup where the extended features are done as a "view" on > top of some other filesystem, so that you can choose to _either_ access > the raw (and supposedly stable, simply by virtue of simplicity) or the > "fancy" interface. Without having to reformat the disk to a filesystem you > don't trust, or you have other reasons you can't use (disk sharing with > other systems, whatever). > > It doesn't have to be "user", btw, in the sense that a lot of the normal > code could be in kernel mode. Same way as Tux handling all the regular > static requests entirely in kernel mode, but having the ability for > calling down to apache.. Stackable file systems are a good fit for doing things like this. They layer on top of an existing directory and intercept all of the vfs_* calls. You can do lots of neat things like encryption versioning, compression, and more. At Stony Brook's Filesystems and Storage Laboratory (http://www.fsl.cs.sunysb.edu), we maintain a toolkit (FiST) that makes developing these file systems easier by providing stackable file system templates for Linux and FreeBSD. There are papers and the FiST toolkit available at http://www.filesystems.org/ Charles ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 19:04 ` Charles P. Wright @ 2004-09-02 19:42 ` Linus Torvalds 2004-09-02 20:05 ` Charles P. Wright 2004-09-02 20:30 ` Erez Zadok 0 siblings, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 19:42 UTC (permalink / raw) To: Charles P. Wright; +Cc: Christoph Hellwig, Jamie Lokier, linux-fsdevel On Thu, 2 Sep 2004, Charles P. Wright wrote: > > Stackable file systems are a good fit for doing things like this. They > layer on top of an existing directory and intercept all of the vfs_* > calls. You can do lots of neat things like encryption versioning, > compression, and more. Stackable filesystems are really hard, and ambiguous on where things go, and what happens when you remove an entry that hides another one. Sure, you can make up any amount of semantics, but the fact is, all the semantics _are_ going to be pretty arbitrary, and fairly complex. I'd frankly _much_ prefer not having stackable filesystems supported by the VFS layer. That said, I think you can make stackable filesystems as a _separate_ filesystem (aka "view" or whatever), which is slightly different from supporting them in the VFS layer itself: make it a filesystem of its own, rather than supporting it on a VFS layer as a merge of two different filesystems. That way you still have totally arbitrary semantics for how the stacking works, but at least it's not codified by the VFS layer - it's codified by nothing more special than one more random filesystem implementation, and you could in fact have _different_ semantics depending on what the user wants. Same end result, different "mindset". I think this is what FiST actually does, but I could be wrong. What's the difference? If the VFS layer supports it (like the VFS supports regular "opaque" overmounting), you'd just do mount -o stack /dir /otherdir the same way you do "bind" mounts. In contrast, if you do it as a third-party "view" filesystem, it would be more like mount -t stackfs -o under=/usr,over=/usr/local /mnt where /usr and /usr/local would be the same old underlying filesystems, and you'd see a totally new filesystem at /mnt. Btw, this kind of non-VFS-supported filesystem would not necessarily have coherency guarantees: if you changed /usr or /usr/local directly (rather than through the stackable /mnt), you could confuse stackfs. Kind of the same way you can confuse a NFSv2 client by operating directly on the server (ie I'm not saying "confuse" in a "blow up the kernel" kind of way, but in a "would return -ESTALE or similar"). Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 19:42 ` Linus Torvalds @ 2004-09-02 20:05 ` Charles P. Wright 2004-09-02 20:30 ` Erez Zadok 1 sibling, 0 replies; 1835+ messages in thread From: Charles P. Wright @ 2004-09-02 20:05 UTC (permalink / raw) To: Linus Torvalds; +Cc: Christoph Hellwig, Jamie Lokier, linux-fsdevel On Thu, 2004-09-02 at 15:42, Linus Torvalds wrote: > On Thu, 2 Sep 2004, Charles P. Wright wrote: > > > > Stackable file systems are a good fit for doing things like this. They > > layer on top of an existing directory and intercept all of the vfs_* > > calls. You can do lots of neat things like encryption versioning, > > compression, and more. > > Stackable filesystems are really hard, and ambiguous on where things go, > and what happens when you remove an entry that hides another one. Sure, > you can make up any amount of semantics, but the fact is, all the > semantics _are_ going to be pretty arbitrary, and fairly complex. I think by stackable you are thinking what we call a Union. By stackable, I mean that you just layer some functionality on an existing file system, without having to modify it. I agree that Unioning semantics are difficult. We've actually implemented a stackable unioning file system, and dealing with an arbitrary number of branches, especially in the face of some operations failing in the middle, is quite complex. Rename is especially hard. I think for the type of problem, where you have the file system jump down into archives or do other like-minded operations, you will also end up with having to make some arbitrary decisions with complex semantics. > I'd frankly _much_ prefer not having stackable filesystems supported by > the VFS layer. > That said, I think you can make stackable filesystems as a _separate_ > filesystem (aka "view" or whatever), which is slightly different from > supporting them in the VFS layer itself: make it a filesystem of its own, > rather than supporting it on a VFS layer as a merge of two different > filesystems. That way you still have totally arbitrary semantics for how > the stacking works, but at least it's not codified by the VFS layer - it's > codified by nothing more special than one more random filesystem > implementation, and you could in fact have _different_ semantics depending > on what the user wants. > > Same end result, different "mindset". I think this is what FiST actually > does, but I could be wrong. Yes, to the VFS FiST behaves as a file system. To the file system it is stacked on, it behaves like the VFS. > What's the difference? If the VFS layer supports it (like the VFS supports > regular "opaque" overmounting), you'd just do > > mount -o stack /dir /otherdir > > the same way you do "bind" mounts. In contrast, if you do it as a > third-party "view" filesystem, it would be more like > > mount -t stackfs -o under=/usr,over=/usr/local /mnt > > where /usr and /usr/local would be the same old underlying filesystems, > and you'd see a totally new filesystem at /mnt. That is very close to how a FiST file system works. However, I think you are more closely describing a Union mount that would provide a merged view of "/user" and "/usr/local" (with /usr/local taking precedence over /user). More concretely, something like: mount -t gzipfs -o dir=/compressed none /mnt Would transparently compress data written to /mnt/a, and physically store it in /compressed/a. When reading /mnt/a, the data is decompressed and returned to the user-space. This a real example that is actually included as part of FiST. > Btw, this kind of non-VFS-supported filesystem would not necessarily have > coherency guarantees: if you changed /usr or /usr/local directly (rather > than through the stackable /mnt), you could confuse stackfs. Kind of the > same way you can confuse a NFSv2 client by operating directly on the > server (ie I'm not saying "confuse" in a "blow up the kernel" kind of > way, but in a "would return -ESTALE or similar"). With a simple stacking file system, one way to avoid this is an overlay mount. mount -t cryptfs -o dir=/home/cpwright none /home/cpwright There are some disadvantages to hiding the lower level (e.g., with Cryptfs you want to backup the encrypted data, not the decrypted data). Charles ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 19:42 ` Linus Torvalds 2004-09-02 20:05 ` Charles P. Wright @ 2004-09-02 20:30 ` Erez Zadok 1 sibling, 0 replies; 1835+ messages in thread From: Erez Zadok @ 2004-09-02 20:30 UTC (permalink / raw) To: Linus Torvalds; +Cc: Charles P. Wright, linux-fsdevel In message <Pine.LNX.4.58.0409021230450.2295@ppc970.osdl.org>, Linus Torvalds writes: [...] > That said, I think you can make stackable filesystems as a _separate_ > filesystem (aka "view" or whatever), which is slightly different from > supporting them in the VFS layer itself: make it a filesystem of its own, > rather than supporting it on a VFS layer as a merge of two different > filesystems. That way you still have totally arbitrary semantics for how > the stacking works, but at least it's not codified by the VFS layer - it's > codified by nothing more special than one more random filesystem > implementation, and you could in fact have _different_ semantics depending > on what the user wants. > > Same end result, different "mindset". I think this is what FiST actually > does, but I could be wrong. > > What's the difference? If the VFS layer supports it (like the VFS supports > regular "opaque" overmounting), you'd just do > > mount -o stack /dir /otherdir > > the same way you do "bind" mounts. In contrast, if you do it as a > third-party "view" filesystem, it would be more like > > mount -t stackfs -o under=/usr,over=/usr/local /mnt > > where /usr and /usr/local would be the same old underlying filesystems, > and you'd see a totally new filesystem at /mnt. [...] Yes, that's exactly how our stacked f/s's work: not touching or modifying VFS or lower f/s's, but of course we use the VFS the way other f/s's use it (calling vfs_* functions and such). Each of our stackable f/s is a separate f/s in its own. And they're a nice way to augment the functionality of any other f/s w/o the hassles of changing a lower-level f/s format or the VFS. We recently did an example f/s that adds EA+ACL functionality on top of anything else, using the lower file system's EA/ACL support if it's there, and for f/s's that don't support EA/ACL natively, we support the ea/acl f/s calls and store the actual ea/acl data in a small database. We've even demonstrated that it worked on v/fat (yes, we know, vfat+ACLs is an an oxymoron :-) We also found out over the years that our stackable file systems are an interesting way to prototype features that might eventually move up to the VFS itself; this is b/c each stacked f/s has to appear to the lower f/s as the VFS itself, essentially emulating most of the VFS's functionality while at the same time appearing to the VFS as a lower-level f/s. It's not easy maintaining this split personality. :-) Erez. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:46 ` Linus Torvalds 2004-09-02 17:50 ` Christoph Hellwig @ 2004-09-02 18:46 ` Alan Cox 2004-09-02 20:22 ` Linus Torvalds 2004-09-02 19:50 ` Spam 2 siblings, 1 reply; 1835+ messages in thread From: Alan Cox @ 2004-09-02 18:46 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 18:46, Linus Torvalds wrote: > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > > it since the late 1990's. > > And nobody has asked for kernel support that I know of. I asked our desktop people. They want something like inotify because dontify doesn't cut it. They have zero interest in the multiple streams and hiding icons in streams type stuff. Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 18:46 ` Alan Cox @ 2004-09-02 20:22 ` Linus Torvalds 0 siblings, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-02 20:22 UTC (permalink / raw) To: Alan Cox Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Alan Cox wrote: > > I asked our desktop people. They want something like inotify because > dontify doesn't cut it. Well, dnotify() really _is_ inotify(), since it does actually work on inodes, not dentries. I think what they are really complaining about is that dnotify() only notifies the _directory_ when a file is changed, and they'd like it to notify the file itself too. Which is a one-liner, really. Does the following make sense? (Totally untested, use-at-your-own-risk, I've-never-actually-used-dnotify-in-user-space, whatever). Linus ===== fs/dnotify.c 1.17 vs edited ===== --- 1.17/fs/dnotify.c 2004-08-09 18:45:22 -07:00 +++ edited/fs/dnotify.c 2004-09-02 13:21:26 -07:00 @@ -160,6 +160,8 @@ if (!dir_notify_enable) return; + __inode_dir_notify(dentry->d_inode, event); + spin_lock(&dentry->d_lock); parent = dentry->d_parent; if (parent->d_inode->i_dnotify_mask & event) { ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:46 ` Linus Torvalds @ 2004-09-02 19:50 ` Spam 2004-09-02 18:46 ` Alan Cox 2004-09-02 19:50 ` Spam 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 19:50 UTC (permalink / raw) To: Linus Torvalds Cc: Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Thu, 2 Sep 2004, Alan Cox wrote: >> >> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: >> > and quite frankly, I think you can do the above pretty much totally in >> > user space with a small library and a daemon (in fact, ignoring security >> > issues you probably don't even need the daemon). And if you can prototype >> > it like that, and people actually find it useful, I suspect kernel support >> > for better performance might be possible. >> >> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported >> it since the late 1990's. > And nobody has asked for kernel support that I know of. Actually. Doesn't matter if it is in kernel or not for the users as long as it works. The problem is that I do not see either Gnome or KDE to ever get along to form one standard that everyone will use. Their libraries are huge and memory hogging which so many Linux users just do not like. What if a user doesn't want KDE or Gnome? Would all files created with either be broken? I doubt that something like file streams and meta-data can successfully be implemented purely in user-space and get the same support (ie be used by many programs) if this change doesn't come from the kernel. I just do not see it happen. > So either "it just works" in user space, or people haven't figured out the > kernel could help them. Or decided it's not worth it, exactly because > they'd still have to support systems/filesystems that can't be converted. > Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) @ 2004-09-02 19:50 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 19:50 UTC (permalink / raw) To: Linus Torvalds Cc: Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Thu, 2 Sep 2004, Alan Cox wrote: >> >> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: >> > and quite frankly, I think you can do the above pretty much totally in >> > user space with a small library and a daemon (in fact, ignoring security >> > issues you probably don't even need the daemon). And if you can prototype >> > it like that, and people actually find it useful, I suspect kernel support >> > for better performance might be possible. >> >> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported >> it since the late 1990's. > And nobody has asked for kernel support that I know of. Actually. Doesn't matter if it is in kernel or not for the users as long as it works. The problem is that I do not see either Gnome or KDE to ever get along to form one standard that everyone will use. Their libraries are huge and memory hogging which so many Linux users just do not like. What if a user doesn't want KDE or Gnome? Would all files created with either be broken? I doubt that something like file streams and meta-data can successfully be implemented purely in user-space and get the same support (ie be used by many programs) if this change doesn't come from the kernel. I just do not see it happen. > So either "it just works" in user space, or people haven't figured out the > kernel could help them. Or decided it's not worth it, exactly because > they'd still have to support systems/filesystems that can't be converted. > Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 16:11 ` Jamie Lokier 2004-09-02 17:46 ` Linus Torvalds @ 2004-09-02 20:38 ` Frank van Maarseveen 2004-09-02 21:36 ` Dave Kleikamp 2 siblings, 1 reply; 1835+ messages in thread From: Frank van Maarseveen @ 2004-09-02 20:38 UTC (permalink / raw) To: Alan Cox Cc: Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 10:46:05AM +0100, Alan Cox wrote: > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > > and quite frankly, I think you can do the above pretty much totally in > > user space with a small library and a daemon (in fact, ignoring security > > issues you probably don't even need the daemon). And if you can prototype > > it like that, and people actually find it useful, I suspect kernel support > > for better performance might be possible. > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > it since the late 1990's. Can it do this: cd FC2-i386-disc1.iso ls or this: cd /dev/sda1 ls cd /dev/floppy ls cd /dev/cdrom ls ? -- Frank ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 20:38 ` Frank van Maarseveen @ 2004-09-02 21:36 ` Dave Kleikamp 0 siblings, 0 replies; 1835+ messages in thread From: Dave Kleikamp @ 2004-09-02 21:36 UTC (permalink / raw) To: Frank van Maarseveen Cc: Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 15:38, Frank van Maarseveen wrote: > Can it do this: > > cd FC2-i386-disc1.iso > ls > > or this: > > cd /dev/sda1 > ls > cd /dev/floppy > ls > cd /dev/cdrom > ls > > ? We have the mount command for that. :^) -- David Kleikamp IBM Linux Technology Center ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-01 20:08 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Jamie Lokier 2004-09-01 20:50 ` Linus Torvalds @ 2004-09-02 14:06 ` Horst von Brand 2004-09-02 17:32 ` Jamie Lokier 1 sibling, 1 reply; 1835+ messages in thread From: Horst von Brand @ 2004-09-02 14:06 UTC (permalink / raw) To: Jamie Lokier Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier <jamie@shareable.org> said: > I'm going to explain why filesystem support for .tar.gz or other > "document container"-like formats is useful. Nobody disagrees there (I think), the disagreement is on whether the usefullness is worth the hassle. > This does _not_ mean tar > in the kernel (I know someone who can't read will think that if I > don't say it isn't); it does mean hooks in the kernel for maintaining > coherency between different views and filesystem support for cacheing. "Coherency" and "different views" implies atomic transactions, and being able to tell that an access to the file requieres updating random junk about it. It requires being able to guess if it is worth updating now (given that the file might be modified a dozen times more before the junk being checked). > The vision I'm going for here is: > > 1. OpenOffice and similar programs store compound documents in > standard file formats, such as .tar.gz, compressed XML and such. And they are doing fine AFAICS. Besides, they won't exactly jump on the possibility of leaving behind all other OSes on which they run to become a Linux-only format. > Fs support can reduce CPU time handling these sorts of files, as > I explain below, while still working with the standard file formats. I don't buy this one. A tar.gz must be uncompressed and unpacked, and whatever you could save is surely dwarfed by those costs. > With appropriate userspace support, programs can be written which > have access to the capabilities on all platforms, but reduced CPU > time occurs only on platforms with the fs support. Userspace support isn't there on any of the platforms right now, if ever it will be a strange-Linux-installation thing for quite some time to come. Not exactly attractive for application writers. > 2. Real-time indexing and local search engine tools. Sure! Gimme the CPU power and disk throughput for that, pretty please. [No, I won't tell I have better use for those right now...] > This isn't > just things like local Google; it's also your MP3 player scanning > for titles & artists, your email program scanning for subject > lines to display the summary fast, your blog server caching built > pages, your programming environment scanning for tags, and your > file transfer program scanning for shared deltas to reduce bandwidth. With no description on how this is supposed to work, this is pure science fiction/wet dreams. > I won't explain how these work as it would make this mail too > long. It should be clear to anyone who thinks about it why the > coherency mechanism is essential for real-time, and a consistent > interface to container internals helps with performance. Coherency is essential, but it isn't free. Far from it. The easiest way of getting coherency is having _one_ authoritative source. That way you don't need coherency, and don't pay for it. Anything in this class must by force be just hints, to be recomputed at a moment's notice. I.e., let the application who might use it check and recompute as needed. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 14:06 ` Horst von Brand @ 2004-09-02 17:32 ` Jamie Lokier 2004-09-02 18:23 ` Christer Weinigel 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-09-02 17:32 UTC (permalink / raw) To: Horst von Brand Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Horst von Brand wrote: > > This does _not_ mean tar > > in the kernel (I know someone who can't read will think that if I > > don't say it isn't); it does mean hooks in the kernel for maintaining > > coherency between different views and filesystem support for cacheing. > > "Coherency" and "different views" implies atomic transactions, Wrong. They imply ordering. > and being able to tell that an access to the file requieres updating > random junk about it. Correct. > It requires being able to guess if it is worth updating now > (given that the file might be modified a dozen times more before the junk > being checked). Wrong. Lazy and deterministic are compatible. > > > The vision I'm going for here is: > > > > 1. OpenOffice and similar programs store compound documents in > > standard file formats, such as .tar.gz, compressed XML and such. > > And they are doing fine AFAICS. Besides, they won't exactly jump on the > possibility of leaving behind all other OSes on which they run to become a > Linux-only format. You replied before reading the rest of the mail again, didn't you? The whole purpose of the idea is to work with formats which are _not_ Linux-only. > > Fs support can reduce CPU time handling these sorts of files, as > > I explain below, while still working with the standard file formats. > > I don't buy this one. A tar.gz must be uncompressed and unpacked, and > whatever you could save is surely dwarfed by those costs. Didn't they explain what "lazy" means in your computer science class? Google for "lazy evaluation". In this context it means _unnecessary_ uncomressions and compressions, i.e. repeated ones, or ones where you never really needed the compressed form, are eliminated. Necessary ones are of course done. Those are things which are currently wasting your CPU because apps cannot eliminate them: to do so implies synchronisation between different apps running at different times. Surprise surprise, that's something filesystems and kernels are good for. > > With appropriate userspace support, programs can be written which > > have access to the capabilities on all platforms, but reduced CPU > > time occurs only on platforms with the fs support. > > Userspace support isn't there on any of the platforms right now, if ever it > will be a strange-Linux-installation thing for quite some time to come. Not > exactly attractive for application writers. Again, you have a strange vision of what userspace support means. I say it means a portable and simple library for accessing components inside a compressed container file in a standard format. Something I believe OpenOffice et al. already has - so to say it doesn't exist is both incorrect and missing the point. The point is to change that library which already exists, so it uses the filesystem facility when available. And to make the library better in standalone form, so that other tools which manipulate container-like formats are inclined to work with it as plugins, instead of creating their own interface as cli tools. _At no point is portability an issue_ - that library is already portable to everything - and it's certainly not a strange-Linux thing nor meant to become one. > > 2. Real-time indexing and local search engine tools. > > Sure! Gimme the CPU power and disk throughput for that, pretty please. [No, > I won't tell I have better use for those right now...] You still haven't grasped the idea of an algorithm which is more complex but reduces CPU time and disk throughput, have you? *Today*, when I open Rhythmbox I have to wait 5 minutes while it wastes my CPU power and disk throughput scanning all the files in my Music directory. The entire point of these indexing schemes is so that programs like Rhythmbox will display the same data without having to scan the disk for 5 minutes every time they start up. That's a near infinite _saving_ of CPU power and time. If you don't see that I'm just going to have to suggest you google for "cache" and learn a bit about them. > > This isn't > > just things like local Google; it's also your MP3 player scanning > > for titles & artists, your email program scanning for subject > > lines to display the summary fast, your blog server caching built > > pages, your programming environment scanning for tags, and your > > file transfer program scanning for shared deltas to reduce bandwidth. > > With no description on how this is supposed to work, this is pure science > fiction/wet dreams. Sigh. If you must continue to hurl your blunt instruments around, here is a description. I didn't want to write this becuase it is off topic, and you insist on not understanding the basic of algorithm complexity and CPU usage, so you'll probably not understand this either, but I'll give it a go. 1. Local Google (by which I mean a search engine on your local machine), Real-time (by which I mean the results are always up to date): Every file modified since last search is known to the query engine. This is a reality: BeOS does it; WinFS is expected to do it. We know that it's possible to update free text indexes with small amounts of known changes quickly, at the time of a query. Thus we have real-time local free text search engine, and other features like searching inside files and for file names. The point is the real-time nature of it: the results you get correspond to exactly the contents of the filesystem at the time of the query (writes which occur _during_ a query are obviously not coherent with this, but writes which complete before the query, even immediately before, appear in the results). Note that file write timing need not be affected - all the work can happen during queries (although it is best to have a nightly re-index or whatever so that the delay at query times is kept moderate). 2. MP3 player scanning artists & titles: Easy. MP3 player does what it does, shows the extracted ID tags from all .mp3 files in your home directory. They do this already! The difference is that with fs coherency hooks, they can _store_ that ID information, retrieve it later without having to scan all the .mp3s again (see Rhythmbox earlier), and keep their lists updated on the screen as soon as any .mp3 is changed or even any new ones created anywhere. Technically it's a simpler subset of real-time queries in 1. 3. Email program scanning for subject lines fast: See Evolution; the only difference is stat() on a thousand files won't be required, it'll be exposed through a standard query instead of an Evolution-only cache method so other mail programs may use it as well as shell commands, and you can pretend you have mbox instead of maildir (mbox is a container format...). 4. Blog server caching built pages: It's a reality already, obviously. The difference is it'll make sense to built the pages through an indexed query on files instead of a database, e.g. one file per article, and (independent of the first point) the process of building can keep track of all the prerequisite files and scripts and templates used to produce the page, and actually expect to know, _with coherent accuracy_, if any of those prerequisites is different the next time the page is needed. In other words, complex script-generated web pages, cached output, and with no overhead when serving each page to check prerequisites (i.e. no 100 stat() calls), because we know the filesystem would have told us if any prerequisite had changed prior to the moment we begin serving the cached output. You can do something close to this already with dnotify (if you ignore that it doesn't tell you about new hard links, which is a dnotify bug), although doing the query part is unrealistic with dnotify and stat() prerequisites unless the directory names are structured thoughtfully. 5. Programming environment scanning for tags: By now this should be obvious. No need to run "exuberant ctags" or whatever class-hierarchy-extraction and documentation-extraction program every so often after making changes, and yet your IDE's clickable list of tags and notes stays up to date with file modifications in real time. I think some IDEs do this already with moderate size trees, using stat(), but it's not realistic when you have tens of thousands of source files. 6. File transfer program scanning for shared deltas. This is nothing more than searching all files for common subsequences which match strong hashes which are expected to be common during file transfer operations. E.g. the GPL header at the start of many source files would have one such hash. A hash of every whole file is another one, along with the name of the file (it is a key which indicates a "likely to match the corresponding hash" condition), as is a hash of every aligned 64k subsequence, or whatever is appropriate to reduce disk I/O. Having an index of these can speed up some file transfer protocols, in the manner of rsync but comparing among a large group of files instead of just two. The point is that kind of protocol can be more efficient than rsync (sometimes dramatically so), but it's only a net gain to use that kind of algorithm if you have a handy _and reliable_ index of likely common subsequences and whole-file hashes, otherwise it uses far too much disk I/O to check the group. The index needs filesystem coherency support, otherwise it is not reliable enough for this bandwidth optimisation. > > I won't explain how these work as it would make this mail too > > long. It should be clear to anyone who thinks about it why the > > coherency mechanism is essential for real-time, and a consistent > > interface to container internals helps with performance. > > Coherency is essential, but it isn't free. Far from it. The easiest way of > getting coherency is having _one_ authoritative source. That way you don't > need coherency, and don't pay for it. Anything in this class must by force > be just hints, to be recomputed at a moment's notice. I.e., let the > application who might use it check and recompute as needed. (a) Without some kind of minimal hook, there is no way for the application to check without re-reading all the files it has used for a computation, starting at the moment where it's about to potentially recompute the result. Sometimes you can get away with stat() on the files (e.g. Evolution and fontconfig do that), but often you can't because that's not trustworthy enough (security checks, protocol optimisations, transparent "as if I didn't cache anything" semantics), and sometimes stat() calls are far too slow anyway (anything that involves a lot of files). (The trustworthiness problem is likely to be solved, for some applications, by an xattr or such which is guaranteed to be deleted or changed when a file is modified. That is really a very minimal hook, but it is one). (b) Even with all the misgivings of stat(), you can't realistically update real time displays, e.g. the lists of artists & albums on the screen in your MP3 player whenever you modify anything in your music collection using another program. At best, you have to expect the display to not notice the change for a while (minutes in the mp3 case), or until you click an "Explicit Refresh" button. Neither should be necessary. It's true that the simplest coherency is one authoratitive source which you have to check all the time (although even that doesn't work for some things). But checking all the time does rule out, due to its high algorithmic complexity, a lot of very interesting applications and even some interesting simple unixish tools which you might like, such as real-time "locate". -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:32 ` Jamie Lokier @ 2004-09-02 18:23 ` Christer Weinigel 0 siblings, 0 replies; 1835+ messages in thread From: Christer Weinigel @ 2004-09-02 18:23 UTC (permalink / raw) To: Jamie Lokier Cc: Horst von Brand, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier <jamie@shareable.org> writes: > 1. Local Google (by which I mean a search engine on your local machine), > Real-time (by which I mean the results are always up to date): > > Every file modified since last search is known to the query engine. > This is a reality: BeOS does it; WinFS is expected to do it. > > Thus we have real-time local free text search engine, and other > features like searching inside files and for file names. The > point is the real-time nature of it: the results you get > correspond to exactly the contents of the filesystem at the time of > the query (writes which occur _during_ a query are obviously > not coherent with this, but writes which complete before the > query, even immediately before, appear in the results). Can be done with dnotify/inotify and a cache daemon keeping track of mtime. Yes, this will need a kernel change to make sure mtime always changed when the file changes, but it does not require anything else. > 2. MP3 player scanning artists & titles: Same. > 3. Email program scanning for subject lines fast: Same here. > 4. Blog server caching built pages: > 5. Programming environment scanning for tags: > 6. File transfer program scanning for shared deltas. And so on. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:49 ` Hans Reiser (?) (?) @ 2004-08-26 18:35 ` Joel Becker 2004-08-26 19:53 ` Jamie Lokier 2004-08-27 9:19 ` Markus Törnqvist -1 siblings, 2 replies; 1835+ messages in thread From: Joel Becker @ 2004-08-26 18:35 UTC (permalink / raw) To: Hans Reiser Cc: Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote: > Yes, this was part of the plan, tar file-directory plugins would be cute. Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute directory or a directory in the tarball named "metas"? Joel -- "I'm so tired of being tired, Sure as night will follow day. Most things I worry about Never happen anyway." Joel Becker Senior Member of Technical Staff Oracle Corporation E-mail: joel.becker@oracle.com Phone: (650) 506-8127 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:35 ` silent semantic changes with reiser4 Joel Becker @ 2004-08-26 19:53 ` Jamie Lokier 2004-08-27 9:19 ` Markus Törnqvist 1 sibling, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 19:53 UTC (permalink / raw) To: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Joel Becker wrote: > On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote: > > Yes, this was part of the plan, tar file-directory plugins would be cute. > > Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute > directory or a directory in the tarball named "metas"? This needs to be designed. Perhaps /foo/bar/baz.tar.gz/tar/metas is the directory in the tarball named "metas". Or perhaps /foo/bar/baz.tar.gz/x/metas is: it's independent of archive format, and I personally tend to extract things into a directory called "x". [*] Or perhaps /foo/bar/baz.tar.gz/metas is, and the attribute directory is /foo/bar/baz.tar.gz/../metas, to be perverse ;) I prefer the second one, ("x/metas"), but not with any conviction. -- Jamie [*] Actually I prefer: /foo/bar/baz.tar.gz/content/metas /foo/bar/baz-0.01.tar.gz/content/baz-0.01/metas Archives always in "content". One layer of decompression always tried for .tar files and other uncompressed archive formats. /foo/bar/baz.tar.gz/x -> content/ /foo/bar/baz-0.01.tar.gz/x -> content/baz-0.01/ If the root of the archive contains a single directory, "x" is a symlink to it. Otherwise "x" is a symlink to the root directory of the archive. This is comfortable with the common practice by which archives are distributed, without making a mess when someone forgets to put everything in a top-level directory. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:35 ` silent semantic changes with reiser4 Joel Becker 2004-08-26 19:53 ` Jamie Lokier @ 2004-08-27 9:19 ` Markus Törnqvist 1 sibling, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-27 9:19 UTC (permalink / raw) To: Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 11:35:32AM -0700, Joel Becker wrote: > Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute >directory or a directory in the tarball named "metas"? This has been fought over on the reiserfs-list ad nauseaum, but it's a valid point. That's why I tend to rename metas/ into ..metas/ to avoid name clashes, even if I've never had a directory named metas/ apart from what Reiser4 ships. It is then debatable if it should be renamed before it's too late, have it renamable in the kernel configs (and the name exported via /sys or something) or just leave it be. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 9:19 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: mjt @ 2004-08-27 9:19 UTC (permalink / raw) To: Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 11:35:32AM -0700, Joel Becker wrote: > Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute >directory or a directory in the tarball named "metas"? This has been fought over on the reiserfs-list ad nauseaum, but it's a valid point. That's why I tend to rename metas/ into ..metas/ to avoid name clashes, even if I've never had a directory named metas/ apart from what Reiser4 ships. It is then debatable if it should be renamed before it's too late, have it renamable in the kernel configs (and the name exported via /sys or something) or just leave it be. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 9:19 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-27 9:19 UTC (permalink / raw) To: Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 11:35:32AM -0700, Joel Becker wrote: > Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute >directory or a directory in the tarball named "metas"? This has been fought over on the reiserfs-list ad nauseaum, but it's a valid point. That's why I tend to rename metas/ into ..metas/ to avoid name clashes, even if I've never had a directory named metas/ apart from what Reiser4 ships. It is then debatable if it should be renamed before it's too late, have it renamable in the kernel configs (and the name exported via /sys or something) or just leave it be. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 9:19 ` Markus Törnqvist @ 2004-08-27 12:26 ` Matthew Wilcox -1 siblings, 0 replies; 1835+ messages in thread From: Matthew Wilcox @ 2004-08-27 12:26 UTC (permalink / raw) To: Markus Törnqvist Cc: Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Fri, Aug 27, 2004 at 12:19:54PM +0300, Markus Törnqvist wrote: > That's why I tend to rename metas/ into ..metas/ to avoid > name clashes, even if I've never had a directory named metas/ > apart from what Reiser4 ships. http://packages.debian.org/cgi-bin/search_contents.pl?word=metas&searchmode=searchfilesanddirs&case=insensitive&version=unstable&arch=i386 OK, those are capital METAS rather than junior metas, but it does show this is not a unique word to reiser4. -- "Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 12:26 ` Matthew Wilcox 0 siblings, 0 replies; 1835+ messages in thread From: Matthew Wilcox @ 2004-08-27 12:26 UTC (permalink / raw) To: Markus Törnqvist Cc: Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Fri, Aug 27, 2004 at 12:19:54PM +0300, Markus Törnqvist wrote: > That's why I tend to rename metas/ into ..metas/ to avoid > name clashes, even if I've never had a directory named metas/ > apart from what Reiser4 ships. http://packages.debian.org/cgi-bin/search_contents.pl?word=metas&searchmode=searchfilesanddirs&case=insensitive&version=unstable&arch=i386 OK, those are capital METAS rather than junior metas, but it does show this is not a unique word to reiser4. -- "Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:00 ` Jamie Lokier 2004-08-26 3:13 ` viro 2004-08-26 8:49 ` Hans Reiser @ 2004-08-26 10:05 ` Chris Wedgwood 2004-08-26 11:02 ` Jamie Lokier 2004-08-27 21:01 ` Pavel Machek 2 siblings, 2 replies; 1835+ messages in thread From: Chris Wedgwood @ 2004-08-26 10:05 UTC (permalink / raw) To: Jamie Lokier Cc: viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote: > One of the big potential uses for file-as-directory is to go inside > archive files, ELF files, .iso files and so on in a convenient way. Arguably this belongs in userspace --- and people have put it there. --cw ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:05 ` Chris Wedgwood @ 2004-08-26 11:02 ` Jamie Lokier 2004-08-26 11:19 ` Spam ` (2 more replies) 2004-08-27 21:01 ` Pavel Machek 1 sibling, 3 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 11:02 UTC (permalink / raw) To: Chris Wedgwood Cc: viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Chris Wedgwood wrote: > On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote: > > > One of the big potential uses for file-as-directory is to go inside > > archive files, ELF files, .iso files and so on in a convenient way. > > Arguably this belongs in userspace --- and people have put it there. I agree that these belong in userspace, and that there's plenty* of userspace code doing a similar thing already. I don't think there's any argument over it. However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility. -- Jamie * plenty == too much. Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:02 ` Jamie Lokier @ 2004-08-26 11:19 ` Spam 2004-08-26 15:50 ` Will Dyson 2004-08-27 21:06 ` Pavel Machek 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:19 UTC (permalink / raw) To: Jamie Lokier Cc: Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Chris Wedgwood wrote: >> On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote: >> >> > One of the big potential uses for file-as-directory is to go inside >> > archive files, ELF files, .iso files and so on in a convenient way. >> >> Arguably this belongs in userspace --- and people have put it there. > I agree that these belong in userspace, and that there's plenty* of > userspace code doing a similar thing already. I don't think there's > any argument over it. > However, as far as I know it's not accessible in a file-as-directory > form as yet. In my opinion that is the most natural form and it would > be very intuitive to use. I hope we can pick a useful semantics for > them, and also provide filesystem-independent plugins with GNU > Hurd-like per-user extensibility. > -- Jamie > * plenty == too much. > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > (All but Bash implement their own virtual filesystem extensions). > That makes them much less useful than they could be. Exactly, and I doubt they have the ability to join together and create a common uniform standard out of it either. They have just far to different views on Linux and how they want things to be done. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 11:19 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:19 UTC (permalink / raw) To: Jamie Lokier Cc: Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Chris Wedgwood wrote: >> On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote: >> >> > One of the big potential uses for file-as-directory is to go inside >> > archive files, ELF files, .iso files and so on in a convenient way. >> >> Arguably this belongs in userspace --- and people have put it there. > I agree that these belong in userspace, and that there's plenty* of > userspace code doing a similar thing already. I don't think there's > any argument over it. > However, as far as I know it's not accessible in a file-as-directory > form as yet. In my opinion that is the most natural form and it would > be very intuitive to use. I hope we can pick a useful semantics for > them, and also provide filesystem-independent plugins with GNU > Hurd-like per-user extensibility. > -- Jamie > * plenty == too much. > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > (All but Bash implement their own virtual filesystem extensions). > That makes them much less useful than they could be. Exactly, and I doubt they have the ability to join together and create a common uniform standard out of it either. They have just far to different views on Linux and how they want things to be done. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:19 ` Spam (?) @ 2004-08-28 0:28 ` Horst von Brand -1 siblings, 0 replies; 1835+ messages in thread From: Horst von Brand @ 2004-08-28 0:28 UTC (permalink / raw) To: Spam Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: > > Chris Wedgwood wrote: [...] > > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > > (All but Bash implement their own virtual filesystem extensions). > > That makes them much less useful than they could be. > Exactly, and I doubt they have the ability to join together and > create a common uniform standard out of it either. They have just > far to different views on Linux and how they want things to be done. And the solution is... inventing yet another, incompatible way of doing things that nobody outside its followers will ever use. Inside the kernel. Should have thought of that before. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:02 ` Jamie Lokier 2004-08-26 11:19 ` Spam @ 2004-08-26 15:50 ` Will Dyson 2004-08-26 20:38 ` Lee Revell 2004-08-27 21:06 ` Pavel Machek 2 siblings, 1 reply; 1835+ messages in thread From: Will Dyson @ 2004-08-26 15:50 UTC (permalink / raw) To: Jamie Lokier Cc: Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier wrote: > However, as far as I know it's not accessible in a file-as-directory > form as yet. In my opinion that is the most natural form and it would > be very intuitive to use. I hope we can pick a useful semantics for > them, and also provide filesystem-independent plugins with GNU > Hurd-like per-user extensibility. > > -- Jamie > > * plenty == too much. > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > (All but Bash implement their own virtual filesystem extensions). > That makes them much less useful than they could be. It has always bugged me that Gnome and KDE implement their own VFS layers. It seems to me that having a standard userspace filesystem layer like FUSE (http://sourceforge.net/projects/avf) would provide a lot of the benefit that HURD users (all 5 of them) see from filesystem "translators". Now, safely allowing unprivileged users to run arbitrary userspace filesystems would be a real trick. But, if it were possible, it could be combined with files-as-directories in some rather neat ways. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:50 ` Will Dyson @ 2004-08-26 20:38 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 20:38 UTC (permalink / raw) To: Will Dyson Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-08-26 at 11:50, Will Dyson wrote: > Jamie Lokier wrote: > > > However, as far as I know it's not accessible in a file-as-directory > > form as yet. In my opinion that is the most natural form and it would > > be very intuitive to use. I hope we can pick a useful semantics for > > them, and also provide filesystem-independent plugins with GNU > > Hurd-like per-user extensibility. > > > > -- Jamie > > > > * plenty == too much. > > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > > (All but Bash implement their own virtual filesystem extensions). > > That makes them much less useful than they could be. > > It has always bugged me that Gnome and KDE implement their own VFS layers. > Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 20:38 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 20:38 UTC (permalink / raw) To: Will Dyson Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-08-26 at 11:50, Will Dyson wrote: > Jamie Lokier wrote: > > > However, as far as I know it's not accessible in a file-as-directory > > form as yet. In my opinion that is the most natural form and it would > > be very intuitive to use. I hope we can pick a useful semantics for > > them, and also provide filesystem-independent plugins with GNU > > Hurd-like per-user extensibility. > > > > -- Jamie > > > > * plenty == too much. > > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > > (All but Bash implement their own virtual filesystem extensions). > > That makes them much less useful than they could be. > > It has always bugged me that Gnome and KDE implement their own VFS layers. > Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:38 ` Lee Revell @ 2004-08-26 20:50 ` Christophe Saout -1 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 20:50 UTC (permalink / raw) To: Lee Revell Cc: Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 801 bytes --] Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell: > > It has always bugged me that Gnome and KDE implement their own VFS layers. > > Same here. This always seemed like something the kernel should be able > to handle. It seems to me that if reiser4 had been available at the > time the Gnome and KDE developers would not have needed to do this. Well, the kernel doesn't have a filesystem that speaks http, scp and those things. GnomeVFS is URL-based. It has some pseudo-protocols that extract a pseudo directory-tree for all installed applications + the changes the used made, created on the fly from a set of XML files that are read-only and system-wide and the user-overridden changes. I don't know if all of these things would really make sense inside the kernel. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 20:50 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 20:50 UTC (permalink / raw) To: Lee Revell Cc: Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 801 bytes --] Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell: > > It has always bugged me that Gnome and KDE implement their own VFS layers. > > Same here. This always seemed like something the kernel should be able > to handle. It seems to me that if reiser4 had been available at the > time the Gnome and KDE developers would not have needed to do this. Well, the kernel doesn't have a filesystem that speaks http, scp and those things. GnomeVFS is URL-based. It has some pseudo-protocols that extract a pseudo directory-tree for all installed applications + the changes the used made, created on the fly from a set of XML files that are read-only and system-wide and the user-overridden changes. I don't know if all of these things would really make sense inside the kernel. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:50 ` Christophe Saout @ 2004-08-26 20:57 ` Lee Revell -1 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 20:57 UTC (permalink / raw) To: Christophe Saout Cc: Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: > Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell: > > > > It has always bugged me that Gnome and KDE implement their own VFS layers. > > > > Same here. This always seemed like something the kernel should be able > > to handle. It seems to me that if reiser4 had been available at the > > time the Gnome and KDE developers would not have needed to do this. > > Well, the kernel doesn't have a filesystem that speaks http, scp and > those things. GnomeVFS is URL-based. It has some pseudo-protocols that > extract a pseudo directory-tree for all installed applications + the > changes the used made, created on the fly from a set of XML files that > are read-only and system-wide and the user-overridden changes. I don't > know if all of these things would really make sense inside the kernel. > True. FWIW, I never use most of those features. It's just too damn slow. Windows seems to implement all of the useful features of GnomeVFS, and they are 10x faster. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 20:57 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 20:57 UTC (permalink / raw) To: Christophe Saout Cc: Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: > Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell: > > > > It has always bugged me that Gnome and KDE implement their own VFS layers. > > > > Same here. This always seemed like something the kernel should be able > > to handle. It seems to me that if reiser4 had been available at the > > time the Gnome and KDE developers would not have needed to do this. > > Well, the kernel doesn't have a filesystem that speaks http, scp and > those things. GnomeVFS is URL-based. It has some pseudo-protocols that > extract a pseudo directory-tree for all installed applications + the > changes the used made, created on the fly from a set of XML files that > are read-only and system-wide and the user-overridden changes. I don't > know if all of these things would really make sense inside the kernel. > True. FWIW, I never use most of those features. It's just too damn slow. Windows seems to implement all of the useful features of GnomeVFS, and they are 10x faster. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:57 ` Lee Revell (?) @ 2004-08-27 9:21 ` Markus Törnqvist -1 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-27 9:21 UTC (permalink / raw) To: Lee Revell Cc: Christophe Saout, Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote: >On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: >> are read-only and system-wide and the user-overridden changes. I don't >> know if all of these things would really make sense inside the kernel. >True. FWIW, I never use most of those features. It's just too damn >slow. Windows seems to implement all of the useful features of >GnomeVFS, and they are 10x faster. Are they in the kernel in Windows? -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 9:21 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: mjt @ 2004-08-27 9:21 UTC (permalink / raw) To: Lee Revell Cc: Christophe Saout, Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote: >On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: >> are read-only and system-wide and the user-overridden changes. I don't >> know if all of these things would really make sense inside the kernel. >True. FWIW, I never use most of those features. It's just too damn >slow. Windows seems to implement all of the useful features of >GnomeVFS, and they are 10x faster. Are they in the kernel in Windows? -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 9:21 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-27 9:21 UTC (permalink / raw) To: Lee Revell Cc: Christophe Saout, Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote: >On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: >> are read-only and system-wide and the user-overridden changes. I don't >> know if all of these things would really make sense inside the kernel. >True. FWIW, I never use most of those features. It's just too damn >slow. Windows seems to implement all of the useful features of >GnomeVFS, and they are 10x faster. Are they in the kernel in Windows? -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 9:21 ` Markus Törnqvist @ 2004-08-27 11:34 ` Spam -1 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-27 11:34 UTC (permalink / raw) To: Markus Törnqvist Cc: Lee Revell, Christophe Saout, Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hello Markus, Friday, August 27, 2004, 11:21:31 AM, you wrote: > On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote: >>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: >>> are read-only and system-wide and the user-overridden changes. I don't >>> know if all of these things would really make sense inside the kernel. >>True. FWIW, I never use most of those features. It's just too damn >>slow. Windows seems to implement all of the useful features of >>GnomeVFS, and they are 10x faster. > Are they in the kernel in Windows? Some are, some aren't :) Many of the features are between the FS and the application layers. Therefore they will work with all programs that use the Windows API to access files etc. I do not think that many programs actually access disks directly at the FS or below level. Exceptions would perhaps be file-recovery tools etc. Others are simply plugins to Windows Explorer. I don't know if all, any, or none, are installed at the kernel level. I doubt they are. But the problem with Linux seem to be that programs actually do access the FS directly. Therefore everything will break if you change the FS structures to much. Like I said, this is IMHO something that Linux should somehow solve. Applications should really not access the filesystems directly. Perhaps Grub and stuff like it should, but not many other. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 11:34 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-27 11:34 UTC (permalink / raw) To: Markus Törnqvist Cc: Lee Revell, Christophe Saout, Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hello Markus, Friday, August 27, 2004, 11:21:31 AM, you wrote: > On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote: >>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: >>> are read-only and system-wide and the user-overridden changes. I don't >>> know if all of these things would really make sense inside the kernel. >>True. FWIW, I never use most of those features. It's just too damn >>slow. Windows seems to implement all of the useful features of >>GnomeVFS, and they are 10x faster. > Are they in the kernel in Windows? Some are, some aren't :) Many of the features are between the FS and the application layers. Therefore they will work with all programs that use the Windows API to access files etc. I do not think that many programs actually access disks directly at the FS or below level. Exceptions would perhaps be file-recovery tools etc. Others are simply plugins to Windows Explorer. I don't know if all, any, or none, are installed at the kernel level. I doubt they are. But the problem with Linux seem to be that programs actually do access the FS directly. Therefore everything will break if you change the FS structures to much. Like I said, this is IMHO something that Linux should somehow solve. Applications should really not access the filesystems directly. Perhaps Grub and stuff like it should, but not many other. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:38 ` Lee Revell (?) (?) @ 2004-08-26 21:00 ` Jamie Lokier -1 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 21:00 UTC (permalink / raw) To: Lee Revell Cc: Will Dyson, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Lee Revell wrote: > Same here. This always seemed like something the kernel should be able > to handle. It seems to me that if reiser4 had been available at the > time the Gnome and KDE developers would not have needed to do this. reiser4 has prompted the discussion again, but it doesn't provide the virtual filesystem capabilities that we're talking about, which Gnome and KDE implement in userspace. The FUSE project is much more relevant for that. We had file-as-directory discussions years ago, and 5 years ago I was experimenting with doing virtual filesystems with that capability over NFS. podfuk offered something close to Gnome and KDE but in kernel space: virtual filesytems accessible through the kernel, but it didn't get too popular. It was based on Midnight Commander's VFS (which has evolved into Gnome's VFS), using the CODA filesystem interface to provide the kernel hooks. It had some problems, and was a bit fiddly to use. We still don't have something that's particularly nice to use for ordinary users. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:02 ` Jamie Lokier 2004-08-26 11:19 ` Spam 2004-08-26 15:50 ` Will Dyson @ 2004-08-27 21:06 ` Pavel Machek 2004-08-31 1:00 ` David Masover 2 siblings, 1 reply; 1835+ messages in thread From: Pavel Machek @ 2004-08-27 21:06 UTC (permalink / raw) To: Jamie Lokier Cc: Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > One of the big potential uses for file-as-directory is to go inside > > > archive files, ELF files, .iso files and so on in a convenient way. > > > > Arguably this belongs in userspace --- and people have put it there. > > I agree that these belong in userspace, and that there's plenty* of > userspace code doing a similar thing already. I don't think there's > any argument over it. > > However, as far as I know it's not accessible in a file-as-directory > form as yet. In my opinion that is the most natural form and it would > be very intuitive to use. It does not work. .deb file is ar archive. It is also debian package. How do you know if I want to see it as a package or as a archive? How do you identify file types, anyway? Same happens for .tar.gz. uservfs does cd foo.deb#uar vs. cd foo.deb#udeb and cd foo.tgz#utar vs. cat foo.tgz#ugz -- 64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 21:06 ` Pavel Machek @ 2004-08-31 1:00 ` David Masover 2004-08-31 8:21 ` Pavel Machek 0 siblings, 1 reply; 1835+ messages in thread From: David Masover @ 2004-08-31 1:00 UTC (permalink / raw) To: Pavel Machek Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Pavel Machek wrote: [...] | uservfs does | | cd foo.deb#uar cd foo.deb/ar | vs. | cd foo.deb#udeb cd foo.deb/deb and why would you want that, instead of just: cd foo.deb # for the ar dpkg -i foo.deb # for the deb | | and | | cd foo.tgz#utar cd foo.tgz | vs. | cat foo.tgz#ugz cat foo.tgz Just want to extract the tar file? Maybe something like cat foo.tgz/gunzip In which case (of course) foo.tgz/gunzip has exactly the same directory contents as foo.tgz Looks different, that's all. In fact, for just about any syntax anyone could suggest, I can't really see why you can't just replace all weird symbols with a slash and a symbol. Instead of foo.tgz#utar you have foo.tgz/#/utar Only difference is, some things which used to require special tools can now be serviced by less than what's in busybox. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTPNpngHNmZLgCUhAQJ0PA//RmPgHL4MgiQn7Li5b52DI/0p+Zp9JUs7 OaA+k4wHeXEIdBProtBQ+3541noEg086bVriYRnajsA9xgd/P9ncE7ci/d9Kagdl 8KmH4eWnCjNkL4x8GnDeR+EM5EfcgoSxR7ezO1KO9uMMcPFnCvmeB1yB5U+Fr9Q1 HVPHfGsh+ZWU2CFJHoJx27Q07UK+bFvzGr4JkhLbRMbPeEROrkHJHp3wj/N2yXXb 5icLgFa0g/b6wM8SuBAaO5xVM8jkmECI/P5Jo3n/d/CTUumb/BoKMIgbMurR61CG hrZ40mizpCMFAnLrgm9FIvjKtZUsigG+oM/2xTtku2Z2nSM16/ChKtJOAj6MoMGT xvWzLdTY6kEL30VDaHtcVvIyUFtUe5oYxEasnrcKseV7hJtccgjTYKB21PnWyZFz /KIdh8sBg6i1nuYOtHncL6agD1M2bg1vBYwbMvIRfa1YbAFUbW1A6hDTftxJIhLn Beso6d68SyfNMdQ8yWloR7sIufmbDut2fx1SHS3wPt2Z5W1e0XBexcpSqdD8KPns lvmC0ggPDuQzXjPCIsS17Uk5vrCtVEdrsCCaj62a8UE2d1mCqu4CyLwMYCFv3Sve SwZ36iOv4fzdvgjPOUg27tMwbpa/WiDAcZJ/31BfZfTDl4E9LbkQjWYT/YgsJuaQ wp3Hi/Cgh/4= =8YVp -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 1:00 ` David Masover @ 2004-08-31 8:21 ` Pavel Machek 2004-08-31 18:28 ` Hubert Chan ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: Pavel Machek @ 2004-08-31 8:21 UTC (permalink / raw) To: David Masover Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > | uservfs does > | > | cd foo.deb#uar > cd foo.deb/ar > | vs. > | cd foo.deb#udeb > cd foo.deb/deb > > and why would you want that, instead of just: > cd foo.deb # for the ar > dpkg -i foo.deb # for the deb Because I want to see contents of that .deb package, nicely parsed? > Just want to extract the tar file? Maybe something like > cat foo.tgz/gunzip > In which case (of course) foo.tgz/gunzip has exactly the same directory > contents as foo.tgz Yes, that would work. > In fact, for just about any syntax anyone could suggest, I can't really > see why you can't just replace all weird symbols with a slash and a > symbol. Instead of > foo.tgz#utar > you have > foo.tgz/#/utar > Only difference is, some things which used to require special tools can > now be serviced by less than what's in busybox. That would work, too. I do not get your comment about busybox. Pavel -- People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl! ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 8:21 ` Pavel Machek @ 2004-08-31 18:28 ` Hubert Chan 2004-08-31 19:31 ` Horst von Brand 2004-09-01 1:10 ` David Masover 2 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-31 18:28 UTC (permalink / raw) To: linux-fsdevel; +Cc: linux-kernel, reiserfs-list >>>>> "Pavel" == Pavel Machek <pavel@ucw.cz> writes: Pavel> Hi! >> | uservfs does >> | >> | cd foo.deb#uar cd foo.deb/ar | vs. | cd foo.deb#udeb cd >> foo.deb/deb >> >> and why would you want that, instead of just: cd foo.deb # for the ar >> dpkg -i foo.deb # for the deb Pavel> Because I want to see contents of that .deb package, nicely Pavel> parsed? What nicely-parsed information do you want to see? A deb package is an ar archive of three files: - debian-binary, which contains the deb format version - control.tar.gz, which contains the control data and related things - data.tar.gz, which contains the actual files to be installed So to get at the format version, you can "cat foo.deb/debian-binary". To look at the control data, you can "ls foo.deb/control.tar.gz". To browse the files, "cd foo.deb/data.tar.gz". -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 8:21 ` Pavel Machek 2004-08-31 18:28 ` Hubert Chan @ 2004-08-31 19:31 ` Horst von Brand 2004-08-31 20:05 ` Linus Torvalds 2004-09-01 1:14 ` David Masover 2004-09-01 1:10 ` David Masover 2 siblings, 2 replies; 1835+ messages in thread From: Horst von Brand @ 2004-08-31 19:31 UTC (permalink / raw) To: Pavel Machek Cc: David Masover, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Pavel Machek <pavel@ucw.cz> said: > David Masover <ninja@slaphack.com> said: [...] > > Just want to extract the tar file? Maybe something like > > cat foo.tgz/gunzip > > In which case (of course) foo.tgz/gunzip has exactly the same directory > > contents as foo.tgz > > Yes, that would work. > > > In fact, for just about any syntax anyone could suggest, I can't really > > see why you can't just replace all weird symbols with a slash and a > > symbol. Instead of > > foo.tgz#utar > > you have > > foo.tgz/#/utar > > Only difference is, some things which used to require special tools can > > now be serviced by less than what's in busybox. > > That would work, too. I do not get your comment about busybox. You do need extra tools anyway, placing them in the kernel is cheating (and absolutely pointless, IMHO). -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 19:31 ` Horst von Brand @ 2004-08-31 20:05 ` Linus Torvalds 2004-08-31 20:29 ` Spam ` (5 more replies) 2004-09-01 1:14 ` David Masover 1 sibling, 6 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-31 20:05 UTC (permalink / raw) To: Horst von Brand Cc: Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 31 Aug 2004, Horst von Brand wrote: > > You do need extra tools anyway, placing them in the kernel is cheating (and > absolutely pointless, IMHO). I agree. There's no point to having the kernel export information that is already inherent in the main stream. I've seen all these examples of exposing MP3 ID information as a "side stream", and that's TOTALLY POINTLESS! The information is already there, it's in a standard format, and exporting it as a stream buys you absolutely nothing. Where named attributes make sense is when they are _independent_ data. Data that is only tangentially related to the main stream itself, and which the main stream _cannot_ encompass because of some real technical issue. In a graphical environment, the "icon" stream is a good example of this. It literally has _nothing_ to do with the data in the main stream. The only linkage is a totally non-technical one, where the user wanted to associate a secondary stream with the main stream _without_ altering the main one. THAT is where named streams make sense. But if you want to look at one particular file inside a tar-file, do so in user space. There are zero advantages to exposing it as a side-stream, and there are absolutely _tons_ of disadvantages. In short, named streams only make sense if: - they are tied to the file some way that is _independent_ of the file contents (since if it's dependent on the file contents, you're just a ton better off regenerating it with a caching server) - there are serious reasons to keep the lookup synchronized (since if there isn't such a reason, you're just better off with a separate shadow tree in user space) And realize that the "separate shadow tree" actually works very well. That's how version control systems like CVS have always worked. It's certainly how you can make icon information work too. If you use a tool for accessing the data, the tool can maintain coherency and you'll never care about the side stream. Which means that normally we really don't _want_ named streams. In 99% of all cases we can use equally good - and _much_ simpler - tool-based solutions. Which means that the only _real_ technical issue for supporting named streams really ends up being things like samba, which want named streams just because the work they do fundamentally is about them, for externally dictated reasons. Doing named streams for any other reason is likely just being stupid. Once you do decide that you have to do named streams, you might then decide to use them for convenient things like icons. But it should very much be a secondary issue at that point. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds @ 2004-08-31 20:29 ` Spam 2004-08-31 20:32 ` Pavel Machek ` (4 subsequent siblings) 5 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-31 20:29 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Tue, 31 Aug 2004, Horst von Brand wrote: >> >> You do need extra tools anyway, placing them in the kernel is cheating (and >> absolutely pointless, IMHO). > I agree. > There's no point to having the kernel export information that is already > inherent in the main stream. There are two things you are saying in the same sentence here. One I do not agree with. There is point to export information from the main stream to secondary file streams. Just it doesn't have to be done by the kernel. Just do not limit things to a few defined types of things that can be stored in a stream. IMHO a file stream should be able to contain just about anything the user wishes. That said, the kernel should not have to know about the contents. > I've seen all these examples of exposing MP3 ID information as a "side > stream", and that's TOTALLY POINTLESS! The information is already there, > it's in a standard format, and exporting it as a stream buys you > absolutely nothing. True. But this is the case for this specific example. I have stated previously about what I think would be legitimate uses as examples. Of course these apply to me and what I work with. One of the examples were to add streams that contained descriptions or keywords. > Where named attributes make sense is when they are _independent_ data. > Data that is only tangentially related to the main stream itself, and > which the main stream _cannot_ encompass because of some real technical > issue. > In a graphical environment, the "icon" stream is a good example of this. > It literally has _nothing_ to do with the data in the main stream. The > only linkage is a totally non-technical one, where the user wanted to > associate a secondary stream with the main stream _without_ altering the > main one. THAT is where named streams make sense. > But if you want to look at one particular file inside a tar-file, do so in > user space. There are zero advantages to exposing it as a side-stream, and > there are absolutely _tons_ of disadvantages. The tar archive support seem (is) to be useless because tar is also so supported and implemented in every type of application out there. It is also easy to script and otherwise work with. But the idea to be able to extend the filesystem to show/represent data as file streams and meta directores is very good. Someone might come up with an idea to allow browsing and changing of data inside XML files. Of course this may already be done with sed and other tools. > In short, named streams only make sense if: > - they are tied to the file some way that is _independent_ of the file > contents (since if it's dependent on the file contents, you're just a > ton better off regenerating it with a caching server) > - there are serious reasons to keep the lookup synchronized (since if > there isn't such a reason, you're just better off with a separate > shadow tree in user space) > And realize that the "separate shadow tree" actually works very well. > That's how version control systems like CVS have always worked. It's > certainly how you can make icon information work too. If you use a tool > for accessing the data, the tool can maintain coherency and you'll never > care about the side stream. Just that with shadow trees (I assume they are stored simply in another folder beside the file) you won't be able to "cp filea /mnt/backup" and keep the extra stuff. Sure icons could also easily be stores as file.icon, just like on the Amiga :). It just isn't as nice. > Which means that normally we really don't _want_ named streams. In 99% of > all cases we can use equally good - and _much_ simpler - tool-based > solutions. > Which means that the only _real_ technical issue for supporting named > streams really ends up being things like samba, which want named streams > just because the work they do fundamentally is about them, for externally > dictated reasons. Doing named streams for any other reason is likely just > being stupid. I doubt it would be stupid. Perhaps today just not needed for many things. > Once you do decide that you have to do named streams, you might then > decide to use them for convenient things like icons. But it should very > much be a secondary issue at that point. I think we focus to much on the actual contents of these streams. Do we really have to know what will in the future be stored inside these streams? For example. Applications may use file streams for locking instead of having a .lock file. The future isn't known, and we shouldn't try to limit what we can put in the streams. You said 99% of the uses won't need them. True. The same is with xattr, ACLs and so many other things. Even if 0.1% of the users today would use it I think we should have the support for named streams. ~S > Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-31 20:29 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-31 20:29 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Tue, 31 Aug 2004, Horst von Brand wrote: >> >> You do need extra tools anyway, placing them in the kernel is cheating (and >> absolutely pointless, IMHO). > I agree. > There's no point to having the kernel export information that is already > inherent in the main stream. There are two things you are saying in the same sentence here. One I do not agree with. There is point to export information from the main stream to secondary file streams. Just it doesn't have to be done by the kernel. Just do not limit things to a few defined types of things that can be stored in a stream. IMHO a file stream should be able to contain just about anything the user wishes. That said, the kernel should not have to know about the contents. > I've seen all these examples of exposing MP3 ID information as a "side > stream", and that's TOTALLY POINTLESS! The information is already there, > it's in a standard format, and exporting it as a stream buys you > absolutely nothing. True. But this is the case for this specific example. I have stated previously about what I think would be legitimate uses as examples. Of course these apply to me and what I work with. One of the examples were to add streams that contained descriptions or keywords. > Where named attributes make sense is when they are _independent_ data. > Data that is only tangentially related to the main stream itself, and > which the main stream _cannot_ encompass because of some real technical > issue. > In a graphical environment, the "icon" stream is a good example of this. > It literally has _nothing_ to do with the data in the main stream. The > only linkage is a totally non-technical one, where the user wanted to > associate a secondary stream with the main stream _without_ altering the > main one. THAT is where named streams make sense. > But if you want to look at one particular file inside a tar-file, do so in > user space. There are zero advantages to exposing it as a side-stream, and > there are absolutely _tons_ of disadvantages. The tar archive support seem (is) to be useless because tar is also so supported and implemented in every type of application out there. It is also easy to script and otherwise work with. But the idea to be able to extend the filesystem to show/represent data as file streams and meta directores is very good. Someone might come up with an idea to allow browsing and changing of data inside XML files. Of course this may already be done with sed and other tools. > In short, named streams only make sense if: > - they are tied to the file some way that is _independent_ of the file > contents (since if it's dependent on the file contents, you're just a > ton better off regenerating it with a caching server) > - there are serious reasons to keep the lookup synchronized (since if > there isn't such a reason, you're just better off with a separate > shadow tree in user space) > And realize that the "separate shadow tree" actually works very well. > That's how version control systems like CVS have always worked. It's > certainly how you can make icon information work too. If you use a tool > for accessing the data, the tool can maintain coherency and you'll never > care about the side stream. Just that with shadow trees (I assume they are stored simply in another folder beside the file) you won't be able to "cp filea /mnt/backup" and keep the extra stuff. Sure icons could also easily be stores as file.icon, just like on the Amiga :). It just isn't as nice. > Which means that normally we really don't _want_ named streams. In 99% of > all cases we can use equally good - and _much_ simpler - tool-based > solutions. > Which means that the only _real_ technical issue for supporting named > streams really ends up being things like samba, which want named streams > just because the work they do fundamentally is about them, for externally > dictated reasons. Doing named streams for any other reason is likely just > being stupid. I doubt it would be stupid. Perhaps today just not needed for many things. > Once you do decide that you have to do named streams, you might then > decide to use them for convenient things like icons. But it should very > much be a secondary issue at that point. I think we focus to much on the actual contents of these streams. Do we really have to know what will in the future be stored inside these streams? For example. Applications may use file streams for locking instead of having a .lock file. The future isn't known, and we shouldn't try to limit what we can put in the streams. You said 99% of the uses won't need them. True. The same is with xattr, ACLs and so many other things. Even if 0.1% of the users today would use it I think we should have the support for named streams. ~S > Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds 2004-08-31 20:29 ` Spam @ 2004-08-31 20:32 ` Pavel Machek 2004-08-31 20:38 ` Linus Torvalds 2004-08-31 21:07 ` Hubert Chan ` (3 subsequent siblings) 5 siblings, 1 reply; 1835+ messages in thread From: Pavel Machek @ 2004-08-31 20:32 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > You do need extra tools anyway, placing them in the kernel is cheating (and > > absolutely pointless, IMHO). > > I agree. > > There's no point to having the kernel export information that is already > inherent in the main stream. > > I've seen all these examples of exposing MP3 ID information as a "side > stream", and that's TOTALLY POINTLESS! The information is already there, > it's in a standard format, and exporting it as a stream buys you > absolutely nothing. It buys me caching. I do quite often bzcat patch.2.6.8.bz2 | less (read the patch) (sometimes repeat that few times because I hit ^c when I should not have etc). cd ...clean; bzcat patch.2.6.8.bz2 | patch -Esp1 cd ...linux; bzcat patch.2.6.8.bz2 | patch -Esp1 Now... that's total waste of cpu. bzip2 decompression takes quite some time. I could do bzcat patch.2.6.8.bz2 > /tmp/delete.me.when.you.are.done ...but I'd probably forget to delete that one and anyway, it requires me to think about it. Nicest way would be cat patch.2.6.8.bz2/ubz | less cd ...clean; cat patch.2.6.8.bz2/ubz | patch -Esp1 cd ...linux; cat patch.2.6.8.bz2/ubz | patch -Esp1 with kernel intelligently caching uncompressed data. I believe this can not be done completely in userspace. Pavel -- People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl! ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:32 ` Pavel Machek @ 2004-08-31 20:38 ` Linus Torvalds 2004-08-31 20:54 ` Pavel Machek 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-31 20:38 UTC (permalink / raw) To: Pavel Machek Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 31 Aug 2004, Pavel Machek wrote: > > It buys me caching. I'll actually buy into that. If only because I consider caching to be one of the more important things that the kernel does (caches are a _classic_ case of "shared data that needs synchronization"). However, that said, user space can trivially cache things in the filesystem, so while this may be a convenient feature, I think you should look at perhaps doing it in the _shell_ instead.. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:38 ` Linus Torvalds @ 2004-08-31 20:54 ` Pavel Machek 2004-08-31 20:59 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: Pavel Machek @ 2004-08-31 20:54 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > It buys me caching. > > I'll actually buy into that. If only because I consider caching to be one > of the more important things that the kernel does (caches are a _classic_ > case of "shared data that needs synchronization"). > > However, that said, user space can trivially cache things in the > filesystem, so while this may be a convenient feature, I think you should > look at perhaps doing it in the _shell_ instead.. That cache should disappear as soon as I need disk space. I.e. userspace should never see -ENOSPC because of this kind of caching. This need some kernel support. Ouch and cached file should atomically go away as soon as main file changes, otherwise I do not see how multiple processes could cooperate on caching... chattr +kill-this-file-when-low-on-disk-space patch.bz2...ubz would solve first problem. Not sure how to do the second one. Pavel -- People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl! ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:54 ` Pavel Machek @ 2004-08-31 20:59 ` Linus Torvalds 2004-08-31 22:07 ` Pavel Machek 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-31 20:59 UTC (permalink / raw) To: Pavel Machek Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 31 Aug 2004, Pavel Machek wrote: > > > > However, that said, user space can trivially cache things in the > > filesystem, so while this may be a convenient feature, I think you should > > look at perhaps doing it in the _shell_ instead.. > > That cache should disappear as soon as I need disk > space. I.e. userspace should never see -ENOSPC because of this kind of > caching. This need some kernel support. Ouch and cached file should > atomically go away as soon as main file changes, otherwise I do not > see how multiple processes could cooperate on caching... Well, what other projects have done is to just reserve a certain amount of disk for caching. See "ccache", which solves both of the above problems (it doesn't shrink the cache on ENOSPC, but reserving diskspace is accepted practice for things like this..) Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:59 ` Linus Torvalds @ 2004-08-31 22:07 ` Pavel Machek 2004-08-31 23:19 ` Christer Weinigel 0 siblings, 1 reply; 1835+ messages in thread From: Pavel Machek @ 2004-08-31 22:07 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > However, that said, user space can trivially cache things in the > > > filesystem, so while this may be a convenient feature, I think you should > > > look at perhaps doing it in the _shell_ instead.. > > > > That cache should disappear as soon as I need disk > > space. I.e. userspace should never see -ENOSPC because of this kind of > > caching. This need some kernel support. Ouch and cached file should > > atomically go away as soon as main file changes, otherwise I do not > > see how multiple processes could cooperate on caching... > > Well, what other projects have done is to just reserve a certain amount of > disk for caching. See "ccache", which solves both of the above problems > (it doesn't shrink the cache on ENOSPC, but reserving diskspace is > accepted practice for things like this..) Okay, that does work, it just is not really nice. Just as reserving fixed ammount of space for disk cache is bad, reserving fixed ammount of space for ccache (and similar) is bad. When there are few of such caches, balancing between them starts to matter... Pavel -- People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl! ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 22:07 ` Pavel Machek @ 2004-08-31 23:19 ` Christer Weinigel 2004-09-01 1:43 ` David Masover 2004-09-01 5:02 ` Pavel Machek 0 siblings, 2 replies; 1835+ messages in thread From: Christer Weinigel @ 2004-08-31 23:19 UTC (permalink / raw) To: Pavel Machek Cc: Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Pavel Machek <pavel@ucw.cz> writes: > Okay, that does work, it just is not really nice. Just as reserving > fixed ammount of space for disk cache is bad, reserving fixed ammount > of space for ccache (and similar) is bad. When there are few of such > caches, balancing between them starts to matter... So try to convince them to use the same cache daemon or the same shared cache manager library then. It isn't all that different from implementing a kernel interface that everyone is supposed to use. A cache manager daemon could sit and watch the free space on the disk every other second and start deleting the cached files (according to some LCU heuristics or whatever) whenever free space is getting low. To see if the original file is newer than the cached file, good old mtime can be used. I can't remember who right now, but someone mentioned a few problems with timestamps: 1. The resolutions is too low in current kernels, but that just means that someone(tm) needs to implement nanosecond timestamps in the kernel and for the filesystems. 2. Even nanosecond timestams may become to coarse in the future. One way of getting around this is to keep track of the latest timestamp written and make sure the next timestamp is one higher than the latest one. This way timestamps are always increasing, and I doubt that we can have modifications each nanosecond for a long time with the hardware available for the next ten years. 3. If time goes backwards beacuse somebody changes the clock, the caching (or makefile rules) can get confued. This is a hard one, but I actually we can use the doctors response "don't do that then". With NTP we can guarantee a monotonically increasing clock. I belive the kernel could give some assistance to make it easier to see if a file has been modified, I remember that a few suggestions were thrown around the last time Samba and dcache aliases were discussed on l-k. I definitely belive that kind of infrastructure belongs in the kernel. But the cache manager itself, no. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 23:19 ` Christer Weinigel @ 2004-09-01 1:43 ` David Masover 2004-09-01 5:02 ` Pavel Machek 1 sibling, 0 replies; 1835+ messages in thread From: David Masover @ 2004-09-01 1:43 UTC (permalink / raw) To: Christer Weinigel Cc: Pavel Machek, Linus Torvalds, Horst von Brand, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christer Weinigel wrote: | Pavel Machek <pavel@ucw.cz> writes: | | |>Okay, that does work, it just is not really nice. Just as reserving |>fixed ammount of space for disk cache is bad, reserving fixed ammount |>of space for ccache (and similar) is bad. When there are few of such |>caches, balancing between them starts to matter... | | | So try to convince them to use the same cache daemon or the same | shared cache manager library then. It isn't all that different from | implementing a kernel interface that everyone is supposed to use. You're right, it's not. Is there some huge performance loss by having a kernel interface that then turns around and calls a userland daemon? If yes, I can understand the argument. If no, then the kernel interface should ultimately be able to replace gnome-vfs -- and it should rely heavily on userland utilities. Caching is very generic. It's so generic that there should be exactly one cache manager, the way there is exactly one VM manager for Linux. If there's a choice, choose it at boot or compile-time. Abiword uses gzipped files. So does gnumeric, gnucash, cube (a cute little game engine), and many others. There's no way every single one will suddenly use a new library/daemon. But with plugins and intelligent caching, any kind of compression could be supported without the app needing to support it natively -- either it goes in a compressed folder or the user browses to it and changes an attribute. | A cache manager daemon could sit and watch the free space on the disk | every other second and start deleting the cached files (according to | some LCU heuristics or whatever) whenever free space is getting low. Agreed. The daemon should go in userspace. But the interface should be in the kernel. Just as no one suggested putting tar in the kernel, I don't think anyone would really insist that the daemon needs to go in the kernel. That is, unless critical system files start actually being cached streams of the actual package from the distro... But this way leads to madness. Any serious issues here are better solved by an initrd. [...] | I belive the kernel could give some assistance to make it easier to | see if a file has been modified, I remember that a few suggestions | were thrown around the last time Samba and dcache aliases were | discussed on l-k. I definitely belive that kind of infrastructure | belongs in the kernel. But the cache manager itself, no. Isn't there already dnotify? Dnotify is wrong here, because I don't think it would be sane in a sufficiently large setup. But wouldn't it be nice if the kernel not only told you whether a file has been modified, but told you immediately after it was modified? Yet another reason why there has to be a common cache and significant kernel management -- if I've got a drive of less than, say, 5 gigs, I am simply not willing to reserve any large amount of it for cache. Take the example of a source tarball being automatically extracted and the contents cached. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUpPHgHNmZLgCUhAQLhIg/9G87W4OCt8hOvTK+XniPgWGn+YC7t4fKy mi4QowX1eLmnFEovprjSUzPgKRMeDSChZ8M1pvXXWJQ/iULE44WH8vNiH8H974zw h5O8dNI/AczuzaCalQ+BYXEGM8pasaCGCI/sLrr+WBy97NOpM2VUUKY+qKxSFWFt x7u7/XCGGWSHVG3t3KzpkqD2K8u3IGSrV/vHIR2PLatEXu3G8K8sUAhfXMgFsTFp gQAuQN/YzQKGUphnMzFWLDMYyEYjAA4VACZFqFz9QJIYeWzP+YmGdLfZ5mzN91JF g5ACGHUxbYdjZsjTh6RbalVr1HpAAy8mvGNydbwUvuOxaXBvH44vUfRLnvIUoZs9 BB0Tq4xPqAwAtP0Z7b5Qem1ef3gqGXo1hfsEJWJZPv1lssYzn1+Jpn99p6N6VkXI fbAHnypbHeYbf30lC5lvID0yngb+s5EUaxzxOYqFA8QpEJD/bl3S9Qx6bpom/9AL U8kdK8QjNbhHSMrOpl8mg77VRLXdINoV7R5iSXXUMdIcA1j+4R1AIPh62JAMwQDF /Vzek8KNyxvJDgF/y78LiGETZcG/l2W4szEsJFFF4iIKLri5+OFGiLFz8jVNViKu Lr7Q7u7TgWSSykdTrHFcR/WRupAHeL5RB5DA9HBZErqgAgJWz/Rjv743G8PYFKs8 oPhRyePnFkE= =cQLE -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 23:19 ` Christer Weinigel 2004-09-01 1:43 ` David Masover @ 2004-09-01 5:02 ` Pavel Machek 1 sibling, 0 replies; 1835+ messages in thread From: Pavel Machek @ 2004-09-01 5:02 UTC (permalink / raw) To: Christer Weinigel Cc: Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > I belive the kernel could give some assistance to make it easier to > see if a file has been modified, I remember that a few suggestions > were thrown around the last time Samba and dcache aliases were > discussed on l-k. I definitely belive that kind of infrastructure > belongs in the kernel. But the cache manager itself, no. Well, callback "filesystem full" would be nice, too; but I guess that is "support user cache", too... Pavel -- When do you have heart between your knees? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds @ 2004-08-31 21:07 ` Hubert Chan 2004-08-31 20:32 ` Pavel Machek ` (4 subsequent siblings) 5 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-31 21:07 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes: [...] Linus> I've seen all these examples of exposing MP3 ID information as a Linus> "side stream", and that's TOTALLY POINTLESS! The information is Linus> already there, it's in a standard format, and exporting it as a Linus> stream buys you absolutely nothing. Exporting it through the filesystem would mean that I could use my favourite editor to edit MP3/OGG tags, instead of having to use some other program that has a piece-of-crap interface. It's a user-interface issue, and not a technical issue. Someone might then tell me to write my own MP3-tag editor that has an interface that I like. How may MP3 tag editors do we need? Besides, I have better things to do with my time. Actually, I would argue that things like ID3 tags for MP3 files, and EXIF for JPEG exist only because filesystems in general can't support associating metadata/attributes with files. If filesystems were built the "right way" to begin with, all those tagging schemes would never have come into existence. Of course, that's just a dream, but one can dream, can't they? [...] Linus> And realize that the "separate shadow tree" actually works very Linus> well. That's how version control systems like CVS have always Linus> worked. VMS (or some other OS -- too lazy to look it up) exposed version control through the filesystem. Their naming sucked (or so says Pike), but that's what they did. Yes, CVS works, but I wouldn't say that it works very well. If I want to get a specific version of a file, I need to drop to the command line and type in some cvs command that I don't remember and then open the file. I can't just do "C-x C-f foo.c/version/2.1" in emacs. Yes, emacs has version-control support, so that I can open up a specific version. But if v-c isn't exposed through the filesystem, we would have to add support to emacs for every v-c system out there. And do the same for vi, gedit, nano, bluefish, conglomerate, Mozilla Composer, glade, etc. Linus> It's certainly how you can make icon information work too. If you Linus> use a tool for accessing the data, the tool can maintain Linus> coherency and you'll never care about the side stream. The point is that I shouldn't need to use a special tool. I should be able to use my favourite text/image/sound/etc. editor. Using a special tool may work for you, but it isn't the optimal user interface in general. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-31 21:07 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-31 21:07 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes: [...] Linus> I've seen all these examples of exposing MP3 ID information as a Linus> "side stream", and that's TOTALLY POINTLESS! The information is Linus> already there, it's in a standard format, and exporting it as a Linus> stream buys you absolutely nothing. Exporting it through the filesystem would mean that I could use my favourite editor to edit MP3/OGG tags, instead of having to use some other program that has a piece-of-crap interface. It's a user-interface issue, and not a technical issue. Someone might then tell me to write my own MP3-tag editor that has an interface that I like. How may MP3 tag editors do we need? Besides, I have better things to do with my time. Actually, I would argue that things like ID3 tags for MP3 files, and EXIF for JPEG exist only because filesystems in general can't support associating metadata/attributes with files. If filesystems were built the "right way" to begin with, all those tagging schemes would never have come into existence. Of course, that's just a dream, but one can dream, can't they? [...] Linus> And realize that the "separate shadow tree" actually works very Linus> well. That's how version control systems like CVS have always Linus> worked. VMS (or some other OS -- too lazy to look it up) exposed version control through the filesystem. Their naming sucked (or so says Pike), but that's what they did. Yes, CVS works, but I wouldn't say that it works very well. If I want to get a specific version of a file, I need to drop to the command line and type in some cvs command that I don't remember and then open the file. I can't just do "C-x C-f foo.c/version/2.1" in emacs. Yes, emacs has version-control support, so that I can open up a specific version. But if v-c isn't exposed through the filesystem, we would have to add support to emacs for every v-c system out there. And do the same for vi, gedit, nano, bluefish, conglomerate, Mozilla Composer, glade, etc. Linus> It's certainly how you can make icon information work too. If you Linus> use a tool for accessing the data, the tool can maintain Linus> coherency and you'll never care about the side stream. The point is that I shouldn't need to use a special tool. I should be able to use my favourite text/image/sound/etc. editor. Using a special tool may work for you, but it isn't the optimal user interface in general. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-31 21:07 ` Hubert Chan @ 2004-08-31 23:02 ` Christer Weinigel 2004-08-31 23:20 ` Spam 2004-09-01 0:00 ` John Newbigin 2004-09-01 12:10 ` Alan Cox 2004-09-01 19:44 ` Markus Törnqvist 5 siblings, 2 replies; 1835+ messages in thread From: Christer Weinigel @ 2004-08-31 23:02 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds <torvalds@osdl.org> writes: > In a graphical environment, the "icon" stream is a good example of this. > It literally has _nothing_ to do with the data in the main stream. The > only linkage is a totally non-technical one, where the user wanted to > associate a secondary stream with the main stream _without_ altering the > main one. THAT is where named streams make sense. I think that the "icon" argument for named streams is a silly argument, since different users may want to have different icons for the same file. Say that I want /usr/bin/emacs to have the enterprise icon and someone else wants the gnu head icon. And besides, root owns the file anyways, so neither of us mortal users should be able to add a stream to it. Another reason for named streams that usually crops up is the ability set a "preferred application" for a certain file, so that when I double click on a document I want to open it with antiword instead of openoffice. But the same contra-argument applies here, different users have different preferences. I can see the argument for having the equivalent of Content-type or Content-transfer-encoding as a named stream though. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 23:02 ` Christer Weinigel @ 2004-08-31 23:20 ` Spam 2004-09-01 0:00 ` John Newbigin 1 sibling, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-31 23:20 UTC (permalink / raw) To: Christer Weinigel Cc: Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Linus Torvalds <torvalds@osdl.org> writes: >> In a graphical environment, the "icon" stream is a good example of this. >> It literally has _nothing_ to do with the data in the main stream. The >> only linkage is a totally non-technical one, where the user wanted to >> associate a secondary stream with the main stream _without_ altering the >> main one. THAT is where named streams make sense. > I think that the "icon" argument for named streams is a silly > argument, since different users may want to have different icons for > the same file. Say that I want /usr/bin/emacs to have the enterprise > icon and someone else wants the gnu head icon. And besides, root owns > the file anyways, so neither of us mortal users should be able to add > a stream to it. Yet again are we thinking in blocking ways. Firstly this was an example. Usually, though, most users accept the default icon for a file. If they do not they can still change the icon for the link they make on their start-menu/home folder/etc. > Another reason for named streams that usually crops up is the ability > set a "preferred application" for a certain file, so that when I > double click on a document I want to open it with antiword instead of > openoffice. But the same contra-argument applies here, different > users have different preferences. I can make the same argument as for the icons. > I can see the argument for having the equivalent of Content-type or > Content-transfer-encoding as a named stream though. That would be a nice thing. > /Christer ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-31 23:20 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-31 23:20 UTC (permalink / raw) To: Christer Weinigel Cc: Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Linus Torvalds <torvalds@osdl.org> writes: >> In a graphical environment, the "icon" stream is a good example of this. >> It literally has _nothing_ to do with the data in the main stream. The >> only linkage is a totally non-technical one, where the user wanted to >> associate a secondary stream with the main stream _without_ altering the >> main one. THAT is where named streams make sense. > I think that the "icon" argument for named streams is a silly > argument, since different users may want to have different icons for > the same file. Say that I want /usr/bin/emacs to have the enterprise > icon and someone else wants the gnu head icon. And besides, root owns > the file anyways, so neither of us mortal users should be able to add > a stream to it. Yet again are we thinking in blocking ways. Firstly this was an example. Usually, though, most users accept the default icon for a file. If they do not they can still change the icon for the link they make on their start-menu/home folder/etc. > Another reason for named streams that usually crops up is the ability > set a "preferred application" for a certain file, so that when I > double click on a document I want to open it with antiword instead of > openoffice. But the same contra-argument applies here, different > users have different preferences. I can make the same argument as for the icons. > I can see the argument for having the equivalent of Content-type or > Content-transfer-encoding as a named stream though. That would be a nice thing. > /Christer ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 23:02 ` Christer Weinigel 2004-08-31 23:20 ` Spam @ 2004-09-01 0:00 ` John Newbigin 1 sibling, 0 replies; 1835+ messages in thread From: John Newbigin @ 2004-09-01 0:00 UTC (permalink / raw) To: linux-fsdevel Christer Weinigel wrote: > > I think that the "icon" argument for named streams is a silly > argument, since different users may want to have different icons for > the same file. Say that I want /usr/bin/emacs to have the enterprise > icon and someone else wants the gnu head icon. And besides, root owns > the file anyways, so neither of us mortal users should be able to add > a stream to it. > > Another reason for named streams that usually crops up is the ability > set a "preferred application" for a certain file, so that when I > double click on a document I want to open it with antiword instead of > openoffice. But the same contra-argument applies here, different > users have different preferences. > > I can see the argument for having the equivalent of Content-type or > Content-transfer-encoding as a named stream though. You still have the same problem with that. What if I want to override the content-type for some reason. (I have had to do this is the past, but not for root owned files. In the end though I had to edit /etc/mailcap to do what I wanted :( ) You still have problems with icons. What format should they be stored in? If all this effort is to make samba work then would this mean a windows icon? That is not going to make your X11 app happy. This could probably all be solved with magic plugins in the filesystem (reiserfs style) but it is probably better to return to simple approach of "everything is a file". The executable /usr/bin/emacs is a file. The way it looks on my desktop is a file ~/.desktop/emacs etc. This is what most apps do currently and it works quite well. Just my 2c -- John Newbigin - Computer Systems Officer School of Information Technology Swinburne University of Technology Melbourne, Australia http://www.it.swin.edu.au/staff/jnewbigin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds ` (3 preceding siblings ...) 2004-08-31 23:02 ` Christer Weinigel @ 2004-09-01 12:10 ` Alan Cox 2004-09-01 20:21 ` Hubert Chan 2004-09-01 19:44 ` Markus Törnqvist 5 siblings, 1 reply; 1835+ messages in thread From: Alan Cox @ 2004-09-01 12:10 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Maw, 2004-08-31 at 21:05, Linus Torvalds wrote: > In a graphical environment, the "icon" stream is a good example of this. > It literally has _nothing_ to do with the data in the main stream. The > only linkage is a totally non-technical one, where the user wanted to > associate a secondary stream with the main stream _without_ altering the > main one. THAT is where named streams make sense. The icon doesn't belong in the document, that was a catastrophic disaster in early MacOS (although they only use an index). Users want to manage their icon choices, tags, tooltip notes and attached labels, and you cannot do that in the file if you don't own the file. Also the icon is *not* unrelated to the file in a modern GUI, eg rox and nautilus uses scaled versions of the content for many media types and will show you pictures, frames from a movie etc to help you remember the content. On top they then add user specific annotations. The things that are more independant are: "This file was created by OpenOffice 1.2" "This is a text/plain file in UTF-8" "This file has a UUID of ...." The type has dragons because you get heirarchical typing within documents (consider XML containing namespaces) (UUIDs being one really useful thing we don't tag everywhere onto files that would be a godsend on the desktop providing they moved with the file on rename) Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 12:10 ` Alan Cox @ 2004-09-01 20:21 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-01 20:21 UTC (permalink / raw) To: linux-fsdevel; +Cc: linux-kernel, reiserfs-list >>>>> "Alan" == Alan Cox <alan@lxorguk.ukuu.org.uk> writes: Alan> The icon doesn't belong in the document, that was a catastrophic Alan> disaster in early MacOS (although they only use an index). Users Alan> want to manage their icon choices, tags, tooltip notes and Alan> attached labels, and you cannot do that in the file if you don't Alan> own the file. Most desktop environments provide defaults, but let the user customize their own settings. The substream *could* (I'm not saying if it's a good idea. It's just a possibility.) be used to provide a default icon. Alan> Also the icon is *not* unrelated to the file in a modern GUI, eg Alan> rox and nautilus uses scaled versions of the content for many Alan> media types and will show you pictures, frames from a movie etc to Alan> help you remember the content. For documents, the icon is not completely unrelated, but it isn't something that can be directly extracted. Generating thumbnails usually takes time, so the filemanagers need to cache it somewhere. Right now, they use ~/.thumbnails (at least that's what freedesktop.org recommends), but then you run into consistency issues -- when you delete the file, the thumbnail should be deleted too, and when you rename/move the file, the thumbnail should be attached to the new filename, and not have to be regenerated. Alan> On top they then add user specific annotations. And I think that would be a good use of substreams/attributes/metas too. You don't really want to store it in some separate location, because you run into the same consistency issues as I mentioned before. Sure, if you move a file using Nautilus, it may be smart enough to move the annotations, but I'd bet that mv doesn't know what to do. Alan> The things that are more independant are: Alan> "This file was created by OpenOffice 1.2" "This is a text/plain Alan> file in UTF-8" "This file has a UUID of ...." Alan> The type has dragons because you get heirarchical typing within Alan> documents (consider XML containing namespaces) Well, I think the web guys are going to have to figure something out, and we can just follow what they do ... -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds ` (4 preceding siblings ...) 2004-09-01 12:10 ` Alan Cox @ 2004-09-01 19:44 ` Markus Törnqvist 5 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-09-01 19:44 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, Aug 31, 2004 at 01:05:40PM -0700, Linus Torvalds wrote: > >There's no point to having the kernel export information that is already >inherent in the main stream. As said before, it can be bounced :) You have support for streams, someone could read the inherent information and write it to the stream. At the risk of these two streams being unsynced temporarily, requiring some syncer. >I've seen all these examples of exposing MP3 ID information as a "side >stream", and that's TOTALLY POINTLESS! The information is already there, No it's not pointless :) >it's in a standard format, and exporting it as a stream buys you >absolutely nothing. There is no nice standard way of finding MP3s by the ID information. find music/ -name *mp3/..metas/id3/ | xargs grep -i "dream theater" or something. Which reveals the metadata to find. The kernel doesn't have to parse the ID3 tags, that can be handled in userspace by a daemon, updated based on mtimes or somesuch, yes? >Which means that normally we really don't _want_ named streams. In 99% of >all cases we can use equally good - and _much_ simpler - tool-based >solutions. Sure, when someone goes through every application from ls to nautilus and rams a tool-based patch down the app maintainer's throat and then we're still dependent on the filesystem layout, which is easier to break than the filesystem itself :) >Which means that the only _real_ technical issue for supporting named >streams really ends up being things like samba, which want named streams So they should be implemented for samba and other guys can come up with alternative means for using them ;) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 19:31 ` Horst von Brand @ 2004-09-01 1:14 ` David Masover 2004-09-01 1:14 ` David Masover 1 sibling, 0 replies; 1835+ messages in thread From: David Masover @ 2004-09-01 1:14 UTC (permalink / raw) To: Horst von Brand Cc: Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Horst von Brand wrote: | Pavel Machek <pavel@ucw.cz> said: | |>David Masover <ninja@slaphack.com> said: | | | [...] [...] | You do need extra tools anyway, placing them in the kernel is cheating (and | absolutely pointless, IMHO). Repeat after me: plugins in kernel does NOT equal tools in kernel. It's not hard to, for instance, imagine a generic plugin for archive manipulation which talks to a userspace daemon/library. The kernel doesn't know anything other than (maybe) a list of extensions that are archives. All else is handled in userspace -- the idea that "this is a zipfile" and "zipfiles can be extracted with the 'zip' command" are all in userspace. It's not about the kernel, it's about the interface. And see my other mail: cat foo.zip/README less foo.zip/contents/bar.c is a lot easier than lynx http://google.com/search?q=zip emerge zip man zip unzip foo.zip cat bar.c which already assumes quite a lot of expertise. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUieXgHNmZLgCUhAQLj8Q//VEuRWg9RB/PVg/M1t36SraKIRdbrgGq6 0DYLZwwVWkfz+UKGhUkACNGQzhHG3ySn/lRjqXCsJhzMOzfgmte2CwgGxunHYqdp W69OSZ1vv0ZLpdpzEkVdMfRhhg8RBL54Ewj6fQu7G1ZzJ0J0HUFCEZIsVoL4zJfk +GFl4M+vvc+152HipEDsvrrPasc0dSxmryyzPbyABrfPRi3l2E3Okfewn/4MCC1E H/L04nmvJQ2OSZYu1eSMSpnLrgQdZJbYefRuSinpqfKhjpwQXjj2rnfcPm+xc3hh FDMlAUPA/DAlthTuhxd9+8iDPEARY57/FAMFTalYVhBEv0NWieMMn9V6R8BbnhZW 8y9EWu0Nvoowg9qElkVjWEt5M1Rbeuq4TDviyVyPYg/KcTKDGpjLuyQQbWv6hMVM WAx6THtRK8Ji/XLCj4Kh6qDOEYP1nuwqLYSWv13BtNAJ3bVeKryOJ4Er4AmuJomA uydBPQVsrtzTNmk7Eul2ZjrxXdESpUnUIAbu7H1Gbn51uP4rQTdlXkdPTv6s/wvG OSOMWKs28GBFsG3+IJLnoN2wxrffpgTBEAaPwbh9TRFKLIH+pkCFSx2cIkTj93nr RjETFObjYdJGhGygrq0xOmN5e7iprdxhxd3/+fvjQxnZGxXT3fZLUwhVX9/zhaXG cxau2Onr3jc= =MrR9 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-01 1:14 ` David Masover 0 siblings, 0 replies; 1835+ messages in thread From: David Masover @ 2004-09-01 1:14 UTC (permalink / raw) To: Horst von Brand Cc: Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Horst von Brand wrote: | Pavel Machek <pavel@ucw.cz> said: | |>David Masover <ninja@slaphack.com> said: | | | [...] [...] | You do need extra tools anyway, placing them in the kernel is cheating (and | absolutely pointless, IMHO). Repeat after me: plugins in kernel does NOT equal tools in kernel. It's not hard to, for instance, imagine a generic plugin for archive manipulation which talks to a userspace daemon/library. The kernel doesn't know anything other than (maybe) a list of extensions that are archives. All else is handled in userspace -- the idea that "this is a zipfile" and "zipfiles can be extracted with the 'zip' command" are all in userspace. It's not about the kernel, it's about the interface. And see my other mail: cat foo.zip/README less foo.zip/contents/bar.c is a lot easier than lynx http://google.com/search?q=zip emerge zip man zip unzip foo.zip cat bar.c which already assumes quite a lot of expertise. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUieXgHNmZLgCUhAQLj8Q//VEuRWg9RB/PVg/M1t36SraKIRdbrgGq6 0DYLZwwVWkfz+UKGhUkACNGQzhHG3ySn/lRjqXCsJhzMOzfgmte2CwgGxunHYqdp W69OSZ1vv0ZLpdpzEkVdMfRhhg8RBL54Ewj6fQu7G1ZzJ0J0HUFCEZIsVoL4zJfk +GFl4M+vvc+152HipEDsvrrPasc0dSxmryyzPbyABrfPRi3l2E3Okfewn/4MCC1E H/L04nmvJQ2OSZYu1eSMSpnLrgQdZJbYefRuSinpqfKhjpwQXjj2rnfcPm+xc3hh FDMlAUPA/DAlthTuhxd9+8iDPEARY57/FAMFTalYVhBEv0NWieMMn9V6R8BbnhZW 8y9EWu0Nvoowg9qElkVjWEt5M1Rbeuq4TDviyVyPYg/KcTKDGpjLuyQQbWv6hMVM WAx6THtRK8Ji/XLCj4Kh6qDOEYP1nuwqLYSWv13BtNAJ3bVeKryOJ4Er4AmuJomA uydBPQVsrtzTNmk7Eul2ZjrxXdESpUnUIAbu7H1Gbn51uP4rQTdlXkdPTv6s/wvG OSOMWKs28GBFsG3+IJLnoN2wxrffpgTBEAaPwbh9TRFKLIH+pkCFSx2cIkTj93nr RjETFObjYdJGhGygrq0xOmN5e7iprdxhxd3/+fvjQxnZGxXT3fZLUwhVX9/zhaXG cxau2Onr3jc= =MrR9 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 1:14 ` David Masover (?) @ 2004-09-01 4:59 ` Pavel Machek 2004-09-01 5:23 ` Hubert Chan 2004-09-01 16:14 ` Jamie Lokier -1 siblings, 2 replies; 1835+ messages in thread From: Pavel Machek @ 2004-09-01 4:59 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > | You do need extra tools anyway, placing them in the kernel is cheating > (and > | absolutely pointless, IMHO). > > Repeat after me: plugins in kernel does NOT equal tools in kernel. > > It's not hard to, for instance, imagine a generic plugin for archive > manipulation which talks to a userspace daemon/library. The kernel > doesn't know anything other than (maybe) a list of extensions that are > archives. All else is handled in userspace -- the idea that "this is a > zipfile" and "zipfiles can be extracted with the 'zip' command" are all > in userspace. > > It's not about the kernel, it's about the interface. And see my other mail: > cat foo.zip/README > less foo.zip/contents/bar.c > is a lot easier than > lynx http://google.com/search?q=zip > emerge zip > man zip > unzip foo.zip > cat bar.c > which already assumes quite a lot of expertise. If you only want nice user interface, you can have that today. Its done using coda, and hosted at uservfs.sf.net. Pavel -- When do you have heart between your knees? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 4:59 ` Pavel Machek @ 2004-09-01 5:23 ` Hubert Chan 2004-09-01 16:14 ` Jamie Lokier 1 sibling, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-01 5:23 UTC (permalink / raw) To: linux-fsdevel; +Cc: linux-kernel, reiserfs-list >>>>> "Pavel" == Pavel Machek <pavel@ucw.cz> writes: Pavel> If you only want nice user interface, you can have that Pavel> today. Its done using coda, and hosted at uservfs.sf.net. Hmmm. Last activity seems to be about three years ago. Any idea if it works with Linux 2.6? Is there any chance at all that it will be merged into the kernel? Can anyone comment on the speed? I don't have the ability right now to try out any new kernel stuff. The user interface isn't that nice either. There's no browsability for the methods. (Hmm. I have this strange .deb file. How do I look at the contents? #utar? #ugz? ...) And honestly, I think that "#" hack is pretty ugly. There's also possible name conflicts -- although it's highly unlikely -- if I have a file that just happens to be called foo.tar.gz#utar, I can't browse foo.tar.gz. There may be some nice ideas in there, but from a quick lookover, it seems to be lacking in execution. I'll probably give it a try anyways, during my next round of kernel compiling/testing. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 4:59 ` Pavel Machek 2004-09-01 5:23 ` Hubert Chan @ 2004-09-01 16:14 ` Jamie Lokier 2004-09-01 20:18 ` Pavel Machek 1 sibling, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 16:14 UTC (permalink / raw) To: Pavel Machek Cc: David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Pavel Machek wrote: > > It's not about the kernel, it's about the interface. And see my other mail: > > cat foo.zip/README > > less foo.zip/contents/bar.c > > is a lot easier than > > lynx http://google.com/search?q=zip > > emerge zip > > man zip > > unzip foo.zip > > cat bar.c > > which already assumes quite a lot of expertise. > > If you only want nice user interface, you can have that today. Its > done using coda, and hosted at uservfs.sf.net. Can I do these already using uservfs? cd less foo.zip/contents/bar.c less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c grep -R obscure_label ~/RPMS No I can't. Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can only do the above with convenient paths, completion etc. if I mount my home directory and distfiles over uservfs, so that all I/O in those directories including ordinary accesses is through uservfs, and have the _real_ storage elsewhere. That's quite illogical, cumbersome to set up, and slows down normal I/O significantly. I can do some of those things in a similar way if I do them inside Midnight Commander, Gnome, KDE, or Emacs -- but even then a path that works in one of those doesn't work in another, and the extensions don't even work in all programs, plugins or scripts for those systems. As we're adding file-as-directories to VFS, I propose it would be good to have *hooks* in place, and non-metadata namespace reserved inside the directories, so that uservfs-like userspace filesystems can be auto-mounted at those places and synchronised with the file. In other words, something that makes the next version of uservfs a whole lot more useful. The good news is that a lot of the hooks are falling into place already. Kernel support is needed to handle the interface, cacheing to trigger deletions, page cacheing so that I/O is a normal speed and mmap() works, moveable mounts so that renames work and the cached data followed them, and coherency so that if the underlying archive is changed, it invalidates cached data with the appropriate degree of synchronisation. Note that the current dnotify cannot provide the latter with any strength; it's too weak an interface (so is inotify). (You can omit a lot of these things, but then you get a flaky interface). The FUSE project looks like it will provide an efficient interface to the page cache, avoiding round trips where possible and supporting mmap. If it doesn't, it should. (Similar VFS facilities are useful for distributed filesystems). The presently discussed file-as-directory semantics will provide a nice view, and force the necessary minimal tool changes, so _all_ tools will be fine with the new uservfs. So all I am asking for is a facility to auto-mount with file-as-directory, and the ability for a userspace daemon to be notified of regular file modifications synchronously. Both can be added later, once file-as-directory and moveable mounts are established. (fcntl leases and dnotify _almost_ provide this, but they don't. Looks like incoherent hacks keep getting added all over the place for Samba... :). That's all. Kernel space needs to provide very little that it doesn't already provide, for this rather nice and universally usable feature to work. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 16:14 ` Jamie Lokier @ 2004-09-01 20:18 ` Pavel Machek 2004-09-01 21:59 ` Jamie Lokier 0 siblings, 1 reply; 1835+ messages in thread From: Pavel Machek @ 2004-09-01 20:18 UTC (permalink / raw) To: Jamie Lokier Cc: David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > It's not about the kernel, it's about the interface. And see my other mail: > > > cat foo.zip/README > > > less foo.zip/contents/bar.c > > > is a lot easier than > > > lynx http://google.com/search?q=zip > > > emerge zip > > > man zip > > > unzip foo.zip > > > cat bar.c > > > which already assumes quite a lot of expertise. > > > > If you only want nice user interface, you can have that today. Its > > done using coda, and hosted at uservfs.sf.net. > > Can I do these already using uservfs? > > cd > less foo.zip/contents/bar.c less foo.zip#uzip/contents/bar.c > less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c > grep -R obscure_label ~/RPMS I do not think you'd want this. How would you tell grep obscure label without entering archives, then? > No I can't. > Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can > only do the above with convenient paths, completion etc. if I mount > my Well, you can... or at least I had version somewhere that automagically prepended /overlay when file was not found. That way, normal requests were fast and uservfs were used whenever it was needed... At the price of little uglyness and slowing down -ENOENTs. > As we're adding file-as-directories to VFS, I propose it would be good > to have *hooks* in place, and non-metadata namespace reserved inside > the directories, so that uservfs-like userspace filesystems can > be auto-mounted at those places and synchronised with the file. > > In other words, something that makes the next version of uservfs a > whole lot more useful. Yep, something less hacky would be nice. > So all I am asking for is a facility to auto-mount with > file-as-directory, and the ability for a userspace daemon to be > notified of regular file modifications synchronously. Both can be > added later, once file-as-directory and moveable mounts are If the userspace daemon is synchronously notified of file modification... will not that lead to very ugly deadlocks? Pavel -- Horseback riding is like software... ...vgf orggre jura vgf serr. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:18 ` Pavel Machek @ 2004-09-01 21:59 ` Jamie Lokier 2004-09-01 22:51 ` Lee Revell 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 21:59 UTC (permalink / raw) To: Pavel Machek Cc: David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Pavel Machek wrote: > > Can I do these already using uservfs? > > > > cd > > less foo.zip/contents/bar.c > > less foo.zip#uzip/contents/bar.c > > > less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c > > less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c > > > grep -R obscure_label ~/RPMS > > I do not think you'd want this. How would you tell grep obscure label > without entering archives, then? Granted, this is where we need "grep --recurse-into-files" or, more generally useful, a find option. > > No I can't. > > > Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can > > only do the above with convenient paths, completion etc. if I mount > > my > > Well, you can... or at least I had version somewhere that > automagically prepended /overlay when file was not found. That way, > normal requests were fast and uservfs were used whenever it was > needed... At the price of little uglyness and slowing down -ENOENTs. There you go... a kernel hook to prepend /overlay works :) These hook I keep mentioning, they are really very simple :) > > So all I am asking for is a facility to auto-mount with > > file-as-directory, and the ability for a userspace daemon to be > > notified of regular file modifications synchronously. Both can be > > added later, once file-as-directory and moveable mounts are > > If the userspace daemon is synchronously notified of file > modification... will not that lead to very ugly deadlocks? Not if it's written properly. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 21:59 ` Jamie Lokier @ 2004-09-01 22:51 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-01 22:51 UTC (permalink / raw) To: Jamie Lokier Cc: Pavel Machek, David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 2004-09-01 at 17:59, Jamie Lokier wrote: > Pavel Machek wrote: > > > Can I do these already using uservfs? > > > > > > cd > > > less foo.zip/contents/bar.c > > > > less foo.zip#uzip/contents/bar.c > > > > > less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c > > > > less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c > > > > > grep -R obscure_label ~/RPMS > > > > I do not think you'd want this. How would you tell grep obscure label > > without entering archives, then? > > Granted, this is where we need "grep --recurse-into-files" or, more > generally useful, a find option. > FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature. Of course, no idea how it's implemented, but users like it. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-01 22:51 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-01 22:51 UTC (permalink / raw) To: Jamie Lokier Cc: Pavel Machek, David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 2004-09-01 at 17:59, Jamie Lokier wrote: > Pavel Machek wrote: > > > Can I do these already using uservfs? > > > > > > cd > > > less foo.zip/contents/bar.c > > > > less foo.zip#uzip/contents/bar.c > > > > > less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c > > > > less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c > > > > > grep -R obscure_label ~/RPMS > > > > I do not think you'd want this. How would you tell grep obscure label > > without entering archives, then? > > Granted, this is where we need "grep --recurse-into-files" or, more > generally useful, a find option. > FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature. Of course, no idea how it's implemented, but users like it. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 22:51 ` Lee Revell (?) @ 2004-09-02 14:25 ` Horst von Brand 2004-09-02 17:47 ` Lee Revell 2004-09-02 19:41 ` Spam -1 siblings, 2 replies; 1835+ messages in thread From: Horst von Brand @ 2004-09-02 14:25 UTC (permalink / raw) To: Lee Revell Cc: Jamie Lokier, Pavel Machek, David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Lee Revell <rlrevell@joe-job.com> said: [...] > FWIW, this is how Windows does it now. As of XP, 'Find files' has an > option, enabled by default, to look inside archives. If you tell it to > look for a driver in a given directory it will also look inside .cab > and .zip files. It's extremely useful, I would imagine someone who uses > XP a lot will come to expect this feature. It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 14:25 ` Horst von Brand @ 2004-09-02 17:47 ` Lee Revell 2004-09-02 19:41 ` Spam 1 sibling, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 17:47 UTC (permalink / raw) To: Horst von Brand Cc: Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 10:25, Horst von Brand wrote: > Lee Revell <rlrevell@joe-job.com> said: > > [...] > > > FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > option, enabled by default, to look inside archives. If you tell it to > > look for a driver in a given directory it will also look inside .cab > > and .zip files. It's extremely useful, I would imagine someone who uses > > XP a lot will come to expect this feature. > > It is trivial to implement this by looking inside the files. I.e., the way > mc has done this for ages. This requires a separate, MC-specific namespace. The point is to unify the namespace, not fragment it. If Hans had a comprehensible web page, maybe more people would understand this aspect of his argument. Would you put digressions about chaos theory and King Arthur in a man page? Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 17:47 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 17:47 UTC (permalink / raw) To: Horst von Brand Cc: Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 10:25, Horst von Brand wrote: > Lee Revell <rlrevell@joe-job.com> said: > > [...] > > > FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > option, enabled by default, to look inside archives. If you tell it to > > look for a driver in a given directory it will also look inside .cab > > and .zip files. It's extremely useful, I would imagine someone who uses > > XP a lot will come to expect this feature. > > It is trivial to implement this by looking inside the files. I.e., the way > mc has done this for ages. This requires a separate, MC-specific namespace. The point is to unify the namespace, not fragment it. If Hans had a comprehensible web page, maybe more people would understand this aspect of his argument. Would you put digressions about chaos theory and King Arthur in a man page? Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 14:25 ` Horst von Brand @ 2004-09-02 19:41 ` Spam 2004-09-02 19:41 ` Spam 1 sibling, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 19:41 UTC (permalink / raw) To: Horst von Brand Cc: Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Lee Revell <rlrevell@joe-job.com> said: > [...] >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an >> option, enabled by default, to look inside archives. If you tell it to >> look for a driver in a given directory it will also look inside .cab >> and .zip files. It's extremely useful, I would imagine someone who uses >> XP a lot will come to expect this feature. > It is trivial to implement this by looking inside the files. I.e., the way > mc has done this for ages. Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 19:41 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 19:41 UTC (permalink / raw) To: Horst von Brand Cc: Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Lee Revell <rlrevell@joe-job.com> said: > [...] >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an >> option, enabled by default, to look inside archives. If you tell it to >> look for a driver in a given directory it will also look inside .cab >> and .zip files. It's extremely useful, I would imagine someone who uses >> XP a lot will come to expect this feature. > It is trivial to implement this by looking inside the files. I.e., the way > mc has done this for ages. Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 19:41 ` Spam (?) @ 2004-09-02 18:55 ` Alan Cox 2004-09-02 20:07 ` Spam 2004-09-02 20:42 ` Pavel Machek -1 siblings, 2 replies; 1835+ messages in thread From: Alan Cox @ 2004-09-02 18:55 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 20:41, Spam wrote: > > It is trivial to implement this by looking inside the files. I.e., the way > > mc has done this for ages. > > Difference is that you can't do "locate" or "find" or "Search".. You > would have to open the files in an archive-supporting application > such as mc. And would you rather that logic was running swappable in shared library space or privileged and unswappable in kernel ? Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 18:55 ` Alan Cox @ 2004-09-02 20:07 ` Spam 2004-09-02 20:42 ` Pavel Machek 1 sibling, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 20:07 UTC (permalink / raw) To: Alan Cox Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Iau, 2004-09-02 at 20:41, Spam wrote: >> > It is trivial to implement this by looking inside the files. I.e., the way >> > mc has done this for ages. >> >> Difference is that you can't do "locate" or "find" or "Search".. You >> would have to open the files in an archive-supporting application >> such as mc. > And would you rather that logic was running swappable in shared library > space or privileged and unswappable in kernel ? I would rather have it as a filesystem/vfs plugin that would allow all my programs to use the features the plugin gives, even if it would mean being totally in kernel space. > Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 20:07 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 20:07 UTC (permalink / raw) To: Alan Cox Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Iau, 2004-09-02 at 20:41, Spam wrote: >> > It is trivial to implement this by looking inside the files. I.e., the way >> > mc has done this for ages. >> >> Difference is that you can't do "locate" or "find" or "Search".. You >> would have to open the files in an archive-supporting application >> such as mc. > And would you rather that logic was running swappable in shared library > space or privileged and unswappable in kernel ? I would rather have it as a filesystem/vfs plugin that would allow all my programs to use the features the plugin gives, even if it would mean being totally in kernel space. > Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:07 ` Spam (?) @ 2004-09-02 19:17 ` Alan Cox 2004-09-02 20:28 ` Spam -1 siblings, 1 reply; 1835+ messages in thread From: Alan Cox @ 2004-09-02 19:17 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 21:07, Spam wrote: > > And would you rather that logic was running swappable in shared library > > space or privileged and unswappable in kernel ? > > I would rather have it as a filesystem/vfs plugin that would allow > all my programs to use the features the plugin gives, even if it > would mean being totally in kernel space. Thats not what I asked. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 19:17 ` Alan Cox @ 2004-09-02 20:28 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 20:28 UTC (permalink / raw) To: Alan Cox Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Iau, 2004-09-02 at 21:07, Spam wrote: >> > And would you rather that logic was running swappable in shared library >> > space or privileged and unswappable in kernel ? >> >> I would rather have it as a filesystem/vfs plugin that would allow >> all my programs to use the features the plugin gives, even if it >> would mean being totally in kernel space. > Thats not what I asked. I failed to understand your question then. :( ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 20:28 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 20:28 UTC (permalink / raw) To: Alan Cox Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Iau, 2004-09-02 at 21:07, Spam wrote: >> > And would you rather that logic was running swappable in shared library >> > space or privileged and unswappable in kernel ? >> >> I would rather have it as a filesystem/vfs plugin that would allow >> all my programs to use the features the plugin gives, even if it >> would mean being totally in kernel space. > Thats not what I asked. I failed to understand your question then. :( ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 18:55 ` Alan Cox 2004-09-02 20:07 ` Spam @ 2004-09-02 20:42 ` Pavel Machek 1 sibling, 0 replies; 1835+ messages in thread From: Pavel Machek @ 2004-09-02 20:42 UTC (permalink / raw) To: Alan Cox Cc: Spam, Horst von Brand, Lee Revell, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > It is trivial to implement this by looking inside the files. I.e., the way > > > mc has done this for ages. > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > would have to open the files in an archive-supporting application > > such as mc. > > And would you rather that logic was running swappable in shared library > space or privileged and unswappable in kernel ? Well, having it in kernel mean that cache gets actually shared between different processes -- and that's usefull. With coda and similar tools you can put most of the logic into userspace (kernel impact is 20 lines). You *will* want it to run priviledged, because otherwise caches are useless. Pavel ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 19:41 ` Spam (?) (?) @ 2004-09-02 19:49 ` Pavel Machek 2004-09-02 20:01 ` Lee Revell -1 siblings, 1 reply; 1835+ messages in thread From: Pavel Machek @ 2004-09-02 19:49 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Lee Revell, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > >> option, enabled by default, to look inside archives. If you tell it to > >> look for a driver in a given directory it will also look inside .cab > >> and .zip files. It's extremely useful, I would imagine someone who uses > >> XP a lot will come to expect this feature. > > > It is trivial to implement this by looking inside the files. I.e., the way > > mc has done this for ages. > > Difference is that you can't do "locate" or "find" or "Search".. You > would have to open the files in an archive-supporting application > such as mc. You really need archive support in find. At the very least you need option "enter archives" vs. "do not enter archives". Entering archives automagically is seriously wrong. Pavel ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 19:49 ` Pavel Machek @ 2004-09-02 20:01 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 20:01 UTC (permalink / raw) To: Pavel Machek Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 15:49, Pavel Machek wrote: > Hi! > > > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > >> option, enabled by default, to look inside archives. If you tell it to > > >> look for a driver in a given directory it will also look inside .cab > > >> and .zip files. It's extremely useful, I would imagine someone who uses > > >> XP a lot will come to expect this feature. > > > > > It is trivial to implement this by looking inside the files. I.e., the way > > > mc has done this for ages. > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > would have to open the files in an archive-supporting application > > such as mc. > > You really need archive support in find. At the very least you need > option "enter archives" vs. "do not enter archives". Entering archives > automagically is seriously wrong. But is it efficient to make every application that reads files have to know how to get inside a tar file, just to read its contents? That seems like a massive duplication of effort. Better to have the contents accessible via a separate stream, in the same namespace. Fix it once in the kernel vs. fix it in umpteen apps. The key point here is that the expressive power of the system is greatly reduced by having a fragmented namespace. Of course there are any number of ways for an app to find out what is in a tar file, but exporting all of that information in a unified namespace is nontrivial and much more interesting. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 20:01 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 20:01 UTC (permalink / raw) To: Pavel Machek Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 15:49, Pavel Machek wrote: > Hi! > > > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > >> option, enabled by default, to look inside archives. If you tell it to > > >> look for a driver in a given directory it will also look inside .cab > > >> and .zip files. It's extremely useful, I would imagine someone who uses > > >> XP a lot will come to expect this feature. > > > > > It is trivial to implement this by looking inside the files. I.e., the way > > > mc has done this for ages. > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > would have to open the files in an archive-supporting application > > such as mc. > > You really need archive support in find. At the very least you need > option "enter archives" vs. "do not enter archives". Entering archives > automagically is seriously wrong. But is it efficient to make every application that reads files have to know how to get inside a tar file, just to read its contents? That seems like a massive duplication of effort. Better to have the contents accessible via a separate stream, in the same namespace. Fix it once in the kernel vs. fix it in umpteen apps. The key point here is that the expressive power of the system is greatly reduced by having a fragmented namespace. Of course there are any number of ways for an app to find out what is in a tar file, but exporting all of that information in a unified namespace is nontrivial and much more interesting. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:01 ` Lee Revell (?) @ 2004-09-02 19:16 ` Alan Cox -1 siblings, 0 replies; 1835+ messages in thread From: Alan Cox @ 2004-09-02 19:16 UTC (permalink / raw) To: Lee Revell Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 21:01, Lee Revell wrote: > But is it efficient to make every application that reads files have to > know how to get inside a tar file, just to read its contents? That > seems like a massive duplication of effort. Better to have the contents > accessible via a separate stream, in the same namespace. Fix it once in > the kernel vs. fix it in umpteen apps. Thats how you get yourself a non useful OS. Fix it in a library and share it between the apps that care. Like say.. gnome-vfs2 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:01 ` Lee Revell (?) (?) @ 2004-09-02 20:04 ` Chris Wedgwood 2004-09-02 20:10 ` Spam -1 siblings, 1 reply; 1835+ messages in thread From: Chris Wedgwood @ 2004-09-02 20:04 UTC (permalink / raw) To: Lee Revell Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote: > Better to have the contents accessible via a separate stream, in the > same namespace. Fix it once in the kernel vs. fix it in umpteen > apps. This is ridiculous. We have shared libraries, 99% of applications manage to use libc for example --- this isn't that different. --cw ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:04 ` Chris Wedgwood @ 2004-09-02 20:10 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 20:10 UTC (permalink / raw) To: Chris Wedgwood Cc: Lee Revell, Pavel Machek, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote: >> Better to have the contents accessible via a separate stream, in the >> same namespace. Fix it once in the kernel vs. fix it in umpteen >> apps. > This is ridiculous. We have shared libraries, 99% of applications > manage to use libc for example --- this isn't that different. Yes if that was the case then programs wouldn't brake if libc and its API was fixed. Sadly, I think it isn't. ~S > --cw ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 20:10 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-09-02 20:10 UTC (permalink / raw) To: Chris Wedgwood Cc: Lee Revell, Pavel Machek, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote: >> Better to have the contents accessible via a separate stream, in the >> same namespace. Fix it once in the kernel vs. fix it in umpteen >> apps. > This is ridiculous. We have shared libraries, 99% of applications > manage to use libc for example --- this isn't that different. Yes if that was the case then programs wouldn't brake if libc and its API was fixed. Sadly, I think it isn't. ~S > --cw ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:01 ` Lee Revell ` (2 preceding siblings ...) (?) @ 2004-09-02 20:43 ` Pavel Machek 2004-09-02 20:47 ` Lee Revell -1 siblings, 1 reply; 1835+ messages in thread From: Pavel Machek @ 2004-09-02 20:43 UTC (permalink / raw) To: Lee Revell Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > > >> option, enabled by default, to look inside archives. If you tell it to > > > >> look for a driver in a given directory it will also look inside .cab > > > >> and .zip files. It's extremely useful, I would imagine someone who uses > > > >> XP a lot will come to expect this feature. > > > > > > > It is trivial to implement this by looking inside the files. I.e., the way > > > > mc has done this for ages. > > > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > > would have to open the files in an archive-supporting application > > > such as mc. > > > > You really need archive support in find. At the very least you need > > option "enter archives" vs. "do not enter archives". Entering archives > > automagically is seriously wrong. > > But is it efficient to make every application that reads files have to > know how to get inside a tar file, just to read its contents? That Application does not have to know how to handle tar/zip/etc, but it has to make distinction between "enter archives" and "do not enter archives". See uservfs.sf.net. Pavel ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:43 ` Pavel Machek @ 2004-09-02 20:47 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 20:47 UTC (permalink / raw) To: Pavel Machek Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 16:43, Pavel Machek wrote: > Hi! > > > > > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > > > >> option, enabled by default, to look inside archives. If you tell it to > > > > >> look for a driver in a given directory it will also look inside .cab > > > > >> and .zip files. It's extremely useful, I would imagine someone who uses > > > > >> XP a lot will come to expect this feature. > > > > > > > > > It is trivial to implement this by looking inside the files. I.e., the way > > > > > mc has done this for ages. > > > > > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > > > would have to open the files in an archive-supporting application > > > > such as mc. > > > > > > You really need archive support in find. At the very least you need > > > option "enter archives" vs. "do not enter archives". Entering archives > > > automagically is seriously wrong. > > > > But is it efficient to make every application that reads files have to > > know how to get inside a tar file, just to read its contents? That > > Application does not have to know how to handle tar/zip/etc, but it > has to make distinction between "enter archives" and "do not enter > archives". See uservfs.sf.net. But how do you cache the information you had to look in the archive for in a way that other apps can use it? How do you synchronize access to the cache and maintain cache coherency in userspace? Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 20:47 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 20:47 UTC (permalink / raw) To: Pavel Machek Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 16:43, Pavel Machek wrote: > Hi! > > > > > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > > > >> option, enabled by default, to look inside archives. If you tell it to > > > > >> look for a driver in a given directory it will also look inside .cab > > > > >> and .zip files. It's extremely useful, I would imagine someone who uses > > > > >> XP a lot will come to expect this feature. > > > > > > > > > It is trivial to implement this by looking inside the files. I.e., the way > > > > > mc has done this for ages. > > > > > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > > > would have to open the files in an archive-supporting application > > > > such as mc. > > > > > > You really need archive support in find. At the very least you need > > > option "enter archives" vs. "do not enter archives". Entering archives > > > automagically is seriously wrong. > > > > But is it efficient to make every application that reads files have to > > know how to get inside a tar file, just to read its contents? That > > Application does not have to know how to handle tar/zip/etc, but it > has to make distinction between "enter archives" and "do not enter > archives". See uservfs.sf.net. But how do you cache the information you had to look in the archive for in a way that other apps can use it? How do you synchronize access to the cache and maintain cache coherency in userspace? Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:47 ` Lee Revell (?) @ 2004-09-02 20:49 ` Chris Wedgwood 2004-09-02 20:57 ` Lee Revell -1 siblings, 1 reply; 1835+ messages in thread From: Chris Wedgwood @ 2004-09-02 20:49 UTC (permalink / raw) To: Lee Revell Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote: > But how do you cache the information you had to look in the archive > for in a way that other apps can use it? ~/.object-cache/ or whatever > How do you synchronize access to the cache and maintain cache > coherency in userspace? coherency doesn't exist for normal files for the most part anyhow ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:49 ` Chris Wedgwood @ 2004-09-02 20:57 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 20:57 UTC (permalink / raw) To: Chris Wedgwood Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 16:49, Chris Wedgwood wrote: > On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote: > > > But how do you cache the information you had to look in the archive > > for in a way that other apps can use it? > > ~/.object-cache/ or whatever > How are permissions handled? If root lists the contents of a tar file that is world readable, then joeuser comes along and does the same, can joeuser sees the cached listing? Would it depend on root's umask? Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 20:57 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-02 20:57 UTC (permalink / raw) To: Chris Wedgwood Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 16:49, Chris Wedgwood wrote: > On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote: > > > But how do you cache the information you had to look in the archive > > for in a way that other apps can use it? > > ~/.object-cache/ or whatever > How are permissions handled? If root lists the contents of a tar file that is world readable, then joeuser comes along and does the same, can joeuser sees the cached listing? Would it depend on root's umask? Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:57 ` Lee Revell (?) @ 2004-09-02 20:58 ` Chris Wedgwood -1 siblings, 0 replies; 1835+ messages in thread From: Chris Wedgwood @ 2004-09-02 20:58 UTC (permalink / raw) To: Lee Revell Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 04:57:06PM -0400, Lee Revell wrote: > How are permissions handled? If root lists the contents of a tar > file that is world readable, then joeuser comes along and does the > same, can joeuser sees the cached listing? everyone has their own cache i guess, works well enough elsewhere also, am i the only person scared by the code complexity when it comes to root and setuid code here? sounds like a disaster waiting to happen ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:47 ` Lee Revell (?) (?) @ 2004-09-02 20:58 ` Pavel Machek -1 siblings, 0 replies; 1835+ messages in thread From: Pavel Machek @ 2004-09-02 20:58 UTC (permalink / raw) To: Lee Revell Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > Application does not have to know how to handle tar/zip/etc, but it > > has to make distinction between "enter archives" and "do not enter > > archives". See uservfs.sf.net. > > But how do you cache the information you had to look in the archive for > in a way that other apps can use it? How do you synchronize access to > the cache and maintain cache coherency in userspace? Uservfs.sf.net. Unlike alan, I do not think that "do it all in library" is good idea. I put it in the userspace as "codafs" server, and let applications see it as a regular filesystem. Pavel ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 1:14 ` David Masover (?) (?) @ 2004-09-01 5:50 ` Hans Reiser 2004-09-01 6:06 ` Linus Torvalds 2004-09-01 11:49 ` Alan Cox -1 siblings, 2 replies; 1835+ messages in thread From: Hans Reiser @ 2004-09-01 5:50 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus, you are looking at this like a lieutenant instead of an HQ staffer, which is unusual of you. You are saying, 1-2% simpler and better, no biggie, why work so hard to get it? And we are saying, 1-2% simpler and better, times thousands of applications, wow! That's a lot! Yes, changing cat to use openat() is no big deal. 1-2% additional coding cost for cat, who cares? But if you add 1-2% coding cost to every application which might access an attribute/stream/whatever, well, that totals more than the effort of authoring emacs. Can you see that? Namespace simplifications and empowerments are force multipliers. They don't add to Linux like writing a new app adds to Linux, they add to Linux like adding percentage improvements to every app in Linux adds to Linux. HQ staffer types know that if you collect enough itty bitty little force multipliers, you win the war. Whether the troops have to spend five minutes a day polishing their shoes because their type of shoe needs polishing, that matters more than losing a tank battalion, when you are a major power. Linux is a major power. This is a software engineering issue. We are discussing improvements that because they are diffused throughout the OS in their impact, seem like no big deal. But they are a big deal. One of the major determinants of an organization's efficiency is whether the management can recognize widely diffused inefficiencies as well as it can recognize focused inefficiencies of the same magnitude. David is so very right about the usability issues. Usability is all about trivia. Usability battles feature no dragons, they feature armies of spiders. If you add one step to what a user needs to do to get to the data with an app, that matters a lot. That's doubling the spider army. Making a 2 step process to access data into a one step process to access the data halves the time cost of accessing the data. Lifestyle efficiency is mostly about reducing the cost of trivia, because trivia is most of what we do. Most people have as much interest in reading the man page for tar as you have in learning how to turn the hand crank to start your car. Just the look of "tar -xf" turns them away. Crypto-Geek gobbledy-gook is what it is. Let's value their time, there are a lot of them. Hans David Masover wrote: > > > It's not about the kernel, it's about the interface. And see my other > mail: > cat foo.zip/README > less foo.zip/contents/bar.c > is a lot easier than > lynx http://google.com/search?q=zip > emerge zip > man zip > unzip foo.zip > cat bar.c > which already assumes quite a lot of expertise. > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 5:50 ` Hans Reiser @ 2004-09-01 6:06 ` Linus Torvalds 2004-09-01 6:32 ` Hubert Chan ` (2 more replies) 2004-09-01 11:49 ` Alan Cox 1 sibling, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-09-01 6:06 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 31 Aug 2004, Hans Reiser wrote: > > You are saying, 1-2% simpler and better, no biggie, why work so hard to > get it? > > And we are saying, 1-2% simpler and better, times thousands of > applications, wow! That's a lot! But would thousands care? Seriously? For example, you could make just _one_ program support "openat()", and you'd get most of the advantages, with no possibility of _breaking_ any of thousands of applications.. I know, you've ignored the "runat" program (which is just a wrapper around the openat() system call), but it _has_ been mentioned several times in this thread. Yes, you'd type a bit more to do runat file ls -l instead of ls -l file/ but at least the openat/runat approach also works for directories, which does actually make it a lot more _generic_ than the "show in the regular filespace" approach. No special cases. So your comparison isn't valid, because you're ignoring the people who shout "runat" at you. You've also not ever actually answered about the problems about directories with attributes. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 6:06 ` Linus Torvalds @ 2004-09-01 6:32 ` Hubert Chan 2004-09-01 7:22 ` Hans Reiser 2004-09-01 23:04 ` Hubert Chan 2 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-01 6:32 UTC (permalink / raw) To: linux-fsdevel; +Cc: linux-kernel, reiserfs-list >>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes: [...] Linus> I know, you've ignored the "runat" program (which is just a Linus> wrapper around the openat() system call), but it _has_ been Linus> mentioned several times in this thread. Yes, you'd type a bit Linus> more to do Linus> runat file ls -l Linus> instead of Linus> ls -l file/ And how do I get GIMP (or insert your favourite image editing program here) to edit xterm/icon.png without having to open a new instance of GIMP? runat isn't going to help here. You'll have to modify every editing program if you want that type of functionality. Linus> but at least the openat/runat approach also works for Linus> directories, which does actually make it a lot more _generic_ Linus> than the "show in the regular filespace" approach. No special Linus> cases. Linus> So your comparison isn't valid, because you're ignoring the Linus> people who shout "runat" at you. You've also not ever actually Linus> answered about the problems about directories with attributes. All attributes should be stuffed in a specially-named subdirectory, like Reiser4 puts all metadata in a metas subdirectory. That's one solution. The only trick here is picking a suitable name for the subdirectory that is both user-friendly and highly unlikely to generate clashes. Both of these I've mentioned before in this thread, and nobody has brought up any objections yet. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 6:06 ` Linus Torvalds @ 2004-09-01 7:22 ` Hans Reiser 2004-09-01 7:22 ` Hans Reiser 2004-09-01 23:04 ` Hubert Chan 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-09-01 7:22 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Alexander Zarochentcev Linus Torvalds wrote: >On Tue, 31 Aug 2004, Hans Reiser wrote: > > >>You are saying, 1-2% simpler and better, no biggie, why work so hard to >>get it? >> >>And we are saying, 1-2% simpler and better, times thousands of >>applications, wow! That's a lot! >> >> > >But would thousands care? Seriously? > > >For example, you could make just _one_ program support "openat()", and >you'd get most of the advantages, with no possibility of _breaking_ any of >thousands of applications.. > >I know, you've ignored the "runat" program (which is just a wrapper around >the openat() system call), but it _has_ been mentioned several times in >this thread. Yes, you'd type a bit more to do > > runat file ls -l > >instead of > > ls -l file/ > >but at least the openat/runat approach also works for directories, which >does actually make it a lot more _generic_ than the "show in the regular >filespace" approach. No special cases. > > So you are saying I can do runat emacs my-compound-document/stream-name? and in place of CTRL-X CTRL-F to open a file in emacs I type what to open an attribute? I think it does not work Linus.... >So your comparison isn't valid, because you're ignoring the people who >shout "runat" at you. You've also not ever actually answered about the >problems about directories with attributes. > > Which problem specifically? I will list them and hope I don't miss what you want addressed. I won't claim to be sure I have the right answers for all of these (because VFS has not been my interest in the past), but these look feasible to me.;-) Cycle detection: We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer). Ambiguity: Linus said: - how do you handle the ambiguity of //attr/usr/bin/emacs/icon (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin"). If you want to avoid crowding the namespace with some piece of meta information that you want to be applicable to both directories and the files in them, and also widely used enough that you care about crowding the namespace, and unambiguous as to what level in the hierarchy it metas, then you put it into the metas subdirectory. There is no principled difference between a file that is a child of a directory and an attribute. The ambiguity is inherent in hiearchical semantics I think, and whether or not you have attributes you have this type of ambiguity. Thus, you have either: /usr/bin/metas/emacs/icon or /usr/bin/emacs/metas/icon and these are easily distinguished and unambiguous. This makes metas a reserved keyword, and there was a long discussion about whether to use "metas" or "..metas" as the reserved subdirectory name which I can summarize for you on request. There is also "pseudos", which contains files implementing special methods of special plugins. Feel free to invent more reserved subdirectories as needed, or to argue for fewer. Seeing two trees, one with attrs and one without: Linus said: If you open a filename in some "secondary" tree (be it /proc or //attr or whatever) based on the filename in the primary one, you have two issues that you need to work out: - how do you handle a name change in the primary tree at the same time as lookup - how do you handle the ambiguity of //attr/usr/bin/emacs/icon (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin"). The ambiguity can be handled by saying that attributes only have one component (ie only the _last_ component of a lookup is the attribute name). But the race between primary tree and secondary tree cannot be handled in a normal name-space. Are you sure that you have to have two trees in order to see two trees? That is, can you not refuse to follow or see a link depending on whether some flag is set and what the type of the link is? Locking: Linus said: let's say that you have filename "a" hard-linked to filename "b", and you have a directory structure of streams under there. So you have a/file1 a/dir1/file2 a/dir2/file3 and (through the hard-link with "b") you have aliases of all these same names available as "b/file1", "b/dir1/file2" etc). Now, imagine that you have two processes doing mv a/dir1 a/dir2/newdir and mv b/dir2 b/dir1/newdir at the same time. Both of them MUST NOT SUCCEED, for pretty obvious reasons (you'd have moved two directories within each other, and now neither would be accessible any more). How do you handle locking for this situation? (Here is where my knowledge of VFS is shakiest.) Making "hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks" seems to solve this. If we want to use the Alexander Smith solution to cycle detection, and allow hard links to directories, then we must make sure that for the hard linked entity, there is a single lock for it somewhere (e.g. the directory inode) that can be taken. If you create two dentries for the same object, then flag the dentries as needing to lock a lock in their shared inode and not in the dentry. If you don't want to read in the inode from disk to do that, then create a pointer to a lock shared by them both. Linus, is this a start on solving some of the issues? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-01 7:22 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-09-01 7:22 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Alexander Zarochentcev Linus Torvalds wrote: >On Tue, 31 Aug 2004, Hans Reiser wrote: > > >>You are saying, 1-2% simpler and better, no biggie, why work so hard to >>get it? >> >>And we are saying, 1-2% simpler and better, times thousands of >>applications, wow! That's a lot! >> >> > >But would thousands care? Seriously? > > >For example, you could make just _one_ program support "openat()", and >you'd get most of the advantages, with no possibility of _breaking_ any of >thousands of applications.. > >I know, you've ignored the "runat" program (which is just a wrapper around >the openat() system call), but it _has_ been mentioned several times in >this thread. Yes, you'd type a bit more to do > > runat file ls -l > >instead of > > ls -l file/ > >but at least the openat/runat approach also works for directories, which >does actually make it a lot more _generic_ than the "show in the regular >filespace" approach. No special cases. > > So you are saying I can do runat emacs my-compound-document/stream-name? and in place of CTRL-X CTRL-F to open a file in emacs I type what to open an attribute? I think it does not work Linus.... >So your comparison isn't valid, because you're ignoring the people who >shout "runat" at you. You've also not ever actually answered about the >problems about directories with attributes. > > Which problem specifically? I will list them and hope I don't miss what you want addressed. I won't claim to be sure I have the right answers for all of these (because VFS has not been my interest in the past), but these look feasible to me.;-) Cycle detection: We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer). Ambiguity: Linus said: - how do you handle the ambiguity of //attr/usr/bin/emacs/icon (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin"). If you want to avoid crowding the namespace with some piece of meta information that you want to be applicable to both directories and the files in them, and also widely used enough that you care about crowding the namespace, and unambiguous as to what level in the hierarchy it metas, then you put it into the metas subdirectory. There is no principled difference between a file that is a child of a directory and an attribute. The ambiguity is inherent in hiearchical semantics I think, and whether or not you have attributes you have this type of ambiguity. Thus, you have either: /usr/bin/metas/emacs/icon or /usr/bin/emacs/metas/icon and these are easily distinguished and unambiguous. This makes metas a reserved keyword, and there was a long discussion about whether to use "metas" or "..metas" as the reserved subdirectory name which I can summarize for you on request. There is also "pseudos", which contains files implementing special methods of special plugins. Feel free to invent more reserved subdirectories as needed, or to argue for fewer. Seeing two trees, one with attrs and one without: Linus said: If you open a filename in some "secondary" tree (be it /proc or //attr or whatever) based on the filename in the primary one, you have two issues that you need to work out: - how do you handle a name change in the primary tree at the same time as lookup - how do you handle the ambiguity of //attr/usr/bin/emacs/icon (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin"). The ambiguity can be handled by saying that attributes only have one component (ie only the _last_ component of a lookup is the attribute name). But the race between primary tree and secondary tree cannot be handled in a normal name-space. Are you sure that you have to have two trees in order to see two trees? That is, can you not refuse to follow or see a link depending on whether some flag is set and what the type of the link is? Locking: Linus said: let's say that you have filename "a" hard-linked to filename "b", and you have a directory structure of streams under there. So you have a/file1 a/dir1/file2 a/dir2/file3 and (through the hard-link with "b") you have aliases of all these same names available as "b/file1", "b/dir1/file2" etc). Now, imagine that you have two processes doing mv a/dir1 a/dir2/newdir and mv b/dir2 b/dir1/newdir at the same time. Both of them MUST NOT SUCCEED, for pretty obvious reasons (you'd have moved two directories within each other, and now neither would be accessible any more). How do you handle locking for this situation? (Here is where my knowledge of VFS is shakiest.) Making "hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks" seems to solve this. If we want to use the Alexander Smith solution to cycle detection, and allow hard links to directories, then we must make sure that for the hard linked entity, there is a single lock for it somewhere (e.g. the directory inode) that can be taken. If you create two dentries for the same object, then flag the dentries as needing to lock a lock in their shared inode and not in the dentry. If you don't want to read in the inode from disk to do that, then create a pointer to a lock shared by them both. Linus, is this a start on solving some of the issues? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 7:22 ` Hans Reiser (?) @ 2004-09-01 11:51 ` Alexander G. M. Smith -1 siblings, 0 replies; 1835+ messages in thread From: Alexander G. M. Smith @ 2004-09-01 11:51 UTC (permalink / raw) To: Hans Reiser Cc: ninja, vonbrand, pavel, jamie, cw, viro, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, zam, torvalds Hans Reiser wrote on Wed, 01 Sep 2004 00:22:49 -0700: > Making "hard links only link to the file aspect of the file-directory > duality, and persons who want to link to the directory aspect must use > symlinks" seems to solve this. If we want to use the Alexander Smith > solution to cycle detection, and allow hard links to directories, then > we must make sure that for the hard linked entity, there is a single > lock for it somewhere (e.g. the directory inode) that can be taken. Oddly enough that's what I did in my RAM file system. No matter how the file/directory node is found (the path can lead through different parents), it has a unique ID and a corresponding single lock for it. Rename/delete goes and locks all the nodes that might change parents and/or children, then it does its operations (though it has to watch out for deadlocks, which are unavoidable when two rename/deletes both sneak up on the same file by different paths). - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 6:06 ` Linus Torvalds 2004-09-01 6:32 ` Hubert Chan 2004-09-01 7:22 ` Hans Reiser @ 2004-09-01 23:04 ` Hubert Chan 2 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-01 23:04 UTC (permalink / raw) To: linux-fsdevel; +Cc: linux-kernel, reiserfs-list >>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes: Linus> For example, you could make just _one_ program support Linus> "openat()", and you'd get most of the advantages, with no Linus> possibility of _breaking_ any of thousands of applications.. I'm starting to think that even if we had openat, it wouldn't solve much. Maybe I'm just not imaginative enough, but I still see problems. Let's pretend, just for the sake of argument, that every program all of a sudden magically was modified to support openat, which is a big assumption. But we'll make that assumption so that we can ignore _that_ problem for a second. The question still remains, how do I tell the thing to open a stream? If cat knows about openat, how do I tell it to cat foo.txt, stream bar? I obviously can't do it using a normal filename, because if I could do that, we could just forget about the openat business. The only way I can think of is to give cat a flag that tells it what stream to open: "cat --stream bar foo.txt". Ugly. It also makes it impossible to do redirection, because you can't pass flags to "<". And before someone yells "runat" at me (which, btw, won't work with redirection either), consider larger programs like GIMP or emacs. Let's look at GIMP first. I've already got an instance of GIMP open, and I don't want to waste memory with another instance. How do I tell it to open the icon.png stream of the file xterm? When I go to File | Open, up pops a dialog that only shows real files. Trying to design a dialog box that will show real files and substreams, and not confuse the users is not going to be fun. Please don't ask the GTK/GNOME developers to design another file selection dialog. There were enough people whining the last round. If they need to add more complexity, we'll have even more whining. Now let's look at emacs. I want to open up the description.txt stream of the file xterm. So I do "C-x C-f", and it prompts me on the status bar for the file name. What do I type? Well, description.txt doesn't live in the normal namespace, so the only way around that is for emacs to ask for the filename, and then ask for the stream name. That's extra-annoying when I usually just want to open just the normal stream, and that second prompt just makes more work for me. Linus> I know, you've ignored the "runat" program (which is just a Linus> wrapper around the openat() system call), but it _has_ been Linus> mentioned several times in this thread. Yes, you'd type a bit Linus> more to do Linus> runat file ls -l Linus> instead of Linus> ls -l file/ Might not be just a "bit more" typing, because you lose handy things like tab completion. Say I want to "cat" file foo.txt, stream bar. Or was that baz? Instead of being able to type cat foo/ba<tab> and have the filename magically come up, I need to runat foo ls ba* runat foo cat baz (which also requires more thinking, since I need to think about how to formulate the query, read the output, and write the new command, and so is much much slower than being able to mindlessly hit the <tab> key.) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 5:50 ` Hans Reiser 2004-09-01 6:06 ` Linus Torvalds @ 2004-09-01 11:49 ` Alan Cox 1 sibling, 0 replies; 1835+ messages in thread From: Alan Cox @ 2004-09-01 11:49 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mer, 2004-09-01 at 06:50, Hans Reiser wrote: > Yes, changing cat to use openat() is no big deal. 1-2% additional coding > cost for cat, who cares? Large cost on its own. The FSF would almost certainly reject such a change until someone had written portable emulation of the feature for every other platform which would mean vendor patches which would essentially mean it wouldn't happen. I know I rarely agree with Hans but I think he's right on this one, it has to work with the existing open() API. Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 8:21 ` Pavel Machek @ 2004-09-01 1:10 ` David Masover 2004-08-31 19:31 ` Horst von Brand 2004-09-01 1:10 ` David Masover 2 siblings, 0 replies; 1835+ messages in thread From: David Masover @ 2004-09-01 1:10 UTC (permalink / raw) To: Pavel Machek Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Pavel Machek wrote: | Hi! | | |>| uservfs does |>| |>| cd foo.deb#uar |>cd foo.deb/ar |>| vs. |>| cd foo.deb#udeb |>cd foo.deb/deb |> |>and why would you want that, instead of just: |>cd foo.deb # for the ar |>dpkg -i foo.deb # for the deb | | | Because I want to see contents of that .deb package, nicely parsed? | | |>Just want to extract the tar file? Maybe something like |>cat foo.tgz/gunzip |>In which case (of course) foo.tgz/gunzip has exactly the same directory |>contents as foo.tgz | | | Yes, that would work. | | |>In fact, for just about any syntax anyone could suggest, I can't really |>see why you can't just replace all weird symbols with a slash and a |>symbol. Instead of |> foo.tgz#utar |>you have |> foo.tgz/#/utar |>Only difference is, some things which used to require special tools can |>now be serviced by less than what's in busybox. | | | That would work, too. I do not get your comment about busybox. Busybox is a bare minimum of unix tools, yet still includes things like tar/bz2. With such a system, you can use the most basic (sub-GNU) versions of tools like cp, cat, sh, ls, and so on, to do much more complex operations. What's more, you can see everything you can do to a file just by looking at it. I can certainly imagine doing cat foo.tgz/metas/README rather than a google search for tgz, followed by a "man tar" and "man gzip". Assuming I know nothing about tar/gzip files, it's a lot easier as a new user to have one common interface for everything -- including docs. Fewer commands to memorize, faster to learn new stuff. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUha3gHNmZLgCUhAQKRJw/7BUEm41gKZGoVj7gOkcUVe/SRdTs9WtTL jkjCX8Zzr/1nnAKTHLqNJEiAJ68Ndf+dRhDfL5y6dDHhOy/Dx+38SzsWz+dQnMJ0 Fiq5fFDO/DTd0UPecXYPhBrtVeCdfjyyENhgRgJud6D8F39qvLWrZFUsIkjAA3lj nzZ/48jRPDWT/mF6JJmv0RfVvNtMWh9HUcW/i0+n4WZU1yFp6jyiF5ExCOP84z3x yXQ4Q9hnjgt52MDJTrGsCDvfCSQscQilxvXbl0iaozquIhG9Jz1XNYpij2XUygFi RDc1LA1k74VvbWahlkdfbTI+3P4qeMYGBblmfwXVDRwW0i4NH7X0/6xcnr1QjaK/ f/7EHBz0wnkIG4OpcGWzu9ik4WOR0WXapKrd938XC/mG+WzIluJlQsxG1A2HoRqg R66E/mVMVvHJWnJjrW6s10WIemPiKnuXgeIVPifsw81DLXlZ3wxjlZU3iuo7xked O6wNDFQBWQR8kv+EMDIt3gtbUdB+5KpxGR+Yli0AjxEdpdk3oZTfe5XQiNdJhcl5 xrcomVKXGvR96IlNUwHy1EsQk5LySMl101vJIS4z6i7lAIM5CQ1d6eOZrPp/DBOP Rw0KoQaiPd3SKNMgoyWSKQL6h85i1PRUN62WH0T2NRU1fZ3EazpiRZr+fuaFqFT9 Wk2lkDvm5yw= =ShWo -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-01 1:10 ` David Masover 0 siblings, 0 replies; 1835+ messages in thread From: David Masover @ 2004-09-01 1:10 UTC (permalink / raw) To: Pavel Machek Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Pavel Machek wrote: | Hi! | | |>| uservfs does |>| |>| cd foo.deb#uar |>cd foo.deb/ar |>| vs. |>| cd foo.deb#udeb |>cd foo.deb/deb |> |>and why would you want that, instead of just: |>cd foo.deb # for the ar |>dpkg -i foo.deb # for the deb | | | Because I want to see contents of that .deb package, nicely parsed? | | |>Just want to extract the tar file? Maybe something like |>cat foo.tgz/gunzip |>In which case (of course) foo.tgz/gunzip has exactly the same directory |>contents as foo.tgz | | | Yes, that would work. | | |>In fact, for just about any syntax anyone could suggest, I can't really |>see why you can't just replace all weird symbols with a slash and a |>symbol. Instead of |> foo.tgz#utar |>you have |> foo.tgz/#/utar |>Only difference is, some things which used to require special tools can |>now be serviced by less than what's in busybox. | | | That would work, too. I do not get your comment about busybox. Busybox is a bare minimum of unix tools, yet still includes things like tar/bz2. With such a system, you can use the most basic (sub-GNU) versions of tools like cp, cat, sh, ls, and so on, to do much more complex operations. What's more, you can see everything you can do to a file just by looking at it. I can certainly imagine doing cat foo.tgz/metas/README rather than a google search for tgz, followed by a "man tar" and "man gzip". Assuming I know nothing about tar/gzip files, it's a lot easier as a new user to have one common interface for everything -- including docs. Fewer commands to memorize, faster to learn new stuff. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUha3gHNmZLgCUhAQKRJw/7BUEm41gKZGoVj7gOkcUVe/SRdTs9WtTL jkjCX8Zzr/1nnAKTHLqNJEiAJ68Ndf+dRhDfL5y6dDHhOy/Dx+38SzsWz+dQnMJ0 Fiq5fFDO/DTd0UPecXYPhBrtVeCdfjyyENhgRgJud6D8F39qvLWrZFUsIkjAA3lj nzZ/48jRPDWT/mF6JJmv0RfVvNtMWh9HUcW/i0+n4WZU1yFp6jyiF5ExCOP84z3x yXQ4Q9hnjgt52MDJTrGsCDvfCSQscQilxvXbl0iaozquIhG9Jz1XNYpij2XUygFi RDc1LA1k74VvbWahlkdfbTI+3P4qeMYGBblmfwXVDRwW0i4NH7X0/6xcnr1QjaK/ f/7EHBz0wnkIG4OpcGWzu9ik4WOR0WXapKrd938XC/mG+WzIluJlQsxG1A2HoRqg R66E/mVMVvHJWnJjrW6s10WIemPiKnuXgeIVPifsw81DLXlZ3wxjlZU3iuo7xked O6wNDFQBWQR8kv+EMDIt3gtbUdB+5KpxGR+Yli0AjxEdpdk3oZTfe5XQiNdJhcl5 xrcomVKXGvR96IlNUwHy1EsQk5LySMl101vJIS4z6i7lAIM5CQ1d6eOZrPp/DBOP Rw0KoQaiPd3SKNMgoyWSKQL6h85i1PRUN62WH0T2NRU1fZ3EazpiRZr+fuaFqFT9 Wk2lkDvm5yw= =ShWo -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:05 ` Chris Wedgwood 2004-08-26 11:02 ` Jamie Lokier @ 2004-08-27 21:01 ` Pavel Machek 1 sibling, 0 replies; 1835+ messages in thread From: Pavel Machek @ 2004-08-27 21:01 UTC (permalink / raw) To: Chris Wedgwood Cc: Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > One of the big potential uses for file-as-directory is to go inside > > archive files, ELF files, .iso files and so on in a convenient way. > > Arguably this belongs in userspace --- and people have put it there. > uservfs.sf.net, using existing coda hooks. There are 2+ very similar projects. Advantage is we can do ftpfs, too. Having ftp done by > /ftp cd /ftp/sunsite would be too ugly for reiser4 plugins... Or not? -- 64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:30 ` viro 2004-08-26 1:00 ` Jamie Lokier @ 2004-08-26 1:13 ` Chris Wright 1 sibling, 0 replies; 1835+ messages in thread From: Chris Wright @ 2004-08-26 1:13 UTC (permalink / raw) To: viro Cc: Jamie Lokier, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List * viro@parcelfarce.linux.theplanet.co.uk (viro@parcelfarce.linux.theplanet.co.uk) wrote: > On Thu, Aug 26, 2004 at 01:11:52AM +0100, Jamie Lokier wrote: > > Is this a problem if we treat entering a file-as-directory as crossing > > a mount point (i.e. like auto-mounting)? > > Yes - mountpoints can't be e.g. unlinked. Could it be essentially MNT_DETACH'd? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:25 ` viro 2004-08-26 0:11 ` Jamie Lokier @ 2004-08-29 9:55 ` Mikaël Cluseau 1 sibling, 0 replies; 1835+ messages in thread From: Mikaël Cluseau @ 2004-08-29 9:55 UTC (permalink / raw) To: reiserfs-list Hello, This thread is very very very very big. I think lots of people spend hours talking here. Isn't it time for a summary of all this please ? (for people like me who are quite busy and try to follow... but also for the health of the debate) Thank! Mikael. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:42 ` viro @ 2004-08-26 8:43 ` Hans Reiser 2004-08-25 21:00 ` Linus Torvalds 2004-08-26 8:43 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Jason Holt viro@parcelfarce.linux.theplanet.co.uk wrote: >On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > > >>On Wed, 25 Aug 2004, Christoph Hellwig wrote: >> >> >>>For one thing _I_ didn't decide about xattrs anyway. And I still >>>haven't seen a design from you on -fsdevel how you try to solve the >>>problems with files as directories. >>> >>> >>Hey, files-as-directories are one of my pet things, so I have to side with >>Hans on this one. I think it just makes sense. A hell of a lot more sense >>than xattrs, anyway, since it allows scripts etc standard tools to touch >>the attributes. >> >>It's the UNIX way. >> >> > >Not if you allow link(2) on them. And not if you design and market your >stuff as a general-purpose backdoor into kernel. > What backdoor? Please spell it out. Plugins are not dynamically loadable. > Note how *EVERY* *DAMN* >*OPERATION* is made possible to override by "plugins". Which is the reason >for deadlocks in question, BTW. > >Don't fool yourself - that's what Hans is selling. Target market: ISV. > > Hunh? No, target market is hackers who like to spend a weekend dreaming up funky new kinds of files. One guy (Jason Holt, clever guy) came up with an idea for write only files for which even root cannot read the parts of the file written prior to the time root was achieved, because the encryption key is changed in a forward computable only direction with every write, and the start key is kept on another computer. Lots of folks will have plugins I would never dream of. Think of photoshop plugins, and what plugins did for photoshop. Same thing will happen to reiser4 next year (or earlier). >Marketed product: a set of hooks, the wider the better, no matter how >little sense it makes. The reason for doing that outside of core kernel: >bypassing any review and being able to control the product being sold (see >above). > >Shame that it got an actual filesystem mixed in with the marketing plans >and general insanity... > > > > I am one of those free software hippy-child anarchists who thinks that random people should come up with ideas and contribute them. You understand me well.;-) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:43 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Jason Holt viro@parcelfarce.linux.theplanet.co.uk wrote: >On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > > >>On Wed, 25 Aug 2004, Christoph Hellwig wrote: >> >> >>>For one thing _I_ didn't decide about xattrs anyway. And I still >>>haven't seen a design from you on -fsdevel how you try to solve the >>>problems with files as directories. >>> >>> >>Hey, files-as-directories are one of my pet things, so I have to side with >>Hans on this one. I think it just makes sense. A hell of a lot more sense >>than xattrs, anyway, since it allows scripts etc standard tools to touch >>the attributes. >> >>It's the UNIX way. >> >> > >Not if you allow link(2) on them. And not if you design and market your >stuff as a general-purpose backdoor into kernel. > What backdoor? Please spell it out. Plugins are not dynamically loadable. > Note how *EVERY* *DAMN* >*OPERATION* is made possible to override by "plugins". Which is the reason >for deadlocks in question, BTW. > >Don't fool yourself - that's what Hans is selling. Target market: ISV. > > Hunh? No, target market is hackers who like to spend a weekend dreaming up funky new kinds of files. One guy (Jason Holt, clever guy) came up with an idea for write only files for which even root cannot read the parts of the file written prior to the time root was achieved, because the encryption key is changed in a forward computable only direction with every write, and the start key is kept on another computer. Lots of folks will have plugins I would never dream of. Think of photoshop plugins, and what plugins did for photoshop. Same thing will happen to reiser4 next year (or earlier). >Marketed product: a set of hooks, the wider the better, no matter how >little sense it makes. The reason for doing that outside of core kernel: >bypassing any review and being able to control the product being sold (see >above). > >Shame that it got an actual filesystem mixed in with the marketing plans >and general insanity... > > > > I am one of those free software hippy-child anarchists who thinks that random people should come up with ideas and contribute them. You understand me well.;-) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-25 20:42 ` viro @ 2004-08-25 21:52 ` Matt Mackall 2004-08-25 22:21 ` Linus Torvalds 2004-08-26 0:18 ` Mikulas Patocka ` (3 subsequent siblings) 7 siblings, 1 reply; 1835+ messages in thread From: Matt Mackall @ 2004-08-25 21:52 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > > > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > haven't seen a design from you on -fsdevel how you try to solve the > > problems with files as directories. > > Hey, files-as-directories are one of my pet things, so I have to side with > Hans on this one. I think it just makes sense. A hell of a lot more sense > than xattrs, anyway, since it allows scripts etc standard tools to touch > the attributes. > > It's the UNIX way. I thought the UNIX way is "everything's a file", not "everything's a directory". > Will it potentially break something? Sure. Do we care? Me, I'll take that > kind of extension _any_ day over xattrs, that are fundamentally flawed in > my opinion and totally useless. There's always the option that they're both broken. -- Mathematics is the supreme nostalgia of our time. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:52 ` Matt Mackall @ 2004-08-25 22:21 ` Linus Torvalds 0 siblings, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-25 22:21 UTC (permalink / raw) To: Matt Mackall Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004, Matt Mackall wrote: > > > > It's the UNIX way. > > I thought the UNIX way is "everything's a file", not "everything's a > directory". It really was. Directories were historically largely just files too, although with the special "lookup" operation. Historic unix didn't have readdir/rmdir/mkdir/rename or really much _any_ special directory handling. Directories were just files, and you read them like files. Of course, even in that early unix, "directories" were very much a reality even apart from the fact that they happened to be implemented pretty much like files. Nobody has ever claimed that the UNIX way is "everything is _one_ file", after all ;) > > Will it potentially break something? Sure. Do we care? Me, I'll take that > > kind of extension _any_ day over xattrs, that are fundamentally flawed in > > my opinion and totally useless. > > There's always the option that they're both broken. Yes. Highly likely. However, something like that _does_ end up what a Windows fileserver wants. IOW, even if it's broken, _something_ is likely forced on us by that nasty thing we call "real users". Damn them. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds ` (3 preceding siblings ...) 2004-08-25 21:52 ` Matt Mackall @ 2004-08-26 0:18 ` Mikulas Patocka 2004-08-26 0:27 ` Linus Torvalds 2004-08-26 0:57 ` Rik van Riel 2004-08-26 8:43 ` Hans Reiser ` (2 subsequent siblings) 7 siblings, 2 replies; 1835+ messages in thread From: Mikulas Patocka @ 2004-08-26 0:18 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004, Linus Torvalds wrote: > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > haven't seen a design from you on -fsdevel how you try to solve the > > problems with files as directories. > > Hey, files-as-directories are one of my pet things, so I have to side with > Hans on this one. I think it just makes sense. A hell of a lot more sense > than xattrs, anyway, since it allows scripts etc standard tools to touch > the attributes. > > It's the UNIX way. > > And yes, the semantics can _easily_ be solved in very unixy ways. > > One way to solve it is to just realize that a final slash at the end > implies pretty strongly that you want to treat it as a directory. So what > you do is: > > - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) > - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) > > Problem solved. Very user-friendly, and very intuitive. Stupid question: who will use it? And why? Anyone can write an userspace library, that implements function set_attribute(char *file, char *attribute, char *value), that creates directory ".attr/file" in file's directory and stores attribute there. (and you can get list of attributes from shell too: ls `echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/'` ). There's no need to add extra functionality to kernel and filesystem. Advantage: - you don't add bloat to kernel or filesystem - you don't need to teach tar/cp -a/mc about attributes - you won't lose attributes after editing file in vim (it creates another file and renames it over original one) > Will it potentially break something? Sure. Do we care? Me, I'll take that > kind of extension _any_ day over xattrs, that are fundamentally flawed in > my opinion and totally useless. The argument that applications like "tar" > won't understand the file-as-directory thing is _flawed_, since legacy > apps won't understand xattrs either. The only way xattrs are useful is that backup/restore software doesn't have to know about every filesystem with it's specific attributes and every magic ioctl for setting them. Instead it can save/restore filesystem-specific attributes without understanding what do they mean. However there's no need why application should use them. And no application does. I can't imagine anyone shipping an application with "this app requires reiser4" prerequisite. Why should anyone use it if he can store attributes in ".attr" directory or whereever and make the application work on any OS and any filesystem? Mikulas ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:18 ` Mikulas Patocka @ 2004-08-26 0:27 ` Linus Torvalds 2004-08-26 0:51 ` Mikulas Patocka 2004-08-26 8:36 ` Hans Reiser 2004-08-26 0:57 ` Rik van Riel 1 sibling, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 0:27 UTC (permalink / raw) To: Mikulas Patocka Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Mikulas Patocka wrote: > > Stupid question: who will use it? And why? > > Anyone can write an userspace library, that implements function > set_attribute(char *file, char *attribute, char *value), that creates > directory ".attr/file" in file's directory and stores attribute there. > (and you can get list of attributes from shell too: > ls `echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/'` > ). There's no need to add extra functionality to kernel and filesystem. ..and the above is, roughly, what I understand samba etc falls back on. The problem ends up being that the above isn't in any way safe from people moving files around (oops, where did those attributes go?) nor does it have any consistency guarantees. So it only works well if _one_ application does this, and that application follows all the locking rules. Is it enough? It may have to be. > The only way xattrs are useful is that backup/restore software doesn't > have to know about every filesystem with it's specific attributes and > every magic ioctl for setting them. Instead it can save/restore > filesystem-specific attributes without understanding what do they mean. > However there's no need why application should use them. And no > application does. If no application does, then why back them up? Why implement them in the first place? In other words - some apps obviously do want to use the. Sadly. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:27 ` Linus Torvalds @ 2004-08-26 0:51 ` Mikulas Patocka 2004-08-26 8:36 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Mikulas Patocka @ 2004-08-26 0:51 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > > The only way xattrs are useful is that backup/restore software doesn't > > have to know about every filesystem with it's specific attributes and > > every magic ioctl for setting them. Instead it can save/restore > > filesystem-specific attributes without understanding what do they mean. > > However there's no need why application should use them. And no > > application does. > > If no application does, then why back them up? Why implement them in the > first place? > > In other words - some apps obviously do want to use the. Sadly. You can add more functionality to filesystem and use xattrs to control it. For example: - acls - compress file - encrypt file (copy user's password into task_struct and use it to encrypt his files) - preallocate file in 4MB contignuous chunks, becuase it needs real time multimedia access - sync/append-only/immutable etc. However there's no need why an application should care whether the file is compressed, whether it has acls, or so. And applications don't. And I think this is the only legitimate use for xattrs. Who else uses them except samba? I don't see how reiser4's hybrids would help. Mikulas ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:27 ` Linus Torvalds @ 2004-08-26 8:36 ` Hans Reiser 2004-08-26 8:36 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:36 UTC (permalink / raw) To: Linus Torvalds Cc: Mikulas Patocka, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Streams are quite ugly. However, if you decompose streams into all of the little pieces that are needed to emulate them, the pieces are quite nice. For instance, inheriting stat data from a common parent is nice, and inheritance is nice, and being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice, and, well, if you decompose all of the features of streams into little features you get a bunch of fun little features much nicer than streams. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:36 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:36 UTC (permalink / raw) To: Linus Torvalds Cc: Mikulas Patocka, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Streams are quite ugly. However, if you decompose streams into all of the little pieces that are needed to emulate them, the pieces are quite nice. For instance, inheriting stat data from a common parent is nice, and inheritance is nice, and being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice, and, well, if you decompose all of the features of streams into little features you get a bunch of fun little features much nicer than streams. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:36 ` Hans Reiser (?) @ 2004-08-26 10:53 ` Jamie Lokier 2004-08-26 10:59 ` Christoph Hellwig -1 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 10:53 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Mikulas Patocka, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser wrote: > being able to cat dirname/pseudos/cat and get a > concatenation of all of the files is nice, and being able to cat > dirname/pseudos/tar and get an archive of the directory is nice Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:53 ` Jamie Lokier @ 2004-08-26 10:59 ` Christoph Hellwig 2004-08-26 11:07 ` Jamie Lokier 2004-08-26 11:17 ` Spam 0 siblings, 2 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 10:59 UTC (permalink / raw) To: Jamie Lokier Cc: Hans Reiser, Linus Torvalds, Mikulas Patocka, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote: > Hans Reiser wrote: > > being able to cat dirname/pseudos/cat and get a > > concatenation of all of the files is nice, and being able to cat > > dirname/pseudos/tar and get an archive of the directory is nice > > Yes. Being able to cd into filename.tar.gz and filename.iso is also > nice, but all of these features should be supported by the VFS > generically, not in any specific filesystem, and there should be a > hook to invoke the various fun filesystem-independent handlers by name. It doesn't belong into the kernel at all. If at all it belongs into a userspace filesystems, but even in that case the magic detection of which one to use is kinda hard. You absoutely don't want to hardcode file formats in the kernel. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:59 ` Christoph Hellwig @ 2004-08-26 11:07 ` Jamie Lokier 2004-08-26 11:17 ` Spam 1 sibling, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 11:07 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, Linus Torvalds, Mikulas Patocka, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Christoph Hellwig wrote: > > > being able to cat dirname/pseudos/cat and get a > > > concatenation of all of the files is nice, and being able to cat > > > dirname/pseudos/tar and get an archive of the directory is nice > > > > Yes. Being able to cd into filename.tar.gz and filename.iso is also > > nice, but all of these features should be supported by the VFS > > generically, not in any specific filesystem, and there should be a > > hook to invoke the various fun filesystem-independent handlers by name. > > It doesn't belong into the kernel at all. If at all it belongs into a > userspace filesystems, but even in that case the magic detection of > which one to use is kinda hard. You absoutely don't want to hardcode > file formats in the kernel. Oh, I agree userspace should be involved. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:59 ` Christoph Hellwig @ 2004-08-26 11:17 ` Spam 2004-08-26 11:17 ` Spam 1 sibling, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:17 UTC (permalink / raw) To: Christoph Hellwig Cc: Jamie Lokier, Hans Reiser, Linus Torvalds, Mikulas Patocka, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote: >> Hans Reiser wrote: >> > being able to cat dirname/pseudos/cat and get a >> > concatenation of all of the files is nice, and being able to cat >> > dirname/pseudos/tar and get an archive of the directory is nice >> >> Yes. Being able to cd into filename.tar.gz and filename.iso is also >> nice, but all of these features should be supported by the VFS >> generically, not in any specific filesystem, and there should be a >> hook to invoke the various fun filesystem-independent handlers by name. > It doesn't belong into the kernel at all. If at all it belongs into a > userspace filesystems, but even in that case the magic detection of > which one to use is kinda hard. You absoutely don't want to hardcode > file formats in the kernel. Do you mean user-level file system as a VFS handled by user applications, or a intermediate file system layer between any application and the real file system? The latter would be good enough as it would still be transparent to the applications. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 11:17 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:17 UTC (permalink / raw) To: Christoph Hellwig Cc: Jamie Lokier, Hans Reiser, Linus Torvalds, Mikulas Patocka, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote: >> Hans Reiser wrote: >> > being able to cat dirname/pseudos/cat and get a >> > concatenation of all of the files is nice, and being able to cat >> > dirname/pseudos/tar and get an archive of the directory is nice >> >> Yes. Being able to cd into filename.tar.gz and filename.iso is also >> nice, but all of these features should be supported by the VFS >> generically, not in any specific filesystem, and there should be a >> hook to invoke the various fun filesystem-independent handlers by name. > It doesn't belong into the kernel at all. If at all it belongs into a > userspace filesystems, but even in that case the magic detection of > which one to use is kinda hard. You absoutely don't want to hardcode > file formats in the kernel. Do you mean user-level file system as a VFS handled by user applications, or a intermediate file system layer between any application and the real file system? The latter would be good enough as it would still be transparent to the applications. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:18 ` Mikulas Patocka @ 2004-08-26 0:57 ` Rik van Riel 2004-08-26 0:57 ` Rik van Riel 1 sibling, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 0:57 UTC (permalink / raw) To: Mikulas Patocka Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Mikulas Patocka wrote: > On Wed, 25 Aug 2004, Linus Torvalds wrote: > > One way to solve it is to just realize that a final slash at the end > > implies pretty strongly that you want to treat it as a directory. So what > > you do is: > > Stupid question: who will use it? And why? I've got a stupid question too. How do you back up these things ? If your backup program reads them as a file and restores them as a file, you might lose your directory-inside-the-file magic. If your backup program dives into the file despite stat() saying it's a file and you restore your backup, how are the "file is a file" semantics preserved ? Obviously this is something that needs to be sorted out at the VFS layer. A filesystem specific backup and restore program isn't desirable, if only because then there'd be no way for Hans's users to switch to reiser5 in 2010 ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 0:57 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 0:57 UTC (permalink / raw) To: Mikulas Patocka Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Mikulas Patocka wrote: > On Wed, 25 Aug 2004, Linus Torvalds wrote: > > One way to solve it is to just realize that a final slash at the end > > implies pretty strongly that you want to treat it as a directory. So what > > you do is: > > Stupid question: who will use it? And why? I've got a stupid question too. How do you back up these things ? If your backup program reads them as a file and restores them as a file, you might lose your directory-inside-the-file magic. If your backup program dives into the file despite stat() saying it's a file and you restore your backup, how are the "file is a file" semantics preserved ? Obviously this is something that needs to be sorted out at the VFS layer. A filesystem specific backup and restore program isn't desirable, if only because then there'd be no way for Hans's users to switch to reiser5 in 2010 ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:57 ` Rik van Riel (?) @ 2004-08-26 7:51 ` Helge Hafting 2004-08-26 9:21 ` Paul Jackson -1 siblings, 1 reply; 1835+ messages in thread From: Helge Hafting @ 2004-08-26 7:51 UTC (permalink / raw) To: Rik van Riel Cc: Mikulas Patocka, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Rik van Riel wrote: >On Thu, 26 Aug 2004, Mikulas Patocka wrote: > > >>On Wed, 25 Aug 2004, Linus Torvalds wrote: >> >> > > > >>>One way to solve it is to just realize that a final slash at the end >>>implies pretty strongly that you want to treat it as a directory. So what >>>you do is: >>> >>> >>Stupid question: who will use it? And why? >> >> > >I've got a stupid question too. How do you back up these >things ? > >If your backup program reads them as a file and restores >them as a file, you might lose your directory-inside-the-file >magic. > >If your backup program dives into the file despite stat() >saying it's a file and you restore your backup, how are the >"file is a file" semantics preserved ? > > > >Obviously this is something that needs to be sorted out at >the VFS layer. A filesystem specific backup and restore >program isn't desirable, if only because then there'd be >no way for Hans's users to switch to reiser5 in 2010 ;) > > > Sure, this sort of thing must be sorted out at the VFS layer. And a backup program working on such a filesystem will need to know that something can be a file, a directory - or both. So an old "tar" won't get this right as it will assume that an object is either file or directory. The change to get it right won't be that big - just notice that an object is both, then backup the ordinary file contents as usual, before recursing into the directory it also provides and backup stuff there as usual. The resulting .tar can of course only be unpacked properly on a fs supporting file-as-directory, similiar to how a .tar of a fs with links only will unpack properly on a fs supporting links. I don't see much problems for userland. Old apps will keep working, as the new features is a superset. Those who care about file-as-directory extras will provide patches for "tar" and friends, after that the extras become useable. Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 7:51 ` Helge Hafting @ 2004-08-26 9:21 ` Paul Jackson 2004-08-26 10:47 ` Jamie Lokier 2004-08-27 12:33 ` Helge Hafting 0 siblings, 2 replies; 1835+ messages in thread From: Paul Jackson @ 2004-08-26 9:21 UTC (permalink / raw) To: Helge Hafting Cc: riel, mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Helge wrote: > So an old "tar" won't get this right as it will assume that an object > is either file or directory. There are many backup apps, not just one. I've written a few myself, none of which will ever be worthy of notice. The sourceforge Topic.System.Archiving.Backup lists 335 projects at present. I find the idea that most backup tools and scripts will silently stop working correctly to be pretty scary. And then there's archiving, installation, distribution, administration, emulation, file system and partition managers, and on and on. === I wonder if we can make this "modal" somehow. The one consistency I see is that apps that want the "enhanced" view need to ask for it, somehow. It is the new views of the data that are being added - let the app announce to the kernel (usually via specialized code in some shared library that the app is using to get the alternate views) that either per-task, or per-file descriptor, it is to see the "enhanced" view, as a side affect of trying to access it. Old stuff, or even new stuff that is content to work with the "classic" view that a file is a single data stream, and that directories only have pathnames, not data, would by default see that view, and see _all_ the data, presented somehow in that view, perhaps as additional files with magic names. This still leaves the breakage that such tools don't know, and don't preserve, the magic linkage between such magic files. But that is much less of an issue, in my view. Programs such as backups that are manipulating the files of apps they know nothing about already have to presume that all the files are important in inscrutable ways, and just be careful to preserve or copy or backup all of them. Yeah - I realize that there will be a few followups denouncing modal architectures. I might even agree with some of them. If this were easy, it would have been done years ago. The onus should be on the new stuff to request the enhanced view, rather than on the old dogs to learn new tricks. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:21 ` Paul Jackson @ 2004-08-26 10:47 ` Jamie Lokier 2004-08-26 11:19 ` Paul Jackson 2004-08-27 12:33 ` Helge Hafting 1 sibling, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 10:47 UTC (permalink / raw) To: Paul Jackson Cc: Helge Hafting, riel, mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Paul Jackson wrote: > I find the idea that most backup tools and scripts will silently > stop working correctly to be pretty scary. Too late. We have xattrs already; many programs don't store them. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:47 ` Jamie Lokier @ 2004-08-26 11:19 ` Paul Jackson 0 siblings, 0 replies; 1835+ messages in thread From: Paul Jackson @ 2004-08-26 11:19 UTC (permalink / raw) To: Jamie Lokier Cc: helge.hafting, riel, mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list > Too late. We have xattrs already; many programs don't store them. If by your "too late" you mean we can stop worrying about any more breakage of file system utilities, because there exists an example in which some were already broken, then you are absolutely wrong. Just because we caused some breakage doesn't give us license to cause even more. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:21 ` Paul Jackson @ 2004-08-27 12:33 ` Helge Hafting 2004-08-27 12:33 ` Helge Hafting 1 sibling, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-27 12:33 UTC (permalink / raw) To: Paul Jackson Cc: riel, mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Paul Jackson wrote: >Helge wrote: > > >>So an old "tar" won't get this right as it will assume that an object >>is either file or directory. >> >> > >There are many backup apps, not just one. I've written a few myself, >none of which will ever be worthy of notice. The sourceforge >Topic.System.Archiving.Backup lists 335 projects at present. > >I find the idea that most backup tools and scripts will silently >stop working correctly to be pretty scary. > > They won't stop working, they will merely not support the new features. That is only a problem if you actually use those features. If all you have is plain files and plain directories - no problem. Not if files-as-directories are implemented right. Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 12:33 ` Helge Hafting 0 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-27 12:33 UTC (permalink / raw) To: Paul Jackson Cc: riel, mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Paul Jackson wrote: >Helge wrote: > > >>So an old "tar" won't get this right as it will assume that an object >>is either file or directory. >> >> > >There are many backup apps, not just one. I've written a few myself, >none of which will ever be worthy of notice. The sourceforge >Topic.System.Archiving.Backup lists 335 projects at present. > >I find the idea that most backup tools and scripts will silently >stop working correctly to be pretty scary. > > They won't stop working, they will merely not support the new features. That is only a problem if you actually use those features. If all you have is plain files and plain directories - no problem. Not if files-as-directories are implemented right. Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:57 ` Rik van Riel @ 2004-08-26 8:40 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:40 UTC (permalink / raw) To: Rik van Riel Cc: Mikulas Patocka, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Rik van Riel wrote: > > >If your backup program dives into the file despite stat() >saying it's a file and you restore your backup, how are the >"file is a file" semantics preserved ? > >Obviously this is something that needs to be sorted out at >the VFS layer. > It needs to be sorted out, whether it is sorted out at the VFS layer is unimportant. > A filesystem specific backup and restore >program isn't desirable, if only because then there'd be >no way for Hans's users to switch to reiser5 in 2010 ;) > > > It might be that we need a filenameA/metas/backup method for all of our file plugins, which if cat'd gives a set of instructions which if executed are adequate for restoring filenameA. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:40 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:40 UTC (permalink / raw) To: Rik van Riel Cc: Mikulas Patocka, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Rik van Riel wrote: > > >If your backup program dives into the file despite stat() >saying it's a file and you restore your backup, how are the >"file is a file" semantics preserved ? > >Obviously this is something that needs to be sorted out at >the VFS layer. > It needs to be sorted out, whether it is sorted out at the VFS layer is unimportant. > A filesystem specific backup and restore >program isn't desirable, if only because then there'd be >no way for Hans's users to switch to reiser5 in 2010 ;) > > > It might be that we need a filenameA/metas/backup method for all of our file plugins, which if cat'd gives a set of instructions which if executed are adequate for restoring filenameA. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:40 ` Hans Reiser (?) @ 2004-08-26 14:46 ` Lars Marowsky-Bree -1 siblings, 0 replies; 1835+ messages in thread From: Lars Marowsky-Bree @ 2004-08-26 14:46 UTC (permalink / raw) To: Hans Reiser, Rik van Riel Cc: Mikulas Patocka, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On 2004-08-26T01:40:29, Hans Reiser <reiser@namesys.com> said: > >Obviously this is something that needs to be sorted out at > >the VFS layer. > > > It needs to be sorted out, whether it is sorted out at the VFS layer is > unimportant. So what exactly is wrong with sorting it out at the VFS layer, and why do you _insist_ on sorting it in the reiserfs4 core? I'm missing something, please fill me in on the details. Sincerely, Lars Marowsky-Brée <lmb@suse.de> -- High Availability & Clustering \\\ /// SUSE Labs, Research and Development \honk/ SUSE LINUX AG - A Novell company \\// ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:57 ` Rik van Riel ` (2 preceding siblings ...) (?) @ 2004-08-26 9:44 ` Paul Jackson -1 siblings, 0 replies; 1835+ messages in thread From: Paul Jackson @ 2004-08-26 9:44 UTC (permalink / raw) To: Rik van Riel Cc: mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel writes: > If your backup program reads them as a file and restores > them as a file, you might lose your directory-inside-the-file > magic. Encode the magic in the names, by stealing a bit of the existing filename space to encode it. Such works pretty well as part of the magic to map long filenames into DOS 8.3 names on my FAT partitions. Apps linked with the appropriate Windows library see nice fancy long names. The rest of the world, including DOS apps and my Unix backup scripts, see the primitive 8.3 names, including one or a few extra files per directory, which are nothing special to them. So long as these other apps don't presume to know that they can keep some of the files in an apps directory, and drop others, then it works well enough. And no self-respecting general purpose backup program is going to presume such knowledge anyway. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds @ 2004-08-26 8:43 ` Hans Reiser 2004-08-25 20:41 ` Hans Reiser ` (6 subsequent siblings) 7 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > >>For one thing _I_ didn't decide about xattrs anyway. And I still >>haven't seen a design from you on -fsdevel how you try to solve the >>problems with files as directories. >> >> > >Hey, files-as-directories are one of my pet things, so I have to side with >Hans on this one. I think it just makes sense. A hell of a lot more sense >than xattrs, anyway, since it allows scripts etc standard tools to touch >the attributes. > >It's the UNIX way. > >And yes, the semantics can _easily_ be solved in very unixy ways. > >One way to solve it is to just realize that a final slash at the end >implies pretty strongly that you want to treat it as a directory. So what >you do is: > > - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) > - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) > >Problem solved. Very user-friendly, and very intuitive. > >Will it potentially break something? Sure. Do we care? Me, I'll take that >kind of extension _any_ day over xattrs, that are fundamentally flawed in >my opinion and totally useless. The argument that applications like "tar" >won't understand the file-as-directory thing is _flawed_, since legacy >apps won't understand xattrs either. > >Oh, add a O_NOXATTRS flag to force a path lookup to only use regular >directories, the same way we have O_NOFOLLOW and friends. That allows >people to see the difference, if they care (ie a file server might decide >that it doesn't want to expose things like this). > > I think we should require people to care enough to supply an O_NOMETAS flag to see the difference. >I never liked the xattr stuff. It makes little sense, and is totally >useless for 99.9999% of everything. I still don't see the point of it, >except for samba. Ugly. > > Linus > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:43 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > >>For one thing _I_ didn't decide about xattrs anyway. And I still >>haven't seen a design from you on -fsdevel how you try to solve the >>problems with files as directories. >> >> > >Hey, files-as-directories are one of my pet things, so I have to side with >Hans on this one. I think it just makes sense. A hell of a lot more sense >than xattrs, anyway, since it allows scripts etc standard tools to touch >the attributes. > >It's the UNIX way. > >And yes, the semantics can _easily_ be solved in very unixy ways. > >One way to solve it is to just realize that a final slash at the end >implies pretty strongly that you want to treat it as a directory. So what >you do is: > > - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) > - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) > >Problem solved. Very user-friendly, and very intuitive. > >Will it potentially break something? Sure. Do we care? Me, I'll take that >kind of extension _any_ day over xattrs, that are fundamentally flawed in >my opinion and totally useless. The argument that applications like "tar" >won't understand the file-as-directory thing is _flawed_, since legacy >apps won't understand xattrs either. > >Oh, add a O_NOXATTRS flag to force a path lookup to only use regular >directories, the same way we have O_NOFOLLOW and friends. That allows >people to see the difference, if they care (ie a file server might decide >that it doesn't want to expose things like this). > > I think we should require people to care enough to supply an O_NOMETAS flag to see the difference. >I never liked the xattr stuff. It makes little sense, and is totally >useless for 99.9999% of everything. I still don't see the point of it, >except for samba. Ugly. > > Linus > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds ` (5 preceding siblings ...) 2004-08-26 8:43 ` Hans Reiser @ 2004-08-26 13:53 ` Michael Halcrow 2004-08-26 22:26 ` Hans Reiser 2004-08-29 14:20 ` Alan Cox 7 siblings, 1 reply; 1835+ messages in thread From: Michael Halcrow @ 2004-08-26 13:53 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 5340 bytes --] If I may chime in here... On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > haven't seen a design from you on -fsdevel how you try to solve > > the problems with files as directories. > > Hey, files-as-directories are one of my pet things, so I have to > side with Hans on this one. I think it just makes sense. A hell of a > lot more sense than xattrs, anyway, since it allows scripts etc > standard tools to touch the attributes. > > It's the UNIX way. This is an issue that directly affects work I am doing in extended cryptfs: http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55 http://halcrow.us/~mhalcrow/ols2004.pdf http://halcrow.us/~mhalcrow/ols_cryptfs.sxi The basic idea is that the cryptographic context for every file is correlated with the individual file via xattr's. A file is a unit of data that should, as it stands, contain all the information requisite for the encrypting filesystem layer to transparently decrypt (and encrypt, when the file is written to). This allows for a key->file granularity, as opposed to a key->block device (dm-crypt) or a key->mount point (CFS) granularity. My grand vision is to have a policy that determines whether or not the encrypted version of the file or the decrypted version of the file is read, dependent on whether or not the file is leaving the security domain (the storage device under the control of the currently running kernel). For example, if the ``cp'' command is copying a file from a filesystem mounted from /dev/hda1 to a filesystem mounted from /dev/fd0, then the policy would indicate that (unless otherwise noted in the .cryptfsrc file in the root of the filesystem mounted from /dev/fd0, which might also contain the default security context for that filesystem or directory - like whose public keys should be used to encrypt the symmetric key for data) the file is leaving the security domain, and the encrypted contents of the file should be given to cp. Same with mutt reading an email attachment (as opposed to, say, .muttrc, where, more likely than not, the unencrypted version is wanted). The goal is to enable an ``encrypted by default'' policy, in which files on the storage devices are independent encrypted units that remain encrypted until an application that actually needs to see the decrypted contents opens them. Then the encryption and decryption is done transparently by the fs layer, as long as the user has the right keys. Extended attributes seem like a natural way to store this context. Once you consider that you can have a crypto context for each file, you can start doing other neat tricks, like keyed hashes over extents within the file, to allow for dynamic integrity verification during the read. If an offset of 1.5 gigabytes into a 2-gigabyte has been tampered with, then that tampering will be caught when that portion of the file is read; you don't have to verify the hash of the entire 2-gigabyte file at the time of the open. Of course, this would very rapidly overrun the available xattr storage size. And so to realistically implement something like this, some new underlying file format is in order. In any case, the issue of userspace applications supporting extended attributes is key to the viability of this approach. If cp, uuencode, tar, or what not do not preserve the extended attributes, then the crypto context is lost, and the file is unreadable. So the $64,000 question is, just how committed is the community to this whole concept of extended attributes? From this point, should I assume that good xattr support is forthcoming, or should I abandon the idea of using xattr's for this altogether? One solution I've been kicking around is to make cryptfs GnuPG-compatible. Not only would this eliminate the need to store some of the crypto context in the xattr set, but it would also preserve the crypto context with apps that don't know about xattr's, and it would be possible for users who are not running cryptfs to read the files with gpg. Keyed hashes over extents would be doable if GnuPG allowed for opaque data blobs in the file that gpg would just ignore when decrypting the file (gnupg-dev list had technical issues last time I tried to post these ideas to it - any gpg guys around that can comment on this?). > I never liked the xattr stuff. It makes little sense, and is totally > useless for 99.9999% of everything. I still don't see the point of it, > except for samba. Ugly. If xattr's wind up getting supported by a certain critical mass of applications, then they are somewhat useful for me, although, as currently implemented, are insufficient for what I really need (keyed hashes over extents require too much space). BTW, early this week I migrated cryptfs over to use David Howell's new keyring, which is working out nicely. Mike .___________________________________________________________________. Michael A. Halcrow Security Software Engineer, IBM Linux Technology Center GnuPG Fingerprint: 05B5 08A8 713A 64C1 D35D 2371 2D3C FDDA 3EB6 601D [-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:53 ` Michael Halcrow @ 2004-08-26 22:26 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 22:26 UTC (permalink / raw) To: Michael Halcrow Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Michael Halcrow wrote: > This is an issue that directly affects work I am doing in extended > >cryptfs: > >http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55 >http://halcrow.us/~mhalcrow/ols2004.pdf >http://halcrow.us/~mhalcrow/ols_cryptfs.sxi > > > Reiser4 has an encryption plugin that will ship sometime this year. You might want to talk to edward@namesys.com about it. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 22:26 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 22:26 UTC (permalink / raw) To: Michael Halcrow Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Michael Halcrow wrote: > This is an issue that directly affects work I am doing in extended > >cryptfs: > >http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55 >http://halcrow.us/~mhalcrow/ols2004.pdf >http://halcrow.us/~mhalcrow/ols_cryptfs.sxi > > > Reiser4 has an encryption plugin that will ship sometime this year. You might want to talk to edward@namesys.com about it. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds ` (6 preceding siblings ...) 2004-08-26 13:53 ` Michael Halcrow @ 2004-08-29 14:20 ` Alan Cox 7 siblings, 0 replies; 1835+ messages in thread From: Alan Cox @ 2004-08-29 14:20 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mer, 2004-08-25 at 21:22, Linus Torvalds wrote: > - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) > - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) > > Problem solved. Very user-friendly, and very intuitive. Breaks bash and almost all other file completion 8) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:08 ` Christoph Hellwig 2004-08-25 20:19 ` Christoph Hellwig 2004-08-25 20:22 ` Linus Torvalds @ 2004-08-25 20:35 ` Alex Zarochentsev 2004-08-25 20:51 ` Christoph Hellwig 2 siblings, 1 reply; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-25 20:35 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Wed, Aug 25, 2004 at 10:08:59PM +0200, Christoph Hellwig wrote: > On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote: > > You ignored everything I said during the discussion of xattrs about how > > there is no need to have attributes when you can just have files and > > directories, and that xattrs reflected a complete ignorance of name > > space design principles. > > Actually in most of the discussion you simply didn't participate. While > xattrs might not be the nicest interface they have the advantag of not > breaking the SuS assumption of what directories vs files are, and they > do not break the Linux O_DIRECTORY semantics that are defined and need > to solve real-world races either. Reiser4 may have a mount option for whoose who like or have to use traditional O_DIRECTORY semantics. There would be no /metas under non-directories, if user wants that. [ ... ] -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:35 ` Alex Zarochentsev @ 2004-08-25 20:51 ` Christoph Hellwig 2004-08-25 23:54 ` Jamie Lokier 2004-08-26 8:43 ` Hans Reiser 0 siblings, 2 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:51 UTC (permalink / raw) To: Alex Zarochentsev Cc: Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote: > Reiser4 may have a mount option for whoose who like or have to use traditional > O_DIRECTORY semantics. There would be no /metas under non-directories, if user > wants that. Again, O_DIRECTORY was added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:51 ` Christoph Hellwig @ 2004-08-25 23:54 ` Jamie Lokier 2004-08-26 8:44 ` Hans Reiser 2004-08-26 12:43 ` Christoph Hellwig 2004-08-26 8:43 ` Hans Reiser 1 sibling, 2 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-25 23:54 UTC (permalink / raw) To: Christoph Hellwig, Alex Zarochentsev, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > > Reiser4 may have a mount option for whoose who like or have to use > > traditional O_DIRECTORY semantics. There would be no /metas under > > non-directories, if user wants that. > > Again, O_DIRECTORY was added to solve a real-world race, not just for > the sake of it. If you really want to add that option I'll research the > CAN number for you so you can named it after that - or just call it -o > insecure directly. man open(2) explains that O_DIRECTORY is used by opendir() to prevent blocking when opening pipes and certain devices*, and should only by used by opendir (of course it isn't only used by opendir, as it's a handy optimisation). In fact O_DIRECTORY is also used by Glibc to optimise away stat() before and fstat() after calls. An O_NODEVICE flag would be equally secure, and more generally useful. It's important that device nodes cannot be opened when O_DIRECTORY is set. This is compatible with reiser4 file-as-directory semantics, but I don't know if reiser4 actually implements this. If it does (and it should) then there is no device blocking problem. That leaves only the optimisation of fstat() in opendir(). But that begs the question: do we want opendir() to succeed on a reiser4 file? It's up to us to decide if we like that semantic, or prefer a different one such as the one I described before (path search enters file-as-directory, but opendir() directly on it fails), or the Sun syscalls someone mentioned. -- Jamie * Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on many systems. It's best if you can avoid opening devices at all though. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:54 ` Jamie Lokier @ 2004-08-26 8:44 ` Hans Reiser 2004-08-26 12:43 ` Christoph Hellwig 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:44 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Jamie Lokier wrote: >Christoph Hellwig wrote: > > >>>Reiser4 may have a mount option for whoose who like or have to use >>>traditional O_DIRECTORY semantics. There would be no /metas under >>>non-directories, if user wants that. >>> >>> >>Again, O_DIRECTORY was added to solve a real-world race, not just for >>the sake of it. If you really want to add that option I'll research the >>CAN number for you so you can named it after that - or just call it -o >>insecure directly. >> >> > >man open(2) explains that O_DIRECTORY is used by opendir() to prevent >blocking when opening pipes and certain devices*, and should only by >used by opendir (of course it isn't only used by opendir, as it's a >handy optimisation). > >In fact O_DIRECTORY is also used by Glibc to optimise away stat() >before and fstat() after calls. > >An O_NODEVICE flag would be equally secure, and more generally useful. > >It's important that device nodes cannot be opened when O_DIRECTORY is >set. This is compatible with reiser4 file-as-directory semantics, but >I don't know if reiser4 actually implements this. If it does (and it >should) then there is no device blocking problem. > >That leaves only the optimisation of fstat() in opendir(). > >But that begs the question: do we want opendir() to succeed on a reiser4 file? > > Yes, we do. Why not? Not is more complex..... >It's up to us to decide if we like that semantic, or prefer a >different one such as the one I described before (path search enters >file-as-directory, but opendir() directly on it fails), or the Sun >syscalls someone mentioned. > >-- Jamie > > >* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on >many systems. It's best if you can avoid opening devices at all though. > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:44 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:44 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Jamie Lokier wrote: >Christoph Hellwig wrote: > > >>>Reiser4 may have a mount option for whoose who like or have to use >>>traditional O_DIRECTORY semantics. There would be no /metas under >>>non-directories, if user wants that. >>> >>> >>Again, O_DIRECTORY was added to solve a real-world race, not just for >>the sake of it. If you really want to add that option I'll research the >>CAN number for you so you can named it after that - or just call it -o >>insecure directly. >> >> > >man open(2) explains that O_DIRECTORY is used by opendir() to prevent >blocking when opening pipes and certain devices*, and should only by >used by opendir (of course it isn't only used by opendir, as it's a >handy optimisation). > >In fact O_DIRECTORY is also used by Glibc to optimise away stat() >before and fstat() after calls. > >An O_NODEVICE flag would be equally secure, and more generally useful. > >It's important that device nodes cannot be opened when O_DIRECTORY is >set. This is compatible with reiser4 file-as-directory semantics, but >I don't know if reiser4 actually implements this. If it does (and it >should) then there is no device blocking problem. > >That leaves only the optimisation of fstat() in opendir(). > >But that begs the question: do we want opendir() to succeed on a reiser4 file? > > Yes, we do. Why not? Not is more complex..... >It's up to us to decide if we like that semantic, or prefer a >different one such as the one I described before (path search enters >file-as-directory, but opendir() directly on it fails), or the Sun >syscalls someone mentioned. > >-- Jamie > > >* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on >many systems. It's best if you can avoid opening devices at all though. > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:54 ` Jamie Lokier 2004-08-26 8:44 ` Hans Reiser @ 2004-08-26 12:43 ` Christoph Hellwig 1 sibling, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:43 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Alex Zarochentsev, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 12:54:09AM +0100, Jamie Lokier wrote: > Christoph Hellwig wrote: > > > Reiser4 may have a mount option for whoose who like or have to use > > > traditional O_DIRECTORY semantics. There would be no /metas under > > > non-directories, if user wants that. > > > > Again, O_DIRECTORY was added to solve a real-world race, not just for > > the sake of it. If you really want to add that option I'll research the > > CAN number for you so you can named it after that - or just call it -o > > insecure directly. > > man open(2) explains that O_DIRECTORY is used by opendir() to prevent > blocking when opening pipes and certain devices*, and should only by > used by opendir (of course it isn't only used by opendir, as it's a > handy optimisation). > > In fact O_DIRECTORY is also used by Glibc to optimise away stat() > before and fstat() after calls. > > An O_NODEVICE flag would be equally secure, and more generally useful. It would prevent the open device issue but you'd need another syscall. Anyways, this is an existing API and chaging it now would suddenly making everything that relies on it for that purpose insecure, so it's completely out of option. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:51 ` Christoph Hellwig @ 2004-08-26 8:43 ` Hans Reiser 2004-08-26 8:43 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: Christoph Hellwig Cc: Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: >On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote: > > >>Reiser4 may have a mount option for whoose who like or have to use traditional >>O_DIRECTORY semantics. There would be no /metas under non-directories, if user >>wants that. >> >> > >Again, O_DIRECTORY was added to solve a real-world race, not just for >the sake of it. > Can you supply more details and we will try to reply concretely? Thanks. > If you really want to add that option I'll research the >CAN number for you so you can named it after that - or just call it -o >insecure directly. > > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:43 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: Christoph Hellwig Cc: Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: >On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote: > > >>Reiser4 may have a mount option for whoose who like or have to use traditional >>O_DIRECTORY semantics. There would be no /metas under non-directories, if user >>wants that. >> >> > >Again, O_DIRECTORY was added to solve a real-world race, not just for >the sake of it. > Can you supply more details and we will try to reply concretely? Thanks. > If you really want to add that option I'll research the >CAN number for you so you can named it after that - or just call it -o >insecure directly. > > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:43 ` Hans Reiser (?) @ 2004-08-26 12:41 ` Christoph Hellwig 2004-08-26 13:48 ` Jamie Lokier -1 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:41 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 01:43:59AM -0700, Hans Reiser wrote: > >Again, O_DIRECTORY was added to solve a real-world race, not just for > >the sake of it. > > > Can you supply more details and we will try to reply concretely? Thanks. It's really userland programming 101 :) you need to fstat after open to really check that no one swtiched the path below you to a fifo or device, O_DIRECTORY gets that directly in the open call, with the additional benefit of making sure you don't get any of the sideeffects at all that would happen if you're opening a device file. the current reiser4 semantics break that and as soon as you're having a world-writeable (e.g. /tmp) dir on it and someone is doing an opendir on it he's lost. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:41 ` Christoph Hellwig @ 2004-08-26 13:48 ` Jamie Lokier 2004-08-26 15:57 ` Christoph Hellwig 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 13:48 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > you need to fstat after open to really check that no one swtiched the > path below you to a fifo or device, O_DIRECTORY gets that directly in > the open call, with the additional benefit of making sure you don't > get any of the sideeffects at all that would happen if you're opening > a device file. > > the current reiser4 semantics break that and as soon as you're having a > world-writeable (e.g. /tmp) dir on it and someone is doing an opendir > on it he's lost. How does the current reiser4 semantics break that? In a reiser4 filesystem, a file _is_ a directory. opendir() is supposed to succeed on it. There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics? -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:48 ` Jamie Lokier @ 2004-08-26 15:57 ` Christoph Hellwig 2004-08-26 16:06 ` Jamie Lokier 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 15:57 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 02:48:12PM +0100, Jamie Lokier wrote: > > the current reiser4 semantics break that and as soon as you're having a > > world-writeable (e.g. /tmp) dir on it and someone is doing an opendir > > on it he's lost. > > How does the current reiser4 semantics break that? > > In a reiser4 filesystem, a file _is_ a directory. > opendir() is supposed to succeed on it. > > There's bound to be some security issue, but I'm not sure what you're > getting at with /tmp. What sort of sort of security problem arises > with a world-writeable directory such as /tmp, that cannot arise with > the standard fs semantics? Actually you are right on that issue because it would open the device/fifo as directory and not device/fifo (in fact I'd had to look at the code again to see whether they actually do this only for files or also for special files) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:57 ` Christoph Hellwig @ 2004-08-26 16:06 ` Jamie Lokier 2004-08-26 16:13 ` Christoph Hellwig 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 16:06 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > > There's bound to be some security issue, but I'm not sure what you're > > getting at with /tmp. What sort of sort of security problem arises > > with a world-writeable directory such as /tmp, that cannot arise with > > the standard fs semantics? > > Actually you are right on that issue because it would open the > device/fifo as directory and not device/fifo (in fact I'd had to look at > the code again to see whether they actually do this only for files or > also for special files) Are you saying that with reiser4, you can open a device or fifo with O_DIRECTORY? -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:06 ` Jamie Lokier @ 2004-08-26 16:13 ` Christoph Hellwig 2004-08-26 16:35 ` Linus Torvalds 2004-08-29 18:43 ` Alex Zarochentsev 0 siblings, 2 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:13 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 05:06:38PM +0100, Jamie Lokier wrote: > Christoph Hellwig wrote: > > > There's bound to be some security issue, but I'm not sure what you're > > > getting at with /tmp. What sort of sort of security problem arises > > > with a world-writeable directory such as /tmp, that cannot arise with > > > the standard fs semantics? > > > > Actually you are right on that issue because it would open the > > device/fifo as directory and not device/fifo (in fact I'd had to look at > > the code again to see whether they actually do this only for files or > > also for special files) > > Are you saying that with reiser4, you can open a device or fifo with > O_DIRECTORY? That's what I thought, but as far as I can follow the code this is not actually true. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:13 ` Christoph Hellwig @ 2004-08-26 16:35 ` Linus Torvalds 2004-08-26 17:20 ` Jamie Lokier 2004-08-27 9:18 ` Giuliano Pochini 2004-08-29 18:43 ` Alex Zarochentsev 1 sibling, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 16:35 UTC (permalink / raw) To: Christoph Hellwig Cc: Jamie Lokier, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Christoph Hellwig wrote: > > > Are you saying that with reiser4, you can open a device or fifo with > > O_DIRECTORY? > > That's what I thought, but as far as I can follow the code this is not > actually true. It should be possible to do, though. There's nothing really different in making the "default" (unnamed) fork be a special device or a fifo. And it would be perfectly ok for O_DIRECTORY to open such a file, as long as it opens the directory branch, not the special device. I advocated (long ago) something like this for /dev handling, just because I think it would make sense to have /dev/hda <- special file /dev/hda/part1 <- partition 1 (aka /dev/hda1) which just seems like a very obvious and intuitive interface to me. Of course, we have so much legacy in /dev that there's no real point to doing this, but it's still an appealing approach, I think. But I do take Al's concerns seriously. I like the notion of supporting "containers", but there are undoubtedly serious issues in the notion. I don't strictly know exactly _how_ to implement it sanely (I can talk about using the vfsmnt structure all I like, but the fact is, it's a different thing from a normal mount, and there may be serious problems indeed there). Still, I really do like the idea of merging the notion of file and directory into one notion of "container". I absolutely _detest_ files with internal structure that tools have to know about (ie I hate seeing all those embedded formats that I can't use "grep" on - MIME being one case). I'd much rather see a "group of files" and a "file with a grouping of information". (Now, flattening that "group of files" is obviously needed for serial protocols, so I think MIME/tar/xxxx are fine for _transporting_ data, but I'm saying that outside of transport I really prefer a "collection of files" approach). Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:35 ` Linus Torvalds @ 2004-08-26 17:20 ` Jamie Lokier 2004-08-26 17:29 ` Linus Torvalds 2004-08-27 9:18 ` Giuliano Pochini 1 sibling, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 17:20 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > And it would be perfectly ok for O_DIRECTORY to open such a file, as long > as it opens the directory branch, not the special device. What about: cd /dev/into_directory_branch/ run_setuid_program -> calls pwd pwd opens("."), (".."), ("../..") etc. -> the setuid program thus ends up opening a device or fifo, when it does pwd's path walk. Yes it could use the getcwd syscall, but some programs do their own path walk. > I advocated (long ago) something like this for /dev handling, just because > I think it would make sense to have > > /dev/hda <- special file > /dev/hda/part1 <- partition 1 (aka /dev/hda1) > > which just seems like a very obvious and intuitive interface to me. Of > course, we have so much legacy in /dev that there's no real point to doing > this, but it's still an appealing approach, I think. It also fits the container idea very well: /dev/hda/part1/ <- the filesystem inside partition 1 That's not a contrivance, it's what I'd expect to happen if /dev/hda is an ordinary file containing a disk image: image.bin <- hard disk image (a regular file) image.bin/part1 <- partition 1 image.bin/part1/ <- the filesystem inside partition 1 That's assuming there's a format handler which recognises that image. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:20 ` Jamie Lokier @ 2004-08-26 17:29 ` Linus Torvalds 2004-08-26 17:54 ` viro ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 17:29 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Jamie Lokier wrote: > > run_setuid_program > > -> calls pwd > pwd opens("."), (".."), ("../..") etc. Ehh.. Not only does pwd not do that.. (hint: there's a getcwd() system call) > -> the setuid program thus ends up opening a device or fifo, > when it does pwd's path walk. Yes it could use the getcwd > syscall, but some programs do their own path walk. .. but even if it did that, it should use O_DIRECTORY when it did so. If it doesn't, it's broken. So no, it would _not_ open the device or fifo when it did so. The fact is, anything that expects to open a directory should already be opening it with O_DIRECTORY. That said, ".." and "." are special already inside the kernel, and it migth be worth making them automatically imply O_DIRECTORY, since nothing else makes sense anyway. That would fix the case where somebody uses ".." _without_ using O_DIRECTORY. > It also fits the container idea very well: > > /dev/hda/part1/ <- the filesystem inside partition 1 I don't think you can do that. The kernel has no idea how to mount the filesystem. If it's already mounted somewhere else, that's a different issue. Although it might be mounted in several places (as a bind mount) with different writability, I guess, so even then it might be "interesting". Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:29 ` Linus Torvalds @ 2004-08-26 17:54 ` viro 2004-08-26 18:16 ` Jamie Lokier 2004-08-30 13:25 ` Alex Zarochentsev 2 siblings, 0 replies; 1835+ messages in thread From: viro @ 2004-08-26 17:54 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 10:29:45AM -0700, Linus Torvalds wrote: > > /dev/hda/part1/ <- the filesystem inside partition 1 > > I don't think you can do that. The kernel has no idea how to mount the > filesystem. ... and what's more, fundamental thing for mount is fs type, what to do with device(s) (if any) and any other arguments is the filesystem's business. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:29 ` Linus Torvalds 2004-08-26 17:54 ` viro @ 2004-08-26 18:16 ` Jamie Lokier 2004-08-26 18:27 ` Linus Torvalds 2004-08-30 13:25 ` Alex Zarochentsev 2 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 18:16 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > > -> the setuid program thus ends up opening a device or fifo, > > when it does pwd's path walk. Yes it could use the getcwd > > syscall, but some programs do their own path walk. > > .. but even if it did that, it should use O_DIRECTORY when it did so. If > it doesn't, it's broken. Didn't someone just say that O_DIRECTORY will succeed on a device, precisely because opendir() is supposed to succeed on the device? I'm not sure. Let me refresh my memory. Linus Torvalds wrote: > And it would be perfectly ok for O_DIRECTORY to open such a file, as long > as it opens the directory branch, not the special device. Oh, I see. You're right. ".." sees the path of directory branches. > > It also fits the container idea very well: > > > > /dev/hda/part1/ <- the filesystem inside partition 1 > > I don't think you can do that. The kernel has no idea how to mount the > filesystem. It is not the kernel which decides. The filesystem containing /dev/hda/part1 opens "the directory branch". If the filesystem is suitably configured, it's exactly the same as opening a .zip or ELF file: it asks userspace for help. Userspace helps by either mounting it, or creating a container virtual directory -- in exactly the same way it helps with .zip and ELF files by creating a container virtual directory for those. It would mount it if the kernel supports the fs, and create a virtual directory if the kernel doesn't support the fs and userspace does. This is exactly the same as when you cd into a filesystem image: the helper mounts (this time with a lookback file mount) if the kernel supports the image fs, otherwise the helper may be able to decode it in userspace. By the way, these mechanisms could allow some of the old filesystems that nobody uses any more to be removed from the kernel altogether and kept in an archive of userspace-only supported filesystems. What do you think of that idea? > If it's already mounted somewhere else, that's a different issue. > Although it might be mounted in several places (as a bind mount) with > different writability, I guess, so even then it might be "interesting". The obvious implementation has the userspace helper just mounting it, end of story. If the mount command fails, it fails. Much like autofs. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:16 ` Jamie Lokier @ 2004-08-26 18:27 ` Linus Torvalds 2004-08-26 19:40 ` Jamie Lokier 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 18:27 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Jamie Lokier wrote: > > > .. but even if it did that, it should use O_DIRECTORY when it did so. If > > it doesn't, it's broken. > > Didn't someone just say that O_DIRECTORY will succeed on a device, > precisely because opendir() is supposed to succeed on the device? It will succeed on the _name_ of the device, but it won't open the device node. It would open the _directory_ node associated with that name. Think of it this way: a pathname always points to one "container". That container is either a directory or a single node - and O_DIRECTORY ends up being the thing that chooses between them. > > I don't think you can do that. The kernel has no idea how to mount the > > filesystem. > > It is not the kernel which decides. The filesystem containing > /dev/hda/part1 opens "the directory branch". But that filesystem cannot know what the _other_ filesystem configurations are. And that's what you'd have to have to mount. > The obvious implementation has the userspace helper just mounting it, > end of story. If the mount command fails, it fails. Much like autofs. Yes, that would work, but it's of questionable use. If you want autofs, then just _use_ autofs. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:27 ` Linus Torvalds @ 2004-08-26 19:40 ` Jamie Lokier 0 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 19:40 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > > It is not the kernel which decides. The filesystem containing > > /dev/hda/part1 opens "the directory branch". > > But that filesystem cannot know what the _other_ filesystem configurations > are. And that's what you'd have to have to mount. At which point, userspace comes in. Who knows what userspace chooses. > > The obvious implementation has the userspace helper just mounting it, > > end of story. If the mount command fails, it fails. Much like autofs. > > Yes, that would work, but it's of questionable use. If you want autofs, > then just _use_ autofs. I don't mean to imply it's useful behaviour: only that it seems to happen by default if you don't do anything special to prevent it, if you see what I mean. It does seem useful to have _files_ mounted automagically (using loopback devices): for example, cd into a .iso file and it would be nice to see the contents. At this point, deciding on the preferred -EBUSY or not behaviour becomes relevant, as does deciding whether the .iso in this example is viewed using the kernel's isofs code, or using a userspace helper in the same way as we'd use one to browse a .zip file. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:29 ` Linus Torvalds 2004-08-26 17:54 ` viro 2004-08-26 18:16 ` Jamie Lokier @ 2004-08-30 13:25 ` Alex Zarochentsev 2 siblings, 0 replies; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-30 13:25 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 10:29:45AM -0700, Linus Torvalds wrote: > > > On Thu, 26 Aug 2004, Jamie Lokier wrote: > > > > run_setuid_program > > > > -> calls pwd > > pwd opens("."), (".."), ("../..") etc. > > Ehh.. Not only does pwd not do that.. > > (hint: there's a getcwd() system call) > > > -> the setuid program thus ends up opening a device or fifo, > > when it does pwd's path walk. Yes it could use the getcwd > > syscall, but some programs do their own path walk. > > .. but even if it did that, it should use O_DIRECTORY when it did so. If > it doesn't, it's broken. > > So no, it would _not_ open the device or fifo when it did so. > > The fact is, anything that expects to open a directory should already be > opening it with O_DIRECTORY. reiser4 files-as-dirs do not depend on that. A file behaves as directory if one calls i_op->lookup() or f_op->readdir(). I am not sure that O_DIRECTORY should be a switch. > That said, ".." and "." are special already inside the kernel, and it > migth be worth making them automatically imply O_DIRECTORY, since nothing > else makes sense anyway. That would fix the case where somebody uses ".." > _without_ using O_DIRECTORY. > > > It also fits the container idea very well: > > > > /dev/hda/part1/ <- the filesystem inside partition 1 > > I don't think you can do that. The kernel has no idea how to mount the > filesystem. > > If it's already mounted somewhere else, that's a different issue. > Although it might be mounted in several places (as a bind mount) with > different writability, I guess, so even then it might be "interesting". > > Linus -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:35 ` Linus Torvalds 2004-08-26 17:20 ` Jamie Lokier @ 2004-08-27 9:18 ` Giuliano Pochini 2004-08-27 10:33 ` Giuseppe Bilotta 2004-08-30 14:10 ` Alex Zarochentsev 1 sibling, 2 replies; 1835+ messages in thread From: Giuliano Pochini @ 2004-08-27 9:18 UTC (permalink / raw) To: Linus Torvalds Cc: ReiserFS List, linux-kernel, linux-fsdevel, Hans Reiser, Jamie Lokier, Christoph Hellwig On 26-Aug-2004 Linus Torvalds wrote: > I advocated (long ago) something like this for /dev handling, just because > I think it would make sense to have > > /dev/hda <- special file > /dev/hda/part1 <- partition 1 (aka /dev/hda1) This breaks the r4 semantics if I understood it correctly. Because /partN are not simply associated to the file, they are part of the file. ie. when I modify /dev/hda I also change /dev/hda/partN and vice-versa. I don't see any pratical problem, though. > Still, I really do like the idea of merging the notion of file and > directory into one notion of "container". I absolutely _detest_ files with > internal structure that tools have to know about (ie I hate seeing all > those embedded formats that I can't use "grep" on - MIME being one case). > I'd much rather see a "group of files" and a "file with a grouping of > information". You're actually looking for a database with a legacy fs-like interface :) -- Giuliano. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 9:18 ` Giuliano Pochini @ 2004-08-27 10:33 ` Giuseppe Bilotta 2004-08-30 14:10 ` Alex Zarochentsev 1 sibling, 0 replies; 1835+ messages in thread From: Giuseppe Bilotta @ 2004-08-27 10:33 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel Giuliano Pochini wrote: > On 26-Aug-2004 Linus Torvalds wrote: > > > I advocated (long ago) something like this for /dev handling, just because > > I think it would make sense to have > > > > /dev/hda <- special file > > /dev/hda/part1 <- partition 1 (aka /dev/hda1) > > This breaks the r4 semantics if I understood it correctly. Because > /partN are not simply associated to the file, they are part of the > file. ie. when I modify /dev/hda I also change /dev/hda/partN and > vice-versa. I don't see any pratical problem, though. > > > > Still, I really do like the idea of merging the notion of file and > > directory into one notion of "container". I absolutely _detest_ files with > > internal structure that tools have to know about (ie I hate seeing all > > those embedded formats that I can't use "grep" on - MIME being one case). > > I'd much rather see a "group of files" and a "file with a grouping of > > information". > > You're actually looking for a database with a legacy fs-like interface :) Hm. Isn't this exactly what MS has in store for Longhorn? -- Giuseppe "Oblomov" Bilotta Can't you see It all makes perfect sense Expressed in dollar and cents Pounds shillings and pence (Roger Waters) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 10:33 ` Giuseppe Bilotta 0 siblings, 0 replies; 1835+ messages in thread From: Giuseppe Bilotta @ 2004-08-27 10:33 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel Giuliano Pochini wrote: > On 26-Aug-2004 Linus Torvalds wrote: > > > I advocated (long ago) something like this for /dev handling, just because > > I think it would make sense to have > > > > /dev/hda <- special file > > /dev/hda/part1 <- partition 1 (aka /dev/hda1) > > This breaks the r4 semantics if I understood it correctly. Because > /partN are not simply associated to the file, they are part of the > file. ie. when I modify /dev/hda I also change /dev/hda/partN and > vice-versa. I don't see any pratical problem, though. > > > > Still, I really do like the idea of merging the notion of file and > > directory into one notion of "container". I absolutely _detest_ files with > > internal structure that tools have to know about (ie I hate seeing all > > those embedded formats that I can't use "grep" on - MIME being one case). > > I'd much rather see a "group of files" and a "file with a grouping of > > information". > > You're actually looking for a database with a legacy fs-like interface :) Hm. Isn't this exactly what MS has in store for Longhorn? -- Giuseppe "Oblomov" Bilotta Can't you see It all makes perfect sense Expressed in dollar and cents Pounds shillings and pence (Roger Waters) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 10:33 ` Giuseppe Bilotta @ 2004-08-27 18:48 ` Hubert Chan -1 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-27 18:48 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel >>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes: Giuseppe> Giuliano Pochini wrote: [...] >> You're actually looking for a database with a legacy fs-like >> interface :) Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn? No. To access the database-like functionality in WinFS, you need to use a special API. Hans as been planning this for a long time before WinFS was ever announced. See http://www.namesys.com/whitepaper.html -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 18:48 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-27 18:48 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel >>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes: Giuseppe> Giuliano Pochini wrote: [...] >> You're actually looking for a database with a legacy fs-like >> interface :) Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn? No. To access the database-like functionality in WinFS, you need to use a special API. Hans as been planning this for a long time before WinFS was ever announced. See http://www.namesys.com/whitepaper.html -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:48 ` Hubert Chan @ 2004-08-28 16:33 ` Giuseppe Bilotta -1 siblings, 0 replies; 1835+ messages in thread From: Giuseppe Bilotta @ 2004-08-28 16:33 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel Hubert Chan wrote: > >>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes: > > Giuseppe> Giuliano Pochini wrote: > > [...] > > >> You're actually looking for a database with a legacy fs-like > >> interface :) > > Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn? > > No. To access the database-like functionality in WinFS, you need to use > a special API. I see, thanks. Might be interesting to know what they plan to make accessible of WinFS to legacy apps via the old API, then (especially since, at according to a /. article, they're planning to retrofit WinFS to WinXP ...) -- Giuseppe "Oblomov" Bilotta Can't you see It all makes perfect sense Expressed in dollar and cents Pounds shillings and pence (Roger Waters) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 16:33 ` Giuseppe Bilotta 0 siblings, 0 replies; 1835+ messages in thread From: Giuseppe Bilotta @ 2004-08-28 16:33 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel Hubert Chan wrote: > >>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes: > > Giuseppe> Giuliano Pochini wrote: > > [...] > > >> You're actually looking for a database with a legacy fs-like > >> interface :) > > Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn? > > No. To access the database-like functionality in WinFS, you need to use > a special API. I see, thanks. Might be interesting to know what they plan to make accessible of WinFS to legacy apps via the old API, then (especially since, at according to a /. article, they're planning to retrofit WinFS to WinXP ...) -- Giuseppe "Oblomov" Bilotta Can't you see It all makes perfect sense Expressed in dollar and cents Pounds shillings and pence (Roger Waters) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 16:33 ` Giuseppe Bilotta @ 2004-08-28 17:27 ` Hubert Chan -1 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-28 17:27 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel >>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes: Giuseppe> Hubert Chan wrote: >> No. To access the database-like functionality in WinFS, you need to >> use a special API. Giuseppe> I see, thanks. Might be interesting to know what they plan to Giuseppe> make accessible of WinFS to legacy apps via the old API, then Giuseppe> (especially since, at according to a /. article, they're Giuseppe> planning to retrofit WinFS to WinXP ...) AFAIK, all files are still accessible through normal NTFS. You just need the special API to do querying and such. So old apps can access all the files normally. They just can't take advantage of all the new whizbang features. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 17:27 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-28 17:27 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel >>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes: Giuseppe> Hubert Chan wrote: >> No. To access the database-like functionality in WinFS, you need to >> use a special API. Giuseppe> I see, thanks. Might be interesting to know what they plan to Giuseppe> make accessible of WinFS to legacy apps via the old API, then Giuseppe> (especially since, at according to a /. article, they're Giuseppe> planning to retrofit WinFS to WinXP ...) AFAIK, all files are still accessible through normal NTFS. You just need the special API to do querying and such. So old apps can access all the files normally. They just can't take advantage of all the new whizbang features. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 9:18 ` Giuliano Pochini 2004-08-27 10:33 ` Giuseppe Bilotta @ 2004-08-30 14:10 ` Alex Zarochentsev 1 sibling, 0 replies; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-30 14:10 UTC (permalink / raw) To: Giuliano Pochini Cc: Linus Torvalds, ReiserFS List, linux-kernel, linux-fsdevel, Hans Reiser, Jamie Lokier, Christoph Hellwig On Fri, Aug 27, 2004 at 11:18:55AM +0200, Giuliano Pochini wrote: > > On 26-Aug-2004 Linus Torvalds wrote: > > > I advocated (long ago) something like this for /dev handling, just because > > I think it would make sense to have > > > > /dev/hda <- special file > > /dev/hda/part1 <- partition 1 (aka /dev/hda1) > > This breaks the r4 semantics if I understood it correctly. Reiser4 currently links /metas/ directory under each reiser4 fs object. Reiser4 "silently" allows regular file to be a directory for that. That is only an example how an fs object may link its internal hierarchical structure to the fs tree. /dev/hda/-as-a-dir is another example, but, I am afraid, it is not a reiser4 business to look into the devices. Devfs or its successor may do it. > Because > /partN are not simply associated to the file, they are part of the > file. ie. when I modify /dev/hda I also change /dev/hda/partN and > vice-versa. I don't see any pratical problem, though. > > > > Still, I really do like the idea of merging the notion of file and > > directory into one notion of "container". I absolutely _detest_ files with > > internal structure that tools have to know about (ie I hate seeing all > > those embedded formats that I can't use "grep" on - MIME being one case). > > I'd much rather see a "group of files" and a "file with a grouping of > > information". > > You're actually looking for a database with a legacy fs-like interface :) > > > -- > Giuliano. -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:13 ` Christoph Hellwig 2004-08-26 16:35 ` Linus Torvalds @ 2004-08-29 18:43 ` Alex Zarochentsev 1 sibling, 0 replies; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-29 18:43 UTC (permalink / raw) To: Christoph Hellwig, Jamie Lokier, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 06:13:03PM +0200, Christoph Hellwig wrote: > On Thu, Aug 26, 2004 at 05:06:38PM +0100, Jamie Lokier wrote: > > Christoph Hellwig wrote: > > > > There's bound to be some security issue, but I'm not sure what you're > > > > getting at with /tmp. What sort of sort of security problem arises > > > > with a world-writeable directory such as /tmp, that cannot arise with > > > > the standard fs semantics? > > > > > > Actually you are right on that issue because it would open the > > > device/fifo as directory and not device/fifo (in fact I'd had to look at > > > the code again to see whether they actually do this only for files or > > > also for special files) > > > > Are you saying that with reiser4, you can open a device or fifo with > > O_DIRECTORY? > > That's what I thought, but as far as I can follow the code this is not > actually true. All reiser4 inodes have i_ops->lookup != NULL, so open(..., O_DIRECTORY) would succeed on them (thanks Nikita for reminding me that). It may be better to pass the control of that to ->i_op->permission() through explicit MAY_LOOKUP flag. It can be possible to eliminate the race by some more complex logic (strict ownership check, for example) in ->permission(). -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser ` (2 preceding siblings ...) (?) @ 2004-08-25 20:20 ` Jeremy Allison 2004-08-26 8:42 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Jeremy Allison @ 2004-08-25 20:20 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote: > > You ignored everything I said during the discussion of xattrs about how > there is no need to have attributes when you can just have files and > directories, and that xattrs reflected a complete ignorance of name > space design principles. When I said we should just add some nice > optional features to files and directories so that they can do > everything that attributes can do if they are used that way, you just > didn't get it. You instead went for the quick ugly hack called xattrs. > You then got that ugly hack done first, because quick hacks are, well, > quick. I then went about doing it the right way for Reiser4, and got > DARPA to fund doing it. I was never silent about it. I don't want to comment on any of the technical issues about VFS etc. as I would be completely out of my depth, however I do want to say 2 things. Firstly, this is a feature that Samba users have been needing for many years to maintain compatibility with NTFS and Windows clients. Microsoft no longer sell any servers or clients without support for multiple data streams per file, and their latest XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad that Hans and Namesys have created something we can use to match this functionality - soon we will need it in order to be able to exist in a Microsoft client-dominated world. My second point is the following. Hans - did you *really* have to reinvent the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001 for Solaris) ? http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5 http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C http://bama.ua.edu/cgi-bin/man-cgi?openat+2 I'm complaining here as someone who will have to write portable code to try and work on all these "files with streams" systems. Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:20 ` Jeremy Allison @ 2004-08-26 8:42 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:42 UTC (permalink / raw) To: Jeremy Allison Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Jeremy Allison wrote: >On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote: > > >>You ignored everything I said during the discussion of xattrs about how >>there is no need to have attributes when you can just have files and >>directories, and that xattrs reflected a complete ignorance of name >>space design principles. When I said we should just add some nice >>optional features to files and directories so that they can do >>everything that attributes can do if they are used that way, you just >>didn't get it. You instead went for the quick ugly hack called xattrs. >>You then got that ugly hack done first, because quick hacks are, well, >>quick. I then went about doing it the right way for Reiser4, and got >>DARPA to fund doing it. I was never silent about it. >> >> > >I don't want to comment on any of the technical issues about VFS etc. as >I would be completely out of my depth, however I do want to say 2 things. Firstly, >this is a feature that Samba users have been needing for many years to maintain >compatibility with NTFS and Windows clients. Microsoft no longer sell any servers >or clients without support for multiple data streams per file, and their latest >XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad >that Hans and Namesys have created something we can use to match this >functionality - soon we will need it in order to be able to exist in >a Microsoft client-dominated world. > > I agree that your work is important without agreeing that MS client domination will last.;-) It is indeed my desire to give you every single feature you need to emulate MS streams within files, but doing it using directories that are files. I would like to support you in emulating windows faster than windows. >My second point is the following. Hans - did you *really* have to reinvent >the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001 >for Solaris) ? > > I interviewed for the file system architect job at Sun in, I think, 1999, and they offered me the job conditional on my giving up on my Linux work. (After much trying and failing to convince them that it would be okay for me to work on Linux also, I declined the job, much to my fiscal loss and work satisfaction.) They do not do a pure job of implementing attributes in the file namespace though. There are far more distinctions between files and attributes than are necessary that are described in these man pages below, and those distinctions cause a loss of closure. I can say more on that if asked. >http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5 >http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C >http://bama.ua.edu/cgi-bin/man-cgi?openat+2 > >I'm complaining here as someone who will have to write portable code >to try and work on all these "files with streams" systems. > >Jeremy. > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:42 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:42 UTC (permalink / raw) To: Jeremy Allison Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Jeremy Allison wrote: >On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote: > > >>You ignored everything I said during the discussion of xattrs about how >>there is no need to have attributes when you can just have files and >>directories, and that xattrs reflected a complete ignorance of name >>space design principles. When I said we should just add some nice >>optional features to files and directories so that they can do >>everything that attributes can do if they are used that way, you just >>didn't get it. You instead went for the quick ugly hack called xattrs. >>You then got that ugly hack done first, because quick hacks are, well, >>quick. I then went about doing it the right way for Reiser4, and got >>DARPA to fund doing it. I was never silent about it. >> >> > >I don't want to comment on any of the technical issues about VFS etc. as >I would be completely out of my depth, however I do want to say 2 things. Firstly, >this is a feature that Samba users have been needing for many years to maintain >compatibility with NTFS and Windows clients. Microsoft no longer sell any servers >or clients without support for multiple data streams per file, and their latest >XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad >that Hans and Namesys have created something we can use to match this >functionality - soon we will need it in order to be able to exist in >a Microsoft client-dominated world. > > I agree that your work is important without agreeing that MS client domination will last.;-) It is indeed my desire to give you every single feature you need to emulate MS streams within files, but doing it using directories that are files. I would like to support you in emulating windows faster than windows. >My second point is the following. Hans - did you *really* have to reinvent >the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001 >for Solaris) ? > > I interviewed for the file system architect job at Sun in, I think, 1999, and they offered me the job conditional on my giving up on my Linux work. (After much trying and failing to convince them that it would be okay for me to work on Linux also, I declined the job, much to my fiscal loss and work satisfaction.) They do not do a pure job of implementing attributes in the file namespace though. There are far more distinctions between files and attributes than are necessary that are described in these man pages below, and those distinctions cause a loss of closure. I can say more on that if asked. >http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5 >http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C >http://bama.ua.edu/cgi-bin/man-cgi?openat+2 > >I'm complaining here as someone who will have to write portable code >to try and work on all these "files with streams" systems. > >Jeremy. > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:42 ` Hans Reiser @ 2004-08-26 13:27 ` Rik van Riel -1 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 13:27 UTC (permalink / raw) To: Hans Reiser Cc: Jeremy Allison, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, 26 Aug 2004, Hans Reiser wrote: > I agree that your work is important without agreeing that MS client > domination will last.;-) It is indeed my desire to give you every > single feature you need to emulate MS streams within files, but doing it > using directories that are files. I would like to support you in > emulating windows faster than windows. 1) how do you back up and restore files with streams inside ? 2) how do standard unix utilities handle them ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 13:27 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 13:27 UTC (permalink / raw) To: Hans Reiser Cc: Jeremy Allison, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, 26 Aug 2004, Hans Reiser wrote: > I agree that your work is important without agreeing that MS client > domination will last.;-) It is indeed my desire to give you every > single feature you need to emulate MS streams within files, but doing it > using directories that are files. I would like to support you in > emulating windows faster than windows. 1) how do you back up and restore files with streams inside ? 2) how do standard unix utilities handle them ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:27 ` Rik van Riel (?) @ 2004-08-26 13:56 ` Markus Törnqvist 2004-08-26 19:58 ` Paul Jackson -1 siblings, 1 reply; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-26 13:56 UTC (permalink / raw) To: Rik van Riel Cc: Hans Reiser, Jeremy Allison, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 09:27:52AM -0400, Rik van Riel wrote: > >1) how do you back up and restore files with streams inside ? With a program that supports files with streams inside. >2) how do standard unix utilities handle them ? Most likely they don't ;) That is, until they are fixed or replaced. I've heard of people who want xattrs to be backed up so they use star, not gnu tar, already. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:56 ` Markus Törnqvist @ 2004-08-26 19:58 ` Paul Jackson 0 siblings, 0 replies; 1835+ messages in thread From: Paul Jackson @ 2004-08-26 19:58 UTC (permalink / raw) To: Markus Törnqvist Cc: riel, reiser, jra, hch, akpm, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Markus wrote: > >2) how do standard unix utilities handle them ? > > Most likely they don't ;) That is, until they are fixed or replaced. Not good, in my view. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 19:58 ` Paul Jackson 0 siblings, 0 replies; 1835+ messages in thread From: Paul Jackson @ 2004-08-26 19:58 UTC (permalink / raw) To: Markus Törnqvist Cc: riel, reiser, jra, hch, akpm, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Markus wrote: > >2) how do standard unix utilities handle them ? > > Most likely they don't ;) That is, until they are fixed or replaced. Not good, in my view. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:27 ` Rik van Riel @ 2004-08-26 23:53 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:53 UTC (permalink / raw) To: Rik van Riel Cc: Jeremy Allison, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Rik van Riel wrote: >On Thu, 26 Aug 2004, Hans Reiser wrote: > > > >>I agree that your work is important without agreeing that MS client >>domination will last.;-) It is indeed my desire to give you every >>single feature you need to emulate MS streams within files, but doing it >>using directories that are files. I would like to support you in >>emulating windows faster than windows. >> >> > >1) how do you back up and restore files with streams inside ? > >2) how do standard unix utilities handle them ? > > > To repeat, I think it would be nice to implement a filename/pseudos/backup method for all the plugins. Guys, we just have the beginnings in place. One plugin method at a time it will all fall into place. What we have now is useful now. The more methods come into existence, the more compelling it becomes --- standard network economic theory applies here. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:53 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:53 UTC (permalink / raw) To: Rik van Riel Cc: Jeremy Allison, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Rik van Riel wrote: >On Thu, 26 Aug 2004, Hans Reiser wrote: > > > >>I agree that your work is important without agreeing that MS client >>domination will last.;-) It is indeed my desire to give you every >>single feature you need to emulate MS streams within files, but doing it >>using directories that are files. I would like to support you in >>emulating windows faster than windows. >> >> > >1) how do you back up and restore files with streams inside ? > >2) how do standard unix utilities handle them ? > > > To repeat, I think it would be nice to implement a filename/pseudos/backup method for all the plugins. Guys, we just have the beginnings in place. One plugin method at a time it will all fall into place. What we have now is useful now. The more methods come into existence, the more compelling it becomes --- standard network economic theory applies here. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser ` (3 preceding siblings ...) (?) @ 2004-08-25 20:22 ` Chris Mason 2004-08-26 8:42 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Chris Mason @ 2004-08-25 20:22 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Wed, 2004-08-25 at 15:53, Hans Reiser wrote: > I had not intended to respond to this because I have nothing positive to > say, but Andrew said I needed to respond and suggested I should copy > Linus. Sigh. > > Dear Christoph, > > Let me see if I can summarize what you and your contingent are saying, > and if I misconstrue anything, let me know.;-) > Just for fun why don't we look at the way things are today: 1) reiser4 has semantics that do belong at the VFS level. They weren't implemented at the VFS level for a variety of reasons, none of which really matter right now. 2) new kernel patches that fragment the application developers between apis are a bad thing. There does need to be one interface here, and it is in Hans' best interest to unify his work by working with people to introduce new kernel wide apis. This starts with exactly what Christoph described in writing a short summary of how you want things to work today. Since we can't resist, we'll also go ahead and rehash all the old flame wars over this, but try to include some new ideas about where you want to see the reiser4 interfaces in 6 months as well. -chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Chris Mason @ 2004-08-26 8:42 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:42 UTC (permalink / raw) To: Chris Mason Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Chris Mason wrote: > > >This starts with exactly what Christoph described in writing a short >summary of how you want things to work today. Since we can't resist, >we'll also go ahead and rehash all the old flame wars over this, but try >to include some new ideas about where you want to see the reiser4 >interfaces in 6 months as well. > >-chris > > > > > > > Did Christoph read the www.namesys.com URL that has for several years been hiding from him? ;-) Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:42 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:42 UTC (permalink / raw) To: Chris Mason Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Chris Mason wrote: > > >This starts with exactly what Christoph described in writing a short >summary of how you want things to work today. Since we can't resist, >we'll also go ahead and rehash all the old flame wars over this, but try >to include some new ideas about where you want to see the reiser4 >interfaces in 6 months as well. > >-chris > > > > > > > Did Christoph read the www.namesys.com URL that has for several years been hiding from him? ;-) Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:42 ` Hans Reiser (?) @ 2004-08-26 9:36 ` Christoph Hellwig -1 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 9:36 UTC (permalink / raw) To: Hans Reiser Cc: Chris Mason, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 01:42:50AM -0700, Hans Reiser wrote: > Did Christoph read the www.namesys.com URL that has for several years > been hiding from him? ;-) Yes, and I'm sick of your marketing bullshit Hans. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser ` (4 preceding siblings ...) (?) @ 2004-08-25 20:23 ` Chris Friesen -1 siblings, 0 replies; 1835+ messages in thread From: Chris Friesen @ 2004-08-25 20:23 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Hans Reiser wrote: > Making files into directories caused only two applications out of the > entire OS to notice the change, and that was because of a bug in what > error code we returned that we are going to fix. You think that was a > disaster; I think it was a triumph. > Since we have such a performance lead, Namesys is about to change its > focus from the storage layer to semantics, look at > www.namesys.com/whitepaper.html for details. Semantic enhancements are > the important stuff, and finally Namesys is where we have all the > storage layer prerequisites done right, and the real work can begin. Just curious about your comments on Jamie Lokier's suggestions for enabling files-as-directories semantics without breaking existing apps. Chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser ` (5 preceding siblings ...) (?) @ 2004-08-25 22:28 ` Andrew Morton 2004-08-25 22:51 ` Spam 2004-08-26 8:31 ` Hans Reiser -1 siblings, 2 replies; 1835+ messages in thread From: Andrew Morton @ 2004-08-25 22:28 UTC (permalink / raw) To: Hans Reiser Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser <reiser@namesys.com> wrote: > > I had not intended to respond to this because I have nothing positive to > say, but Andrew said I needed to respond and suggested I should copy > Linus. Yes, but I didn't say "flame Christoph and ignore the issues" ;) There are lots of little things to do with implementation, coding style, module exports, deadlocks, what code goes where, etc. These are all normal daily kernel business and we should set them aside for now and concentrate on the bigger issues. And as I see it, there are two big issues: a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not anticipate and b) it does this within the context of just a single filesystem. I see three possible responses: a) accept the reiser4-only extensions as-is (possibly with post-review modifications, of course) or b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or c) reject the extensions. My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason. b) means that at some time in the future we need to hoist the reiser4 extensions (at a conceptual level) (and probably with restrictions) up into the VFS. This will involve much thought, argument and work. To get us started on this route it would really help me (and, probably, others) if you could describe what these API extensions are in a very simple way, without referring to incomprehehsible web pages, and without using terms which non-reiser4 people don't understand. It would be best if each extension was addressed in a separate email thread. We also need to discuss what a reiser4 "module" is, what its capabilities are, and what licensing implications they have. Then, we can look at each one and say "yup, that makes sense - we want Linux to do that" and we can also think about how we would implement it at the VFS level. If we follow the above route I believe we can make progress in a technical direction and not get deadlocked on personal/political stuff. Now, an alternative to all the above is to just merge reiser4 as-is, after addressing all the lower-level coding issues. And see what happens. That may be a thing which Linus wishes to do. I'm easy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:28 ` Andrew Morton @ 2004-08-25 22:51 ` Spam 2004-08-26 8:31 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-25 22:51 UTC (permalink / raw) To: Andrew Morton Cc: Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > And as I see it, there are two big issues: > a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not > anticipate and > b) it does this within the context of just a single filesystem. > I see three possible responses: > a) accept the reiser4-only extensions as-is (possibly with post-review > modifications, of course) or > b) accept the reiser4-only extensions with a view to turning them into > kernel-wide extensions at some time in the future, so all filesystems > will offer the extensions (as much as poss) or > c) reject the extensions. > My own order of preference is b) c) a). The fact that one filesystem will > offer features which other filesystems do not and cannot offer makes me > queasy for some reason. This last sentence makes me wonder. Where is Linux heading? The idea that a FS cannot contain features that no other FS has is very scary. I am all for uniformity, but not at the expense of shutting down advanced progress that Linux is so badly needing. This talk about old UNIX seems like people want to still live in the 70'ies and not look forward. Please wake up! ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-25 22:51 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-25 22:51 UTC (permalink / raw) To: Andrew Morton Cc: Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > And as I see it, there are two big issues: > a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not > anticipate and > b) it does this within the context of just a single filesystem. > I see three possible responses: > a) accept the reiser4-only extensions as-is (possibly with post-review > modifications, of course) or > b) accept the reiser4-only extensions with a view to turning them into > kernel-wide extensions at some time in the future, so all filesystems > will offer the extensions (as much as poss) or > c) reject the extensions. > My own order of preference is b) c) a). The fact that one filesystem will > offer features which other filesystems do not and cannot offer makes me > queasy for some reason. This last sentence makes me wonder. Where is Linux heading? The idea that a FS cannot contain features that no other FS has is very scary. I am all for uniformity, but not at the expense of shutting down advanced progress that Linux is so badly needing. This talk about old UNIX seems like people want to still live in the 70'ies and not look forward. Please wake up! ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:51 ` Spam (?) @ 2004-08-25 22:51 ` Christoph Hellwig -1 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-25 22:51 UTC (permalink / raw) To: Spam Cc: Andrew Morton, Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > This last sentence makes me wonder. Where is Linux heading? The idea > that a FS cannot contain features that no other FS has is very > scary. > > I am all for uniformity, but not at the expense of shutting down > advanced progress that Linux is so badly needing. > > This talk about old UNIX seems like people want to still live in the > 70'ies and not look forward. Please wake up! Just because semantics are at the VFS layer doesn't mean every filesystem has to implement them. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:51 ` Spam (?) (?) @ 2004-08-25 22:59 ` Linus Torvalds 2004-08-25 23:19 ` Spam -1 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-25 22:59 UTC (permalink / raw) To: Spam Cc: Andrew Morton, Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Spam wrote: > > > My own order of preference is b) c) a). The fact that one filesystem will > > offer features which other filesystems do not and cannot offer makes me > > queasy for some reason. > > This last sentence makes me wonder. Where is Linux heading? The idea > that a FS cannot contain features that no other FS has is very > scary. That's not what Andrew said or meant. Note the "cannot offer". As in "there is no way to offer them even if the filesystem could support it otherwise". We have tons of filesystems that do things other filesystems cannot do. Most filesystems support writing to a file - despite the fact that some filesystems (iso9600 being an obvious one) cannot. The infrastructure is there in the VFS layer, and it becomes a _choice_ for the filesystem whether it offers certain capabilities. So look at what Andrew said, again: his top choice would be (b). Let's see what that was again, shall we? > b) accept the reiser4-only extensions with a view to turning them into > kernel-wide extensions at some time in the future, so all filesystems > will offer the extensions (as much as poss) or In other words, if reiserfs does something special, we should make standard interfaces for doing that special thing, so that everybody can do it without stepping on other peoples toes. That doesn't mean that we'd _force_ everybody to do it. The same way we don't force iso9660 to write to a CD-ROM. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:59 ` Linus Torvalds @ 2004-08-25 23:19 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-25 23:19 UTC (permalink / raw) To: Linus Torvalds Cc: Andrew Morton, Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Thu, 26 Aug 2004, Spam wrote: >> >> > My own order of preference is b) c) a). The fact that one filesystem will >> > offer features which other filesystems do not and cannot offer makes me >> > queasy for some reason. >> >> This last sentence makes me wonder. Where is Linux heading? The idea >> that a FS cannot contain features that no other FS has is very >> scary. > That's not what Andrew said or meant. > Note the "cannot offer". As in "there is no way to offer them even if the > filesystem could support it otherwise". > We have tons of filesystems that do things other filesystems cannot do. > Most filesystems support writing to a file - despite the fact that some > filesystems (iso9600 being an obvious one) cannot. The infrastructure is > there in the VFS layer, and it becomes a _choice_ for the filesystem > whether it offers certain capabilities. > So look at what Andrew said, again: his top choice would be (b). Let's see > what that was again, shall we? >> b) accept the reiser4-only extensions with a view to turning them into >> kernel-wide extensions at some time in the future, so all filesystems >> will offer the extensions (as much as poss) or > In other words, if reiserfs does something special, we should make > standard interfaces for doing that special thing, so that everybody can > do it without stepping on other peoples toes. Agreed that would be the best. But how much time and effort will it be, and how much of the original ideas would be lost on the way to implement them in the VFS? Especially with new and very advanced FS's like Reiser4. Isn't the line between the actual file system and the virtual one very hair thin? Where would the separation lay in Reiser4? > That doesn't mean that we'd _force_ everybody to do it. The same way we > don't force iso9660 to write to a CD-ROM. > Linus I got caught in the moment of flame war. My appologies. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-25 23:19 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-25 23:19 UTC (permalink / raw) To: Linus Torvalds Cc: Andrew Morton, Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Thu, 26 Aug 2004, Spam wrote: >> >> > My own order of preference is b) c) a). The fact that one filesystem will >> > offer features which other filesystems do not and cannot offer makes me >> > queasy for some reason. >> >> This last sentence makes me wonder. Where is Linux heading? The idea >> that a FS cannot contain features that no other FS has is very >> scary. > That's not what Andrew said or meant. > Note the "cannot offer". As in "there is no way to offer them even if the > filesystem could support it otherwise". > We have tons of filesystems that do things other filesystems cannot do. > Most filesystems support writing to a file - despite the fact that some > filesystems (iso9600 being an obvious one) cannot. The infrastructure is > there in the VFS layer, and it becomes a _choice_ for the filesystem > whether it offers certain capabilities. > So look at what Andrew said, again: his top choice would be (b). Let's see > what that was again, shall we? >> b) accept the reiser4-only extensions with a view to turning them into >> kernel-wide extensions at some time in the future, so all filesystems >> will offer the extensions (as much as poss) or > In other words, if reiserfs does something special, we should make > standard interfaces for doing that special thing, so that everybody can > do it without stepping on other peoples toes. Agreed that would be the best. But how much time and effort will it be, and how much of the original ideas would be lost on the way to implement them in the VFS? Especially with new and very advanced FS's like Reiser4. Isn't the line between the actual file system and the virtual one very hair thin? Where would the separation lay in Reiser4? > That doesn't mean that we'd _force_ everybody to do it. The same way we > don't force iso9660 to write to a CD-ROM. > Linus I got caught in the moment of flame war. My appologies. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:19 ` Spam (?) @ 2004-08-25 23:32 ` Andrew Morton 2004-08-25 23:37 ` Jeremy Allison 2004-08-26 9:03 ` Anton Altaparmakov -1 siblings, 2 replies; 1835+ messages in thread From: Andrew Morton @ 2004-08-25 23:32 UTC (permalink / raw) To: Spam Cc: torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > > In other words, if reiserfs does something special, we should make > > standard interfaces for doing that special thing, so that everybody can > > do it without stepping on other peoples toes. > > Agreed that would be the best. But how much time and effort will it > be Zero. We can add these new features tomorrow, as reiser4-only features, with a plan in hand to generalise them later. -->>__if__<<-- we think these are features which Linux should offer. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:32 ` Andrew Morton @ 2004-08-25 23:37 ` Jeremy Allison 2004-08-25 23:46 ` Wichert Akkerman 2004-08-26 9:03 ` Anton Altaparmakov 1 sibling, 1 reply; 1835+ messages in thread From: Jeremy Allison @ 2004-08-25 23:37 UTC (permalink / raw) To: Andrew Morton Cc: Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, Aug 25, 2004 at 04:32:25PM -0700, Andrew Morton wrote: > > We can add these new features tomorrow, as reiser4-only features, with a > plan in hand to generalise them later. > > -->>__if__<<-- we think these are features which Linux should offer. Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers. Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:37 ` Jeremy Allison @ 2004-08-25 23:46 ` Wichert Akkerman 2004-08-26 0:01 ` Tierra ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: Wichert Akkerman @ 2004-08-25 23:46 UTC (permalink / raw) To: Jeremy Allison Cc: Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Jeremy Allison wrote: > Multiple-data-stream files are something we should offer, definately (IMHO). > I don't care how we do it, but I know it's something we need as application > developers. Aside from samba, is there any other application that has a use for them? Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:46 ` Wichert Akkerman @ 2004-08-26 0:01 ` Tierra 2004-08-26 0:05 ` Wichert Akkerman 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 9:40 ` Spam 2 siblings, 1 reply; 1835+ messages in thread From: Tierra @ 2004-08-26 0:01 UTC (permalink / raw) To: Wichert Akkerman, ReiserFS On Thu, 26 Aug 2004 01:46:29 +0200, Wichert Akkerman <wichert@wiggy.net> wrote: > Previously Jeremy Allison wrote: > > Multiple-data-stream files are something we should offer, definately (IMHO). > > I don't care how we do it, but I know it's something we need as application > > developers. > > Aside from samba, is there any other application that has a use for > them? > > Wichert. > > -- > Wichert Akkerman <wichert@wiggy.net> It is simple to make things. > http://www.wiggy.net/ It is hard to make things simple. > It's not so much about samba having a use for them as it is about other applications using them between platforms in the same way. I'm glad to see Jeremy is on top of this because it's not something I had even thought about having available on the linux platform in the sense that my work deals with data streams on NTFS, HFS, and whatever other filesystems join the venture. NTFS had very poor support for them for a long time. At one point I was convinced it was going to be taken out eventually, but SP2 just showed with even better support for them, and more MS software has started using them. They can't turn around at this point and take them out, so they must be planning on using them to a wider extent. Bryan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:01 ` Tierra @ 2004-08-26 0:05 ` Wichert Akkerman [not found] ` <c7ef32a70408251732442bff16@mail.gmail.com> 0 siblings, 1 reply; 1835+ messages in thread From: Wichert Akkerman @ 2004-08-26 0:05 UTC (permalink / raw) To: Tierra; +Cc: ReiserFS Previously Tierra wrote: > It's not so much about samba having a use for them as it is about > other applications using them between platforms in the same way. But what for? Personally I still haven't seen a good use for them. Apple's resource fork comes close though. > NTFS had very poor support for them for a long time. At one point I > was convinced it was going to be taken out eventually, but SP2 just > showed with even better support for them, and more MS software has > started using them. I wonder how they intend to shoehorn that into distribution across other media such as emails or zip files. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1835+ messages in thread
[parent not found: <c7ef32a70408251732442bff16@mail.gmail.com>]
* Re: silent semantic changes with reiser4 [not found] ` <c7ef32a70408251732442bff16@mail.gmail.com> @ 2004-08-26 0:33 ` Tierra 0 siblings, 0 replies; 1835+ messages in thread From: Tierra @ 2004-08-26 0:33 UTC (permalink / raw) To: ReiserFS On Thu, 26 Aug 2004 02:05:40 +0200, Wichert Akkerman <wichert@wiggy.net> wrote: > But what for? Personally I still haven't seen a good use for them. > Apple's resource fork comes close though. There's one way where it can really come in handy, and that's with specific application file types. Certain media could be saved in a better format and read more efficiently. The reason you haven't seen this done yet though is because multiple data streams isn't a standard and isn't available in most enviroments. There also isn't any file transfer protocols that will handle transfer of multiple data streams except through Windows File Sharing. Take into account though that I have absolutely no experience or knowledge of how it all works in the backend, and don't know the implications involved in the implementation. As for applications that already use data streams, you mentioned Apple's resource fork already, so I assume you know about file types there. IIS currently uses ADS for image thumbnailing and it's also possible to hide an alternate website right inline with a public one. I've heard of a couple other applications on Windows that uses them. But you are right, none of them _appropriately_ use data streams for any specific advantage over regular files except to keep system controlled information hidden. Bryan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:46 ` Wichert Akkerman @ 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 9:40 ` Spam 2 siblings, 0 replies; 1835+ messages in thread From: Nicholas Miell @ 2004-08-26 0:42 UTC (permalink / raw) To: Wichert Akkerman Cc: Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: > Previously Jeremy Allison wrote: > > Multiple-data-stream files are something we should offer, definately (IMHO). > > I don't care how we do it, but I know it's something we need as application > > developers. > > Aside from samba, is there any other application that has a use for > them? > Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc. As to how to do it, I think the Solaris interface is reasonably decent. The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view (An important detail for those who want to access their multiple-data-streams from non-MDS aware apps is the runat shell command, which basically does a chdir into the specified file's attribute directory and then runs a command. i.e. 'runat ~/blah ls' will list the ~/blah's attributes.) The only real problem I have with their design is the calling them attributes and using "at" everywhere. "Attributes", because it will get confused with the current Linux xattr implementation (which is still useful for things that actually are file attributes, like security labels, ACLs, weird attributess that FAT/NTFS/whatever have, etc.). I don't like "at" because the API changes don't have anything to do with the actual attributes. It's a general set of changes to allow paths relative to a fd instead of the cwd, and doesn't really have anything specifically to do with attributes (with the exception of the O_XATTR flag). Replace "at" with "rel" and O_XATTR with O_FORK or O_MULTI or something, and it's all good. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 0:42 ` Nicholas Miell 0 siblings, 0 replies; 1835+ messages in thread From: Nicholas Miell @ 2004-08-26 0:42 UTC (permalink / raw) To: Wichert Akkerman Cc: Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: > Previously Jeremy Allison wrote: > > Multiple-data-stream files are something we should offer, definately (IMHO). > > I don't care how we do it, but I know it's something we need as application > > developers. > > Aside from samba, is there any other application that has a use for > them? > Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc. As to how to do it, I think the Solaris interface is reasonably decent. The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view (An important detail for those who want to access their multiple-data-streams from non-MDS aware apps is the runat shell command, which basically does a chdir into the specified file's attribute directory and then runs a command. i.e. 'runat ~/blah ls' will list the ~/blah's attributes.) The only real problem I have with their design is the calling them attributes and using "at" everywhere. "Attributes", because it will get confused with the current Linux xattr implementation (which is still useful for things that actually are file attributes, like security labels, ACLs, weird attributess that FAT/NTFS/whatever have, etc.). I don't like "at" because the API changes don't have anything to do with the actual attributes. It's a general set of changes to allow paths relative to a fd instead of the cwd, and doesn't really have anything specifically to do with attributes (with the exception of the O_XATTR flag). Replace "at" with "rel" and O_XATTR with O_FORK or O_MULTI or something, and it's all good. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:42 ` Nicholas Miell (?) @ 2004-08-26 1:03 ` Jamie Lokier 2004-08-26 1:26 ` Nicholas Miell ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 1:03 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Nicholas Miell wrote: > Anything that currently stores a file's metadata in another file really > wants this right now. Things like image thumbnails, document summaries, > digital signatures, etc. Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed. Whereas there are some other kinds of metadata which should not be deleted if the file is modified. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:03 ` Jamie Lokier @ 2004-08-26 1:26 ` Nicholas Miell 2004-08-26 1:53 ` Jamie Lokier 2004-08-26 8:41 ` Hans Reiser 2004-08-26 19:06 ` Giuseppe Bilotta 2 siblings, 1 reply; 1835+ messages in thread From: Nicholas Miell @ 2004-08-26 1:26 UTC (permalink / raw) To: Jamie Lokier Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 2004-08-25 at 18:03, Jamie Lokier wrote: > Nicholas Miell wrote: > > Anything that currently stores a file's metadata in another file really > > wants this right now. Things like image thumbnails, document summaries, > > digital signatures, etc. > > Additionally, all of those things you describe should be deleted if > the file is modified -- to indicate that they're no longer valid and > should be regenerated if needed. > > Whereas there are some other kinds of metadata which should not be > deleted if the file is modified. > > -- Jamie Presumably the app which uses the metadata will be smart enough to compare the st_mtime of the MDS/stream/attribute/whatever (can we choose a name for these things now?) to the st_mtime of the file and do the right thing. thumbnail - regenerate it summary - keep it, it's relatively independent of the file's exact contents signature - always verify it etc. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:26 ` Nicholas Miell @ 2004-08-26 1:53 ` Jamie Lokier 2004-08-26 2:02 ` Nicholas Miell 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 1:53 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Nicholas Miell wrote: > > Additionally, all of those things you describe should be deleted if > > the file is modified -- to indicate that they're no longer valid and > > should be regenerated if needed. > > > > Whereas there are some other kinds of metadata which should not be > > deleted if the file is modified. > > > > -- Jamie > > Presumably the app which uses the metadata will be smart enough to > compare the st_mtime of the MDS/stream/attribute/whatever (can we choose > a name for these things now?) to the st_mtime of the file and do the > right thing. [This is straying off the topic of files-as-directories]. st_mtime tests are weak. They break sometimes, and are not suitable for strong data models such as transparently caching generated data from a file's contents. They're especially breakable if you change a file and then read it within a second. Sometimes, more than a second. As has been raised before, nanosecond timestamps (a) don't solve this unless they're stored in the filesystem; (b) even then they will fail one day, on a sufficiently fast box; (c) don't work when there are changes to wall clock time. They're fine where you don't care if the generated data is wrong sometimes. If you want the _equivalent_ behaviour to opening and parsing the file, but fast, then they're no good. Modification serial numbers would work, though. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:53 ` Jamie Lokier @ 2004-08-26 2:02 ` Nicholas Miell 0 siblings, 0 replies; 1835+ messages in thread From: Nicholas Miell @ 2004-08-26 2:02 UTC (permalink / raw) To: Jamie Lokier Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 2004-08-25 at 18:53, Jamie Lokier wrote: > Nicholas Miell wrote: > > > Additionally, all of those things you describe should be deleted if > > > the file is modified -- to indicate that they're no longer valid and > > > should be regenerated if needed. > > > > > > Whereas there are some other kinds of metadata which should not be > > > deleted if the file is modified. > > > > > > -- Jamie > > > > Presumably the app which uses the metadata will be smart enough to > > compare the st_mtime of the MDS/stream/attribute/whatever (can we choose > > a name for these things now?) to the st_mtime of the file and do the > > right thing. > > [This is straying off the topic of files-as-directories]. > > st_mtime tests are weak. They break sometimes, and are not suitable > for strong data models such as transparently caching generated data > from a file's contents. > > They're especially breakable if you change a file and then read it > within a second. Sometimes, more than a second. > > As has been raised before, nanosecond timestamps (a) don't solve this > unless they're stored in the filesystem; (b) even then they will fail > one day, on a sufficiently fast box; (c) don't work when there are > changes to wall clock time. > > They're fine where you don't care if the generated data is wrong sometimes. > > If you want the _equivalent_ behaviour to opening and parsing the > file, but fast, then they're no good. > > Modification serial numbers would work, though. > > -- Jamie Remember that the Solaris attribute model is just another filesystem tree rooted in a hidden directory associated with a regular file. If you can come up with semantics that work in general for all files, that's fine. inodes that delete themselves when other inodes are changed creep me out. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:03 ` Jamie Lokier @ 2004-08-26 8:41 ` Hans Reiser 2004-08-26 8:41 ` Hans Reiser 2004-08-26 19:06 ` Giuseppe Bilotta 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:41 UTC (permalink / raw) To: Jamie Lokier Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie Lokier wrote: >Nicholas Miell wrote: > > >>Anything that currently stores a file's metadata in another file really >>wants this right now. Things like image thumbnails, document summaries, >>digital signatures, etc. >> >> > >Additionally, all of those things you describe should be deleted if >the file is modified -- to indicate that they're no longer valid and >should be regenerated if needed. > >Whereas there are some other kinds of metadata which should not be >deleted if the file is modified. > >-- Jamie > > > > Yes, I agree. Actually we plan to have a whole link taxonomy, and one expected feature is that some links don't count towards the refcount needed to keep an object in existence (For instance, the links between key words and text documents, you don't want to have to explicitly unlink every keyword in order to delete a document). ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:41 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:41 UTC (permalink / raw) To: Jamie Lokier Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie Lokier wrote: >Nicholas Miell wrote: > > >>Anything that currently stores a file's metadata in another file really >>wants this right now. Things like image thumbnails, document summaries, >>digital signatures, etc. >> >> > >Additionally, all of those things you describe should be deleted if >the file is modified -- to indicate that they're no longer valid and >should be regenerated if needed. > >Whereas there are some other kinds of metadata which should not be >deleted if the file is modified. > >-- Jamie > > > > Yes, I agree. Actually we plan to have a whole link taxonomy, and one expected feature is that some links don't count towards the refcount needed to keep an object in existence (For instance, the links between key words and text documents, you don't want to have to explicitly unlink every keyword in order to delete a document). ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:03 ` Jamie Lokier @ 2004-08-26 19:06 ` Giuseppe Bilotta 2004-08-26 8:41 ` Hans Reiser 2004-08-26 19:06 ` Giuseppe Bilotta 2 siblings, 0 replies; 1835+ messages in thread From: Giuseppe Bilotta @ 2004-08-26 19:06 UTC (permalink / raw) To: linux-kernel; +Cc: reiserfs-list, linux-fsdevel, reiserfs-list, linux-kernel Jamie Lokier wrote: > Nicholas Miell wrote: > > Anything that currently stores a file's metadata in another file really > > wants this right now. Things like image thumbnails, document summaries, > > digital signatures, etc. > > Additionally, all of those things you describe should be deleted if > the file is modified -- to indicate that they're no longer valid and > should be regenerated if needed. In principle, not all of them. For example, a document summary for a text document or a long (textual) description of a video clip might remain the same when the user is only working on the finishing details. Maybe the metadata needs an attribute to determine how 'immutable' it should be wrt changes on the file? (Can you spell meta-meta-data <g>) -- Giuseppe "Oblomov" Bilotta Can't you see It all makes perfect sense Expressed in dollar and cents Pounds shillings and pence (Roger Waters) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 19:06 ` Giuseppe Bilotta 0 siblings, 0 replies; 1835+ messages in thread From: Giuseppe Bilotta @ 2004-08-26 19:06 UTC (permalink / raw) To: linux-kernel; +Cc: reiserfs-list, linux-fsdevel, reiserfs-list Jamie Lokier wrote: > Nicholas Miell wrote: > > Anything that currently stores a file's metadata in another file really > > wants this right now. Things like image thumbnails, document summaries, > > digital signatures, etc. > > Additionally, all of those things you describe should be deleted if > the file is modified -- to indicate that they're no longer valid and > should be regenerated if needed. In principle, not all of them. For example, a document summary for a text document or a long (textual) description of a video clip might remain the same when the user is only working on the finishing details. Maybe the metadata needs an attribute to determine how 'immutable' it should be wrt changes on the file? (Can you spell meta-meta-data <g>) -- Giuseppe "Oblomov" Bilotta Can't you see It all makes perfect sense Expressed in dollar and cents Pounds shillings and pence (Roger Waters) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:42 ` Nicholas Miell (?) (?) @ 2004-08-26 1:22 ` Jeremy Allison 2004-08-26 11:01 ` Wichert Akkerman -1 siblings, 1 reply; 1835+ messages in thread From: Jeremy Allison @ 2004-08-26 1:22 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > > Anything that currently stores a file's metadata in another file really > wants this right now. Things like image thumbnails, document summaries, > digital signatures, etc. Which is exactly how Windows apps have started to use streams. > As to how to do it, I think the Solaris interface is reasonably decent. > The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view I agree. This is an interface Samba can live with I think. I was thinking of implementing it anyway, just so I could piss off the Linux kernel developers by saying "oh if you need proper Windows semantics on Samba you have to use an advanced OS like Solaris, Linux doesn't cut it" :-) :-). > The only real problem I have with their design is the calling them > attributes and using "at" everywhere. Yep - they're different from xattrs. The easiest way to remember this is that file streams are seekable and get a fd, xattrs aren't and don't :-). Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:22 ` Jeremy Allison @ 2004-08-26 11:01 ` Wichert Akkerman [not found] ` <c7ef32a70408260942c8efa54@mail.gmail.com> 0 siblings, 1 reply; 1835+ messages in thread From: Wichert Akkerman @ 2004-08-26 11:01 UTC (permalink / raw) To: Jeremy Allison Cc: Nicholas Miell, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Jeremy Allison wrote: > Which is exactly how Windows apps have started to use streams. Interesting I can not find a way using the .NET framework, which is supposed to be The Future, to use streams. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1835+ messages in thread
[parent not found: <c7ef32a70408260942c8efa54@mail.gmail.com>]
* Re: silent semantic changes with reiser4 [not found] ` <c7ef32a70408260942c8efa54@mail.gmail.com> @ 2004-08-26 16:44 ` Tierra 0 siblings, 0 replies; 1835+ messages in thread From: Tierra @ 2004-08-26 16:44 UTC (permalink / raw) To: linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004 13:01:41 +0200, Wichert Akkerman <wichert@wiggy.net> wrote: > Previously Jeremy Allison wrote: > > Which is exactly how Windows apps have started to use streams. > > Interesting I can not find a way using the .NET framework, which is > supposed to be The Future, to use streams. That's because you don't even need .NET... http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/q105/7/63.asp&NoWebContent=1&NoWebContent=1 http://tinyurl.com/45mfa Bryan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:42 ` Nicholas Miell ` (2 preceding siblings ...) (?) @ 2004-08-26 4:44 ` Matt Mackall 2004-08-26 5:09 ` Nicholas Miell ` (3 more replies) -1 siblings, 4 replies; 1835+ messages in thread From: Matt Mackall @ 2004-08-26 4:44 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: > > Previously Jeremy Allison wrote: > > > Multiple-data-stream files are something we should offer, definately (IMHO). > > > I don't care how we do it, but I know it's something we need as application > > > developers. > > > > Aside from samba, is there any other application that has a use for > > them? > > > > Anything that currently stores a file's metadata in another file really > wants this right now. Things like image thumbnails, document summaries, > digital signatures, etc. That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already. While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE. A file is a stream of bytes. -- Mathematics is the supreme nostalgia of our time. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 4:44 ` Matt Mackall @ 2004-08-26 5:09 ` Nicholas Miell 2004-08-26 5:17 ` James Morris ` (3 more replies) 2004-08-26 5:23 ` Jamie Lokier ` (2 subsequent siblings) 3 siblings, 4 replies; 1835+ messages in thread From: Nicholas Miell @ 2004-08-26 5:09 UTC (permalink / raw) To: Matt Mackall Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 2004-08-25 at 21:44, Matt Mackall wrote: > On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: > > > Previously Jeremy Allison wrote: > > > > Multiple-data-stream files are something we should offer, definately (IMHO). > > > > I don't care how we do it, but I know it's something we need as application > > > > developers. > > > > > > Aside from samba, is there any other application that has a use for > > > them? > > > > > > > Anything that currently stores a file's metadata in another file really > > wants this right now. Things like image thumbnails, document summaries, > > digital signatures, etc. > > That is _highly_ debatable. I would much rather have my cp and grep > and cat and tar and such continue to work than have to rewrite every > tool because we've thrown the file-is-a-stream-of-bytes concept out > the window. Never mind that I've got thumbnails, document summaries, > and digital signatures already. > > While the number of annoying properties of files with forks is > practically endless, the biggest has got to be utter lack of > portability. How do you stick the thing in an attachment or on an ftp > site? Well you can't because it's NOT A FILE. > > A file is a stream of bytes. "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress. cp, grep, cat, and tar will continue to work just fine on files with multiple streams. tar and cp will lose the extra streams until somebody fixes them, but they lose ACLs and xattrs right now, and I don't hear anybody suggesting that ACLs and xattrs be removed from the kernel because of this. Fixing programs that do recursive filesystem traversals is a matter of a glibc patch to nftw(3) and a modification to their option processing. Holding back a useful feature because you don't want to upgrade coreutils is just plain dumb. (BTW, for email, multipart/parallel is a start, but a specific multipart content type for multi-stream file attachments would probably be more appropriate.) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:09 ` Nicholas Miell @ 2004-08-26 5:17 ` James Morris 2004-08-26 5:32 ` Matt Mackall ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: James Morris @ 2004-08-26 5:17 UTC (permalink / raw) To: Nicholas Miell Cc: Matt Mackall, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 25 Aug 2004, Nicholas Miell wrote: > (BTW, for email, multipart/parallel is a start, but a specific multipart > content type for multi-stream file attachments would probably be more > appropriate.) I don't understand why something like this is so important that it needs underlying VFS support. Multipart messages seem to have survived fine up until now. Where is the justification? - James -- James Morris <jmorris@redhat.com> ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 5:17 ` James Morris 0 siblings, 0 replies; 1835+ messages in thread From: James Morris @ 2004-08-26 5:17 UTC (permalink / raw) To: Nicholas Miell Cc: Matt Mackall, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 25 Aug 2004, Nicholas Miell wrote: > (BTW, for email, multipart/parallel is a start, but a specific multipart > content type for multi-stream file attachments would probably be more > appropriate.) I don't understand why something like this is so important that it needs underlying VFS support. Multipart messages seem to have survived fine up until now. Where is the justification? - James -- James Morris <jmorris@redhat.com> ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:09 ` Nicholas Miell 2004-08-26 5:17 ` James Morris @ 2004-08-26 5:32 ` Matt Mackall 2004-08-26 7:34 ` Denis Vlasenko ` (4 more replies) 2004-08-26 6:53 ` Paul Jackson 2004-08-26 9:54 ` Spam 3 siblings, 5 replies; 1835+ messages in thread From: Matt Mackall @ 2004-08-26 5:32 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, Aug 25, 2004 at 10:09:08PM -0700, Nicholas Miell wrote: > On Wed, 2004-08-25 at 21:44, Matt Mackall wrote: > > On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > > > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: > > > > Previously Jeremy Allison wrote: > > > > > Multiple-data-stream files are something we should offer, definately (IMHO). > > > > > I don't care how we do it, but I know it's something we need as application > > > > > developers. > > > > > > > > Aside from samba, is there any other application that has a use for > > > > them? > > > > > > > > > > Anything that currently stores a file's metadata in another file really > > > wants this right now. Things like image thumbnails, document summaries, > > > digital signatures, etc. > > > > That is _highly_ debatable. I would much rather have my cp and grep > > and cat and tar and such continue to work than have to rewrite every > > tool because we've thrown the file-is-a-stream-of-bytes concept out > > the window. Never mind that I've got thumbnails, document summaries, > > and digital signatures already. > > > > While the number of annoying properties of files with forks is > > practically endless, the biggest has got to be utter lack of > > portability. How do you stick the thing in an attachment or on an ftp > > site? Well you can't because it's NOT A FILE. > > > > A file is a stream of bytes. > > "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all > the time and are fixed. It's called progress. What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was. Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated. > cp, grep, cat, and tar will continue to work just fine on files with > multiple streams. Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work? I like cat < a > b. You can keep your progress. -- Mathematics is the supreme nostalgia of our time. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall @ 2004-08-26 7:34 ` Denis Vlasenko 2004-08-26 11:42 ` Rik van Riel 2004-08-26 7:53 ` Markus Törnqvist ` (3 subsequent siblings) 4 siblings, 1 reply; 1835+ messages in thread From: Denis Vlasenko @ 2004-08-26 7:34 UTC (permalink / raw) To: Matt Mackall, Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > > > A file is a stream of bytes. > > > > "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all > > the time and are fixed. It's called progress. > > What it breaks is the concept of a file. In ways that are ill-defined, > not portable, hard to work with, and needlessly complex. Along the > way, it breaks every single application that ever thought it knew what > a file was. > > Progress? No, this has been done before. Various dead operating > systems have done it or similar and regretted it. Most recently MacOS, > which jumped through major hurdles to begin purging themselves of > resource forks when they went to OS X. They're still there, but > heavily deprecated. > > > cp, grep, cat, and tar will continue to work just fine on files with > > multiple streams. > > Find some silly person with an iBook and open a shell on OS X. Use cp > to copy a file with a resource fork. Oh look, the Finder has no idea > what the new file is, even though it looks exactly identical in the > shell. Isn't that _wonderful_? Now try cat < a > b on a file with a > fork. How is that ever going to work? > > I like cat < a > b. You can keep your progress. cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions. -- vda ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 7:34 ` Denis Vlasenko @ 2004-08-26 11:42 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 11:42 UTC (permalink / raw) To: Denis Vlasenko Cc: Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > I like cat < a > b. You can keep your progress. > > cat <a >b does not preserve following file properties even on standard > UNIX filesystems: name,owner,group,permissions. Losing permissions is one thing. Annoying, mostly. However, actual losing file data during such a copy is nothing short of a disaster, IMHO. In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 11:42 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 11:42 UTC (permalink / raw) To: Denis Vlasenko Cc: Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > I like cat < a > b. You can keep your progress. > > cat <a >b does not preserve following file properties even on standard > UNIX filesystems: name,owner,group,permissions. Losing permissions is one thing. Annoying, mostly. However, actual losing file data during such a copy is nothing short of a disaster, IMHO. In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:42 ` Rik van Riel @ 2004-08-26 12:00 ` Christophe Saout -1 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 12:00 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 3556 bytes --] Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel: > > > I like cat < a > b. You can keep your progress. > > > > cat <a >b does not preserve following file properties even on standard > > UNIX filesystems: name,owner,group,permissions. > > Losing permissions is one thing. Annoying, mostly. > > However, actual losing file data during such a copy is > nothing short of a disaster, IMHO. > > In my opinion we shouldn't merge file-as-a-directory > semantics into the kernel until we figure out how to > fix the backup/restore problem and keep standard unix > utilities work. Well, again, what about xattrs and ACLs...? Actually, reiser4 doesn't currently implement storage of arbitrary user data under a file. It's just some sysfs-like information that can be retrieved, some properties can be changed. If you copy a file the worst thing that can happen right now is that you lose the information whether that file should be encrypted or compressed. You don't lose data. In case some people are wondering, this is what you can find in the reiser4 metas: leto:/home/chtephan/.muttrc/metas > la insgesamt 1 dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . -rwx------ 1 chtephan users 290 12. Jan 2004 .. -r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid -r--r--r-- 1 chtephan users 0 26. Aug 13:52 items -r--r--r-- 1 chtephan users 0 26. Aug 13:52 key -r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality --w------- 1 chtephan users 0 26. Aug 13:52 new -r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink -r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin -r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo -r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx -r--r--r-- 1 chtephan users 0 26. Aug 13:52 size -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid Some of them are reiser4 specific things (items, key, locality, oid) which can only be queried anyway. The rest is what you can also get using other VFS calls. leto:/home/chtephan/.muttrc/metas/plugin > la insgesamt 0 dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd Here you can change reiser4 specific things. Here you can change some properties how the file is stored in the reiser4 tree. For example you can activate compression, encryption or authentication or set some hints to optimize speed. You can't create or remove these pseudo-files, just like in sysfs. I don't know, but if someone wants to store user data I think you would have to implement a directory "userdata" or something where you can store it. Not all of the information does make a lot sense inside a modified tar. Who is interested in the bmap information of the file or some other read-only information? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 12:00 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 12:00 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 3556 bytes --] Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel: > > > I like cat < a > b. You can keep your progress. > > > > cat <a >b does not preserve following file properties even on standard > > UNIX filesystems: name,owner,group,permissions. > > Losing permissions is one thing. Annoying, mostly. > > However, actual losing file data during such a copy is > nothing short of a disaster, IMHO. > > In my opinion we shouldn't merge file-as-a-directory > semantics into the kernel until we figure out how to > fix the backup/restore problem and keep standard unix > utilities work. Well, again, what about xattrs and ACLs...? Actually, reiser4 doesn't currently implement storage of arbitrary user data under a file. It's just some sysfs-like information that can be retrieved, some properties can be changed. If you copy a file the worst thing that can happen right now is that you lose the information whether that file should be encrypted or compressed. You don't lose data. In case some people are wondering, this is what you can find in the reiser4 metas: leto:/home/chtephan/.muttrc/metas > la insgesamt 1 dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . -rwx------ 1 chtephan users 290 12. Jan 2004 .. -r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid -r--r--r-- 1 chtephan users 0 26. Aug 13:52 items -r--r--r-- 1 chtephan users 0 26. Aug 13:52 key -r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality --w------- 1 chtephan users 0 26. Aug 13:52 new -r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink -r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin -r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo -r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx -r--r--r-- 1 chtephan users 0 26. Aug 13:52 size -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid Some of them are reiser4 specific things (items, key, locality, oid) which can only be queried anyway. The rest is what you can also get using other VFS calls. leto:/home/chtephan/.muttrc/metas/plugin > la insgesamt 0 dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd Here you can change reiser4 specific things. Here you can change some properties how the file is stored in the reiser4 tree. For example you can activate compression, encryption or authentication or set some hints to optimize speed. You can't create or remove these pseudo-files, just like in sysfs. I don't know, but if someone wants to store user data I think you would have to implement a directory "userdata" or something where you can store it. Not all of the information does make a lot sense inside a modified tar. Who is interested in the bmap information of the file or some other read-only information? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:00 ` Christophe Saout @ 2004-08-28 11:00 ` Yury Umanets -1 siblings, 0 replies; 1835+ messages in thread From: Yury Umanets @ 2004-08-28 11:00 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel: > > > >>>>I like cat < a > b. You can keep your progress. >>>> >>>> >>>cat <a >b does not preserve following file properties even on standard >>>UNIX filesystems: name,owner,group,permissions. >>> >>> >>Losing permissions is one thing. Annoying, mostly. >> >>However, actual losing file data during such a copy is >>nothing short of a disaster, IMHO. >> >>In my opinion we shouldn't merge file-as-a-directory >>semantics into the kernel until we figure out how to >>fix the backup/restore problem and keep standard unix >>utilities work. >> >> > >Well, again, what about xattrs and ACLs...? > >Actually, reiser4 doesn't currently implement storage of arbitrary user >data under a file. > >It's just some sysfs-like information that can be retrieved, some >properties can be changed. If you copy a file the worst thing that can >happen right now is that you lose the information whether that file >should be encrypted or compressed. You don't lose data. > >In case some people are wondering, this is what you can find in the >reiser4 metas: > >leto:/home/chtephan/.muttrc/metas > la >insgesamt 1 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >-rwx------ 1 chtephan users 290 12. Jan 2004 .. >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 items >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 key >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality >--w------- 1 chtephan users 0 26. Aug 13:52 new >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 size >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid > >Some of them are reiser4 specific things (items, key, locality, oid) >which can only be queried anyway. > >The rest is what you can also get using other VFS calls. > >leto:/home/chtephan/.muttrc/metas/plugin > la >insgesamt 0 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd > >Here you can change reiser4 specific things. Here you can change some >properties how the file is stored in the reiser4 tree. For example you >can activate compression, encryption or authentication or set some hints >to optimize speed. > > I suppose that you are talking about the prinsiple ability to change these things. Because in real life it will also need some kind of convertion from one look (say with old plugin) to another one. Just like tail conversion rebuilds file's body after tail policy plugin gets changed. -- umka ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 11:00 ` Yury Umanets 0 siblings, 0 replies; 1835+ messages in thread From: Yury Umanets @ 2004-08-28 11:00 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel: > > > >>>>I like cat < a > b. You can keep your progress. >>>> >>>> >>>cat <a >b does not preserve following file properties even on standard >>>UNIX filesystems: name,owner,group,permissions. >>> >>> >>Losing permissions is one thing. Annoying, mostly. >> >>However, actual losing file data during such a copy is >>nothing short of a disaster, IMHO. >> >>In my opinion we shouldn't merge file-as-a-directory >>semantics into the kernel until we figure out how to >>fix the backup/restore problem and keep standard unix >>utilities work. >> >> > >Well, again, what about xattrs and ACLs...? > >Actually, reiser4 doesn't currently implement storage of arbitrary user >data under a file. > >It's just some sysfs-like information that can be retrieved, some >properties can be changed. If you copy a file the worst thing that can >happen right now is that you lose the information whether that file >should be encrypted or compressed. You don't lose data. > >In case some people are wondering, this is what you can find in the >reiser4 metas: > >leto:/home/chtephan/.muttrc/metas > la >insgesamt 1 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >-rwx------ 1 chtephan users 290 12. Jan 2004 .. >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 items >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 key >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality >--w------- 1 chtephan users 0 26. Aug 13:52 new >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 size >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid > >Some of them are reiser4 specific things (items, key, locality, oid) >which can only be queried anyway. > >The rest is what you can also get using other VFS calls. > >leto:/home/chtephan/.muttrc/metas/plugin > la >insgesamt 0 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd > >Here you can change reiser4 specific things. Here you can change some >properties how the file is stored in the reiser4 tree. For example you >can activate compression, encryption or authentication or set some hints >to optimize speed. > > I suppose that you are talking about the prinsiple ability to change these things. Because in real life it will also need some kind of convertion from one look (say with old plugin) to another one. Just like tail conversion rebuilds file's body after tail policy plugin gets changed. -- umka ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:00 ` Christophe Saout @ 2004-08-28 11:12 ` Yury Umanets -1 siblings, 0 replies; 1835+ messages in thread From: Yury Umanets @ 2004-08-28 11:12 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel: > > > >>>>I like cat < a > b. You can keep your progress. >>>> >>>> >>>cat <a >b does not preserve following file properties even on standard >>>UNIX filesystems: name,owner,group,permissions. >>> >>> >>Losing permissions is one thing. Annoying, mostly. >> >>However, actual losing file data during such a copy is >>nothing short of a disaster, IMHO. >> >>In my opinion we shouldn't merge file-as-a-directory >>semantics into the kernel until we figure out how to >>fix the backup/restore problem and keep standard unix >>utilities work. >> >> > >Well, again, what about xattrs and ACLs...? > >Actually, reiser4 doesn't currently implement storage of arbitrary user >data under a file. > >It's just some sysfs-like information that can be retrieved, some >properties can be changed. If you copy a file the worst thing that can >happen right now is that you lose the information whether that file >should be encrypted or compressed. You don't lose data. > >In case some people are wondering, this is what you can find in the >reiser4 metas: > >leto:/home/chtephan/.muttrc/metas > la >insgesamt 1 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >-rwx------ 1 chtephan users 290 12. Jan 2004 .. >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 items >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 key >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality >--w------- 1 chtephan users 0 26. Aug 13:52 new >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 size >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid > >Some of them are reiser4 specific things (items, key, locality, oid) >which can only be queried anyway. > >The rest is what you can also get using other VFS calls. > >leto:/home/chtephan/.muttrc/metas/plugin > la >insgesamt 0 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd > >Here you can change reiser4 specific things. Here you can change some >properties how the file is stored in the reiser4 tree. For example you >can activate compression, encryption or authentication or set some hints >to optimize speed. > > I suppose that you are talking about the principle ability to change these things. Because in real life it will also need some kind of conversion from one look (say with old plugin) to another one. Just like tail conversion rebuilds file's body after tail policy plugin gets changed. -- umka ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 11:12 ` Yury Umanets 0 siblings, 0 replies; 1835+ messages in thread From: Yury Umanets @ 2004-08-28 11:12 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel: > > > >>>>I like cat < a > b. You can keep your progress. >>>> >>>> >>>cat <a >b does not preserve following file properties even on standard >>>UNIX filesystems: name,owner,group,permissions. >>> >>> >>Losing permissions is one thing. Annoying, mostly. >> >>However, actual losing file data during such a copy is >>nothing short of a disaster, IMHO. >> >>In my opinion we shouldn't merge file-as-a-directory >>semantics into the kernel until we figure out how to >>fix the backup/restore problem and keep standard unix >>utilities work. >> >> > >Well, again, what about xattrs and ACLs...? > >Actually, reiser4 doesn't currently implement storage of arbitrary user >data under a file. > >It's just some sysfs-like information that can be retrieved, some >properties can be changed. If you copy a file the worst thing that can >happen right now is that you lose the information whether that file >should be encrypted or compressed. You don't lose data. > >In case some people are wondering, this is what you can find in the >reiser4 metas: > >leto:/home/chtephan/.muttrc/metas > la >insgesamt 1 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >-rwx------ 1 chtephan users 290 12. Jan 2004 .. >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 items >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 key >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality >--w------- 1 chtephan users 0 26. Aug 13:52 new >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 size >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid > >Some of them are reiser4 specific things (items, key, locality, oid) >which can only be queried anyway. > >The rest is what you can also get using other VFS calls. > >leto:/home/chtephan/.muttrc/metas/plugin > la >insgesamt 0 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd > >Here you can change reiser4 specific things. Here you can change some >properties how the file is stored in the reiser4 tree. For example you >can activate compression, encryption or authentication or set some hints >to optimize speed. > > I suppose that you are talking about the principle ability to change these things. Because in real life it will also need some kind of conversion from one look (say with old plugin) to another one. Just like tail conversion rebuilds file's body after tail policy plugin gets changed. -- umka ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:42 ` Rik van Riel (?) (?) @ 2004-08-26 12:12 ` Jamie Lokier 2004-08-26 12:25 ` Wichert Akkerman -1 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 12:12 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > cat <a >b does not preserve following file properties even on standard > > UNIX filesystems: name,owner,group,permissions. > > Losing permissions is one thing. Annoying, mostly. I find losing mtime annoying. That's why I don't use Mozilla's download manager (and Navigator's before it) because they presumed that mtime isn't worth copying. ncftp and wget do the right thing, but it's annoying that I can't use the fancy tools and get the metadata _I_ want transferred by them. > However, actual losing file data during such a copy is > nothing short of a disaster, IMHO. Note that file-as-directory doesn't imply that you can store just anything into those directories. Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"? > In my opinion we shouldn't merge file-as-a-directory > semantics into the kernel until we figure out how to > fix the backup/restore problem and keep standard unix > utilities work. Same as xattrs, for data that should be stored. But, unlike xattrs, while permissions and C-source-encoding should be stored, some of this metadata should _not_ be stored when making an archive and perhaps not when making a backup either. Personally I wouldn't want an archive containing a file to save and restore recomputable metadata like cached, extracted-on-demand MP3 id tags and email headers-as-metadata. I'd rather they were recalculated when needed, implying a stronger guarantee about their relationship to the data from which they're extracted - and implying that a user program might not be allowed to set them to inconsistent values. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:12 ` Jamie Lokier @ 2004-08-26 12:25 ` Wichert Akkerman 0 siblings, 0 replies; 1835+ messages in thread From: Wichert Akkerman @ 2004-08-26 12:25 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Jamie Lokier wrote: > I find losing mtime annoying. That's why I don't use Mozilla's > download manager (and Navigator's before it) because they presumed > that mtime isn't worth copying. It's free software, you could fix that :) > Is it a problem to decree that "file data MUST NOT be stored in a > metadata directory; only non-essential metadata and data computed from > the file data may be stored"? That would put us right back at the samba-and-other-apps-want-streams point in the discussion, but it certainly would not be a bad starting point. From there we could always add streams later on if the API is right. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:42 ` Rik van Riel @ 2004-08-26 20:35 ` Lee Revell -1 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 20:35 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 2004-08-26 at 07:42, Rik van Riel wrote: > On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > > > I like cat < a > b. You can keep your progress. > > > > cat <a >b does not preserve following file properties even on standard > > UNIX filesystems: name,owner,group,permissions. > > Losing permissions is one thing. Annoying, mostly. > cat <a >b does preserve permissions - the permissions of b. I always considered this a feature. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 20:35 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 20:35 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 2004-08-26 at 07:42, Rik van Riel wrote: > On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > > > I like cat < a > b. You can keep your progress. > > > > cat <a >b does not preserve following file properties even on standard > > UNIX filesystems: name,owner,group,permissions. > > Losing permissions is one thing. Annoying, mostly. > cat <a >b does preserve permissions - the permissions of b. I always considered this a feature. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:42 ` Rik van Riel ` (3 preceding siblings ...) (?) @ 2004-08-27 0:02 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 0:02 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > >In my opinion we shouldn't merge file-as-a-directory >semantics into the kernel until we figure out how to >fix the backup/restore problem and keep standard unix >utilities work. > > > One incremental piece at a time our solution will fall into place. Pieces should go in when they are individually ready. Warning users that some pieces might be changed out from under them up until when 2.8 ships is ok with me though. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall 2004-08-26 7:34 ` Denis Vlasenko @ 2004-08-26 7:53 ` Markus Törnqvist 2004-08-26 8:31 ` Helge Hafting ` (2 subsequent siblings) 4 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-26 7:53 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: > >What it breaks is the concept of a file. In ways that are ill-defined, >not portable, hard to work with, and needlessly complex. Along the >way, it breaks every single application that ever thought it knew what >a file was. It breaks the concept of a file. In ways that offer more versatility, challenge the imagination to make even better progress and keeps Linux competing with competitors who are implementing this stuff as we speak. I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file. Were I any richer, I'd pay Namesys to have this work for me :) >Find some silly person with an iBook and open a shell on OS X. Use cp >to copy a file with a resource fork. Oh look, the Finder has no idea >what the new file is, even though it looks exactly identical in the >shell. Isn't that _wonderful_? Now try cat < a > b on a file with a >fork. How is that ever going to work? Then I guess OS X ships a broken implementation of cp, yes? On the cat example, what if cat < a > b simply copies the "main stream" and not the metadata, as a feature. The key being, "as a feature" The metadata streams could get file descriptors of their own OR another program, streamcat or something, could be written to compensate. >I like cat < a > b. You can keep your progress. With all due respect, I hope not too many people agree with you :) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 7:53 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: mjt @ 2004-08-26 7:53 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: > >What it breaks is the concept of a file. In ways that are ill-defined, >not portable, hard to work with, and needlessly complex. Along the >way, it breaks every single application that ever thought it knew what >a file was. It breaks the concept of a file. In ways that offer more versatility, challenge the imagination to make even better progress and keeps Linux competing with competitors who are implementing this stuff as we speak. I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file. Were I any richer, I'd pay Namesys to have this work for me :) >Find some silly person with an iBook and open a shell on OS X. Use cp >to copy a file with a resource fork. Oh look, the Finder has no idea >what the new file is, even though it looks exactly identical in the >shell. Isn't that _wonderful_? Now try cat < a > b on a file with a >fork. How is that ever going to work? Then I guess OS X ships a broken implementation of cp, yes? On the cat example, what if cat < a > b simply copies the "main stream" and not the metadata, as a feature. The key being, "as a feature" The metadata streams could get file descriptors of their own OR another program, streamcat or something, could be written to compensate. >I like cat < a > b. You can keep your progress. With all due respect, I hope not too many people agree with you :) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 7:53 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-26 7:53 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: > >What it breaks is the concept of a file. In ways that are ill-defined, >not portable, hard to work with, and needlessly complex. Along the >way, it breaks every single application that ever thought it knew what >a file was. It breaks the concept of a file. In ways that offer more versatility, challenge the imagination to make even better progress and keeps Linux competing with competitors who are implementing this stuff as we speak. I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file. Were I any richer, I'd pay Namesys to have this work for me :) >Find some silly person with an iBook and open a shell on OS X. Use cp >to copy a file with a resource fork. Oh look, the Finder has no idea >what the new file is, even though it looks exactly identical in the >shell. Isn't that _wonderful_? Now try cat < a > b on a file with a >fork. How is that ever going to work? Then I guess OS X ships a broken implementation of cp, yes? On the cat example, what if cat < a > b simply copies the "main stream" and not the metadata, as a feature. The key being, "as a feature" The metadata streams could get file descriptors of their own OR another program, streamcat or something, could be written to compensate. >I like cat < a > b. You can keep your progress. With all due respect, I hope not too many people agree with you :) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 7:53 ` Markus Törnqvist (?) (?) @ 2004-08-26 16:32 ` Jan Harkes 2004-08-26 16:48 ` Linus Torvalds 2004-08-26 17:14 ` Hubert Chan -1 siblings, 2 replies; 1835+ messages in thread From: Jan Harkes @ 2004-08-26 16:32 UTC (permalink / raw) To: Markus Törnqvist Cc: Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus Törnqvist wrote: > On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: > >What it breaks is the concept of a file. In ways that are ill-defined, > >not portable, hard to work with, and needlessly complex. Along the > >way, it breaks every single application that ever thought it knew what > >a file was. > > It breaks the concept of a file. In ways that offer more versatility, > challenge the imagination to make even better progress and keeps > Linux competing with competitors who are implementing this stuff > as we speak. Sigh, it really isn't all that groundbreaking. The only special feature is the fact that it introduces directory hardlinks. Just turn the phrase 'file-as-dir' around and you might see the point. As far as userspace is concerned the implementation would behave 99% identical if we provided a 'dir-as-file' semantics. i.e. opening a directory without 'O_DIRECTORY' would return a filehandle to a default file within the directory. Something like open("/path/to/dir", O_RDONLY) would be translated to open('/path/to/dir/_contents', O_RDONLY). So I could simply replace all my files with directories and move the original file contents into a '_contents' file. Applications can still open them by the same name, although the attributes would look a bit different. Poof, we have the same file-as-dir concept, but now we play fair with the existing VFS and don't introduced races/deadlocks and other issues. Also existing backup tools will be able to back up and restore these 'streams'. (btw. the same could be implemented completely in userspace, for instance in glibc. Whenever an open call gets an EISDIR error, simply retry the open, but this time with /_contents appended). > I for one would truly welcome the coming of thumbnails and descriptions > in picture files, because I have a real-life project going on where > that would be extremely handy to have in the actual file. Ehhh, that already exists and doesn't require meta-data streams, http://www.exif.org/ > >Find some silly person with an iBook and open a shell on OS X. Use cp > >to copy a file with a resource fork. Oh look, the Finder has no idea > >what the new file is, even though it looks exactly identical in the > >shell. Isn't that _wonderful_? Now try cat < a > b on a file with a > >fork. How is that ever going to work? > > Then I guess OS X ships a broken implementation of cp, yes? No cp is working just fine, there is no copy syscall in the VFS, so any file system specific features such meta data streams will have to be handled by the application. So at some point 'cp' will have to be taught about macos resource forks, posix acls, xattrs, afs acls, reiserfs file-as-dir, freebsd attributes, samba/ntfs streams. And possibly even learn the semantical differences between these so that it can copy metadata between the various file systems types. Jan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:32 ` Jan Harkes @ 2004-08-26 16:48 ` Linus Torvalds 2004-08-26 18:07 ` Jan Harkes 2004-08-31 3:39 ` Tom Vier 2004-08-26 17:14 ` Hubert Chan 1 sibling, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 16:48 UTC (permalink / raw) To: Jan Harkes Cc: Markus Törnqvist, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jan Harkes wrote: > > (btw. the same could be implemented completely in userspace, for > instance in glibc. Whenever an open call gets an EISDIR error, simply > retry the open, but this time with /_contents appended). Yes, and no - just to make it obvious before people jump on this, a lot of things can be prototyped in user space with things like this, but once you have to deal with races and mixed tool environments, user space suddenly doesn't work so well any more. I think Jan understands this distinction, I just wanted to make sure everybody else is aware of the _one_ thing that kernel land does well: - safely synchronize globally visible data structures That's quite fundamental. 99% of what a kernel does is exactly that. TCP would be in user space too, if it wasn't for _exactly_ this issue. A lot of people think that kernels are about hardware access, and yes, that's the other 99% of the picture (I see the _big_ picture, remember?), but the "safe access to common data" is really very fundamental. The kernel is literally the thing that makes sure that you don't have - and _cannot_ have - user programs that confuse each other by modifying data unsynchronized. For example, a filesystem is really nothing but a way to access a disk in a controlled manner - it's not so much about hardware access, as it is about maintaining a coherent view of how some shared data (disk or whatever) is maintained. Same goes for caches. We could cache things in user space, but if you want to _share_ your caches (so that you don't have to re-load them for every new application), you need some entity that manages those shared data structures in a secure manner. In other words, you need the kernel. The same goes for something like a "container file". Whether you see it as "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are totally equivalent), the point of having the capability in the kernel is not that the operations cannot be done in user space - the point is that they cannot be done in user space _safely_. The kernel is kind of the thing that guarantees that everybody follows the rules. Imagine the security problems if a set-uid program were to (unwittingly) depend on a user-space library that implements what Jan's prototype library would do? Races galore, since a user-space implementation wouldn't have _any_ way to do tests like the above atomically. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:48 ` Linus Torvalds @ 2004-08-26 18:07 ` Jan Harkes 2004-08-31 3:39 ` Tom Vier 1 sibling, 0 replies; 1835+ messages in thread From: Jan Harkes @ 2004-08-26 18:07 UTC (permalink / raw) To: Linus Torvalds Cc: Andrew Morton, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, viro I trimmed/adjusted the CC list somewhat in the hope of getting a more productive discussion going. On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: > On Thu, 26 Aug 2004, Jan Harkes wrote: > > > > (btw. the same could be implemented completely in userspace, for > > instance in glibc. Whenever an open call gets an EISDIR error, simply > > retry the open, but this time with /_contents appended). > > Yes, and no - just to make it obvious before people jump on this, a lot of > things can be prototyped in user space with things like this, but once you > have to deal with races and mixed tool environments, user space suddenly > doesn't work so well any more. Agreed. > The same goes for something like a "container file". Whether you see it as > "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are > totally equivalent), the point of having the capability in the kernel is Only to some point. I was mostly trying to make the point that similar functionality is already achievable within the current VFS framework. However, files and directories still have some semantical differences, and these are probably the points that we would have to address to get Hans's efforts work well in cooperation with the Linux VFS. We cannot atomically unlink a dir-as-file, but can unlink a file-as-dir. In this case, the dcache seems to have no problems unlinking/unhashing a non-empty directory, so I'm assuming that the unlink problem is mostly enforced to avoid unreachable, but still allocated files in a file system. (I'm sure that Viro can probably come up with other reasons why unlinking a non-empty directory can be evil). Also, we cannot create hardlinks to a directory (-as-file), but I guess reiserfs4 allows hardlinks to files (-as-directory). Luckily as far as hardlinked directories are concerned the file-as-dir implementation should be a relatively benign subset. I assume that we are not allowed to rename or link arbitrary objects from the regular fs into it, Coda returns EXDEV in such situations. As a result there will be no cycles and this simple premise should get rid of at least a couple of the possible deadlock scenarios that I've seen mentioned. Jan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:48 ` Linus Torvalds 2004-08-26 18:07 ` Jan Harkes @ 2004-08-31 3:39 ` Tom Vier 2004-08-31 4:08 ` Linus Torvalds 1 sibling, 1 reply; 1835+ messages in thread From: Tom Vier @ 2004-08-31 3:39 UTC (permalink / raw) To: Linus Torvalds; +Cc: linux-kernel On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: > - safely synchronize globally visible data structures > That's quite fundamental. 99% of what a kernel does is exactly that. TCP > would be in user space too, if it wasn't for _exactly_ this issue. A lot What about microkernels? They do tcp in userspace. So did winsock, iirc. As long as a trusted process keeps data such as free ports, what's the problem? Perhaps a process could even simply hand off proto:port handles and let a library do the rest, but i know little about exokernels. (There's probably some catches.) -- Tom Vier <tmv@comcast.net> DSA Key ID 0x15741ECE ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 3:39 ` Tom Vier @ 2004-08-31 4:08 ` Linus Torvalds 2004-08-31 4:38 ` Jeff Garzik ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-31 4:08 UTC (permalink / raw) To: Tom Vier; +Cc: linux-kernel On Mon, 30 Aug 2004, Tom Vier wrote: > On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: > > - safely synchronize globally visible data structures > > That's quite fundamental. 99% of what a kernel does is exactly that. TCP > > would be in user space too, if it wasn't for _exactly_ this issue. A lot > > What about microkernels? They do tcp in userspace. No they don't. They do TCP in a separate address space from user space, that just also happens to be separate from the "microkernel address space". So a microkernel will have _more_ address spaces, and they won't be "user space". They'll be "server deamon space" or something. Now, that's also why they tend to have performance problems - because you need to copy the data between different address spaces, and switch the CPU context etc around. Not user space. They may be "ring 3" from a CPU standpoint, but they aren't user space from a _user_ standpoint - it's still very much a separate address space, with domain protection. > So did winsock, iirc. Now that is a different case. Things like the PalmOS TCP stack (and, I believe, Winsock) are true "user space" TCP stacks, in that they really do live as libraries in the same address space as the user app. It sucks. Exactly because now the data structures are _not_ protected, and they are _not_ shared. So your library basically ends up being a "single client" library, with no protection between clients (or no sharing: you can have "protected" single-stream TCP, but then you won't share the TCP state that needs to be shared like listen queues etc). This works in an environment like Palm or Win-3.0, which are really just single-client _anyway_, without any protection. But notice how Windows started doing TCP in the kernel, and notice how you can actually use it as a server these days? In short: you _need_ to have a separate address space (either kernel, or "TCP server" or whatever) if you want to have reliable, secure and generally usable TCP. > As long as a trusted process keeps data such as free ports, what's the > problem? None - because it's not user space any more. Well, performance might still suck, of course. And it does. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 4:08 ` Linus Torvalds @ 2004-08-31 4:38 ` Jeff Garzik 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen 2004-08-31 10:57 ` Alan Cox 2004-08-31 19:45 ` Toon van der Pas 2 siblings, 2 replies; 1835+ messages in thread From: Jeff Garzik @ 2004-08-31 4:38 UTC (permalink / raw) To: Linus Torvalds; +Cc: Tom Vier, linux-kernel Linus Torvalds wrote: > > On Mon, 30 Aug 2004, Tom Vier wrote: > > >>On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: >> >>> - safely synchronize globally visible data structures >>>That's quite fundamental. 99% of what a kernel does is exactly that. TCP >>>would be in user space too, if it wasn't for _exactly_ this issue. A lot >> >>What about microkernels? They do tcp in userspace. > > > No they don't. They do TCP in a separate address space from user space, > that just also happens to be separate from the "microkernel address > space". > > So a microkernel will have _more_ address spaces, and they won't be "user > space". They'll be "server deamon space" or something. Now, that's also > why they tend to have performance problems - because you need to copy the > data between different address spaces, and switch the CPU context etc > around. Man, this thread came a long way. I peek in and see a pet topic... _My_ own definition of microkernel, which differs from traditional CS, is where I would love to see Linux go: move as much as humanly possible to userspace, such that, the kernel only contains pagecache operations, driver fast paths, and security checks. Move slow paths, including ACPI probing, PCI bus walking, driver init/reset paths, some of the ioctl handling, to userspace. Be willing to accept extra context switches as a cost in all but the fast paths. What you have left after you move all the slow paths to userspace is my version of a microkernel. The kernel is still monolithic and a single address space, but a lot smaller. Now this (a) is likely just a pipe dream and (b) will increase complexity because each driver will _require_ a userspace component, but hey... you gotta have goals in life. Jeff ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 4:38 ` Jeff Garzik @ 2004-08-31 5:30 ` Bill Huey 2004-08-31 5:56 ` Jeff Garzik 2004-08-31 9:01 ` Tonnerre 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen 1 sibling, 2 replies; 1835+ messages in thread From: Bill Huey @ 2004-08-31 5:30 UTC (permalink / raw) To: Jeff Garzik; +Cc: Linus Torvalds, Tom Vier, linux-kernel On Tue, Aug 31, 2004 at 12:38:14AM -0400, Jeff Garzik wrote: > _My_ own definition of microkernel, which differs from traditional CS, > is where I would love to see Linux go: move as much as humanly possible > to userspace, such that, the kernel only contains pagecache operations, > driver fast paths, and security checks. Move slow paths, including ACPI > probing, PCI bus walking, driver init/reset paths, some of the ioctl > handling, to userspace. Be willing to accept extra context switches as > a cost in all but the fast paths. http://www.dragonflybsd.org DragonFly BSD, the only remotely functional open source BSD project on this planet, has plans in place to push certain kernel components like their VFS layer into userspace for easier debugging, testing and other things that go with developing file systems easily. If they back it with something like C++ for doing constructor style type conversion on top of overloaded operators to back VFS data structure translation, could easily import stuff like most Linux file systems without major restructuring, say, if they had their translation library written. In this case, userspace kernel systems have some serious programming advantages over traditional kernels. They're also pushing an async syscall model to support a non-1:1 threading system for userspace unlike what Linux has done with futexes. It'll allow for them to plug in various userspace schedulers at will for whatever scheduling policy you want dynamically, even while the program is running. This isn't mentioning their single system image clustering goals that have abstracted various kernel systems like their VFS stuff to a messaging model which should get them proccess migration as an intrinsic part of their OS design. They've made a lot of progress over the last year and it's largely because of Matt Dillon's ability to create a non-overly political group that creates trust. He's a true leader, not some random loud mouth(s) unlike other BSD efforts. It's much like the Linux community in this regard with Linus. bill ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey @ 2004-08-31 5:56 ` Jeff Garzik 2004-08-31 6:50 ` Bill Huey [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> 2004-08-31 9:01 ` Tonnerre 1 sibling, 2 replies; 1835+ messages in thread From: Jeff Garzik @ 2004-08-31 5:56 UTC (permalink / raw) To: Bill Huey (hui); +Cc: Linus Torvalds, Tom Vier, linux-kernel Bill Huey (hui) wrote: > DragonFly BSD, the only remotely functional open source BSD project on this > planet, has plans in place to push certain kernel components like their VFS > layer into userspace for easier debugging, testing and other things that go That violates Jeff's First Rule, put the fast path stuff in the kernel. > with developing file systems easily. If they back it with something like C++ > for doing constructor style type conversion on top of overloaded operators > to back VFS data structure translation, could easily import stuff like most > Linux file systems without major restructuring, say, if they had their > translation library written. In this case, userspace kernel systems have > some serious programming advantages over traditional kernels. Sounds like security would be a pain in the ass :) > They're also pushing an async syscall model to support a non-1:1 threading > system for userspace unlike what Linux has done with futexes. It'll allow messaging passing is also known as "really slow C function calls" My PCI bus passes messages all the time. A message is in the eye of the beholder. Jeff ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 5:56 ` Jeff Garzik @ 2004-08-31 6:50 ` Bill Huey [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> 1 sibling, 0 replies; 1835+ messages in thread From: Bill Huey @ 2004-08-31 6:50 UTC (permalink / raw) To: Jeff Garzik; +Cc: Bill Huey (hui), Linus Torvalds, Tom Vier, linux-kernel On Tue, Aug 31, 2004 at 01:56:45AM -0400, Jeff Garzik wrote: > Bill Huey (hui) wrote: > >DragonFly BSD, the only remotely functional open source BSD project on this > >planet, has plans in place to push certain kernel components like their VFS > >layer into userspace for easier debugging, testing and other things that go > > That violates Jeff's First Rule, put the fast path stuff in the kernel. It's not for native file system, but one that are coming around. I kind of misspoke when I said that the VFS layer is going to be pushed into userspace. It's not. Access to the VFS layer is going to be exported to userspace. The system is interesting because of its potential programming flexibility. > >with developing file systems easily. If they back it with something like > >C++ > >for doing constructor style type conversion on top of overloaded operators > >to back VFS data structure translation, could easily import stuff like most > >Linux file systems without major restructuring, say, if they had their > >translation library written. In this case, userspace kernel systems have > >some serious programming advantages over traditional kernels. > > Sounds like security would be a pain in the ass :) Don't know. But if you can get away from the typical C programming errors, then it could be a win. > >They're also pushing an async syscall model to support a non-1:1 threading > >system for userspace unlike what Linux has done with futexes. It'll allow > > messaging passing is also known as "really slow C function calls" > > My PCI bus passes messages all the time. A message is in the eye of the > beholder. Message passing is used as a method of concurrency in that system outside of the typical use of function calles. It makes access to kernel facilities, remote or local, transparent. They also don't use a mutex for their locking, but something called a token which is per-CPU kind of locking primitive. It's definitely an interesting system with a lot of potential. bill ^ permalink raw reply [flat|nested] 1835+ messages in thread
[parent not found: <20040831155613.2b25df1e.diegocg@teleline.es>]
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> @ 2004-08-31 20:52 ` Bill Huey 2004-09-02 17:20 ` Diego Calleja 2004-08-31 21:23 ` Bill Huey 1 sibling, 1 reply; 1835+ messages in thread From: Bill Huey @ 2004-08-31 20:52 UTC (permalink / raw) To: Diego Calleja; +Cc: Jeff Garzik, bhuey, torvalds, tmv, linux-kernel On Tue, Aug 31, 2004 at 03:56:13PM +0200, Diego Calleja wrote: > El Tue, 31 Aug 2004 01:56:45 -0400 Jeff Garzik <jgarzik@pobox.com> escribi?: > > messaging passing is also known as "really slow C function calls" > > > > My PCI bus passes messages all the time. A message is in the eye of the > > beholder. > > There're some numbers for that: > > http://lists.freebsd.org/pipermail/freebsd-hackers/2003-August/002426.html As you can see the numbers are very fast for a general purpose system like that. Add that with their XIO framework for data propagation and you have a really amazingly fast system that combines both aggressive design and optimized low level facilities for both remote and local commuication. Matt's one of the very few open source engineers on this planet that can do both well. bill ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 20:52 ` Bill Huey @ 2004-09-02 17:20 ` Diego Calleja 2004-09-02 21:37 ` Bill Huey 0 siblings, 1 reply; 1835+ messages in thread From: Diego Calleja @ 2004-09-02 17:20 UTC (permalink / raw) To: Bill Huey; +Cc: jgarzik, bhuey, torvalds, tmv, linux-kernel El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <bhuey@lnxw.com> escribió: > As you can see the numbers are very fast for a general purpose system > like that. Add that with their XIO framework for data propagation and For the syscall case, I remember that they would be able to "batch" syscalls, so this framework could have even better performance in some cases. The good thing is that they didn't do it just for the sake of doing it (like microkernel people) but to achieve their goals (SSI, etc) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-09-02 17:20 ` Diego Calleja @ 2004-09-02 21:37 ` Bill Huey 0 siblings, 0 replies; 1835+ messages in thread From: Bill Huey @ 2004-09-02 21:37 UTC (permalink / raw) To: Diego Calleja; +Cc: Bill Huey, jgarzik, torvalds, tmv, linux-kernel On Thu, Sep 02, 2004 at 07:20:58PM +0200, Diego Calleja wrote: > El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <bhuey@lnxw.com> escribi?: > > > As you can see the numbers are very fast for a general purpose system > > like that. Add that with their XIO framework for data propagation and > > For the syscall case, I remember that they would be able to "batch" syscalls, > so this framework could have even better performance in some cases. > The good thing is that they didn't do it just for the sake of doing it > (like microkernel people) but to achieve their goals (SSI, etc) For what they're doing, they have minimize the overhead cross boundaries as much as possible for usage in both user and kernel spaces. If dfBSD achieves what they intend to do, they'll have on of the best SSI systems on the planet and it'll be able to use the VFS layer in ways that other systems can't even touch with iSCSI drivers, etc... They've got process suspend and restore already in order to support serialization across machines. Having it VFS driven, VM objects, etc... makes the concept of migration a function of the VFS layer. It's quite clever and I'm sure that it's driven by Matt's experience with distributed databases, specifically Backplane. All locking in his system is CPU local, no mutexes of any sort, so migration as a function of messaging, scheduling and other things are done in per-CPU chunks. It's quite clever make handling in per-process resources chunks. http://twiki.im.ufba.br/pub/MAT154/WebHome/duality78.pdf Talks about part of this and how "procedural" is isomorphic to "message-passing" concurrency system. One can be transformed to another. It's a project that's move much faster in the area of SMP than FreeBSD 5.x (Solaris style SMP) and really the only BSD project on this planet that has the talent and social chemistry to make it happen. bill ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> 2004-08-31 20:52 ` Bill Huey @ 2004-08-31 21:23 ` Bill Huey 1 sibling, 0 replies; 1835+ messages in thread From: Bill Huey @ 2004-08-31 21:23 UTC (permalink / raw) To: Diego Calleja; +Cc: Jeff Garzik, bhuey, torvalds, tmv, linux-kernel On Tue, Aug 31, 2004 at 03:56:13PM +0200, Diego Calleja wrote: > There're some numbers for that: > > http://lists.freebsd.org/pipermail/freebsd-hackers/2003-August/002426.html Link about XIO and things hijacked from FreeBSD-current. http://leaf.dragonflybsd.org/mailarchive/kernel/2004-03/msg00469.html bill ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 2004-08-31 5:56 ` Jeff Garzik @ 2004-08-31 9:01 ` Tonnerre 2004-08-31 11:15 ` Userspace file systems Luca Ferroni 2004-08-31 20:53 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 1 sibling, 2 replies; 1835+ messages in thread From: Tonnerre @ 2004-08-31 9:01 UTC (permalink / raw) To: Bill Huey; +Cc: Jeff Garzik, Linus Torvalds, Tom Vier, linux-kernel [-- Attachment #1: Type: text/plain, Size: 841 bytes --] Salut, On Mon, Aug 30, 2004 at 10:30:55PM -0700, Bill Huey wrote: > DragonFly BSD, the only remotely functional open source BSD project on this > planet, has plans in place to push certain kernel components like their VFS > layer into userspace for easier debugging, testing and other things that go > with developing file systems easily. If they back it with something like C++ > for doing constructor style type conversion on top of overloaded operators > to back VFS data structure translation, could easily import stuff like most > Linux file systems without major restructuring, say, if they had their > translation library written. In this case, userspace kernel systems have > some serious programming advantages over traditional kernels. Uh, what about enhancing lufs code in this regard and porting all fs'es to it? Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Userspace file systems 2004-08-31 9:01 ` Tonnerre @ 2004-08-31 11:15 ` Luca Ferroni 2004-08-31 20:53 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 1 sibling, 0 replies; 1835+ messages in thread From: Luca Ferroni @ 2004-08-31 11:15 UTC (permalink / raw) To: Tonnerre; +Cc: linux-kernel, miklos Il Tue, 31 Aug 2004 11:01:58 +0200, Tonnerre <tonnerre@thundrix.ch> ha scritto: > Uh, what about enhancing lufs code in this regard and porting all fs'es to it? > I think that porting all file systems to LUFS is a mad idea :), but providing a facility in the kernel for user-space file system implementations could be great. I am working on PackageFS (http://packagefs.sourceforge.net) which is an user-space file system to manage packages: it relies on LUFS, but I plan to port it soon to FUSE ( http://sourceforge.net/projects/avf) that, at present, is maintained by Miklos Szeredi and works with latest kernels. PackageFS relies also on existent package managers, it will be transparent to all package formats. I. e.: you can install a package by copying, and see package informations by a simple cat. Now it works only in Debian distro, and it is far from complete (it is 0.09 version :) ). So, what do you think about including a kernel module to make a user able to develop, install and use an user-space file system ? Maybe Miklos Szeredi will write more details about FUSE which is the interesting part for kernel inclusion. Bye Luca -- ----------------------------------------------- They'll never stole us our.... FREEDOM!!! Luca Ferroni ICQ #317977679 www.cs.unibo.it/~fferroni/ ----------------------------------------------- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 9:01 ` Tonnerre 2004-08-31 11:15 ` Userspace file systems Luca Ferroni @ 2004-08-31 20:53 ` Bill Huey 1 sibling, 0 replies; 1835+ messages in thread From: Bill Huey @ 2004-08-31 20:53 UTC (permalink / raw) To: Tonnerre; +Cc: Bill Huey, Jeff Garzik, Linus Torvalds, Tom Vier, linux-kernel On Tue, Aug 31, 2004 at 11:01:58AM +0200, Tonnerre wrote: > Salut, > > On Mon, Aug 30, 2004 at 10:30:55PM -0700, Bill Huey wrote: ... > > translation library written. In this case, userspace kernel systems have > > some serious programming advantages over traditional kernels. > > Uh, what about enhancing lufs code in this regard and porting all fs'es to it? I wasn't aware that your project has gotten this far. I'll tell Hiten about this to see if some of this can be used in their project. Thanks bill ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 4:38 ` Jeff Garzik 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey @ 2004-08-31 19:01 ` Ryan Breen 2004-08-31 19:12 ` Jeff Garzik 2004-09-01 15:37 ` Clay Haapala 1 sibling, 2 replies; 1835+ messages in thread From: Ryan Breen @ 2004-08-31 19:01 UTC (permalink / raw) To: Jeff Garzik, linux-kernel On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote: > > Man, this thread came a long way. > You said it -- from Reiser to microkernel. If we can only figure out a way to get a BitKeeper discussion going, we'll have the Grand Unified Flamewar. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen @ 2004-08-31 19:12 ` Jeff Garzik 2004-09-01 15:37 ` Clay Haapala 1 sibling, 0 replies; 1835+ messages in thread From: Jeff Garzik @ 2004-08-31 19:12 UTC (permalink / raw) To: Ryan Breen; +Cc: linux-kernel Ryan Breen wrote: > On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote: > >>Man, this thread came a long way. >> > > > You said it -- from Reiser to microkernel. If we can only figure out > a way to get a BitKeeper discussion going, we'll have the Grand > Unified Flamewar. If you can find a link between the reiserfs thread and the webcam driver thread, you've gone all the way from filesystems to Nazis, thus invoking Godwin's Law ;-) Jeff [1] http://en.wikipedia.org/wiki/Godwin's_law ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen 2004-08-31 19:12 ` Jeff Garzik @ 2004-09-01 15:37 ` Clay Haapala 1 sibling, 0 replies; 1835+ messages in thread From: Clay Haapala @ 2004-09-01 15:37 UTC (permalink / raw) To: Ryan Breen; +Cc: Jeff Garzik, linux-kernel On Tue, 31 Aug 2004, Ryan Breen verbalised: > On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote: >> >> Man, this thread came a long way. >> > > You said it -- from Reiser to microkernel. If we can only figure out > a way to get a BitKeeper discussion going, we'll have the Grand > Unified Flamewar. OK, I'll byte: consider file versioning of an SCM implemented as a ReiserFS 4 plug-in, as in: diff -u ultrastore.c/delta/3.2.3 ultrastore.c/delta/3.2.4 (Brrr! Just had a VMS flash-back shiver.) -- Clay Haapala (chaapala@cisco.com) Cisco Systems Storage +1 763-398-1056 6450 Wedgwood Rd, Suite 130 Maple Grove MN 55311 PGP: C89240AD The mysterious blob seen in satellite photos off the east coast of Florida during the passing of Hurricane Charley has now been positively identified as a large cloud of Democrat absentee-ballot applications and voter lists. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 4:08 ` Linus Torvalds 2004-08-31 4:38 ` Jeff Garzik @ 2004-08-31 10:57 ` Alan Cox 2004-08-31 17:15 ` Linus Torvalds 2004-08-31 19:45 ` Toon van der Pas 2 siblings, 1 reply; 1835+ messages in thread From: Alan Cox @ 2004-08-31 10:57 UTC (permalink / raw) To: Linus Torvalds; +Cc: Tom Vier, Linux Kernel Mailing List On Maw, 2004-08-31 at 05:08, Linus Torvalds wrote: > > What about microkernels? They do tcp in userspace. > > No they don't. They do TCP in a separate address space from user space, > that just also happens to be separate from the "microkernel address > space". Several do TCP in user space. The only thing you need in kernel for TCP/IP is enough decode to decide who gets the packet. Even some non microkernel embedded OS's do this in order to keep kernel size down. Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 10:57 ` Alan Cox @ 2004-08-31 17:15 ` Linus Torvalds 2004-08-31 21:13 ` Frank van Maarseveen 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-31 17:15 UTC (permalink / raw) To: Alan Cox; +Cc: Tom Vier, Linux Kernel Mailing List On Tue, 31 Aug 2004, Alan Cox wrote: > > Several do TCP in user space. The only thing you need in kernel for > TCP/IP is enough decode to decide who gets the packet. Only thing? I don't think so. You also want to make sure that regular users cannot send "impossible" packets. Think about the old "ping of death" kind of thing, where a normal mis-behaving (and I'm not saying intentionally so: it might be a small bug that just overwrites some data) program should _not_ be able to cause problems on the network. Admins absolutely _hate_ that. They will ban an OS if it sends out packets that cause troublem. You should remember that - we used to do strange things on the net (long long time ago), and we brought down servers by mistake, and nobody ever considered it a server bug: it was a Linux bug that it wouldn't do the right thing. Things like not sending FIN-packets when a program suddenly goes away is NOT acceptable behaviour! Neither is it acceptable behaviour to allow user programs to make up their own packets. > Even some non microkernel embedded OS's do this in order to keep kernel > size down. ..and I'm not disagreeing that it doesn't happen. I explicitly mentioned PalmOS, I bet it happens in other cases too. But I'd strongly argue that it's a bug, not a feature. It's a bug that people tend to accept in a "single-client" environment. NOTE! This is totally ignoring the fact that you can't be called "UNIX" any more. You _need_ to have sequence numbers etc be shared between multiple programs that all write to the stream. Again, that _does_ mean that you have another protection domain (aka "kernel" or "TCP deamon") that keeps track of the sequence number. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 17:15 ` Linus Torvalds @ 2004-08-31 21:13 ` Frank van Maarseveen 2004-08-31 21:25 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: Frank van Maarseveen @ 2004-08-31 21:13 UTC (permalink / raw) To: Linus Torvalds; +Cc: Alan Cox, Tom Vier, Linux Kernel Mailing List On Tue, Aug 31, 2004 at 10:15:25AM -0700, Linus Torvalds wrote: > > Admins absolutely _hate_ that. They will ban an OS if it sends out packets > that cause troublem. You should remember that - we used to do strange > things on the net (long long time ago), and we brought down servers by > mistake, and nobody ever considered it a server bug: it was a Linux bug > that it wouldn't do the right thing. > > Things like not sending FIN-packets when a program suddenly goes away is > NOT acceptable behaviour! Neither is it acceptable behaviour to allow user > programs to make up their own packets. The user/kernel distinction is not always (heck, maybe almost never) present in the embedded world (a large world). The notion of a "regular user" does not apply at all in such a case. This is not a bug but merely the state of technology. It will slowly go away I think because embedded software becomes more complex and hardware becomes cheaper. To implement multiple TCP clients _and_ the TCP/IP stack in one space is perfectly possible and it's actually done in practice. It has advantages (speed, when done properly) and disadvantages (complexity/bug-prone). > > NOTE! This is totally ignoring the fact that you can't be called "UNIX" > any more. You _need_ to have sequence numbers etc be shared between > multiple programs that all write to the stream. Again, that _does_ mean > that you have another protection domain (aka "kernel" or "TCP deamon") > that keeps track of the sequence number. There is nothing in the networking or UNIX standards that prescibe another protection domain for this. Would be insane to leave that out in a hosted environment but it _can_ be done without. -- Frank ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 21:13 ` Frank van Maarseveen @ 2004-08-31 21:25 ` Linus Torvalds 0 siblings, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-31 21:25 UTC (permalink / raw) To: Frank van Maarseveen; +Cc: Alan Cox, Tom Vier, Linux Kernel Mailing List On Tue, 31 Aug 2004, Frank van Maarseveen wrote: > > There is nothing in the networking or UNIX standards that prescibe another > protection domain for this. Would be insane to leave that out in a hosted > environment but it _can_ be done without. My point is that TCP _does_ have a lot of state that needs to be handled in a safe manner by a proper operating system. The fact that there are OS's out there that are crap doesn't change that matter. There are lots of embedded OS's out there that still do multitasking in a purely cooperative way. I don't think it's a valid model for anything but toys. Same goes for putting TCP in user space. It's doable, but it's not an "OS". It's a program loader. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 4:08 ` Linus Torvalds 2004-08-31 4:38 ` Jeff Garzik 2004-08-31 10:57 ` Alan Cox @ 2004-08-31 19:45 ` Toon van der Pas 2 siblings, 0 replies; 1835+ messages in thread From: Toon van der Pas @ 2004-08-31 19:45 UTC (permalink / raw) To: linux-kernel On Mon, Aug 30, 2004 at 09:08:07PM -0700, Linus Torvalds wrote: > > > On Mon, 30 Aug 2004, Tom Vier wrote: > > > On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: > > > - safely synchronize globally visible data structures > > > That's quite fundamental. 99% of what a kernel does is exactly > > > that. TCP would be in user space too, if it wasn't for > > > _exactly_ this issue. A lot > > > > What about microkernels? They do tcp in userspace. > > No they don't. They do TCP in a separate address space from user > space, that just also happens to be separate from the "microkernel > address space". The HP NonStop Server (formerly called Tandem) has an MPP architecture and a message passing OS (the NonStop Kernel). On this OS the TCP/IP stack is implemented as a library against which the application software is linked. The ethernet controllers/drivers maintain a routing table for routing incoming traffic to the right process. So the TCP/IP stack is implemented in userspace on the NonStop Server, I would say. Regards, Toon. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:32 ` Jan Harkes @ 2004-08-26 17:14 ` Hubert Chan 2004-08-26 17:14 ` Hubert Chan 1 sibling, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-26 17:14 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Jan" == Jan Harkes <jaharkes@cs.cmu.edu> writes: Jan> On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus Törnqvist wrote: [...] >> I for one would truly welcome the coming of thumbnails and >> descriptions in picture files, because I have a real-life project >> going on where that would be extremely handy to have in the actual >> file. Jan> Ehhh, that already exists and doesn't require meta-data streams, Jan> http://www.exif.org/ but then you need an extra library to access the exif data. You can't just do something like "display foo.jpg/exif/thumbnail". Or even better, "echo 'this is a comment' > foo.jpg/comment" to change the jpeg comment without an external program. And I don't think exif tags work with all image formats. >> On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: >> >Find some silly person with an iBook and open a shell on OS X. Use >> >cp to copy a file with a resource fork. Oh look, the Finder has no >> >idea what the new file is, even though it looks exactly identical in >> >the shell. Isn't that _wonderful_? Now try cat < a > b on a file >> >with a fork. How is that ever going to work? Under file-as-dir, it would not look the same under the shell if cp didn't copy everything -- the directory contents would be different. To copy the metadata as well, you would do something like "cp -r". Similarly with "cat < a > b". -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 17:14 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-26 17:14 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Jan" == Jan Harkes <jaharkes@cs.cmu.edu> writes: Jan> On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus Törnqvist wrote: [...] >> I for one would truly welcome the coming of thumbnails and >> descriptions in picture files, because I have a real-life project >> going on where that would be extremely handy to have in the actual >> file. Jan> Ehhh, that already exists and doesn't require meta-data streams, Jan> http://www.exif.org/ but then you need an extra library to access the exif data. You can't just do something like "display foo.jpg/exif/thumbnail". Or even better, "echo 'this is a comment' > foo.jpg/comment" to change the jpeg comment without an external program. And I don't think exif tags work with all image formats. >> On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: >> >Find some silly person with an iBook and open a shell on OS X. Use >> >cp to copy a file with a resource fork. Oh look, the Finder has no >> >idea what the new file is, even though it looks exactly identical in >> >the shell. Isn't that _wonderful_? Now try cat < a > b on a file >> >with a fork. How is that ever going to work? Under file-as-dir, it would not look the same under the shell if cp didn't copy everything -- the directory contents would be different. To copy the metadata as well, you would do something like "cp -r". Similarly with "cat < a > b". -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall @ 2004-08-26 8:31 ` Helge Hafting 2004-08-26 7:53 ` Markus Törnqvist ` (3 subsequent siblings) 4 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-26 8:31 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Matt Mackall wrote: [...] >What it breaks is the concept of a file. In ways that are ill-defined, >not portable, hard to work with, and needlessly complex. Along the >way, it breaks every single application that ever thought it knew what >a file was. > >Progress? No, this has been done before. Various dead operating >systems have done it or similar and regretted it. Most recently MacOS, >which jumped through major hurdles to begin purging themselves of >resource forks when they went to OS X. They're still there, but >heavily deprecated. > > > >>cp, grep, cat, and tar will continue to work just fine on files with >>multiple streams. >> >> > >Find some silly person with an iBook and open a shell on OS X. Use cp >to copy a file with a resource fork. Oh look, the Finder has no idea >what the new file is, even though it looks exactly identical in the >shell. Isn't that _wonderful_? > It is what I'd expect. Now, use cp -R to copy the file _with its directory_, and see if that fares better. If not - bad implementation of fs and/or cp. The way I see file-as -directory is that _file_ operations (like the reads issued by cat) only work on the file part. You want the directory part? Use directory operations such as those "cp -R" use. >Now try cat < a > b on a file with a >fork. How is that ever going to work? > > It is going to copy the _file_ part, of course. I wouldn't expect anything else - "cat" doesn't deal with directories. This also raise the question of when to use file-as-directory. A usage where you need everything to follow the file, even when using "cat" calls out for an application that puts everything into one file. Directory-as-file is the wrong tool for that job, so don't worry about such problems. Sticking thumbnails in a file-as-dir is another story though. Move the file (with "mv a b", not "cat a>b;rm a") moves the image file _and_ the thumbnail. No time wasted on regenerating thumbnails, no disk space or cleanup time wasted on dangling thumbs. Use the image file for its intended purposes (view it, mail it off, serve it on the web, edit it, print it) and the thumbnail doesn't get in the way because it isn't embedded in the file format. Embedding it in the file might work for jpeg which support generic embedded data, it surely won't work for every image format out there. This is my idea of how file-as-directory should work. Helge Hafting Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:31 ` Helge Hafting 0 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-26 8:31 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Matt Mackall wrote: [...] >What it breaks is the concept of a file. In ways that are ill-defined, >not portable, hard to work with, and needlessly complex. Along the >way, it breaks every single application that ever thought it knew what >a file was. > >Progress? No, this has been done before. Various dead operating >systems have done it or similar and regretted it. Most recently MacOS, >which jumped through major hurdles to begin purging themselves of >resource forks when they went to OS X. They're still there, but >heavily deprecated. > > > >>cp, grep, cat, and tar will continue to work just fine on files with >>multiple streams. >> >> > >Find some silly person with an iBook and open a shell on OS X. Use cp >to copy a file with a resource fork. Oh look, the Finder has no idea >what the new file is, even though it looks exactly identical in the >shell. Isn't that _wonderful_? > It is what I'd expect. Now, use cp -R to copy the file _with its directory_, and see if that fares better. If not - bad implementation of fs and/or cp. The way I see file-as -directory is that _file_ operations (like the reads issued by cat) only work on the file part. You want the directory part? Use directory operations such as those "cp -R" use. >Now try cat < a > b on a file with a >fork. How is that ever going to work? > > It is going to copy the _file_ part, of course. I wouldn't expect anything else - "cat" doesn't deal with directories. This also raise the question of when to use file-as-directory. A usage where you need everything to follow the file, even when using "cat" calls out for an application that puts everything into one file. Directory-as-file is the wrong tool for that job, so don't worry about such problems. Sticking thumbnails in a file-as-dir is another story though. Move the file (with "mv a b", not "cat a>b;rm a") moves the image file _and_ the thumbnail. No time wasted on regenerating thumbnails, no disk space or cleanup time wasted on dangling thumbs. Use the image file for its intended purposes (view it, mail it off, serve it on the web, edit it, print it) and the thumbnail doesn't get in the way because it isn't embedded in the file format. Embedding it in the file might work for jpeg which support generic embedded data, it surely won't work for every image format out there. This is my idea of how file-as-directory should work. Helge Hafting Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Helge Hafting (?) @ 2004-08-26 18:17 ` Horst von Brand 2004-08-28 16:49 ` Helge Hafting -1 siblings, 1 reply; 1835+ messages in thread From: Horst von Brand @ 2004-08-26 18:17 UTC (permalink / raw) To: Helge Hafting Cc: Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Helge Hafting <helge.hafting@hist.no> said: > Matt Mackall wrote: [...] > >Find some silly person with an iBook and open a shell on OS X. Use cp > >to copy a file with a resource fork. Oh look, the Finder has no idea > >what the new file is, even though it looks exactly identical in the > >shell. Isn't that _wonderful_? > It is what I'd expect. Great. That means that all the tools of the trade stop working. Sounds like show-killer feature to me. > Now, use cp -R to copy the file > _with its directory_, Either it is a file or a directory. Make up your mind. If you have no clear distinction, you'll only get messed up. Badly. > and see if that fares better. If not - bad > implementation of fs and/or cp. The way I see file-as -directory > is that _file_ operations (like the reads issued by cat) only > work on the file part. You want the directory part? Use > directory operations such as those "cp -R" use. Excuse me, I must grab my sickness bag here. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:17 ` Horst von Brand @ 2004-08-28 16:49 ` Helge Hafting 0 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-28 16:49 UTC (permalink / raw) To: Horst von Brand Cc: Helge Hafting, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 02:17:03PM -0400, Horst von Brand wrote: > Helge Hafting <helge.hafting@hist.no> said: > > Matt Mackall wrote: > > [...] > > > >Find some silly person with an iBook and open a shell on OS X. Use cp > > >to copy a file with a resource fork. Oh look, the Finder has no idea > > >what the new file is, even though it looks exactly identical in the > > >shell. Isn't that _wonderful_? > > > It is what I'd expect. > > Great. That means that all the tools of the trade stop working. Sounds like > show-killer feature to me. > All wrong. Feel free to dislike the concept all you want, but don't provide false arguments. I am sure you can think up some _good_ ones, if you feel so strongly about this. _No_ tool "stop working" with such a fs. Of course the new feature (file-as-directory) may not be supported - which is a problem only for files that use the new feature. Stick a lot of ordinary files and directories in such a fs, and the tools works as before with no breakage. People obviously won't deploy new features before they are useable - the fs have to come first. Then the tools, and widespread use only after that. This is similiar to other extensions like ACLs which many tools don't understand. They're still useable though. Of course file-as-directory can have pilot uses like the often mentioned thumbnails - because it doesn't matter if those aren't backed up or get lost at random during the experimental phase. > > Now, use cp -R to copy the file > > _with its directory_, > > Either it is a file or a directory. Make up your mind. Wrong again. The file-as-directory concept means that a filename refers to something that is a file, or a directory, or both. That's the point of it. Being both isn't possible with current filesystems (except reiser4) of course, but file-as-directory is a proposed _extension_. > If you have no clear > distinction, you'll only get messed up. Badly. > The distinction is clear enough - three options instead of two. > Excuse me, I must grab my sickness bag here. Oh, very impressive. Please take a look at what we're talking about before attacking. Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall @ 2004-08-26 9:59 ` Spam 2004-08-26 7:53 ` Markus Törnqvist ` (3 subsequent siblings) 4 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 9:59 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list >> cp, grep, cat, and tar will continue to work just fine on files with >> multiple streams. > Find some silly person with an iBook and open a shell on OS X. Use cp > to copy a file with a resource fork. Oh look, the Finder has no idea > what the new file is, even though it looks exactly identical in the > shell. Isn't that _wonderful_? Now try cat < a > b on a file with a > fork. How is that ever going to work? It should work on the main file, the extra streams and meta data won't be copied. Just as it is now with special attributes etc. Cat reads the content of a file stream, not the file container. > I like cat < a > b. You can keep your progress. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 9:59 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 9:59 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list >> cp, grep, cat, and tar will continue to work just fine on files with >> multiple streams. > Find some silly person with an iBook and open a shell on OS X. Use cp > to copy a file with a resource fork. Oh look, the Finder has no idea > what the new file is, even though it looks exactly identical in the > shell. Isn't that _wonderful_? Now try cat < a > b on a file with a > fork. How is that ever going to work? It should work on the main file, the extra streams and meta data won't be copied. Just as it is now with special attributes etc. Cat reads the content of a file stream, not the file container. > I like cat < a > b. You can keep your progress. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall ` (3 preceding siblings ...) 2004-08-26 9:59 ` Spam @ 2004-08-26 13:05 ` Joseph Fannin 4 siblings, 0 replies; 1835+ messages in thread From: Joseph Fannin @ 2004-08-26 13:05 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2248 bytes --] On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: > What it breaks is the concept of a file. In ways that are ill-defined, > not portable, hard to work with, and needlessly complex. Along the > way, it breaks every single application that ever thought it knew what > a file was. > > Progress? No, this has been done before. Various dead operating > systems have done it or similar and regretted it. Most recently MacOS, > which jumped through major hurdles to begin purging themselves of > resource forks when they went to OS X. They're still there, but > heavily deprecated. > > > cp, grep, cat, and tar will continue to work just fine on files with > > multiple streams. > > Find some silly person with an iBook and open a shell on OS X. Use cp > to copy a file with a resource fork. Oh look, the Finder has no idea > what the new file is, even though it looks exactly identical in the > shell. Isn't that _wonderful_? Now try cat < a > b on a file with a > fork. How is that ever going to work? I've been known to be silly now and then, for sure. Apple says this about the next version of OS X, due next year: "Tiger provides a standard, Darwin-level API for managing resource forks, filesystem metadata, security information, properties and other attributes in a consistent, cross-platform manner. For example, common UNIX utilities such as cp, tar and rsync can properly handle HFS+ resource forks." That's marketing copy, but I think it's worth noting. It seems to be that resource forks aren't deprecated across the board, just... changed. Apple has found a way to make this work, at least partially. I don't know if it's a good way, but there it is. I'd like to be able to keep my Macintosh stuff on my Linux fileserver, without losing the resource forks. That would require that the network fs support it of course, and whether I could use tar to back up my linux box would depend on how streams might be implemented there. > I like cat < a > b. You can keep your progress. Can't we have both? -- Joseph Fannin jhf@rivenstone.net "Bull in pure form is rare; there is usually some contamination by data." -- William Graves Perry Jr. [-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:09 ` Nicholas Miell 2004-08-26 5:17 ` James Morris 2004-08-26 5:32 ` Matt Mackall @ 2004-08-26 6:53 ` Paul Jackson 2004-08-26 9:54 ` Spam 3 siblings, 0 replies; 1835+ messages in thread From: Paul Jackson @ 2004-08-26 6:53 UTC (permalink / raw) To: Nicholas Miell Cc: mpm, wichert, jra, akpm, spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all > the time and are fixed. It's called progress. Yes - we break things all the time. But we take notice, when we are breaking things, of how long standing and deeply embedded they are. The deeper the roots, the more respect we show it, and the harder it will be to change. Heaping scorn on someone who is reluctant to change something as deeply embedded as "a file is a byte stream" does not further the discussion. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:09 ` Nicholas Miell @ 2004-08-26 9:54 ` Spam 2004-08-26 5:32 ` Matt Mackall ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 9:54 UTC (permalink / raw) To: Nicholas Miell Cc: Matt Mackall, Wichert Akkerman, Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > cp, grep, cat, and tar will continue to work just fine on files with > multiple streams. > tar and cp will lose the extra streams until somebody fixes them, but > they lose ACLs and xattrs right now, and I don't hear anybody suggesting > that ACLs and xattrs be removed from the kernel because of this. I thought that the underlaying FS/VFS would make sure the extra streams would be preserved. Normal applications that doesn't have support for them should not destroy the extra streams (and meta data,attributes etc). Of course tar is different, but any probgram that uses the normal OS API to copy, write and read files should not affect the extra info. ~S > Fixing programs that do recursive filesystem traversals is a matter of a > glibc patch to nftw(3) and a modification to their option processing. > Holding back a useful feature because you don't want to upgrade > coreutils is just plain dumb. > (BTW, for email, multipart/parallel is a start, but a specific multipart > content type for multi-stream file attachments would probably be more > appropriate.) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 9:54 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 9:54 UTC (permalink / raw) To: Nicholas Miell Cc: Matt Mackall, Wichert Akkerman, Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > cp, grep, cat, and tar will continue to work just fine on files with > multiple streams. > tar and cp will lose the extra streams until somebody fixes them, but > they lose ACLs and xattrs right now, and I don't hear anybody suggesting > that ACLs and xattrs be removed from the kernel because of this. I thought that the underlaying FS/VFS would make sure the extra streams would be preserved. Normal applications that doesn't have support for them should not destroy the extra streams (and meta data,attributes etc). Of course tar is different, but any probgram that uses the normal OS API to copy, write and read files should not affect the extra info. ~S > Fixing programs that do recursive filesystem traversals is a matter of a > glibc patch to nftw(3) and a modification to their option processing. > Holding back a useful feature because you don't want to upgrade > coreutils is just plain dumb. > (BTW, for email, multipart/parallel is a start, but a specific multipart > content type for multi-stream file attachments would probably be more > appropriate.) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 4:44 ` Matt Mackall 2004-08-26 5:09 ` Nicholas Miell @ 2004-08-26 5:23 ` Jamie Lokier 2004-08-26 8:16 ` Helge Hafting 2004-08-26 9:51 ` Spam 3 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 5:23 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Matt Mackall wrote: > > Anything that currently stores a file's metadata in another file really > > wants this right now. Things like image thumbnails, document summaries, > > digital signatures, etc. > > That is _highly_ debatable. I would much rather have my cp and grep > and cat and tar and such continue to work than have to rewrite every > tool because we've thrown the file-is-a-stream-of-bytes concept out > the window. Never mind that I've got thumbnails, document summaries, > and digital signatures already. > > While the number of annoying properties of files with forks is > practically endless, the biggest has got to be utter lack of > portability. How do you stick the thing in an attachment or on an ftp > site? Well you can't because it's NOT A FILE. > > A file is a stream of bytes. I couldn't agree more. Metadata which is easily lost is a terrible place to put a "document summary". However, it's not a bad place to put thumbnails which are generated from the file contents, if they can be regenerated after transporting. It's slightly better than a ".thumbnails" directory because the latter won't follow renames and things like that. It's a very good place to put metadata which is semantically bound to the file in special ways not intended for transport, such as security attributes, auto-invalidated digests of the file's contents, an auto-unpacked view of an archive, a virtual tree of an XML or other structured file, or an auto-generated textual representation of a binary file. The key concept is "not intended for transport". Generally that means permissions metadata, and things which are re-generated from the file's contents on demand.* It's a very bad place to put an abstract of a text document, or the names of authors, or the MIME content type, or the character encoding of the document, or the name of the shell to run the file as a script, unless these things are also deducible from the file's contents. However, sometimes the content doesn't give any clue, and then something like the character encoding of a text file is usefully stored in the metadata simply because that's better than nothing. Of course the facility can be abused. What does Windows XP do? Does it only store the appropriate kind of metadata in the alternate streams? Even Windows XP users expect files transfered over HTTP or WebDAV to work, don't they? -- Jamie * - By my definition, modification time arguably shouldn't be metadata. Personally I like downloads to be stamped with their source's modification time, which is why I don't use Mozilla's download manager but cut-and-paste URLs to "wget -N" instead. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 4:44 ` Matt Mackall @ 2004-08-26 8:16 ` Helge Hafting 2004-08-26 5:23 ` Jamie Lokier ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-26 8:16 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Matt Mackall wrote: >On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > > >>On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: >> >> >>>Previously Jeremy Allison wrote: >>> >>> >>>>Multiple-data-stream files are something we should offer, definately (IMHO). >>>>I don't care how we do it, but I know it's something we need as application >>>>developers. >>>> >>>> >>>Aside from samba, is there any other application that has a use for >>>them? >>> >>> >>> >>Anything that currently stores a file's metadata in another file really >>wants this right now. Things like image thumbnails, document summaries, >>digital signatures, etc. >> >> > >That is _highly_ debatable. I would much rather have my cp and grep >and cat and tar and such continue to work than have to rewrite every >tool because we've thrown the file-is-a-stream-of-bytes concept out >the window. Never mind that I've got thumbnails, document summaries, >and digital signatures already. > > Utilities that works on files only, such as cat, should keep working. No problem there. If you cat a file that also is a directory, then the file contents is all you get - by design. Utilities that _only_ traverse the directory tree, such as find, should keep working too. Perhaps with a very minor update so they don't mistake a file-directory for a file only. I.e. find must recurse into anything that support directory semantics. Something that both recurse and operate on files (cp -a, tar, grep, ... will need minor updating. They already knows how to handle files and directories, now they will need an update so they're open for objects that are both. I.e. let grep scan the file as usual, then recurse into its directory part in the usual way too. Performance problems should be avoided by not supporting directory operations on files with no directory content, which I believe will be many of them. The "file-as-directory" thing will not be that useful before the tools gets these relatively simple updates. Till then it'll be a toy, which shouldn't stop it from getting into the VFS. Updating the tools will be a task for the file-as-dir fans. (I don't know wether reiser4 does things this way - it is certainly they way I would want file-as-directory though.) >While the number of annoying properties of files with forks is >practically endless, the biggest has got to be utter lack of >portability. How do you stick the thing in an attachment or on an ftp >site? Well you can't because it's NOT A FILE. > > Stick the file in an attachment and you get the file only. No problem, it is designed that way. An app that really wants everything in a single file should use a file structured for that, not file-as-dir. File-as-dir attach stuff to a file in a more loose way. If you want to attach the directory contents too, do what you usually do when you want to mail someone a directory tree. You can't stick a directory in an attachment because it is not a file. So you either attach every file in the tree, or use tar. In this case, an updated tar. The ftp server shouldn't be a problem. It supports both files and directories already. It may need a minor update in order to not mistake directory for file or vice-versa when someone request an operation. ftp> get filename #Will get you the contents of the file part only - by design. ftp> cd filename #Will change into the directory (if the file indeed provides one.) >A file is a stream of bytes. > > Sure. And a file with a directory support both directory and file operations. You can get the stream of bytes as usual - that's the file part. Or you can cd into the directory as usual. There isn't much overlap between file operations and directory operations, so there is little conflict the way I see it. Merely letting the tools know that being a file no longer rule out the possibilities of directory operations. Getting the VFS right is another matter of course, but I don't worry about userland tools. Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:16 ` Helge Hafting 0 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-26 8:16 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Matt Mackall wrote: >On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > > >>On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: >> >> >>>Previously Jeremy Allison wrote: >>> >>> >>>>Multiple-data-stream files are something we should offer, definately (IMHO). >>>>I don't care how we do it, but I know it's something we need as application >>>>developers. >>>> >>>> >>>Aside from samba, is there any other application that has a use for >>>them? >>> >>> >>> >>Anything that currently stores a file's metadata in another file really >>wants this right now. Things like image thumbnails, document summaries, >>digital signatures, etc. >> >> > >That is _highly_ debatable. I would much rather have my cp and grep >and cat and tar and such continue to work than have to rewrite every >tool because we've thrown the file-is-a-stream-of-bytes concept out >the window. Never mind that I've got thumbnails, document summaries, >and digital signatures already. > > Utilities that works on files only, such as cat, should keep working. No problem there. If you cat a file that also is a directory, then the file contents is all you get - by design. Utilities that _only_ traverse the directory tree, such as find, should keep working too. Perhaps with a very minor update so they don't mistake a file-directory for a file only. I.e. find must recurse into anything that support directory semantics. Something that both recurse and operate on files (cp -a, tar, grep, ... will need minor updating. They already knows how to handle files and directories, now they will need an update so they're open for objects that are both. I.e. let grep scan the file as usual, then recurse into its directory part in the usual way too. Performance problems should be avoided by not supporting directory operations on files with no directory content, which I believe will be many of them. The "file-as-directory" thing will not be that useful before the tools gets these relatively simple updates. Till then it'll be a toy, which shouldn't stop it from getting into the VFS. Updating the tools will be a task for the file-as-dir fans. (I don't know wether reiser4 does things this way - it is certainly they way I would want file-as-directory though.) >While the number of annoying properties of files with forks is >practically endless, the biggest has got to be utter lack of >portability. How do you stick the thing in an attachment or on an ftp >site? Well you can't because it's NOT A FILE. > > Stick the file in an attachment and you get the file only. No problem, it is designed that way. An app that really wants everything in a single file should use a file structured for that, not file-as-dir. File-as-dir attach stuff to a file in a more loose way. If you want to attach the directory contents too, do what you usually do when you want to mail someone a directory tree. You can't stick a directory in an attachment because it is not a file. So you either attach every file in the tree, or use tar. In this case, an updated tar. The ftp server shouldn't be a problem. It supports both files and directories already. It may need a minor update in order to not mistake directory for file or vice-versa when someone request an operation. ftp> get filename #Will get you the contents of the file part only - by design. ftp> cd filename #Will change into the directory (if the file indeed provides one.) >A file is a stream of bytes. > > Sure. And a file with a directory support both directory and file operations. You can get the stream of bytes as usual - that's the file part. Or you can cd into the directory as usual. There isn't much overlap between file operations and directory operations, so there is little conflict the way I see it. Merely letting the tools know that being a file no longer rule out the possibilities of directory operations. Getting the VFS right is another matter of course, but I don't worry about userland tools. Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 4:44 ` Matt Mackall @ 2004-08-26 9:51 ` Spam 2004-08-26 5:23 ` Jamie Lokier ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 9:51 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: >> On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: >> > Previously Jeremy Allison wrote: >> > > Multiple-data-stream files are something we should offer, definately (IMHO). >> > > I don't care how we do it, but I know it's something we need as application >> > > developers. >> > >> > Aside from samba, is there any other application that has a use for >> > them? >> > >> >> Anything that currently stores a file's metadata in another file really >> wants this right now. Things like image thumbnails, document summaries, >> digital signatures, etc. > That is _highly_ debatable. I would much rather have my cp and grep > and cat and tar and such continue to work than have to rewrite every > tool because we've thrown the file-is-a-stream-of-bytes concept out > the window. Never mind that I've got thumbnails, document summaries, > and digital signatures already. In Windows, the extra file streams are not lost or removed if you use a program that doesn't support them. They are only lost if you move the file to a file system that doesn't support the streams. Even RAR support the NTFS file streams. > While the number of annoying properties of files with forks is > practically endless, the biggest has got to be utter lack of > portability. How do you stick the thing in an attachment or on an ftp > site? Well you can't because it's NOT A FILE. > A file is a stream of bytes. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 9:51 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 9:51 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: >> On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: >> > Previously Jeremy Allison wrote: >> > > Multiple-data-stream files are something we should offer, definately (IMHO). >> > > I don't care how we do it, but I know it's something we need as application >> > > developers. >> > >> > Aside from samba, is there any other application that has a use for >> > them? >> > >> >> Anything that currently stores a file's metadata in another file really >> wants this right now. Things like image thumbnails, document summaries, >> digital signatures, etc. > That is _highly_ debatable. I would much rather have my cp and grep > and cat and tar and such continue to work than have to rewrite every > tool because we've thrown the file-is-a-stream-of-bytes concept out > the window. Never mind that I've got thumbnails, document summaries, > and digital signatures already. In Windows, the extra file streams are not lost or removed if you use a program that doesn't support them. They are only lost if you move the file to a file system that doesn't support the streams. Even RAR support the NTFS file streams. > While the number of annoying properties of files with forks is > practically endless, the biggest has got to be utter lack of > portability. How do you stick the thing in an attachment or on an ftp > site? Well you can't because it's NOT A FILE. > A file is a stream of bytes. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:42 ` Nicholas Miell ` (3 preceding siblings ...) (?) @ 2004-08-26 17:59 ` Shawn Rutledge 2004-08-26 18:17 ` Valdis.Kletnieks -1 siblings, 1 reply; 1835+ messages in thread From: Shawn Rutledge @ 2004-08-26 17:59 UTC (permalink / raw) To: reiserfs-list --- Nicholas Miell <nmiell@gmail.com> wrote: > Anything that currently stores a file's metadata in another file > really > wants this right now. Things like image thumbnails, document > summaries, > digital signatures, etc. As well as cases where the metadata is currently _in_ the file, right? Like MP3ID tags, EXIF tags, etc. And so many MP3 jukebox-server programs currently use a database to store metadata, just because they don't trust the MP3ID tags or need to store supplemental info. Being able to put this stuff in the filesystem along with the files would be so much cleaner. So I think there are plenty of uses for it. IMO the Mac resource fork was always a good idea too, and I haven't a clue why it got left out of OSX (maybe Woz "got it" but Jobs doesn't). Somebody said "we don't have to teach tar about it"; does that mean that tar in its present form would already successfully find the alternate streams? ===== . _______ Shawn T. Rutledge / KB7PWD ecloud@bigfoot.com (_ | |_) http://ecloud.org/ kb7pwd@kb7pwd.ampr.org __) | | \______________________________________________ ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:59 ` Shawn Rutledge @ 2004-08-26 18:17 ` Valdis.Kletnieks 2004-08-27 8:06 ` Hans Reiser 0 siblings, 1 reply; 1835+ messages in thread From: Valdis.Kletnieks @ 2004-08-26 18:17 UTC (permalink / raw) To: e_cloud; +Cc: reiserfs-list [-- Attachment #1: Type: text/plain, Size: 352 bytes --] On Thu, 26 Aug 2004 10:59:52 PDT, Shawn Rutledge said: > Somebody said "we don't have to teach tar about it"; does that mean > that tar in its present form would already successfully find the > alternate streams? Look at history - did anybody teach tar about xattrs or acls? If nobody cared about those, why will reiser4 metadata be any different? [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:17 ` Valdis.Kletnieks @ 2004-08-27 8:06 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: Valdis.Kletnieks; +Cc: e_cloud, reiserfs-list Valdis.Kletnieks@vt.edu wrote: > Look at history - did anybody teach tar about xattrs or acls? If nobody > >cared about those, why will reiser4 metadata be any different? > > > My funding proposals often contain budget for modifying the most important user space apps to know how to use the new features. Somebody should fund me.... ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:46 ` Wichert Akkerman @ 2004-08-26 9:40 ` Spam 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 9:40 ` Spam 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 9:40 UTC (permalink / raw) To: Wichert Akkerman Cc: Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Previously Jeremy Allison wrote: >> Multiple-data-stream files are something we should offer, definately (IMHO). >> I don't care how we do it, but I know it's something we need as application >> developers. > Aside from samba, is there any other application that has a use for > them? Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more. Also, just because we do not see all the benefits right now, doesn't mean there won't be any. ~S > Wichert. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 9:40 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 9:40 UTC (permalink / raw) To: Wichert Akkerman Cc: Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Previously Jeremy Allison wrote: >> Multiple-data-stream files are something we should offer, definately (IMHO). >> I don't care how we do it, but I know it's something we need as application >> developers. > Aside from samba, is there any other application that has a use for > them? Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more. Also, just because we do not see all the benefits right now, doesn't mean there won't be any. ~S > Wichert. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:40 ` Spam (?) @ 2004-08-26 9:49 ` Andrew Morton 2004-08-26 10:20 ` Spam ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Andrew Morton @ 2004-08-26 9:49 UTC (permalink / raw) To: Spam Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > Yes, for example documents, image files etc. The multiple data > streams can contain thumbnails, info about who is editing the file > (useful for networked files) etc. Could be used for version handling > and much more. All of which can be handled in userspace library code. What compelling reason is there for doing this in the kernel? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:49 ` Andrew Morton @ 2004-08-26 10:20 ` Spam 2004-08-26 13:36 ` Rik van Riel 2004-08-26 17:32 ` Jeremy Allison 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 10:20 UTC (permalink / raw) To: Andrew Morton Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> wrote: >> >> Yes, for example documents, image files etc. The multiple data >> streams can contain thumbnails, info about who is editing the file >> (useful for networked files) etc. Could be used for version handling >> and much more. > All of which can be handled in userspace library code. > What compelling reason is there for doing this in the kernel? Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications. Applications that support the new features will benefit, all others will continue to work without destroying data. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 10:20 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 10:20 UTC (permalink / raw) To: Andrew Morton Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> wrote: >> >> Yes, for example documents, image files etc. The multiple data >> streams can contain thumbnails, info about who is editing the file >> (useful for networked files) etc. Could be used for version handling >> and much more. > All of which can be handled in userspace library code. > What compelling reason is there for doing this in the kernel? Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications. Applications that support the new features will benefit, all others will continue to work without destroying data. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:20 ` Spam (?) @ 2004-08-26 10:24 ` Andrew Morton 2004-08-26 10:51 ` Spam -1 siblings, 1 reply; 1835+ messages in thread From: Andrew Morton @ 2004-08-26 10:24 UTC (permalink / raw) To: Spam Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > > > > Spam <spam@tnonline.net> wrote: > >> > >> Yes, for example documents, image files etc. The multiple data > >> streams can contain thumbnails, info about who is editing the file > >> (useful for networked files) etc. Could be used for version handling > >> and much more. > > > All of which can be handled in userspace library code. > > > What compelling reason is there for doing this in the kernel? > > > Because having user space tools and code will make it not work with > everything. Keeping stuff in the kernel should make the new features > transparent to the applications. > > Applications that support the new features will benefit, all others > will continue to work without destroying data. Sorry, but that all sounds a bit fluffy. Please provide some examples. (Generally, getting all of userspace to agree on a particular library is socially hard [*], but I don't see that as a reason for putting the functionality into the kernel) [*] Example: where's the library to manipulate /etc/whatever.conf? [**] [**] yes, I know about gconf. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:24 ` Andrew Morton @ 2004-08-26 10:51 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 10:51 UTC (permalink / raw) To: Andrew Morton Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> wrote: >> >> >> >> > Spam <spam@tnonline.net> wrote: >> >> >> >> Yes, for example documents, image files etc. The multiple data >> >> streams can contain thumbnails, info about who is editing the file >> >> (useful for networked files) etc. Could be used for version handling >> >> and much more. >> >> > All of which can be handled in userspace library code. >> >> > What compelling reason is there for doing this in the kernel? >> >> >> Because having user space tools and code will make it not work with >> everything. Keeping stuff in the kernel should make the new features >> transparent to the applications. >> >> Applications that support the new features will benefit, all others >> will continue to work without destroying data. > Sorry, but that all sounds a bit fluffy. Please provide some examples. We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API. However, if things are to implemented in user-space, then old applications will not work correctly and there is risk that all the extra information will be lost or corrupted. You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk. If we want things to work from the start, including with old applications then these features should be at below-application level. If it is in VFS or FS doesn't matter to me, as long as it doesn't hinder or delay progress to long. > (Generally, getting all of userspace to agree on a particular library is > socially hard [*], but I don't see that as a reason for putting the > functionality into the kernel) > [*] Example: where's the library to manipulate /etc/whatever.conf? [**] > [**] yes, I know about gconf. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 10:51 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 10:51 UTC (permalink / raw) To: Andrew Morton Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> wrote: >> >> >> >> > Spam <spam@tnonline.net> wrote: >> >> >> >> Yes, for example documents, image files etc. The multiple data >> >> streams can contain thumbnails, info about who is editing the file >> >> (useful for networked files) etc. Could be used for version handling >> >> and much more. >> >> > All of which can be handled in userspace library code. >> >> > What compelling reason is there for doing this in the kernel? >> >> >> Because having user space tools and code will make it not work with >> everything. Keeping stuff in the kernel should make the new features >> transparent to the applications. >> >> Applications that support the new features will benefit, all others >> will continue to work without destroying data. > Sorry, but that all sounds a bit fluffy. Please provide some examples. We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API. However, if things are to implemented in user-space, then old applications will not work correctly and there is risk that all the extra information will be lost or corrupted. You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk. If we want things to work from the start, including with old applications then these features should be at below-application level. If it is in VFS or FS doesn't matter to me, as long as it doesn't hinder or delay progress to long. > (Generally, getting all of userspace to agree on a particular library is > socially hard [*], but I don't see that as a reason for putting the > functionality into the kernel) > [*] Example: where's the library to manipulate /etc/whatever.conf? [**] > [**] yes, I know about gconf. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:51 ` Spam (?) @ 2004-08-26 10:55 ` Andrew Morton 2004-08-26 11:15 ` Spam ` (3 more replies) -1 siblings, 4 replies; 1835+ messages in thread From: Andrew Morton @ 2004-08-26 10:55 UTC (permalink / raw) To: Spam Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > > > > Spam <spam@tnonline.net> wrote: > >> > >> > >> > >> > Spam <spam@tnonline.net> wrote: > >> >> > >> >> Yes, for example documents, image files etc. The multiple data > >> >> streams can contain thumbnails, info about who is editing the file > >> >> (useful for networked files) etc. Could be used for version handling > >> >> and much more. > >> > >> > All of which can be handled in userspace library code. > >> > >> > What compelling reason is there for doing this in the kernel? > >> > >> > >> Because having user space tools and code will make it not work with > >> everything. Keeping stuff in the kernel should make the new features > >> transparent to the applications. > >> > >> Applications that support the new features will benefit, all others > >> will continue to work without destroying data. > > > Sorry, but that all sounds a bit fluffy. Please provide some examples. > > We already had the examples with cp and mv. Both should continue to > work and the files will still be copied. The same with Konqueror and > Nautilus. Files and their meta-files/streams/attributes will be > retained as long as applications are using the OS API. > > However, if things are to implemented in user-space, then old > applications will not work correctly and there is risk that all the > extra information will be lost or corrupted. > > You said it would be socially hard. I think it would be very much > close to impossible to get it right. Imagine that Gnome and Nautilus > would implement support for these. I doubt that cp, mv, KDE, mc, > app-xyz would implement this anytime soon and in the meantime the > data is at risk. > No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file. For example, some image file formats already support embedded metadata, do they not? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:55 ` Andrew Morton @ 2004-08-26 11:15 ` Spam 2004-08-26 11:48 ` Jamie Lokier ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:15 UTC (permalink / raw) To: Andrew Morton Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> wrote: >> >> >> >> > Spam <spam@tnonline.net> wrote: >> >> >> >> >> >> >> >> > Spam <spam@tnonline.net> wrote: >> >> >> >> >> >> Yes, for example documents, image files etc. The multiple data >> >> >> streams can contain thumbnails, info about who is editing the file >> >> >> (useful for networked files) etc. Could be used for version handling >> >> >> and much more. >> >> >> >> > All of which can be handled in userspace library code. >> >> >> >> > What compelling reason is there for doing this in the kernel? >> >> >> >> >> >> Because having user space tools and code will make it not work with >> >> everything. Keeping stuff in the kernel should make the new features >> >> transparent to the applications. >> >> >> >> Applications that support the new features will benefit, all others >> >> will continue to work without destroying data. >> >> > Sorry, but that all sounds a bit fluffy. Please provide some examples. >> >> We already had the examples with cp and mv. Both should continue to >> work and the files will still be copied. The same with Konqueror and >> Nautilus. Files and their meta-files/streams/attributes will be >> retained as long as applications are using the OS API. >> >> However, if things are to implemented in user-space, then old >> applications will not work correctly and there is risk that all the >> extra information will be lost or corrupted. >> >> You said it would be socially hard. I think it would be very much >> close to impossible to get it right. Imagine that Gnome and Nautilus >> would implement support for these. I doubt that cp, mv, KDE, mc, >> app-xyz would implement this anytime soon and in the meantime the >> data is at risk. >> > No. All of the applications which you initially identified can be > implemented by putting the various bits of data into a single file and > getting applications to agree on the format of that file. > For example, some image file formats already support embedded metadata, do > they not? Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them. Also, you are thinking _very_ narrowly now. There are thousands of file formats. Implementing the support for meta-data/ streams/ attributes in the kernel will make it possible to use this generically for all files. I use this in Windows quite much. I put information description fields for lots of different files. These descriptions are then searchable etc. I could use the command prompt to copy the files and the descriptions would still be there. Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents? ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 11:15 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:15 UTC (permalink / raw) To: Andrew Morton Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> wrote: >> >> >> >> > Spam <spam@tnonline.net> wrote: >> >> >> >> >> >> >> >> > Spam <spam@tnonline.net> wrote: >> >> >> >> >> >> Yes, for example documents, image files etc. The multiple data >> >> >> streams can contain thumbnails, info about who is editing the file >> >> >> (useful for networked files) etc. Could be used for version handling >> >> >> and much more. >> >> >> >> > All of which can be handled in userspace library code. >> >> >> >> > What compelling reason is there for doing this in the kernel? >> >> >> >> >> >> Because having user space tools and code will make it not work with >> >> everything. Keeping stuff in the kernel should make the new features >> >> transparent to the applications. >> >> >> >> Applications that support the new features will benefit, all others >> >> will continue to work without destroying data. >> >> > Sorry, but that all sounds a bit fluffy. Please provide some examples. >> >> We already had the examples with cp and mv. Both should continue to >> work and the files will still be copied. The same with Konqueror and >> Nautilus. Files and their meta-files/streams/attributes will be >> retained as long as applications are using the OS API. >> >> However, if things are to implemented in user-space, then old >> applications will not work correctly and there is risk that all the >> extra information will be lost or corrupted. >> >> You said it would be socially hard. I think it would be very much >> close to impossible to get it right. Imagine that Gnome and Nautilus >> would implement support for these. I doubt that cp, mv, KDE, mc, >> app-xyz would implement this anytime soon and in the meantime the >> data is at risk. >> > No. All of the applications which you initially identified can be > implemented by putting the various bits of data into a single file and > getting applications to agree on the format of that file. > For example, some image file formats already support embedded metadata, do > they not? Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them. Also, you are thinking _very_ narrowly now. There are thousands of file formats. Implementing the support for meta-data/ streams/ attributes in the kernel will make it possible to use this generically for all files. I use this in Windows quite much. I put information description fields for lots of different files. These descriptions are then searchable etc. I could use the command prompt to copy the files and the descriptions would still be there. Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents? ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:15 ` Spam (?) @ 2004-08-26 11:20 ` Andrew Morton 2004-08-26 11:29 ` Wichert Akkerman 2004-08-28 0:24 ` Horst von Brand -1 siblings, 2 replies; 1835+ messages in thread From: Andrew Morton @ 2004-08-26 11:20 UTC (permalink / raw) To: Spam Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > Secondly, do you expect file managers like Nautilus and Konqueror to > support every piece of file format on the planet so they could read > information directly from the documents? Sure. You're proposing that we implement a single, golden multi-stream file format in the kernel. We could just as well do that in libmultistreamfileformat.so. But I'll grant that one cannot go adding new metadata to, say, C files this way. I don't know how useful such a thing is though. Remember that my main point is that there's a lack of coordination in userspace. Hell, there's none. Putting it in-kernel forces that coordination, and may be the way to go, but it's pretty sad. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:20 ` Andrew Morton @ 2004-08-26 11:29 ` Wichert Akkerman 2004-08-28 0:24 ` Horst von Brand 1 sibling, 0 replies; 1835+ messages in thread From: Wichert Akkerman @ 2004-08-26 11:29 UTC (permalink / raw) To: Andrew Morton Cc: Spam, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Andrew Morton wrote: > But I'll grant that one cannot go adding new metadata to, say, C files this > way. I don't know how useful such a thing is though. That is actually one of the few places where a bit of metadata would be very useful. Right now there is no way to indicate in what encoding a source is written: ascii, utf-8, ucs16, etc. are all possible. But a compiler or interpreter has no good way to figure that out. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:20 ` Andrew Morton 2004-08-26 11:29 ` Wichert Akkerman @ 2004-08-28 0:24 ` Horst von Brand 1 sibling, 0 replies; 1835+ messages in thread From: Horst von Brand @ 2004-08-28 0:24 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Andrew Morton <akpm@osdl.org> said: > Spam <spam@tnonline.net> wrote: > > > > Secondly, do you expect file managers like Nautilus and Konqueror to > > support every piece of file format on the planet so they could read > > information directly from the documents? > > Sure. You're proposing that we implement a single, golden multi-stream file > format in the kernel. Which won't be able to cater for all needs, I'm afraid. Just look at /etc, and contemplate why there are so many different configuration file formats. Sure, some are just random coder preferences set in stone, but others do make sense being different. The same will happen with file formats. Heck, it is already that way, with file formats for C source code, ELF shared libraries, various graphics and audio and video formats, ... > We could just as well do that in libmultistreamfileformat.so. Right. > But I'll grant that one cannot go adding new metadata to, say, C files this > way. I don't know how useful such a thing is though. You can never know. Perhaps annotations in swahili? Alternative versions for different architectures? In object format(s)? Copyright info? Changelog? > Remember that my main point is that there's a lack of coordination in > userspace. Hell, there's none. Putting it in-kernel forces that > coordination, and may be the way to go, but it's pretty sad. I think that forcing coordination is wrong. For example, I do value having ar/tar/cpio/zip doing essentially the same thing (and being able to define and use my own way of doing it if it pleases me), and I'd strongly oppose the kernel forcing one way of doing it down my throat. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:15 ` Spam (?) (?) @ 2004-08-26 11:27 ` Wichert Akkerman -1 siblings, 0 replies; 1835+ messages in thread From: Wichert Akkerman @ 2004-08-26 11:27 UTC (permalink / raw) To: Spam Cc: Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Spam wrote: > Also, you are thinking _very_ narrowly now. There are thousands of > file formats. Implementing the support for meta-data/ streams/ > attributes in the kernel will make it possible to use this > generically for all files. And how many different formats will be used for thumbnails? How many different names to indicate a summary for a document? Having the ability to add attributes standardizes a way to access that data, but not the format and naming of those attributes. It could be a step closer to a more flexible way of handling storage, but you have to realize that a lot (if not most) of the work still needs to be done in userspace. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:15 ` Spam @ 2004-08-26 13:39 ` Rik van Riel -1 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 13:39 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Spam wrote: > Yes, JPEG, TIFF and PNG files for example. But, if you modify any of > these with an application that doesn't support the extensions then > you will loose them. OK, so we've got a choice. Either we will lose the extensions when modifying the file with an unaware program, or you lose the extensions when copying (or restoring from backup) using an unaware program. Personally I'd prefer to keep the file intact when not modifying it... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 13:39 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 13:39 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Spam wrote: > Yes, JPEG, TIFF and PNG files for example. But, if you modify any of > these with an application that doesn't support the extensions then > you will loose them. OK, so we've got a choice. Either we will lose the extensions when modifying the file with an unaware program, or you lose the extensions when copying (or restoring from backup) using an unaware program. Personally I'd prefer to keep the file intact when not modifying it... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:39 ` Rik van Riel @ 2004-08-26 13:59 ` Spam -1 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 13:59 UTC (permalink / raw) To: Rik van Riel Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Thu, 26 Aug 2004, Spam wrote: >> Yes, JPEG, TIFF and PNG files for example. But, if you modify any of >> these with an application that doesn't support the extensions then >> you will loose them. > OK, so we've got a choice. > Either we will lose the extensions when modifying the file > with an unaware program, or you lose the extensions when > copying (or restoring from backup) using an unaware program. > Personally I'd prefer to keep the file intact when not > modifying it... Backup to me is a special case, not simply a copy of files, but also retaining all the extra data, info, attributes, etc, that comes with the file. Enabling support to have all of this extra stuff below the application level it will be possible to retain everything even though applications do not support them. Backup tools, however, must know about these new features to be able to backup them. And so will every tool that directly accesses the filesystem instead of using the OS API for it. And from I learned in an earlier message, tools like cp actually work on the fs level and themselves move the data to the new file instead of letting the OS handle it. This seem to me as it could be dangerous and could also prevent any kind of enhancements to the FS unless every tool like cp is patched. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 13:59 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 13:59 UTC (permalink / raw) To: Rik van Riel Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Thu, 26 Aug 2004, Spam wrote: >> Yes, JPEG, TIFF and PNG files for example. But, if you modify any of >> these with an application that doesn't support the extensions then >> you will loose them. > OK, so we've got a choice. > Either we will lose the extensions when modifying the file > with an unaware program, or you lose the extensions when > copying (or restoring from backup) using an unaware program. > Personally I'd prefer to keep the file intact when not > modifying it... Backup to me is a special case, not simply a copy of files, but also retaining all the extra data, info, attributes, etc, that comes with the file. Enabling support to have all of this extra stuff below the application level it will be possible to retain everything even though applications do not support them. Backup tools, however, must know about these new features to be able to backup them. And so will every tool that directly accesses the filesystem instead of using the OS API for it. And from I learned in an earlier message, tools like cp actually work on the fs level and themselves move the data to the new file instead of letting the OS handle it. This seem to me as it could be dangerous and could also prevent any kind of enhancements to the FS unless every tool like cp is patched. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:59 ` Spam (?) @ 2004-08-26 14:39 ` Felipe Alfaro Solana -1 siblings, 0 replies; 1835+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 14:39 UTC (permalink / raw) To: Spam Cc: Rik van Riel, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 15:59, Spam wrote: > Backup to me is a special case, not simply a copy of files, but also > retaining all the extra data, info, attributes, etc, that comes with > the file. > > Enabling support to have all of this extra stuff below the > application level it will be possible to retain everything even > though applications do not support them. > > Backup tools, however, must know about these new features to be able > to backup them. And so will every tool that directly accesses the > filesystem instead of using the OS API for it. Thus, this kind of metadata API must be made accessible through the VFS layer, if we want to make it FS-independent (not tied to Reiser4). If the real FS does not support metadata operations, just ignore them (much like in ISO9660 ignoring write operations). > And from I learned in an earlier message, tools like cp actually > work on the fs level and themselves move the data to the new file > instead of letting the OS handle it. This seem to me as it could > be dangerous and could also prevent any kind of enhancements to the > FS unless every tool like cp is patched. I still don't understand why a copy_file() syscall is still missing. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:15 ` Spam ` (3 preceding siblings ...) (?) @ 2004-08-27 23:58 ` Horst von Brand 2004-08-29 15:17 ` Markus Törnqvist ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Horst von Brand @ 2004-08-27 23:58 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> said: > Andrew Morton <akpm@osdl.org> said: [...] > > For example, some image file formats already support embedded metadata, do > > they not? > Yes, JPEG, TIFF and PNG files for example. But, if you modify any of > these with an application that doesn't support the extensions then > you will loose them. As you will each time you muck around with your files-are-directories. Nothing new there. > Also, you are thinking _very_ narrowly now. There are thousands of > file formats. Implementing the support for meta-data/ streams/ > attributes in the kernel will make it possible to use this > generically for all files. So the _kernel_ has to know about thousands of formats, just in case it some blue day it comes across a strange file? Better leave that to the applications. You will _not_ be able to define a single format for extra data about the file, there will be differing extra data for different users (do you suggest a root-only file with special writable pouches for "graphical icon for the file" for each user on the system?!), so the idea in itself is doomed from the start. > I use this in Windows quite much. Then use it and be happy. No need to screw up Linux for that. > I put information description > fields for lots of different files. These descriptions are then > searchable etc. I could use the command prompt to copy the files and > the descriptions would still be there. The descriptions might make sense to _you_, _now_. No guarantee they make any sense (or are in the least useful) for other users on your system, and I might want them in arabic or some such. The descriptions might make no sense to you in a couple of years. > Secondly, do you expect file managers like Nautilus and Konqueror to > support every piece of file format on the planet so they could read > information directly from the documents? That's their (self-selected) job, yes. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 23:58 ` Horst von Brand @ 2004-08-29 15:17 ` Markus Törnqvist 2004-08-29 16:18 ` Spam 2004-08-29 19:10 ` Tonnerre 2 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-29 15:17 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: > >So the _kernel_ has to know about thousands of formats, just in case it >some blue day it comes across a strange file? Better leave that to the >applications. Or come up with a way to export some interface to userspace. Or make a standard description format in something like XML that can be used for arbitrary files and have userspace interface with that metadata stream that is the XML. This is, however, dangerously close to what we have now and does. not. work, so the only added bonus we might have is the fact that it's endorsed by the kernel. I don't like this idea except as a last fallback solution... >> I use this in Windows quite much. >Then use it and be happy. No need to screw up Linux for that. Linux does not need to be screwed up by that. It can be done right. >The descriptions might make sense to _you_, _now_. No guarantee they make >any sense (or are in the least useful) for other users on your system, and They also make perfect sense to me, and I would like this to have had this yesterday. or last year ;) >I might want them in arabic or some such. The descriptions might make no >sense to you in a couple of years. So what do you do if you use someone else's system and he doesn't have Arabic man pages but you want them in Arabic? And a description that doesn't make sense in many years is badly formed. >> Secondly, do you expect file managers like Nautilus and Konqueror to >> support every piece of file format on the planet so they could read >> information directly from the documents? >That's their (self-selected) job, yes. Sure, but they store it differently in different places. Having them in the actual file makes it universal. I think the biggest issue is that you are arguing against this out of some principle. There is a certain want for these new features among the users, there are people who are willing to implement them, there is Linus who seems to agree and accept. What do you, and people who agree with you, lose if this gets implemented? Drive space when you have the slightly-grown kernel sources downloaded? -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 23:58 ` Horst von Brand @ 2004-08-29 16:18 ` Spam 2004-08-29 16:18 ` Spam 2004-08-29 19:10 ` Tonnerre 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-29 16:18 UTC (permalink / raw) To: Horst von Brand Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Andrew Morton <akpm@osdl.org> said: > [...] >> > For example, some image file formats already support embedded metadata, do >> > they not? >> Yes, JPEG, TIFF and PNG files for example. But, if you modify any of >> these with an application that doesn't support the extensions then >> you will loose them. > As you will each time you muck around with your files-are-directories. > Nothing new there. >> Also, you are thinking _very_ narrowly now. There are thousands of >> file formats. Implementing the support for meta-data/ streams/ >> attributes in the kernel will make it possible to use this >> generically for all files. > So the _kernel_ has to know about thousands of formats, just in case it > some blue day it comes across a strange file? Better leave that to the > applications. No, not at all. The only think the kernel would need to support is the actual meta files and streams - not the contents of the streams. That is up to the applications to know and use. The idea with plugins to reiser4 would allow someone to create a plugin that would export the thumbnails from TIFF and JPEG images as separate streams. > You will _not_ be able to define a single format for extra data about the > file, there will be differing extra data for different users (do you > suggest a root-only file with special writable pouches for "graphical icon > for the file" for each user on the system?!), so the idea in itself is > doomed from the start. We do not need to define a single format. You are locking yourself to the contents of streams and meta files. If someone likes the idea of a plugin that would allow for icons then that may be it. It would be up to application developers to know about this and use it. Perhaps they could agree on a standard even. >> I use this in Windows quite much. > Then use it and be happy. No need to screw up Linux for that. This wouldn't screw anything up besides allowing for extending the functionality. There is noone that forces you to enable and instll plugins, use extra attributes or meta info. All of this always was optional (as I see it). >> I put information description >> fields for lots of different files. These descriptions are then >> searchable etc. I could use the command prompt to copy the files and >> the descriptions would still be there. > The descriptions might make sense to _you_, _now_. No guarantee they make > any sense (or are in the least useful) for other users on your system, and > I might want them in arabic or some such. The descriptions might make no > sense to you in a couple of years. So? The same may be for your own word documents etc. It doesn't mean I do not find it useful now, or that anyone else might not. Shall we not include features because "someone" on the same system may not use it? Seems very backward to me. If I include a filestream, description.xml, then that wouldn't affect anyone. I would know how to read the file, and that would be enough. >> Secondly, do you expect file managers like Nautilus and Konqueror to >> support every piece of file format on the planet so they could read >> information directly from the documents? > That's their (self-selected) job, yes. Yes and no. I used them as example but there are lots of other programs that can access files. Midnight commander, cp, tar, email programs and lots of others. besides, I think it would be easier for them to 1) write plugins and 2) just use normal file access method to provide the extra data to the users. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 16:18 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-29 16:18 UTC (permalink / raw) To: Horst von Brand Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Andrew Morton <akpm@osdl.org> said: > [...] >> > For example, some image file formats already support embedded metadata, do >> > they not? >> Yes, JPEG, TIFF and PNG files for example. But, if you modify any of >> these with an application that doesn't support the extensions then >> you will loose them. > As you will each time you muck around with your files-are-directories. > Nothing new there. >> Also, you are thinking _very_ narrowly now. There are thousands of >> file formats. Implementing the support for meta-data/ streams/ >> attributes in the kernel will make it possible to use this >> generically for all files. > So the _kernel_ has to know about thousands of formats, just in case it > some blue day it comes across a strange file? Better leave that to the > applications. No, not at all. The only think the kernel would need to support is the actual meta files and streams - not the contents of the streams. That is up to the applications to know and use. The idea with plugins to reiser4 would allow someone to create a plugin that would export the thumbnails from TIFF and JPEG images as separate streams. > You will _not_ be able to define a single format for extra data about the > file, there will be differing extra data for different users (do you > suggest a root-only file with special writable pouches for "graphical icon > for the file" for each user on the system?!), so the idea in itself is > doomed from the start. We do not need to define a single format. You are locking yourself to the contents of streams and meta files. If someone likes the idea of a plugin that would allow for icons then that may be it. It would be up to application developers to know about this and use it. Perhaps they could agree on a standard even. >> I use this in Windows quite much. > Then use it and be happy. No need to screw up Linux for that. This wouldn't screw anything up besides allowing for extending the functionality. There is noone that forces you to enable and instll plugins, use extra attributes or meta info. All of this always was optional (as I see it). >> I put information description >> fields for lots of different files. These descriptions are then >> searchable etc. I could use the command prompt to copy the files and >> the descriptions would still be there. > The descriptions might make sense to _you_, _now_. No guarantee they make > any sense (or are in the least useful) for other users on your system, and > I might want them in arabic or some such. The descriptions might make no > sense to you in a couple of years. So? The same may be for your own word documents etc. It doesn't mean I do not find it useful now, or that anyone else might not. Shall we not include features because "someone" on the same system may not use it? Seems very backward to me. If I include a filestream, description.xml, then that wouldn't affect anyone. I would know how to read the file, and that would be enough. >> Secondly, do you expect file managers like Nautilus and Konqueror to >> support every piece of file format on the planet so they could read >> information directly from the documents? > That's their (self-selected) job, yes. Yes and no. I used them as example but there are lots of other programs that can access files. Midnight commander, cp, tar, email programs and lots of others. besides, I think it would be easier for them to 1) write plugins and 2) just use normal file access method to provide the extra data to the users. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 23:58 ` Horst von Brand 2004-08-29 15:17 ` Markus Törnqvist 2004-08-29 16:18 ` Spam @ 2004-08-29 19:10 ` Tonnerre 2004-08-30 0:39 ` Alexander G. M. Smith 2 siblings, 1 reply; 1835+ messages in thread From: Tonnerre @ 2004-08-29 19:10 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 889 bytes --] Salut, On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: > > Also, you are thinking _very_ narrowly now. There are thousands of > > file formats. Implementing the support for meta-data/ streams/ > > attributes in the kernel will make it possible to use this > > generically for all files. > > So the _kernel_ has to know about thousands of formats, just in case it > some blue day it comes across a strange file? Better leave that to the > applications. I'd do that quite differently: Collect metadata about files in extended attributes (which are rather universally useable), and do it from an userspace daemon/application/file manager plugin which uses dynamically loadable plugins for the different file types differentiated by libmagic. This has nothing to do in the kernel. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 19:10 ` Tonnerre @ 2004-08-30 0:39 ` Alexander G. M. Smith 2004-08-30 18:39 ` Spam 2004-08-31 8:15 ` Tonnerre 0 siblings, 2 replies; 1835+ messages in thread From: Alexander G. M. Smith @ 2004-08-30 0:39 UTC (permalink / raw) To: Tonnerre Cc: spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200: > On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: > > So the _kernel_ has to know about thousands of formats, just in case it > > some blue day it comes across a strange file? Better leave that to the > > applications. > > I'd do that quite differently: Collect metadata about files in > extended attributes (which are rather universally useable), and do it > from an userspace daemon/application/file manager plugin which uses > dynamically loadable plugins for the different file types > differentiated by libmagic. > > This has nothing to do in the kernel. I mostly agree, like I was saying earlier, file types are needed! The kernel doesn't have to know about all of them, just some of them. It should be possible to attach a file type to everything so you know what kind of thing it is, not just block or character device or file, but something like a MIME type. If space is a concern, add to the inode a unique 8 byte character code for each popular type and a global translation table to get back the MIME string. The kernel (includes file systems and plug-ins) would have to know some standard file types and how to compare them for indexing purposes. For example, an integer attribute like the year a music album was released would be stored as a 4 byte attribute "file" with a type like "primitive/int32", which tells the kernel to use 32 bit integer number comparison for indexing order. Another type the kernel could be aware of are the file types used for security attributes. A third kind it might need to know about are the computed attributes, which get set whenever related attributes they are computed from change. For any unknown types, the kernel just stores the data and doesn't do any extra processing. Optionally the kernel could also maintain the global list of all file types and their properties (such as which ones are indexed, which are computed), though that could also be done in userland if you aren't doing indexing or computed attributes or other fancy operations. The determination of the types would be in userland (smart programs would write the type when they created the file). Like you said, a daemon could classify unknown files in the background (BeOS does this). Most of the uses of the file type would also be in userland, such as deciding which program can handle a given file, or what icon to display, or to hint that a file/directory/attribute thing should be interpreted as an old style directory to the user, or to mark a file thing as being worth backing up (don't need to back up the computed ones). - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 0:39 ` Alexander G. M. Smith 2004-08-30 18:39 ` Spam @ 2004-08-30 18:39 ` Spam 1 sibling, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-30 18:39 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Tonnerre, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand > Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200: >> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: >> > So the _kernel_ has to know about thousands of formats, just in case it >> > some blue day it comes across a strange file? Better leave that to the >> > applications. >> >> I'd do that quite differently: Collect metadata about files in >> extended attributes (which are rather universally useable), and do it >> from an userspace daemon/application/file manager plugin which uses >> dynamically loadable plugins for the different file types >> differentiated by libmagic. >> >> This has nothing to do in the kernel. > I mostly agree, like I was saying earlier, file types are needed! > The kernel doesn't have to know about all of them, just some of > them. It should be possible to attach a file type to everything so > you know what kind of thing it is, not just block or character > device or file, but something like a MIME type. If space is a > concern, add to the inode a unique 8 byte character code for each > popular type and a global translation table to get back the MIME > string. The kernel doesn't have anything to know about normal files.It is up to applications to determine and know the contents. Isn't it? Exception would be of course special meta files that define attributes and permissions for the filesystem. > The kernel (includes file systems and plug-ins) would have to > know some standard file types and how to compare them for indexing > purposes. For example, an integer attribute like the year a music > album was released would be stored as a 4 byte attribute "file" with > a type like "primitive/int32", which tells the kernel to use 32 bit > integer number comparison for indexing order. Another type the > kernel could be aware of are the file types used for security > attributes. A third kind it might need to know about are the > computed attributes, which get set whenever related attributes they > are computed from change. For any unknown types, the kernel just > stores the data and doesn't do any extra processing. > Optionally the kernel could also maintain the global list of all > file types and their properties (such as which ones are indexed, > which are computed), though that could also be done in userland if > you aren't doing indexing or computed attributes or other fancy > operations. > The determination of the types would be in userland (smart > programs would write the type when they created the file). Like you > said, a daemon could classify unknown files in the background (BeOS > does this). Most of the uses of the file type would also be in > userland, such as deciding which program can handle a given file, or > what icon to display, or to hint that a file/directory/attribute > thing should be interpreted as an old style directory to the user, > or to mark a file thing as being worth backing up (don't need to > back up the computed ones). > - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-30 18:39 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-30 18:39 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Tonnerre, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand > Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200: >> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: >> > So the _kernel_ has to know about thousands of formats, just in case it >> > some blue day it comes across a strange file? Better leave that to the >> > applications. >> >> I'd do that quite differently: Collect metadata about files in >> extended attributes (which are rather universally useable), and do it >> from an userspace daemon/application/file manager plugin which uses >> dynamically loadable plugins for the different file types >> differentiated by libmagic. >> >> This has nothing to do in the kernel. > I mostly agree, like I was saying earlier, file types are needed! > The kernel doesn't have to know about all of them, just some of > them. It should be possible to attach a file type to everything so > you know what kind of thing it is, not just block or character > device or file, but something like a MIME type. If space is a > concern, add to the inode a unique 8 byte character code for each > popular type and a global translation table to get back the MIME > string. The kernel doesn't have anything to know about normal files.It is up to applications to determine and know the contents. Isn't it? Exception would be of course special meta files that define attributes and permissions for the filesystem. > The kernel (includes file systems and plug-ins) would have to > know some standard file types and how to compare them for indexing > purposes. For example, an integer attribute like the year a music > album was released would be stored as a 4 byte attribute "file" with > a type like "primitive/int32", which tells the kernel to use 32 bit > integer number comparison for indexing order. Another type the > kernel could be aware of are the file types used for security > attributes. A third kind it might need to know about are the > computed attributes, which get set whenever related attributes they > are computed from change. For any unknown types, the kernel just > stores the data and doesn't do any extra processing. > Optionally the kernel could also maintain the global list of all > file types and their properties (such as which ones are indexed, > which are computed), though that could also be done in userland if > you aren't doing indexing or computed attributes or other fancy > operations. > The determination of the types would be in userland (smart > programs would write the type when they created the file). Like you > said, a daemon could classify unknown files in the background (BeOS > does this). Most of the uses of the file type would also be in > userland, such as deciding which program can handle a given file, or > what icon to display, or to hint that a file/directory/attribute > thing should be interpreted as an old style directory to the user, > or to mark a file thing as being worth backing up (don't need to > back up the computed ones). > - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-30 18:39 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-30 18:39 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Tonnerre, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand > Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200: >> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: >> > So the _kernel_ has to know about thousands of formats, just in case it >> > some blue day it comes across a strange file? Better leave that to the >> > applications. >> >> I'd do that quite differently: Collect metadata about files in >> extended attributes (which are rather universally useable), and do it >> from an userspace daemon/application/file manager plugin which uses >> dynamically loadable plugins for the different file types >> differentiated by libmagic. >> >> This has nothing to do in the kernel. > I mostly agree, like I was saying earlier, file types are needed! > The kernel doesn't have to know about all of them, just some of > them. It should be possible to attach a file type to everything so > you know what kind of thing it is, not just block or character > device or file, but something like a MIME type. If space is a > concern, add to the inode a unique 8 byte character code for each > popular type and a global translation table to get back the MIME > string. The kernel doesn't have anything to know about normal files.It is up to applications to determine and know the contents. Isn't it? Exception would be of course special meta files that define attributes and permissions for the filesystem. > The kernel (includes file systems and plug-ins) would have to > know some standard file types and how to compare them for indexing > purposes. For example, an integer attribute like the year a music > album was released would be stored as a 4 byte attribute "file" with > a type like "primitive/int32", which tells the kernel to use 32 bit > integer number comparison for indexing order. Another type the > kernel could be aware of are the file types used for security > attributes. A third kind it might need to know about are the > computed attributes, which get set whenever related attributes they > are computed from change. For any unknown types, the kernel just > stores the data and doesn't do any extra processing. > Optionally the kernel could also maintain the global list of all > file types and their properties (such as which ones are indexed, > which are computed), though that could also be done in userland if > you aren't doing indexing or computed attributes or other fancy > operations. > The determination of the types would be in userland (smart > programs would write the type when they created the file). Like you > said, a daemon could classify unknown files in the background (BeOS > does this). Most of the uses of the file type would also be in > userland, such as deciding which program can handle a given file, or > what icon to display, or to hint that a file/directory/attribute > thing should be interpreted as an old style directory to the user, > or to mark a file thing as being worth backing up (don't need to > back up the computed ones). > - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 0:39 ` Alexander G. M. Smith 2004-08-30 18:39 ` Spam @ 2004-08-31 8:15 ` Tonnerre 2004-09-01 20:16 ` Jamie Lokier 1 sibling, 1 reply; 1835+ messages in thread From: Tonnerre @ 2004-08-31 8:15 UTC (permalink / raw) To: Alexander G. M. Smith Cc: spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand [-- Attachment #1: Type: text/plain, Size: 1351 bytes --] Salut, On Sun, Aug 29, 2004 at 08:39:25PM -0400, Alexander G. M. Smith wrote: > I mostly agree, like I was saying earlier, file types are needed! > The kernel doesn't have to know about all of them, just some of > them. It should be possible to attach a file type to everything so > you know what kind of thing it is, not just block or character > device or file, but something like a MIME type. Then store the value computed by libmagic into an extended attribute, if you like. I still don't see why the kernel should even care. It shouldn't read things from files anyway, as a kernel shall be bootable on a busybox system (and *Step) as well, so we don't rely on the fs layout in any way. > Optionally the kernel could also maintain the global list of all > file types and their properties (such as which ones are indexed, > which are computed), though that could also be done in userland if > you aren't doing indexing or computed attributes or other fancy > operations. Quel horreur! Do it in userland, really. If I get the time, I'll write you a small daemon based on libmagic which stores the file attributes in xattrs, or if they're not supported, in some MacOS/Xish per-directory files. Even a file manager ("finder") can do that, there's not even the need for a daemon. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 8:15 ` Tonnerre @ 2004-09-01 20:16 ` Jamie Lokier 2004-09-01 20:28 ` Dave Kleikamp ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 20:16 UTC (permalink / raw) To: Tonnerre Cc: Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Tonnerre wrote: > Quel horreur! > Do it in userland, really. I'm amazed that after all this discussion where all the realistic implementations are done in userland with kernel support for calling out to it, there are people who think the kernel is supposed to decode MP3 files or whatever. Nobody is advocating that! > If I get the time, I'll write you a small daemon based on libmagic > which stores the file attributes in xattrs, or if they're not > supported, in some MacOS/Xish per-directory files. Even a file manager > ("finder") can do that, there's not even the need for a daemon. How are you going to do the part where the xattr changes when the file is modified? (For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) I don't see how you can do that without kernel support. Don't say dnotify or inotify, because neither would work. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:16 ` Jamie Lokier @ 2004-09-01 20:28 ` Dave Kleikamp 2004-09-02 3:42 ` Hubert Chan ` (2 more replies) 2004-09-02 9:50 ` Alan Cox 2004-09-02 18:16 ` Christer Weinigel 2 siblings, 3 replies; 1835+ messages in thread From: Dave Kleikamp @ 2004-09-01 20:28 UTC (permalink / raw) To: Jamie Lokier Cc: Tonnerre, Alexander G. M. Smith, spam, Andrew Morton, wichert, jra, Linus Torvalds, reiser, Christoph Hellwig, fsdevel, linux-kernel, flx, reiserfs-list, vonbrand On Wed, 2004-09-01 at 15:16, Jamie Lokier wrote: > (For example, if I edit an HTML file which is encoded in iso-8859-1, > change it to utf-8 and indicate that in a META element, and save it > under the same name, the full content-type should change from > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > I don't see how you can do that without kernel support. Your html editor should do that. -- David Kleikamp IBM Linux Technology Center ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:28 ` Dave Kleikamp @ 2004-09-02 3:42 ` Hubert Chan 2004-09-02 3:52 ` David Masover 2004-09-02 5:57 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-02 3:42 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Dave" == Dave Kleikamp <shaggy@austin.ibm.com> writes: Dave> On Wed, 2004-09-01 at 15:16, Jamie Lokier wrote: >> (For example, if I edit an HTML file which is encoded in iso-8859-1, >> change it to utf-8 and indicate that in a META element, and save it >> under the same name, the full content-type should change from >> "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) >> >> I don't see how you can do that without kernel support. Dave> Your html editor should do that. And if it doesn't, you'll need to patch your editor. Every editor will need to be patched to make sure that it is always consistent. Even sed. And you'll have to do that for every dynamically-changing attribute that people want. Sure, you can go that route if you want, but I wish you luck. You'll need it. (Just for the record, I don't particularly think that the example given by Jamie is a good one. The "meta" element to me is a hack necessitated by not being able to attach attributes to a file in a consistent manner. The "proper" way to do it would be to set the content-type as an attribute, and let the webserver read that and send it in the HTTP header, where it's supposed to be. Another example could be thumbnails. There are probably more, but I'm not feeling very creative right now.) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 3:42 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-02 3:42 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Dave" == Dave Kleikamp <shaggy@austin.ibm.com> writes: Dave> On Wed, 2004-09-01 at 15:16, Jamie Lokier wrote: >> (For example, if I edit an HTML file which is encoded in iso-8859-1, >> change it to utf-8 and indicate that in a META element, and save it >> under the same name, the full content-type should change from >> "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) >> >> I don't see how you can do that without kernel support. Dave> Your html editor should do that. And if it doesn't, you'll need to patch your editor. Every editor will need to be patched to make sure that it is always consistent. Even sed. And you'll have to do that for every dynamically-changing attribute that people want. Sure, you can go that route if you want, but I wish you luck. You'll need it. (Just for the record, I don't particularly think that the example given by Jamie is a good one. The "meta" element to me is a hack necessitated by not being able to attach attributes to a file in a consistent manner. The "proper" way to do it would be to set the content-type as an attribute, and let the webserver read that and send it in the HTTP header, where it's supposed to be. Another example could be thumbnails. There are probably more, but I'm not feeling very creative right now.) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:28 ` Dave Kleikamp 2004-09-02 3:42 ` Hubert Chan @ 2004-09-02 3:52 ` David Masover 2004-09-02 5:57 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: David Masover @ 2004-09-02 3:52 UTC (permalink / raw) To: Dave Kleikamp Cc: Jamie Lokier, Tonnerre, Alexander G. M. Smith, spam, Andrew Morton, wichert, jra, Linus Torvalds, reiser, Christoph Hellwig, fsdevel, linux-kernel, flx, reiserfs-list, vonbrand -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dave Kleikamp wrote: | On Wed, 2004-09-01 at 15:16, Jamie Lokier wrote: | |>(For example, if I edit an HTML file which is encoded in iso-8859-1, |>change it to utf-8 and indicate that in a META element, and save it |>under the same name, the full content-type should change from |>"text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) |> |>I don't see how you can do that without kernel support. | | | Your html editor should do that. Isn't the kernel faced with the same issues here as userland programs without meta attribute support? How do you read the META element before you know the encoding? Is there property attached to each file descriptor that defines what encoding it's using? I know there is in perl, but I doubt it's at the OS level. If it is at the OS level, the kernel should use that. Otherwise I agree -- the html editor should do that. Although I'd hate to see Vim patched to do that. Maybe save it as file/utf8? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTaZEHgHNmZLgCUhAQLmHxAAif/DzI3HDPPixHEJURTW8LMSBYi+RhgC Rsldit8G6N9Sx1wKReSZqoTI27OYUodVcXjgtSYbzr9w0dw2ifuLaU87550nCcwl p/3ivEiPww3Pqc6PgN2PpAhYT2b/xnZrWJSU3g9siGp9D4APBZ4yxz8GO9N8pAgg a/juHtLCNCtPKD8/IdjCYRn9W7Ka7uWF7cEHp58Yyj0R70EG4ljIRbJRhqKGT2Ew 6EIFcieStmuveA2Kna4Od4gAXIE5CrKEO8whRvplOsvHAlsZMp0zkgUGiKzBzQU0 K7ICkddo3Bnos1pc7BQ/7F595tNCNREqnlvCkWGVifLPdcyujULZ/U/OAVdD7zSJ AvhZs5A1/hPFYTEqrzSNsYteDsTSc0TJsiWS8gx+qyGzOzaVMAOIuf86jQGKzBT7 WSYutU4itrE1QYvdJ9XXeess8m9aSY1A9G1EkhtWmaRDq/SuCpSf01cNjZZVu52v Q1h6faCC9JTHM1pDPMBlMjeQOnhrhv/AuhrZnOfXmrODK06l+ZhQ/LSUTRAZwzkv UV6PSc+Pbq7ijYrurH4attPI8IkelzExhIPHziGQCTMnIN3pXSr6RT2zEtDG7LNR 98MHtKkuNC8bsuYFhyqGRbMLr9GANiByzu3rh1ccHuMbB42FR8/Z7X6UCDVZtP80 Bp4g6WwVG6A= =xNYt -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:28 ` Dave Kleikamp 2004-09-02 3:42 ` Hubert Chan 2004-09-02 3:52 ` David Masover @ 2004-09-02 5:57 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-02 5:57 UTC (permalink / raw) To: Dave Kleikamp Cc: Tonnerre, Alexander G. M. Smith, spam, Andrew Morton, wichert, jra, Linus Torvalds, reiser, Christoph Hellwig, fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Tonnerre wrote: > > > I'll write you a small daemon based on libmagic which stores the > > > file attributes in xattrs, or if they're not supported, in some > > > MacOS/Xish per-directory files. Even a file manager ("finder") can > > > do that, there's not even the need for a daemon. Jamie Lokier wrote: > > (For example, if I edit an HTML file which is encoded in iso-8859-1, > > change it to utf-8 and indicate that in a META element, and save it > > under the same name, the full content-type should change from > > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > > > I don't see how you can do that without kernel support. Dave Kleikamp wrote: > Your html editor should do that. My html editor is "vi". If I am supposed to manually set the content-type attribute after exiting vi, doesn't that rather invalidate the idea of a "small daemon based on libmagic" which sets it for me? That was the sole point of my statement. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:16 ` Jamie Lokier 2004-09-01 20:28 ` Dave Kleikamp @ 2004-09-02 9:50 ` Alan Cox 2004-09-02 20:25 ` Jamie Lokier 2004-09-02 18:16 ` Christer Weinigel 2 siblings, 1 reply; 1835+ messages in thread From: Alan Cox @ 2004-09-02 9:50 UTC (permalink / raw) To: Jamie Lokier Cc: Tonnerre, Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list, vonbrand On Mer, 2004-09-01 at 21:16, Jamie Lokier wrote: > (For example, if I edit an HTML file which is encoded in iso-8859-1, > change it to utf-8 and indicate that in a META element, and save it > under the same name, the full content-type should change from > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > I don't see how you can do that without kernel support. > > Don't say dnotify or inotify, because neither would work. inotify done right is useful here as well as in a lot of other desktop cases where dnotify doesn't really scale. Its enough to let me - Find the new file - Virus scan it - Classify its possible type heirarchies - Index it Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 9:50 ` Alan Cox @ 2004-09-02 20:25 ` Jamie Lokier 0 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-02 20:25 UTC (permalink / raw) To: Alan Cox Cc: Tonnerre, Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list, vonbrand Alan Cox wrote: > On Mer, 2004-09-01 at 21:16, Jamie Lokier wrote: > > (For example, if I edit an HTML file which is encoded in iso-8859-1, > > change it to utf-8 and indicate that in a META element, and save it > > under the same name, the full content-type should change from > > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > > > I don't see how you can do that without kernel support. > > > > Don't say dnotify or inotify, because neither would work. > > inotify done right is useful here as well as in a lot of other desktop > cases where dnotify doesn't really scale. Its enough to let me > > - Find the new file > - Virus scan it > - Classify its possible type heirarchies > - Index it Indeed it does, but it fails for the example I was commenting on to which you replied.. 1. The file /var/www/site/index.html's written in vi. 2. "The daemon" (that's what I objected to) receives inotify. blocks waiting for scheduler, however... 3. Seeing that vi is now finished, I phone person on other side of world and say the updated file is now available. 4. Person fetches http://site/index.html ...oh! They receive the wrong content-type (bollocks!) 5. Eventually after paging & scheduling deems it appropriate, "The daemon" gets to run, looks at file, updates content-type. In other words I was criticising Tonnerre's idea of a daemon which updates the xattrs by looking at file contents... by definition, a daemon runs in the background, and the whole point of it updating a content-type xattr was obviously so that programs like httpd could just _use_ that xattr. That doesn't work, even if that daemon uses inotify. Basically, anytime where you want an ordering guarantee that something will be recalculated in the interval after a file is modified and before the calculated result is used, you have to be very careful about exactly which code is using dnotify or inotify to achieve that. Single threaded programs can use it easily, but a multi-threaded file server has to be very careful about ordering if it's to avoid glitches. It is exactly a real problem I have faced in a multi-threaded HTTP server which uses dnotify to detect changes to prerequisite files and also changes to the path walked and permissions on it, and thus invalidate cached generated pages, thus giving strong cache guarantees. Performance is great, if you ignore the race conditions... (One of the more popular small web servers, lighttpd (PHP people seem to like it), also uses dnotify. Apparently it provides a big performance boost, just by avoiding stat() calls. So I'm not alone in using dnotify for such things.) -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:16 ` Jamie Lokier 2004-09-01 20:28 ` Dave Kleikamp 2004-09-02 9:50 ` Alan Cox @ 2004-09-02 18:16 ` Christer Weinigel 2004-09-02 19:58 ` Jamie Lokier 2 siblings, 1 reply; 1835+ messages in thread From: Christer Weinigel @ 2004-09-02 18:16 UTC (permalink / raw) To: Jamie Lokier Cc: Tonnerre, Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Jamie Lokier <jamie@shareable.org> writes: > (For example, if I edit an HTML file which is encoded in iso-8859-1, > change it to utf-8 and indicate that in a META element, and save it > under the same name, the full content-type should change from > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > I don't see how you can do that without kernel support. charset_cache = dbm(os.getenv('HOME') + '/.charset_cache') def get_charset(path): file_mtime = get_mtime(path) cache_tuple = dbm.get(path, (None, None)) if file_mtime != cache_tuple[0]: cache_tuple = (file_mtime, figure_out_charset(path)) dbm.put(path, cache_tuple) return cache_tuple[1] This code is guaranteed to always give you an up to date charset for a file, provided that the mtime is guaranteed to change every time the file changes. If the file can change during the call to get_charset you'll have to lock the file while you are working with it or handle it some other way (comparing mtime before and after and retrying if they differ), but that is true even if you do it all in the kernel. This has the added advantage that the cache is updated lazily, so the charset is never calculated unless someone really needs it. If the disk gets full, it is time to shrink the cache and preferably stale entries are evicted first. If that's not good enough, a notifier such as dnotify or inotify can be used to invalidate the cache immediately after the file has changed. If this gets popular, the cache management can be moved to a daemon (naturally with all the security aspects in mind). > Don't say dnotify or inotify, because neither would work. Why not? The approach above works on any filesystem, even without dnotify or inotify but will be more efficient with them. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 18:16 ` Christer Weinigel @ 2004-09-02 19:58 ` Jamie Lokier 0 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-02 19:58 UTC (permalink / raw) To: Christer Weinigel Cc: Tonnerre, Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Christer Weinigel wrote: > Jamie Lokier <jamie@shareable.org> writes: > > > (For example, if I edit an HTML file which is encoded in iso-8859-1, > > change it to utf-8 and indicate that in a META element, and save it > > under the same name, the full content-type should change from > > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > > > I don't see how you can do that without kernel support. > > [code which runs in the program which needs to read the content-type] > > This code is guaranteed to always give you an up to date charset for a > file, provided that the mtime is guaranteed to change every time the > file changes. ...which it isn't, but ignoring that. Yes we all know that code will work, turning a blind eye to occasional mtime failures. What I objected to isn't that, but Tonerre's idea of a daemon (i.e. in the background, that's what daemons do) updating xattrs. Your code will work. Tonerre's has a serious race condition. > > Don't say dnotify or inotify, because neither would work. > > Why not? The approach above works on any filesystem, even without > dnotify or inotify but will be more efficient with them. The above approach is in fact better with dnotify/inotify, when they area available, because then it doesn't have mtime problems. My objection is that dnotify & inotify don't fix the problems of a "daemon" (i.e. in the background) updating an xattr - because the daemon might not run scheduled between the write and when the file needs to be used. That said, yours has practical problems too. The intended purpose (*) of storing content-type in an xattr was so that programs which need to read content without understanding it may just do so (e.g. mail agents, httpd), and are distinct from the programs which compute and set it (e.g. programs which understand specific content formats). (*) I don't necessarily agree with that purpose. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:55 ` Andrew Morton 2004-08-26 11:15 ` Spam @ 2004-08-26 11:48 ` Jamie Lokier 2004-08-26 16:07 ` James Bruce 2004-08-26 23:52 ` Hans Reiser 3 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 11:48 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Andrew Morton wrote: > No. All of the applications which you initially identified can be > implemented by putting the various bits of data into a single file and > getting applications to agree on the format of that file. > > For example, some image file formats already support embedded metadata, do > they not? Same for MP3s and their metadata tags. _That_ kind of data should be part of the contents of the file: it is intended to be transported. It is useful to auto-extract it from the file's data stream, using format-specific programs, and offer that metadata in separate files where it is easy to use e.g. by shell scripts and indexing programs, which can then work independently of file formats. That kind of metadata should be _derived_ from the file's data stream, and ideally the extraction should be automatic, with userspace assistance. It may be useful for the filesystem to record what's extracted, for fast access and indexing. The same applies to thumbnails. The only difference is that these may take longer to compute. But they're the same thing: something calculated from the file's data stream, which may (or may not) be stored in the filesystem so it doesn't have to be recomputed each time. It might even be worth expiring the metadata from disk if it hasn't been used for a while and can be recomputed. These are the same as looking into archives or structured files. The only difference is that the unpacked views might not be stored on the filesystem (but they might be, if the filesystem knows how to cache views for faster future access).[*] Those are all examples of data derived from the data stream of a flat file. They are useful applications, and all work fine with FTP, cp, cat <a >b, and email attachments, _no application changes required_. There's another kind of metadata which isn't intended for transport. That would be things like the classic attributes: permissions, timestamps. It would also include things like author information when the file format cannot carry it. -- Jamie [*] - A fancier variant, which is hard without good fs support, it writing to structured files through a view. I.e. modifying files inside a .tar.gz, or (this is useful) inside an {MS,Open,Gnome,K}Office container file or *SQL database table. To do this efficiently, the file's main data stream is regenerated on demand from the view's data. As I said, it requires good, special fs support to do it efficiently (fancy caching). With appropriate userspace libraries you can make apps work with container files on all other OSes (just like .tar.gz and *Office containers at the moment) but are significantly faster to work with on Linux (no need to serialise and (de-)compress except on demand) and more visible to the standard unix scripting tools. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:55 ` Andrew Morton 2004-08-26 11:15 ` Spam 2004-08-26 11:48 ` Jamie Lokier @ 2004-08-26 16:07 ` James Bruce 2004-08-26 16:58 ` Jamie Lokier 2004-08-26 23:52 ` Hans Reiser 3 siblings, 1 reply; 1835+ messages in thread From: James Bruce @ 2004-08-26 16:07 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Andrew Morton wrote: >No. All of the applications which you initially identified can be >implemented by putting the various bits of data into a single file and >getting applications to agree on the format of that file. > > So in order to avoid breaking backup and file utilities, we'd instead break every application that reads files? That way surely lies madness :) This /could/ possibly work with lots of cooperation from glibc, but we'd also need a way to insert and delete bytes in a file... moving 1MB of attributes/streams just to change a 20 character author attribute seems a bit silly. In addition, there are little things, such as how adding a thumbnail image shouldn't change the modification time of the original picture IMO, but it does if you embed it in the file. If it should be in userspace, we could take the OS-X approach of using directories for everything, and the "data" that would be in a hybrid directory-file is a specially named file under that directory. Programs for instance are actually directories, with the elf file underneath it. When you click on the "directory-program" in the gui it runs the associated elf file rather than opening the directory. I think that approach is promising, but so few unix programs have any indirection for file access it'd be hell to teach them all how it works. Of course with emminent breakage of some sort, maybe now is the time... - Jim Bruce ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:07 ` James Bruce @ 2004-08-26 16:58 ` Jamie Lokier 2004-08-26 21:34 ` Felipe Alfaro Solana 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 16:58 UTC (permalink / raw) To: James Bruce Cc: Andrew Morton, Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list James Bruce wrote: > moving 1MB of attributes/streams just to change a 20 character > author attribute seems a bit silly. That's what we do now. Try editing an MP3's id tags. > If it should be in userspace, we could take the OS-X approach of using > directories for everything, and the "data" that would be in a hybrid > directory-file is a specially named file under that directory. Programs > for instance are actually directories, with the elf file underneath it. > When you click on the "directory-program" in the gui it runs the > associated elf file rather than opening the directory. I think that > approach is promising, but so few unix programs have any indirection for > file access it'd be hell to teach them all how it works. The interfaces we've been talking about are very close to that. The indirection is automatic: instead of a document (or program) looking like a directory, it'll look like a file, and reading (or executing) the file will do the indirection, with kernel control (and so consistency and performance) via an appropriate flat<->structured relation helper. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:58 ` Jamie Lokier @ 2004-08-26 21:34 ` Felipe Alfaro Solana 0 siblings, 0 replies; 1835+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 21:34 UTC (permalink / raw) To: Jamie Lokier Cc: James Bruce, Andrew Morton, Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 18:58, Jamie Lokier wrote: > James Bruce wrote: > > moving 1MB of attributes/streams just to change a 20 character > > author attribute seems a bit silly. > > That's what we do now. Try editing an MP3's id tags. What's the problem? Let's say a file has three streams/forks/whatever: the unnamed one, and two additional named. The filesystem can treat these three streams like three independent (but related) files on disk so, if the author attribute belongs to the first named stream, only the first named stream (and its corresponding file on the backend) would need to be modified. No need to move MBs of data around. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:55 ` Andrew Morton @ 2004-08-26 23:52 ` Hans Reiser 2004-08-26 11:48 ` Jamie Lokier ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:52 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Andrew Morton wrote: > >No. All of the applications which you initially identified can be >implemented by putting the various bits of data into a single file and >getting applications to agree on the format of that file. > >For example, some image file formats already support embedded metadata, do >they not? > >- > > and that leads to vasts numbers of file formats, all of which require special code to parse them, and apps that don't duplicate that code can't access the data, and it is a software engineering mistake. Uniform interfaces greatly reduce the cost of an OS and increase its expressive power. Closure is the most important and least understood principle of OS design. By contrast, suppose everything was stored in files and directories. Every app can afford the coding cost to learn about files and directories. Data can freely bounce from one object to another in the OS because the APIs for the objects are all the same. That's increased expressive power. The number of connections between objects determines the expressive power of the OS, not the number of objects in it. Unified namespaces do a lot for an OS. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:52 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:52 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Andrew Morton wrote: > >No. All of the applications which you initially identified can be >implemented by putting the various bits of data into a single file and >getting applications to agree on the format of that file. > >For example, some image file formats already support embedded metadata, do >they not? > >- > > and that leads to vasts numbers of file formats, all of which require special code to parse them, and apps that don't duplicate that code can't access the data, and it is a software engineering mistake. Uniform interfaces greatly reduce the cost of an OS and increase its expressive power. Closure is the most important and least understood principle of OS design. By contrast, suppose everything was stored in files and directories. Every app can afford the coding cost to learn about files and directories. Data can freely bounce from one object to another in the OS because the APIs for the objects are all the same. That's increased expressive power. The number of connections between objects determines the expressive power of the OS, not the number of objects in it. Unified namespaces do a lot for an OS. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:51 ` Spam (?) (?) @ 2004-08-26 11:11 ` Bernd Petrovitsch -1 siblings, 0 replies; 1835+ messages in thread From: Bernd Petrovitsch @ 2004-08-26 11:11 UTC (permalink / raw) To: Spam; +Cc: Andrew Morton, jra, hch, linux-fsdevel, linux-kernel [ shortened Cc: of reiser- and other folks since it is no longer reiser-specific - sorry, if it was too much ] [ fixed missing attribution ] On Thu, 2004-08-26 at 12:51 +0200, Spam wrote: > On Thu, 2004-08-26 at 03:24 -0700, Andrew Morton wrote: > > Spam <spam@tnonline.net> wrote: [...] > >> Applications that support the new features will benefit, all others > >> will continue to work without destroying data. > > > Sorry, but that all sounds a bit fluffy. Please provide some examples. It is too fluffy. > We already had the examples with cp and mv. Both should continue to > work and the files will still be copied. The same with Konqueror and ... after they are patched to support streams. As stated each stream has an own fd, so - from the user-space perspective - one has to open *all* streams of this file, read them and write them into the destination. > Nautilus. Files and their meta-files/streams/attributes will be > retained as long as applications are using the OS API. The OS-API is *one* file descriptor where you can read, write, mmap, ... on. Therefore stream selection must happen with open() since open() returns a fd (which uniquely identify a stream). Voila. And actually this makes transition actually possible: Old apps simply ignore streams (yes, there will be a default stream, which is used id nothing else is specified) and throw them away (think of e.g. patch which opens a new file, writes into it, unlink's the old one and mv's the new one on the old place) until they are modified to support them. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:51 ` Spam @ 2004-08-26 23:59 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:59 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list What is important is that it all be one packaged solution from the same maintainer. That is necessary for coherency of solution. There is also benefit to it being one coherent body of code, but this is harder to articulate. It is always hard to articulate why functionality belongs in the same program rather than split into multiple programs. Filesystems should provide primitives for ordering, structuring, and naming objects. It is hard to say why we should do all of that, but on some level I just know it is our task. Hans Spam wrote: > > > You said it would be socially hard. I think it would be very much > close to impossible to get it right. Imagine that Gnome and Nautilus > would implement support for these. I doubt that cp, mv, KDE, mc, > app-xyz would implement this anytime soon and in the meantime the > data is at risk. > > > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:59 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:59 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list What is important is that it all be one packaged solution from the same maintainer. That is necessary for coherency of solution. There is also benefit to it being one coherent body of code, but this is harder to articulate. It is always hard to articulate why functionality belongs in the same program rather than split into multiple programs. Filesystems should provide primitives for ordering, structuring, and naming objects. It is hard to say why we should do all of that, but on some level I just know it is our task. Hans Spam wrote: > > > You said it would be socially hard. I think it would be very much > close to impossible to get it right. Imagine that Gnome and Nautilus > would implement support for these. I doubt that cp, mv, KDE, mc, > app-xyz would implement this anytime soon and in the meantime the > data is at risk. > > > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:20 ` Spam (?) (?) @ 2004-08-26 10:54 ` Wichert Akkerman 2004-08-26 23:48 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Wichert Akkerman @ 2004-08-26 10:54 UTC (permalink / raw) To: Spam Cc: Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Spam wrote: > Because having user space tools and code will make it not work with > everything. Keeping stuff in the kernel should make the new features > transparent to the applications. But having it in the kernel has the same problem. If you read the Solaris documentation you will see that a bunch of utilties had to get a new commandline option to be able to access the metadata and a special utility was added for other applications. If you look at windows you will see that you need to use a filename like <realfilename>:<streamname>:$DATA (which obviously does not work for single-character filenames). Ignoring samba for a bit which just needs streams to stay compatible with windows I see few reasons for using streams: * files are more complex these days and tend to include multiple different things: images with thumbnails and exif data, 'office' documents containing both text and images * standard way to add common metadata to a file which can be used for searching tools (author, copyright, keywords, etc.) But both can already be done in userland (modern image formats can store thumbnails and exif data internally, applications use tar or zip-like files for documents, etc.). The metadata part is a lot more complicated as well since the behaviour of attributes might need to be complex: if I change an image using an application that is not stream-aware, what should happen to its thumbnail? The only common benefits I can see are standardisation and optimization: instead of every file format or application defining a way to specify metadata for a file you get a common API defined by the OS (but you'll still need to standardize on attribute names and formats, so plenty of room that will still not help), and instead of parsing different files or XML streams you can directly access a bit of metadata. So far I'm not convinced that streams are worth the effort. Not that my opinion is all that relevant here, but still :) Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:54 ` Wichert Akkerman @ 2004-08-26 23:48 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:48 UTC (permalink / raw) To: Wichert Akkerman Cc: Spam, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Wichert Akkerman wrote: >Previously Spam wrote: > > >> Because having user space tools and code will make it not work with >> everything. Keeping stuff in the kernel should make the new features >> transparent to the applications. >> >> > >But having it in the kernel has the same problem. If you read the >Solaris documentation you will see that a bunch of utilties had to >get a new commandline option to be able to access the metadata and >a special utility was added for other applications. > They did it wrong. > If you look >at windows you will see that you need to use a filename like ><realfilename>:<streamname>:$DATA (which obviously does not work for >single-character filenames). > > again, they did it wrong. See Rob Pike's "The Hideous Name" for why that is wrong. >Ignoring samba for a bit which just needs streams to stay compatible >with windows I see few reasons for using streams: > >* files are more complex these days and tend to include multiple > different things: images with thumbnails and exif data, 'office' > documents containing both text and images > >* standard way to add common metadata to a file which can be used for > searching tools (author, copyright, keywords, etc.) > >But both can already be done in userland (modern image formats can store >thumbnails and exif data internally, applications use tar or zip-like >files for documents, etc.). The metadata part is a lot more complicated >as well since the behaviour of attributes might need to be complex: >if I change an image using an application that is not stream-aware, what >should happen to its thumbnail? > >The only common benefits I can see are standardisation and optimization: >instead of every file format or application defining a way to specify >metadata for a file you get a common API defined by the OS (but you'll >still need to standardize on attribute names and formats, so plenty of >room that will still not help), and instead of parsing different files >or XML streams you can directly access a bit of metadata. > >So far I'm not convinced that streams are worth the effort. Not that my >opinion is all that relevant here, but still :) > >Wichert. > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:48 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:48 UTC (permalink / raw) To: Wichert Akkerman Cc: Spam, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Wichert Akkerman wrote: >Previously Spam wrote: > > >> Because having user space tools and code will make it not work with >> everything. Keeping stuff in the kernel should make the new features >> transparent to the applications. >> >> > >But having it in the kernel has the same problem. If you read the >Solaris documentation you will see that a bunch of utilties had to >get a new commandline option to be able to access the metadata and >a special utility was added for other applications. > They did it wrong. > If you look >at windows you will see that you need to use a filename like ><realfilename>:<streamname>:$DATA (which obviously does not work for >single-character filenames). > > again, they did it wrong. See Rob Pike's "The Hideous Name" for why that is wrong. >Ignoring samba for a bit which just needs streams to stay compatible >with windows I see few reasons for using streams: > >* files are more complex these days and tend to include multiple > different things: images with thumbnails and exif data, 'office' > documents containing both text and images > >* standard way to add common metadata to a file which can be used for > searching tools (author, copyright, keywords, etc.) > >But both can already be done in userland (modern image formats can store >thumbnails and exif data internally, applications use tar or zip-like >files for documents, etc.). The metadata part is a lot more complicated >as well since the behaviour of attributes might need to be complex: >if I change an image using an application that is not stream-aware, what >should happen to its thumbnail? > >The only common benefits I can see are standardisation and optimization: >instead of every file format or application defining a way to specify >metadata for a file you get a common API defined by the OS (but you'll >still need to standardize on attribute names and formats, so plenty of >room that will still not help), and instead of parsing different files >or XML streams you can directly access a bit of metadata. > >So far I'm not convinced that streams are worth the effort. Not that my >opinion is all that relevant here, but still :) > >Wichert. > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:20 ` Spam @ 2004-08-26 11:16 ` Christer Weinigel -1 siblings, 0 replies; 1835+ messages in thread From: Christer Weinigel @ 2004-08-26 11:16 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> writes: > Keeping stuff in the kernel should make the new features > transparent to the applications. No, it will make just one special case, rename within the same filesystem, work. (Well, two special cases, rm will also delete the other forks). Unless we add a new copy(2) syscall (which would be nice for other reasons) all applicatons that copy files will fail to copy the streams. So no working cp command, no nautilus nor konqueror without changes to the application. And if you have to change the applications anyway, isn't it much better to agree on a shared library in userspace that everyone uses? Which has the added bonus that it can be made to work on top of Linux, Windows, Ultrix and VMS? /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 11:16 ` Christer Weinigel 0 siblings, 0 replies; 1835+ messages in thread From: Christer Weinigel @ 2004-08-26 11:16 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> writes: > Keeping stuff in the kernel should make the new features > transparent to the applications. No, it will make just one special case, rename within the same filesystem, work. (Well, two special cases, rm will also delete the other forks). Unless we add a new copy(2) syscall (which would be nice for other reasons) all applicatons that copy files will fail to copy the streams. So no working cp command, no nautilus nor konqueror without changes to the application. And if you have to change the applications anyway, isn't it much better to agree on a shared library in userspace that everyone uses? Which has the added bonus that it can be made to work on top of Linux, Windows, Ultrix and VMS? /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:16 ` Christer Weinigel @ 2004-08-26 11:35 ` Spam -1 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:35 UTC (permalink / raw) To: Christer Weinigel Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> writes: >> Keeping stuff in the kernel should make the new features >> transparent to the applications. > No, it will make just one special case, rename within the same > filesystem, work. (Well, two special cases, rm will also delete > the other forks). > Unless we add a new copy(2) syscall (which would be nice for other > reasons) all applicatons that copy files will fail to copy the > streams. How so? The whole idea is that the underlaying OS that handles the copying should also know to copy everything, otherwise you can implement everything into applications and just skip the whole filesystem part. Of course, it won't work if applications themselves are actually transferring the data and create the new files directly onto the filesystem. This just seem so wrong. ~S > So no working cp command, no nautilus nor konqueror without > changes to the application. And if you have to change the > applications anyway, isn't it much better to agree on a shared library > in userspace that everyone uses? Which has the added bonus that it > can be made to work on top of Linux, Windows, Ultrix and VMS? > /Christer ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 11:35 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:35 UTC (permalink / raw) To: Christer Weinigel Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> writes: >> Keeping stuff in the kernel should make the new features >> transparent to the applications. > No, it will make just one special case, rename within the same > filesystem, work. (Well, two special cases, rm will also delete > the other forks). > Unless we add a new copy(2) syscall (which would be nice for other > reasons) all applicatons that copy files will fail to copy the > streams. How so? The whole idea is that the underlaying OS that handles the copying should also know to copy everything, otherwise you can implement everything into applications and just skip the whole filesystem part. Of course, it won't work if applications themselves are actually transferring the data and create the new files directly onto the filesystem. This just seem so wrong. ~S > So no working cp command, no nautilus nor konqueror without > changes to the application. And if you have to change the > applications anyway, isn't it much better to agree on a shared library > in userspace that everyone uses? Which has the added bonus that it > can be made to work on top of Linux, Windows, Ultrix and VMS? > /Christer ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:35 ` Spam (?) @ 2004-08-26 11:33 ` Wichert Akkerman 2004-08-26 11:43 ` Spam ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Wichert Akkerman @ 2004-08-26 11:33 UTC (permalink / raw) To: Spam Cc: Christer Weinigel, Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Spam wrote: > How so? The whole idea is that the underlaying OS that handles the > copying should also know to copy everything, otherwise you can > implement everything into applications and just skip the whole > filesystem part. UNIX doesn't have a copy systemcall, applications copy the data manually. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:33 ` Wichert Akkerman @ 2004-08-26 11:43 ` Spam 2004-08-26 23:56 ` Hans Reiser 2004-08-27 0:00 ` Stelian Ionescu 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:43 UTC (permalink / raw) To: Wichert Akkerman Cc: Christer Weinigel, Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Previously Spam wrote: >> How so? The whole idea is that the underlaying OS that handles the >> copying should also know to copy everything, otherwise you can >> implement everything into applications and just skip the whole >> filesystem part. > UNIX doesn't have a copy systemcall, applications copy the data > manually. Oh, this is very unfortunate and should be a bigger issue to fix. > Wichert. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 11:43 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-26 11:43 UTC (permalink / raw) To: Wichert Akkerman Cc: Christer Weinigel, Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Previously Spam wrote: >> How so? The whole idea is that the underlaying OS that handles the >> copying should also know to copy everything, otherwise you can >> implement everything into applications and just skip the whole >> filesystem part. > UNIX doesn't have a copy systemcall, applications copy the data > manually. Oh, this is very unfortunate and should be a bigger issue to fix. > Wichert. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:43 ` Spam (?) @ 2004-08-27 7:41 ` Bernd Petrovitsch 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:49 ` Christoph Hellwig -1 siblings, 2 replies; 1835+ messages in thread From: Bernd Petrovitsch @ 2004-08-27 7:41 UTC (permalink / raw) To: Spam Cc: Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 2004-08-26 at 13:43 +0200, Spam wrote: > > Previously Spam wrote: > >> How so? The whole idea is that the underlaying OS that handles the > >> copying should also know to copy everything, otherwise you can > >> implement everything into applications and just skip the whole > >> filesystem part. > > > UNIX doesn't have a copy systemcall, applications copy the data > > manually. > > Oh, this is very unfortunate and should be a bigger issue to fix. Then you have to rewrite POSIX und SuSv3. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 7:41 ` Bernd Petrovitsch @ 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:49 ` Christoph Hellwig 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:05 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Spam, Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Bernd Petrovitsch wrote: >On Thu, 2004-08-26 at 13:43 +0200, Spam wrote: > > >>>Previously Spam wrote: >>> >>> >>>> How so? The whole idea is that the underlaying OS that handles the >>>> copying should also know to copy everything, otherwise you can >>>> implement everything into applications and just skip the whole >>>> filesystem part. >>>> >>>> >>>UNIX doesn't have a copy systemcall, applications copy the data >>>manually. >>> >>> >> Oh, this is very unfortunate and should be a bigger issue to fix. >> >> > >Then you have to rewrite POSIX und SuSv3. > > Bernd > > Spam is right. Posix is a standard, not a vision, and the future is always a vision not a standard. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:05 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:05 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Spam, Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Bernd Petrovitsch wrote: >On Thu, 2004-08-26 at 13:43 +0200, Spam wrote: > > >>>Previously Spam wrote: >>> >>> >>>> How so? The whole idea is that the underlaying OS that handles the >>>> copying should also know to copy everything, otherwise you can >>>> implement everything into applications and just skip the whole >>>> filesystem part. >>>> >>>> >>>UNIX doesn't have a copy systemcall, applications copy the data >>>manually. >>> >>> >> Oh, this is very unfortunate and should be a bigger issue to fix. >> >> > >Then you have to rewrite POSIX und SuSv3. > > Bernd > > Spam is right. Posix is a standard, not a vision, and the future is always a vision not a standard. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:05 ` Hans Reiser (?) @ 2004-08-27 8:51 ` Christoph Hellwig -1 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-27 8:51 UTC (permalink / raw) To: Hans Reiser Cc: Bernd Petrovitsch, Spam, Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 01:05:02AM -0700, Hans Reiser wrote: > Spam is right. Posix is a standard, not a vision, and the future is > always a vision not a standard. An old friend of mine always said "if someone has visions he needs to stop somking mushrooms.." ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 7:41 ` Bernd Petrovitsch 2004-08-27 8:05 ` Hans Reiser @ 2004-08-27 8:49 ` Christoph Hellwig 2004-08-27 9:03 ` Bernd Petrovitsch 1 sibling, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-27 8:49 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Spam, Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 09:41:07AM +0200, Bernd Petrovitsch wrote: > > > UNIX doesn't have a copy systemcall, applications copy the data > > > manually. > > > > Oh, this is very unfortunate and should be a bigger issue to fix. > > Then you have to rewrite POSIX und SuSv3. They don't say 'you must now have a copy syscall'. Having one that's actually used by system tools would be a great optimization for many network or distributed filesystems. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:49 ` Christoph Hellwig @ 2004-08-27 9:03 ` Bernd Petrovitsch 0 siblings, 0 replies; 1835+ messages in thread From: Bernd Petrovitsch @ 2004-08-27 9:03 UTC (permalink / raw) To: Christoph Hellwig Cc: Spam, Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 2004-08-27 at 10:49 +0200, Christoph Hellwig wrote: > On Fri, Aug 27, 2004 at 09:41:07AM +0200, Bernd Petrovitsch wrote: > > > > UNIX doesn't have a copy systemcall, applications copy the data > > > > manually. > > > > > > Oh, this is very unfortunate and should be a bigger issue to fix. > > > > Then you have to rewrite POSIX und SuSv3. > > They don't say 'you must now have a copy syscall'. Having one that's > actually used by system tools would be a great optimization for many > network or distributed filesystems. ACK. But actually spam apparently assumes that the kernel - userspace interface is at the wrong abstraction level. And this is conceptually bound IMHO to POSIX. And it does not solve the problem, that the read() and write() sys-calls will not vanish. Even if a copy() sys-call exists. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:33 ` Wichert Akkerman @ 2004-08-26 23:56 ` Hans Reiser 2004-08-26 23:56 ` Hans Reiser 2004-08-27 0:00 ` Stelian Ionescu 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:56 UTC (permalink / raw) To: Wichert Akkerman Cc: Spam, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Wichert Akkerman wrote: >Previously Spam wrote: > > >> How so? The whole idea is that the underlaying OS that handles the >> copying should also know to copy everything, otherwise you can >> implement everything into applications and just skip the whole >> filesystem part. >> >> > >UNIX doesn't have a copy systemcall, applications copy the data >manually. > >Wichert. > > > See sys_reiser4()..... ;-) you can go "A<-B". I have hopes of getting drive vendors to support us in doing single disk copies without even leaving the disk drive..... ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:56 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:56 UTC (permalink / raw) To: Wichert Akkerman Cc: Spam, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Wichert Akkerman wrote: >Previously Spam wrote: > > >> How so? The whole idea is that the underlaying OS that handles the >> copying should also know to copy everything, otherwise you can >> implement everything into applications and just skip the whole >> filesystem part. >> >> > >UNIX doesn't have a copy systemcall, applications copy the data >manually. > >Wichert. > > > See sys_reiser4()..... ;-) you can go "A<-B". I have hopes of getting drive vendors to support us in doing single disk copies without even leaving the disk drive..... ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:56 ` Hans Reiser (?) @ 2004-08-27 1:05 ` Andreas Dilger -1 siblings, 0 replies; 1835+ messages in thread From: Andreas Dilger @ 2004-08-27 1:05 UTC (permalink / raw) To: Hans Reiser Cc: Wichert Akkerman, Spam, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1124 bytes --] On Aug 26, 2004 16:56 -0700, Hans Reiser wrote: > Wichert Akkerman wrote: > >UNIX doesn't have a copy systemcall, applications copy the data > >manually. > > See sys_reiser4()..... ;-) you can go "A<-B". I have hopes of getting > drive vendors to support us in doing single disk copies without even > leaving the disk drive..... Yes, we've wanted to have a "copy" syscall for a while now also, so that Lustre (or other network FS for that matter) can do remote copies of large files without having to go through the VM of the client. The work on sendfile() in 2.6 is a step in that direction, I eagerly await the day when "cp" will try to do a sys_sendfile() call first, before doing the read+write looping it does now. Consider for a network filesystem we are limited to 1/2 of the client network bandwidth (say 50-75MB/s) for copying instead of the server-local or multiple server-server bandwidth for copies (400-600MB/s or more). Cheers, Andreas -- Andreas Dilger http://sourceforge.net/projects/ext2resize/ http://members.shaw.ca/adilger/ http://members.shaw.ca/golinux/ [-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:33 ` Wichert Akkerman 2004-08-26 11:43 ` Spam 2004-08-26 23:56 ` Hans Reiser @ 2004-08-27 0:00 ` Stelian Ionescu 2 siblings, 0 replies; 1835+ messages in thread From: Stelian Ionescu @ 2004-08-27 0:00 UTC (permalink / raw) To: Wichert Akkerman Cc: Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 01:33:32PM +0200, Wichert Akkerman wrote: >Previously Spam wrote: >> How so? The whole idea is that the underlaying OS that handles the >> copying should also know to copy everything, otherwise you can >> implement everything into applications and just skip the whole >> filesystem part. > >UNIX doesn't have a copy systemcall, applications copy the data >manually. couldn't sendfile(2) be used for that ? perhaps a flag parameter could be added to sendfile(2) which specifies whether to copy _all_ metadata if both fd's correspond to files, something like either O_METADATA or O_NOMETADATA -- Stelian Ionescu aka fe[nl]ix Quidquid latine dictum sit, altum viditur. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:16 ` Christer Weinigel (?) (?) @ 2004-08-26 11:58 ` Jamie Lokier -1 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 11:58 UTC (permalink / raw) To: Christer Weinigel Cc: Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christer Weinigel wrote: > all applicatons that copy files will fail to copy the streams. So > no working cp command, no nautilus nor konqueror without changes to > the application. No true. A lot of the fancy stuff depends on metadata which is merely another view of the serialised contents in the flat file. Nautilus & Konqueror thumbnails are an example of that (I gave a few more in other mails). cp works fine with that, and so does ftp. The metadata is just recalculated when it's needed from the target file. > And if you have to change the applications anyway, isn't it much > better to agree on a shared library in userspace that everyone uses? > Which has the added bonus that it can be made to work on top of > Linux, Windows, Ultrix and VMS? Ideally yes, a shared library _or_ at least an agreed representation. However, with filesystem support you can improve performance by avoiding unneed serialisation (write to your huge *Office presentation, read it from another program, no need to wait for the slow tar+compress stage yet it's _as if_ those were done), retain computed metadata with coherency guarantees (e.g. real time search indexes, crypto digests and such), and let the filesystem decide when best to prune computed metadata or convert representations. All of these things can work with a combination of userspace plugins (not the same as reiser4 plugins), and filesystem support. Without the filesystem support, you can still use the userspace plugins -- so apps would still be portable -- but you don't get the extras mentioned above. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:16 ` Christer Weinigel @ 2004-08-26 14:00 ` Denis Vlasenko -1 siblings, 0 replies; 1835+ messages in thread From: Denis Vlasenko @ 2004-08-26 14:00 UTC (permalink / raw) To: Christer Weinigel, Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 14:16, Christer Weinigel wrote: > Spam <spam@tnonline.net> writes: > > Keeping stuff in the kernel should make the new features > > transparent to the applications. > > No, it will make just one special case, rename within the same > filesystem, work. (Well, two special cases, rm will also delete > the other forks). In traditional UNIX filesystems, copy loses some info too (owner,group,mode bits). cp and tar was taught about this and they now can preserve that info. However, I do not think that hacking on cp, tar, etc everytime we need to add something to our filesystems (xattrs, ACLs, whatever) is a terribly attractive idea. I think reiserfs is going to handle that by giving uniform access to such ancillary data. It will be visible as file/meta/* filelets ('small files') or something like that and can be saved/restored by e.g. tar *without* needing to know what data is stored there, and in what format. I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc... Note that even today's files aren't simple "streams of bytes" because they also have names,mode bits etc. -- vda ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 14:00 ` Denis Vlasenko 0 siblings, 0 replies; 1835+ messages in thread From: Denis Vlasenko @ 2004-08-26 14:00 UTC (permalink / raw) To: Christer Weinigel, Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 14:16, Christer Weinigel wrote: > Spam <spam@tnonline.net> writes: > > Keeping stuff in the kernel should make the new features > > transparent to the applications. > > No, it will make just one special case, rename within the same > filesystem, work. (Well, two special cases, rm will also delete > the other forks). In traditional UNIX filesystems, copy loses some info too (owner,group,mode bits). cp and tar was taught about this and they now can preserve that info. However, I do not think that hacking on cp, tar, etc everytime we need to add something to our filesystems (xattrs, ACLs, whatever) is a terribly attractive idea. I think reiserfs is going to handle that by giving uniform access to such ancillary data. It will be visible as file/meta/* filelets ('small files') or something like that and can be saved/restored by e.g. tar *without* needing to know what data is stored there, and in what format. I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc... Note that even today's files aren't simple "streams of bytes" because they also have names,mode bits etc. -- vda ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:00 ` Denis Vlasenko @ 2004-08-26 14:12 ` Rik van Riel -1 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 14:12 UTC (permalink / raw) To: Denis Vlasenko Cc: Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Denis Vlasenko wrote: > I think Hans is not planning turning old "file is a stream of bytes" > into eight-stream octopus. One stream will remain as a 'main' one, > which contains actual data. Others will keep metadata, etc... This is exactly what the Samba people want, though. Office suites can store a document with embedded images and spread sheets "easily" by putting the text, the images and spread sheets all in different file streams. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 14:12 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 14:12 UTC (permalink / raw) To: Denis Vlasenko Cc: Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Denis Vlasenko wrote: > I think Hans is not planning turning old "file is a stream of bytes" > into eight-stream octopus. One stream will remain as a 'main' one, > which contains actual data. Others will keep metadata, etc... This is exactly what the Samba people want, though. Office suites can store a document with embedded images and spread sheets "easily" by putting the text, the images and spread sheets all in different file streams. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:12 ` Rik van Riel (?) @ 2004-08-26 14:24 ` Christophe Saout 2004-08-26 15:44 ` Jamie Lokier -1 siblings, 1 reply; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 14:24 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1259 bytes --] Am Donnerstag, den 26.08.2004, 10:12 -0400 schrieb Rik van Riel: > > I think Hans is not planning turning old "file is a stream of bytes" > > into eight-stream octopus. One stream will remain as a 'main' one, > > which contains actual data. Others will keep metadata, etc... > > This is exactly what the Samba people want, though. > > Office suites can store a document with embedded images > and spread sheets "easily" by putting the text, the > images and spread sheets all in different file streams. Ouch. I think Hans' idea is (I don't know if it is a good idea nor if it is doable, but at least it sounds interesting) to have special compound files where you can do something like this: cp text.txt test.compound/test.txt cp image.jpg test.compound/image.jpg And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access test.compound/test.txt again. The only thing I'm worrying about with this approach is what happens if someone tries to simultaneously open test.compound and test.compound/test.txt. Hans has an example somewhere where he does something like that with /etc/passwd. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:24 ` Christophe Saout @ 2004-08-26 15:44 ` Jamie Lokier 2004-08-26 15:54 ` Rik van Riel 2004-08-27 2:14 ` Hans Reiser 0 siblings, 2 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 15:44 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: > I think Hans' idea is (I don't know if it is a good idea nor if it is > doable, but at least it sounds interesting) to have special compound > files where you can do something like this: > > cp text.txt test.compound/test.txt > cp image.jpg test.compound/image.jpg > > And if you read test.compound (the main stream) you get a special format > that contains all the components. You can copy that single stream of > bytes to another (reiser4) fs and then access test.compound/test.txt > again. (To Rik especially), this is the design which more or less satisfies lots of different goals at once. I wrote: > Properly implemented metadata can: > > (1) operate in both modes simultaneously; > (2) work with unaware applications; > (3) provide performance enhancements to aware applications; > (4) provide storage enhancements to both; > (5) provide useful features that work with standard unmodified unix tools, (1) is by design. You can copy these "files" safely using cp or cat or ftp. You can email them. They're flat files. That's (2). You can poke around inside them. Applications which do poke around inside get performance gains, because they don't have to serialise the flat format every time they "load" or "save" a file. For example, if you have a huge OpenOffice document containing lots of sub-documents and pictures. We can imagine a version of OpenOffice which just opens the sub-documents, and writes the ones which are changed after editing. Also, another program such as a viewer, post-processor or search indexing program only needs to read the parts which have changed; it doesn't have to decode the whole thing after a change. That's (3). We already know how useful this is with big mailboxes, hence people switching to maildir. Now imagine the attachments in each message are also viewable, cat'able etc., and that the filesystem stores each message compressed automatically, or in aggregate (your choice) and that you can use both mbox and maildir tools on the same mailbox. That's (4). You can ls, cp, grep, less, vi, gqview etc. the pieces of files, and every supported file format needs just one tool to provide the interface to its contents. That's (5). While I'm here I might as well add: (6) improve support for real time search indexing. The "view coherency" mechanism which is needed for efficient userspace VFS support has just the right properties to keep informing an indexer of changes. They could even enforce operation orderings if we cared about _perfect_ real-timeness, although we probably don't. (7) yet another way of doing distros. Why not just install .tgz or .zip files of entire binary packages, and let the filesystem unpack the bits that are needed when they're needed. Some platforms (i.e. Java) do this already with .jar files and it's quite popular. This would extend it in a versatile way. Rik, I hope I've explained how those 5 properties are satisfied by Hans Reiser's brilliant design (ahem, great minds think alike, and remember Linus likes files-as-directories too :). But if I haven't please do pick holes. > The only thing I'm worrying about with this approach is what happens if > someone tries to simultaneously open test.compound and > test.compound/test.txt. It is possible to implement it so that every change to one file immediately affects the other in the appropriate way, _as if_ the other file is regenerated every time before reading it. This is what I mean by coherent views. It can be done even when userspace is doing the work, and the obvious mechanisms just happen to have the property of being useful in other ways (like a better dnotify, but I digress...). They're just an ordinary cache coherency protocols. However, if you are writing a few bytes to test.compound, then the coherent view in test.compound/test.txt is probably a corrupt or non-existent file, because test.compound's intermediate state is a truncated and/or corrupt archive file, until you've written some more. That doesn't look nice, but it's exactly the same problem as you get when you're writing to a file that another program is reading. If you do that on a live http server, for example editing a .html file, very occasionally someone will see a truncated file. Even if your editor uses renames to update atomically, sometimes you need to update more than one file atomically. It can be done, but rarely is. Is there a solution to this? The obvious one is when you're writing to test.compound, don't read from test.compound/test.txt. That's _no different_ from the advice which says when you're saving a file from OpenOffice, don't load it into a document viewer during the time you're writing it, and don't serve it. The other advice is to save to a different file and call rename(). Again, you get pretty much the same results as now: it atomically changes, and currently open references to the old contents will continue to reference the old contents. There's another solution: transactions. reiser4 offers some kind of transaction facility, doesn't it? This is a fine example of when might want to declare that a sequence of writes needs to be a single transaction, visible only when the whole sequence is complete. Perfect! I don't know reiser4's interface to that, but it would make a lot of sense to have a simple shell tool called "atomic", to wrap a sequence of operations. That would be useful for writing files _and_ useful for those pesky "update several files on the web server at once" moments. (Yes I know you can build a hard link tree and rename the root directory, although even that doesn't always work). -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:44 ` Jamie Lokier @ 2004-08-26 15:54 ` Rik van Riel 2004-08-27 2:14 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 15:54 UTC (permalink / raw) To: Jamie Lokier Cc: Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > Christophe Saout wrote: > > And if you read test.compound (the main stream) you get a special format > > that contains all the components. You can copy that single stream of > > bytes to another (reiser4) fs and then access test.compound/test.txt > > again. > > (To Rik especially), this is the design which more or less satisfies > lots of different goals at once. And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ? Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 15:54 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 15:54 UTC (permalink / raw) To: Jamie Lokier Cc: Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > Christophe Saout wrote: > > And if you read test.compound (the main stream) you get a special format > > that contains all the components. You can copy that single stream of > > bytes to another (reiser4) fs and then access test.compound/test.txt > > again. > > (To Rik especially), this is the design which more or less satisfies > lots of different goals at once. And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ? Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:54 ` Rik van Riel @ 2004-08-26 16:04 ` Christophe Saout -1 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 16:04 UTC (permalink / raw) To: Rik van Riel Cc: Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1291 bytes --] Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel: > > > And if you read test.compound (the main stream) you get a special format > > > that contains all the components. You can copy that single stream of > > > bytes to another (reiser4) fs and then access test.compound/test.txt > > > again. > > > > (To Rik especially), this is the design which more or less satisfies > > lots of different goals at once. > > And if an unaware application reads the compound file > and then writes it out again, does the filesystem > interpret the contents and create the other streams ? > > Unless I overlook something (please tell me what), the > scheme just proposed requires filesystems to look at > the content of files that is being written out, in > order to make the streams work. Yes, the main compound stream unaware applications would see would just be a writable view of its contents. Probably not trivial to implement but doable. It should be a simple format. Something simliar to tar. The worst thing that can happen is people start writing plugins for every existing compound format out there. It should be the other way around, agree on a simple compound format and encourage applications to use this one if the want to use this advantage. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 16:04 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 16:04 UTC (permalink / raw) To: Rik van Riel Cc: Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1291 bytes --] Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel: > > > And if you read test.compound (the main stream) you get a special format > > > that contains all the components. You can copy that single stream of > > > bytes to another (reiser4) fs and then access test.compound/test.txt > > > again. > > > > (To Rik especially), this is the design which more or less satisfies > > lots of different goals at once. > > And if an unaware application reads the compound file > and then writes it out again, does the filesystem > interpret the contents and create the other streams ? > > Unless I overlook something (please tell me what), the > scheme just proposed requires filesystems to look at > the content of files that is being written out, in > order to make the streams work. Yes, the main compound stream unaware applications would see would just be a writable view of its contents. Probably not trivial to implement but doable. It should be a simple format. Something simliar to tar. The worst thing that can happen is people start writing plugins for every existing compound format out there. It should be the other way around, agree on a simple compound format and encourage applications to use this one if the want to use this advantage. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:04 ` Christophe Saout (?) @ 2004-08-26 16:07 ` Christoph Hellwig 2004-08-26 16:44 ` Jamie Lokier -1 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:07 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 06:04:42PM +0200, Christophe Saout wrote: > It should be a simple format. Something simliar to tar. The worst thing > that can happen is people start writing plugins for every existing > compound format out there. It should be the other way around, agree on a > simple compound format and encourage applications to use this one if the > want to use this advantage. or placing this into a userspace helper that the kernel can invoke transparently. If you're doing funny thing like that for your swap device deadlocks are the last of your worries :) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:07 ` Christoph Hellwig @ 2004-08-26 16:44 ` Jamie Lokier 0 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 16:44 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Rik van Riel, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Christoph Hellwig wrote: > or placing this into a userspace helper that the kernel can invoke > transparently. Yes, exactly this for most file formats. Ideally, with _some_ user control over the handlers for their own files a bit like Hurd. Not just archive files: metadata pulled from audio files, images, distro packages, etc. In other words much the same as Gnome and KDE's things, but with more emphasis on metadata extractors and efficient on-disk cache interaction, and a sane page-cache coherency interface (possibly FUSE). Metadata extraction is useful for search and indexing purposes. (That, and a way to hook change notifications up with indexers -- again comes with any proper coherency interface). For new applications using a container, it's tempting to want one format which is optimised for this. But really many applications want to write XML or compressed XML, or something else that isn't especially efficient but is easy to work with on many OSes. We should support applications whatever format they need to work with, if providing an interior view as a directory is useful for them. (This implies applications can introduce their own formats for files they need to operate on.) By the way, lazy updating of containers is an even bigger advantage if you have to use XML as the flat format, because the savings from not serialising when you don't need to are more pronounced. For some applications the "content" isn't structured in a way which maps naturally to a plain archive file. Hans' /etc/passwd is a simple example: at least one of the "flat" representations, when there are enough entries, probably ought to be a DB file for faster lookup. (Though, it would be great if /etc/passwd could automagically get fast lookup due to an automagic underlying representation, while still appearing to be a flat file when you cat it). Also, imagine "cd" into a MySQL table file. It might just work. By the way, we don't need containers to waste huge amounts of space like the Microsoft Word "fast save" documents did. Because our containers are created on demand, there is no performance advantage in them having unused holes, unlike the case were Word was saving changes periodically. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:04 ` Christophe Saout @ 2004-08-26 18:40 ` David Masover -1 siblings, 0 replies; 1835+ messages in thread From: David Masover @ 2004-08-26 18:40 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christophe Saout wrote: | Yes, the main compound stream unaware applications would see would just | be a writable view of its contents. Probably not trivial to implement | but doable. Not the main stream, I think, but we do need a way to make "eight-stream octopus" files portable. But there needs to be a copy(2) syscall, and apps need to start supporting it. After all, copy-on-write was planned for Reiser4, now 4.1*. And don't anyone dare suggest "userland library". | It should be a simple format. Something simliar to tar. The worst thing Simple, yes. Similar to tar, no. How about a serializer plugin? foo.mp3/serialize (say) should return some single stream which contains all of foo.mp3. It could work for directories as well, recursively, at least on the same filesystem -- you'd back up by doing cp /serialize /backup/20040826 This has two potential problems. One is security -- I want to be able to serialize foo.mp3 without being root, but root doesn't want me to be able to serialize /etc and thus get access to shadow. The second problem is that in order to actually be used for backup, we need snapshots, which are not done yet*. Currently, it'd have to lock the file/directory (recursively), which you don't want to do to entire live filesystems... So, no massive backups of live systems yet, but definitely useful for sending around mp3s and such. | that can happen is people start writing plugins for every existing | compound format out there. It should be the other way around, agree on a | simple compound format and encourage applications to use this one if the | want to use this advantage. For the format, I vote for whatever format we're using at the storage layer. It'd be a lot faster that way, both in execution time (I think) and in development time (I'm sure). Plus, imagine the restore: cat /backup/20040826 > /dev/hda5 resizefs_reiser4 /dev/hda5 * Disclaimer -- all this is "I think, last I checked". -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQS4uhngHNmZLgCUhAQKTaw//WCHt0Umgaf0GaMUpEr0PLhiXhFTYQqKV MKgpVt6SjP7lC9k3RbuCnPCLyOJGb9lIqo+Wd+B3b4TcQAQd4OYsSI6WiMPQjxXY OdNnB7/xYkkd/ULw7/7IGcokyc4FRDjhTyOMe4KscQS/6uhdX1dCriCCzVq7iELx u3HLulunHv1bwHbnZ6CyEZdQEr3j8XUu62+IXPTc6hWnfRRCvskV8Y4qyyVd8wFY Kp44TxbJJSzDeWmuI/YYkxVJsPmSoxRLxmw9/B8D72KGNg1SYLnM4ueIc5IV8RH0 kG/o8tzwTRxE/eqfcHP+UBt6yXZR/i84DDogAmJFKP73g9X+U+J5o/kONOBQODhY qY5LUwQu4UjlzXttMxC4NmZPESdUw85DgC/GhTQlsBMYBcmQHmpLAK67qt83+Kg0 1QqGPqJm4HL4weHLNknJ0DukLNKH7Bw5CVg335IwruaoiKUG7HDgthUhA319FCab VvYa+bEfeBEw9RorSE/gWM9Hg73luM7eoSzKc9zHgx++cACTi8SBsTFWqI2FU6DX UyWG5sNPybOiczpPsFu5S5FJBBNM3fLjkfeYgfqxUIjxGTNeUn/g/9I5ShUrCF1o T+3nd1jFxfoehYGkfGB2YLpdeNVBv4EdKK16j39+iducfJrXnRlL1yjdo/Kte2uz tTQZw3uTnhU= =7bBF -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 18:40 ` David Masover 0 siblings, 0 replies; 1835+ messages in thread From: David Masover @ 2004-08-26 18:40 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christophe Saout wrote: | Yes, the main compound stream unaware applications would see would just | be a writable view of its contents. Probably not trivial to implement | but doable. Not the main stream, I think, but we do need a way to make "eight-stream octopus" files portable. But there needs to be a copy(2) syscall, and apps need to start supporting it. After all, copy-on-write was planned for Reiser4, now 4.1*. And don't anyone dare suggest "userland library". | It should be a simple format. Something simliar to tar. The worst thing Simple, yes. Similar to tar, no. How about a serializer plugin? foo.mp3/serialize (say) should return some single stream which contains all of foo.mp3. It could work for directories as well, recursively, at least on the same filesystem -- you'd back up by doing cp /serialize /backup/20040826 This has two potential problems. One is security -- I want to be able to serialize foo.mp3 without being root, but root doesn't want me to be able to serialize /etc and thus get access to shadow. The second problem is that in order to actually be used for backup, we need snapshots, which are not done yet*. Currently, it'd have to lock the file/directory (recursively), which you don't want to do to entire live filesystems... So, no massive backups of live systems yet, but definitely useful for sending around mp3s and such. | that can happen is people start writing plugins for every existing | compound format out there. It should be the other way around, agree on a | simple compound format and encourage applications to use this one if the | want to use this advantage. For the format, I vote for whatever format we're using at the storage layer. It'd be a lot faster that way, both in execution time (I think) and in development time (I'm sure). Plus, imagine the restore: cat /backup/20040826 > /dev/hda5 resizefs_reiser4 /dev/hda5 * Disclaimer -- all this is "I think, last I checked". -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQS4uhngHNmZLgCUhAQKTaw//WCHt0Umgaf0GaMUpEr0PLhiXhFTYQqKV MKgpVt6SjP7lC9k3RbuCnPCLyOJGb9lIqo+Wd+B3b4TcQAQd4OYsSI6WiMPQjxXY OdNnB7/xYkkd/ULw7/7IGcokyc4FRDjhTyOMe4KscQS/6uhdX1dCriCCzVq7iELx u3HLulunHv1bwHbnZ6CyEZdQEr3j8XUu62+IXPTc6hWnfRRCvskV8Y4qyyVd8wFY Kp44TxbJJSzDeWmuI/YYkxVJsPmSoxRLxmw9/B8D72KGNg1SYLnM4ueIc5IV8RH0 kG/o8tzwTRxE/eqfcHP+UBt6yXZR/i84DDogAmJFKP73g9X+U+J5o/kONOBQODhY qY5LUwQu4UjlzXttMxC4NmZPESdUw85DgC/GhTQlsBMYBcmQHmpLAK67qt83+Kg0 1QqGPqJm4HL4weHLNknJ0DukLNKH7Bw5CVg335IwruaoiKUG7HDgthUhA319FCab VvYa+bEfeBEw9RorSE/gWM9Hg73luM7eoSzKc9zHgx++cACTi8SBsTFWqI2FU6DX UyWG5sNPybOiczpPsFu5S5FJBBNM3fLjkfeYgfqxUIjxGTNeUn/g/9I5ShUrCF1o T+3nd1jFxfoehYGkfGB2YLpdeNVBv4EdKK16j39+iducfJrXnRlL1yjdo/Kte2uz tTQZw3uTnhU= =7bBF -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:40 ` David Masover (?) @ 2004-08-26 21:17 ` David Lang -1 siblings, 0 replies; 1835+ messages in thread From: David Lang @ 2004-08-26 21:17 UTC (permalink / raw) To: David Masover Cc: Christophe Saout, Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, David Masover wrote: > Christophe Saout wrote: > > | Yes, the main compound stream unaware applications would see would just > | be a writable view of its contents. Probably not trivial to implement > | but doable. > > Not the main stream, I think, but we do need a way to make "eight-stream > octopus" files portable. But there needs to be a copy(2) syscall, and > apps need to start supporting it. After all, copy-on-write was planned > for Reiser4, now 4.1*. And don't anyone dare suggest "userland library". > > | It should be a simple format. Something simliar to tar. The worst thing > > Simple, yes. Similar to tar, no. > > How about a serializer plugin? foo.mp3/serialize (say) should return > some single stream which contains all of foo.mp3. It could work for > directories as well, recursively, at least on the same filesystem -- > you'd back up by doing > > cp /serialize /backup/20040826 > > This has two potential problems. One is security -- I want to be able > to serialize foo.mp3 without being root, but root doesn't want me to be > able to serialize /etc and thus get access to shadow. > > The second problem is that in order to actually be used for backup, we > need snapshots, which are not done yet*. Currently, it'd have to lock > the file/directory (recursively), which you don't want to do to entire > live filesystems... So, no massive backups of live systems yet, but > definitely useful for sending around mp3s and such. > > | that can happen is people start writing plugins for every existing > | compound format out there. It should be the other way around, agree on a > | simple compound format and encourage applications to use this one if the > | want to use this advantage. > > For the format, I vote for whatever format we're using at the storage > layer. It'd be a lot faster that way, both in execution time (I think) > and in development time (I'm sure). Plus, imagine the restore: > > cat /backup/20040826 > /dev/hda5 > resizefs_reiser4 /dev/hda5 > one of the virtual metatags should be something about the method to serialize the data if you are viewing a tarfile as a directory the VFS/Filesystem shoud know that the link between the whole thing and the contents is tar, and it should be possible for userspace programs to find this out by reading teh appropriate thing inside the virtual directory (and eventually change it as well, which would trigger regeneration of the base object, or at least noting that the current version is wrong and needs to be regenerated) David Lang -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:04 ` Christophe Saout ` (2 preceding siblings ...) (?) @ 2004-08-26 21:05 ` David Lang 2004-08-26 21:19 ` Jamie Lokier 2004-08-26 22:30 ` Felipe Alfaro Solana -1 siblings, 2 replies; 1835+ messages in thread From: David Lang @ 2004-08-26 21:05 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Christophe Saout wrote: > Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel: > >>>> And if you read test.compound (the main stream) you get a special format >>>> that contains all the components. You can copy that single stream of >>>> bytes to another (reiser4) fs and then access test.compound/test.txt >>>> again. >>> >>> (To Rik especially), this is the design which more or less satisfies >>> lots of different goals at once. >> >> And if an unaware application reads the compound file >> and then writes it out again, does the filesystem >> interpret the contents and create the other streams ? >> >> Unless I overlook something (please tell me what), the >> scheme just proposed requires filesystems to look at >> the content of files that is being written out, in >> order to make the streams work. > > Yes, the main compound stream unaware applications would see would just > be a writable view of its contents. Probably not trivial to implement > but doable. > > It should be a simple format. Something simliar to tar. The worst thing > that can happen is people start writing plugins for every existing > compound format out there. It should be the other way around, agree on a > simple compound format and encourage applications to use this one if the > want to use this advantage. > I also don't see why the VFS/Filesystem can't decide that (for example) this tar.gz is so active that instead of storing it as a tar.gz and providing a virtual directory of the contents that it instead stores the directory of the contents and makes the tar.gz virtual (regenerating it as needed or as extra system resources are available) implementation wise I see headaches in doing this, but conceptually this is just an optimization that could take place in the future if we fine that it's needed. David Lang -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:05 ` David Lang @ 2004-08-26 21:19 ` Jamie Lokier 2004-08-26 21:31 ` Linus Torvalds 2004-08-26 22:30 ` Felipe Alfaro Solana 1 sibling, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 21:19 UTC (permalink / raw) To: David Lang Cc: Christophe Saout, Rik van Riel, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list David Lang wrote: > I also don't see why the VFS/Filesystem can't decide that (for example) > this tar.gz is so active that instead of storing it as a tar.gz and > providing a virtual directory of the contents that it instead stores the > directory of the contents and makes the tar.gz virtual (regenerating it as > needed or as extra system resources are available) Absolutely. It could keep both views, if they're both actively being used. Or more than both, if there are more. (You could think of compression being an alternate view, and both compressed and uncompressed may as well remain on disk if there's space and it's being actively accessed. > implementation wise I see headaches in doing this, but conceptually this > is just an optimization that could take place in the future if we fine > that it's needed. untarring a big source tree takes ages. I wouldn't like to do it after every reboot, if there was a .tar.bz2 tree I looked at often. That's why I have things like glibc untarred in my home directory. With good cacheing, I could cd into the .tar.bz2 files and _effectively_ have the performance of untarred source trees for the ones I look at often on my disk -- automatically cleaned if the space if needed for something else, too. It would be quite nice. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:19 ` Jamie Lokier @ 2004-08-26 21:31 ` Linus Torvalds 0 siblings, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 21:31 UTC (permalink / raw) To: Jamie Lokier Cc: David Lang, Christophe Saout, Rik van Riel, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > > With good cacheing, I could cd into the .tar.bz2 files and > _effectively_ have the performance of untarred source trees for the > ones I look at often on my disk -- automatically cleaned if the space > if needed for something else, too. It would be quite nice. What you're talking about is just a caching filesystem. Then you can have any deamon do whatever it wants to fill it - whether from tar-files or over the network or anything else. I think you should ask David Howells about it.. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:05 ` David Lang 2004-08-26 21:19 ` Jamie Lokier @ 2004-08-26 22:30 ` Felipe Alfaro Solana 2004-08-26 22:32 ` Chris Dukes 2004-08-26 22:52 ` David Lang 1 sibling, 2 replies; 1835+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 22:30 UTC (permalink / raw) To: David Lang Cc: Christophe Saout, Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 23:05, David Lang wrote: > I also don't see why the VFS/Filesystem can't decide that (for example) > this tar.gz is so active that instead of storing it as a tar.gz and > providing a virtual directory of the contents that it instead stores the > directory of the contents and makes the tar.gz virtual (regenerating it as > needed or as extra system resources are available) Because that would mean the kernel should "talk" the tar format, which is, IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to perform this task, instead. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:30 ` Felipe Alfaro Solana @ 2004-08-26 22:32 ` Chris Dukes 2004-08-26 22:52 ` David Lang 1 sibling, 0 replies; 1835+ messages in thread From: Chris Dukes @ 2004-08-26 22:32 UTC (permalink / raw) To: Felipe Alfaro Solana Cc: David Lang, Christophe Saout, Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 12:30:19AM +0200, Felipe Alfaro Solana wrote: > On Thursday 26 August 2004 23:05, David Lang wrote: > > > I also don't see why the VFS/Filesystem can't decide that (for example) > > this tar.gz is so active that instead of storing it as a tar.gz and > > providing a virtual directory of the contents that it instead stores the > > directory of the contents and makes the tar.gz virtual (regenerating it as > > needed or as extra system resources are available) > > Because that would mean the kernel should "talk" the tar format, which is, > IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to > perform this task, instead. Wasn't HURD supposed to do this for us? -- Chris Dukes Warning: Do not use the reflow toaster oven to prepare foods after it has been used for solder paste reflow. http://www.stencilsunlimited.com/stencil_article_page5.htm ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:30 ` Felipe Alfaro Solana 2004-08-26 22:32 ` Chris Dukes @ 2004-08-26 22:52 ` David Lang 1 sibling, 0 replies; 1835+ messages in thread From: David Lang @ 2004-08-26 22:52 UTC (permalink / raw) To: Felipe Alfaro Solana Cc: Christophe Saout, Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Felipe Alfaro Solana wrote: > On Thursday 26 August 2004 23:05, David Lang wrote: > >> I also don't see why the VFS/Filesystem can't decide that (for example) >> this tar.gz is so active that instead of storing it as a tar.gz and >> providing a virtual directory of the contents that it instead stores the >> directory of the contents and makes the tar.gz virtual (regenerating it as >> needed or as extra system resources are available) > > Because that would mean the kernel should "talk" the tar format, which is, > IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to > perform this task, instead. > the kernel will definantly need the ability to use user-space code to do the transformations from one version to the other (if nothing else think of the thumbnail version of images, we don't want the image manipulation code in the kernel and we definantly want this sort of option available) the interesting issue is going to be defining the kernel->user-space interface for doing the extractions. David Lang -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:54 ` Rik van Riel (?) (?) @ 2004-08-26 16:53 ` Jamie Lokier 2004-08-26 19:32 ` John Stoffel 2004-08-30 17:37 ` Alex Zarochentsev -1 siblings, 2 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 16:53 UTC (permalink / raw) To: Rik van Riel Cc: Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > And if an unaware application reads the compound file > and then writes it out again, does the filesystem > interpret the contents and create the other streams ? Yes, exactly that. The streams are created on demand of course, and by userspace helpers when that's appropriate which I suspect it almost always is. > Unless I overlook something (please tell me what), the > scheme just proposed requires filesystems to look at > the content of files that is being written out, in > order to make the streams work. Yes. Hence the idea of coherent views between two files: writing to one affects the content of the other, although the calcalation is only done on demand (or when the fs wants to migrate the representation -- for example, creating the flat container prior to deleting the regeneratable pieces in order to save space). I haven't seen anything from Namesys that says they'll do that. I have the impression the streams are just generated in memory on the fly, not stored on disk with a cacheing policy, but that's just an impression. (We've all seen the Namesys white papers, they're not _that_ revealing). :) I'm just pointing out how to do it _right_. I think it will turn out like this eventually, either next year or some time over the next decade after some iterations. Inevitable. Mark my words, etc. :) -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:53 ` Jamie Lokier @ 2004-08-26 19:32 ` John Stoffel 2004-08-30 17:37 ` Alex Zarochentsev 1 sibling, 0 replies; 1835+ messages in thread From: John Stoffel @ 2004-08-26 19:32 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie> Rik van Riel wrote: >> And if an unaware application reads the compound file >> and then writes it out again, does the filesystem >> interpret the contents and create the other streams ? Jamie> Yes, exactly that. The streams are created on demand of Jamie> course, and by userspace helpers when that's appropriate which Jamie> I suspect it almost always is. So how would a program that converts between a JPEG file (with exif data) and a PNG work, such as ImageMagick? Are we proposing to teach the VFS (or worse yet each filesystem) how to do this? I've been following this discussion a bit and I'm not sure that I've actually seen any concrete examples of where this is a *good* thing to have. People talk about only having to modify 20 bytes at a time instead of reading and writing 1mb of data. Isn't that what mmap() does? Now I can sorta understand the idea that having a directory look like a file is neat, and certainly simplifies some aspects, but I think that going all the way down to the logical conclusion here is a bit silly. To use the principle of blowing things up to make them very large or very small, what happens if I decide that the best idea is to make all files just be directories which contain single byte files? Isn't that the logical extreme here? So my 1mb JPEG file is not just some image data and header info in multiple files, but it's really just 1 million (ok 1024 * 1024) individual files that the VFS knows how to put together. Seems like the logical extreme. Oh wait, maybe we should be exposing a single file per bit instead! >From my point of view, there lies madness. As Rik pointed out, how do backup and restore tools work with this stuff? Most people could care less about how their data is organized, but they certainly care when they can't restore it from backups. I'd really like to see a concrete example from Hans or other proponents about why this makes things easier/faster/better to do. Mostly, I've just seen "proof by vigorous handwaving" that it's a good thing. In alot of ways, I think people are going in the wrong direction, you want to excapsulate and hide the details more, not expose them. That's what a good API does, it hides the details while giving you a rich set of semantics to manage your data. God knows I'm not smart enough or driven enough to actually come up with my own ideas, but I certainly haven't seen anyone else (even Linus) come up with an earth shattering arguement to say why this is the right move to make. As Linus says, most of the OS's job is to mediate access to objects/data. Why do we want to expose such low level data then? John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies stoffel@lucent.com - http://www.lucent.com - 978-952-7548 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 19:32 ` John Stoffel 0 siblings, 0 replies; 1835+ messages in thread From: John Stoffel @ 2004-08-26 19:32 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie> Rik van Riel wrote: >> And if an unaware application reads the compound file >> and then writes it out again, does the filesystem >> interpret the contents and create the other streams ? Jamie> Yes, exactly that. The streams are created on demand of Jamie> course, and by userspace helpers when that's appropriate which Jamie> I suspect it almost always is. So how would a program that converts between a JPEG file (with exif data) and a PNG work, such as ImageMagick? Are we proposing to teach the VFS (or worse yet each filesystem) how to do this? I've been following this discussion a bit and I'm not sure that I've actually seen any concrete examples of where this is a *good* thing to have. People talk about only having to modify 20 bytes at a time instead of reading and writing 1mb of data. Isn't that what mmap() does? Now I can sorta understand the idea that having a directory look like a file is neat, and certainly simplifies some aspects, but I think that going all the way down to the logical conclusion here is a bit silly. To use the principle of blowing things up to make them very large or very small, what happens if I decide that the best idea is to make all files just be directories which contain single byte files? Isn't that the logical extreme here? So my 1mb JPEG file is not just some image data and header info in multiple files, but it's really just 1 million (ok 1024 * 1024) individual files that the VFS knows how to put together. Seems like the logical extreme. Oh wait, maybe we should be exposing a single file per bit instead! From my point of view, there lies madness. As Rik pointed out, how do backup and restore tools work with this stuff? Most people could care less about how their data is organized, but they certainly care when they can't restore it from backups. I'd really like to see a concrete example from Hans or other proponents about why this makes things easier/faster/better to do. Mostly, I've just seen "proof by vigorous handwaving" that it's a good thing. In alot of ways, I think people are going in the wrong direction, you want to excapsulate and hide the details more, not expose them. That's what a good API does, it hides the details while giving you a rich set of semantics to manage your data. God knows I'm not smart enough or driven enough to actually come up with my own ideas, but I certainly haven't seen anyone else (even Linus) come up with an earth shattering arguement to say why this is the right move to make. As Linus says, most of the OS's job is to mediate access to objects/data. Why do we want to expose such low level data then? John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies stoffel@lucent.com - http://www.lucent.com - 978-952-7548 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:32 ` John Stoffel (?) @ 2004-08-26 19:51 ` Denis Vlasenko -1 siblings, 0 replies; 1835+ messages in thread From: Denis Vlasenko @ 2004-08-26 19:51 UTC (permalink / raw) To: John Stoffel, Jamie Lokier Cc: Rik van Riel, Christophe Saout, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 22:32, John Stoffel wrote: > Jamie> Rik van Riel wrote: > >> And if an unaware application reads the compound file > >> and then writes it out again, does the filesystem > >> interpret the contents and create the other streams ? > > Jamie> Yes, exactly that. The streams are created on demand of > Jamie> course, and by userspace helpers when that's appropriate which > Jamie> I suspect it almost always is. > > So how would a program that converts between a JPEG file (with exif > data) and a PNG work, such as ImageMagick? Are we proposing to teach > the VFS (or worse yet each filesystem) how to do this? > > I've been following this discussion a bit and I'm not sure that I've > actually seen any concrete examples of where this is a *good* thing to > have. People talk about only having to modify 20 bytes at a time > instead of reading and writing 1mb of data. Isn't that what mmap() > does? > > Now I can sorta understand the idea that having a directory look like > a file is neat, and certainly simplifies some aspects, but I think > that going all the way down to the logical conclusion here is a bit > silly. > > To use the principle of blowing things up to make them very large or > very small, what happens if I decide that the best idea is to make all > files just be directories which contain single byte files? Isn't that > the logical extreme here? So my 1mb JPEG file is not just some image > data and header info in multiple files, but it's really just 1 > million (ok 1024 * 1024) individual files that the VFS knows how to > put together. Seems like the logical extreme. Oh wait, maybe we > should be exposing a single file per bit instead! It is doable. It doesn't mean we shall do this. I think some reiser4 supporters try to do too much too soon. It is more sensible to do it in small steps. I think we can start small and make all file metadata to be accessible via file/meta/{uid,gid,mode,mtime,atime,...}. Normal tools will be unable to see file/meta/ because file is not a directory and file/ is not a directory (i.e. open(O_DIRECTORY) will fail on them). Hardlinking to file/meta/* is not allowed. However, file/. _is_ a directory. We can make a tar-like tool which can do its work and save/restore metadata along with data by just tarring/untarring file/meta/*. Notice how this 'new tar' does not need to know about exact kind of metadata supported by fs. You can add ACLs to the fs, and this 'new tar' will be able to save/restore ACLs. Without any modufications. Contrast this with existing tar which has hardcoded knowlendge about uid,gid,mode,etc... -- vda ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:32 ` John Stoffel (?) (?) @ 2004-08-26 20:13 ` Jamie Lokier -1 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 20:13 UTC (permalink / raw) To: John Stoffel Cc: Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list John Stoffel wrote: > Jamie> Yes, exactly that. The streams are created on demand of > Jamie> course, and by userspace helpers when that's appropriate which > Jamie> I suspect it almost always is. > > So how would a program that converts between a JPEG file (with exif > data) and a PNG work, such as ImageMagick? Are we proposing to teach > the VFS (or worse yet each filesystem) how to do this? No. (1) That's what "userspace helpers" are for. (2) What does image format conversion have to do with viewing files in their component parts? (3) I suspect someone will write a plugin that does indeed convert virtually every known image format to a common format (probably PNG and something "raw") at some point. Why not? It's just a small script (it would run ImageMagick!). > I've been following this discussion a bit and I'm not sure that I've > actually seen any concrete examples of where this is a *good* thing to > have. People talk about only having to modify 20 bytes at a time > instead of reading and writing 1mb of data. Isn't that what mmap() > does? Sure, if you think mmap() is an easy substitute for "parse the author & license tags from this [unknown format] audio, video or font file". > Now I can sorta understand the idea that having a directory look like > a file is neat, and certainly simplifies some aspects, but I think > that going all the way down to the logical conclusion here is a bit > silly. > > To use the principle of blowing things up to make them very large or > very small, what happens if I decide that the best idea is to make all > files just be directories which contain single byte files? Isn't that > the logical extreme here? So my 1mb JPEG file is not just some image > data and header info in multiple files, but it's really just 1 > million (ok 1024 * 1024) individual files that the VFS knows how to > put together. Seems like the logical extreme. Oh wait, maybe we > should be exposing a single file per bit instead! If you decide to do that, you are welcome to write the userspace helper which creates that view in your directory. Why not? It's very silly, but it's no sillier then writing a program to read your 1mb JPEG and write it out one file per bit, which you're welcome to write right now. > >From my point of view, there lies madness. As Rik pointed out, how do > backup and restore tools work with this stuff? Most people could care > less about how their data is organized, but they certainly care when > they can't restore it from backups. Generated views are something which should _not_ be backed up and restored. That's not what they are for. Auxiliary metadata, such as author info and signatures which cannot be stored in the main file for some reason -- that should be backed up. Permissions, that should be backed up. But not views which are computed from the main file. You don't need to back them up, and they don't need to take any real space. They're virtual, just like an infinitely deep directory on a web server can be virtual. You can make those very silly too, if you want to. (Some help from the filesystem with storing temporarily cached values is fine, for performance, but we shouldn't pretend that generated views are anything other than virtual). > I'd really like to see a concrete example from Hans or other > proponents about why this makes things easier/faster/better to do. > Mostly, I've just seen "proof by vigorous handwaving" that it's a good > thing. > > In alot of ways, I think people are going in the wrong direction, you > want to excapsulate and hide the details more, not expose them. > That's what a good API does, it hides the details while giving you a > rich set of semantics to manage your data. Would you like to propose an alternate API? I was under the impression that we were discussing one, right here in this thread. It's called POSIX with an extension where you can cd into a file. It has a rich set of semantics to manage your data, and works with a lot of familiar programs. A better one would be welcome, if you have an idea. > God knows I'm not smart enough or driven enough to actually come up > with my own ideas, but I certainly haven't seen anyone else (even > Linus) come up with an earth shattering arguement to say why this is > the right move to make. Come up with a better one otherwise it's right ;) > As Linus says, most of the OS's job is to mediate access to > objects/data. Why do we want to expose such low level data then? Eh? In this case the OS is mediating by providing a uniform interface between programs that access the data and programs that handle file formats. What is this "low level" data you speak of? I would say something like the "designer" of a font, or the "from address" and "subject" of an email are very high level abstractions, and that's among the sorts of things which are to be exposed by these interfaces. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:53 ` Jamie Lokier 2004-08-26 19:32 ` John Stoffel @ 2004-08-30 17:37 ` Alex Zarochentsev 2004-08-30 18:51 ` Hans Reiser 1 sibling, 1 reply; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-30 17:37 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote: > Rik van Riel wrote: > > And if an unaware application reads the compound file > > and then writes it out again, does the filesystem > > interpret the contents and create the other streams ? > > Yes, exactly that. The streams are created on demand of course, and > by userspace helpers when that's appropriate which I suspect it almost > always is. > > > Unless I overlook something (please tell me what), the > > scheme just proposed requires filesystems to look at > > the content of files that is being written out, in > > order to make the streams work. > > Yes. Hence the idea of coherent views between two files: writing to > one affects the content of the other, although the calcalation is only > done on demand (or when the fs wants to migrate the representation -- > for example, creating the flat container prior to deleting the > regeneratable pieces in order to save space). > > I haven't seen anything from Namesys that says they'll do that. I > have the impression the streams are just generated in memory on the > fly, not stored on disk with a cacheing policy, but that's just an > impression. (We've all seen the Namesys white papers, they're not > _that_ revealing). :) would it be a tautology if I say that the streams implementation depends on (reiser4 now, may be vfs later) plugins? All proposals including user-level helpers, shadow caching/indexing may be implemented, why not? > I'm just pointing out how to do it _right_. I think it will turn out > like this eventually, either next year or some time over the next > decade after some iterations. Inevitable. Mark my words, etc. :) > > -- Jamie -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 17:37 ` Alex Zarochentsev @ 2004-08-30 18:51 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-30 18:51 UTC (permalink / raw) To: Alex Zarochentsev Cc: Jamie Lokier, Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Alex Zarochentsev wrote: >On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote: > > >>Rik van Riel wrote: >> >> >>>And if an unaware application reads the compound file >>>and then writes it out again, does the filesystem >>>interpret the contents and create the other streams ? >>> >>> >>Yes, exactly that. The streams are created on demand of course, and >>by userspace helpers when that's appropriate which I suspect it almost >>always is. >> >> >> >>>Unless I overlook something (please tell me what), the >>>scheme just proposed requires filesystems to look at >>>the content of files that is being written out, in >>>order to make the streams work. >>> >>> >>Yes. Hence the idea of coherent views between two files: writing to >>one affects the content of the other, although the calcalation is only >>done on demand (or when the fs wants to migrate the representation -- >>for example, creating the flat container prior to deleting the >>regeneratable pieces in order to save space). >> >>I haven't seen anything from Namesys that says they'll do that. >> We do that, err, it is our design to do that. Hmm, the plugins to do that seem to all be in the next to implement list and not on the implemented already list..... However, with respect to streams, those are just files in a file-directory object in our scheme. If you want them unified, well, then you cat compound-document/pseudos/glued, or you link compound-document/pseudos/glued to compound-document/pseudos/body, where body is our term for the default stream, and then you just cat compound-document If you have the impression I am being vaporwarish, well, you are right.... glued is not yet on the already implemented list. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-30 18:51 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-30 18:51 UTC (permalink / raw) To: Alex Zarochentsev Cc: Jamie Lokier, Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Alex Zarochentsev wrote: >On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote: > > >>Rik van Riel wrote: >> >> >>>And if an unaware application reads the compound file >>>and then writes it out again, does the filesystem >>>interpret the contents and create the other streams ? >>> >>> >>Yes, exactly that. The streams are created on demand of course, and >>by userspace helpers when that's appropriate which I suspect it almost >>always is. >> >> >> >>>Unless I overlook something (please tell me what), the >>>scheme just proposed requires filesystems to look at >>>the content of files that is being written out, in >>>order to make the streams work. >>> >>> >>Yes. Hence the idea of coherent views between two files: writing to >>one affects the content of the other, although the calcalation is only >>done on demand (or when the fs wants to migrate the representation -- >>for example, creating the flat container prior to deleting the >>regeneratable pieces in order to save space). >> >>I haven't seen anything from Namesys that says they'll do that. >> We do that, err, it is our design to do that. Hmm, the plugins to do that seem to all be in the next to implement list and not on the implemented already list..... However, with respect to streams, those are just files in a file-directory object in our scheme. If you want them unified, well, then you cat compound-document/pseudos/glued, or you link compound-document/pseudos/glued to compound-document/pseudos/body, where body is our term for the default stream, and then you just cat compound-document If you have the impression I am being vaporwarish, well, you are right.... glued is not yet on the already implemented list. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:54 ` Rik van Riel ` (2 preceding siblings ...) (?) @ 2004-08-26 17:05 ` Diego Calleja 2004-08-26 17:16 ` Rik van Riel -1 siblings, 1 reply; 1835+ messages in thread From: Diego Calleja @ 2004-08-26 17:05 UTC (permalink / raw) To: Rik van Riel Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list El Thu, 26 Aug 2004 11:54:51 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió: > And if an unaware application reads the compound file > and then writes it out again, does the filesystem > interpret the contents and create the other streams ? If the old utils doesn't support streams there's no way to make them create streams. There's not a real solution for that except fixing the old tools. Why instead of being "compound" you just make test.compound to behave as a real directory? ie: # echo "foo" > streamfile # echo "fake metadata" > streamfile/somemetadata # cat streamfile foo # cat streamfile/somemetadata fake metadata Now the old backup utils could see "streamfile" as a directory, ie: old apps could see that "streamfile" is a directory and then make their backups with the following contents: streamfile/somemetadata streamfile/defaultstream The defaultstream would be what its name means. There could be more default streams like "streamfile/compoundstream" All this looks like reinventing the file/directory concept wheel. Instead of adding support for streams and "use files as directories", why not orientate it to "use directories as files? Streams could very well be provided by directories containing files, with the addon of being able to cat a directory (which would cat the output of a designed file inside every directory like "defaultstream", then there could be other "special files" like "compoundstream", although the idea of "fixed name- special files" doesn't sounds very cool) All this could be very well provided done userspace, but then couldn't userspace also supply the whole VFS functionality and still it lives in the kernel...? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:05 ` Diego Calleja @ 2004-08-26 17:16 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 17:16 UTC (permalink / raw) To: Diego Calleja Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Diego Calleja wrote: > All this looks like reinventing the file/directory concept wheel. > Instead of adding support for streams and "use files as directories", > why not orientate it to "use directories as files? Streams could very > well be provided by directories containing files, So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? This is getting surreal. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 17:16 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 17:16 UTC (permalink / raw) To: Diego Calleja Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Diego Calleja wrote: > All this looks like reinventing the file/directory concept wheel. > Instead of adding support for streams and "use files as directories", > why not orientate it to "use directories as files? Streams could very > well be provided by directories containing files, So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? This is getting surreal. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:16 ` Rik van Riel (?) @ 2004-08-26 17:20 ` Linus Torvalds 2004-08-26 17:27 ` Jamie Lokier ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 17:20 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > On Thu, 26 Aug 2004, Diego Calleja wrote: > > > All this looks like reinventing the file/directory concept wheel. > > Instead of adding support for streams and "use files as directories", > > why not orientate it to "use directories as files? Streams could very > > well be provided by directories containing files, > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:20 ` Linus Torvalds @ 2004-08-26 17:27 ` Jamie Lokier 2004-08-26 17:32 ` Linus Torvalds 2004-08-26 17:57 ` Rik van Riel 2004-08-27 8:08 ` Hans Reiser 2 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 17:27 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > No no. The stream you get from a directory is totally _independent_ of the > contents of the directory. Anything else would be pointless. Surely it depends on the directory? The stream from a "classic directory", i.e. something made with mkdir(), mode drwxr-xr-x filled with files, might well be a reproducible representation of its contents. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:27 ` Jamie Lokier @ 2004-08-26 17:32 ` Linus Torvalds 2004-08-26 19:09 ` Jamie Lokier 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 17:32 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > Linus Torvalds wrote: > > > > No no. The stream you get from a directory is totally _independent_ of the > > contents of the directory. Anything else would be pointless. > > Surely it depends on the directory? Well, you _could_ make it do a "tar", but what's the point? That really doesn't add any value as far as user space is concerned. Think of it this way: if the directory and the "default stream" associated with it aren't independent, then user space already has the information, and the stream is pointless. In fact, the stream is _worse_ than pointless, because it hides the fact that there is no independent information. See my point? Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:32 ` Linus Torvalds @ 2004-08-26 19:09 ` Jamie Lokier 0 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 19:09 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > Well, you _could_ make it do a "tar", but what's the point? That really > doesn't add any value as far as user space is concerned. > > Think of it this way: if the directory and the "default stream" associated > with it aren't independent, then user space already has the information, > and the stream is pointless. In fact, the stream is _worse_ than > pointless, because it hides the fact that there is no independent > information. > > See my point? I see your point, but I agree 50%. (1) One of the uses of "streams" is to have extra information which accompanies a file, and is separate from the flat file's information. For example, metadata like "this C source file is in UTF-8", "the author is Jamie (nowhere to put it in the data file)" and "Jamie's signature". (2) Filesystem settings like "compress this", and perhaps "permissions". (3) Filesystem information like "this file has been compressed by 59%, uses 5 1k blocks". (4) Another use, which is more akin to containers, is to _present_ the information of the flat file in a different way. For example, the id tags and comments of any media file, the component parts of any media file, license (of course :), and expanded or decoded views such as inside an archive file, filesystem image, or other structured storage file (.doc :). Those alternate views are useful just because they can be accessed using ordinary unix tools, no matter what the obscure codec format provided it's supported. With a bit of effort a standard presentation for certain kinds of data may arise. (5) There is also data which is _calculated_ from the flat file or its extra metadata, and which the kernel guarantees is either consistent with the file, not set at all, or at least is invalidated if the file is changed. It may be calculated on demand through the kernel interface, or applications may simply have to calculate it themselves when they don't have an up to date value. For example MD5 or SHA1 digests of the flat file, or of its parts. The key thing is that these are always up to date, not dependent on weak tests like mtime. Partial checksums for speeding up rsync and other delta algorithms. Computed thumbnails, MP3 average power levels, character encoding extracted from parsing old HTML META tags. Anything really that can be calculated and usefully cached. This is similar to (4), but I'm emphasising things that may take a while to compute, may be cached with the filesystem itself helping to decide when to evict, and writing them won't affect the data they're calculated from, if writing is allowed at all. The only really important of these are MD5 or SHA1 digests, or some other way of tracking modifications like a strong serial number. All other computed cached data can be maintained by applications elsewhere, if they know when the data has been modified since the last time. (Though there are advantages to making the data disappear if the file is deleted, or move if it's renamed). The thing with these different categories is that we want different behaviour when serialising the data for a backup or to copy it. We probably always (1) to be copied. (2) is questionable: it's like unix file permissions, you might want to copy it; you might not. (3) Shouldn't be copied or stored except for special reasons. It's a bit like /proc: it's not meant to be stored on your backup CD and it's definitely not meant to be written during a restore. :) (4) Should never be copied if you're copying the flat file, because it's exactly the same information, just accessed differently. You might want to copy it by itself for some reason though, and you certainly would like to work with it from scripts and applications. These streams are special in another way: they might themselves decompose into further streams. Just think of a AVI file containing an OGG file containing a FLAC containing metadata and data streams, or something like that. They happen. (5) shouldn't be copied if you're transmitting a file, as a matter of data integrity among other things. You might want it on the backup, then again maybe only some of the values. Probably never the crypto digests. To summarise: most data inside a directory-as-file should _not_ usually be copied and transported. A little of it, (1), should be copied if the serialised form will have it, and some of the other metadata should be copied under special circumstances. I've suggested that we could simply not have (1), and recommend that only reproducible, non-essential data appears inside the directory. Everything essential would be part of the serialised, flat file. However, you seem to like (1), metadata which should be serialised and copied around, and which is not part of the flat file. Note that (1) will only be copied around on systems which handle "extra forks" anyway -- things like Samba and MacOS compatibility. FTP and HTTP obviously won't copy the metadata of (1), although HTTP WebDAV might allow it to be visible as properties. All that suggests a common structure for the directory inside a "container" -- one where the metadata to be copied by default is clearly distinguished from the metadata which should never by copied, or which is cached, or which is a view. Possibly it should be distinguished by name, although permission bits or (not sure I like this) meta-metadata could serve the same role without being name dependent. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:20 ` Linus Torvalds @ 2004-08-26 17:57 ` Rik van Riel 2004-08-26 17:57 ` Rik van Riel 2004-08-27 8:08 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 17:57 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > No no. The stream you get from a directory is totally _independent_ of the > contents of the directory. Anything else would be pointless. It's a relief to know that nobody's taking my humorous suggestion seriously, but now we still have the "standard Unix tools can't manipulate files" problem... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 17:57 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 17:57 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > No no. The stream you get from a directory is totally _independent_ of the > contents of the directory. Anything else would be pointless. It's a relief to know that nobody's taking my humorous suggestion seriously, but now we still have the "standard Unix tools can't manipulate files" problem... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:57 ` Rik van Riel (?) @ 2004-08-26 18:15 ` Linus Torvalds 2004-08-26 18:41 ` Rik van Riel ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 18:15 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > It's a relief to know that nobody's taking my humorous > suggestion seriously, but now we still have the "standard > Unix tools can't manipulate files" problem... I disagree. They can manipulate the files a whole lot better than they can manipulate xattr's. For example, you _could_ probably (but hey, maybe "tar" tries to strip slashes off the end of filenames, so this might not work due to silly reasons like that) back up a compound file with tar cvf file.tar file file/ although unpacking it would require that tar be taught about the thing. And you definitely could write a script to do the thing, ie even with an unmodified tar you could do tar cvf file-archive.tar file cd file tar cvf ../attribute-archive.tar . which is a hell of a lot better than what you can do with the fsattr interfaces and unmodified legacy applications. So one of the advantages of "dir-as-file/file-as-dir" is exactly that you _can_ manipulate the data with legacy tools. Sure, things that traverse a directory tree might need some (likely fairly trivial) modifications if they really want to take advantage of the subfiles, but that's still likely to be _much_ less of an issue than with fsattr's that have a totally different model entirely. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:15 ` Linus Torvalds @ 2004-08-26 18:41 ` Rik van Riel 2004-08-26 19:02 ` David Masover 2004-08-26 20:44 ` Martin J. Bligh 2 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 18:41 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > For example, you _could_ probably (but hey, maybe "tar" tries to strip > slashes off the end of filenames, so this might not work due to silly > reasons like that) back up a compound file with > > tar cvf file.tar file file/ So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 18:41 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 18:41 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > For example, you _could_ probably (but hey, maybe "tar" tries to strip > slashes off the end of filenames, so this might not work due to silly > reasons like that) back up a compound file with > > tar cvf file.tar file file/ So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:41 ` Rik van Riel (?) @ 2004-08-26 18:55 ` Linus Torvalds 2004-08-26 18:59 ` Rik van Riel 2004-08-27 8:46 ` Herbert Poetzl -1 siblings, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 18:55 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > So you'd have both a file and a directory that just happen > to have the same name ? How would this work in the dcache? There would be only one entry in the dcache. The lookup will select whether it opens the file or the directory based on O_DIRECTORY (and usage, of course - if it's in the middle of a path, it obviously needs to be opened as a directory regardless). That's not the problem. The problem from a dcache standpoint ends up being when the file has a link, and you have two paths to the same sub-file through two different ways: .. create file 'x' with named stream 'y' ... ln x z ls -l x/y z/y /* it's the same attribute!! */ but this is actually exactly the same thing that we already have with mounts, ie it is equivalent (from a dentry standpoint) to .. create directory 'x' with file 'y' .. mkdir z mount --bind x z ls -l x/y z/y /* It's the same file!! */ so none of this is really anything "new" from a dcache standpoint. Except for all the details, of course ;) Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:55 ` Linus Torvalds @ 2004-08-26 18:59 ` Rik van Riel 2004-08-27 8:46 ` Herbert Poetzl 1 sibling, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 18:59 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > On Thu, 26 Aug 2004, Rik van Riel wrote: > > > > So you'd have both a file and a directory that just happen > > to have the same name ? How would this work in the dcache? > > There would be only one entry in the dcache. The lookup will select > whether it opens the file or the directory based on O_DIRECTORY (and > usage, of course - if it's in the middle of a path, it obviously needs to > be opened as a directory regardless). Hmmm, I just straced "cp /bin/bash /tmp". One line stood out as a potential problem: open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4 What do we do with O_CREAT ? Do we always allow both a directory and a file to be created with the same name ? Does this create a new class of "symlink attack" style security holes ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 18:59 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 18:59 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > On Thu, 26 Aug 2004, Rik van Riel wrote: > > > > So you'd have both a file and a directory that just happen > > to have the same name ? How would this work in the dcache? > > There would be only one entry in the dcache. The lookup will select > whether it opens the file or the directory based on O_DIRECTORY (and > usage, of course - if it's in the middle of a path, it obviously needs to > be opened as a directory regardless). Hmmm, I just straced "cp /bin/bash /tmp". One line stood out as a potential problem: open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4 What do we do with O_CREAT ? Do we always allow both a directory and a file to be created with the same name ? Does this create a new class of "symlink attack" style security holes ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:59 ` Rik van Riel @ 2004-08-26 19:14 ` Christophe Saout -1 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 19:14 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 673 bytes --] Am Donnerstag, den 26.08.2004, 14:59 -0400 schrieb Rik van Riel: > open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4 > > What do we do with O_CREAT ? > > Do we always allow both a directory and a file to be created with > the same name ? I would say that the directory under a file is implicit. There are still the pseudo-files a filesystem might want to provide or the VFS if there should be an agreement to generally provide file/pseudo/uid or possibly some security hooks. Some filesystem might want to provide those pseudo files but doesn't have the technical possibility to store non-pseudeo files from the user under such a directory. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 19:14 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 19:14 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 673 bytes --] Am Donnerstag, den 26.08.2004, 14:59 -0400 schrieb Rik van Riel: > open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4 > > What do we do with O_CREAT ? > > Do we always allow both a directory and a file to be created with > the same name ? I would say that the directory under a file is implicit. There are still the pseudo-files a filesystem might want to provide or the VFS if there should be an agreement to generally provide file/pseudo/uid or possibly some security hooks. Some filesystem might want to provide those pseudo files but doesn't have the technical possibility to store non-pseudeo files from the user under such a directory. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:59 ` Rik van Riel (?) (?) @ 2004-08-26 19:23 ` Linus Torvalds 2004-08-26 19:44 ` Rik van Riel ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 19:23 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > Hmmm, I just straced "cp /bin/bash /tmp". > One line stood out as a potential problem: > > open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4 > > What do we do with O_CREAT ? > > Do we always allow both a directory and a file to be created with > the same name ? Either I am confused, or you are. To me, a filesystem that allows this thing doesn't really _have_ the concept of "directory vs file". It's just a "filesystem object", and it can act as _both_ a directory and a file. So when you create "/tmp/bash" - assuming /tmp supports the file-as-dir semantics at all, you're just creating a new inode - the same you always have. When you write to that inode, it writes to the default stream. There's no special cases here. Now, after you have your regular /tmp/bash, you can then start adding named streams to it, ie you can do open("/tmp/bash/icon", O_WRONLY|O_CREAT|O_LARGEFILE, 0755); and that will create the "icon" named stream. See? So "/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part). Hey, think of it as a wave-particle duality. Both "modes" exist at the same time, and cannot be separated from each other. Which one you see depends entirely on your "experiment", ie how you open the file. > Does this create a new class of "symlink attack" style security > holes ? I don't believe so. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:23 ` Linus Torvalds @ 2004-08-26 19:44 ` Rik van Riel 2004-08-26 20:10 ` Rik van Riel 2004-08-27 8:06 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 19:44 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > contains both a standard data stream (the "file" part) _and_ pointers to > other named streams (the "directory" part). OK, that makes sense. > Hey, think of it as a wave-particle duality. Both "modes" exist at the > same time, and cannot be separated from each other. Which one you see > depends entirely on your "experiment", ie how you open the file. Guess I'm scared again now. We need to make sure that backup programs don't fall victim to the uncertainty principle ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 19:44 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 19:44 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > contains both a standard data stream (the "file" part) _and_ pointers to > other named streams (the "directory" part). OK, that makes sense. > Hey, think of it as a wave-particle duality. Both "modes" exist at the > same time, and cannot be separated from each other. Which one you see > depends entirely on your "experiment", ie how you open the file. Guess I'm scared again now. We need to make sure that backup programs don't fall victim to the uncertainty principle ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:44 ` Rik van Riel (?) @ 2004-08-26 19:47 ` Christophe Saout -1 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 19:47 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 526 bytes --] Am Donnerstag, den 26.08.2004, 15:44 -0400 schrieb Rik van Riel: > > Hey, think of it as a wave-particle duality. Both "modes" exist at the > > same time, and cannot be separated from each other. Which one you see > > depends entirely on your "experiment", ie how you open the file. > > Guess I'm scared again now. We need to make sure that > backup programs don't fall victim to the uncertainty > principle ;) At least you can be sure that the backup will go into a defined state as soon as you observe it. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:23 ` Linus Torvalds @ 2004-08-26 20:10 ` Rik van Riel 2004-08-26 20:10 ` Rik van Riel 2004-08-27 8:06 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 20:10 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > contains both a standard data stream (the "file" part) _and_ pointers to > other named streams (the "directory" part). Thinking about it some more, how would file managers and file chosers handle this situation ? Currently the user browses the directory tree and when the user clicks on something, one of the following happens: 1) if it is a directory, the file manager/choser changes into that directory 2) if it is a file, the file is opened Now how do we present things to users ? How will users know when an object can only be chdired into, or only be opened ? For objects that do both, how does the user choose ? Do we really want to have a file paradigm that's different from the other OSes out there ? What happens when users want to transfer data from Linux to another system ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 20:10 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 20:10 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > contains both a standard data stream (the "file" part) _and_ pointers to > other named streams (the "directory" part). Thinking about it some more, how would file managers and file chosers handle this situation ? Currently the user browses the directory tree and when the user clicks on something, one of the following happens: 1) if it is a directory, the file manager/choser changes into that directory 2) if it is a file, the file is opened Now how do we present things to users ? How will users know when an object can only be chdired into, or only be opened ? For objects that do both, how does the user choose ? Do we really want to have a file paradigm that's different from the other OSes out there ? What happens when users want to transfer data from Linux to another system ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel (?) @ 2004-08-26 20:17 ` Christophe Saout 2004-08-26 20:21 ` Jamie Lokier -1 siblings, 1 reply; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 20:17 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1317 bytes --] Am Donnerstag, den 26.08.2004, 16:10 -0400 schrieb Rik van Riel: > > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > > contains both a standard data stream (the "file" part) _and_ pointers to > > other named streams (the "directory" part). > > Thinking about it some more, how would file managers and > file chosers handle this situation ? I would say they don't. That's where metadata and pseudo-files sit. It's up to the applications to do something useful with the data. > Do we really want to have a file paradigm that's different > from the other OSes out there ? Some other OSes have similar things. > What happens when users want to transfer data from Linux > to another system ? No one should store really important data inside file/ that makes the file completely useless. I just love these postscript fonts on the Mac. The actual main file stream is 0 bytes in size. The actual font is inside the resource fork. When I want to convert it the Mac user needs to pack it into a compound file using a special tool and then I can decode it using another tool to access the PFB file. Crap. It's useful for metadata like keywords, ACLs and these things. If you can pack it using an aware backup program, fine. If you don't, you still have the data. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:17 ` Christophe Saout @ 2004-08-26 20:21 ` Jamie Lokier 0 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 20:21 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Linus Torvalds, Diego Calleja, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: > I just love these postscript fonts on the Mac. The actual main file > stream is 0 bytes in size. The actual font is inside the resource > fork. When I want to convert it the Mac user needs to pack it into a > compound file using a special tool and then I can decode it using > another tool to access the PFB file. Crap. I thought it was a cunning way to stop me from copying those nice fonts onto my Linux box. The original DRM. Thanks for the tip :) -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel (?) (?) @ 2004-08-26 20:35 ` Stephen Wille Padnos 2004-08-31 11:01 ` Claus Färber -1 siblings, 1 reply; 1835+ messages in thread From: Stephen Wille Padnos @ 2004-08-26 20:35 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: >On Thu, 26 Aug 2004, Linus Torvalds wrote: > > >>So "/tmp/bash" is _not_ two different things. It is _one_ entity, that >>contains both a standard data stream (the "file" part) _and_ pointers to >>other named streams (the "directory" part). >> >> >Thinking about it some more, how would file managers and >file chosers handle this situation ? > >Currently the user browses the directory tree and when >the user clicks on something, one of the following >happens: > >1) if it is a directory, the file manager/choser changes > into that directory > > How does the file manager / chooser decide whether you're trying to move into a directory, or the meta-data-directory for a directory? It's not just files that should have metadata - directories need* them too. Making it possible to see attributes as a directory under a file is great, but you'd still need an O_META flag for accessing directory metadata (since there are already files under a directory). >2) if it is a file, the file is opened > >Now how do we present things to users ? > >How will users know when an object can only be chdired >into, or only be opened ? > >For objects that do both, how does the user choose ? > >Do we really want to have a file paradigm that's different >from the other OSes out there ? > > MacOS does, Be did (sort of). I'm not sure it would be the end of the world, as long as the data can be extracted. >What happens when users want to transfer data from Linux >to another system ? > > That would depend on the other system. Data is easy, metadata is hard. It would be possible to create an XML schema for metadata, and if requested (O_EVERYTHING?), the file data is returned with all metadata in XML tags. (not advocating this, just an idea :) - Steve * I say need in the same way as one *needs* to upgrade their 2GHz computer - it would be nice :) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:35 ` Stephen Wille Padnos 2004-08-31 11:01 ` Claus Färber @ 2004-08-31 11:01 ` Claus Färber 0 siblings, 0 replies; 1835+ messages in thread From: Claus Färber @ 2004-08-31 11:01 UTC (permalink / raw) To: linux-kernel; +Cc: reiserfs-list, linux-fsdevel Stephen Wille Padnos <spadnos@sover.net> schrieb/wrote: > How does the file manager / chooser decide whether you're trying to > move into a directory, or the meta-data-directory for a directory? > It's not just files that should have metadata - directories need* them > too. Making it possible to see attributes as a directory under a file > is great, but you'd still need an O_META flag for accessing directory > metadata (since there are already files under a directory). A simple convention that meta data files start with, say ".$", would be enough. Claus -- http://www.faerber.muc.de ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-31 11:01 ` Claus Färber 0 siblings, 0 replies; 1835+ messages in thread From: Claus Färber @ 2004-08-31 11:01 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel Stephen Wille Padnos <spadnos@sover.net> schrieb/wrote: > How does the file manager / chooser decide whether you're trying to > move into a directory, or the meta-data-directory for a directory? > It's not just files that should have metadata - directories need* them > too. Making it possible to see attributes as a directory under a file > is great, but you'd still need an O_META flag for accessing directory > metadata (since there are already files under a directory). A simple convention that meta data files start with, say ".$", would be enough. Claus -- http://www.faerber.muc.de ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-31 11:01 ` Claus Färber 0 siblings, 0 replies; 1835+ messages in thread From: Claus Färber @ 2004-08-31 11:01 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-kernel, linux-fsdevel Stephen Wille Padnos <spadnos@sover.net> schrieb/wrote: > How does the file manager / chooser decide whether you're trying to > move into a directory, or the meta-data-directory for a directory? > It's not just files that should have metadata - directories need* them > too. Making it possible to see attributes as a directory under a file > is great, but you'd still need an O_META flag for accessing directory > metadata (since there are already files under a directory). A simple convention that meta data files start with, say ".$", would be enough. Claus -- http://www.faerber.muc.de ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 11:01 ` Claus Färber @ 2004-09-01 15:13 ` Alan Cox -1 siblings, 0 replies; 1835+ messages in thread From: Alan Cox @ 2004-09-01 15:13 UTC (permalink / raw) To: Claus Färber; +Cc: Linux Kernel Mailing List, reiserfs-list, linux-fsdevel On Maw, 2004-08-31 at 12:01, Claus Färber wrote: > A simple convention that meta data files start with, say ".$", would be > enough. POSIX/SuS don't permit this. The only "free" namespace is that starting "//" (and not as some desktops seem to think foo://). Remember always send GUI desktop users files called http://read.me .. its fun 8) The // doesn't really help because you don't want extensions at the path top. Another interesting question btw with substreams of a file is what the semantics of fchown, fstat, fchmod, fchdir etc are, or of mounting a file over a substream. Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-01 15:13 ` Alan Cox 0 siblings, 0 replies; 1835+ messages in thread From: Alan Cox @ 2004-09-01 15:13 UTC (permalink / raw) To: Claus Färber; +Cc: Linux Kernel Mailing List, reiserfs-list, linux-fsdevel On Maw, 2004-08-31 at 12:01, Claus Färber wrote: > A simple convention that meta data files start with, say ".$", would be > enough. POSIX/SuS don't permit this. The only "free" namespace is that starting "//" (and not as some desktops seem to think foo://). Remember always send GUI desktop users files called http://read.me .. its fun 8) The // doesn't really help because you don't want extensions at the path top. Another interesting question btw with substreams of a file is what the semantics of fchown, fstat, fchmod, fchdir etc are, or of mounting a file over a substream. Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel ` (2 preceding siblings ...) (?) @ 2004-08-26 20:40 ` Linus Torvalds 2004-08-26 22:53 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 20:40 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > Thinking about it some more, how would file managers and file chosers > handle this situation ? The same way they already handle it on other platforms that support it? By taking advantage of it.. People literally use this for icon and preview information, so some of the stuff shows up very much in file managers. And I'd assume that a normal double-click would just open the main file, and you can right-click for management information, including opening the file. If you want an entity that acts as a directory, you create a directory. Directories don't go away - you still have S_ISDIR() and S_ISREG(), and they still return information. A file that has associated information is still a _file_, and people should treat it that way, it's just that it also has a list of named sub-streams. You can open it as a directory, but the stat information clearly says it is a file, and the "directory" view is very much associated with _that_ file. I definitely don't think you want the file manager to act as if a named stream is exactly the same as a stand-alone file. They have all the same operations, but there's no question that there are differences too. Especially on a conceptual level - but for most filesystems there are likely real technical differences too. For example, it's likely that most filesystems would _not_ allow linking of a named stream anywhere else. And you might not be able to change the permissions or date on the named stream either, since it may or may not have a separate date/permission thing from the container. So don't believe that just because the named streams are _named_ like real files, that they suddenly have any existence beyond the container that they are part of. There may be other limitations too - again depending on how the filesystem actually implements named streams. It might not support more than one level of naming, for example - so you might not be able to create a directory structure within the named streams, for example. In short: the fact that the VFS layer exposes the _capability_ to see the named streams as a full POSIX filesystem of its own does _not_ mean that the low-level filesystem necessarily allows the full possible semantics. So you shouldn't design your apps that know about named streams to think they are normal directories. The directory thing is just a very powerful naming scheme, and one that fits into the existing UNIX model. > Do we really want to have a file paradigm that's different > from the other OSes out there ? Different from what other OSes? Last I looked, Windows, Solaris, and OSX all supported named streams. What other OS's are out there that you care about? In other words, this is _not_ a different paradigm from what others do. The discussion is whether we want to implement it at all, and more importantly about syntactic issues (ie we clearly already implement extended attributes, just with a much more limited syntactic power). We don't have to go all the way. Solaris has "openat()", which is kind of a half-way there - not really directly available in the same namespace, but at least the result is available as a real file interface (as opposed to the Linux "xattr" interfaces that are _totally_ special-cased system calls). In other words - the paradigm is already there. It's just that currently it's pretty much unusable under Linux, because it requires so much specialized knowledge that it's not worth it to modify existing apps. And the interfaces are really very limited, so even if you _do_ end up using the specialized Linux interfaces, you can't actually do a lot of what you might want to do. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:40 ` Linus Torvalds @ 2004-08-26 22:53 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 22:53 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > >For example, it's likely that most filesystems would _not_ allow linking >of a named stream anywhere else. > Why? It is just a file that inherits its stat data and is referenced by a concatenation method of its parent directory. The link will also inherit its stat data from the original parent directory. > And you might not be able to change the >permissions or date on the named stream either, since it may or may not >have a separate date/permission thing from the container. > > You should be able to change the permission and data, but when you change it, you change it for the whole container. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 22:53 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 22:53 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > >For example, it's likely that most filesystems would _not_ allow linking >of a named stream anywhere else. > Why? It is just a file that inherits its stat data and is referenced by a concatenation method of its parent directory. The link will also inherit its stat data from the original parent directory. > And you might not be able to change the >permissions or date on the named stream either, since it may or may not >have a separate date/permission thing from the container. > > You should be able to change the permission and data, but when you change it, you change it for the whole container. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:53 ` Hans Reiser (?) @ 2004-08-26 23:27 ` Linus Torvalds 2004-08-26 23:47 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 23:27 UTC (permalink / raw) To: Hans Reiser Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Hans Reiser wrote: > > Linus Torvalds wrote: > > > >For example, it's likely that most filesystems would _not_ allow linking > >of a named stream anywhere else. > > Why? It is just a file that inherits its stat data and is referenced by > a concatenation method of its parent directory. The link will also > inherit its stat data from the original parent directory. In reiser4, maybe. Not in general. Look at what other filesystems do for attributes.. > > And you might not be able to change the > >permissions or date on the named stream either, since it may or may not > >have a separate date/permission thing from the container. > > You should be able to change the permission and data, but when you > change it, you change it for the whole container. Why? I'd much rather disallow it, than allow it and then have "nonlocal" effects. I can also see a filesystem validly having _extra_ permissions (ie apart from the container) on it too. Not all the world is reiser4. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:27 ` Linus Torvalds @ 2004-08-26 23:47 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:47 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > > >>>And you might not be able to change the >>>permissions or date on the named stream either, since it may or may not >>>have a separate date/permission thing from the container. >>> >>> >>You should be able to change the permission and data, >> ^data^date >> but when you >>change it, you change it for the whole container. >> >> > >Why? I'd much rather disallow it, than allow it and then have "nonlocal" >effects. > > Sometimes you want the nonlocal effects and sometimes you don't, and by decomposing streams into smaller primitives we can let users choose as they want. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:47 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:47 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > > >>>And you might not be able to change the >>>permissions or date on the named stream either, since it may or may not >>>have a separate date/permission thing from the container. >>> >>> >>You should be able to change the permission and data, >> ^data^date >> but when you >>change it, you change it for the whole container. >> >> > >Why? I'd much rather disallow it, than allow it and then have "nonlocal" >effects. > > Sometimes you want the nonlocal effects and sometimes you don't, and by decomposing streams into smaller primitives we can let users choose as they want. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:47 ` Hans Reiser (?) @ 2004-08-27 2:03 ` viro 2004-08-27 8:10 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-27 2:03 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 04:47:39PM -0700, Hans Reiser wrote: > Sometimes you want the nonlocal effects and sometimes you don't, and by > decomposing streams into smaller primitives we can let users choose as > they want. Right. Now, would you kindly post the detailed technical analysis of your wonderful design that handles that stuff safely? With proof of correctness, please. Since we are expected to take your code and use it as "Uber" replacement for the existing one, it surely would not be too much to expect, would it? And no, "we have a nice presentation somewhere on namesys" does not qualify. I apologize for the obvious posting in a thread already full of noise, but I would like to make *very* sure that lack of ripping you another one does *not* mutate into "no objections from viro" in a lovely thread on SlashDot in a week or so. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 2:03 ` viro @ 2004-08-27 8:10 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:10 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list viro@parcelfarce.linux.theplanet.co.uk wrote: >On Thu, Aug 26, 2004 at 04:47:39PM -0700, Hans Reiser wrote: > > > >>Sometimes you want the nonlocal effects and sometimes you don't, and by >>decomposing streams into smaller primitives we can let users choose as >>they want. >> >> > >Right. Now, would you kindly post the detailed technical analysis of your >wonderful design that handles that stuff safely? With proof of correctness, >please. Since we are expected to take your code and use it as "Uber" >replacement for the existing one, it surely would not be too much to expect, >would it? > >And no, "we have a nice presentation somewhere on namesys" does not >qualify. > >I apologize for the obvious posting in a thread already full of noise, >but I would like to make *very* sure that lack of ripping you another >one does *not* mutate into "no objections from viro" in a lovely thread >on SlashDot in a week or so. > > > > Over the next 6-18 months, one piece at a time, it will all fall into place. Reiser4 as it is shipping today, consists of close to the minimal functionality necessary to replace reiser3, albeit done according to our plugin model, and with vastly higher performance. The full solution is not yet written, wait for it. I understand why you think deadlocks are everything, but avoiding them is only a small, albeit necessary, piece of the task. Thanks though for pointing out issues with unlinking files that have been cd'd to. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:10 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:10 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list viro@parcelfarce.linux.theplanet.co.uk wrote: >On Thu, Aug 26, 2004 at 04:47:39PM -0700, Hans Reiser wrote: > > > >>Sometimes you want the nonlocal effects and sometimes you don't, and by >>decomposing streams into smaller primitives we can let users choose as >>they want. >> >> > >Right. Now, would you kindly post the detailed technical analysis of your >wonderful design that handles that stuff safely? With proof of correctness, >please. Since we are expected to take your code and use it as "Uber" >replacement for the existing one, it surely would not be too much to expect, >would it? > >And no, "we have a nice presentation somewhere on namesys" does not >qualify. > >I apologize for the obvious posting in a thread already full of noise, >but I would like to make *very* sure that lack of ripping you another >one does *not* mutate into "no objections from viro" in a lovely thread >on SlashDot in a week or so. > > > > Over the next 6-18 months, one piece at a time, it will all fall into place. Reiser4 as it is shipping today, consists of close to the minimal functionality necessary to replace reiser3, albeit done according to our plugin model, and with vastly higher performance. The full solution is not yet written, wait for it. I understand why you think deadlocks are everything, but avoiding them is only a small, albeit necessary, piece of the task. Thanks though for pointing out issues with unlinking files that have been cd'd to. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel ` (3 preceding siblings ...) (?) @ 2004-08-26 20:48 ` Jamie Lokier 2004-08-26 20:52 ` Jeremy Allison -1 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 20:48 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > For objects that do both, how does the user choose ? > > Do we really want to have a file paradigm that's different > from the other OSes out there ? What does MacOS X do? Someone said that documents are directories in it; it must know how to handle that. What does Windows do when you click on a .zip file and WinZIP is installed? It opens the .zip file and lets you explore inside. When you click on a .doc file, though, it opens a viewer or editor -- you don't get the option to look inside. > What happens when users want to transfer data from Linux > to another system ? This is why I favour storing all essential metadata (about the file's content) inside the file's contents, the primary stream. We have another problem: what happens when users want to transfer data from Windows (with secondary streams) and MacOS (with resource forks)? -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:48 ` Jamie Lokier @ 2004-08-26 20:52 ` Jeremy Allison 2004-08-27 9:19 ` Christoph Hellwig 0 siblings, 1 reply; 1835+ messages in thread From: Jeremy Allison @ 2004-08-26 20:52 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 09:48:41PM +0100, Jamie Lokier wrote: > > This is why I favour storing all essential metadata (about the file's > content) inside the file's contents, the primary stream. > > We have another problem: what happens when users want to transfer data > from Windows (with secondary streams) and MacOS (with resource forks)? I can tell you that - right now we (Samba) has to drop all but the primary stream, because the Linux filesystems don't have data stream support. Oh look, we're losing user data (that's a *bad* thing :-). Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:52 ` Jeremy Allison @ 2004-08-27 9:19 ` Christoph Hellwig 2004-08-28 3:56 ` Jeremy Allison 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-27 9:19 UTC (permalink / raw) To: Jeremy Allison Cc: Jamie Lokier, Rik van Riel, Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 01:52:18PM -0700, Jeremy Allison wrote: > On Thu, Aug 26, 2004 at 09:48:41PM +0100, Jamie Lokier wrote: > > > > This is why I favour storing all essential metadata (about the file's > > content) inside the file's contents, the primary stream. > > > > We have another problem: what happens when users want to transfer data > > from Windows (with secondary streams) and MacOS (with resource forks)? > > I can tell you that - right now we (Samba) has to drop all but the primary stream, > because the Linux filesystems don't have data stream support. Oh look, > we're losing user data (that's a *bad* thing :-). Maybe you should learn from netatalk. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 9:19 ` Christoph Hellwig @ 2004-08-28 3:56 ` Jeremy Allison 2004-08-28 19:18 ` Brad Boyer 0 siblings, 1 reply; 1835+ messages in thread From: Jeremy Allison @ 2004-08-28 3:56 UTC (permalink / raw) To: Christoph Hellwig, Jeremy Allison, Jamie Lokier, Rik van Riel, Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 10:19:56AM +0100, Christoph Hellwig wrote: > > Maybe you should learn from netatalk. I am well aware of what netatalk does. However netatalk isn't as widely used as Samba. Things they can get away with would cause our user community to flay us alive. We need a proper solution, not a nasty hack. That's like me telling you to "learn from *BSD". You have different user constituencies, you have to serve yours, I have to serve mine. Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 3:56 ` Jeremy Allison @ 2004-08-28 19:18 ` Brad Boyer 0 siblings, 0 replies; 1835+ messages in thread From: Brad Boyer @ 2004-08-28 19:18 UTC (permalink / raw) To: Jeremy Allison Cc: Christoph Hellwig, Jamie Lokier, Rik van Riel, Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 08:56:29PM -0700, Jeremy Allison wrote: > I am well aware of what netatalk does. However netatalk > isn't as widely used as Samba. Things they can get away > with would cause our user community to flay us alive. > > We need a proper solution, not a nasty hack. > > That's like me telling you to "learn from *BSD". You have > different user constituencies, you have to serve yours, > I have to serve mine. While I do think that there are some hacks in netatalk, the hacks there are not nearly as bad as the hacks that made it into the kernel to support exporting a native Mac filesystem. Take a look at the fork code in the hfs filesystem from 2.4 or before for exporting to netatalk. It's disgusting, and mostly because there is no easy way to export the resource fork and extra metadata that is essential to the client. The 2.6 code doesn't have that anymore, but I don't think it actually works with netatalk either. Perhaps someone can correct me on that, but I don't see how it could work. I would hope that something comes out of this whole discussion that will allow a much better interface between the hfs/hfsplus code in the kernel and netatalk. Brad Boyer flar@allandria.com ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel ` (4 preceding siblings ...) (?) @ 2004-08-26 22:20 ` Felipe Alfaro Solana -1 siblings, 0 replies; 1835+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 22:20 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 22:10, Rik van Riel wrote: > Do we really want to have a file paradigm that's different > from the other OSes out there ? > > What happens when users want to transfer data from Linux > to another system ? I think it depends on the transport being used: some transports could allow for metadata (i.e. a MIME-compatible transport) while some others don't. Of course, the remote side needs to also support metadata (i,e. using a MIME-compatible transport against a remote host that doesn't understand metadata is certainly impossible). For example, with FTP transports, I guess the only possible option is to transfer the unnamed/default stream. NFS should allow metadata, but this probably needs some kind of extensions to the NFS protocol. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel ` (5 preceding siblings ...) (?) @ 2004-08-29 0:04 ` Horst von Brand 2004-09-01 22:45 ` Jamie Lokier -1 siblings, 1 reply; 1835+ messages in thread From: Horst von Brand @ 2004-08-29 0:04 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel <riel@redhat.com> said: > On Thu, 26 Aug 2004, Linus Torvalds wrote: > > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > > contains both a standard data stream (the "file" part) _and_ pointers to > > other named streams (the "directory" part). > Thinking about it some more, how would file managers and > file chosers handle this situation ? > > Currently the user browses the directory tree and when > the user clicks on something, one of the following > happens: > > 1) if it is a directory, the file manager/choser changes > into that directory > > 2) if it is a file, the file is opened And now you have a mess. Is it "really" a file, or a directory? Why not just keep both well apart, and stay happy? I just fail to see what could be gained by having directories that really aren't, and files that aren't either. Use a directory if you want one, use a file elsewhere. > Now how do we present things to users ? > > How will users know when an object can only be chdired > into, or only be opened ? Easy: It is a directory, or it isn't. > For objects that do both, how does the user choose ? Don't give silly choices. > Do we really want to have a file paradigm that's different > from the other OSes out there ? I vote "no". There are/have been OSes with weird "files", none of them survived to get anywhere as popular as Unix and "file == stream of bytes". Even with much simpler variants like files as sequences of records. For a good reason: The Unix way is simple, and extremely flexible, as my proggie can define at its own whim how to handle what's inside. If a single stream isn't enough, we have directories. No need to innovate there. > What happens when users want to transfer data from Linux > to another system ? Or between Linux systems with different kernels that happen to implement different views/metadata on files. Please do remember devfs: It sounded like a cool idea, got into the kernel just to be thrown out later because nobody used it. Much heat was generated, nothing of permanent value. This looks the same: A very vocal tiny minority is clamoring for something completely non-Unix for totally bogus reasons. What happened to "code talks, bullshit walks"? There is _no_ code (== real-world, user programs that can't be done efficiently enough without this), so this nonsense should just be thrown out, and everybody go back to real hacking. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 0:04 ` Horst von Brand @ 2004-09-01 22:45 ` Jamie Lokier 2004-09-02 5:23 ` Olaf Hering 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 22:45 UTC (permalink / raw) To: Horst von Brand; +Cc: linux-fsdevel, linux-kernel, reiserfs-list Horst von Brand wrote: > Please do remember devfs: It sounded like a cool idea, got into the kernel > just to be thrown out later because nobody used it. Are you kidding? Lots of distros and many embedded Linuxes use devfs. And udev _still_ doesn't create device nodes properly. (Hint: I have to run two modprobe commands before pppd works. I have to run modprobe before openvpn or bochs work.) > What happened to "code talks, bullshit walks"? devfs is a fine example of why code isn't enough. With devfs the code came first, the >1 year of strategic bullshit politics from the "it's not traditional unix" crowd came later, then it went in, then lots of people used it, then it was replaced by something which still doesn't work as well as 2.4 does with or without devfs, and people are still using it despite it's faults. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 22:45 ` Jamie Lokier 2004-09-02 5:23 ` Olaf Hering @ 2004-09-02 5:23 ` Olaf Hering 0 siblings, 0 replies; 1835+ messages in thread From: Olaf Hering @ 2004-09-02 5:23 UTC (permalink / raw) To: Jamie Lokier; +Cc: Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list On Wed, Sep 01, Jamie Lokier wrote: > And udev _still_ doesn't create device nodes properly. (Hint: I have > to run two modprobe commands before pppd works. I have to run > modprobe before openvpn or bochs work.) Just because it is supposed to work that way. -- USB is for mice, FireWire is for men! sUse lINUX ag, nÜRNBERG ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 5:23 ` Olaf Hering 0 siblings, 0 replies; 1835+ messages in thread From: Olaf Hering @ 2004-09-02 5:23 UTC (permalink / raw) To: Jamie Lokier; +Cc: Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list On Wed, Sep 01, Jamie Lokier wrote: > And udev _still_ doesn't create device nodes properly. (Hint: I have > to run two modprobe commands before pppd works. I have to run > modprobe before openvpn or bochs work.) Just because it is supposed to work that way. -- USB is for mice, FireWire is for men! sUse lINUX ag, n√úRNBERG - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-02 5:23 ` Olaf Hering 0 siblings, 0 replies; 1835+ messages in thread From: Olaf Hering @ 2004-09-02 5:23 UTC (permalink / raw) To: Jamie Lokier; +Cc: Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list On Wed, Sep 01, Jamie Lokier wrote: > And udev _still_ doesn't create device nodes properly. (Hint: I have > to run two modprobe commands before pppd works. I have to run > modprobe before openvpn or bochs work.) Just because it is supposed to work that way. -- USB is for mice, FireWire is for men! sUse lINUX ag, nÜRNBERG - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:23 ` Linus Torvalds @ 2004-08-27 8:06 ` Hans Reiser 2004-08-26 20:10 ` Rik van Riel 2004-08-27 8:06 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > >To me, a filesystem that allows this thing doesn't really _have_ the >concept of "directory vs file". It's just a "filesystem object", and it >can act as _both_ a directory and a file. > > > I agree. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:06 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > >To me, a filesystem that allows this thing doesn't really _have_ the >concept of "directory vs file". It's just a "filesystem object", and it >can act as _both_ a directory and a file. > > > I agree. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:59 ` Rik van Riel ` (2 preceding siblings ...) (?) @ 2004-08-26 19:29 ` Jamie Lokier 2004-08-26 19:42 ` Christophe Saout -1 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 19:29 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > What do we do with O_CREAT ? > > Do we always allow both a directory and a file to be created with > the same name ? I haven't analysed it thoroughly. I suspect it's fine as long as: (1) O_CREAT creates something with "file-like attributes", meaning stat() says it's a regular file. (2) File-like means it can be unlinked, linked and renamed, even if someone has something inside it open. Nothing that can be created inside it will prevent it from being unlinked (unlike a directory-like object, where rmdir() will refuse if it's not empty). File-like also means that programs like "cp from to_my_new_file" won't do anything so silly as to write _inside_, the way that "cp from to_my_new_file/from" would. Basic utilities will need to be checked to make sure they don't try appending "/" as their method of deciding if the target object is a directory and should be entered. > Does this create a new class of "symlink attack" style security > holes ? Yes, but they don't need O_CREAT. An adversary creates a symlink to metadata inside your file. You write to it: it has interesting effects that weren't anticipated, such as either modifying another (virtual) file, or altering permissions or other parameters which writing doesn't normally do. This is very difficult to prevent. We are creating a way for scripts and classic unix tools to have easy access to fancy attributes which may affect the security: things like being able to change a file's permissions just by writing to the appropriate path, or reading characteristics of files that shouldn't be visible. In Hans Reiser's example of expanded /etc/passwd, atimes and mtimes of individual passwd entries is security information that perhaps shouldn't be exposed. In a way, these holes are similar to /proc, in that just writing has side effects which might not be acceptable. The solution is the same as for /proc (I hope): make sure the read permissions on all metadata inside a directory branch are restricted to the permissions of the file branch, and write permissions even more restricted at least by default. For the paranoid, one way is to make new files "not readable or executable as directories". It's possible for the file and directory branches to have different permission bits, though I could see that getting a bit annoying. Other possibilities abound. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:29 ` Jamie Lokier @ 2004-08-26 19:42 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 19:42 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Linus Torvalds, Diego Calleja, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2037 bytes --] Am Donnerstag, den 26.08.2004, 20:29 +0100 schrieb Jamie Lokier: > (1) O_CREAT creates something with "file-like > attributes", meaning stat() says it's a regular file. > > (2) File-like means it can be unlinked, linked and renamed, even if > someone has something inside it open. Nothing that can be created > inside it will prevent it from being unlinked (unlike a > directory-like object, where rmdir() will refuse if it's not empty). I would say hat the file and all of its directory gets invalidated and deleted as soon as the last opener of the file's main stream or some of its directory content disappears. How would that be? > > Does this create a new class of "symlink attack" style security > > holes ? > > Yes, but they don't need O_CREAT. > > An adversary creates a symlink to metadata inside your file. You > write to it: it has interesting effects that weren't anticipated, such > as either modifying another (virtual) file, or altering permissions or > other parameters which writing doesn't normally do. Hmm. > This is very difficult to prevent. Right. But to change some properties you usually need to be the owner of that file anyway, or root. > In Hans Reiser's example of expanded /etc/passwd, atimes > and mtimes of individual passwd entries is security information that > perhaps shouldn't be exposed. Only assuming mtime/ctime/atime/rwx/uid/gid of file contents can actually be independant of those from the main stream. I would say that the rights (read and write) for non-pseudo sub-files should the same as the file, uid and gid 644 (only owner can change these), something like that. mtime, atime and ctime always show the same as the file itself. Hmm. > The solution is the same as for /proc (I hope): make sure the read > permissions on all metadata inside a directory branch are restricted > to the permissions of the file branch, and write permissions even more > restricted at least by default. That's what I was thinking. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 19:42 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 19:42 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Linus Torvalds, Diego Calleja, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2037 bytes --] Am Donnerstag, den 26.08.2004, 20:29 +0100 schrieb Jamie Lokier: > (1) O_CREAT creates something with "file-like > attributes", meaning stat() says it's a regular file. > > (2) File-like means it can be unlinked, linked and renamed, even if > someone has something inside it open. Nothing that can be created > inside it will prevent it from being unlinked (unlike a > directory-like object, where rmdir() will refuse if it's not empty). I would say hat the file and all of its directory gets invalidated and deleted as soon as the last opener of the file's main stream or some of its directory content disappears. How would that be? > > Does this create a new class of "symlink attack" style security > > holes ? > > Yes, but they don't need O_CREAT. > > An adversary creates a symlink to metadata inside your file. You > write to it: it has interesting effects that weren't anticipated, such > as either modifying another (virtual) file, or altering permissions or > other parameters which writing doesn't normally do. Hmm. > This is very difficult to prevent. Right. But to change some properties you usually need to be the owner of that file anyway, or root. > In Hans Reiser's example of expanded /etc/passwd, atimes > and mtimes of individual passwd entries is security information that > perhaps shouldn't be exposed. Only assuming mtime/ctime/atime/rwx/uid/gid of file contents can actually be independant of those from the main stream. I would say that the rights (read and write) for non-pseudo sub-files should the same as the file, uid and gid 644 (only owner can change these), something like that. mtime, atime and ctime always show the same as the file itself. Hmm. > The solution is the same as for /proc (I hope): make sure the read > permissions on all metadata inside a directory branch are restricted > to the permissions of the file branch, and write permissions even more > restricted at least by default. That's what I was thinking. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:55 ` Linus Torvalds 2004-08-26 18:59 ` Rik van Riel @ 2004-08-27 8:46 ` Herbert Poetzl 1 sibling, 0 replies; 1835+ messages in thread From: Herbert Poetzl @ 2004-08-27 8:46 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 11:55:07AM -0700, Linus Torvalds wrote: > > > On Thu, 26 Aug 2004, Rik van Riel wrote: > > > > So you'd have both a file and a directory that just happen > > to have the same name ? How would this work in the dcache? > > There would be only one entry in the dcache. The lookup will select > whether it opens the file or the directory based on O_DIRECTORY (and > usage, of course - if it's in the middle of a path, it obviously needs to > be opened as a directory regardless). hmm, that might be interesting for applications like apache which allow to append arguments by 'building' a 'virtual' path ... http://www.kernel.org/list.cgi/some/more/args where the web server actually decides where the script ends and the arguments start by checking the filesystem ... > That's not the problem. The problem from a dcache standpoint ends up being > when the file has a link, and you have two paths to the same sub-file > through two different ways: > > .. create file 'x' with named stream 'y' ... > ln x z > ls -l x/y z/y /* it's the same attribute!! */ > > but this is actually exactly the same thing that we already have with > mounts, ie it is equivalent (from a dentry standpoint) to > > .. create directory 'x' with file 'y' .. > mkdir z > mount --bind x z > ls -l x/y z/y /* It's the same file!! */ > > so none of this is really anything "new" from a dcache standpoint. > > Except for all the details, of course ;) if the file is removed, but some attributes are 'locked' by access, will the 'other' attributes remain visible or disappear ... leaving a partial 'view' of the contents? or did I get the idea completely wrong? TIA, Herbert > Linus > - > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:15 ` Linus Torvalds 2004-08-26 18:41 ` Rik van Riel @ 2004-08-26 19:02 ` David Masover 2004-08-27 8:06 ` Hans Reiser 2004-08-26 20:44 ` Martin J. Bligh 2 siblings, 1 reply; 1835+ messages in thread From: David Masover @ 2004-08-26 19:02 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Linus Torvalds wrote: [...] | tar cvf file.tar file file/ I don't like it. Imagine this: $ cat file/perm 040700 drwx------ $ cat file/perm/user/writable 1 $ cat file/perm/uid 0 Do that tar, and you lose the uid. This is an example, and has plenty of flaws. The point is that often there may be files/directories within files/directories. In order to make this work, you either have to patch tar or split every file/directory into a file/directory pair, recursively, using some sort of mangling. "file" becomes file.f and file.d. Then tar works. To be honest, I like my "serializer" better. It just feels faster and cleaner. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQS4z1XgHNmZLgCUhAQKXAQ//eJ9tPFFEf1kpFY7gYUhMSUAAs1wIvVrN pOVyLPJwGTv4oIaq8/8Op+XmQt1BWfnFdgwSal5R7vuLlrj0/Lp714I4sTOb35+6 q4hB1Q6X1oKbS1QIVWM9zw8PaebTY1GZDYITqWEN8xFN2cx41m4HZZ9P2TXNGPWx 1TqeBAaR07N1LaDzDHWvXu2v8pP6Wjy8mmWbtHbjN7U8oLEAUVXmT/ZJ5MpVK6z6 FTTA6NnARpEDtWebTJit14IQYG88syl31+lNL/N87iTveYI3+6gAQniajkMda565 PnPbVCd/aLHpx4hNLYDiNZwGkhmCgA/V3/vHJGT1vbTWN56zFLF7XQqE0BMfR5k6 ozwzBZ368+suEs0pDplCJq7ru0BF3j49r2Ad7aILCJOENvN0uVzLNxmu+LfvARTm gG7UNkJYcAjwmUaE1vXEACnwKhtyyxIpRbsb9ALqbDS9KXB7eAaI2FwLPXCU/L61 qHWHN47KllWdfT4fttwPd/Lag7UpKu9McBPnp9ZOTuYGY6QTgflBG1C7kallcJUw 7cZh831PRqmnj2hWcAlSvinD3ClxaF9swewMPfQJ9I10n7p6w9Q9EJEXUX6+gUhv 5MD6q2WxqNVM/i5Y/2s3GQ+3lJ2VvnWHkJgc7JkmP0H1N0vocYWhKaofMOQXAy57 /ocTFfcwzUc= =YrfX -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:02 ` David Masover @ 2004-08-27 8:06 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: David Masover Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:06 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: David Masover Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:06 ` Hans Reiser @ 2004-08-27 14:12 ` Rik van Riel -1 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-27 14:12 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Hans Reiser wrote: > Why are you guys even considering going to any pain at all to distort > semantics for the sake of backup? tar is easy, we'll fix it and send in > a patch. Because not everybody uses tar. Quite a few people use a network backup system, while others use duplicity, RPM uses cpio internally and big companies tend to use proprietary network backup suites. Breaking people's setup is something to worry about. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 14:12 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-27 14:12 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Hans Reiser wrote: > Why are you guys even considering going to any pain at all to distort > semantics for the sake of backup? tar is easy, we'll fix it and send in > a patch. Because not everybody uses tar. Quite a few people use a network backup system, while others use duplicity, RPM uses cpio internally and big companies tend to use proprietary network backup suites. Breaking people's setup is something to worry about. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 14:12 ` Rik van Riel (?) @ 2004-08-27 18:27 ` Hans Reiser 2004-08-27 18:44 ` Lee Revell 2004-08-27 19:13 ` Rik van Riel -1 siblings, 2 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 18:27 UTC (permalink / raw) To: Rik van Riel Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: >On Fri, 27 Aug 2004, Hans Reiser wrote: > > > >>Why are you guys even considering going to any pain at all to distort >>semantics for the sake of backup? tar is easy, we'll fix it and send in >>a patch. >> >> > >Because not everybody uses tar. Quite a few people use a >network backup system, while others use duplicity, RPM uses >cpio internally and big companies tend to use proprietary >network backup suites. > >Breaking people's setup is something to worry about. > > > It is the tail that should not wag the dog. New semantics are going to break backups other than dd. We need a LOT of new semantics if we don't want to be inferior to Apple and MS. We should implement backup plugin methods though, so that we only have to break things once.... ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:27 ` Hans Reiser @ 2004-08-27 18:44 ` Lee Revell 2004-08-27 19:13 ` Rik van Riel 1 sibling, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-27 18:44 UTC (permalink / raw) To: Hans Reiser Cc: Rik van Riel, David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 2004-08-27 at 14:27, Hans Reiser wrote: > Rik van Riel wrote: > >On Fri, 27 Aug 2004, Hans Reiser wrote: > > > >>Why are you guys even considering going to any pain at all to distort > >>semantics for the sake of backup? tar is easy, we'll fix it and send in > >>a patch. > > > >Because not everybody uses tar. Quite a few people use a > >network backup system, while others use duplicity, RPM uses > >cpio internally and big companies tend to use proprietary > >network backup suites. > > > >Breaking people's setup is something to worry about. > > > It is the tail that should not wag the dog. New semantics are going to > break backups other than dd. We need a LOT of new semantics if we > don't want to be inferior to Apple and MS. > > We should implement backup plugin methods though, so that we only have > to break things once.... Every Linux shop I know of uses amanda anyway, which will not be affected by these changes, as it uses dump/restore. restore will need minor changes, I think dump might not have to be changed at all. If your backup system can't easily handle these changes, that's a pretty good indication that it's poorly designed. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:27 ` Hans Reiser @ 2004-08-27 19:13 ` Rik van Riel 2004-08-27 19:13 ` Rik van Riel 1 sibling, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-27 19:13 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Hans Reiser wrote: > It is the tail that should not wag the dog. That's no reason for dismissing technical questions with vigorous handwaving. > New semantics are going to break backups other than dd. We need a LOT > of new semantics if we don't want to be inferior to Apple and MS. Technical objections need to be addressed, otherwise Linux will end up inferior to Apple and MS. Whatever choice we end up making, it should be something that's both workable and elegant. It would be really nice if all the technical issues were worked out on linux-kernel before the files-as-directories stuff gets merged. There are quite a few technical questions in the thread that haven't been answered yet... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 19:13 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-27 19:13 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Hans Reiser wrote: > It is the tail that should not wag the dog. That's no reason for dismissing technical questions with vigorous handwaving. > New semantics are going to break backups other than dd. We need a LOT > of new semantics if we don't want to be inferior to Apple and MS. Technical objections need to be addressed, otherwise Linux will end up inferior to Apple and MS. Whatever choice we end up making, it should be something that's both workable and elegant. It would be really nice if all the technical issues were worked out on linux-kernel before the files-as-directories stuff gets merged. There are quite a few technical questions in the thread that haven't been answered yet... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 19:13 ` Rik van Riel (?) @ 2004-08-28 6:10 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 6:10 UTC (permalink / raw) To: Rik van Riel Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > >It would be really nice if all the technical issues were >worked out on linux-kernel before the files-as-directories >stuff gets merged. There are quite a few technical questions >in the thread that haven't been answered yet... > > > Well, did I answer the technical question with the filename/pseudos/backup method answer in your opinion? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 14:12 ` Rik van Riel (?) (?) @ 2004-08-27 20:56 ` Christian Mayrhuber -1 siblings, 0 replies; 1835+ messages in thread From: Christian Mayrhuber @ 2004-08-27 20:56 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel On Friday 27 August 2004 16:12, Rik van Riel wrote: > Because not everybody uses tar. Quite a few people use a > network backup system, while others use duplicity, RPM uses > cpio internally and big companies tend to use proprietary > network backup suites. Big backup systems support scripts that can be run prior to backup and post restore. You aready have to use star to do a metadata backup the acl's/ea into some tgz, which can be unpacked after restore. > Breaking people's setup is something to worry about. Yeah, maybe. But as usual on UN*X like systems most admins are able to come up with a quick shell/perl script to solve their particular problem. -- lg, Chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:06 ` Hans Reiser @ 2004-08-27 14:45 ` Rik van Riel -1 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-27 14:45 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Hans Reiser wrote: > Why are you guys even considering going to any pain at all to distort > semantics for the sake of backup? tar is easy, we'll fix it and send in > a patch. It's not as easy as you make it out, and not just because there are a few dozen backup programs that need fixing. The problem is more fundamental than that. Some of the file streams proposed need to be backed up, while others are alternative presentations of the file, which should not be backed up. Currently I see no way to distinguish between the stuff that should be backed up and the stuff that shouldn't. That problem needs to be resolved before we can even start thinking about fixing archivers... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 14:45 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-27 14:45 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Hans Reiser wrote: > Why are you guys even considering going to any pain at all to distort > semantics for the sake of backup? tar is easy, we'll fix it and send in > a patch. It's not as easy as you make it out, and not just because there are a few dozen backup programs that need fixing. The problem is more fundamental than that. Some of the file streams proposed need to be backed up, while others are alternative presentations of the file, which should not be backed up. Currently I see no way to distinguish between the stuff that should be backed up and the stuff that shouldn't. That problem needs to be resolved before we can even start thinking about fixing archivers... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 14:45 ` Rik van Riel (?) @ 2004-08-27 15:15 ` Spam -1 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-27 15:15 UTC (permalink / raw) To: Rik van Riel Cc: Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Fri, 27 Aug 2004, Hans Reiser wrote: >> Why are you guys even considering going to any pain at all to distort >> semantics for the sake of backup? tar is easy, we'll fix it and send in >> a patch. > It's not as easy as you make it out, and not just because > there are a few dozen backup programs that need fixing. > The problem is more fundamental than that. Some of the > file streams proposed need to be backed up, while others > are alternative presentations of the file, which should > not be backed up. No, not really. This is a user decision and should be options in the backup software. I don't think it is up to the kernel, filesystem, or the OS in general to decide what information the user want to retain or not. > Currently I see no way to distinguish between the stuff > that should be backed up and the stuff that shouldn't. > That problem needs to be resolved before we can even start > thinking about fixing archivers... The archivers should, as I said, allow the user to choose. It shouldn't be automatic. Default, should IMO be to store everything. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 15:15 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-27 15:15 UTC (permalink / raw) To: Rik van Riel Cc: Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Fri, 27 Aug 2004, Hans Reiser wrote: >> Why are you guys even considering going to any pain at all to distort >> semantics for the sake of backup? tar is easy, we'll fix it and send in >> a patch. > It's not as easy as you make it out, and not just because > there are a few dozen backup programs that need fixing. > The problem is more fundamental than that. Some of the > file streams proposed need to be backed up, while others > are alternative presentations of the file, which should > not be backed up. No, not really. This is a user decision and should be options in the backup software. I don't think it is up to the kernel, filesystem, or the OS in general to decide what information the user want to retain or not. > Currently I see no way to distinguish between the stuff > that should be backed up and the stuff that shouldn't. > That problem needs to be resolved before we can even start > thinking about fixing archivers... The archivers should, as I said, allow the user to choose. It shouldn't be automatic. Default, should IMO be to store everything. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 15:15 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-27 15:15 UTC (permalink / raw) To: Rik van Riel Cc: Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Fri, 27 Aug 2004, Hans Reiser wrote: >> Why are you guys even considering going to any pain at all to distort >> semantics for the sake of backup? tar is easy, we'll fix it and send in >> a patch. > It's not as easy as you make it out, and not just because > there are a few dozen backup programs that need fixing. > The problem is more fundamental than that. Some of the > file streams proposed need to be backed up, while others > are alternative presentations of the file, which should > not be backed up. No, not really. This is a user decision and should be options in the backup software. I don't think it is up to the kernel, filesystem, or the OS in general to decide what information the user want to retain or not. > Currently I see no way to distinguish between the stuff > that should be backed up and the stuff that shouldn't. > That problem needs to be resolved before we can even start > thinking about fixing archivers... The archivers should, as I said, allow the user to choose. It shouldn't be automatic. Default, should IMO be to store everything. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [SPAM] Re: silent semantic changes with reiser4 2004-08-27 15:15 ` Spam (?) (?) @ 2004-08-27 15:41 ` Jamie Lokier 2004-08-27 17:59 ` Hendrik Visage 2004-08-28 0:19 ` Spam -1 siblings, 2 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-27 15:41 UTC (permalink / raw) To: Spam Cc: Rik van Riel, Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam wrote: > > The problem is more fundamental than that. Some of the > > file streams proposed need to be backed up, while others > > are alternative presentations of the file, which should > > not be backed up. > > No, not really. This is a user decision and should be options in the > backup software. I don't think it is up to the kernel, filesystem, > or the OS in general to decide what information the user want to > retain or not. It is helpful for the OS, or a naming convention, to indicate what _is information_ though. It makes no sense to backup two or more copies of the _same information_, and it makes even less sense to try to restore them as it'll either be slow, fail (you can't always write to alternative presentations), or cause unwanted side effects. Just like when you backup a dynamic web site. You store the files which the server is using. You don't use "wget" to store the generated pages, that's not a useful backup and you can't restore from it. > > Currently I see no way to distinguish between the stuff > > that should be backed up and the stuff that shouldn't. > > > That problem needs to be resolved before we can even start > > thinking about fixing archivers... > > The archivers should, as I said, allow the user to choose. It > shouldn't be automatic. Default, should IMO be to store everything. Don't try to store different views of the same thing. When you try to restore, you _won't_ necessarily get back what you stored. Whereas if you follow the OS's advice, and skip virtual files, then backup and restore will recreate the filesystem, which is what you want. _That's_ storing everything. It's what you want from a backup. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [SPAM] Re: silent semantic changes with reiser4 2004-08-27 15:41 ` [SPAM] " Jamie Lokier @ 2004-08-27 17:59 ` Hendrik Visage 2004-08-28 0:19 ` Spam 1 sibling, 0 replies; 1835+ messages in thread From: Hendrik Visage @ 2004-08-27 17:59 UTC (permalink / raw) To: Jamie Lokier Cc: Spam, Rik van Riel, Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 04:41:56PM +0100, Jamie Lokier wrote: > > It is helpful for the OS, or a naming convention, to indicate what > _is information_ though. > > It makes no sense to backup two or more copies of the _same > information_, and it makes even less sense to try to restore them as > it'll either be slow, fail (you can't always write to alternative > presentations), or cause unwanted side effects. You sure you don't mean what is data (on disk) and what is information, ie. interpretted data? Following this whole threat is quite interesting, but I think there are two things that need to be pointed out (which I might've missed) is that hardlinks would be backed up multiple times if the archiver doesn't know the semantics of the filesystem. This is the reason why the <fs-of-choice>dump usually are faster than tar/cpio/etc. as they understand the low level and can optimize at that level. Also, a user might only have access to a certain subset of information, which he might to backup but not the rest of the stream/data, and such the choice should be the user's. > Just like when you backup a dynamic web site. You store the files > which the server is using. You don't use "wget" to store the > generated pages, that's not a useful backup and you can't restore from it. That depends ;^) But in the general case it's true. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 15:41 ` [SPAM] " Jamie Lokier 2004-08-27 17:59 ` Hendrik Visage @ 2004-08-28 0:19 ` Spam 1 sibling, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-28 0:19 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam wrote: >> > The problem is more fundamental than that. Some of the >> > file streams proposed need to be backed up, while others >> > are alternative presentations of the file, which should >> > not be backed up. >> >> No, not really. This is a user decision and should be options in the >> backup software. I don't think it is up to the kernel, filesystem, >> or the OS in general to decide what information the user want to >> retain or not. > It is helpful for the OS, or a naming convention, to indicate what > _is information_ though. > It makes no sense to backup two or more copies of the _same > information_, and it makes even less sense to try to restore them as > it'll either be slow, fail (you can't always write to alternative > presentations), or cause unwanted side effects. > Just like when you backup a dynamic web site. You store the files > which the server is using. You don't use "wget" to store the > generated pages, that's not a useful backup and you can't restore from it. I do not agree. Everything can be considered information, even if it is derived from an already existing file. If the user wants to not backup certain things then that is an option to tell the backup program. >> > Currently I see no way to distinguish between the stuff >> > that should be backed up and the stuff that shouldn't. >> >> > That problem needs to be resolved before we can even start >> > thinking about fixing archivers... >> >> The archivers should, as I said, allow the user to choose. It >> shouldn't be automatic. Default, should IMO be to store everything. > Don't try to store different views of the same thing. > When you try to restore, you _won't_ necessarily get back what you stored. No, it would be restored as it were before with the meta information intact. > Whereas if you follow the OS's advice, and skip virtual files, then > backup and restore will recreate the filesystem, which is what you want. > _That's_ storing everything. It's what you want from a backup. It would be storing everything but the virtual files. > -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 0:19 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-28 0:19 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam wrote: >> > The problem is more fundamental than that. Some of the >> > file streams proposed need to be backed up, while others >> > are alternative presentations of the file, which should >> > not be backed up. >> >> No, not really. This is a user decision and should be options in the >> backup software. I don't think it is up to the kernel, filesystem, >> or the OS in general to decide what information the user want to >> retain or not. > It is helpful for the OS, or a naming convention, to indicate what > _is information_ though. > It makes no sense to backup two or more copies of the _same > information_, and it makes even less sense to try to restore them as > it'll either be slow, fail (you can't always write to alternative > presentations), or cause unwanted side effects. > Just like when you backup a dynamic web site. You store the files > which the server is using. You don't use "wget" to store the > generated pages, that's not a useful backup and you can't restore from it. I do not agree. Everything can be considered information, even if it is derived from an already existing file. If the user wants to not backup certain things then that is an option to tell the backup program. >> > Currently I see no way to distinguish between the stuff >> > that should be backed up and the stuff that shouldn't. >> >> > That problem needs to be resolved before we can even start >> > thinking about fixing archivers... >> >> The archivers should, as I said, allow the user to choose. It >> shouldn't be automatic. Default, should IMO be to store everything. > Don't try to store different views of the same thing. > When you try to restore, you _won't_ necessarily get back what you stored. No, it would be restored as it were before with the meta information intact. > Whereas if you follow the OS's advice, and skip virtual files, then > backup and restore will recreate the filesystem, which is what you want. > _That's_ storing everything. It's what you want from a backup. It would be storing everything but the virtual files. > -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 0:19 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-28 0:19 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam wrote: >> > The problem is more fundamental than that. Some of the >> > file streams proposed need to be backed up, while others >> > are alternative presentations of the file, which should >> > not be backed up. >> >> No, not really. This is a user decision and should be options in the >> backup software. I don't think it is up to the kernel, filesystem, >> or the OS in general to decide what information the user want to >> retain or not. > It is helpful for the OS, or a naming convention, to indicate what > _is information_ though. > It makes no sense to backup two or more copies of the _same > information_, and it makes even less sense to try to restore them as > it'll either be slow, fail (you can't always write to alternative > presentations), or cause unwanted side effects. > Just like when you backup a dynamic web site. You store the files > which the server is using. You don't use "wget" to store the > generated pages, that's not a useful backup and you can't restore from it. I do not agree. Everything can be considered information, even if it is derived from an already existing file. If the user wants to not backup certain things then that is an option to tell the backup program. >> > Currently I see no way to distinguish between the stuff >> > that should be backed up and the stuff that shouldn't. >> >> > That problem needs to be resolved before we can even start >> > thinking about fixing archivers... >> >> The archivers should, as I said, allow the user to choose. It >> shouldn't be automatic. Default, should IMO be to store everything. > Don't try to store different views of the same thing. > When you try to restore, you _won't_ necessarily get back what you stored. No, it would be restored as it were before with the meta information intact. > Whereas if you follow the OS's advice, and skip virtual files, then > backup and restore will recreate the filesystem, which is what you want. > _That's_ storing everything. It's what you want from a backup. It would be storing everything but the virtual files. > -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 0:19 ` Spam @ 2004-08-28 1:59 ` Rik van Riel -1 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-28 1:59 UTC (permalink / raw) To: Spam Cc: Jamie Lokier, Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, 28 Aug 2004, Spam wrote: > > Whereas if you follow the OS's advice, and skip virtual files, then > > backup and restore will recreate the filesystem, which is what you want. > > > _That's_ storing everything. It's what you want from a backup. > > It would be storing everything but the virtual files. Thing is, there is no way to distinguish between what are virtual files and what are actual streams hidden inside a file. You don't know what should and shouldn't be backed up... Hans says there must be a way, somehow, but I haven't seen him tell this mailing list what exactly it is ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 1:59 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-28 1:59 UTC (permalink / raw) To: Spam Cc: Jamie Lokier, Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, 28 Aug 2004, Spam wrote: > > Whereas if you follow the OS's advice, and skip virtual files, then > > backup and restore will recreate the filesystem, which is what you want. > > > _That's_ storing everything. It's what you want from a backup. > > It would be storing everything but the virtual files. Thing is, there is no way to distinguish between what are virtual files and what are actual streams hidden inside a file. You don't know what should and shouldn't be backed up... Hans says there must be a way, somehow, but I haven't seen him tell this mailing list what exactly it is ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 1:59 ` Rik van Riel (?) @ 2004-08-28 2:05 ` Linus Torvalds 2004-08-28 9:53 ` Hans Reiser 2004-08-28 17:05 ` Helge Hafting -1 siblings, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-28 2:05 UTC (permalink / raw) To: Rik van Riel Cc: Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Fri, 27 Aug 2004, Rik van Riel wrote: > > Thing is, there is no way to distinguish between what are > virtual files and what are actual streams hidden inside a > file. You don't know what should and shouldn't be backed > up... I think that lack of distinguishing poiwer is more serious for directories. The more I think I think about it, the more I wonder whether Solaris did things right - having a special operation to "cross the boundary". I suspect Solaris did it that way because it's a hell of a lot easier to do it like that, but regardless, it would solve the issue of real directories having both real children _and_ the "extra streams". Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 2:05 ` Linus Torvalds @ 2004-08-28 9:53 ` Hans Reiser 2004-08-28 17:05 ` Helge Hafting 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 9:53 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Just have a special name instead of a special boundary, or, better, have a filename/pseudos/backup method that outputs everything needed to backup the object "filename". Hans Linus Torvalds wrote: >On Fri, 27 Aug 2004, Rik van Riel wrote: > > >>Thing is, there is no way to distinguish between what are >>virtual files and what are actual streams hidden inside a >>file. You don't know what should and shouldn't be backed >>up... >> >> > >I think that lack of distinguishing poiwer is more serious for >directories. The more I think I think about it, the more I wonder whether >Solaris did things right - having a special operation to "cross the >boundary". > >I suspect Solaris did it that way because it's a hell of a lot easier to >do it like that, but regardless, it would solve the issue of real >directories having both real children _and_ the "extra streams". > > Linus > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 9:53 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 9:53 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Just have a special name instead of a special boundary, or, better, have a filename/pseudos/backup method that outputs everything needed to backup the object "filename". Hans Linus Torvalds wrote: >On Fri, 27 Aug 2004, Rik van Riel wrote: > > >>Thing is, there is no way to distinguish between what are >>virtual files and what are actual streams hidden inside a >>file. You don't know what should and shouldn't be backed >>up... >> >> > >I think that lack of distinguishing poiwer is more serious for >directories. The more I think I think about it, the more I wonder whether >Solaris did things right - having a special operation to "cross the >boundary". > >I suspect Solaris did it that way because it's a hell of a lot easier to >do it like that, but regardless, it would solve the issue of real >directories having both real children _and_ the "extra streams". > > Linus > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 9:53 ` Hans Reiser @ 2004-08-28 11:21 ` Spam -1 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-28 11:21 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Rik van Riel, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro I am thinking that is there was a proper API for accessing the filesystem then this problem wouldn't arise because things could be done behind the curtains inside the API, instead of having all the tools to be rewritten to know. Think of FAT32, for example, the new filenames will be retained even if you use old tools on them. It is only if you mount it as a FAT16 volume they will be lost. Even if this is fixed for Reiser4 now. Next time someone wants to make changes like this we have the exact same problem yet again. > Just have a special name instead of a special boundary, or, better, have > a filename/pseudos/backup method that outputs everything needed to > backup the object "filename". > Hans > Linus Torvalds wrote: >>On Fri, 27 Aug 2004, Rik van Riel wrote: >> >> >>>Thing is, there is no way to distinguish between what are >>>virtual files and what are actual streams hidden inside a >>>file. You don't know what should and shouldn't be backed >>>up... >>> >>> >> >>I think that lack of distinguishing poiwer is more serious for >>directories. The more I think I think about it, the more I wonder whether >>Solaris did things right - having a special operation to "cross the >>boundary". >> >>I suspect Solaris did it that way because it's a hell of a lot easier to >>do it like that, but regardless, it would solve the issue of real >>directories having both real children _and_ the "extra streams". >> >> Linus >> >> >> >> ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 11:21 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-28 11:21 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Rik van Riel, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro I am thinking that is there was a proper API for accessing the filesystem then this problem wouldn't arise because things could be done behind the curtains inside the API, instead of having all the tools to be rewritten to know. Think of FAT32, for example, the new filenames will be retained even if you use old tools on them. It is only if you mount it as a FAT16 volume they will be lost. Even if this is fixed for Reiser4 now. Next time someone wants to make changes like this we have the exact same problem yet again. > Just have a special name instead of a special boundary, or, better, have > a filename/pseudos/backup method that outputs everything needed to > backup the object "filename". > Hans > Linus Torvalds wrote: >>On Fri, 27 Aug 2004, Rik van Riel wrote: >> >> >>>Thing is, there is no way to distinguish between what are >>>virtual files and what are actual streams hidden inside a >>>file. You don't know what should and shouldn't be backed >>>up... >>> >>> >> >>I think that lack of distinguishing poiwer is more serious for >>directories. The more I think I think about it, the more I wonder whether >>Solaris did things right - having a special operation to "cross the >>boundary". >> >>I suspect Solaris did it that way because it's a hell of a lot easier to >>do it like that, but regardless, it would solve the issue of real >>directories having both real children _and_ the "extra streams". >> >> Linus >> >> >> >> ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 9:53 ` Hans Reiser @ 2004-08-28 17:19 ` Hubert Chan -1 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-28 17:19 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Hans" == Hans Reiser <reiser@namesys.com> writes: Hans> Just have a special name instead of a special boundary, or, Hans> better, have a filename/pseudos/backup method that outputs Hans> everything needed to backup the object "filename". For the Reiser4 case, this could also be solved using views, right? Just create a view that exposes everything that should be backed up, and nothing else? What "instructions" would a backup program need, other than knowing what files to backup, and what files to not backup? Where are views going to be handled? It seems like something that would be useful for every other filesystem, and so shouldn't be handled too close to Reiser4. (Or is that a debate/discussion/flamewar we don't want to get into right now? ;-) ) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 17:19 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-28 17:19 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Hans" == Hans Reiser <reiser@namesys.com> writes: Hans> Just have a special name instead of a special boundary, or, Hans> better, have a filename/pseudos/backup method that outputs Hans> everything needed to backup the object "filename". For the Reiser4 case, this could also be solved using views, right? Just create a view that exposes everything that should be backed up, and nothing else? What "instructions" would a backup program need, other than knowing what files to backup, and what files to not backup? Where are views going to be handled? It seems like something that would be useful for every other filesystem, and so shouldn't be handled too close to Reiser4. (Or is that a debate/discussion/flamewar we don't want to get into right now? ;-) ) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 2:05 ` Linus Torvalds @ 2004-08-28 17:05 ` Helge Hafting 2004-08-28 17:05 ` Helge Hafting 1 sibling, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-28 17:05 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Fri, Aug 27, 2004 at 07:05:35PM -0700, Linus Torvalds wrote: > > > On Fri, 27 Aug 2004, Rik van Riel wrote: > > > > Thing is, there is no way to distinguish between what are > > virtual files and what are actual streams hidden inside a > > file. You don't know what should and shouldn't be backed > > up... > > I think that lack of distinguishing poiwer is more serious for > directories. The more I think I think about it, the more I wonder whether > Solaris did things right - having a special operation to "cross the > boundary". > > I suspect Solaris did it that way because it's a hell of a lot easier to > do it like that, but regardless, it would solve the issue of real > directories having both real children _and_ the "extra streams". There are many ways of doing this. Several extra streams to a directory that aren't ordinary files in the directory? It seems to me that we can get a lot of nice functionality in a simpler way: Instead of thinking about a number of streams attached to something that is either an ordinary file or directory, just say that the only change will be that a directory may have a _single_ file stream in addition to being a plain directory. The conceptual change is smaller. Still you achieve multiple streams, the "main" stream in such a dir+file is the single stream attached to the directory. All the "extra streams" are files inside the (normal) directory. * This way, the extra streams aren't "special" in any way and should work fine with existing tools. * Serving such a filesystem to clients that use multiple streams will allow any number of streams per file for them * Moving the directory+file thing will move both the directory and the special stream, and all the extra streams follow because they're �part of the directory as usual�. If the VFS is to be extended in order to support file-as-directory (or vice versa) then hopefully it can be done in a simple way. Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 17:05 ` Helge Hafting 0 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-28 17:05 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Fri, Aug 27, 2004 at 07:05:35PM -0700, Linus Torvalds wrote: > > > On Fri, 27 Aug 2004, Rik van Riel wrote: > > > > Thing is, there is no way to distinguish between what are > > virtual files and what are actual streams hidden inside a > > file. You don't know what should and shouldn't be backed > > up... > > I think that lack of distinguishing poiwer is more serious for > directories. The more I think I think about it, the more I wonder whether > Solaris did things right - having a special operation to "cross the > boundary". > > I suspect Solaris did it that way because it's a hell of a lot easier to > do it like that, but regardless, it would solve the issue of real > directories having both real children _and_ the "extra streams". There are many ways of doing this. Several extra streams to a directory that aren't ordinary files in the directory? It seems to me that we can get a lot of nice functionality in a simpler way: Instead of thinking about a number of streams attached to something that is either an ordinary file or directory, just say that the only change will be that a directory may have a _single_ file stream in addition to being a plain directory. The conceptual change is smaller. Still you achieve multiple streams, the "main" stream in such a dir+file is the single stream attached to the directory. All the "extra streams" are files inside the (normal) directory. * This way, the extra streams aren't "special" in any way and should work fine with existing tools. * Serving such a filesystem to clients that use multiple streams will allow any number of streams per file for them * Moving the directory+file thing will move both the directory and the special stream, and all the extra streams follow because they're �part of the directory as usual�. If the VFS is to be extended in order to support file-as-directory (or vice versa) then hopefully it can be done in a simple way. Helge Hafting - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 17:05 ` Helge Hafting (?) @ 2004-08-28 18:09 ` Linus Torvalds 2004-08-28 18:20 ` Rik van Riel ` (4 more replies) -1 siblings, 5 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-28 18:09 UTC (permalink / raw) To: Helge Hafting Cc: Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sat, 28 Aug 2004, Helge Hafting wrote: > > > I think that lack of distinguishing power is more serious for > > directories. The more I think I think about it, the more I wonder whether > > Solaris did things right - having a special operation to "cross the > > boundary". > > > > I suspect Solaris did it that way because it's a hell of a lot easier to > > do it like that, but regardless, it would solve the issue of real > > directories having both real children _and_ the "extra streams". > > There are many ways of doing this. Several extra streams to a directory > that aren't ordinary files in the directory? Well.. Yes. We already have "." and "..", which are "special extra streams" in a sense. However, people expect them, and know to ignore them. The same wouldn't be true of new naming. > It seems to me that we can get a lot of nice functionality in a simpler way: > Instead of thinking about a number of streams attached to something > that is either an ordinary file or directory, just say that the only > change will be that a directory may have a _single_ file stream in > addition to being a plain directory. That doesn't really help us. What would the name be, and how could you avoid clashes? > If the VFS is to be extended in order to support file-as-directory (or > vice versa) then hopefully it can be done in a simple way. I'm pretty confident that we can extend the VFS layer to support named streams (see the technical discussion with Al, rather than the flames in this thread). I also clearly believe that it is worth it, but I'm starting to wonder if we should have a special open flag to make people select the stream. If you look at the Solaris interface, the _nice_ part about "openat()" is that you can do something like file = open(filename, O_RDONLY); if (file < 0) return -ENOENT; icon = openat(file, "icon", O_RDONLY | O_XATTR); if (icon < 0) icon = default_icon_file; .. and it will work regardless of whether "filename" is a directory or a regular file, if I've understood correctly. Now, I think that makes sense for several reasons: - single case - race-free (think "stat()" vs "fstat()" races). - I think we want to do "openat()" regardless of whether we ever support extended attributes or not ("openat()" is nice for doing "namei()" in user space even in the absense of any attributes or named streams). So what we can do is - implement openat() regardless, and expect to do the Solaris thing for it if we ever do streams. - _also_ support the "implied named attributes" for regular files, so that you don't have to use "openat()" to access them. Comments? Does anybody hate "openat()" for any reason (regardless of attributes)? We can easily support it, we'd just need to pass in the file to use as part of the "nameidata" thing or add an argument (it would also possibly be cleaner if we made "fs->pwd" be a "struct file"). Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds @ 2004-08-28 18:20 ` Rik van Riel 2004-08-28 18:29 ` viro ` (3 subsequent siblings) 4 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-28 18:20 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sat, 28 Aug 2004, Linus Torvalds wrote: > Well.. Yes. We already have "." and "..", which are "special extra > streams" in a sense. However, people expect them, and know to ignore them. > The same wouldn't be true of new naming. If the streams that don't contain data weren't real files, but instead fifos, backup programs would ignore them fine. Tar seems to ignore special nodes ... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 18:20 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-28 18:20 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sat, 28 Aug 2004, Linus Torvalds wrote: > Well.. Yes. We already have "." and "..", which are "special extra > streams" in a sense. However, people expect them, and know to ignore them. > The same wouldn't be true of new naming. If the streams that don't contain data weren't real files, but instead fifos, backup programs would ignore them fine. Tar seems to ignore special nodes ... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds 2004-08-28 18:20 ` Rik van Riel @ 2004-08-28 18:29 ` viro 2004-08-28 18:44 ` Linus Torvalds 2004-08-28 22:29 ` Hans Reiser ` (2 subsequent siblings) 4 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-28 18:29 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 11:09:38AM -0700, Linus Torvalds wrote: > Comments? Does anybody hate "openat()" for any reason (regardless of > attributes)? We can easily support it, we'd just need to pass in the file > to use as part of the "nameidata" thing or add an argument (it would also > possibly be cleaner if we made "fs->pwd" be a "struct file"). What would your openat() produce? Normal struct file? Then what's going to be its vfsmount/dentry and what will they be attached to? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:29 ` viro @ 2004-08-28 18:44 ` Linus Torvalds 2004-08-28 18:56 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-28 18:44 UTC (permalink / raw) To: viro Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > What would your openat() produce? Normal struct file? Then what's going > to be its vfsmount/dentry and what will they be attached to? Normal file descriptor, exactly like "open()". And it's going to have all the same vfsmount/dentry thing it would have if you looked it up the whole way. I don't understand your question.. Ignore the O_XATTR thing for a while, and assume it's just a convenient combination of "fchdir + open" (plus "fchdir back" of course, but that's beside the point - openat() doesn't ever really change cwd). Going back to O_XATTR: that would end up doing the "special vfsmount" magic at the beginning of the lookup. If the dentry you started with wasn't marked D_HYBRID, it would just return -ENOTDIR. So we could do openat() _without_ any of the lookup_mnt() etc special cases. This interface is independent of whether we want to expose the attributes through a normal lookup - we can do either, both, or neither as we choose. The nice thing about "openat()" is - people can definitely find uses for it even without attributes. - it's "portable". Well, at least somebody else does the same thing, which is nice for user-space developers. You don't use a Linux-only interface, you use a Linux/Solaris one, which makes a lot of people a lot more happy. Remember, portability has always been very important to Linux, and Linux-only features while nice are certainly not as nice as features you can also find in other places. NIH is a disease. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:44 ` Linus Torvalds @ 2004-08-28 18:56 ` viro 2004-08-28 19:16 ` Linus Torvalds 2004-08-29 0:46 ` Paul Jakma 0 siblings, 2 replies; 1835+ messages in thread From: viro @ 2004-08-28 18:56 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 11:44:52AM -0700, Linus Torvalds wrote: > Going back to O_XATTR: that would end up doing the "special vfsmount" > magic at the beginning of the lookup. If the dentry you started with > wasn't marked D_HYBRID, it would just return -ENOTDIR. OK, let me restate the question - what do we get from pwd if we do fchdir() to such beast? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:56 ` viro @ 2004-08-28 19:16 ` Linus Torvalds 2004-08-28 19:41 ` Andries Brouwer 2004-08-28 19:45 ` viro 2004-08-29 0:46 ` Paul Jakma 1 sibling, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-28 19:16 UTC (permalink / raw) To: viro Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > On Sat, Aug 28, 2004 at 11:44:52AM -0700, Linus Torvalds wrote: > > Going back to O_XATTR: that would end up doing the "special vfsmount" > > magic at the beginning of the lookup. If the dentry you started with > > wasn't marked D_HYBRID, it would just return -ENOTDIR. > > OK, let me restate the question - what do we get from pwd if we do > fchdir() to such beast? We'll see the "attribute path". We could (if we want to) mark the point where we walked into attribute space somehow (since we can see it by just looking at the vfsmount: d_root/d_mountpoint being the same), but even if we don't, we'll get a sane-looking path. (It's not just "getcwd()", it's /proc interfaces too, for opening files, and we already have the notion of magic markers like "(deleted)" to show human-readable information). The question about what to do at the "attribute point" (and there may actually be several, if the filesystem supports attributes on attribute files) likely depends on whether we support the previously discussed "lookup()" magic for attributes. So if we do support it, I think we should just make the attribute point look exactly like a normal directory, since that path would work as-is. If we don't support it (and at directory points), we might want to just consider it a special root. NOTE! Anybody who tries to use the "getcwd()" string as a real path is already broken - you have pathname permissions that may not make it possible to look up the path you see. So we have multiple options: - support "file/attribute" lookup: show the path as-is, so you'd show it as "/path/to/file/attribute" - alternatively, even if you _do_ support the normal lookup, show it with a double slash (which will still be a valid path), just as a visual clue: "/path/to/file//attribute" - for directories, or if we do _not_ support the extended lookup format, we could show it the same way we show deleted files, something like this: "/path/to/file/attribute (attr)" - using "http notation" for non-standard-namespaces (we already do this for sockets and pipes, for example) "attr:attribute@/path/to/file" pick your poison. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:16 ` Linus Torvalds @ 2004-08-28 19:41 ` Andries Brouwer 2004-08-28 19:46 ` Linus Torvalds 2004-08-28 19:45 ` viro 1 sibling, 1 reply; 1835+ messages in thread From: Andries Brouwer @ 2004-08-28 19:41 UTC (permalink / raw) To: Linus Torvalds Cc: viro, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 12:16:47PM -0700, Linus Torvalds wrote: If I see it correctly, you want to group a file and some ancillary files together. The Unix way would be to make a directory and put them all there: xterm/xterm xterm/xterm.icon But you are unsatisfied and want xterm xterm/xterm.icon As long as we agree that the latter really means the former, there are no problems in finding out what should happen. The conclusion is, that a directory carries an additional bit that says "if I am opened as a regular file then use the file of the same name inside". Now there is no attribute space, just a shorthand. Andries ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:41 ` Andries Brouwer @ 2004-08-28 19:46 ` Linus Torvalds 2004-08-29 13:56 ` Andries Brouwer 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-28 19:46 UTC (permalink / raw) To: Andries Brouwer Cc: viro, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004, Andries Brouwer wrote: > > If I see it correctly, you want to group a file and some > ancillary files together. > > The Unix way would be to make a directory and put them all there: > xterm/xterm > xterm/xterm.icon > > But you are unsatisfied and want > xterm > xterm/xterm.icon > > As long as we agree that the latter really means the former, > there are no problems in finding out what should happen. > > The conclusion is, that a directory carries an additional bit > that says "if I am opened as a regular file then use the file > of the same name inside". > > Now there is no attribute space, just a shorthand. It's more than a shorthand, though. _Much_ more. There's the small issue of atomicity and locking. There's the small issue of hardlinks. Both of those are why it would need special support. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:46 ` Linus Torvalds @ 2004-08-29 13:56 ` Andries Brouwer 0 siblings, 0 replies; 1835+ messages in thread From: Andries Brouwer @ 2004-08-29 13:56 UTC (permalink / raw) To: Linus Torvalds Cc: Andries Brouwer, viro, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 12:46:10PM -0700, Linus Torvalds wrote: > > Now there is no attribute space, just a shorthand. > > It's more than a shorthand, though. _Much_ more. > ... > Both of those are why it would need special support. Yes - support, I do not argue against that. But I argue against an attribute space. This xterm.icon was xterm.icons - with both large and small icons. And these icons have copyrights. Etc. There is just an arbitrary tree below this xterm filesystem object. The files below are in no way different from the files above. Unix says: everything is a file. Now we get: everything is a directory. And some directories have a file attached. Andries ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:16 ` Linus Torvalds 2004-08-28 19:41 ` Andries Brouwer @ 2004-08-28 19:45 ` viro 2004-08-29 5:12 ` Linus Torvalds 1 sibling, 1 reply; 1835+ messages in thread From: viro @ 2004-08-28 19:45 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 12:16:47PM -0700, Linus Torvalds wrote: > So if we do support it, I think we should just make the attribute point > look exactly like a normal directory, since that path would work as-is. If > we don't support it (and at directory points), we might want to just > consider it a special root. > > NOTE! Anybody who tries to use the "getcwd()" string as a real path is > already broken - you have pathname permissions that may not make it > possible to look up the path you see. OK, forget getcwd(). What does lookup of .. do from that point? *Especially* for stuff you've got from regular files. That's the decision that needs to be made. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:45 ` viro @ 2004-08-29 5:12 ` Linus Torvalds 2004-08-29 5:44 ` Jeremy Allison 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-29 5:12 UTC (permalink / raw) To: viro Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > OK, forget getcwd(). What does lookup of .. do from that point? *Especially* > for stuff you've got from regular files. That's the decision that needs to > be made. I think that will decide on whether we expose attributes through the normal namespace or not. If we do expose them in the normal namespace, then ".." should work the way the namespace looks: if you do ".." on the "attribute directory" of a file, you get the directory that the file was in. Ie an old-style user-space "getcwd()" would give the right path (well, an old-style user-space getcwd() would probably refuse the file on the base that it is S_IFREG, but ignoring that..) If we _don't_ expose it in the normal namespace, we should should either just error out (logically you'd get the file itself, but I really don't wan tto have ".." return a non-directory, because _that_ really might confuse things), or you'd just return the same directory (ie it would be a "local root" in the namespace you got moved to). So let's try to be self-consistent with how we expose it in the normal namespace. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 5:12 ` Linus Torvalds @ 2004-08-29 5:44 ` Jeremy Allison 0 siblings, 0 replies; 1835+ messages in thread From: Jeremy Allison @ 2004-08-29 5:44 UTC (permalink / raw) To: Linus Torvalds Cc: viro, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 10:12:39PM -0700, Linus Torvalds wrote: > > If we do expose them in the normal namespace, then ".." should work the > way the namespace looks: if you do ".." on the "attribute directory" of a > file, you get the directory that the file was in. Ie an old-style > user-space "getcwd()" would give the right path (well, an old-style > user-space getcwd() would probably refuse the file on the base that it is > S_IFREG, but ignoring that..) Errr, why don't we check what Solaris does in these cases and see if that makes sense. They do seem to have made some progress along this path and it wouldn't make sense to create new semantics just for the sake of it. Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:56 ` viro 2004-08-28 19:16 ` Linus Torvalds @ 2004-08-29 0:46 ` Paul Jakma 1 sibling, 0 replies; 1835+ messages in thread From: Paul Jakma @ 2004-08-29 0:46 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > OK, let me restate the question - what do we get from pwd if we do > fchdir() to such beast? On solaris: ENOTDIR from getcwd(), with 'pwd' executed inside a shell started with runat. .. entry exists, but doesnt lead anywhere, cd .. returning ENOTDIR too. regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: If you didn't get caught, did you really do it? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds @ 2004-08-28 22:29 ` Hans Reiser 2004-08-28 18:29 ` viro ` (3 subsequent siblings) 4 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 22:29 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro I object to openat()..... My reason is that the things that distinguish between objects should be the names, not the choice of system call. The reason for this is that it improves closure and namespace unification to do so, because it allows all the objects to be accessed within the same namespace. Yes, it can be useful to allow a namespace to exclude some objects, but that exclusion should not be mandated. If you want to exclude, you should cd or chroot to /proc/nopseudos and find a view of the filesystem that excludes metas, or mount with -nopseuodos. Do you see why I say this? I can say a lot more about the damage of fragmenting namespaces into multiple apis.... Why look at xattrs....;-) Hans Linus Torvalds wrote: > > >I'm pretty confident that we can extend the VFS layer to support named >streams (see the technical discussion with Al, rather than the flames in >this thread). I also clearly believe that it is worth it, but I'm starting >to wonder if we should have a special open flag to make people select the >stream. > >If you look at the Solaris interface, the _nice_ part about "openat()" is >that you can do something like > > file = open(filename, O_RDONLY); > if (file < 0) > return -ENOENT; > icon = openat(file, "icon", O_RDONLY | O_XATTR); > if (icon < 0) > icon = default_icon_file; > .. > >and it will work regardless of whether "filename" is a directory or a >regular file, if I've understood correctly. > >Now, I think that makes sense for several reasons: > - single case > - race-free (think "stat()" vs "fstat()" races). > - I think we want to do "openat()" regardless of whether we ever > support extended attributes or not ("openat()" is nice for doing > "namei()" in user space even in the absense of any attributes or > named streams). > >So what we can do is > - implement openat() regardless, and expect to do the Solaris thing for > it if we ever do streams. > - _also_ support the "implied named attributes" for regular files, so > that you don't have to use "openat()" to access them. > >Comments? Does anybody hate "openat()" for any reason (regardless of >attributes)? We can easily support it, we'd just need to pass in the file >to use as part of the "nameidata" thing or add an argument (it would also >possibly be cleaner if we made "fs->pwd" be a "struct file"). > > Linus >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 22:29 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 22:29 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro I object to openat()..... My reason is that the things that distinguish between objects should be the names, not the choice of system call. The reason for this is that it improves closure and namespace unification to do so, because it allows all the objects to be accessed within the same namespace. Yes, it can be useful to allow a namespace to exclude some objects, but that exclusion should not be mandated. If you want to exclude, you should cd or chroot to /proc/nopseudos and find a view of the filesystem that excludes metas, or mount with -nopseuodos. Do you see why I say this? I can say a lot more about the damage of fragmenting namespaces into multiple apis.... Why look at xattrs....;-) Hans Linus Torvalds wrote: > > >I'm pretty confident that we can extend the VFS layer to support named >streams (see the technical discussion with Al, rather than the flames in >this thread). I also clearly believe that it is worth it, but I'm starting >to wonder if we should have a special open flag to make people select the >stream. > >If you look at the Solaris interface, the _nice_ part about "openat()" is >that you can do something like > > file = open(filename, O_RDONLY); > if (file < 0) > return -ENOENT; > icon = openat(file, "icon", O_RDONLY | O_XATTR); > if (icon < 0) > icon = default_icon_file; > .. > >and it will work regardless of whether "filename" is a directory or a >regular file, if I've understood correctly. > >Now, I think that makes sense for several reasons: > - single case > - race-free (think "stat()" vs "fstat()" races). > - I think we want to do "openat()" regardless of whether we ever > support extended attributes or not ("openat()" is nice for doing > "namei()" in user space even in the absense of any attributes or > named streams). > >So what we can do is > - implement openat() regardless, and expect to do the Solaris thing for > it if we ever do streams. > - _also_ support the "implied named attributes" for regular files, so > that you don't have to use "openat()" to access them. > >Comments? Does anybody hate "openat()" for any reason (regardless of >attributes)? We can easily support it, we'd just need to pass in the file >to use as part of the "nameidata" thing or add an argument (it would also >possibly be cleaner if we made "fs->pwd" be a "struct file"). > > Linus >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* The Necessity of File Types (was silent semantic changes with reiser4) 2004-08-28 22:29 ` Hans Reiser (?) @ 2004-08-29 0:40 ` Alexander G. M. Smith 2004-08-31 0:42 ` David Masover -1 siblings, 1 reply; 1835+ messages in thread From: Alexander G. M. Smith @ 2004-08-29 0:40 UTC (permalink / raw) To: Hans Reiser; +Cc: linux-fsdevel, linux-kernel, reiserfs-list, torvalds Linus Torvalds wrote: > I'm pretty confident that we can extend the VFS layer to support > named streams [...] Hans Reiser wrote on Sat, 28 Aug 2004 15:29:33 -0700: > I object to openat()..... > > My reason is that the things that distinguish between objects should be > the names, not the choice of system call. The reason for this is that > it improves closure and namespace unification to do so, because it > allows all the objects to be accessed within the same namespace. I have to agree with Hans there. Just have one main kind of file system object. No separate stream system, other than as an optional alternative way of viewing things. Treat it as a file or a directory or as an attribute as the context implies. Have just one open call to open it, another to enumerate its children, and another to read and write data to it. Paths to file system objects, no matter what their purpose, are all just plain paths (you can get fancy later with sub-name-spaces and name space boundaries). But how do you tell what kind of thing it is? How do you tell if it is important enough to back up? How do you tell that a thing is intended to be treated as a directory containing other things rather than as a stand-alone object? File types are needed for that! In BeOS we had a MIME type database and all important files were given an attribute of BEOS:TYPE which contained their MIME string. A system-wide database associated applications that handled each type, and also listed the standard attributes for each MIME type (such as Subject/From/To/... for "text/e-mail" files). It even included some extra info about which attributes should be shown to the user and which were editable by the user in the desktop directory window GUI. By the way, the database was just a tree of files (one for each MIME type using the same structure as MIME names) with attributes attached to them containing the relevant meta-meta-data. Unclassified files (ones with no type, such as those generated by "cp" and other non-attribute aware programs) were treated as application/octet-stream. There was also a background task that would try to identify them when the computer was idle, or if you tried to open them in a file requestor. It used the file extension and also looked for characteristic signature bytes to pick the type. Unfortunately BeOS had fragmented the type system by having a separate system of 4 character codes to identify the type of attributes (since attributes had their own API separate from files). That would tell you the attribute was an INT32 or C-String or Date. I'd like to see one unified type system for the unified file/directory/attribute thing (has anyone named those multipurpose file system objects yet or can I call them Fildirutes?). That means a global list of types and an attribute attached to all files needing a type. That standard child file (used as an attribute) would be named something like "..meta.mimetype". The global list would contain information about whether that type of thing needed to be backed up, what its standard computed attributes would be, and all sorts of common properties that all files of that type need. Plus I'd add the primitive types, so "primitive/int32" and "primitive/date64" would be possible types for things which are simple attributes. If you're worried about extra files, those file types could be stored as a unique ID (I recommend 8 character codes, using semi-readable text, for portability reasons) in each file inode. The type would only appear as a file to users of the file system. Another need for the type system is the need to index attributes. Someone has to tell the OS that "MAIL:date-received" is a 64 bit number of microseconds since the start of time, so that the OS knows how to sort the index for that attribute. Jan Harkes wrote on Sat, 28 Aug 2004 12:11:14 -0400: > From what I saw in one part of the discussion, it allows for infinite > depth recursion (file/metas/metas/metas/...). Some applications are > going to have a lot of fun with that. Some of the type information can be faked out as appearing as files. So looking for the MIME type of a MIME type file (thing/..meta.mimetype/..meta.mimetype) would pretend to be a file that contained "text/mime-string", but not actually represent anything on disk. Instead stock answers for the really primitive data types would be generated by the file system, recursively if needed. The recursion should be there. Just like it is for "." and "..", which "ls -R" for some reason doesn't traverse :-) By the way, I disagree on Hans with using a directory for the meta data, I'd prefer to just have "..meta." as the name prefix of the metadata things. Otherwise there's an extra directory level in there, making ..meta/ different from other directories (making browsing it a bit awkward since it doesn't have metadata of its own - like window coordinates for displaying the directory). - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: The Necessity of File Types (was silent semantic changes with reiser4) 2004-08-29 0:40 ` The Necessity of File Types (was silent semantic changes with reiser4) Alexander G. M. Smith @ 2004-08-31 0:42 ` David Masover 0 siblings, 0 replies; 1835+ messages in thread From: David Masover @ 2004-08-31 0:42 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Hans Reiser, linux-fsdevel, linux-kernel, reiserfs-list, torvalds -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alexander G. M. Smith wrote: [...] | By the way, I disagree on Hans with using a directory for the meta data, I'd prefer to just have "..meta." as the name prefix of the metadata things. Otherwise there's an extra directory level in there, making ..meta/ different from other directories (making browsing it a bit awkward since it doesn't have metadata of its own - like window coordinates for displaying the directory). You'd rather do "cat ..meta.pseudo" than "ls ..metas/"? And we already have weird directories. /proc doesn't do things that /tmp does. I also doubt proc will ever store that kind of metadata. I don't like either "metas" or "..metas", but you'll have to look in the archives for why. I am NOT starting that again. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTPJW3gHNmZLgCUhAQKuEw/+PjrDfSlZvuTUwiOSa/kU05qjTKkIp05F nBkrSIUKSUfoYN3gILucIGOjjJ5qBRW3wLIwWyO+raOAzHFTdylvR1rEWP9h/JUG Psz3X8RE8cubiuNwjyJawI87cPX8HW1xQxgkiiquZN2L5XbvHsFsbwAJV1tJLnrM 2lL6IBUXcxDIh7pdB0/z+4t945ElT3Rl09CLP/MfP4GNvt/UWjhbeHNWdeLq93C+ w0irn6VIsm+1+GKqEhpYLLMj9V4cbBY6m7XMuBzH+gEyu5exAMqGzPLGDiG/I6y/ rXI6IF94i2p4sLMv0LmtEvQsV37LFIKRBrWVScV5oiQ93LPVLG6pYmAi6RLsHN03 QazMmcRwjVvrTAf0KS21KXaE0qYbvgoQpteUEkPZLRkczSmiry2d3e6gBQlCV7xj 9Hzkm9HxR0BjxmRaMVLEWZ86t08OHkl0Ab186DiHUC5RlL2FPIvdyYthXo9wwtI/ XlA03Rvc3kl6DVRO8UM/+fa8v5MDn58ZMI2QgdPrU0V2Q9bGv/woByszXaDA+IVJ ZiHQ8IgShYlkA279DffOrZnDPnZMOWa7iRYetX5PYPoAyfkrmFKofT8va+dDL7pf HA5crUR5ws1EZZ8lUfrtXXnsBeEn9PS0lUHvJY7zZ4tJfZ65Fa+IiDpK5p+1tJZi ylCBNEidK/8= =CiZ9 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 22:29 ` Hans Reiser (?) (?) @ 2004-08-29 5:01 ` Linus Torvalds 2004-08-29 9:36 ` Hans Reiser ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-29 5:01 UTC (permalink / raw) To: Hans Reiser Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sat, 28 Aug 2004, Hans Reiser wrote: > > I object to openat()..... Sound slike you object to O_XATTRS, not openat() itself. Realize that openat() works independently of any special streams, it's fundamentally a "look up name starting from this file" (rather than "starting from root" or "starting from cwd"). Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds @ 2004-08-29 9:36 ` Hans Reiser 2004-08-29 10:44 ` CaT 2004-08-29 17:57 ` Linus Torvalds 2004-08-29 15:12 ` Alexander Lyamin 2004-08-29 16:20 ` viro 2 siblings, 2 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-29 9:36 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Linus Torvalds wrote: >On Sat, 28 Aug 2004, Hans Reiser wrote: > > >>I object to openat()..... >> >> > >Sound slike you object to O_XATTRS, not openat() itself. > >Realize that openat() works independently of any special streams, it's >fundamentally a "look up name starting from this file" (rather than >"starting from root" or "starting from cwd"). > > Linus > > > > well, isn't that namespace fragmentation by definition? If you can't go cat filenameA/metas/permissions > filenameB/metas/permissions find / -exec cat {}/permissions \; | grep 4777 | wc -l then aren't you missing the whole point of this exercise which is to allow the whole OS to be better integrated into a more unified namespace so that data can leap from one tool to another, and one container to another, with the greatest of ease? If cat cannot access the metadata without changing the code of cat, then all the elegance goes poof. It completely baffles me what disabling filenameA/.. does for us. Why add asymmetries? Ease of implementation is no excuse for such asymmetry. Tomorrow I am going to send a little essay I wrote this evening on these metafiles. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 9:36 ` Hans Reiser @ 2004-08-29 10:44 ` CaT 2004-08-29 21:05 ` Hubert Chan 2004-08-29 17:57 ` Linus Torvalds 1 sibling, 1 reply; 1835+ messages in thread From: CaT @ 2004-08-29 10:44 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sun, Aug 29, 2004 at 02:36:50AM -0700, Hans Reiser wrote: > Linus Torvalds wrote: > >On Sat, 28 Aug 2004, Hans Reiser wrote: > >>I object to openat()..... > > > >Sound slike you object to O_XATTRS, not openat() itself. > > > >Realize that openat() works independently of any special streams, it's > >fundamentally a "look up name starting from this file" (rather than > >"starting from root" or "starting from cwd"). > > well, isn't that namespace fragmentation by definition? If you can't go > > cat filenameA/metas/permissions > filenameB/metas/permissions > > find / -exec cat {}/permissions \; | grep 4777 | wc -l My apologies if this question has been raised (and answered) elsewhere. I tried to keep up with the flamefest/discussion but it wasn't always possible. How would you do it for directories without making certain names illegal? ie taking the above you'd do: cat dirA/metas/permissions > dirB/metas/permissions But how does that distinguish an access to the metas for dirA and dirB from an access to a file called permissions within the metas directories of dirA and dirB? I think what Linus is trying to do, as far as naming, is create a consistant access scheme that works for all types of objects in the fs without making certain names illegal (and with plugins, those names are not even a fixed thing I believe). Feel free to correct me if I'm wrong, ofcourse. :) -- Red herrings strewn hither and yon. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 10:44 ` CaT @ 2004-08-29 21:05 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-29 21:05 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "CaT" == CaT <cat@zip.com.au> writes: CaT> My apologies if this question has been raised (and answered) CaT> elsewhere. I tried to keep up with the flamefest/discussion but it CaT> wasn't always possible. CaT> How would you do it for directories without making certain names CaT> illegal? ie taking the above you'd do: CaT> cat dirA/metas/permissions > dirB/metas/permissions Yes. Directories are not allowed to contain objects with a filename "metas". The naming of "metas" has been discussed quite a bit on the Reiserfs list, and I don't think there was much consensus that came out of it. Currently for Reiser4, AFAIK, the "metas" name is a compile-time option that you can change by changing a #define, if you're worried about name clashes. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 21:05 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-29 21:05 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "CaT" == CaT <cat@zip.com.au> writes: CaT> My apologies if this question has been raised (and answered) CaT> elsewhere. I tried to keep up with the flamefest/discussion but it CaT> wasn't always possible. CaT> How would you do it for directories without making certain names CaT> illegal? ie taking the above you'd do: CaT> cat dirA/metas/permissions > dirB/metas/permissions Yes. Directories are not allowed to contain objects with a filename "metas". The naming of "metas" has been discussed quite a bit on the Reiserfs list, and I don't think there was much consensus that came out of it. Currently for Reiser4, AFAIK, the "metas" name is a compile-time option that you can change by changing a #define, if you're worried about name clashes. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 21:05 ` Hubert Chan (?) @ 2004-08-30 6:31 ` Markus Törnqvist -1 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-30 6:31 UTC (permalink / raw) To: Hubert Chan; +Cc: reiserfs-list, linux-fsdevel, linux-kernel On Sun, Aug 29, 2004 at 05:05:17PM -0400, Hubert Chan wrote: >Reiserfs list, and I don't think there was much consensus that came out It's like Gentoo users and patch sets ;) >of it. Currently for Reiser4, AFAIK, the "metas" name is a compile-time >option that you can change by changing a #define, if you're worried >about name clashes. http://mjt.nysv.org/reiser/reiser4.metas.patch -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 9:36 ` Hans Reiser 2004-08-29 10:44 ` CaT @ 2004-08-29 17:57 ` Linus Torvalds 2004-08-29 19:53 ` Hans Reiser 1 sibling, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-29 17:57 UTC (permalink / raw) To: Hans Reiser Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sun, 29 Aug 2004, Hans Reiser wrote: > > > >Realize that openat() works independently of any special streams, it's > >fundamentally a "look up name starting from this file" (rather than > >"starting from root" or "starting from cwd"). > > well, isn't that namespace fragmentation by definition? No. There's no difference between fd = open("/usr/bin/yes", O_RDWR); and dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY); fd = openat(dirfd, "yes", O_RDWR); apart from error checking and permissions.. No new namespace. Only O_XATTR implies a "namespace change". Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 17:57 ` Linus Torvalds @ 2004-08-29 19:53 ` Hans Reiser 2004-08-29 20:14 ` Nicholas Miell 2004-08-29 21:01 ` Linus Torvalds 0 siblings, 2 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-29 19:53 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Linus Torvalds wrote: >On Sun, 29 Aug 2004, Hans Reiser wrote: > > >>>Realize that openat() works independently of any special streams, it's >>>fundamentally a "look up name starting from this file" (rather than >>>"starting from root" or "starting from cwd"). >>> >>> >>well, isn't that namespace fragmentation by definition? >> >> > >No. > >There's no difference between > > fd = open("/usr/bin/yes", O_RDWR); > >and > > dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY); > fd = openat(dirfd, "yes", O_RDWR); > > The difference is that cat uses open() not openat(). With your model, can I do: cat filenameA/metas/permissions > filenameB/metas/permissions find / -exec cat {}/permissions \; | grep 4777 | wc -l If yes, then we are talking past each other somehow rather than disagreeing. If metafiles can be opened with both open and openat() in your model, then we are discussing some small detail. I think the answer is no though, in which case you are missing the point of the new design. Is the answer no? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 19:53 ` Hans Reiser @ 2004-08-29 20:14 ` Nicholas Miell 2004-08-29 21:01 ` Linus Torvalds 1 sibling, 0 replies; 1835+ messages in thread From: Nicholas Miell @ 2004-08-29 20:14 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sun, 2004-08-29 at 12:53, Hans Reiser wrote: > With your model, can I do: > > cat filenameA/metas/permissions > filenameB/metas/permissions > runat filenameA "cat permissions" | runat filenameB "cat > permissions" > find / -exec cat {}/permissions \; | grep 4777 | wc -l find / -exec runat {} "cat permissions" \; | grep 4777 | wc -l Although, whether or not the kernel exposes file permissions as an attribute named permissions is up to the filesystem. (And the wrong thing to do, in my opinion, but that's irrelevant.) > If yes, then we are talking past each other somehow rather than > disagreeing. If metafiles can be opened with both open and openat() in > your model, then we are discussing some small detail. It can be opened by both, but in order to do it via open(2), you need to fchdir(2) to the attribute directory as returned by openat(2) and then use a relative pathname. > I think the answer is no though, in which case you are missing the point > of the new design. Is the answer no? -- Nicholas Miell <nmiell@comcast.net> ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 20:14 ` Nicholas Miell 0 siblings, 0 replies; 1835+ messages in thread From: Nicholas Miell @ 2004-08-29 20:14 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sun, 2004-08-29 at 12:53, Hans Reiser wrote: > With your model, can I do: > > cat filenameA/metas/permissions > filenameB/metas/permissions > runat filenameA "cat permissions" | runat filenameB "cat > permissions" > find / -exec cat {}/permissions \; | grep 4777 | wc -l find / -exec runat {} "cat permissions" \; | grep 4777 | wc -l Although, whether or not the kernel exposes file permissions as an attribute named permissions is up to the filesystem. (And the wrong thing to do, in my opinion, but that's irrelevant.) > If yes, then we are talking past each other somehow rather than > disagreeing. If metafiles can be opened with both open and openat() in > your model, then we are discussing some small detail. It can be opened by both, but in order to do it via open(2), you need to fchdir(2) to the attribute directory as returned by openat(2) and then use a relative pathname. > I think the answer is no though, in which case you are missing the point > of the new design. Is the answer no? -- Nicholas Miell <nmiell@comcast.net> ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 20:14 ` Nicholas Miell @ 2004-08-29 21:35 ` Hubert Chan -1 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-29 21:35 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Nicholas" == Nicholas Miell <nmiell@comcast.net> writes: Nicholas> On Sun, 2004-08-29 at 12:53, Hans Reiser wrote: >> With your model, can I do: >> >> cat filenameA/metas/permissions > filenameB/metas/permissions >> Nicholas> runat filenameA "cat permissions" | runat filenameB "cat > permissions" The world is more than the commandline. Say I have an application icon stored in xterm/icon.png. Can I just open xterm/icon.png in Gimp without having to open up a new instance of Gimp, and just use the old instance that I was using for other images? Or Can I use emacs to edit xterm/description.txt without having to open up a new instance of emacs? Will scripts be able to use metafiles and normal files at the same time? -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 21:35 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-29 21:35 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Nicholas" == Nicholas Miell <nmiell@comcast.net> writes: Nicholas> On Sun, 2004-08-29 at 12:53, Hans Reiser wrote: >> With your model, can I do: >> >> cat filenameA/metas/permissions > filenameB/metas/permissions >> Nicholas> runat filenameA "cat permissions" | runat filenameB "cat > permissions" The world is more than the commandline. Say I have an application icon stored in xterm/icon.png. Can I just open xterm/icon.png in Gimp without having to open up a new instance of Gimp, and just use the old instance that I was using for other images? Or Can I use emacs to edit xterm/description.txt without having to open up a new instance of emacs? Will scripts be able to use metafiles and normal files at the same time? -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 19:53 ` Hans Reiser 2004-08-29 20:14 ` Nicholas Miell @ 2004-08-29 21:01 ` Linus Torvalds 1 sibling, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-29 21:01 UTC (permalink / raw) To: Hans Reiser Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sun, 29 Aug 2004, Hans Reiser wrote: > > The difference is that cat uses open() not openat(). Hans. Step back a bit. Stop this idiocy about thinking that this is all about you, and all about your reiser4 "metas" etc. This is about VFS interfaces, and about being generic and _useful_. Quite frankly, there is absolutely _zero_ point to make attributes just because reiser4 can implement them. Zero. Nada. Zilch. It just won't matter, because people are NOT GOING TO SWITCH TO REISER4 IN MASS. So in order to make this useful, it has to be: - stable. This means that if Al Viro asks about locking and aliasing issues, you don't ignore it, you ask "how high?" - portable. Preferably across operating systems, but at the very _least_ across filesystems. See the NFSv4 thing, for example. - useful to the people who would actually use it. Right now that's samba, and not a whole lot else. For everybody else, the pain of not having _all_ major filesystems support it is probably so insurmountable that it's just not going to happen. Do you get it? If you can't get your mind out of your little current reiser4 issues, don't even bother talking about it. Look through the thread on "possible design issues for hybrids" for me actually talking with Al on how to handle problem #1. That's one important thing, and in fact, that one is important whether we expose it to a normal "open()" call or not, since all the same aliasing issues exist _regardless_ of interfaces. "openat()" happens to be _one_ solution to #2 and #3. You haven't given any constructive input to _any_ of the problems, and you keep on raising totally irrelevant issues. Face the music, Hans. If you can't solve those three problems, whatever you do simply DOES NOT MATTER. In particular, in the two last emails to you, I tried to show that "openat()" is _independent_ of streams/attributes. That it has semantics that have absolutely _nothing_ to do with metas. And in BOTH CASES you could not wrap your brain around that fact, and kept on harping about meta's. There are bigger issues here. I'm perfectly happy to _also_ expose the streams as regular filenames, but I've tried to explain that that doesn't work for directories, for example, so YOU CANNOT -JUST- talk about the filename-only thing. You do need to have a different namespace. Full stop. What's so hard to understand about that? So stop ignoring the issues. Stop ignoring the fact that not everything revolves around how you happened to implement something. Get with the program. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds 2004-08-29 9:36 ` Hans Reiser @ 2004-08-29 15:12 ` Alexander Lyamin 2004-08-29 16:20 ` viro 2 siblings, 0 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-29 15:12 UTC (permalink / raw) To: Linus Torvalds Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Sat, Aug 28, 2004 at 10:01:23PM -0700, Linus Torvalds wrote: > > > On Sat, 28 Aug 2004, Hans Reiser wrote: > > > > I object to openat()..... > > Sound slike you object to O_XATTRS, not openat() itself. xattr deserve to be objected. quickiehackie design practice. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds 2004-08-29 9:36 ` Hans Reiser 2004-08-29 15:12 ` Alexander Lyamin @ 2004-08-29 16:20 ` viro 2 siblings, 0 replies; 1835+ messages in thread From: viro @ 2004-08-29 16:20 UTC (permalink / raw) To: Linus Torvalds Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 10:01:23PM -0700, Linus Torvalds wrote: > > > On Sat, 28 Aug 2004, Hans Reiser wrote: > > > > I object to openat()..... > > Sound slike you object to O_XATTRS, not openat() itself. > > Realize that openat() works independently of any special streams, it's > fundamentally a "look up name starting from this file" (rather than > "starting from root" or "starting from cwd"). Linus, openat() is trivial to implement in userland *IF* we are talking about stuff already in namespace. Proof: int openat(int fd, char *path, int flags, mode_t mode) { int len = snprintf(NULL, 0, "/proc/self/%d/%s", fd, path); char name[len + 1]; sprintf(name, "/proc/self/%d/%s", fd, path); return open(name, flags, mode); } 7 lines. Replace VLA with alloca() or malloc() in pre-C99 environment. In other words, there's nothing magical about syscall itself - compatibility with Solaris is not an issue here. And basic functionality (i.e. what you've described above) can be trivially achieved with no extra primitives at all. Now, if you want to use that puppy as explicit indicator of magic steps taken, fine, but that again says nothing about the primitives we want to express the magic steps in question - reproducing openat() as one of the libc functions won't be hard anyway. So I have to agree with Hans on that one - I believe that as a fundamental primitive it's wrong choice. Convenient to have in libc - sure, why not? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 22:29 ` Hans Reiser ` (2 preceding siblings ...) (?) @ 2004-08-30 15:00 ` John Stoffel 2004-08-31 0:26 ` Hubert Chan -1 siblings, 1 reply; 1835+ messages in thread From: John Stoffel @ 2004-08-30 15:00 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Hans> My reason is that the things that distinguish between objects Hans> should be the names, not the choice of system call. The reason Hans> for this is that it improves closure and namespace unification Hans> to do so, because it allows all the objects to be accessed Hans> within the same namespace. So who decides what the names mean in a specific context? How do we get consistency across applications? In the Apple and MicroSoft cases, it's a single company which sets the standards for the names of the streams and what they mean. I have no knowledge of how Sun is doing this. I don't see us setting up a single naming authority and enforcement mechanism. I went and read the Pike paper on namespaces which Hans quotes much earlier in this thread, and I think it isn't applicable to this thread. In the paper Pike compares the MVS, VMS, DOS and Unix name spaces for disk layouts. It's an interesting paper, though a bit dated since it's 20 years old. The concepts are good, but the examples aren't any more. How many Unix systems today have /n/faces/... to link email addresses to digitized images of people faces? The big point of this paper though was to make the syntax simple, and the semantics clear and unambiguous. For example, they point out how the "Cedar" file system added in VMS style version numbers to filenames with the syntax "<name>!<version>" and they pointedly ask what does "/usr/rob!3/bin/cat-v mean? Which I think just points out that the syntax of streams (metas) needs be carefully thought out, and how that syntax implies various semantics. Lots of people keep asking for ACLs, but time and time again people point out that while the Unix permission model isn't as flexible, it is _simpler_ and easier to keep track of. As an example of complexity, just keeping track of our /etc/sudoers file with all the users, hosts and commands that can be defined is a huge hassle. You pretty much need to have a 'lint' style program to do the consistency checking. Hans> Yes, it can be useful to allow a namespace to exclude some Hans> objects, but that exclusion should not be mandated. Conversely, should a namespace *require* that objects always be there? What happens when I mount an NFS filesystem onto a Resierfs4 tree? How does the user/apps determine when the semantics of the underlying namespace have changed? Hans> If you want to exclude, you should cd or chroot to Hans> /proc/nopseudos and find a view of the filesystem that excludes Hans> metas, or mount with -nopseuodos. Hans> Do you see why I say this? I can say a lot more about the Hans> damage of fragmenting namespaces into multiple apis.... Why look Hans> at xattrs....;-) I think one of the advantages of the xattrs is that the core namespace doesn't change, nor do the semantics. They are layered on top and can be explicitly looked at. A real life problem of namespaces is Network Appliance Filers. If you have a filer with both NFS and CIFS exports, it will nicely manage the CIFS ACLs in harmony (for some level you determine) with the NFS and Unix permissions. But to backup that data, and to store that information, you need to use either their 'dump' tool or NDMP. If you NFS mount the filesystem to your backup host and then use a simple tar to back it up, you lose the information since that window you use to look into the namespace is restricted. It's a gotcha. Until we can (or maybe computing in general) come up with a new namespace paradym which we can all agree to, this is going to be a big issue. John ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 15:00 ` John Stoffel @ 2004-08-31 0:26 ` Hubert Chan 2004-08-31 1:10 ` Hubert Chan 0 siblings, 1 reply; 1835+ messages in thread From: Hubert Chan @ 2004-08-31 0:26 UTC (permalink / raw) To: linux-fsdevel; +Cc: linux-kernel, reiserfs-list >>>>> "John" == John Stoffel <stoffel@lucent.com> writes: [...] John> I went and read the Pike paper on namespaces which Hans quotes John> much earlier in this thread, and I think it isn't applicable to John> this thread. In the paper Pike compares the MVS, VMS, DOS and John> Unix name spaces for disk layouts. It's an interesting paper, John> though a bit dated since it's 20 years old. The concepts are John> good, but the examples aren't any more. How many Unix systems John> today have /n/faces/... to link email addresses to digitized John> images of people faces? IIRC, gdm uses some directory in /usr/share to match usernames to pictures (along with ~<username>/<somefile>). But I can't find the directory nor remember exactly what the name is. Then there's the gconf database, which uses the filesystem to store its hierarchical structure. Or the .cddb/ database that uses filenames to match the discid to the data. Or on Debian systems, there's /var/lib/dpkg/info database that attaches (post|pre)(inst|rm) scripts, file lists, etc. to packages. (Although I would have organized that directory differently.) And then there's the fact that there would be more examples if filesystems were more expressive. Like my LDAP database that sometimes gets corrupted when the power gets pulled. It's stored in a number of opaque files, so I don't know how it gets corrupted, and I can't use the normal fsck to check it. Wouldn't it be nice if the database could be stored as normal files, using the filesystem hierarchy to mimic the LDAP hierarchy? Then if something gets corrupted, fsck will fix it, or I can easily see which record contains the garbage. The problem is that the filesystem can't do queries, so I'll have to wait until Reiser6. John> The big point of this paper though was to make the syntax simple, John> and the semantics clear and unambiguous. For example, they point John> out how the "Cedar" file system added in VMS style version numbers John> to filenames with the syntax "<name>!<version>" and they pointedly John> ask what does "/usr/rob!3/bin/cat-v mean? Which I think just John> points out that the syntax of streams (metas) needs be carefully John> thought out, and how that syntax implies various semantics. The problem isn't just with things being confusing. It's also about making things accessible through a unified interface. If we take the attributes example, if we need to use a special xattr syscall, then I can't use GIMP to edit icon attribute of xterm, or use emacs/vim/nano/ed to edit the comment attribute of sound_of_silence.ogg (or the lyrics attribute of sound_of_silence.ogg). I need to fire up some stupid program to do batch edits of the attributes of my ogg collection because the FreeDB database had the wrong CD title info for the CD that I just ripped and I didn't catch the error before I ripped it. If attributes were exposed through the regular filesystem interface (possibly through some userspace helper to do the actual OGG parsing), I could just type a couple of lines on the command line, and be done with it. John> Lots of people keep asking for ACLs, but time and time again John> people point out that while the Unix permission model isn't as John> flexible, it is _simpler_ and easier to keep track of. Not if you need to add a whole bunch of groups just to do the things you want to do. I personally haven't done that. Why? Because I would rather deal with suboptimal permissions than having to add all those groups. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 0:26 ` Hubert Chan @ 2004-08-31 1:10 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-31 1:10 UTC (permalink / raw) To: linux-fsdevel; +Cc: linux-kernel, reiserfs-list >>>>> "Hubert" == Hubert Chan <hubert@uhoreg.ca> writes: >>>>> "John" == John Stoffel <stoffel@lucent.com> writes: [...] John> The big point of this paper though was to make the syntax simple, John> and the semantics clear and unambiguous. For example, they point John> out how the "Cedar" file system added in VMS style version numbers John> to filenames with the syntax "<name>!<version>" and they pointedly John> ask what does "/usr/rob!3/bin/cat-v mean? Which I think just John> points out that the syntax of streams (metas) needs be carefully John> thought out, and how that syntax implies various semantics. Hubert> The problem isn't just with things being confusing. It's also Hubert> about making things accessible through a unified interface. ... By the way, "The Hideous Name" mentions this just briefly (in the "Properties of Name Spaces" section). A better paper, IMHO, about this is "The Use of Name Spaces in Plan 9" (Rob Pike, Dave Presotto, Ken Thompson, Howard Trickey, Phil Winterbottom; Operating Systems Review, Num. 2, Vol 27, April 1993 pp. 72-76) http://www.cs.bell-labs.com/sys/doc/names.pdf or http://www.cs.bell-labs.com/sys/doc/names.html -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-28 22:29 ` Hans Reiser @ 2004-08-28 23:47 ` Jeremy Allison 2004-08-30 13:56 ` Helge Hafting 4 siblings, 0 replies; 1835+ messages in thread From: Jeremy Allison @ 2004-08-28 23:47 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sat, Aug 28, 2004 at 11:09:38AM -0700, Linus Torvalds wrote: > > If you look at the Solaris interface, the _nice_ part about "openat()" is > that you can do something like > > file = open(filename, O_RDONLY); > if (file < 0) > return -ENOENT; > icon = openat(file, "icon", O_RDONLY | O_XATTR); > if (icon < 0) > icon = default_icon_file; > .. > > and it will work regardless of whether "filename" is a directory or a > regular file, if I've understood correctly. > > Now, I think that makes sense for several reasons: > - single case > - race-free (think "stat()" vs "fstat()" races). > - I think we want to do "openat()" regardless of whether we ever > support extended attributes or not ("openat()" is nice for doing > "namei()" in user space even in the absense of any attributes or > named streams). > > So what we can do is > - implement openat() regardless, and expect to do the Solaris thing for > it if we ever do streams. Hurrah ! Definately the best way to go as far as Samba is concerned. Makes our portability easier. Now the NT interface allows an open on a named data stream without an open on the containing file (ie. you can do CreateFile("filename::STREAMNAME",....) without doing CreateFile("filename",..." first, but I personally think those semantics are a nasty hack. It makes much more sense (IMHO) to ensure that you already have an open fd on a file before allowing access to the named streams inside, Windows be damned. If we have some semantic issues mapping from the Windows semantics to Solaris/Linux semantics then so be it, it's a Samba problem so to speak :-). Thanks, Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds @ 2004-08-30 13:56 ` Helge Hafting 2004-08-28 18:29 ` viro ` (3 subsequent siblings) 4 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-30 13:56 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Linus Torvalds wrote: >On Sat, 28 Aug 2004, Helge Hafting wrote: > > >>>I think that lack of distinguishing power is more serious for >>>directories. The more I think I think about it, the more I wonder whether >>>Solaris did things right - having a special operation to "cross the >>>boundary". >>> >>>I suspect Solaris did it that way because it's a hell of a lot easier to >>>do it like that, but regardless, it would solve the issue of real >>>directories having both real children _and_ the "extra streams". >>> >>> >>There are many ways of doing this. Several extra streams to a directory >>that aren't ordinary files in the directory? >> >> > >Well.. Yes. We already have "." and "..", which are "special extra >streams" in a sense. However, people expect them, and know to ignore them. >The same wouldn't be true of new naming. > > > >>It seems to me that we can get a lot of nice functionality in a simpler way: >>Instead of thinking about a number of streams attached to something >>that is either an ordinary file or directory, just say that the only >>change will be that a directory may have a _single_ file stream in >>addition to being a plain directory. >> >> > >That doesn't really help us. What would the name be, and how could you >avoid clashes? > > The name for the file stream and the directory would be the same, distinguished by how they're used. I.e. fopen("filedirname", "rw") gets the stream, while chdir("filedirname") changes into the directory. fopen("filedirname/substream", "rw"); opens some stream inside the directory - as usual. My idea is that the "extra streams" aren't special at all, they are simply files in a directory and will therefore work with any normal tool. cd'ing into such a directory won't be special either - because it is a plain directory. A collection of named streams is usually a bunch of files in a directory - nothing new there. Someone serving files for clients using other os'es may attach special meaning to them, but I don't see the need for a special meaning under linux. So the only new thing is that the directory have a stream of its own. Then there must be a way to turn a ordinary file into such a directory - people may then implement thumbnails, summaries and so on as substreams. Where the substreams really are plain files in this slightly special directory. Perhaps mkdir with an existing filename could succeed, creating a mixed file/directory, if the underlying fs support it. > > >>If the VFS is to be extended in order to support file-as-directory (or >>vice versa) then hopefully it can be done in a simple way. >> >> > >I'm pretty confident that we can extend the VFS layer to support named >streams (see the technical discussion with Al, rather than the flames in >this thread). I also clearly believe that it is worth it, but I'm starting >to wonder if we should have a special open flag to make people select the >stream. > >If you look at the Solaris interface, the _nice_ part about "openat()" is >that you can do something like > > file = open(filename, O_RDONLY); > if (file < 0) > return -ENOENT; > icon = openat(file, "icon", O_RDONLY | O_XATTR); > if (icon < 0) > icon = default_icon_file; > .. > >and it will work regardless of whether "filename" is a directory or a >regular file, if I've understood correctly. > > An alternative: support open("filename/icon", O_CREAT); and have this turn "filename" into file-as-directory if it wasn't so already. Then it works both with files and directories. A new flag might be needed if old programs somehow depend on the stuff above failing for plain files. >Now, I think that makes sense for several reasons: > - single case > - race-free (think "stat()" vs "fstat()" races). > - I think we want to do "openat()" regardless of whether we ever > support extended attributes or not ("openat()" is nice for doing > "namei()" in user space even in the absense of any attributes or > named streams). > >So what we can do is > - implement openat() regardless, and expect to do the Solaris thing for > it if we ever do streams. > - _also_ support the "implied named attributes" for regular files, so > that you don't have to use "openat()" to access them. > >Comments? Does anybody hate "openat()" for any reason (regardless of >attributes)? We can easily support it, we'd just need to pass in the file >to use as part of the "nameidata" thing or add an argument (it would also >possibly be cleaner if we made "fs->pwd" be a "struct file"). > > > No problem with openat() Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-30 13:56 ` Helge Hafting 0 siblings, 0 replies; 1835+ messages in thread From: Helge Hafting @ 2004-08-30 13:56 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Linus Torvalds wrote: >On Sat, 28 Aug 2004, Helge Hafting wrote: > > >>>I think that lack of distinguishing power is more serious for >>>directories. The more I think I think about it, the more I wonder whether >>>Solaris did things right - having a special operation to "cross the >>>boundary". >>> >>>I suspect Solaris did it that way because it's a hell of a lot easier to >>>do it like that, but regardless, it would solve the issue of real >>>directories having both real children _and_ the "extra streams". >>> >>> >>There are many ways of doing this. Several extra streams to a directory >>that aren't ordinary files in the directory? >> >> > >Well.. Yes. We already have "." and "..", which are "special extra >streams" in a sense. However, people expect them, and know to ignore them. >The same wouldn't be true of new naming. > > > >>It seems to me that we can get a lot of nice functionality in a simpler way: >>Instead of thinking about a number of streams attached to something >>that is either an ordinary file or directory, just say that the only >>change will be that a directory may have a _single_ file stream in >>addition to being a plain directory. >> >> > >That doesn't really help us. What would the name be, and how could you >avoid clashes? > > The name for the file stream and the directory would be the same, distinguished by how they're used. I.e. fopen("filedirname", "rw") gets the stream, while chdir("filedirname") changes into the directory. fopen("filedirname/substream", "rw"); opens some stream inside the directory - as usual. My idea is that the "extra streams" aren't special at all, they are simply files in a directory and will therefore work with any normal tool. cd'ing into such a directory won't be special either - because it is a plain directory. A collection of named streams is usually a bunch of files in a directory - nothing new there. Someone serving files for clients using other os'es may attach special meaning to them, but I don't see the need for a special meaning under linux. So the only new thing is that the directory have a stream of its own. Then there must be a way to turn a ordinary file into such a directory - people may then implement thumbnails, summaries and so on as substreams. Where the substreams really are plain files in this slightly special directory. Perhaps mkdir with an existing filename could succeed, creating a mixed file/directory, if the underlying fs support it. > > >>If the VFS is to be extended in order to support file-as-directory (or >>vice versa) then hopefully it can be done in a simple way. >> >> > >I'm pretty confident that we can extend the VFS layer to support named >streams (see the technical discussion with Al, rather than the flames in >this thread). I also clearly believe that it is worth it, but I'm starting >to wonder if we should have a special open flag to make people select the >stream. > >If you look at the Solaris interface, the _nice_ part about "openat()" is >that you can do something like > > file = open(filename, O_RDONLY); > if (file < 0) > return -ENOENT; > icon = openat(file, "icon", O_RDONLY | O_XATTR); > if (icon < 0) > icon = default_icon_file; > .. > >and it will work regardless of whether "filename" is a directory or a >regular file, if I've understood correctly. > > An alternative: support open("filename/icon", O_CREAT); and have this turn "filename" into file-as-directory if it wasn't so already. Then it works both with files and directories. A new flag might be needed if old programs somehow depend on the stuff above failing for plain files. >Now, I think that makes sense for several reasons: > - single case > - race-free (think "stat()" vs "fstat()" races). > - I think we want to do "openat()" regardless of whether we ever > support extended attributes or not ("openat()" is nice for doing > "namei()" in user space even in the absense of any attributes or > named streams). > >So what we can do is > - implement openat() regardless, and expect to do the Solaris thing for > it if we ever do streams. > - _also_ support the "implied named attributes" for regular files, so > that you don't have to use "openat()" to access them. > >Comments? Does anybody hate "openat()" for any reason (regardless of >attributes)? We can easily support it, we'd just need to pass in the file >to use as part of the "nameidata" thing or add an argument (it would also >possibly be cleaner if we made "fs->pwd" be a "struct file"). > > > No problem with openat() Helge Hafting ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 13:56 ` Helge Hafting @ 2004-08-30 23:56 ` Hubert Chan -1 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-30 23:56 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Helge" == Helge Hafting <helge.hafting@hist.no> writes: Helge> Linus Torvalds wrote: [...] >> That doesn't really help us. What would the name be, and how could you >> avoid clashes? Helge> The name for the file stream and the directory would be the same, Helge> distinguished by how they're used. I.e. fopen("filedirname", Helge> "rw") gets the stream, while chdir("filedirname") changes into Helge> the directory. fopen("filedirname/substream", "rw"); opens some Helge> stream inside the directory - as usual. The problem is that filedirname now cannot contain a file called "substream", or else it will clash with the name of the substream. One solution is to put all streams into another subdirectory, much like "metas" in Reiser4. So to access a stream named "substream", you open filedirname/streams/substream (replace "streams" with some carefully chosen name that is user-friendly and highly unlikely to produce clashes, such as "..streams", following the original "metas" proposal which has been discussed to death on the ReiserFS list. Or "pseudos/streams", or one of countless variations). -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-30 23:56 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-30 23:56 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Helge" == Helge Hafting <helge.hafting@hist.no> writes: Helge> Linus Torvalds wrote: [...] >> That doesn't really help us. What would the name be, and how could you >> avoid clashes? Helge> The name for the file stream and the directory would be the same, Helge> distinguished by how they're used. I.e. fopen("filedirname", Helge> "rw") gets the stream, while chdir("filedirname") changes into Helge> the directory. fopen("filedirname/substream", "rw"); opens some Helge> stream inside the directory - as usual. The problem is that filedirname now cannot contain a file called "substream", or else it will clash with the name of the substream. One solution is to put all streams into another subdirectory, much like "metas" in Reiser4. So to access a stream named "substream", you open filedirname/streams/substream (replace "streams" with some carefully chosen name that is user-friendly and highly unlikely to produce clashes, such as "..streams", following the original "metas" proposal which has been discussed to death on the ReiserFS list. Or "pseudos/streams", or one of countless variations). -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 15:15 ` Spam @ 2004-08-27 18:40 ` Hubert Chan -1 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-27 18:40 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Spam" == Spam <spam@tnonline.net> writes: > Rik van Riel wrote: >> On Fri, 27 Aug 2004, Hans Reiser wrote: >>> Why are you guys even considering going to any pain at all to >>> distort semantics for the sake of backup? tar is easy, we'll fix it >>> and send in a patch. >> It's not as easy as you make it out, and not just because there are a >> few dozen backup programs that need fixing. >> The problem is more fundamental than that. Some of the file streams >> proposed need to be backed up, while others are alternative >> presentations of the file, which should not be backed up. Spam> No, not really. This is a user decision and should be options in Spam> the backup software. I don't think it is up to the kernel, Spam> filesystem, or the OS in general to decide what information the Spam> user want to retain or not. Why not just define an attribute named something like "do-not-backup"? Then whatever program that generates the thumbnail can automatically add the do-not-backup bit, and the backup software knows to ignore it. (Obviously, that bit should apply recursively down the subtree.) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 18:40 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-27 18:40 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Spam" == Spam <spam@tnonline.net> writes: > Rik van Riel wrote: >> On Fri, 27 Aug 2004, Hans Reiser wrote: >>> Why are you guys even considering going to any pain at all to >>> distort semantics for the sake of backup? tar is easy, we'll fix it >>> and send in a patch. >> It's not as easy as you make it out, and not just because there are a >> few dozen backup programs that need fixing. >> The problem is more fundamental than that. Some of the file streams >> proposed need to be backed up, while others are alternative >> presentations of the file, which should not be backed up. Spam> No, not really. This is a user decision and should be options in Spam> the backup software. I don't think it is up to the kernel, Spam> filesystem, or the OS in general to decide what information the Spam> user want to retain or not. Why not just define an attribute named something like "do-not-backup"? Then whatever program that generates the thumbnail can automatically add the do-not-backup bit, and the backup software knows to ignore it. (Obviously, that bit should apply recursively down the subtree.) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:40 ` Hubert Chan (?) @ 2004-08-28 0:02 ` Spam 2004-08-28 0:32 ` Hubert Chan 2004-08-28 1:05 ` Hubert Chan -1 siblings, 2 replies; 1835+ messages in thread From: Spam @ 2004-08-28 0:02 UTC (permalink / raw) To: Hubert Chan; +Cc: reiserfs-list, linux-fsdevel, linux-kernel >>>>>> "Spam" == Spam <spam@tnonline.net> writes: >> Rik van Riel wrote: >>> On Fri, 27 Aug 2004, Hans Reiser wrote: >>>> Why are you guys even considering going to any pain at all to >>>> distort semantics for the sake of backup? tar is easy, we'll fix it >>>> and send in a patch. >>> It's not as easy as you make it out, and not just because there are a >>> few dozen backup programs that need fixing. >>> The problem is more fundamental than that. Some of the file streams >>> proposed need to be backed up, while others are alternative >>> presentations of the file, which should not be backed up. Spam>> No, not really. This is a user decision and should be options in Spam>> the backup software. I don't think it is up to the kernel, Spam>> filesystem, or the OS in general to decide what information the Spam>> user want to retain or not. > Why not just define an attribute named something like "do-not-backup"? > Then whatever program that generates the thumbnail can automatically add > the do-not-backup bit, and the backup software knows to ignore it. > (Obviously, that bit should apply recursively down the subtree.) This is about the same idea as the archive flag in DOS/Windows environments. It didn't really work to well IMO. If meta files are definable by users/application then the backup system could create its own meta files with the specific information it needs. ~S ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 0:02 ` Spam @ 2004-08-28 0:32 ` Hubert Chan 2004-08-28 1:05 ` Hubert Chan 1 sibling, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-28 0:32 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Spam" == Spam <spam@tnonline.net> writes: > Hubert Chan wrote: >> Why not just define an attribute named something like >> "do-not-backup"? Then whatever program that generates the thumbnail >> can automatically add the do-not-backup bit, and the backup software >> knows to ignore it. (Obviously, that bit should apply recursively >> down the subtree.) Spam> This is about the same idea as the archive flag in DOS/Windows Spam> environments. It didn't really work to well IMO. Similar, but not the same. IIRC, the purpose of the archive flag was to mark which files had been modified since the last backup, wherein you get into conflicts if you have several backup-like utilities. (I'm trying to forget about the DOS days. ;-) ) A do-not-backup attribute would flag that a file should never be backed up. Spam> If meta files are definable by users/application then the backup Spam> system could create its own meta files with the specific Spam> information it needs. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 0:32 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-28 0:32 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Spam" == Spam <spam@tnonline.net> writes: > Hubert Chan wrote: >> Why not just define an attribute named something like >> "do-not-backup"? Then whatever program that generates the thumbnail >> can automatically add the do-not-backup bit, and the backup software >> knows to ignore it. (Obviously, that bit should apply recursively >> down the subtree.) Spam> This is about the same idea as the archive flag in DOS/Windows Spam> environments. It didn't really work to well IMO. Similar, but not the same. IIRC, the purpose of the archive flag was to mark which files had been modified since the last backup, wherein you get into conflicts if you have several backup-like utilities. (I'm trying to forget about the DOS days. ;-) ) A do-not-backup attribute would flag that a file should never be backed up. Spam> If meta files are definable by users/application then the backup Spam> system could create its own meta files with the specific Spam> information it needs. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 0:02 ` Spam @ 2004-08-28 1:05 ` Hubert Chan 2004-08-28 1:05 ` Hubert Chan 1 sibling, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-28 1:05 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Spam" == Spam <spam@tnonline.net> writes: Spam> This is about the same idea as the archive flag in DOS/Windows Spam> environments. It didn't really work to well IMO. If meta files Spam> are definable by users/application then the backup system could Spam> create its own meta files with the specific information it needs. True enough. Sometimes you'll want to backup the thumbnail, sometimes you won't. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 1:05 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-28 1:05 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Spam" == Spam <spam@tnonline.net> writes: Spam> This is about the same idea as the archive flag in DOS/Windows Spam> environments. It didn't really work to well IMO. If meta files Spam> are definable by users/application then the backup system could Spam> create its own meta files with the specific information it needs. True enough. Sometimes you'll want to backup the thumbnail, sometimes you won't. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 14:45 ` Rik van Riel @ 2004-08-27 18:28 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 18:28 UTC (permalink / raw) To: Rik van Riel Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > >Currently I see no way to distinguish between the stuff >that should be backed up and the stuff that shouldn't. > > > We create filename/pseudos/backup, and that tells the archiver what to do..... ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 18:28 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 18:28 UTC (permalink / raw) To: Rik van Riel Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > >Currently I see no way to distinguish between the stuff >that should be backed up and the stuff that shouldn't. > > > We create filename/pseudos/backup, and that tells the archiver what to do..... ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:28 ` Hans Reiser (?) @ 2004-08-28 6:08 ` Paul Jackson 2004-08-29 15:02 ` Alexander Lyamin 2004-08-30 17:43 ` Paul Stewart -1 siblings, 2 replies; 1835+ messages in thread From: Paul Jackson @ 2004-08-28 6:08 UTC (permalink / raw) To: Hans Reiser Cc: riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Hans wrote: > We create filename/pseudos/backup, and that tells the archiver what to > do..... Instead of exposing the old semantics under a new interface, why not expose the new semantics under a new interface. There exist plenty of programs that know the old Unix semantics. There don't exist many working programs that use the new semantics that you're adding. I raise again the example of how Windows adapted to long filenames. Old DOS and FAT programs, including my Unix backups of today, see a 8.3 name space. Only code that knows the new magic sees the long names. If given the choice of breaking much old, existing stuff, or some new, mostly not yet existing stuff, does not it make more sense to break what mostly doesn't exist yet? One possible way to do this, of no doubt many: * Stealing a corner of the existing filename space for some magic names with the new semantics. * A new option on open(2), hence opendir(3), that lights up these magic names. * Doing any of the classic pathname calls with such a new magic name exposes the new semantics - such calls as: access execve mkdir mknod mount readlink rename rmdir stat truncate unlink This means essentially constructing a map between old and new, such that changes made in either view are sane and visible from the other view. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 6:08 ` Paul Jackson @ 2004-08-29 15:02 ` Alexander Lyamin 2004-08-29 18:28 ` Hans Reiser 2004-08-30 17:43 ` Paul Stewart 1 sibling, 1 reply; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-29 15:02 UTC (permalink / raw) To: Paul Jackson Cc: Hans Reiser, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Fri, Aug 27, 2004 at 11:08:57PM -0700, Paul Jackson wrote: > Hans wrote: > > We create filename/pseudos/backup, and that tells the archiver what to > > do..... > > Instead of exposing the old semantics under a new interface, why not > expose the new semantics under a new interface. yeah. that could work. > There exist plenty of programs that know the old Unix semantics. There > don't exist many working programs that use the new semantics that you're > adding. > > I raise again the example of how Windows adapted to long filenames. Old > DOS and FAT programs, including my Unix backups of today, see a 8.3 name > space. Only code that knows the new magic sees the long names. > > If given the choice of breaking much old, existing stuff, or some new, > mostly not yet existing stuff, does not it make more sense to break what > mostly doesn't exist yet? > > One possible way to do this, of no doubt many: > > * Stealing a corner of the existing filename space for > some magic names with the new semantics. > > * A new option on open(2), hence opendir(3), that lights up > these magic names. > > * Doing any of the classic pathname calls with such a > new magic name exposes the new semantics - such calls > as: > access execve mkdir mknod mount readlink > rename rmdir stat truncate unlink > > This means essentially constructing a map between old and new, > such that changes made in either view are sane and visible > from the other view. It would be intresting to hear comments from Hans Reiser on proposals stated above... -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 15:02 ` Alexander Lyamin @ 2004-08-29 18:28 ` Hans Reiser 2004-08-29 18:36 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: Hans Reiser @ 2004-08-29 18:28 UTC (permalink / raw) To: flx Cc: Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Alexander Lyamin wrote: >Fri, Aug 27, 2004 at 11:08:57PM -0700, Paul Jackson wrote: > > >> >>One possible way to do this, of no doubt many: >> >> * Stealing a corner of the existing filename space for >> some magic names with the new semantics. >> >> * A new option on open(2), hence opendir(3), that lights up >> these magic names. >> >> * Doing any of the classic pathname calls with such a >> new magic name exposes the new semantics - such calls >> as: >> access execve mkdir mknod mount readlink >> rename rmdir stat truncate unlink >> >>This means essentially constructing a map between old and new, >>such that changes made in either view are sane and visible >>from the other view. >> >> > >It would be intresting to hear comments from Hans Reiser on proposals stated above... > > > > just use a view, and skip the options on the system calls. if you cd to /nometas/your_home_directory_path you don't see the metafiles. Why is a view better than a syscall flag? Because it lets the user choose what he wants without recompiling to do it. This kind of a view requires no coding because you can just mount the root filesystem two ways, one with the -nopseudos mount option, and one without it. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 18:28 ` Hans Reiser @ 2004-08-29 18:36 ` viro 2004-08-29 18:57 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-29 18:36 UTC (permalink / raw) To: Hans Reiser Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 11:28:42AM -0700, Hans Reiser wrote: > just use a view, and skip the options on the system calls. if you cd to > /nometas/your_home_directory_path you don't see the metafiles. Why is a > view better than a syscall flag? Because it lets the user choose what > he wants without recompiling to do it. This kind of a view requires no > coding because you can just mount the root filesystem two ways, one with > the -nopseudos mount option, and one without it. *What*? OK, now I want detailed explanation of the reasons why that doesn't create cache coherency problems. Do you have an analysis of locking in the entire thing? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 18:36 ` viro @ 2004-08-29 18:57 ` viro 2004-08-29 20:06 ` Hans Reiser 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-29 18:57 UTC (permalink / raw) To: Hans Reiser Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 07:36:29PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > he wants without recompiling to do it. This kind of a view requires no > > coding because you can just mount the root filesystem two ways, one with > > the -nopseudos mount option, and one without it. > > *What*? > > OK, now I want detailed explanation of the reasons why that doesn't create > cache coherency problems. > > Do you have an analysis of locking in the entire thing? And I am very, very serious about that - we are talking about very nasty minefield and design choices in that area have fundamental impact on the entire layer, wherever it is located. It's *NOT* something that you can leave until later and hope it somehow falls into place - it can be merged in steps, but you MUST know the goal on that level. To rearchitect later might be possible (even though you will a hell of a time avoiding plugins breakage), but it will be *hard*. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 18:57 ` viro @ 2004-08-29 20:06 ` Hans Reiser 2004-08-29 21:27 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: Hans Reiser @ 2004-08-29 20:06 UTC (permalink / raw) To: viro Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list viro@parcelfarce.linux.theplanet.co.uk wrote: >On Sun, Aug 29, 2004 at 07:36:29PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > >>>he wants without recompiling to do it. This kind of a view requires no >>>coding because you can just mount the root filesystem two ways, one with >>>the -nopseudos mount option, and one without it. >>> >>> >>*What*? >> >>OK, now I want detailed explanation of the reasons why that doesn't create >>cache coherency problems. >> >>Do you have an analysis of locking in the entire thing? >> >> > >And I am very, very serious about that - we are talking about very nasty >minefield and design choices in that area have fundamental impact on the >entire layer, wherever it is located. > >It's *NOT* something that you can leave until later and hope it somehow >falls into place - it can be merged in steps, but you MUST know the goal >on that level. To rearchitect later might be possible (even though you >will a hell of a time avoiding plugins breakage), but it will be *hard*. > > > > How about if you educate me on the problems you see for a bit before I respond? I think it might help us move into a constructive discussion. You are more experienced with your code than I am.... but I firmly believe that having different views of the same filesystem in which some files are visible in one view but not in another is technically feasible for some VFS, though I'd like to learn about the problems you see for doing it in the current VFS. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 20:06 ` Hans Reiser @ 2004-08-29 21:27 ` viro 2004-08-29 21:50 ` Linus Torvalds ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: viro @ 2004-08-29 21:27 UTC (permalink / raw) To: Hans Reiser Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 01:06:41PM -0700, Hans Reiser wrote: > How about if you educate me on the problems you see for a bit before I > respond? I think it might help us move into a constructive discussion. A slew of cache coherency issues (and memory consumption on top of that). Rigth now we have a signle dentry tree for all fs instances, no matter how many times it and its subtrees are visible in the tree. Which, obviously, avoids all that crap. As soon as we start trying to have multiple trees over the same fs, we are in for a *lot* of fun. The basic question is how do you propagate the changes from one tree to another and what do you do when change to "invisible" entry happens via a tree where it is visible. Unfortunately, all obvious solutions either hit the pathname resolution and hit it *hard* (both in scalability and in price of uncontended case) or create a mess for stuff like NFS silly-rename semantics, VFAT aliases, etc. Saying that some unspecified data structures might work doesn't help. obviously - any variant will have tradeoffs of its own and has to be discussed individually. fsdevel is there for exactly that sort of stuff, so if you have any specific proposals you want to discuss, you are more than welcome. Again, right now all cache coherency issues are sidestepped by having a single instance of dentry tree per superblock. Each mount/binding is represented by vfsmount and _refers_ to a (sub)tree of dentry tree of that fs - dentries and inodes are the same (ditto for private data structures owned by filesystem, obviously). So we have a forest of dentry trees (one per filesystem) and forest of vfsmount trees (one tree per namespace). Each vfsmount corresponds to a chunk of namespace - think of the full user-visible tree cut into pieces by mountpoints. For every piece we know a) what fs it's from (->mnt_sb) b) what subtree of that fs it corresponds (->mnt_root) c) what piece it's attached to (->mnt_parent) d) where in that piece it's attached (->mnt_mountpoint) e) where to find its siblings and children f) which namespace (== tree in vfsmount forest) it's from Point in a namespace is determined by pair (vfsmount, dentry) - which piece we are looking at and where in that piece we are. Pathname lookups operate on such pairs - as long as we do not cross into another piece we just step from dentry to dentry, when we cross mountpoint towards root, we flip to (mnt_parent, mnt_mountpoint) first, when we cross mountpoint into mounted filesystem, we do hash lookup (lookup_mnt()) by our pair, find vfsmount with such mnt_parent and mnt_mountpoint and step into (vfsmount, vfsmount->mnt_root). Lifetime of vfsmounts is controlled by a simple refcount. Having vfsmount attached to tree contributes +1, so does having it pinned down by lookup/ chdir/chroot/opened file. When the last vfsmount goes away, filesystem is shut down (basically, that's what happens on umount). Binding simply creates a vfsmount pointing to target and attaches it at source. Cloning a namespace copies the vfsmount tree of parent process' namespace and flips vfsmounts of our root and cwd into the corresponding nodes in that copy. Normally, fork()/clone() just increments namespace refcount instead of cloning it, so all children share namespace with parent. When all processes in a namespace are gone its refcount goes to zero. At that point we dissolve all attachments in its vfsmount tree (which drops a reference to each vfsmount and can in turn lead to fs shutdowns). Lazy umount does similar operation with specified subtree - it detaches all pieces mounted anywhere in that subtree and drops reference to each. Ones that are not busy will be shut down immediately, ones that are will go away as soon as they stop being busy. vfsmounts can have flags of their own - one of the pending projects is to allow individual vfsmounts to be read-only. Noexec, nosuid and nodev are already per-mountpoint. Tree of vfsmounts is protected by a) vfsmount spinlock (protects hash lookups, mostly) b) per-namespace semaphore (that one is obviously blocking) c) in cases when we can go from 0 to 1 vfsmounts with given dentry as a mountpoint, we also hold ->d_inode->i_sem on the mountpoint-to-be (that closes races between rmdir/mount, rename/mount, etc.) Dentry tree is messier - these days we have dcache_lock and RCU stuff. If you can help getting the documentation out of these guys, you've got a *lot* of thanks from a lot of people. As it is, it's RTFS country. I can answer specific questions there, but I won't even try to produce readable manual covering the entire thing. Directory operations have exclusion based on ->i_sem - see Documentation/filesystems/directory-locking for description and proof of correctness. For operations that can destroy an object (unlink/rmdir/overwriting rename) we try to unhash the victim dentry first, so the filesystems that can't handle unlinked-but-busy stuff can detect such attempt by seeing the victim still hashed _and_ can be sure that if it's not hashed, nobody will come and see it (lookup would have to go to filesystem code, since we have the sucker unhashed and we have exclusion there). If operation is unsuccessful, we rehash the victim. Files involved: fs/dcache.c fs/super.c fs/inode.c fs/namespace.c include/linux/dcache.h include/linux/fs.h include/linux/mount.h If you need more details on any specific area (or gaps in the above) - just ask. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 21:27 ` viro @ 2004-08-29 21:50 ` Linus Torvalds [not found] ` <Pine.LNX.4.60.0408300009001.10533@alpha.polcom.net> 2004-08-29 23:17 ` Trond Myklebust 2004-08-30 2:18 ` Hans Reiser 2004-08-30 7:42 ` Hans Reiser 2 siblings, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-29 21:50 UTC (permalink / raw) To: viro Cc: Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sun, 29 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > A slew of cache coherency issues (and memory consumption on top of that). > Rigth now we have a signle dentry tree for all fs instances, no matter > how many times it and its subtrees are visible in the tree. Which, > obviously, avoids all that crap. As soon as we start trying to have > multiple trees over the same fs, we are in for a *lot* of fun. Al, I think you should make the argument a bit more specific, because I doubt a lot of people understand just what the problems are with aliased names. Just a few examples of the problems involved will illuminate things very well, I think. People who haven't been intimate with the name caches probably simply don't understand why you worry so much. I'll start out with some trivial examples, just to let Hans and others get an idea about what the issues are. I think examples of problems are often better ways to explain them than the abstract issues themselves. Aliases: let's say that you have filename "a" hard-linked to filename "b", and you have a directory structure of streams under there. So you have a/file1 a/dir1/file2 a/dir2/file3 and (through the hard-link with "b") you have aliases of all these same names available as "b/file1", "b/dir1/file2" etc). Now, imagine that you have two processes doing mv a/dir1 a/dir2/newdir and mv b/dir2 b/dir1/newdir at the same time. Both of them MUST NOT SUCCEED, for pretty obvious reasons (you'd have moved two directories within each other, and now neither would be accessible any more). How do you handle locking for this situation? Another interesting case is what happens when you have looked up and cache the filename "a/file1" and then another process does "rm b/file1". How do you update the _other_ cached copy, since they had two different names, but _both_ names went away at the same time. Also again, how do you handle locking? The general VFS layer has a lot of rules, and avoids these problems by simply never having aliases between two directories. If the same directory shows up multiple times (which can happen with bind mounts), they have the exact same dentry for the directory, it's just found through two different vfsmount instances. That's why vfsmounts exist - they allow the same name cache entry to show up in different places at the same time. So when we do a bind mount, and the same directory shows up under two different names "a" and "b", and we do a "rm b/file1", it _automatically_ disappears from "a/file1" too, simply by virtue of "a" and "b" literally being the same dentry. No aliasing ever happens, and this makes coherency and locking much easier (which is not to say that they are trivial, but they are pretty damn clear in comparison to the alternatives). What Al (and others) worries about is that the reiser4 name handling has _none_ of these issues figured out and protected against. You can protect against them by taking very heavy locks (you can trivially protect against all races by taking one large lock around any operation), but the fact is, that is just not an option for high-performance name lookup. These aliasing/locking rules need to be global and wll-though-out. Not just fix the two examples above, but be shown to be safe _in_general_. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
[parent not found: <Pine.LNX.4.60.0408300009001.10533@alpha.polcom.net>]
* Re: silent semantic changes with reiser4 [not found] ` <Pine.LNX.4.60.0408300009001.10533@alpha.polcom.net> @ 2004-08-29 22:37 ` Linus Torvalds 2004-08-29 23:28 ` viro ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-29 22:37 UTC (permalink / raw) To: Grzegorz Kulewski; +Cc: Kernel Mailing List [ Linux-kernel cc'd, because I don't think the question is stupid, and I can't even fully answer the kNFSd thing other than point to it as a problem. ] On Mon, 30 Aug 2004, Grzegorz Kulewski wrote: > > Sorry if my qestion is stupid, but why can't we deal with (hard)links to > directories in (nearly) same way we deal with bind mounts (= making > exactly one object representing target and only referencing to it)? On a VFS level we could, these days, I think. But realize that bind mounts and the vfsmounts are pretty recent things. We don't have any filesystems that support the notion, though, and we don't have any interfaces for the filesystem to tell us about it right now. The VFS layer could try to figure it out on its own from aliasing information, so the latter may be a non-issue, but the former is why nobody does it. And even if Linux _these days_ could handle hardlinked directories, the fact is that they would cause slightly more memory usage (due to the vfsmounts), and that nobody else can handle such filesystems - including older versions of Linux. So nobody would likely use the feature (not to mention that nobody is even really asking for it ;). And the lack of filesystem support is not theoretical. It's not easy to just retrofit directory hardlinks on a UNIX filesystem. The ".." entry actually exists on _disk_ on traditional unix filesystems, and with hardlinks on directories, that's a real problem. A hardlinked directory has multiple parents. Also, while the VFS layer no longer cares (to it, ".." is purely virtual, and it never uses it), the NFS export routines still do actually want to get the on-disk parent. A filesystem that can't do that may be unable to be exported with full semantics (ie you might get ESTALE errors after server reboots, although you'd have to ask somebody with more kNFSd knowledge than me on exactly why that is the case ;) Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 22:37 ` Linus Torvalds @ 2004-08-29 23:28 ` viro 2004-08-30 1:43 ` Neil Brown 2004-09-02 10:51 ` Stephan von Krawczynski 2 siblings, 0 replies; 1835+ messages in thread From: viro @ 2004-08-29 23:28 UTC (permalink / raw) To: Linus Torvalds; +Cc: Grzegorz Kulewski, Kernel Mailing List On Sun, Aug 29, 2004 at 03:37:16PM -0700, Linus Torvalds wrote: > > Sorry if my qestion is stupid, but why can't we deal with (hard)links to > > directories in (nearly) same way we deal with bind mounts (= making > > exactly one object representing target and only referencing to it)? > > On a VFS level we could, these days, I think. But realize that bind mounts > and the vfsmounts are pretty recent things. Bindings won't replace hardlinks. a) lifetime rules and keeping stuff busy b) who's bound on top? Note that for real hardlinks to directories (not just "directory on top of file" hybrids) it's a serious question c) for real hardlinks we would want at least rename() working ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 22:37 ` Linus Torvalds 2004-08-29 23:28 ` viro @ 2004-08-30 1:43 ` Neil Brown 2004-09-02 10:51 ` Stephan von Krawczynski 2 siblings, 0 replies; 1835+ messages in thread From: Neil Brown @ 2004-08-30 1:43 UTC (permalink / raw) To: Linus Torvalds; +Cc: Grzegorz Kulewski, Kernel Mailing List On Sunday August 29, torvalds@osdl.org wrote: > > Also, while the VFS layer no longer cares (to it, ".." is purely virtual, > and it never uses it), the NFS export routines still do actually want to > get the on-disk parent. A filesystem that can't do that may be unable to > be exported with full semantics (ie you might get ESTALE errors after > server reboots, although you'd have to ask somebody with more kNFSd > knowledge than me on exactly why that is the case ;) > The VFS requires all directories to have full paths from the filesystem root in the dcache. This is needed for loop detection when renaming directories. suppose and NFS client: (UPPERCASE words refer to file handles) Says "lookup ROOT 'a'" and gets A (file handle for /a) Says "lookup A 'b'" and gets B (file handle for /a/b) waits for /a and /a/b to be flushed from the dcache Says "RENAME ROOT 'a' to B 'z'" (i.e. asks for /a to be moved to /a/b/c) how can the VFS detect that that is not allowed? It needs to know that B is a subdirectory of /a. The complete path cannot fit in filehandle, so to needs to be computed by repeated lookup of "..", which must be on disk. Even without that, storing ".." on disk makes lots of sense for fsck-like tools. NeilBrown ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 22:37 ` Linus Torvalds 2004-08-29 23:28 ` viro 2004-08-30 1:43 ` Neil Brown @ 2004-09-02 10:51 ` Stephan von Krawczynski 2004-09-02 11:26 ` Frank van Maarseveen 2 siblings, 1 reply; 1835+ messages in thread From: Stephan von Krawczynski @ 2004-09-02 10:51 UTC (permalink / raw) To: Linus Torvalds; +Cc: kangur, linux-kernel On Sun, 29 Aug 2004 15:37:16 -0700 (PDT) Linus Torvalds <torvalds@osdl.org> wrote: > [...] > And even if Linux _these days_ could handle hardlinked directories, the > fact is that they would cause slightly more memory usage (due to the > vfsmounts), and that nobody else can handle such filesystems - including > older versions of Linux. So nobody would likely use the feature (not to > mention that nobody is even really asking for it ;). Huh? Me about a year ago ;-) Been in fact pretty much boo'd for it :-) I therefore declare as this years hot issue: How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being available for years I guess, but with 2.6 supporting lots of GIDs becoming very actual... :-))) Sorry for being a bit off-topic but shouldn't we first solve the obvious (broken) and simple issues around fs? Regards, Stephan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 10:51 ` Stephan von Krawczynski @ 2004-09-02 11:26 ` Frank van Maarseveen 2004-09-02 12:58 ` Stephan von Krawczynski 0 siblings, 1 reply; 1835+ messages in thread From: Frank van Maarseveen @ 2004-09-02 11:26 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Linus Torvalds, kangur, linux-kernel On Thu, Sep 02, 2004 at 12:51:56PM +0200, Stephan von Krawczynski wrote: > > I therefore declare as this years hot issue: > How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being ^^ The limit for NFS is 16. > available for years I guess, but with 2.6 supporting lots of GIDs becoming very > actual... thank you for reminding me that I still need to port it to 2.6. The patch came into existence around 2.2.17 IIRC. -- Frank ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 11:26 ` Frank van Maarseveen @ 2004-09-02 12:58 ` Stephan von Krawczynski 0 siblings, 0 replies; 1835+ messages in thread From: Stephan von Krawczynski @ 2004-09-02 12:58 UTC (permalink / raw) To: Frank van Maarseveen; +Cc: torvalds, kangur, linux-kernel On Thu, 2 Sep 2004 13:26:23 +0200 Frank van Maarseveen <frankvm@xs4all.nl> wrote: > On Thu, Sep 02, 2004 at 12:51:56PM +0200, Stephan von Krawczynski wrote: > > > > I therefore declare as this years hot issue: > > How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being > ^^ > The limit for NFS is 16. Yes, of course. Sorry for that exaggeration ;-) > > available for years I guess, but with 2.6 supporting lots of GIDs becoming > > very actual... > > thank you for reminding me that I still need to port it to 2.6. The patch > came into existence around 2.2.17 IIRC. Oh it will be very welcome once your done, the only way to circumvent the situation currently is to give world-rights, which is of course _bad_. Thanks in advance for this patch, Stephan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 21:50 ` Linus Torvalds @ 2004-08-29 23:17 ` Trond Myklebust 2004-08-29 23:17 ` Trond Myklebust 1 sibling, 0 replies; 1835+ messages in thread From: Trond Myklebust @ 2004-08-29 23:17 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds: > The general VFS layer has a lot of rules, and avoids these problems by > simply never having aliases between two directories. If the same directory > shows up multiple times (which can happen with bind mounts), they have the > exact same dentry for the directory, it's just found through two different > vfsmount instances. That's why vfsmounts exist - they allow the same name > cache entry to show up in different places at the same time. So could you explain what is stopping us from reducing the whole problem to the bind mount problem? IOW have "a/" be a directory that acts as if it is dynamically bind mounted on top of the file "a". Is it just the fantasy of supporting hard-links across "stream boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure nobody wants to have to add cyclic graph detection to their filesystems anyway. 8-) What other issues would need to be addressed? Cheers, Trond ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 23:17 ` Trond Myklebust 0 siblings, 0 replies; 1835+ messages in thread From: Trond Myklebust @ 2004-08-29 23:17 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds: > The general VFS layer has a lot of rules, and avoids these problems by > simply never having aliases between two directories. If the same directory > shows up multiple times (which can happen with bind mounts), they have the > exact same dentry for the directory, it's just found through two different > vfsmount instances. That's why vfsmounts exist - they allow the same name > cache entry to show up in different places at the same time. So could you explain what is stopping us from reducing the whole problem to the bind mount problem? IOW have "a/" be a directory that acts as if it is dynamically bind mounted on top of the file "a". Is it just the fantasy of supporting hard-links across "stream boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure nobody wants to have to add cyclic graph detection to their filesystems anyway. 8-) What other issues would need to be addressed? Cheers, Trond - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 23:17 ` Trond Myklebust (?) @ 2004-08-29 23:25 ` viro -1 siblings, 0 replies; 1835+ messages in thread From: viro @ 2004-08-29 23:25 UTC (permalink / raw) To: Trond Myklebust Cc: Linus Torvalds, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 07:17:10PM -0400, Trond Myklebust wrote: > På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds: > > The general VFS layer has a lot of rules, and avoids these problems by > > simply never having aliases between two directories. If the same directory > > shows up multiple times (which can happen with bind mounts), they have the > > exact same dentry for the directory, it's just found through two different > > vfsmount instances. That's why vfsmounts exist - they allow the same name > > cache entry to show up in different places at the same time. > > So could you explain what is stopping us from reducing the whole problem > to the bind mount problem? IOW have "a/" be a directory that acts as if > it is dynamically bind mounted on top of the file "a". > > > Is it just the fantasy of supporting hard-links across "stream > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure > nobody wants to have to add cyclic graph detection to their filesystems > anyway. 8-) > > What other issues would need to be addressed? Go upthread and read. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 23:17 ` Trond Myklebust (?) (?) @ 2004-08-29 23:54 ` Linus Torvalds 2004-08-30 1:42 ` Trond Myklebust -1 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-29 23:54 UTC (permalink / raw) To: Trond Myklebust Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, 29 Aug 2004, Trond Myklebust wrote: > > So could you explain what is stopping us from reducing the whole problem > to the bind mount problem? IOW have "a/" be a directory that acts as if > it is dynamically bind mounted on top of the file "a". Hey, I suggested people do exactly that. See the other thread between me and Al Viro on exact implementation issues - even some code snippets ;) The problem really ends up being directories with attributes (where we can't just overmount the existing directory). That's where "openat()" helps us. (The other problem is the purely _practical_ problem of reiser4 going behind the VFS layer, and thus _not_ getting the aliasing and locking right, but that's an implementation issue in my book, and nothing really fundamental). > Is it just the fantasy of supporting hard-links across "stream > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure > nobody wants to have to add cyclic graph detection to their filesystems > anyway. 8-) It's easy enough to do the graph detection at the VFS layer, exactly because of the density of the dentry graph. (Of course, nfs-exporting a filesystem breaks that density, thanks to the "lookup by fh" stuff, so we might not allow it for an NFS client). I suspect most filesystems wouldn't allow hard-links across "stream boundaries" in the first place. I _suspect_ that most stream implementations would bind the attributes more tightly than that to the file that owns them. reiser4 migth be the only one that might ever support something like that. > What other issues would need to be addressed? - Whether we want it in the first place - whether we need the "separate namespace" thing that O_XATTR and openat() brings us (they certainly solve the problem, but maybe it is solvable another way too) - how to actually test this out in practice (ie getting reiser4 to do the proper thing wrt the VFS layer, but preferably _also_ having another filesystem like NFSv4 or cifs that actually uses this and shows what the problems are). - whether it makes any sense at all unless we also make at least a few other filesystems support it, so that people start using it as an "expected feature" rather than a "works only on a couple of machines". And probably others. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 23:54 ` Linus Torvalds @ 2004-08-30 1:42 ` Trond Myklebust 0 siblings, 0 replies; 1835+ messages in thread From: Trond Myklebust @ 2004-08-30 1:42 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list På su , 29/08/2004 klokka 19:54, skreiv Linus Torvalds: > The problem really ends up being directories with attributes (where we > can't just overmount the existing directory). That's where "openat()" > helps us. Well, yes there has to be a distinction between a true bind mount which actually covers the file or directory, and something like the stream "bind mount" which doesn't. The stream "bind mount" is just there to allow you to root the attributes in a single tree. It can be made functionally entirely equivalent to the openat(), but uses pathname semantics (e.g., "//") to denote the attribute fork instead of an extra function call. > > Is it just the fantasy of supporting hard-links across "stream > > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure > > nobody wants to have to add cyclic graph detection to their filesystems > > anyway. 8-) > > It's easy enough to do the graph detection at the VFS layer, exactly > because of the density of the dentry graph. Don't you end up having to lock the entire paths b/c/d and a/e/f in order to prevent "ln a b/c/d/a; ln b a/e/f/b"? > > What other issues would need to be addressed? > - how to actually test this out in practice (ie getting reiser4 to do the > proper thing wrt the VFS layer, but preferably _also_ having another > filesystem like NFSv4 or cifs that actually uses this and shows what > the problems are). As I said, NFSv4 can be made ready pretty quickly: Bruce is already finishing up the xattr implementation. > - whether it makes any sense at all unless we also make at least a few > other filesystems support it, so that people start using it as an > "expected feature" rather than a "works only on a couple of machines". NTFS? ;-) Cheers, Trond ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-30 1:42 ` Trond Myklebust 0 siblings, 0 replies; 1835+ messages in thread From: Trond Myklebust @ 2004-08-30 1:42 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list På su , 29/08/2004 klokka 19:54, skreiv Linus Torvalds: > The problem really ends up being directories with attributes (where we > can't just overmount the existing directory). That's where "openat()" > helps us. Well, yes there has to be a distinction between a true bind mount which actually covers the file or directory, and something like the stream "bind mount" which doesn't. The stream "bind mount" is just there to allow you to root the attributes in a single tree. It can be made functionally entirely equivalent to the openat(), but uses pathname semantics (e.g., "//") to denote the attribute fork instead of an extra function call. > > Is it just the fantasy of supporting hard-links across "stream > > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure > > nobody wants to have to add cyclic graph detection to their filesystems > > anyway. 8-) > > It's easy enough to do the graph detection at the VFS layer, exactly > because of the density of the dentry graph. Don't you end up having to lock the entire paths b/c/d and a/e/f in order to prevent "ln a b/c/d/a; ln b a/e/f/b"? > > What other issues would need to be addressed? > - how to actually test this out in practice (ie getting reiser4 to do the > proper thing wrt the VFS layer, but preferably _also_ having another > filesystem like NFSv4 or cifs that actually uses this and shows what > the problems are). As I said, NFSv4 can be made ready pretty quickly: Bruce is already finishing up the xattr implementation. > - whether it makes any sense at all unless we also make at least a few > other filesystems support it, so that people start using it as an > "expected feature" rather than a "works only on a couple of machines". NTFS? ;-) Cheers, Trond ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 1:42 ` Trond Myklebust (?) @ 2004-08-30 2:31 ` Linus Torvalds 2004-08-30 3:01 ` viro ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-30 2:31 UTC (permalink / raw) To: Trond Myklebust Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, 29 Aug 2004, Trond Myklebust wrote: > > Well, yes there has to be a distinction between a true bind mount which > actually covers the file or directory, and something like the stream > "bind mount" which doesn't. > > The stream "bind mount" is just there to allow you to root the > attributes in a single tree. It can be made functionally entirely > equivalent to the openat(), but uses pathname semantics (e.g., "//") to > denote the attribute fork instead of an extra function call. Using '//' would be nice, but would break real apps. If I remember correctly, POSIX specifies that '//' can be special at the _beginning_ of a path, but in the middle, it has to act like a single '/'. And that's not just theory - it's quite common for programs to just concatenate a directory name (which may or may not end with a slash) with another path-name that starts with a slash. So you _will_ see existing scripts and programs using things like "/usr/include//sys/type.h", and they'd break if "//" would switch from "regular namespace" to "attribute namespace". So I don't see any way to extend pathname semantics to distinguish between "directory contents" and "directory attribute stream". > > It's easy enough to do the graph detection at the VFS layer, exactly > > because of the density of the dentry graph. > > Don't you end up having to lock the entire paths b/c/d and a/e/f in > order to prevent "ln a b/c/d/a; ln b a/e/f/b"? That's not the problem - since it's in memory, we can just get the dcache lock, and do it locked for t least local filesystems. However, being prodded by Andries, I think I'm wrong _anyway_. Since the dcache is only "dense" down one path to the root, and doesn't contain all the alternate ways of getting to a particular directory, I came to the conclusion that the VFS layer can't actually do cyclic detection after all... So together with the fact that nobody really _wants_ hardlinks to directories, I think the right answer is "no". It's not a problem as long as the attributes streams are always tied to the file/directory they are attributes of - then the "directory link" is really just a file link, and can't cause any cycles. > > - how to actually test this out in practice (ie getting reiser4 to do the > > proper thing wrt the VFS layer, but preferably _also_ having another > > filesystem like NFSv4 or cifs that actually uses this and shows what > > the problems are). > > As I said, NFSv4 can be made ready pretty quickly: Bruce is already > finishing up the xattr implementation. Do we have any servers that implement it? I think NFSv4 might be a good test-case if so. > > - whether it makes any sense at all unless we also make at least a few > > other filesystems support it, so that people start using it as an > > "expected feature" rather than a "works only on a couple of machines". > > NTFS? ;-) Hey, I see the smiley, but I'd still like to point out that not many people use it under Linux, and while I think writing to it might be stable these days, I don't believe named streams are necessarily going to materialize all that quickly.. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 2:31 ` Linus Torvalds @ 2004-08-30 3:01 ` viro 2004-08-30 3:55 ` Linus Torvalds 2004-08-30 3:12 ` Trond Myklebust 2004-08-30 5:42 ` Brad Boyer 2 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-30 3:01 UTC (permalink / raw) To: Linus Torvalds Cc: Trond Myklebust, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote: > And that's not just theory - it's quite common for programs to just > concatenate a directory name (which may or may not end with a slash) with > another path-name that starts with a slash. So you _will_ see existing > scripts and programs using things like "/usr/include//sys/type.h", and > they'd break if "//" would switch from "regular namespace" to "attribute > namespace". > > So I don't see any way to extend pathname semantics to distinguish between > "directory contents" and "directory attribute stream". I do, actually. There might be a way and it is kinda-sorta similar to your openat() variant, but lives in normal namespace. No, I'm not too fond of that, but since we are discussing weird variants anyway... a) associated directory tree of object is not automounted on top of it. Instead of that, we always do detached vfsmount (and do it on demand - see below) b) we have a bunch of pseudo-symlinks in /proc/<pid>/fd/ - same kind as what we already have there, but instead of (file->f_vfsmount, file->f_dentry) they lead to associated vfsmount (allocated if needed). Once we get a reference to such guy, we can * do further lookups * chdir there and poke around * hell, we can even bind it someplace (that will require slight change in attach_mnt() logics, but it's not hard) and get it permanently mounted Since it's not attached anywhere, normal GC logics works just fine. And yes, they are usable from scripts, etc. - exec 42<foo/bar/baz cat /proc/self/fd/#42/whatever/crap/you/want and enjoy. > However, being prodded by Andries, I think I'm wrong _anyway_. Since the > dcache is only "dense" down one path to the root, and doesn't contain all > the alternate ways of getting to a particular directory, I came to the > conclusion that the VFS layer can't actually do cyclic detection after > all... <blinks> <rereads a bunch of earlier postings> Oh, _that_'s what you meant... No, we definitely have no chance in hell to catch loops, dcache or not. It costs too much - we need to examine a *lot* of nodes to do that and all of them would have to be read at some point. We either need entire fs tree locked in core or we might have to reread it on every rename(). The former will kill us on memory use, the latter - on amount of IO. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 3:01 ` viro @ 2004-08-30 3:55 ` Linus Torvalds 2004-08-30 4:46 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-30 3:55 UTC (permalink / raw) To: viro Cc: Trond Myklebust, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Mon, 30 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > So I don't see any way to extend pathname semantics to distinguish between > > "directory contents" and "directory attribute stream". > > I do, actually. There might be a way and it is kinda-sorta similar to > your openat() variant, but lives in normal namespace. No, I'm not too > fond of that, but since we are discussing weird variants anyway... > > a) associated directory tree of object is not automounted on top > of it. Instead of that, we always do detached vfsmount (and do it on demand - > see below) > b) we have a bunch of pseudo-symlinks in /proc/<pid>/fd/ - same > kind as what we already have there, but instead of (file->f_vfsmount, > file->f_dentry) they lead to associated vfsmount (allocated if needed). > Once we get a reference to such guy, we can > * do further lookups > * chdir there and poke around > * hell, we can even bind it someplace (that will require slight change > in attach_mnt() logics, but it's not hard) and get it permanently mounted Well, the above _is_ the same as "openat()", really. It's just using a filesystem starting point to emulate a new system call. Same thing conceptually. You could do pretty much any system call as a filesystem action if you wanted to ;) I don't disagree with doing so - as a way to expose the new system call to scripts. But I don't think you're being entirely intellectually honest if you think this suddendly makes it be "one namespace". It's still a secondary namespace rooted in an entry in the normal ones - exactly like "openat()". For a non-script, a native "openat()" interface would be more efficient and less confusing, and conceptually no different from yours. No reason we couldn't have both, since they are 100% equivalent and would share the same code anyway... Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 3:55 ` Linus Torvalds @ 2004-08-30 4:46 ` viro 2004-08-30 6:33 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-30 4:46 UTC (permalink / raw) To: Linus Torvalds Cc: Trond Myklebust, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 08:55:02PM -0700, Linus Torvalds wrote: > Well, the above _is_ the same as "openat()", really. It's just using a > filesystem starting point to emulate a new system call. Same thing > conceptually. You could do pretty much any system call as a filesystem > action if you wanted to ;) Umm... Yes and no - open() is not the only thing you can do there. You can emulate some of the other stuff with open() + fchdir() + syscall, but... Note that it also gives you access to other tasks' files (provided that tasks are yours, so you can get to their /proc/.../fd). > I don't disagree with doing so - as a way to expose the new system call to > scripts. But I don't think you're being entirely intellectually honest if > you think this suddendly makes it be "one namespace". It's still a > secondary namespace rooted in an entry in the normal ones - exactly like > "openat()". Well - it *does* expose these objects to all normal syscalls. E.g. you can unlink() a component in there without fd2 = open(".", ...); fd3 = openat(fd, ".", ....) fchdir(fd3); unlink("blah"); fchdir(fd2); and similar horrors (now have fun adding locking for multithreaded process, etc.). In that sense it is, indeed, the same namespace. > For a non-script, a native "openat()" interface would be more efficient > and less confusing, and conceptually no different from yours. No reason > we couldn't have both, since they are 100% equivalent and would share the > same code anyway... I'm not sure that openat() is the right interface for e.g. fileservers. And no, I'm not saying that above is suitable for them - IMO neither variant is what they need. The thing is, fileserver almost certainly wants to create and remove objects. And you either end up with new syscalls for doing *that* relative to opened fd or you do tons of fchdir(), which makes benefits of openat() dubious in the best case. Arguments about O_NOFOLLOW on the intermediate stages are bullshit, IMNSHO - if they want to make some parts of tree inaccessible, they should simply mkdir /tmp/FOAD; chmod 0 /tmp/FOAD; mount --bind /tmp/FOAD <blocked path> in the namespace their daemon is running in. And forget all that crap about filtering pathnames and blocking symlinks on intermediate stages (the latter is obviously worthless without the former since one can simply substitute the symlink body in the pathname). ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 4:46 ` viro @ 2004-08-30 6:33 ` viro 0 siblings, 0 replies; 1835+ messages in thread From: viro @ 2004-08-30 6:33 UTC (permalink / raw) To: Linus Torvalds Cc: Trond Myklebust, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Mon, Aug 30, 2004 at 05:46:37AM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > Arguments about O_NOFOLLOW on the intermediate stages are bullshit, IMNSHO - > if they want to make some parts of tree inaccessible, they should simply > mkdir /tmp/FOAD; chmod 0 /tmp/FOAD; mount --bind /tmp/FOAD <blocked path> > in the namespace their daemon is running in. And forget all that crap > about filtering pathnames and blocking symlinks on intermediate stages > (the latter is obviously worthless without the former since one can simply > substitute the symlink body in the pathname). Ehh... After looking at that for a while... No, it's not that simple and removing the stuff that way won't do what these guys want, at least not without something else. Frankly, what I've seen worries me a lot - it looks like there is a missing primitive here that would be saner than this sort of filtering. It appears that most of this stuff would be covered by a fast way to tell if the resulting object belongs to given subtree. That could be arranged (not without some changes, but doable), but I'm not sure that it's enough to cover the stuff they are really trying to do. It does look like an interesting problem and current solutions certainly suck. And I very much doubt that "do a lookup if it doesn't run into anything that could be too tricky for our pathname-based checks, otherwise let's do it step-by-step from userland" is the right approach here. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 2:31 ` Linus Torvalds @ 2004-08-30 3:12 ` Trond Myklebust 2004-08-30 3:12 ` Trond Myklebust 2004-08-30 5:42 ` Brad Boyer 2 siblings, 0 replies; 1835+ messages in thread From: Trond Myklebust @ 2004-08-30 3:12 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list På su , 29/08/2004 klokka 22:31, skreiv Linus Torvalds: > So together with the fact that nobody really _wants_ hardlinks to > directories, I think the right answer is "no". It's not a problem as > long as the attributes streams are always tied to the file/directory > they are attributes of - then the "directory link" is really just a > file link, and can't cause any cycles. Right. > Do we have any servers that implement it? I think NFSv4 might be a good > test-case if so. I believe Solaris Express already has an implementation, so I should have at least one server implementation available in the basement. So probably does Hummingbird, since their NFSv4 server runs on Windows. CITI is working on the Linux code, but I'm not sure how far they've come on the server side. I'll ask Bruce in the morning... Cheers, Trond ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-30 3:12 ` Trond Myklebust 0 siblings, 0 replies; 1835+ messages in thread From: Trond Myklebust @ 2004-08-30 3:12 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list På su , 29/08/2004 klokka 22:31, skreiv Linus Torvalds: > So together with the fact that nobody really _wants_ hardlinks to > directories, I think the right answer is "no". It's not a problem as > long as the attributes streams are always tied to the file/directory > they are attributes of - then the "directory link" is really just a > file link, and can't cause any cycles. Right. > Do we have any servers that implement it? I think NFSv4 might be a good > test-case if so. I believe Solaris Express already has an implementation, so I should have at least one server implementation available in the basement. So probably does Hummingbird, since their NFSv4 server runs on Windows. CITI is working on the Linux code, but I'm not sure how far they've come on the server side. I'll ask Bruce in the morning... Cheers, Trond - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 2:31 ` Linus Torvalds 2004-08-30 3:01 ` viro 2004-08-30 3:12 ` Trond Myklebust @ 2004-08-30 5:42 ` Brad Boyer 2 siblings, 0 replies; 1835+ messages in thread From: Brad Boyer @ 2004-08-30 5:42 UTC (permalink / raw) To: Linus Torvalds Cc: Trond Myklebust, Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote: > On Sun, 29 Aug 2004, Trond Myklebust wrote: > > > - how to actually test this out in practice (ie getting reiser4 to do the > > > proper thing wrt the VFS layer, but preferably _also_ having another > > > filesystem like NFSv4 or cifs that actually uses this and shows what > > > the problems are). > > > > As I said, NFSv4 can be made ready pretty quickly: Bruce is already > > finishing up the xattr implementation. > > Do we have any servers that implement it? I think NFSv4 might be a good > test-case if so. I know the mention of it in the past has brought up ridicule, but netatalk requires some level of multi-stream support in the export to the client, and currently just emulates it on the server side by hiding the extra files in various directories. It should be possible to modify it to use named streams/attributes for the extra data. If that happens, it would be nice if hfs and hfsplus could export the data so it could be used. The view exported by netatalk does match the internal structure of hfs, after all. (For those of you that aren't familiar with the Macintosh, a Mac style filesystem has two peer streams, data and resource. They are each a seekable bytestream, and are actually at the same level inside the filesystem. In addition, there is a large set of fixed length attributes stored, such as the file type info and icon position. In addition, hfsplus also supports full named streams and arbitrary metadata, tho that isn't used much.) Brad Boyer flar@allandria.com ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 23:17 ` Trond Myklebust ` (2 preceding siblings ...) (?) @ 2004-08-30 0:07 ` Alexander G. M. Smith -1 siblings, 0 replies; 1835+ messages in thread From: Alexander G. M. Smith @ 2004-08-30 0:07 UTC (permalink / raw) To: Trond Myklebust Cc: viro, reiser, flx, pj, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, torvalds Trond Myklebust wrote on Sun, 29 Aug 2004 19:17:10 -0400: > Is it just the fantasy of supporting hard-links across "stream > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure > nobody wants to have to add cyclic graph detection to their filesystems > anyway. 8-) Been there, done that. It works in a rough form in an experimental RAM file system for BeOS. But deleting files requires a graph traversal to see if it is isolating just one item or a whole bunch of them. It also implies keeping track of multiple parent directories for everything. It does make it useful for the user - you can have files and directories in multiple places, so you don't have to decide on one classification for a file (or directory). One other downside is a lot of locking complexity (or not if you don't need fine grained locking) to guarantee atomicity of the delete. For user convenience, the low level delete function returns an error code if there is more than one item cut off from the root, but it could just as well delete all of them (sort of like rmdir giving an error if the directory isn't empty rather than just deleting everything). - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 21:27 ` viro 2004-08-29 21:50 ` Linus Torvalds @ 2004-08-30 2:18 ` Hans Reiser 2004-08-30 7:42 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-30 2:18 UTC (permalink / raw) To: viro Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list You are assuming that the hiding has to be embodied by the dentry cache. That sounds hard. Let's not do that; I am too lazy. We can put it at a layer above that. Let us have a "mask" concept. For the purposes of this discussion, the mask can be made totally trivial and hardcoded, with none of the expressive power of my new DoD project, and can probably be coded in a few hours to exclude the metas. For my new DoD project we are creating a configurable, scalable, sophisticated "mask" abstraction, and you must successfully pass through the mask before you can ask if the filesystem contains X/Y/Z. (For the darpa project with the general purpose masks with specifications and all that we will take 6 months, not a few hours.) The rest shall wait until after I get a bite to eat, ok? Hans viro@parcelfarce.linux.theplanet.co.uk wrote: >On Sun, Aug 29, 2004 at 01:06:41PM -0700, Hans Reiser wrote: > > > >>How about if you educate me on the problems you see for a bit before I >>respond? I think it might help us move into a constructive discussion. >> >> > >A slew of cache coherency issues (and memory consumption on top of that). >Rigth now we have a signle dentry tree for all fs instances, no matter >how many times it and its subtrees are visible in the tree. Which, >obviously, avoids all that crap. As soon as we start trying to have >multiple trees over the same fs, we are in for a *lot* of fun. > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 21:27 ` viro 2004-08-29 21:50 ` Linus Torvalds 2004-08-30 2:18 ` Hans Reiser @ 2004-08-30 7:42 ` Hans Reiser 2004-08-30 13:17 ` Alexander G. M. Smith 2 siblings, 1 reply; 1835+ messages in thread From: Hans Reiser @ 2004-08-30 7:42 UTC (permalink / raw) To: viro Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list I think we have to either: 1) Link only to the file and not the directory in the file-directory duality. This means we change the semantics of hard link so that it only links to the data and the standard metadata, and the optional attributes/streams/files-in-directory are not seen by the second link. You can think of this as meaning that hard links only connect to the file and not the directory in the file-directory wave-particle duality or, 2) we should ask Alexander Smith to help with applying the graph traversal cycle detection code that he wrote. I can go either way contentedly for now. 2) is the right long term solution. 1) is probably the right short term solution. Ok, Linus and Viro, now I see why it was hard. Being able to effectively connect to compound documents only with symlinks is a bit distasteful, but it is quite livable, and I very much hope you decide it is better than fragmenting the namespace. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 7:42 ` Hans Reiser @ 2004-08-30 13:17 ` Alexander G. M. Smith 0 siblings, 0 replies; 1835+ messages in thread From: Alexander G. M. Smith @ 2004-08-30 13:17 UTC (permalink / raw) To: Hans Reiser Cc: flx, pj, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, viro Hans Reiser wrote on Mon, 30 Aug 2004 00:42:13 -0700: > or, 2) we should ask Alexander Smith to help with applying the graph > traversal cycle detection code that he wrote. I put my foot into it, didn't I? It's not too bad, one trick to simplify things is to have the concept of a main parent and step parents. The file system name space graph is kept structured so that by always going upward through the main parent, you're guaranteed to reach the root. It doesn't save anything on traversal when deleting files (you still have to visit all the relevant nodes to check for cycles), but allows you to replace hard links with a new kind of dynamic symbolic link. The hard link pretends to be a symbolic link presenting the "true path" to its target, which changes as the target moves around (so it's not as reliable as real hard link). Old software doesn't see the cycles in the graph, just follows the single true path to any particular object like it currently does, and sees a bunch of symbolic links to everything else. New software can see the full parent list (not just a single "..") and the actual hard links. > Ok, Linus and Viro, now I see why it was hard. Being able to effectively > connect to compound documents only with symlinks is a bit distasteful, > but it is quite livable, and I very much hope you decide it is better > than fragmenting the namespace. Still, having hard links to attributes or directories of attributes is useful. For example, if several contact manager entries reference the same person, it would be nice to have that represented as a hard link to the attribute directory tree describing that person. That's assuming you implement contacts (name, address, etc) as a bunch of attributes in a directory. - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 6:08 ` Paul Jackson 2004-08-29 15:02 ` Alexander Lyamin @ 2004-08-30 17:43 ` Paul Stewart 2004-08-30 17:58 ` Linus Torvalds 1 sibling, 1 reply; 1835+ messages in thread From: Paul Stewart @ 2004-08-30 17:43 UTC (permalink / raw) To: Paul Jackson Cc: Hans Reiser, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On 2004.08.27 23:08, Paul Jackson wrote: > Hans wrote: > > We create filename/pseudos/backup, and that tells the archiver what > > to do..... > > Instead of exposing the old semantics under a new interface, why not > expose the new semantics under a new interface. Here's another take on the same theme. To see attrs on files, one can either use a newly developed application which can use special new syscalls/flags on syscalls a Paul Jackson recommends. However from an old shell or application one can also open the attribute node on /home/myself/foo.txt by checking out /attr/home/myself/foo.txt/, which points to the "as directory" node on the filesystem that foo.txt points to. The strange part of this idea is that the /attr filesystem wouldn't be conventionally browsable. An opendir() on any path would return the attributes, not the subdirectory contents, even if the target was a conventional directory. File based operations (open(), unlink(), etc) would operate on the "as directory" node of the dirname() of the file argument. The /attr filesystem would be completely virtual, honoring the permissions, crossing filesystems, and inheriting the root directory of the calling process. Just a thought from a Linux fs layman -- certainly one of the Reiser unwashed masses. -- Paul ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 17:43 ` Paul Stewart @ 2004-08-30 17:58 ` Linus Torvalds 0 siblings, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-30 17:58 UTC (permalink / raw) To: Paul Stewart Cc: Paul Jackson, Hans Reiser, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Mon, 30 Aug 2004, Paul Stewart wrote: > > Here's another take on the same theme. To see attrs on files, one can > either use a newly developed application which can use special new > syscalls/flags on syscalls a Paul Jackson recommends. However from an > old shell or application one can also open the attribute node on > /home/myself/foo.txt by checking out /attr/home/myself/foo.txt/, which > points to the "as directory" node on the filesystem that foo.txt points > to. This is the same idea as Al Viro's /proc/self/fd/#42/attr issue, except yours has two fundamental problem: races and ambiguities. If you open a filename in some "secondary" tree (be it /proc or //attr or whatever) based on the filename in the primary one, you have two issues that you need to work out: - how do you handle a name change in the primary tree at the same time as lookup - how do you handle the ambiguity of //attr/usr/bin/emacs/icon (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin"). The ambiguity can be handled by saying that attributes only have one component (ie only the _last_ component of a lookup is the attribute name). But the race between primary tree and secondary tree cannot be handled in a normal name-space. What Al did was to avoid both by "fixing" the attribute lookup point with another open - _exactly_ the same way "openat()" handles it. So Al's naming convention avoids both the ambiguity and the primary tree name change races by first opening the primary tree file, and then explicitly using that file as the "anchor" in the secondary tree. He did it in /proc, where we obviously already do export an open fd as an anchor-point. > The strange part of this idea is that the /attr filesystem wouldn't be > conventionally browsable. That may make it non-intuitive to use, but that's not the real problem. See above. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:28 ` Hans Reiser (?) (?) @ 2004-08-29 15:00 ` Alexander Lyamin 2004-08-31 0:46 ` David Masover -1 siblings, 1 reply; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-29 15:00 UTC (permalink / raw) To: Hans Reiser Cc: Rik van Riel, David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Fri, Aug 27, 2004 at 11:28:51AM -0700, Hans Reiser wrote: > Rik van Riel wrote: > > > > > > >Currently I see no way to distinguish between the stuff > >that should be backed up and the stuff that shouldn't. > > > > > > > We create filename/pseudos/backup, and that tells the archiver what to > do..... its ugly idea. representations should be distinguished from sources. and its reasonably to put this distinguisher somewhere in namespace. but filename/pseudos/backup is plain ugly. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 15:00 ` Alexander Lyamin @ 2004-08-31 0:46 ` David Masover 2004-08-31 6:47 ` Hans Reiser 0 siblings, 1 reply; 1835+ messages in thread From: David Masover @ 2004-08-31 0:46 UTC (permalink / raw) To: flx Cc: Hans Reiser, Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alexander Lyamin wrote: | Fri, Aug 27, 2004 at 11:28:51AM -0700, Hans Reiser wrote: | |>Rik van Riel wrote: |> |> |>> |>>Currently I see no way to distinguish between the stuff |>>that should be backed up and the stuff that shouldn't. |>> |>> |>> |> |>We create filename/pseudos/backup, and that tells the archiver what to |>do..... | | | its ugly idea. | | representations should be distinguished from sources. and its reasonably | to put this distinguisher somewhere in namespace. but filename/pseudos/backup | is plain ugly. How about filename/metas/serialized, or filename/metas/fschunk? And I did mention before about how it'd be nice for it to be an fs chunk... Imagine 5000 tiny little files that, by the time reiser's done with them, fit inside 5 blocks or so. It's insanely cheaper to copy the five blocks than to tar them up. It also means that we don't have to worry about supporting tar on other platforms -- if they don't have reiser4, they need a separate converter. If apps are going to use reiser4 metadata for anything significant, they probably would link against some sort of userland implementation of reiser4 as a Windows library, say, to access the files elsewhere. That's extreme, but you get the idea. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTPKc3gHNmZLgCUhAQJ5xw//T+kLDCHlfQEaGM9XAeFDZU9f4m4PWjet n2xODVIxaT7WmnZ4nQ3rRTiJjsXvRBcPdWUTFRZl6NbLqa/K6ESpUJAZsZynRhXd NLJ9gEpeqcUrXBLrmTD5AbzENOLdqDIo2qQ+hde68H5vo+naa5xtTyrMiHNHQWbf 9P45WKXYt36madg4KjAhZcYB+T843fc/ItjLQBTwxBNOykBcZ82UUNwGeGbiqczm uXftCD6BNtKsXnJxu2jVc8CK93lOw4SIf7BY09PFXgK2XTpAsKzOwwqpBtdxSzqD lkz+nGEeYZk6GRA3N1OTbPJnboee7WHZzMyvXDMBsExfR6puV1SYA2NWhj9uGDmS nYNZwdN+RimiouD1D9DD+oCQe/Gr66THicex5FM37cMBdaDf0jGblvpDdQRfkhcX 2KGLCt1ax620ia2945/+/+6iQqM7+oTQsVNziYjVGQ4gCNRzZh8SBtM0yUZuecNO H84fWYLICGos+Qk7FyCuQOquTSVRWKadkiDaOSre2hFk+S9VmDWkghJSNGrnqoKi pFYFrUzL+NXPHs6G8b8EKAaQQt+Z8iHE6xgdc+yS56Pdrq8apRzeQv2zhgZaRq9J kCRgvYo40xFMX4B+EqexxIiwc6nPUD5dFbt0OnJMEk4pduFq64lMVdoQA8Qp4X8a 0feuVuaWHr0= =ymeH -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 0:46 ` David Masover @ 2004-08-31 6:47 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-31 6:47 UTC (permalink / raw) To: David Masover Cc: flx, Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list David Masover wrote: > > > And I did mention before about how it'd be nice for it to be an fs > chunk... Please define fs chunk. My mind has an open door.;-) > > Imagine 5000 tiny little files that, by the time reiser's done with > them, fit inside 5 blocks or so. It's insanely cheaper to copy the five > blocks than to tar them up. It also means that we don't have to worry > about supporting tar on other platforms -- if they don't have reiser4, > they need a separate converter. If apps are going to use reiser4 > metadata for anything significant, they probably would link against some > sort of userland implementation of reiser4 as a Windows library, say, to > access the files elsewhere. > > That's extreme, but you get the idea. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-31 6:47 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-31 6:47 UTC (permalink / raw) To: David Masover Cc: flx, Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list David Masover wrote: > > > And I did mention before about how it'd be nice for it to be an fs > chunk... Please define fs chunk. My mind has an open door.;-) > > Imagine 5000 tiny little files that, by the time reiser's done with > them, fit inside 5 blocks or so. It's insanely cheaper to copy the five > blocks than to tar them up. It also means that we don't have to worry > about supporting tar on other platforms -- if they don't have reiser4, > they need a separate converter. If apps are going to use reiser4 > metadata for anything significant, they probably would link against some > sort of userland implementation of reiser4 as a Windows library, say, to > access the files elsewhere. > > That's extreme, but you get the idea. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 6:47 ` Hans Reiser @ 2004-09-01 2:04 ` David Masover -1 siblings, 0 replies; 1835+ messages in thread From: David Masover @ 2004-09-01 2:04 UTC (permalink / raw) To: Hans Reiser Cc: flx, Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hans Reiser wrote: | David Masover wrote: | |> |> |> And I did mention before about how it'd be nice for it to be an fs |> chunk... | | | Please define fs chunk. My mind has an open door.;-) I think this is sort of like dump/restore, but for a subset of the tree. ~ So it's faster for all the reasons dump/restore are faster than tar for a whole filesystem. But I don't know all that much about either dump/restore or reiser4 format plugins. I'm defining an "fs chunk" as a complete, read-only reiser4 filesystem of the data in question. As a snapshot. I'm imagining it being faster to create and ultimately smaller and more intelligently packed than a tarball. What if you could get a blocklist for a directory? Of course some things would have to be changed to completely divorce it from the parent filesystem and have it still be usable. I'm imagining that literally grabbing a chunk of the filesystem -- using its own storage layer -- would be faster than tarring. Think like if directories had blocklists as well as files. It would also be smaller - -- see below. And it completely eliminates the question of "what may I never back up" -- the fs already knows that. Without having to be told by plugins. If the fs stores it to disk, the backup stores it, unless it's part of the "cache" proposal. This is a rough idea, and has issues. For instance, how do we know what's cache and what's not? If you have foo.gz/ugz cached and someone writes to foo.gz/ugz, I think that should be written to the cache and the original foo.gz should be removed -- regenerated if necessary from foo.gz/ugz. But the reverse is true -- write to foo.gz and foo.gz/ugz should go away, to be regenerated if necessary. Only one should go into the backup, though. I think foo.gz goes into the backup and is regenerated from foo.gz/ugz if necessary. More generally, when you have two redundant streams, backup the smaller one. |> |> Imagine 5000 tiny little files that, by the time reiser's done with |> them, fit inside 5 blocks or so. It's insanely cheaper to copy the five |> blocks than to tar them up. It also means that we don't have to worry |> about supporting tar on other platforms -- if they don't have reiser4, |> they need a separate converter. If apps are going to use reiser4 |> metadata for anything significant, they probably would link against some |> sort of userland implementation of reiser4 as a Windows library, say, to |> access the files elsewhere. |> |> That's extreme, but you get the idea. | | P.S. Sorry for the dupe, Hans, accidently hit "reply" instead of "reply-to-all"... -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUuS3gHNmZLgCUhAQKWCQ/+LTNqHvndRAMqzdAJ0rugZl30bqqPmrFs vNqSwEx09RUxN6caOd+C4F7oVj/as0AJrNqyN5YR6/t6BKGxh+xEDnKdKiIRR5qp /TJmsMSkh54MSqaZZPL5EB6/DjWscBN7zKCaaCN6f8xB6KU2BCOK5pNSHDvYhISA uyD8AXrsOLRGC82dNtmh68KdrG+7VcTZMohulXNnosLLRJSuKeYEphB0i5HrvXQn nZ010yb6LnmKtNKD+w7bFxt3ul+lQo8gti7zBJqb7mV4pa1MnQNhGZRgvey/Pq/l ahXrrABauUFNwM4PfFWUoY9zjOdCufQpWev/mNGuoo9BS2ZiRg8LXoUpf2bOvi6I yOTMRb37w6+w37Jv8B0vYPehW3bmuJpb+jDyZ5AYS0GygJmm3bEIFZYupYt2pU/v 7EBIzLszCEnqPbm0glGdWT5jyYiZGucvDK+W4qQ3ymEqaBIdGIti4G56mQ7bTQer DPVT4dv38OE/PEtKb6xLQUaos7hBg+sWM5+mT9B+7kX/cYDhtdBkZpYqf10jAg34 4ufl2NS48CxqCcml7RayPVAuKT1ZN9sDcfXVYnPw33R94/lziZnfJrIInbnf9MZK +P9v6WlFi6+9GZjwqFK9ZCJBoTCKRq1lGiy5flTDf1APAgpmU6CS/0Da4s1QiLLI YRoeX8Nv8Us= =WFSh -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-01 2:04 ` David Masover 0 siblings, 0 replies; 1835+ messages in thread From: David Masover @ 2004-09-01 2:04 UTC (permalink / raw) To: Hans Reiser Cc: flx, Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hans Reiser wrote: | David Masover wrote: | |> |> |> And I did mention before about how it'd be nice for it to be an fs |> chunk... | | | Please define fs chunk. My mind has an open door.;-) I think this is sort of like dump/restore, but for a subset of the tree. ~ So it's faster for all the reasons dump/restore are faster than tar for a whole filesystem. But I don't know all that much about either dump/restore or reiser4 format plugins. I'm defining an "fs chunk" as a complete, read-only reiser4 filesystem of the data in question. As a snapshot. I'm imagining it being faster to create and ultimately smaller and more intelligently packed than a tarball. What if you could get a blocklist for a directory? Of course some things would have to be changed to completely divorce it from the parent filesystem and have it still be usable. I'm imagining that literally grabbing a chunk of the filesystem -- using its own storage layer -- would be faster than tarring. Think like if directories had blocklists as well as files. It would also be smaller - -- see below. And it completely eliminates the question of "what may I never back up" -- the fs already knows that. Without having to be told by plugins. If the fs stores it to disk, the backup stores it, unless it's part of the "cache" proposal. This is a rough idea, and has issues. For instance, how do we know what's cache and what's not? If you have foo.gz/ugz cached and someone writes to foo.gz/ugz, I think that should be written to the cache and the original foo.gz should be removed -- regenerated if necessary from foo.gz/ugz. But the reverse is true -- write to foo.gz and foo.gz/ugz should go away, to be regenerated if necessary. Only one should go into the backup, though. I think foo.gz goes into the backup and is regenerated from foo.gz/ugz if necessary. More generally, when you have two redundant streams, backup the smaller one. |> |> Imagine 5000 tiny little files that, by the time reiser's done with |> them, fit inside 5 blocks or so. It's insanely cheaper to copy the five |> blocks than to tar them up. It also means that we don't have to worry |> about supporting tar on other platforms -- if they don't have reiser4, |> they need a separate converter. If apps are going to use reiser4 |> metadata for anything significant, they probably would link against some |> sort of userland implementation of reiser4 as a Windows library, say, to |> access the files elsewhere. |> |> That's extreme, but you get the idea. | | P.S. Sorry for the dupe, Hans, accidently hit "reply" instead of "reply-to-all"... -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUuS3gHNmZLgCUhAQKWCQ/+LTNqHvndRAMqzdAJ0rugZl30bqqPmrFs vNqSwEx09RUxN6caOd+C4F7oVj/as0AJrNqyN5YR6/t6BKGxh+xEDnKdKiIRR5qp /TJmsMSkh54MSqaZZPL5EB6/DjWscBN7zKCaaCN6f8xB6KU2BCOK5pNSHDvYhISA uyD8AXrsOLRGC82dNtmh68KdrG+7VcTZMohulXNnosLLRJSuKeYEphB0i5HrvXQn nZ010yb6LnmKtNKD+w7bFxt3ul+lQo8gti7zBJqb7mV4pa1MnQNhGZRgvey/Pq/l ahXrrABauUFNwM4PfFWUoY9zjOdCufQpWev/mNGuoo9BS2ZiRg8LXoUpf2bOvi6I yOTMRb37w6+w37Jv8B0vYPehW3bmuJpb+jDyZ5AYS0GygJmm3bEIFZYupYt2pU/v 7EBIzLszCEnqPbm0glGdWT5jyYiZGucvDK+W4qQ3ymEqaBIdGIti4G56mQ7bTQer DPVT4dv38OE/PEtKb6xLQUaos7hBg+sWM5+mT9B+7kX/cYDhtdBkZpYqf10jAg34 4ufl2NS48CxqCcml7RayPVAuKT1ZN9sDcfXVYnPw33R94/lziZnfJrIInbnf9MZK +P9v6WlFi6+9GZjwqFK9ZCJBoTCKRq1lGiy5flTDf1APAgpmU6CS/0Da4s1QiLLI YRoeX8Nv8Us= =WFSh -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:15 ` Linus Torvalds 2004-08-26 18:41 ` Rik van Riel 2004-08-26 19:02 ` David Masover @ 2004-08-26 20:44 ` Martin J. Bligh 2004-08-26 20:54 ` Linus Torvalds 2 siblings, 1 reply; 1835+ messages in thread From: Martin J. Bligh @ 2004-08-26 20:44 UTC (permalink / raw) To: Linus Torvalds, Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list --On Thursday, August 26, 2004 11:15:19 -0700 Linus Torvalds <torvalds@osdl.org> wrote: > On Thu, 26 Aug 2004, Rik van Riel wrote: >> >> It's a relief to know that nobody's taking my humorous >> suggestion seriously, but now we still have the "standard >> Unix tools can't manipulate files" problem... > > I disagree. They can manipulate the files a whole lot better than they can > manipulate xattr's. > > For example, you _could_ probably (but hey, maybe "tar" tries to strip > slashes off the end of filenames, so this might not work due to silly > reasons like that) back up a compound file with > > tar cvf file.tar file file/ > > although unpacking it would require that tar be taught about the thing. > And you definitely could write a script to do the thing, ie even with an > unmodified tar you could do > > tar cvf file-archive.tar file > cd file > tar cvf ../attribute-archive.tar . > > which is a hell of a lot better than what you can do with the fsattr > interfaces and unmodified legacy applications. > > So one of the advantages of "dir-as-file/file-as-dir" is exactly that you > _can_ manipulate the data with legacy tools. Sure, things that traverse a > directory tree might need some (likely fairly trivial) modifications if > they really want to take advantage of the subfiles, but that's still > likely to be _much_ less of an issue than with fsattr's that have a > totally different model entirely. What would "test -d" and "test -f" return on these magic beasties? I can't think of any combinations that wouldn't confuse the crap out of userspace. M. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:44 ` Martin J. Bligh @ 2004-08-26 20:54 ` Linus Torvalds 2004-08-26 21:00 ` Martin J. Bligh ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 20:54 UTC (permalink / raw) To: Martin J. Bligh Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Martin J. Bligh wrote: > > What would "test -d" and "test -f" return on these magic beasties? I can't > think of any combinations that wouldn't confuse the crap out of userspace. "It's a feature". The S_ISDIR/S_ISREG tests show real information: it shows not only user intent ("you should consider this a file, even if it has attributes"), but also whether it is a directory or a container. And there's a real technical difference there: the streams contained within a file are bound to that file. The files contained within a directory are _independent_ of that directory. Big difference. HUGE difference. So it's not confusing. If it tests as a file, you think of it as a file. It may have attributes aka named streams associated with it, and you may be able to open those attributes by treating the file as a directory, but that doesn't really change the fact that it's a file. The _big_ difference is that when you can make the compound object _look_ like a directory, that means that you can now manage the attributes with standard tools. They are still attributes, though. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:54 ` Linus Torvalds @ 2004-08-26 21:00 ` Martin J. Bligh 2004-08-26 21:09 ` Christophe Saout ` (2 more replies) 2004-08-26 21:29 ` Chris Wright 2004-08-27 8:19 ` Anton Altaparmakov 2 siblings, 3 replies; 1835+ messages in thread From: Martin J. Bligh @ 2004-08-26 21:00 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Thu, 26 Aug 2004, Martin J. Bligh wrote: >> >> What would "test -d" and "test -f" return on these magic beasties? I can't >> think of any combinations that wouldn't confuse the crap out of userspace. > > "It's a feature". > > The S_ISDIR/S_ISREG tests show real information: it shows not only user > intent ("you should consider this a file, even if it has attributes"), but > also whether it is a directory or a container. > > And there's a real technical difference there: the streams contained > within a file are bound to that file. The files contained within a > directory are _independent_ of that directory. Big difference. HUGE > difference. > > So it's not confusing. If it tests as a file, you think of it as a file. > It may have attributes aka named streams associated with it, and you may > be able to open those attributes by treating the file as a directory, but > that doesn't really change the fact that it's a file. > > The _big_ difference is that when you can make the compound object _look_ > like a directory, that means that you can now manage the attributes with > standard tools. They are still attributes, though. I think what you're saying is that they'd both return positive, right? Unfortunately I think lots of code in userspace assumes files / directories are mutually exclusive conditions, and acts accordingly ... eg: if [ -f file ] do file stuff else do directory stuff fi. Now I'm not saying they're not broken ... but seems like a dangerous thing to change in the midst of a stable kernel series. M. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:00 ` Martin J. Bligh @ 2004-08-26 21:09 ` Christophe Saout 2004-08-26 21:13 ` Linus Torvalds 2004-08-26 22:44 ` Miquel van Smoorenburg 2 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 21:09 UTC (permalink / raw) To: Martin J. Bligh Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1459 bytes --] Am Donnerstag, den 26.08.2004, 14:00 -0700 schrieb Martin J. Bligh: > > "It's a feature". > > > > The S_ISDIR/S_ISREG tests show real information: it shows not only user > > intent ("you should consider this a file, even if it has attributes"), but > > also whether it is a directory or a container. > > > > And there's a real technical difference there: the streams contained > > within a file are bound to that file. The files contained within a > > directory are _independent_ of that directory. Big difference. HUGE > > difference. > > > > So it's not confusing. If it tests as a file, you think of it as a file. > > It may have attributes aka named streams associated with it, and you may > > be able to open those attributes by treating the file as a directory, but > > that doesn't really change the fact that it's a file. > > > > The _big_ difference is that when you can make the compound object _look_ > > like a directory, that means that you can now manage the attributes with > > standard tools. They are still attributes, though. > > I think what you're saying is that they'd both return positive, right? No. A file is still a file and S_ISREG will thus return false. The application should just try to open it as a directory if it wants to access the assiociated embedded attribute files. If the open fails, well, it knows that the filesystem (or Linux version, whatever) doesn't support this feature. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 21:09 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 21:09 UTC (permalink / raw) To: Martin J. Bligh Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1459 bytes --] Am Donnerstag, den 26.08.2004, 14:00 -0700 schrieb Martin J. Bligh: > > "It's a feature". > > > > The S_ISDIR/S_ISREG tests show real information: it shows not only user > > intent ("you should consider this a file, even if it has attributes"), but > > also whether it is a directory or a container. > > > > And there's a real technical difference there: the streams contained > > within a file are bound to that file. The files contained within a > > directory are _independent_ of that directory. Big difference. HUGE > > difference. > > > > So it's not confusing. If it tests as a file, you think of it as a file. > > It may have attributes aka named streams associated with it, and you may > > be able to open those attributes by treating the file as a directory, but > > that doesn't really change the fact that it's a file. > > > > The _big_ difference is that when you can make the compound object _look_ > > like a directory, that means that you can now manage the attributes with > > standard tools. They are still attributes, though. > > I think what you're saying is that they'd both return positive, right? No. A file is still a file and S_ISREG will thus return false. The application should just try to open it as a directory if it wants to access the assiociated embedded attribute files. If the open fails, well, it knows that the filesystem (or Linux version, whatever) doesn't support this feature. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:00 ` Martin J. Bligh 2004-08-26 21:09 ` Christophe Saout @ 2004-08-26 21:13 ` Linus Torvalds 2004-08-26 21:24 ` Jamie Lokier 2004-08-27 9:36 ` David Greaves 2004-08-26 22:44 ` Miquel van Smoorenburg 2 siblings, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 21:13 UTC (permalink / raw) To: Martin J. Bligh Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Martin J. Bligh wrote: > > I think what you're saying is that they'd both return positive, right? No. I'd say that a file would look like a file, even if it has attributes. It wouldn't show as a directory at all - unless you start looking at attributes. Because it really _is_ a file, and it's "directory aspect" is really nothing but a way to make its named streams visible. So you really should consider it a perfectly regular file, and so only S_ISREG() will return true, and S_ISDIR() will return false. Think of it this way: when you add a named attribute to current files using the xattr interfaces, do you start thinking of the file as a directory? No. Even though it actually now is a starting point for finding more information. It's just that thanks to it's directory aspect, people and apps that _care_ about the attributes suddenly can trivially access them. You can access them in shell scripts, you can access them in programs, you can access them in perl. With no special knowledge necessary. Remember: in the file-as-directory model, we're always talking about just _one_ object. That one object just happens to have namd streams or attributes or whatever you want to call them associated with it, and you can _see_ them through the normal directory interfaces. But that doesn't really make the object a "directory". It's still a file. In other words, the "directory" part is just a _view_ into the file. A view that potentially exposes a lot _more_ of the file, but we're still talking about the same file. In contrast, a S_IFDIR-like _directory_ is something else entirely. When you view the things in that, you aren't looking at data "inside" the directory. You're looking at somethign totally independent. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:13 ` Linus Torvalds @ 2004-08-26 21:24 ` Jamie Lokier 2004-08-27 9:36 ` David Greaves 1 sibling, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 21:24 UTC (permalink / raw) To: Linus Torvalds Cc: Martin J. Bligh, Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > In other words, the "directory" part is just a _view_ into the file. A > view that potentially exposes a lot _more_ of the file, but we're still > talking about the same file. > > In contrast, a S_IFDIR-like _directory_ is something else entirely. When > you view the things in that, you aren't looking at data "inside" the > directory. You're looking at somethign totally independent. One of the constraints is that you can create any name in an S_IFDIR-like directory, but you probably can't create any name "in" an S_IFREG-like file, because the names have meaning -- except in designated subtrees of the view intended for holding arbitrary names. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:13 ` Linus Torvalds 2004-08-26 21:24 ` Jamie Lokier @ 2004-08-27 9:36 ` David Greaves 1 sibling, 0 replies; 1835+ messages in thread From: David Greaves @ 2004-08-27 9:36 UTC (permalink / raw) To: Linus Torvalds Cc: Martin J. Bligh, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: >On Thu, 26 Aug 2004, Martin J. Bligh wrote: > > >>I think what you're saying is that they'd both return positive, right? >> >> > >No. I'd say that a file would look like a file, even if it has attributes. > >It wouldn't show as a directory at all - unless you start looking at >attributes. Because it really _is_ a file, and it's "directory aspect" is >really nothing but a way to make its named streams visible. > >So you really should consider it a perfectly regular file, and so only >S_ISREG() will return true, and S_ISDIR() will return false. > > > ie, logically: S_ISREG() no longer precludes O_DIRECTORY from succeeding conversley O_DIRECTORY succeeding no longer implies S_ISDIR() However, apps making those assumptions will not notice new behaviour. ? David ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:00 ` Martin J. Bligh 2004-08-26 21:09 ` Christophe Saout 2004-08-26 21:13 ` Linus Torvalds @ 2004-08-26 22:44 ` Miquel van Smoorenburg 2 siblings, 0 replies; 1835+ messages in thread From: Miquel van Smoorenburg @ 2004-08-26 22:44 UTC (permalink / raw) To: linux-kernel In article <57730000.1093554054@flay>, Martin J. Bligh <mbligh@aracnet.com> wrote: >> On Thu, 26 Aug 2004, Martin J. Bligh wrote: >>> >>> What would "test -d" and "test -f" return on these magic beasties? I can't >>> think of any combinations that wouldn't confuse the crap out of userspace. >> >I think what you're saying is that they'd both return positive, right? test -f file -> true test -f file/ -> false test -d file -> false test -d file/ -> true Mike. -- "In times of universal deceit, telling the truth becomes a revolutionary act." -- George Orwell. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:54 ` Linus Torvalds 2004-08-26 21:00 ` Martin J. Bligh @ 2004-08-26 21:29 ` Chris Wright 2004-08-27 8:19 ` Anton Altaparmakov 2 siblings, 0 replies; 1835+ messages in thread From: Chris Wright @ 2004-08-26 21:29 UTC (permalink / raw) To: Linus Torvalds Cc: Martin J. Bligh, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list * Linus Torvalds (torvalds@osdl.org) wrote: > The _big_ difference is that when you can make the compound object _look_ > like a directory, that means that you can now manage the attributes with > standard tools. They are still attributes, though. For things that actually use such attributes to determine file access, do the attributes get attributes? thanks, -chris -- Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:54 ` Linus Torvalds @ 2004-08-27 8:19 ` Anton Altaparmakov 2004-08-26 21:29 ` Chris Wright 2004-08-27 8:19 ` Anton Altaparmakov 2 siblings, 0 replies; 1835+ messages in thread From: Anton Altaparmakov @ 2004-08-27 8:19 UTC (permalink / raw) To: Linus Torvalds Cc: Martin J. Bligh, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Thu, 2004-08-26 at 21:54, Linus Torvalds wrote: > The S_ISDIR/S_ISREG tests show real information: it shows not only user > intent ("you should consider this a file, even if it has attributes"), but > also whether it is a directory or a container. > > And there's a real technical difference there: the streams contained > within a file are bound to that file. The files contained within a > directory are _independent_ of that directory. Big difference. HUGE > difference. > > So it's not confusing. If it tests as a file, you think of it as a file. > It may have attributes aka named streams associated with it, and you may > be able to open those attributes by treating the file as a directory, but > that doesn't really change the fact that it's a file. What about the attributes/named streams of a directory though? If you open it as a directory, you would get the files inside the directory. So how do you get at the attributes and named streams of the directory itself using this interface? Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:19 ` Anton Altaparmakov 0 siblings, 0 replies; 1835+ messages in thread From: Anton Altaparmakov @ 2004-08-27 8:19 UTC (permalink / raw) To: Linus Torvalds Cc: Martin J. Bligh, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Thu, 2004-08-26 at 21:54, Linus Torvalds wrote: > The S_ISDIR/S_ISREG tests show real information: it shows not only user > intent ("you should consider this a file, even if it has attributes"), but > also whether it is a directory or a container. > > And there's a real technical difference there: the streams contained > within a file are bound to that file. The files contained within a > directory are _independent_ of that directory. Big difference. HUGE > difference. > > So it's not confusing. If it tests as a file, you think of it as a file. > It may have attributes aka named streams associated with it, and you may > be able to open those attributes by treating the file as a directory, but > that doesn't really change the fact that it's a file. What about the attributes/named streams of a directory though? If you open it as a directory, you would get the files inside the directory. So how do you get at the attributes and named streams of the directory itself using this interface? Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:57 ` Rik van Riel @ 2004-08-26 18:28 ` Denis Vlasenko -1 siblings, 0 replies; 1835+ messages in thread From: Denis Vlasenko @ 2004-08-26 18:28 UTC (permalink / raw) To: Rik van Riel, Linus Torvalds Cc: Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 20:57, Rik van Riel wrote: > On Thu, 26 Aug 2004, Linus Torvalds wrote: > > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > > > No no. The stream you get from a directory is totally _independent_ of > > the contents of the directory. Anything else would be pointless. > > It's a relief to know that nobody's taking my humorous > suggestion seriously, but now we still have the "standard > Unix tools can't manipulate files" problem... Is it possible to sufficiently hide "dirs inside files" so that old tools will be unable to see them? I just checked: ls -d /foo does lstat64("/foo", ...) ls -d /foo/ does lstat64("/foo", ...) but ls -d /foo/. does lstat64("/foo/.", ...) Will it work out if "dir inside file" will only be visible when referred as "file/."? -- vda ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 18:28 ` Denis Vlasenko 0 siblings, 0 replies; 1835+ messages in thread From: Denis Vlasenko @ 2004-08-26 18:28 UTC (permalink / raw) To: Rik van Riel, Linus Torvalds Cc: Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 20:57, Rik van Riel wrote: > On Thu, 26 Aug 2004, Linus Torvalds wrote: > > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > > > No no. The stream you get from a directory is totally _independent_ of > > the contents of the directory. Anything else would be pointless. > > It's a relief to know that nobody's taking my humorous > suggestion seriously, but now we still have the "standard > Unix tools can't manipulate files" problem... Is it possible to sufficiently hide "dirs inside files" so that old tools will be unable to see them? I just checked: ls -d /foo does lstat64("/foo", ...) ls -d /foo/ does lstat64("/foo", ...) but ls -d /foo/. does lstat64("/foo/.", ...) Will it work out if "dir inside file" will only be visible when referred as "file/."? -- vda ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:28 ` Denis Vlasenko (?) @ 2004-08-26 18:46 ` Linus Torvalds 2004-08-26 19:13 ` viro -1 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 18:46 UTC (permalink / raw) To: Denis Vlasenko Cc: Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > Is it possible to sufficiently hide "dirs inside files" > so that old tools will be unable to see them? Certainly possible. > I just checked: > > ls -d /foo does lstat64("/foo", ...) > ls -d /foo/ does lstat64("/foo", ...) > but > ls -d /foo/. does lstat64("/foo/.", ...) > > Will it work out if "dir inside file" will only be visible when referred as "file/."? That would likely be the _easiest_ approach for the kernel, and does solve the problem with some apps knowingly removing the trailing '/'. Note that we could try this out with existing filesystems with very minimal changes: - make directory bind mounts work on top of files ("graft_tree()") - make open_namei() and friend _not_ do the mount-point following for the last component if it's a non-directory. - probably some trivial fixups I haven't thought about. There might be some places that use "S_ISDIR()" to check for whether something can be looked up, but the main path walking already just checks whether there is a ".lookup" operation or not. This would already allow people to "try out" how different applications would react to a file that can show up both as a directory and a file. The patch might end up being less than 25 lines or so, the difficulty is in finding all the right places. Al, anything I missed? (And yes, it's a quick hack, but it's a quick hack that would probably mimic a good part of what we would have to do internally in the VFS layer to support this notion anyway). Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:46 ` Linus Torvalds @ 2004-08-26 19:13 ` viro 2004-08-26 20:32 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-26 19:13 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [reposted in thread] On Thu, Aug 26, 2004 at 11:46:33AM -0700, Linus Torvalds wrote: > Note that we could try this out with existing filesystems with very > minimal changes: > > - make directory bind mounts work on top of files ("graft_tree()") > - make open_namei() and friend _not_ do the mount-point following for the > last component if it's a non-directory. > - probably some trivial fixups I haven't thought about. There might be > some places that use "S_ISDIR()" to check for whether something can be > looked up, but the main path walking already just checks whether there > is a ".lookup" operation or not. > > This would already allow people to "try out" how different applications > would react to a file that can show up both as a directory and a file. The > patch might end up being less than 25 lines or so, the difficulty is in > finding all the right places. The real issue is what to do with unlink() et.al. on these guys. Note that "unlink is OK if all we have there is a bunch of directory mounts" won't work well - we have no good way to check that condition. Even funnier one is what we do if we have directory mounted there *and* have something mounted on stuff in that directory. Yes, that's one of the probable directions for such stuff, but there's a lot of fun semantics questions and answers to them will matter a lot. Hey, if we lose the "can't unlink/rmdir/rename over something that is a mountpoint in other life" - I'm happy and we can get a lot of much more interesting stuff to work. It will take some work (e.g. making sure we can find all vfsmounts over given mountpoint and sorting out the locking issues, which won't be trivial), but the main obstacle in that direction is not in architecture - it's in SuS and tradition; as the matter of fact, our life would be much easier if we stopped trying to give -EBUSY here and just dissolved all subtrees mounted on anything that has that dentry. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:13 ` viro @ 2004-08-26 20:32 ` viro 2004-08-26 20:36 ` Christophe Saout 2004-08-26 20:47 ` Linus Torvalds 0 siblings, 2 replies; 1835+ messages in thread From: viro @ 2004-08-26 20:32 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 08:13:23PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > Hey, if we lose the "can't unlink/rmdir/rename over something that is > a mountpoint in other life" - I'm happy and we can get a lot of much > more interesting stuff to work. It will take some work (e.g. making > sure we can find all vfsmounts over given mountpoint and sorting out > the locking issues, which won't be trivial), but the main obstacle in > that direction is not in architecture - it's in SuS and tradition; as > the matter of fact, our life would be much easier if we stopped trying > to give -EBUSY here and just dissolved all subtrees mounted on anything > that has that dentry. Argh... OK, now I remember why I went for -EBUSY for unlink() (we obviously are not bound by SuS on that one). Consider the following scenario: * local file foo got something else bound on it for a while * we are tight on space - time to clean up * oh, look - contents of foo is junk * rm foo * ... oh, fuck, there goes the underlying file. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:32 ` viro @ 2004-08-26 20:36 ` Christophe Saout 2004-08-26 20:47 ` Linus Torvalds 1 sibling, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 20:36 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 619 bytes --] Am Donnerstag, den 26.08.2004, 21:32 +0100 schrieb : > Argh... OK, now I remember why I went for -EBUSY for unlink() (we obviously > are not bound by SuS on that one). Consider the following scenario: > * local file foo got something else bound on it for a while > * we are tight on space - time to clean up > * oh, look - contents of foo is junk > * rm foo > * ... oh, fuck, there goes the underlying file. Right. Good thinking. You can't delete a directory while there regular files in it either. Another question: /mnt/test is a mountpoint. Why can I rename `mnt' but can't rename `test'? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 20:36 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 20:36 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 619 bytes --] Am Donnerstag, den 26.08.2004, 21:32 +0100 schrieb : > Argh... OK, now I remember why I went for -EBUSY for unlink() (we obviously > are not bound by SuS on that one). Consider the following scenario: > * local file foo got something else bound on it for a while > * we are tight on space - time to clean up > * oh, look - contents of foo is junk > * rm foo > * ... oh, fuck, there goes the underlying file. Right. Good thinking. You can't delete a directory while there regular files in it either. Another question: /mnt/test is a mountpoint. Why can I rename `mnt' but can't rename `test'? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:32 ` viro 2004-08-26 20:36 ` Christophe Saout @ 2004-08-26 20:47 ` Linus Torvalds 2004-08-26 21:28 ` [some sanity for a change] possible design issues for hybrids viro 1 sibling, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 20:47 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > Argh... OK, now I remember why I went for -EBUSY for unlink() (we obviously > are not bound by SuS on that one). Consider the following scenario: > * local file foo got something else bound on it for a while > * we are tight on space - time to clean up > * oh, look - contents of foo is junk > * rm foo > * ... oh, fuck, there goes the underlying file. Hey, that's a valid reason for doing -EBUSY for normal bind-mounts, but it actually _is_ what we want for an "implied-by-way-of-container-mount". After all, when you do a "rm foo", you do mean "remove the container foo". I replied to your earlier off-list mail in private, so let's re-iterate for the list: the easiest way to handle this is to just have a "mount option", and have "MNT_ALLOWUNLINK" that gets set for containers, and that users could possibly choose to set for regular mounts too (as a mount option) if they really want to (and if we want them to). So there's no reason we'd have to drop existing mount behaviour only because we also have special files that look like mountpoints. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* [some sanity for a change] possible design issues for hybrids 2004-08-26 20:47 ` Linus Torvalds @ 2004-08-26 21:28 ` viro 2004-08-26 22:04 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-26 21:28 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [subject changed and *please* let's keep the wankfest out of that branch; we are talking about possible ways to handle hybrids, *NOT* their desirability, effect on DARPA funding, size of KDE developers' genitals experienced by Aunt Tillie, etc.] On Thu, Aug 26, 2004 at 01:47:37PM -0700, Linus Torvalds wrote: > Hey, that's a valid reason for doing -EBUSY for normal bind-mounts, but it > actually _is_ what we want for an "implied-by-way-of-container-mount". > After all, when you do a "rm foo", you do mean "remove the container foo". > > I replied to your earlier off-list mail in private, so let's re-iterate > for the list: the easiest way to handle this is to just have a "mount > option", and have "MNT_ALLOWUNLINK" that gets set for containers, and that > users could possibly choose to set for regular mounts too (as a mount > option) if they really want to (and if we want them to). > > So there's no reason we'd have to drop existing mount behaviour only > because we also have special files that look like mountpoints. All right, let's see where that would take us. 1) we would need to find all vfsmounts over given dentry. Probably a cyclic list (we want to check if there are normal mounts/bindings among those and we want to dissolve them if there's none). 2) we would need to do something about locking, since mount trees in other guys' namespaces are protected by semaphores of their own. 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off it. MNT_DETACH will have no problems, but normal umount() is a different story. Note that it's not just hybrid-related problem - implementing the mount traps will cause the same kind of trouble, 4) OK, we have those hybrids and want to create vfsmounts when crossing a mountpoint. When do they go away, anyway? When we don't reference them anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching happens explicitly - outside of the "dropping the final reference" path. Might become a locking issue. 5) Creation of these vfsmounts: fs should somehow tell us whether it wants one or not (at the very least, we should stop *somewhere*). Can we use the same dentry/inode? I'm not sure and I really doubt that we'd like that. 6) if it's a method, where should it live, *especially* if we want them on device nodes. Note that inode_operations belongs to underlying fs, so it's not particulary good place for device case. 7) automount folks want partially shared mount trees (well, mirrored, actually). The basic idea is that while namespace boundary is a trust boundary, we might want to be able to say "I trust this guy to handle that subtree under /home/stuff/foobar/mounts". It's the same situation as with shared mappings (I want separate address space, but I'm willing to share that chunk of memory with other process), except that we want it to be allowed to become asymmetric (shared r/o mapping with somebody else having it r/w). That stuff (and mount traps) is the next pending major work on the mount trees. We probably want hybrids work to go with it, since they affect the same data structures and need more or less the same changes. [And yes, this is the open season on design discussions for shared subtrees - automount folks are welcome to join] 8) what should happen when something is mounted on top of directory-over-file? How do we treat such beasts? What are the implications? 9) how do we recognize such mountpoints in the path lookups? It *is* a hot path, so we should be careful in that area; the impact will be felt by everything in the system. 10) how do we deal with directories, anyway? Mixing "attributes" with normal directory contents is going to be fun, what with lseek() insanity. That's not an issue for hybrids, but it is one for anybody who wants any sort of common metadata exported that way. Note that it's really important for fs writers - having two different pieces of code to export the same information (for directories and non-directories resp.) is going to become a prime breeding ground for bugs. 11) if we go for your "here's stuff that belongs in device node viewed as directory", how would that play with fs metadata exporters? Again, due to the insanity of lseek() on directories it's *very* hard to deal with unions, when parts of directory come from different chunks of code. That's it for starters. Technical answers/questions/comments are welcome. Generic masturbation => over there in the parent thread, please. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 21:28 ` [some sanity for a change] possible design issues for hybrids viro @ 2004-08-26 22:04 ` Linus Torvalds 2004-08-26 22:36 ` viro 2004-08-27 8:56 ` Anton Altaparmakov 0 siblings, 2 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 22:04 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [ This is quite possibly just impossible and buggy, but here's my implementation notes. You asked for them. ] On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > All right, let's see where that would take us. > > 1) we would need to find all vfsmounts over given dentry. Probably a cyclic > list (we want to check if there are normal mounts/bindings among those and > we want to dissolve them if there's none). Not per-inode? dentries are a bitmore memory-constrained than inodes, and we only need this for filesystems that want to support it, so we wouldn't need to put this information in each dentry. Since the vfsmounts have back-pointers to the dentry they are mounted on, you can still do a "per-dentry" traversal, by just doign the inode list and checking the dentry pointer. No? Alternatively, we could just put the list on an external hash-chain entirely, and hash off the dentry. It depends on how often we end up needing it. Or we could just put it in the dentry itself. I'd hate to make it any bigger than it already is, but maybe it doesn't matter that much. > 2) we would need to do something about locking, since mount trees in other > guys' namespaces are protected by semaphores of their own. Ok, I'll admit that I don't know how to handle namespaces. These things should just go into a global namespace, and I was kind of assuming it would happen automatically in "lookup_mnt()" or something like that. A special case in lookup_mnt which says something like "if you didn't find a vfsmount, we create a new one for you". It should be reasonably easy to create new ones on-the-fly, since we'd have all the information (the parent vfsmount comes stated, and the vfsmount we create would point to the same things that the "base" one would). > 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off > it. MNT_DETACH will have no problems, but normal umount() is a different > story. Note that it's not just hybrid-related problem - implementing the > mount traps will cause the same kind of trouble, Don't allow umount. It's not something the user can unmount - the mount is "implied" in the file. > 4) OK, we have those hybrids and want to create vfsmounts when crossing a > mountpoint. When do they go away, anyway? When we don't reference them > anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching > happens explicitly - outside of the "dropping the final reference" path. > Might become a locking issue. Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously clossed something over when I blathered about the "create the vfsmount on the fly" thing above ;) > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants > one or not (at the very least, we should stop *somewhere*). Can we use > the same dentry/inode? I'm not sure and I really doubt that we'd like that. Why not? When doing the ->lookup() operation, the filesystem would create the vfsmount and bind it to the current vfsmount. That guarantees that it has a vfsmount, and will mean that it will show up positive with the "d_mountpoint()" query, which in turn will cause us to do the "lookup_mnt()". Which in turn will create the other vfsmounts as needed, if you have multiple namespaces. So I _think_ creation is easy. Getting rid of the dang things migth be harder. > 6) if it's a method, where should it live, *especially* if we want them on > device nodes. Note that inode_operations belongs to underlying fs, so it's > not particulary good place for device case. Why not just let the existing .lookup method initialize the mount-point thing? After that, it's all in the VFS layer (I'd hate to have filesystems mess around with vfsmounts - they'll just get it wrong). > 7) automount folks want partially shared mount trees (well, mirrored, > actually). I don't think you can get partial sharing on one of these puppies. You'd always have one vfsmount per namespace (well, lazily created, so maybe in practice you'd see a lot fewer). > 8) what should happen when something is mounted on top of directory-over-file? > How do we treat such beasts? What are the implications? Allow file-on-file mounts - it will just totally hide the thing (in that namespace, at least). But don't allow the dir-on-file thing (that we already don't allow). > 9) how do we recognize such mountpoints in the path lookups? It *is* a > hot path, so we should be careful in that area; the impact will be felt > by everything in the system. I don't think you'll have any special cases. Same d_mountpount(), same lookup_mnt(). > 10) how do we deal with directories, anyway? Mixing "attributes" with > normal directory contents is going to be fun, what with lseek() insanity. You couldn't get at the attributes that way anyway, so I think the point is moot. The "real" directory always takes over. Crazy people could try to just use the regular "xattrs" interfaces if they really want attributes on directories. You wouldn't ever be able to use the "easy" one. > 11) if we go for your "here's stuff that belongs in device node viewed > as directory", how would that play with fs metadata exporters? Again, > due to the insanity of lseek() on directories it's *very* hard to deal > with unions, when parts of directory come from different chunks of code. Don't go there. See above. Directories would be just plain directories, you could never see their metadata. Same goes for at least symlinks, and possibly other filetypes too (ie at least initially, a block or character special device will just take over the whole "file_operations", which includes "readdir", so it's actually hard to have the filesystem do anything about those). Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:04 ` Linus Torvalds @ 2004-08-26 22:36 ` viro 2004-08-26 22:45 ` Linus Torvalds 2004-08-27 8:56 ` Anton Altaparmakov 1 sibling, 1 reply; 1835+ messages in thread From: viro @ 2004-08-26 22:36 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 03:04:21PM -0700, Linus Torvalds wrote: > > 2) we would need to do something about locking, since mount trees in other > > guys' namespaces are protected by semaphores of their own. > > Ok, I'll admit that I don't know how to handle namespaces. These things > should just go into a global namespace, and I was kind of assuming it > would happen automatically in "lookup_mnt()" or something like that. A > special case in lookup_mnt which says something like "if you didn't find a > vfsmount, we create a new one for you". > > It should be reasonably easy to create new ones on-the-fly, since we'd > have all the information (the parent vfsmount comes stated, and the > vfsmount we create would point to the same things that the "base" one > would). Erm... What do we do upon unlink()? I'm killing a file, fs it's in is mounted in a dozen of places (no namespaces, just chroot jails, whatever). We need to find all vfsmounts to be killed by that. And BTW that's an argument against anchoring that list in inode - unlink() on foo should not screw bar/... even if bar and foo are links to the same file. So we'll need to check for dentry match anyway. > > 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off > > it. MNT_DETACH will have no problems, but normal umount() is a different > > story. Note that it's not just hybrid-related problem - implementing the > > mount traps will cause the same kind of trouble, > > Don't allow umount. It's not something the user can unmount - the mount is > "implied" in the file. See below. > > 4) OK, we have those hybrids and want to create vfsmounts when crossing a > > mountpoint. When do they go away, anyway? When we don't reference them > > anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching > > happens explicitly - outside of the "dropping the final reference" path. > > Might become a locking issue. > > Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously > clossed something over when I blathered about the "create the vfsmount on > the fly" thing above ;) > > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants > > one or not (at the very least, we should stop *somewhere*). Can we use > > the same dentry/inode? I'm not sure and I really doubt that we'd like that. > > Why not? When doing the ->lookup() operation, the filesystem would create > the vfsmount and bind it to the current vfsmount. That guarantees that it > has a vfsmount, and will mean that it will show up positive with the > "d_mountpoint()" query, which in turn will cause us to do the > "lookup_mnt()". Several paragraphs below you are saying that you don't like fs messing with vfsmounts. Use of ->lookup() would mean that we should not only create and attach vfsmounts from within fs code, but would actually have to make ->lookup() return vfsmount+dentry, AFAICS. > > 6) if it's a method, where should it live, *especially* if we want them on > > device nodes. Note that inode_operations belongs to underlying fs, so it's > > not particulary good place for device case. > > Why not just let the existing .lookup method initialize the mount-point > thing? After that, it's all in the VFS layer (I'd hate to have filesystems > mess around with vfsmounts - they'll just get it wrong). > Allow file-on-file mounts - it will just totally hide the thing (in that > namespace, at least). But don't allow the dir-on-file thing (that we > already don't allow). Err... What about dir-on-dir-that-is-on-file? I.e. mount on foo/. when foo is a file? > > 9) how do we recognize such mountpoints in the path lookups? It *is* a > > hot path, so we should be careful in that area; the impact will be felt > > by everything in the system. > I don't think you'll have any special cases. Same d_mountpount(), same > lookup_mnt(). See above on use ->lookup() ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:36 ` viro @ 2004-08-26 22:45 ` Linus Torvalds 2004-08-26 22:53 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 22:45 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > It should be reasonably easy to create new ones on-the-fly, since we'd > > have all the information (the parent vfsmount comes stated, and the > > vfsmount we create would point to the same things that the "base" one > > would). > > Erm... What do we do upon unlink()? I'm killing a file, fs it's in is > mounted in a dozen of places (no namespaces, just chroot jails, whatever). > We need to find all vfsmounts to be killed by that. But that should be trivial: that's what the per-inode vfsmount list was (your first question in the last email). > And BTW that's an argument against anchoring that list in inode - unlink() > on foo should not screw bar/... even if bar and foo are links to the same > file. So we'll need to check for dentry match anyway. And again - I talked about this in the previous email. Even if you anchor the list in "struct inode", or you do it with a totally external hash-list, you'll always have the "vfsmount->mnt_mountpoint" pointer to point to the dentry. So you can just iterate over the list, and cherry-pick the ones that point to the dentry you are removing. > > > > 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off > > > it. MNT_DETACH will have no problems, but normal umount() is a different > > > story. Note that it's not just hybrid-related problem - implementing the > > > mount traps will cause the same kind of trouble, > > > > Don't allow umount. It's not something the user can unmount - the mount is > > "implied" in the file. > > See below. > > > > 4) OK, we have those hybrids and want to create vfsmounts when crossing a > > > mountpoint. When do they go away, anyway? When we don't reference them > > > anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching > > > happens explicitly - outside of the "dropping the final reference" path. > > > Might become a locking issue. > > > > Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously > > clossed something over when I blathered about the "create the vfsmount on > > the fly" thing above ;) > > > > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants > > > one or not (at the very least, we should stop *somewhere*). Can we use > > > the same dentry/inode? I'm not sure and I really doubt that we'd like that. > > > > Why not? When doing the ->lookup() operation, the filesystem would create > > the vfsmount and bind it to the current vfsmount. That guarantees that it > > has a vfsmount, and will mean that it will show up positive with the > > "d_mountpoint()" query, which in turn will cause us to do the > > "lookup_mnt()". > > Several paragraphs below you are saying that you don't like fs messing with > vfsmounts. Use of ->lookup() would mean that we should not only create > and attach vfsmounts from within fs code, but would actually have to make > ->lookup() return vfsmount+dentry, AFAICS. No, lookup would just return the dentry, but the dentry would already be filled in with the mount-point information. And you can do that with a simple vfs helper function, ie the filesystem itself would just need to do pseudo_mount(dentry, inode); thing - which just fills in dentry->d_mountpoint with a new vfsmount thing. It would allocate a new root dentry (for the pseudo-mount) and a new vfsmount, and make dentry->d_mountpoint point to it. IOW, the filesystem itself would never mess around with d_mountpoint itself. > Err... What about dir-on-dir-that-is-on-file? I.e. mount on foo/. when foo > is a file? Hmm.. We might as well allow it, I suspect. It's not like it should hurt. We'd end up following the mount-chain twice, but we already have that issue with multi-mount cases.. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:45 ` Linus Torvalds @ 2004-08-26 22:53 ` viro 2004-08-26 23:24 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-26 22:53 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 03:45:09PM -0700, Linus Torvalds wrote: > No, lookup would just return the dentry, but the dentry would already be > filled in with the mount-point information. > > And you can do that with a simple vfs helper function, ie the filesystem > itself would just need to do > > pseudo_mount(dentry, inode); > > thing - which just fills in dentry->d_mountpoint with a new vfsmount > thing. It would allocate a new root dentry (for the pseudo-mount) and a > new vfsmount, and make dentry->d_mountpoint point to it. What dentry->d_mountpoint? No such thing... Note that we can't get vfsmount by dentry - that's the point of having these guys in the first place. So I'm not sure what you are trying to do here - dentry + inode is definitely not enough to attach any vfsmounts anywhere. That's not about namespaces - same fs mounted in several places will give the same problem - one dentry, many vfsmounts. And we obviously *can't* have one vfsmount for all of them - if the same fs is mounted on /foo and /bar, we will have the same dentry for /foo/splat and /bar/splat. So what should we get for /foo/splat/. and /bar/splat/.? Same dentry *and* same vfsmount? I'd expect .. from the former to give /foo and from the latter - /bar... ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:53 ` viro @ 2004-08-26 23:24 ` Linus Torvalds 2004-08-26 23:40 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 23:24 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > What dentry->d_mountpoint? No such thing... Sorry - set "dentry->d_mounted++" + "add vfsmount/dentry to hashes". Yes, it's not a direct list off the dentry, but it effectively is the same thing. So basically: the "d_mounted++" just makes sure we get into "lookup_mnt()". That's where we will usually find the actual mount thing. And that's also where the special case comes in: if we _don't_ find the mount thing there, that's where we need to create it. That will only happen if somebody looks it up using another namespace, though, so it should be rare. And when it does happen, we can just create a new vfsmount - we have all the information there (we'll have to walk the per-inode-or-whatever vfsmount list to find all the information to populate the thing with, of course. But we need that list _anyway_, so it should be a fairly straightforward special case). Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 23:24 ` Linus Torvalds @ 2004-08-26 23:40 ` viro 2004-08-26 23:57 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-26 23:40 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 04:24:51PM -0700, Linus Torvalds wrote: > So basically: the "d_mounted++" just makes sure we get into > "lookup_mnt()". That's where we will usually find the actual mount thing. > > And that's also where the special case comes in: if we _don't_ find the > mount thing there, that's where we need to create it. That will only > happen if somebody looks it up using another namespace, though, so it > should be rare. No. Trivial example: mount --bind /foo /bar mount /dev/sda1 /bar/baz do lookup for /foo/baz. No namespaces involved, no vfsmounts found, d_mounted positive and we certainly do *not* want anything to be created at that point. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 23:40 ` viro @ 2004-08-26 23:57 ` Linus Torvalds 2004-08-27 1:01 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-26 23:57 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > On Thu, Aug 26, 2004 at 04:24:51PM -0700, Linus Torvalds wrote: > > So basically: the "d_mounted++" just makes sure we get into > > "lookup_mnt()". That's where we will usually find the actual mount thing. > > > > And that's also where the special case comes in: if we _don't_ find the > > mount thing there, that's where we need to create it. That will only > > happen if somebody looks it up using another namespace, though, so it > > should be rare. > > No. Trivial example: > > mount --bind /foo /bar > mount /dev/sda1 /bar/baz > > do lookup for /foo/baz. No namespaces involved, no vfsmounts found, d_mounted > positive and we certainly do *not* want anything to be created at that point. Right. We obviously need to mark the dentry somehow, and only do the "create vfsmount" special case in this special case. If we didn't do that, then it wouldn't be a special case, now would it? So clearly lookup_mnt() needs to check the dentry in the failure case. The marking could be in any of three places - mark the dentry itself by just using a dentry flag ("DCACHE_AUTOVFSMNT") or by having a dentry operation for this. - mark the inode itself (same logic as dentry) - look up the first vfsmount (on the inode list), and look if that one is of the automatic type. Clearly we should not _always_ create a vfsmount, that would just break the existign logic. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 23:57 ` Linus Torvalds @ 2004-08-27 1:01 ` viro 2004-08-27 2:46 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-27 1:01 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 04:57:01PM -0700, Linus Torvalds wrote: > Right. We obviously need to mark the dentry somehow, and only do the > "create vfsmount" special case in this special case. If we didn't do that, > then it wouldn't be a special case, now would it? > > So clearly lookup_mnt() needs to check the dentry in the failure case. The > marking could be in any of three places > - mark the dentry itself by just using a dentry flag ("DCACHE_AUTOVFSMNT") > or by having a dentry operation for this. > - mark the inode itself (same logic as dentry) > - look up the first vfsmount (on the inode list), and look if that one is > of the automatic type. > > Clearly we should not _always_ create a vfsmount, that would just break > the existign logic. Hmm... IOW, you are suggesting to use normal trigger for lookup_mnt() and then have extra check in lookup_mnt() failure exit. That will probably work (and I'd prefer to mark dentry here). Freeing these guys will not be fun. The final reference could be given up under very different locking conditions - it's certainly a blocking operation (it can trigger final umount, among other things), but having it trigger removal from mount tree(s) can get ugly. We might be able to tweak that, but it will probably need require major surgery in several places. Right now we assume that vfsmount lock *and* per-namespace semaphore are held when detaching from the tree (attaching what could be the first at given dentry --- same + i_sem on its ->i_sem). We also assume that vfsmount is detached before refcount reaches zero. The thing is, that final refcount can be dropped while holding namespace semaphore. It's not fatal (we'll need to hold onto these guys until dropping that semaphore or add an analog of mntput() for places under namespace sem), but it can get messy. Situation with unlink(): we need to (a) check that all vfsmounts over given point are "hybrid" ones (locking is not a problem at that point) and (b) after successful operation (unlink() and rename() alike, AFAICS) we want to go and kill everything under the victim dentry. *That* can get very ugly, since fs could be mounted in a lot of places and/or in a lot of namespaces. Note that we also need to do in VFS what NFS tries to do in revalidation - if dentry is invalidated, everything under it should be detached and dropped. So it's not something new - we do need something that would be called outside of any superblock/namespace/vfsmount locks (both unlink() and revalidation qualify) and would kill the vfsmounts that become unreachable. Locking rules for attaching might need adjustment (the logics with ->i_sem is that we want !d_mountpoint() stay true around call of ->unlink(), ->rmdir() and ->rename(); so cloning or replacing doesn't need that protection. Now it might become interesting - we'll need to be careful in pivot_root(2) and possibly some other places). I'll see what can be done (it'll definitely take careful looking at the code), but there is a chance of things getting very ugly. BTW, shared subtrees *are* compatible with that puppy - there's a lot of corner cases to look at, but AFAICS they are all doable. Lifetime rules can get ugly in some of them, but that's on par with the situation without sharing. One thing that looks like a bad interface: we get forcible "use same dentry for file and directory" with that design. That can lead to a big can of worms - without a large audit I wouldn't bet a dime on ability to get that right (and same applies to reiser4 code as-is, for the same reasons). No idea how bad it will turn out to be - right now all I can say is that we might run into trouble and that we almost certainly will get extra "be careful not to do <list of things>" rules out of that. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-27 1:01 ` viro @ 2004-08-27 2:46 ` Linus Torvalds 2004-08-27 3:45 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-27 2:46 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > Hmm... IOW, you are suggesting to use normal trigger for lookup_mnt() and > then have extra check in lookup_mnt() failure exit. Yes. That should be an _extremely_ rare failure case anyway, unless people start using namespaces more. > Freeing these guys will not be fun. I agree. I frankly have no idea how to do it. I think we need to keep them around for as long as the dentry lifetime, I suspect. That would likely imply that the vfsmount can't increase the dentry refcount - and we'd have some special code in dentry_iput() or something to get rid of them. > Situation with unlink(): we need to (a) check that all vfsmounts over > given point are "hybrid" ones (locking is not a problem at that point) > and (b) after successful operation (unlink() and rename() alike, AFAICS) > we want to go and kill everything under the victim dentry. *That* > can get very ugly, since fs could be mounted in a lot of places and/or > in a lot of namespaces. It _might_ be acceptable to just do the normal shrink_dentry thing, and then return EBUSY if somebody is still there. > Locking rules for attaching might need adjustment (the logics with ->i_sem > is that we want !d_mountpoint() stay true around call of ->unlink(), > ->rmdir() and ->rename() That should be ok. Since we'd only do the d_mount stuff on a "->lookup()", and since the lookup will take ->i_sem anyway, we should be fine, no? > One thing that looks like a bad interface: we get forcible "use same > dentry for file and directory" with that design. Hmm.. Wouldn't the "directory" dentry always be the fake-mountpoint root one? So we'd always have two dentries, although they are tied together pretty tightly.. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-27 2:46 ` Linus Torvalds @ 2004-08-27 3:45 ` viro 2004-08-27 3:57 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-27 3:45 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 07:46:41PM -0700, Linus Torvalds wrote: > > > On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > Hmm... IOW, you are suggesting to use normal trigger for lookup_mnt() and > > then have extra check in lookup_mnt() failure exit. > > Yes. That should be an _extremely_ rare failure case anyway, unless people > start using namespaces more. Namespaces actually are irrelevant - a box with several dozens of chroot jail is not particulary exotic thing and it's equivalent to our case. However, extra check of a flag at that point won't be too much overhead anyway. > > Freeing these guys will not be fun. > > I agree. I frankly have no idea how to do it. I think we need to keep them > around for as long as the dentry lifetime, I suspect. That would likely > imply that the vfsmount can't increase the dentry refcount - and we'd have > some special code in dentry_iput() or something to get rid of them. My plan is to drop them as soon as their refcount goes to zero. In other words, make the final mntput() do removal from tree for such vfsmounts. Will take some surgery, but it's definitely safer than playing with ->d_iput() et.al. > > can get very ugly, since fs could be mounted in a lot of places and/or > > in a lot of namespaces. > > It _might_ be acceptable to just do the normal shrink_dentry thing, and > then return EBUSY if somebody is still there. We need it for graceful invalidation in case of NFS and friends, though. I think it might be doable; if it isn't, we'll see what else can be done, but it's worth investigating. > > Locking rules for attaching might need adjustment (the logics with ->i_sem > > is that we want !d_mountpoint() stay true around call of ->unlink(), > > ->rmdir() and ->rename() > > That should be ok. Since we'd only do the d_mount stuff on a "->lookup()", > and since the lookup will take ->i_sem anyway, we should be fine, no? Oh, I'm not worried about that kind of incrementing ->d_mount. It's OK. It's existing places that worry me - e.g. if we get pivot_root() moving that hybrid-created vfsmount around and suddenly find that we had replaced "killable" vfsmount with normal one, while holding only vfsmount lock. That would screw the logics in unlink() if we get a race. It's not that hard to do, just that we need to take a good look at that kind of corner cases. > > One thing that looks like a bad interface: we get forcible "use same > > dentry for file and directory" with that design. > > Hmm.. Wouldn't the "directory" dentry always be the fake-mountpoint root > one? So we'd always have two dentries, although they are tied together > pretty tightly.. No. We don't want to allocate dentries outside of fs code - if nothing else, they may need to have proper ->d_op set, etc. So the only way I see here is to have lookup_mnt() set the dentry it got to ->mnt_root of new vfsmount. Anything else opens such a can of worms that I don't want to think of audit involved. We *might* have a secondary dentry allocated by ->lookup() and stored somewhere in the main one. I'm not sure it's a good way to handle the things, though. I'll try to put together what we got so far + notes on mount-traps and post more coherent and detailed variant of original post. Hopefully, writing it down would help to sort the things out... ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-27 3:45 ` viro @ 2004-08-27 3:57 ` Linus Torvalds 2004-08-27 4:30 ` viro 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-27 3:57 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > I agree. I frankly have no idea how to do it. I think we need to keep them > > around for as long as the dentry lifetime, I suspect. That would likely > > imply that the vfsmount can't increase the dentry refcount - and we'd have > > some special code in dentry_iput() or something to get rid of them. > > My plan is to drop them as soon as their refcount goes to zero. In other > words, make the final mntput() do removal from tree for such vfsmounts. Can't do that. At least not if you do my plan. Even with a zero refcount, the dentry is still valid, and won't see a new "->lookup()". And since in my scheme the vfsmount is populated only at lookup (and copied later), you can't drop _all_ the vfsmount's. You'd have to at least leave one of them, in order to know how to re-create the others. Or do you have a cunning plan? > > > can get very ugly, since fs could be mounted in a lot of places and/or > > > in a lot of namespaces. > > > > It _might_ be acceptable to just do the normal shrink_dentry thing, and > > then return EBUSY if somebody is still there. > > We need it for graceful invalidation in case of NFS and friends, though. > I think it might be doable; if it isn't, we'll see what else can be done, > but it's worth investigating. Hey, succeeding and leaving it pending is obviously the user-friendly thing to do, so I won't argue against it. It's more of a "if it's too hard to do right, EBUSY is probably an acceptable fallback". > > Hmm.. Wouldn't the "directory" dentry always be the fake-mountpoint root > > one? So we'd always have two dentries, although they are tied together > > pretty tightly.. > > No. We don't want to allocate dentries outside of fs code - if nothing > else, they may need to have proper ->d_op set, etc. So the only way I > see here is to have lookup_mnt() set the dentry it got to ->mnt_root of > new vfsmount. Anything else opens such a can of worms that I don't want > to think of audit involved. Hmm. If you use the same dentry for mnt_root as for mnt_mountpoint, how do you avoid the infinite loop in follow_mount? Another magic special case? Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-27 3:57 ` Linus Torvalds @ 2004-08-27 4:30 ` viro 2004-08-27 4:42 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: viro @ 2004-08-27 4:30 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 08:57:40PM -0700, Linus Torvalds wrote: > Can't do that. At least not if you do my plan. > > Even with a zero refcount, the dentry is still valid, and won't see a new > "->lookup()". And since in my scheme the vfsmount is populated only at > lookup (and copied later), you can't drop _all_ the vfsmount's. You'd have > to at least leave one of them, in order to know how to re-create the > others. > > Or do you have a cunning plan? a) set ->d_mount to 1 and set DCACHE_HYBRID at ->lookup() time b) in lookup_mnt(): if (unlikely(dentry has DCACHE_HYBRID)) { if (linkely(dentry->d_mount == 1)) goto New; } do the normal series of hash lookups if (likely(dentry doesn't have DCACHE_HYBRID)) return; if (vfsmount is marked as hybrid && dentry == vfsmount->mnt_dentry) return; New: allocate new vfsmount and mark it as hybrid attach it on top of our one and go there Does that qualify as cunning? With that approach we have nothing non-trivial to do at lookup time and lookup_mnt() will DTRT with minimal fuss. > Hmm. If you use the same dentry for mnt_root as for mnt_mountpoint, how do > you avoid the infinite loop in follow_mount? Another magic special case? See above. And yes, I'd love to get a separate dentry (and inode, while we are at it). But I don't see how to do that. One note: we might split lookup_mnt() in two functions - one for the middle of lookup, another for final (non-triggering) step. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-27 4:30 ` viro @ 2004-08-27 4:42 ` Linus Torvalds 2004-08-27 4:46 ` Linus Torvalds 0 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-27 4:42 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > > Or do you have a cunning plan? > > a) set ->d_mount to 1 and set DCACHE_HYBRID at ->lookup() time > b) in lookup_mnt(): > if (unlikely(dentry has DCACHE_HYBRID)) { > if (linkely(dentry->d_mount == 1)) > goto New; > } > do the normal series of hash lookups > if (likely(dentry doesn't have DCACHE_HYBRID)) > return; > if (vfsmount is marked as hybrid && dentry == vfsmount->mnt_dentry) > return; Hmm.. Here we shouldn't have a vfsmount. If we have one, that means that we found it in the hashes, and we should just have returned it immediately (whether hybrid or not doesn't matter at that point). Also, I don't see the magic with "dentry->d_mount == 1", since we may have allowed the dentry to be mounted on for real at one of the places, without losing it's "hydridity" (it's not a word, but dammit, it should be one) for that. Is that part just an optimization to avoid the hash lookups? > New: > allocate new vfsmount and mark it as hybrid > attach it on top of our one and go there > > Does that qualify as cunning? With that approach we have nothing non-trivial > to do at lookup time and lookup_mnt() will DTRT with minimal fuss. Ok, I think it qualifies as cunning enough. It has the nice property of not apparently needing any vfsmount-related code at all at lookup() time. Which is a bonus. > > Hmm. If you use the same dentry for mnt_root as for mnt_mountpoint, how do > > you avoid the infinite loop in follow_mount? Another magic special case? > > See above. And yes, I'd love to get a separate dentry (and inode, while > we are at it). But I don't see how to do that. > > One note: we might split lookup_mnt() in two functions - one for the > middle of lookup, another for final (non-triggering) step. Yes. We don't necessarily even want to trigger the special mountpoint case at all if we don't need it - which normal usage probably wouldn't ever trigger. So I agree 100% - that way I think you actually allocate the vfsmount until somebody really starts doing the "new things". Was that your plan? Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-27 4:42 ` Linus Torvalds @ 2004-08-27 4:46 ` Linus Torvalds 0 siblings, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-27 4:46 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > > > if (vfsmount is marked as hybrid && dentry == vfsmount->mnt_dentry) > > return; > > Hmm.. Here we shouldn't have a vfsmount. If we have one, that means that > we found it in the hashes, and we should just have returned it > immediately (whether hybrid or not doesn't matter at that point). Ahh, I misunderstood you. You mean that if the _old_ vfsmount was already a hybrid one, we won't try to recurse if we didn't find any other, so we return NULL at that point. Got it. Yes, I bow before your cunningness. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:04 ` Linus Torvalds @ 2004-08-27 8:56 ` Anton Altaparmakov 2004-08-27 8:56 ` Anton Altaparmakov 1 sibling, 0 replies; 1835+ messages in thread From: Anton Altaparmakov @ 2004-08-27 8:56 UTC (permalink / raw) To: Linus Torvalds Cc: viro, Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Thu, 2004-08-26 at 23:04, Linus Torvalds wrote: > [ This is quite possibly just impossible and buggy, but here's my > implementation notes. You asked for them. ] > On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > All right, let's see where that would take us. > > 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off > > it. MNT_DETACH will have no problems, but normal umount() is a different > > story. Note that it's not just hybrid-related problem - implementing the > > mount traps will cause the same kind of trouble, > > Don't allow umount. It's not something the user can unmount - the mount is > "implied" in the file. > > > 4) OK, we have those hybrids and want to create vfsmounts when crossing a > > mountpoint. When do they go away, anyway? When we don't reference them > > anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching > > happens explicitly - outside of the "dropping the final reference" path. > > Might become a locking issue. > > Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously > clossed something over when I blathered about the "create the vfsmount on > the fly" thing above ;) If it is created on the fly, it should be "easy" to destroy on the fly using time-based expiry, i.e. a kernel daemon going over all of those beasts every X seconds (X = 5 perhaps?) and doing something like: for (each vfsmount) { lock_vfsmount(vfsmount); if (MOUNT_IS_BUSY(vfsmount)) { unlock_vfsmount(vfsmount); continue; } if (current_time() < (vfsmount->last_used_time + vfsmount->expire_after)) { unlock_vfsmount(vfsmount); continue; } destroy_locked_vfsmount(vfsmount); } Wouldn't that work? > > 10) how do we deal with directories, anyway? Mixing "attributes" with > > normal directory contents is going to be fun, what with lseek() insanity. > > You couldn't get at the attributes that way anyway, so I think the point > is moot. The "real" directory always takes over. > > Crazy people could try to just use the regular "xattrs" interfaces if they > really want attributes on directories. You wouldn't ever be able to use > the "easy" one. But that defeats the whole point of the hybrid objects! We might as well just keep the xattrs interface and throw away the new one if we will have to keep the old one anyway so that we can do named streams/attributes on directories. Windows (and other OS?) certainly do allow them and do use them on directories as well as files... > > 11) if we go for your "here's stuff that belongs in device node viewed > > as directory", how would that play with fs metadata exporters? Again, > > due to the insanity of lseek() on directories it's *very* hard to deal > > with unions, when parts of directory come from different chunks of code. > > Don't go there. See above. Directories would be just plain directories, > you could never see their metadata. Same goes for at least symlinks, and > possibly other filetypes too (ie at least initially, a block or character > special device will just take over the whole "file_operations", which > includes "readdir", so it's actually hard to have the filesystem do > anything about those). Ah, but at least in other OS (Windows is the one I know about) directories _also_ have the same semantics as files with respect to named streams/attributes. Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids @ 2004-08-27 8:56 ` Anton Altaparmakov 0 siblings, 0 replies; 1835+ messages in thread From: Anton Altaparmakov @ 2004-08-27 8:56 UTC (permalink / raw) To: Linus Torvalds Cc: viro, Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Thu, 2004-08-26 at 23:04, Linus Torvalds wrote: > [ This is quite possibly just impossible and buggy, but here's my > implementation notes. You asked for them. ] > On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > All right, let's see where that would take us. > > 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off > > it. MNT_DETACH will have no problems, but normal umount() is a different > > story. Note that it's not just hybrid-related problem - implementing the > > mount traps will cause the same kind of trouble, > > Don't allow umount. It's not something the user can unmount - the mount is > "implied" in the file. > > > 4) OK, we have those hybrids and want to create vfsmounts when crossing a > > mountpoint. When do they go away, anyway? When we don't reference them > > anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching > > happens explicitly - outside of the "dropping the final reference" path. > > Might become a locking issue. > > Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously > clossed something over when I blathered about the "create the vfsmount on > the fly" thing above ;) If it is created on the fly, it should be "easy" to destroy on the fly using time-based expiry, i.e. a kernel daemon going over all of those beasts every X seconds (X = 5 perhaps?) and doing something like: for (each vfsmount) { lock_vfsmount(vfsmount); if (MOUNT_IS_BUSY(vfsmount)) { unlock_vfsmount(vfsmount); continue; } if (current_time() < (vfsmount->last_used_time + vfsmount->expire_after)) { unlock_vfsmount(vfsmount); continue; } destroy_locked_vfsmount(vfsmount); } Wouldn't that work? > > 10) how do we deal with directories, anyway? Mixing "attributes" with > > normal directory contents is going to be fun, what with lseek() insanity. > > You couldn't get at the attributes that way anyway, so I think the point > is moot. The "real" directory always takes over. > > Crazy people could try to just use the regular "xattrs" interfaces if they > really want attributes on directories. You wouldn't ever be able to use > the "easy" one. But that defeats the whole point of the hybrid objects! We might as well just keep the xattrs interface and throw away the new one if we will have to keep the old one anyway so that we can do named streams/attributes on directories. Windows (and other OS?) certainly do allow them and do use them on directories as well as files... > > 11) if we go for your "here's stuff that belongs in device node viewed > > as directory", how would that play with fs metadata exporters? Again, > > due to the insanity of lseek() on directories it's *very* hard to deal > > with unions, when parts of directory come from different chunks of code. > > Don't go there. See above. Directories would be just plain directories, > you could never see their metadata. Same goes for at least symlinks, and > possibly other filetypes too (ie at least initially, a block or character > special device will just take over the whole "file_operations", which > includes "readdir", so it's actually hard to have the filesystem do > anything about those). Ah, but at least in other OS (Windows is the one I know about) directories _also_ have the same semantics as files with respect to named streams/attributes. Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-27 8:56 ` Anton Altaparmakov (?) @ 2004-08-27 9:00 ` Christoph Hellwig -1 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-27 9:00 UTC (permalink / raw) To: Anton Altaparmakov Cc: Linus Torvalds, viro, Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Fri, Aug 27, 2004 at 09:56:39AM +0100, Anton Altaparmakov wrote: > If it is created on the fly, it should be "easy" to destroy on the fly > using time-based expiry, i.e. a kernel daemon going over all of those > beasts every X seconds (X = 5 perhaps?) and doing something like: > > for (each vfsmount) { > lock_vfsmount(vfsmount); > if (MOUNT_IS_BUSY(vfsmount)) { > unlock_vfsmount(vfsmount); > continue; > } > if (current_time() < (vfsmount->last_used_time + > vfsmount->expire_after)) { > unlock_vfsmount(vfsmount); > continue; > } > destroy_locked_vfsmount(vfsmount); > } > > Wouldn't that work? That would work for a low number of them. But with Hans' "visions" we'd have a damn lot of them at which point this isn't really scalable. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:28 ` Denis Vlasenko (?) (?) @ 2004-08-26 19:36 ` Jonathan Abbey 2004-08-26 20:16 ` Jamie Lokier -1 siblings, 1 reply; 1835+ messages in thread From: Jonathan Abbey @ 2004-08-26 19:36 UTC (permalink / raw) To: Denis Vlasenko Cc: Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 930 bytes --] On Thu, Aug 26, 2004 at 09:28:41PM +0300, Denis Vlasenko wrote: | | Is it possible to sufficiently hide "dirs inside files" | so that old tools will be unable to see them? | | I just checked: | | ls -d /foo does lstat64("/foo", ...) | ls -d /foo/ does lstat64("/foo", ...) | but | ls -d /foo/. does lstat64("/foo/.", ...) | | Will it work out if "dir inside file" will only be visible when referred as "file/."? I'm used to using ls symlink/. to get ls to show me the directory on the far side of a symbolic link. That's a pretty analagous case to the one we're discussing here, I think? Jon -- ------------------------------------------------------------------------------- Jonathan Abbey jonabbey@arlut.utexas.edu Applied Research Laboratories The University of Texas at Austin GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg [-- Attachment #2: Type: application/pgp-signature, Size: 185 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:36 ` silent semantic changes with reiser4 Jonathan Abbey @ 2004-08-26 20:16 ` Jamie Lokier 2004-08-26 20:25 ` Christophe Saout 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 20:16 UTC (permalink / raw) To: Jonathan Abbey Cc: Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jonathan Abbey wrote: > | Will it work out if "dir inside file" will only be visible when > referred as "file/."? > > I'm used to using ls symlink/. to get ls to show me the directory on > the far side of a symbolic link. That's a pretty analagous case to > the one we're discussing here, I think? By the way, do symlinks have metadata? Where do you find it? :) -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:16 ` Jamie Lokier @ 2004-08-26 20:25 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 20:25 UTC (permalink / raw) To: Jamie Lokier Cc: Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 733 bytes --] Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier: > > | Will it work out if "dir inside file" will only be visible when > > referred as "file/."? > > > > I'm used to using ls symlink/. to get ls to show me the directory on > > the far side of a symbolic link. That's a pretty analagous case to > > the one we're discussing here, I think? > > By the way, do symlinks have metadata? Where do you find it? :) Oops. :) At least in reiser4 they don't have, or at least you can't access them. ln -s foo bar; cd bar/metas shows me the content of foo/metas. symlinks are special anyway. Their rights are 777. The only thing they can have is an owner. No chance to do a symlink/metas/uid then. Hmm. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 20:25 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 20:25 UTC (permalink / raw) To: Jamie Lokier Cc: Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 733 bytes --] Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier: > > | Will it work out if "dir inside file" will only be visible when > > referred as "file/."? > > > > I'm used to using ls symlink/. to get ls to show me the directory on > > the far side of a symbolic link. That's a pretty analagous case to > > the one we're discussing here, I think? > > By the way, do symlinks have metadata? Where do you find it? :) Oops. :) At least in reiser4 they don't have, or at least you can't access them. ln -s foo bar; cd bar/metas shows me the content of foo/metas. symlinks are special anyway. Their rights are 777. The only thing they can have is an owner. No chance to do a symlink/metas/uid then. Hmm. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:25 ` Christophe Saout (?) @ 2004-08-26 21:45 ` Nikita Danilov 2004-08-26 21:48 ` Christophe Saout -1 siblings, 1 reply; 1835+ messages in thread From: Nikita Danilov @ 2004-08-26 21:45 UTC (permalink / raw) To: Christophe Saout Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout writes: > Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier: > > > > | Will it work out if "dir inside file" will only be visible when > > > referred as "file/."? > > > > > > I'm used to using ls symlink/. to get ls to show me the directory on > > > the far side of a symbolic link. That's a pretty analagous case to > > > the one we're discussing here, I think? > > > > By the way, do symlinks have metadata? Where do you find it? :) > > Oops. :) > > At least in reiser4 they don't have, or at least you can't access them. They do. > ln -s foo bar; cd bar/metas shows me the content of foo/metas. That's because lookup for "bar" performs symlink resolution. > > symlinks are special anyway. Their rights are 777. The only thing they Symlink is represented by normal inode in UNIX. It has full set of attributes. It can even be hard-linked. > can have is an owner. No chance to do a symlink/metas/uid then. Hmm. > Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:45 ` Nikita Danilov @ 2004-08-26 21:48 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 21:48 UTC (permalink / raw) To: Nikita Danilov Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 453 bytes --] Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > At least in reiser4 they don't have, or at least you can't access them. > > They do. > > > ln -s foo bar; cd bar/metas shows me the content of foo/metas. > > That's because lookup for "bar" performs symlink resolution. So I can't access them and it is pointless. ;-) BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think this makes sense. :) [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 21:48 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 21:48 UTC (permalink / raw) To: Nikita Danilov Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 453 bytes --] Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > At least in reiser4 they don't have, or at least you can't access them. > > They do. > > > ln -s foo bar; cd bar/metas shows me the content of foo/metas. > > That's because lookup for "bar" performs symlink resolution. So I can't access them and it is pointless. ;-) BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think this makes sense. :) [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:48 ` Christophe Saout (?) @ 2004-08-26 22:17 ` Emil Larsson -1 siblings, 0 replies; 1835+ messages in thread From: Emil Larsson @ 2004-08-26 22:17 UTC (permalink / raw) To: Christophe Saout Cc: Nikita Danilov, Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: >Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > > >> > At least in reiser4 they don't have, or at least you can't access them. >> >>They do. >> >> > ln -s foo bar; cd bar/metas shows me the content of foo/metas. >> >>That's because lookup for "bar" performs symlink resolution. >> >> > >So I can't access them and it is pointless. ;-) > >BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think >this makes sense. :) > > > I prefer this one: hepburn:/# mount /dev/sda3 on / type reiser4 (rw) hepburn:/# touch apa hepburn:/# chmod +x apa hepburn:/# cd apa hepburn:/apa# ls ls: reading directory .: Not a directory So there is already file-as-dir with legacy "ls" - and it works just like people seem to want it to do with legacy apps. Not at all. /Emil ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:48 ` Christophe Saout (?) (?) @ 2004-08-26 22:21 ` Nikita Danilov 2004-08-26 22:29 ` Christophe Saout 2004-08-26 23:35 ` Hans Reiser -1 siblings, 2 replies; 1835+ messages in thread From: Nikita Danilov @ 2004-08-26 22:21 UTC (permalink / raw) To: Christophe Saout Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout writes: > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > > > At least in reiser4 they don't have, or at least you can't access them. > > > > They do. > > > > > ln -s foo bar; cd bar/metas shows me the content of foo/metas. > > > > That's because lookup for "bar" performs symlink resolution. > > So I can't access them and it is pointless. ;-) > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think > this makes sense. :) Why? foo/metas is a file system object just like foo. It has owner, permission bits, so access to its meta-data should be provided, and uniform way to provide access to the file system object meta-data is to have these little magic files inside metas directory, which is a file system object just like metas. It has owner^@^@^@^@*** - Lisp stack overflow. RESET Nikita. > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:21 ` Nikita Danilov @ 2004-08-26 22:29 ` Christophe Saout 2004-08-26 23:35 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 22:29 UTC (permalink / raw) To: Nikita Danilov Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1323 bytes --] Am Freitag, den 27.08.2004, 02:21 +0400 schrieb Nikita Danilov: > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think > > this makes sense. :) > > Why? foo/metas is a file system object just like foo. It has owner, > permission bits, so access to its meta-data should be provided, and > uniform way to provide access to the file system object meta-data is to > have these little magic files inside metas directory, which is a file > system object just like metas. It has owner^@^@^@^@*** - Lisp stack > overflow. RESET Yes, of course. But who's going to need that? The meta files are there to describe regular unix files. They're just exported using a file system name space. I don't think we need metadata to describe metadata. Assuming someone adds a directory inside a file where user can add some attributes/streams/resource forks. As Linus said, they are just attributes, even if they seem to behave like normal files, they're not, they belong to the real file. Adding attributes to attributes to attributes? That would be like adding ACLS to ACLS. Hello the madness. ;-) There really should be a point where you can't recurse further. Assume there is a backup application that always tries to open every dentry it finds as directory and then recurse. Oops. :-) [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 22:29 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 22:29 UTC (permalink / raw) To: Nikita Danilov Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1323 bytes --] Am Freitag, den 27.08.2004, 02:21 +0400 schrieb Nikita Danilov: > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think > > this makes sense. :) > > Why? foo/metas is a file system object just like foo. It has owner, > permission bits, so access to its meta-data should be provided, and > uniform way to provide access to the file system object meta-data is to > have these little magic files inside metas directory, which is a file > system object just like metas. It has owner^@^@^@^@*** - Lisp stack > overflow. RESET Yes, of course. But who's going to need that? The meta files are there to describe regular unix files. They're just exported using a file system name space. I don't think we need metadata to describe metadata. Assuming someone adds a directory inside a file where user can add some attributes/streams/resource forks. As Linus said, they are just attributes, even if they seem to behave like normal files, they're not, they belong to the real file. Adding attributes to attributes to attributes? That would be like adding ACLS to ACLS. Hello the madness. ;-) There really should be a point where you can't recurse further. Assume there is a backup application that always tries to open every dentry it finds as directory and then recurse. Oops. :-) [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:21 ` Nikita Danilov @ 2004-08-26 23:35 ` Hans Reiser 2004-08-26 23:35 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:35 UTC (permalink / raw) To: Nikita Danilov Cc: Christophe Saout, Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Nikita Danilov wrote: >Christophe Saout writes: > > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > > > > > At least in reiser4 they don't have, or at least you can't access them. > > > > > > They do. > > > > > > > ln -s foo bar; cd bar/metas shows me the content of foo/metas. > > > > > > That's because lookup for "bar" performs symlink resolution. > > > > So I can't access them and it is pointless. ;-) > > > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think > > this makes sense. :) > >Why? foo/metas is a file system object just like foo. It has owner, >permission bits, so access to its meta-data should be provided, and >uniform way to provide access to the file system object meta-data is to >have these little magic files inside metas directory, which is a file >system object just like metas. It has owner^@^@^@^@*** - Lisp stack >overflow. RESET > >Nikita. > > > >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > I think Christophe is a bit right here. While in general having meta-meta objects makes sense, in this particular instance, I don't see the functional need for it. Can you supply an example of where it would be useful? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:35 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:35 UTC (permalink / raw) To: Nikita Danilov Cc: Christophe Saout, Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Nikita Danilov wrote: >Christophe Saout writes: > > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > > > > > At least in reiser4 they don't have, or at least you can't access them. > > > > > > They do. > > > > > > > ln -s foo bar; cd bar/metas shows me the content of foo/metas. > > > > > > That's because lookup for "bar" performs symlink resolution. > > > > So I can't access them and it is pointless. ;-) > > > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think > > this makes sense. :) > >Why? foo/metas is a file system object just like foo. It has owner, >permission bits, so access to its meta-data should be provided, and >uniform way to provide access to the file system object meta-data is to >have these little magic files inside metas directory, which is a file >system object just like metas. It has owner^@^@^@^@*** - Lisp stack >overflow. RESET > >Nikita. > > > >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > I think Christophe is a bit right here. While in general having meta-meta objects makes sense, in this particular instance, I don't see the functional need for it. Can you supply an example of where it would be useful? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:35 ` Hans Reiser (?) @ 2004-08-27 2:00 ` Stephen Wille Padnos -1 siblings, 0 replies; 1835+ messages in thread From: Stephen Wille Padnos @ 2004-08-27 2:00 UTC (permalink / raw) To: Hans Reiser Cc: Nikita Danilov, Christophe Saout, Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Hans Reiser wrote: > Nikita Danilov wrote: > >> Christophe Saout writes: >> > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: >> > > > > At least in reiser4 they don't have, or at least you can't >> access them. >> > > > > They do. >> > > > > > ln -s foo bar; cd bar/metas shows me the content of >> foo/metas. >> > > > > That's because lookup for "bar" performs symlink resolution. >> > > So I can't access them and it is pointless. ;-) >> > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I >> don't think >> > this makes sense. :) >> >> Why? foo/metas is a file system object just like foo. It has owner, >> permission bits, so access to its meta-data should be provided, and >> uniform way to provide access to the file system object meta-data is to >> have these little magic files inside metas directory, which is a file >> system object just like metas. It has owner^@^@^@^@*** - Lisp stack >> overflow. RESET >> >> Nikita. >> >> > - >> To unsubscribe from this list: send the line "unsubscribe >> linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ >> >> >> >> > I think Christophe is a bit right here. While in general having > meta-meta objects makes sense, in this particular instance, I don't > see the functional need for it. Can you supply an example of where it > would be useful? One example is to supply hints about what to do with the attribute if the file is copied / moved. Several types of attributes have been mentioned: 1) Those that are essentially a cached object, which can be re-derived from the source file (e.g. thumbnail image) 2) Those that become invalid if the source file changes (e.g. hash value) 3) Those that should be maintained on file copy (e.g. author, copyright, etc) 4) Those that may not want to be copied with the file (e.g. security keys or the like) etc. This could be accomplished without meta-meta information (like permissions), but that wouldn't be very extensible, would it :) - Steve ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:20 ` Linus Torvalds 2004-08-26 17:27 ` Jamie Lokier 2004-08-26 17:57 ` Rik van Riel @ 2004-08-27 8:08 ` Hans Reiser 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:08 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > > >No no. The stream you get from a directory is totally _independent_ of the >contents of the directory. > but the user can link filename/metas/body to filename/metas/tar, if we implement a tar plugin, if it happens that the user wants it to be totally dependent. > Anything else would be pointless. > > Linus >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:16 ` Rik van Riel @ 2004-08-26 17:40 ` Diego Calleja -1 siblings, 0 replies; 1835+ messages in thread From: Diego Calleja @ 2004-08-26 17:40 UTC (permalink / raw) To: Rik van Riel Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió: > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? /bin could be separated (like linus said) but cat /bin/.compound could do it. This is the /etc/passwd Hans' example, I think: "If /new_syntax_access_path/big_directory_of_small_files/.glued is a plugin that aggregates every file in big_directory_of_small_files with a delimiter separating every file within the aggregation, then one can simply type emacs /new_syntax_access_path/big_directory_of_small_files/.glued, and the filesystem has done all the work emacs needs to be effective at this. Not a line of emacs needs to be changed." ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 17:40 ` Diego Calleja 0 siblings, 0 replies; 1835+ messages in thread From: Diego Calleja @ 2004-08-26 17:40 UTC (permalink / raw) To: Rik van Riel Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió: > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? /bin could be separated (like linus said) but cat /bin/.compound could do it. This is the /etc/passwd Hans' example, I think: "If /new_syntax_access_path/big_directory_of_small_files/.glued is a plugin that aggregates every file in big_directory_of_small_files with a delimiter separating every file within the aggregation, then one can simply type emacs /new_syntax_access_path/big_directory_of_small_files/.glued, and the filesystem has done all the work emacs needs to be effective at this. Not a line of emacs needs to be changed." - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:40 ` Diego Calleja (?) @ 2004-08-26 17:55 ` Christophe Saout -1 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 17:55 UTC (permalink / raw) To: Diego Calleja Cc: Rik van Riel, jamie, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 07:40:10PM +0200, Diego Calleja wrote: > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > /bin could be separated (like linus said) but cat /bin/.compound could do > it. This is the /etc/passwd Hans' example, I think: Yes, but what about locking? If we have compound files with individually accessible components a lock on the compound files could lock its components at the same time. But what with /usr/.compound? Is it read-only? What if you truncate it? Does it atomically delete the whole /usr directory tree? What about reading? It has to go through all these independent files, what if they change while you are reading the compound file? Does it lock everything? Point-in-time snapshot? If reiser4 can't do this. ;-) I don't say that it's impossible to do but I think it's a lot harder than doing it the other way round. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:40 ` Diego Calleja @ 2004-08-26 17:59 ` Rik van Riel -1 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 17:59 UTC (permalink / raw) To: Diego Calleja Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Diego Calleja wrote: > El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió: > > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > /bin could be separated (like linus said) but cat /bin/.compound could > do it. This is the /etc/passwd Hans' example, I think: Arghhhh. I wrote it down to ridicule the idea and now people are taking it seriously ;( It should be obvious enough that anything depending on the kernel parsing file contents will lead to problems. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 17:59 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 17:59 UTC (permalink / raw) To: Diego Calleja Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Diego Calleja wrote: > El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió: > > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > /bin could be separated (like linus said) but cat /bin/.compound could > do it. This is the /etc/passwd Hans' example, I think: Arghhhh. I wrote it down to ridicule the idea and now people are taking it seriously ;( It should be obvious enough that anything depending on the kernel parsing file contents will lead to problems. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:59 ` Rik van Riel (?) @ 2004-08-26 18:20 ` Jamie Lokier 2004-08-27 8:06 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 18:20 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > /bin could be separated (like linus said) but cat /bin/.compound could > > do it. This is the /etc/passwd Hans' example, I think: > > Arghhhh. I wrote it down to ridicule the idea and now people > are taking it seriously ;( > > It should be obvious enough that anything depending on the > kernel parsing file contents will lead to problems. This is one case where the kernel _isn't_ parsing file contents, but I agree it's ridiculous :) -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:20 ` Jamie Lokier @ 2004-08-27 8:06 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie Lokier wrote: >Rik van Riel wrote: > > >>>/bin could be separated (like linus said) but cat /bin/.compound could >>>do it. This is the /etc/passwd Hans' example, I think: >>> >>> >>Arghhhh. I wrote it down to ridicule the idea and now people >>are taking it seriously ;( >> >>It should be obvious enough that anything depending on the >>kernel parsing file contents will lead to problems. >> >> > >This is one case where the kernel _isn't_ parsing file contents, >but I agree it's ridiculous :) > >-- Jamie > > Umm, no, when you write the glued /etc/passwd file it parses the delimiters in it to determine what subfile to put the parts of /etc/passwd into. Inheritance with delimiters is the best way to make emacs effective for editing lots of small files, Stallman was right and deserves credit for it. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:06 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie Lokier wrote: >Rik van Riel wrote: > > >>>/bin could be separated (like linus said) but cat /bin/.compound could >>>do it. This is the /etc/passwd Hans' example, I think: >>> >>> >>Arghhhh. I wrote it down to ridicule the idea and now people >>are taking it seriously ;( >> >>It should be obvious enough that anything depending on the >>kernel parsing file contents will lead to problems. >> >> > >This is one case where the kernel _isn't_ parsing file contents, >but I agree it's ridiculous :) > >-- Jamie > > Umm, no, when you write the glued /etc/passwd file it parses the delimiters in it to determine what subfile to put the parts of /etc/passwd into. Inheritance with delimiters is the best way to make emacs effective for editing lots of small files, Stallman was right and deserves credit for it. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:44 ` Jamie Lokier 2004-08-26 15:54 ` Rik van Riel @ 2004-08-27 2:14 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 2:14 UTC (permalink / raw) To: Jamie Lokier Cc: Christophe Saout, Rik van Riel, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie Lokier wrote: > >Why not just install .tgz or .zip files of entire binary packages, and >let the filesystem unpack the bits that are needed when they're >needed. Some platforms (i.e. Java) do this already with .jar files >and it's quite popular. This would extend it in a versatile way. > > I like this notion of a plugin that would install a package if one of the package's executables was invoked by its name. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:12 ` Rik van Riel @ 2004-08-26 15:19 ` Denis Vlasenko -1 siblings, 0 replies; 1835+ messages in thread From: Denis Vlasenko @ 2004-08-26 15:19 UTC (permalink / raw) To: Rik van Riel Cc: Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 17:12, Rik van Riel wrote: > On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > I think Hans is not planning turning old "file is a stream of bytes" > > into eight-stream octopus. One stream will remain as a 'main' one, > > which contains actual data. Others will keep metadata, etc... > > This is exactly what the Samba people want, though. > > Office suites can store a document with embedded images > and spread sheets "easily" by putting the text, the > images and spread sheets all in different file streams. Kinda far reaching. It's hard to assess whether that is good or bad idea. Can we start small, like putting metadata (e.g. ACLs) into these streams? -- vda ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 15:19 ` Denis Vlasenko 0 siblings, 0 replies; 1835+ messages in thread From: Denis Vlasenko @ 2004-08-26 15:19 UTC (permalink / raw) To: Rik van Riel Cc: Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 17:12, Rik van Riel wrote: > On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > I think Hans is not planning turning old "file is a stream of bytes" > > into eight-stream octopus. One stream will remain as a 'main' one, > > which contains actual data. Others will keep metadata, etc... > > This is exactly what the Samba people want, though. > > Office suites can store a document with embedded images > and spread sheets "easily" by putting the text, the > images and spread sheets all in different file streams. Kinda far reaching. It's hard to assess whether that is good or bad idea. Can we start small, like putting metadata (e.g. ACLs) into these streams? -- vda ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:19 ` Denis Vlasenko (?) @ 2004-08-29 14:30 ` Alan Cox 2004-08-29 17:42 ` Jeremy Allison ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Alan Cox @ 2004-08-29 14:30 UTC (permalink / raw) To: Denis Vlasenko Cc: Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list On Iau, 2004-08-26 at 16:19, Denis Vlasenko wrote: > > Office suites can store a document with embedded images > > and spread sheets "easily" by putting the text, the > > images and spread sheets all in different file streams. > > Kinda far reaching. It's hard to assess whether that is good or bad idea. > Can we start small, like putting metadata (e.g. ACLs) into these streams? Openoffice does this in user space and the user space vfs code desktops use can handle zips so this "just works" already including over NFS, unlike a kernel proposed method. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 14:30 ` Alan Cox @ 2004-08-29 17:42 ` Jeremy Allison 2004-08-29 18:41 ` Trond Myklebust 2004-09-01 20:25 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Jeremy Allison @ 2004-08-29 17:42 UTC (permalink / raw) To: Alan Cox Cc: Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list On Sun, Aug 29, 2004 at 03:30:03PM +0100, Alan Cox wrote: > > Openoffice does this in user space and the user space vfs code desktops > use can handle zips so this "just works" already including over NFS, > unlike a kernel proposed method. Hurrah for OpenOffice. Now all you need to do is to persuade Microsoft to store Word files in the same format...... Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 14:30 ` Alan Cox @ 2004-08-29 18:41 ` Trond Myklebust 2004-08-29 18:41 ` Trond Myklebust 2004-09-01 20:25 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Trond Myklebust @ 2004-08-29 18:41 UTC (permalink / raw) To: Alan Cox Cc: Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list På su , 29/08/2004 klokka 10:30, skreiv Alan Cox: > Openoffice does this in user space and the user space vfs code desktops > use can handle zips so this "just works" already including over NFS, > unlike a kernel proposed method. NFS is not the roadblock here. Sure v2 and v3 support would be impossible to do, but since this is all 2.6 work anyway, people have the option of using NFSv4, which has full support for streams... NFSv4 has an OPENATTR call which acts on files to return a filehandle that works fine with both READDIR and LOOKUP, so if a VFS interface for streams existed, we could code up full support tomorrow. As it is, we're having to shoehorn this into the xattr interface. 8-( Cheers, Trond ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 18:41 ` Trond Myklebust 0 siblings, 0 replies; 1835+ messages in thread From: Trond Myklebust @ 2004-08-29 18:41 UTC (permalink / raw) To: Alan Cox Cc: Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list På su , 29/08/2004 klokka 10:30, skreiv Alan Cox: > Openoffice does this in user space and the user space vfs code desktops > use can handle zips so this "just works" already including over NFS, > unlike a kernel proposed method. NFS is not the roadblock here. Sure v2 and v3 support would be impossible to do, but since this is all 2.6 work anyway, people have the option of using NFSv4, which has full support for streams... NFSv4 has an OPENATTR call which acts on files to return a filehandle that works fine with both READDIR and LOOKUP, so if a VFS interface for streams existed, we could code up full support tomorrow. As it is, we're having to shoehorn this into the xattr interface. 8-( Cheers, Trond ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 18:41 ` Trond Myklebust (?) @ 2004-08-29 19:38 ` Jeremy Allison 2004-09-01 20:19 ` Jamie Lokier -1 siblings, 1 reply; 1835+ messages in thread From: Jeremy Allison @ 2004-08-29 19:38 UTC (permalink / raw) To: Trond Myklebust Cc: Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Sun, Aug 29, 2004 at 02:41:04PM -0400, Trond Myklebust wrote: > > NFSv4 has an OPENATTR call which acts on files to return a filehandle > that works fine with both READDIR and LOOKUP, so if a VFS interface for > streams existed, we could code up full support tomorrow. > As it is, we're having to shoehorn this into the xattr interface. 8-( Yeah, that's what I'm really trying to avoid for Samba. We're currently using the xattr interface for DOS attributes, and soon for NT ACL support, but it doesn't really fit as a streams interface (no lseek() support) although someone in HP coded up a test interface to NT streams that map into xattrs (most of the streams in a Word file for instance are quite small) - but I really don't want to have to do this :-). Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 19:38 ` Jeremy Allison @ 2004-09-01 20:19 ` Jamie Lokier 2004-09-01 20:26 ` Jeremy Allison 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 20:19 UTC (permalink / raw) To: Jeremy Allison Cc: Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list Jeremy Allison wrote: > (most of the streams in a Word file for instance are quite small) Streams in a Word file? Are you saying that when I copy a .doc file onto my Linux box and off, I lose part of a Word document? -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:19 ` Jamie Lokier @ 2004-09-01 20:26 ` Jeremy Allison 2004-09-01 20:31 ` Jamie Lokier 2004-09-02 9:48 ` Alan Cox 0 siblings, 2 replies; 1835+ messages in thread From: Jeremy Allison @ 2004-09-01 20:26 UTC (permalink / raw) To: Jamie Lokier Cc: Jeremy Allison, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, Sep 01, 2004 at 09:19:45PM +0100, Jamie Lokier wrote: > Jeremy Allison wrote: > > (most of the streams in a Word file for instance are quite small) > > Streams in a Word file? Yep. > Are you saying that when I copy a .doc file onto my Linux box and off, > I lose part of a Word document? Right now no, because when Samba refuses the stream open, Word falls back into a "tar"-like mode where it linearises the streams into the data (it's a legacy mode for storing data on a FAT drive, not an NTFS drive). However, the problem is that no currently supported Microsoft OS doesn't have streams-capable NTFS support. This means that in a future MS-Office revision, this backwards support may be broken by accident or by design (less likely, Microsoft really don't do that kind of thing without very high level requests :-) and no testers at Microsoft will notice (because they only test against MS servers). Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:26 ` Jeremy Allison @ 2004-09-01 20:31 ` Jamie Lokier 2004-09-01 20:35 ` Jeremy Allison 2004-09-01 21:09 ` David Lang 2004-09-02 9:48 ` Alan Cox 1 sibling, 2 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 20:31 UTC (permalink / raw) To: Jeremy Allison Cc: Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list Jeremy Allison wrote: > > Streams in a Word file? > > Yep. > > > Are you saying that when I copy a .doc file onto my Linux box and off, > > I lose part of a Word document? > > Right now no, because when Samba refuses the stream open, Word falls > back into a "tar"-like mode where it linearises the streams into the > data (it's a legacy mode for storing data on a FAT drive, not an NTFS > drive). However, the problem is that no currently supported Microsoft > OS doesn't have streams-capable NTFS support. I meant when I copy not using Samba. For example, I copy the .doc file in Windows NT to an FTP server. Does the FTP operation magically linearise the .doc streams on demand? Or does FTP lose part of the Word document? -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:31 ` Jamie Lokier @ 2004-09-01 20:35 ` Jeremy Allison 2004-09-01 20:47 ` Jamie Lokier 2004-09-01 22:05 ` Hubert Chan 2004-09-01 21:09 ` David Lang 1 sibling, 2 replies; 1835+ messages in thread From: Jeremy Allison @ 2004-09-01 20:35 UTC (permalink / raw) To: Jamie Lokier Cc: Jeremy Allison, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, Sep 01, 2004 at 09:31:01PM +0100, Jamie Lokier wrote: > > I meant when I copy not using Samba. For example, I copy the .doc > file in Windows NT to an FTP server. > > Does the FTP operation magically linearise the .doc streams on demand? > > Or does FTP lose part of the Word document? Good question. It depends if the Microsoft ftp client is streams-aware, and understands the Microsoft OLE structured storage format and will do the linearisation on demand or not. I must confess I haven't tested this, as I don't ever run Windows other than on vmware sessions for Samba testing these days :-). Probably a non-Microsoft ftp client would lose part of the word doc. Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:35 ` Jeremy Allison @ 2004-09-01 20:47 ` Jamie Lokier 2004-09-01 20:51 ` Jeremy Allison 2004-09-01 22:05 ` Hubert Chan 1 sibling, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 20:47 UTC (permalink / raw) To: Jeremy Allison Cc: Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list Jeremy Allison wrote: > > I meant when I copy not using Samba. For example, I copy the .doc > > file in Windows NT to an FTP server. > > > > Does the FTP operation magically linearise the .doc streams on demand? > > Or does FTP lose part of the Word document? > > Good question. It depends if the Microsoft ftp client is streams-aware, > and understands the Microsoft OLE structured storage format and will do > the linearisation on demand or not. I must confess I haven't tested this, > as I don't ever run Windows other than on vmware sessions for Samba testing > these days :-). > > Probably a non-Microsoft ftp client would lose part of the word doc. So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd will _all_ lose part of a Word document when they handle it on a Window box? Ouch! The only sensible implementation I can imagine would be if the OS linearised multi-stream Word documents into the non-stream format automatically for all programs which don't know about streams. Which is of course what I would like to implement for Linux... - Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:47 ` Jamie Lokier @ 2004-09-01 20:51 ` Jeremy Allison 2004-09-01 21:35 ` Lee Revell ` (2 more replies) 0 siblings, 3 replies; 1835+ messages in thread From: Jeremy Allison @ 2004-09-01 20:51 UTC (permalink / raw) To: Jamie Lokier Cc: Jeremy Allison, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote: > Jeremy Allison wrote: > > > I meant when I copy not using Samba. For example, I copy the .doc > > > file in Windows NT to an FTP server. > > > > > > Does the FTP operation magically linearise the .doc streams on demand? > > > Or does FTP lose part of the Word document? > > > > Good question. It depends if the Microsoft ftp client is streams-aware, > > and understands the Microsoft OLE structured storage format and will do > > the linearisation on demand or not. I must confess I haven't tested this, > > as I don't ever run Windows other than on vmware sessions for Samba testing > > these days :-). > > > > Probably a non-Microsoft ftp client would lose part of the word doc. > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > will _all_ lose part of a Word document when they handle it on a > Window box? > > Ouch! Yep. It's the meta data that Word stores in streams that will get lost. Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:51 ` Jeremy Allison @ 2004-09-01 21:35 ` Lee Revell 2004-09-01 23:22 ` Oliver Hunt 2004-09-02 12:54 ` Theodore Ts'o 2 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-01 21:35 UTC (permalink / raw) To: Jeremy Allison Cc: Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, 2004-09-01 at 16:51, Jeremy Allison wrote: > On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote: > > Jeremy Allison wrote: > > > > I meant when I copy not using Samba. For example, I copy the .doc > > > > file in Windows NT to an FTP server. > > > > > > > > Does the FTP operation magically linearise the .doc streams on demand? > > > > Or does FTP lose part of the Word document? > > > > > > Good question. It depends if the Microsoft ftp client is streams-aware, > > > and understands the Microsoft OLE structured storage format and will do > > > the linearisation on demand or not. I must confess I haven't tested this, > > > as I don't ever run Windows other than on vmware sessions for Samba testing > > > these days :-). > > > > > > Probably a non-Microsoft ftp client would lose part of the word doc. > > > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > will _all_ lose part of a Word document when they handle it on a > > Window box? > > > > Ouch! > > Yep. It's the meta data that Word stores in streams that will get lost. > This is shocking. When was this behavior introduced? Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-01 21:35 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-09-01 21:35 UTC (permalink / raw) To: Jeremy Allison Cc: Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, 2004-09-01 at 16:51, Jeremy Allison wrote: > On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote: > > Jeremy Allison wrote: > > > > I meant when I copy not using Samba. For example, I copy the .doc > > > > file in Windows NT to an FTP server. > > > > > > > > Does the FTP operation magically linearise the .doc streams on demand? > > > > Or does FTP lose part of the Word document? > > > > > > Good question. It depends if the Microsoft ftp client is streams-aware, > > > and understands the Microsoft OLE structured storage format and will do > > > the linearisation on demand or not. I must confess I haven't tested this, > > > as I don't ever run Windows other than on vmware sessions for Samba testing > > > these days :-). > > > > > > Probably a non-Microsoft ftp client would lose part of the word doc. > > > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > will _all_ lose part of a Word document when they handle it on a > > Window box? > > > > Ouch! > > Yep. It's the meta data that Word stores in streams that will get lost. > This is shocking. When was this behavior introduced? Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:51 ` Jeremy Allison 2004-09-01 21:35 ` Lee Revell @ 2004-09-01 23:22 ` Oliver Hunt 2004-09-02 12:54 ` Theodore Ts'o 2 siblings, 0 replies; 1835+ messages in thread From: Oliver Hunt @ 2004-09-01 23:22 UTC (permalink / raw) To: Jeremy Allison; +Cc: Linux Kernel Mailing List, reiserfs-list The loss of forks in the file is exxactly the problem you used to have when transferring native Mac files to a PC... This meant in order to transfer files to different filesystem you often needed to tar/zip/whatever them first. Bare in mind this would let us do the whole MacOS thing of putting an entire application(plus plugins, etc) inside one "file"... --Oliver On Wed, 1 Sep 2004 13:51:40 -0700, Jeremy Allison <jra@samba.org> wrote: > On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote: > > Jeremy Allison wrote: > > > > I meant when I copy not using Samba. For example, I copy the .doc > > > > file in Windows NT to an FTP server. > > > > > > > > Does the FTP operation magically linearise the .doc streams on demand? > > > > Or does FTP lose part of the Word document? > > > > > > Good question. It depends if the Microsoft ftp client is streams-aware, > > > and understands the Microsoft OLE structured storage format and will do > > > the linearisation on demand or not. I must confess I haven't tested this, > > > as I don't ever run Windows other than on vmware sessions for Samba testing > > > these days :-). > > > > > > Probably a non-Microsoft ftp client would lose part of the word doc. > > > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > will _all_ lose part of a Word document when they handle it on a > > Window box? > > > > Ouch! > > Yep. It's the meta data that Word stores in streams that will get lost. > > > > Jeremy. > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:51 ` Jeremy Allison 2004-09-01 21:35 ` Lee Revell 2004-09-01 23:22 ` Oliver Hunt @ 2004-09-02 12:54 ` Theodore Ts'o 2004-09-02 14:45 ` Stuart Young 2 siblings, 1 reply; 1835+ messages in thread From: Theodore Ts'o @ 2004-09-02 12:54 UTC (permalink / raw) To: Jeremy Allison Cc: Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote: > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > will _all_ lose part of a Word document when they handle it on a > > Window box? > > > > Ouch! > > Yep. It's the meta data that Word stores in streams that will get lost. And this is why I believe that using streams in application is well, ill-advised. Indeed, one of my concerns with providing streams support is that application authors may make the mistake of using it, and we will be back to the bad old days (when MacOS made this mistake) where you will need to binhex files before you ftp them (and unbinhex them on the otherside) --- and if you forget, the resulting file will be useless. I understand why the Samba folks want this feature very badly; however, hopefully other projects will know enough *not* to use streams once they become available in Linux.... - Ted ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 12:54 ` Theodore Ts'o @ 2004-09-02 14:45 ` Stuart Young 0 siblings, 0 replies; 1835+ messages in thread From: Stuart Young @ 2004-09-02 14:45 UTC (permalink / raw) To: linux-kernel Cc: Theodore Ts'o, Jeremy Allison, Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, flx, reiserfs-list On Thu, 2 Sep 2004 22:54, Theodore Ts'o wrote: > On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote: > > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > > will _all_ lose part of a Word document when they handle it on a > > > Window box? > > > > > > Ouch! > > > > Yep. It's the meta data that Word stores in streams that will get lost. > > And this is why I believe that using streams in application is well, > ill-advised. Indeed, one of my concerns with providing streams > support is that application authors may make the mistake of using it, > and we will be back to the bad old days (when MacOS made this mistake) > where you will need to binhex files before you ftp them (and unbinhex > them on the otherside) --- and if you forget, the resulting file will > be useless. At least currently (to my knowledge anyway) all stream support in Windows is data that is not important, and that can be either regenerated from filesystem metadata or (more usually) the main file stream itself. This sort of data is really where streams excel, by providing a way to access data that would otherwise take time/cpu to regenerate over and over, but that in itself is not indispensable. Good examples of this are indexes of data within a document, details of who owns/created/modified the document, common views or reformatting of the data, etc. With audio/video/graphics, you could store lower quality transforms of data (eg: stereo to mono, resolution reduction, thumbnails, etc) in the streams for a file. With a word document, it could be things like an index (assuming it's auto-generated from section headings). With a database, it could be the indexes, and a few views that are expensive time-wise to generate. All of these are easily regenerated from the original data stream, but takes a while. And if you've got the disk, why not use it? If streams were always to be considered volatile, then you could do all sorts of interesting things with them. Any disk cleanup mechanism you have could also reap old streams specifically if the disk gets below a certain amount free. This means that old streams that are hanging about don't end up wasting all your disk space. Of course, you'd want a way to disable this (for servers mainly), and streams would have to be considered volatile on more than just Linux as a platform for this to be useful. Note that I'm not particularly advocating streams here. I'm just pointing out 'how' it could be useful. It could be very easy to misuse streams and cause huge problems (as per Ted's comments), but it's always good to know the other side of the argument. -- Stuart Young (aka Cef) cef-lkml@optusnet.com.au is for LKML and related email only ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:35 ` Jeremy Allison @ 2004-09-01 22:05 ` Hubert Chan 2004-09-01 22:05 ` Hubert Chan 1 sibling, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-01 22:05 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Jeremy" == Jeremy Allison <jra@samba.org> writes: Jeremy> Good question. It depends if the Microsoft ftp client is Jeremy> streams-aware, and understands the Microsoft OLE structured Jeremy> storage format and will do the linearisation on demand or not. I Jeremy> must confess I haven't tested this, as I don't ever run Windows Jeremy> other than on vmware sessions for Samba testing these days :-). Jeremy> Probably a non-Microsoft ftp client would lose part of the word Jeremy> doc. What about email. If someone uses a non-MS mail client, it'll lose part of the Word doc? Does that mean nobody can email me Word docs any more? <pause> Wait a sec, that's a good thing! ;-) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-09-01 22:05 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-09-01 22:05 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Jeremy" == Jeremy Allison <jra@samba.org> writes: Jeremy> Good question. It depends if the Microsoft ftp client is Jeremy> streams-aware, and understands the Microsoft OLE structured Jeremy> storage format and will do the linearisation on demand or not. I Jeremy> must confess I haven't tested this, as I don't ever run Windows Jeremy> other than on vmware sessions for Samba testing these days :-). Jeremy> Probably a non-Microsoft ftp client would lose part of the word Jeremy> doc. What about email. If someone uses a non-MS mail client, it'll lose part of the Word doc? Does that mean nobody can email me Word docs any more? <pause> Wait a sec, that's a good thing! ;-) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:31 ` Jamie Lokier 2004-09-01 20:35 ` Jeremy Allison @ 2004-09-01 21:09 ` David Lang 1 sibling, 0 replies; 1835+ messages in thread From: David Lang @ 2004-09-01 21:09 UTC (permalink / raw) To: Jamie Lokier Cc: Jeremy Allison, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list what happens when you ue one of the many software packages around that lets windows access a NFS server. when you copy the file to a NFS accessed drive does it loose part of the data? David Lang On Wed, 1 Sep 2004, Jamie Lokier wrote: > Date: Wed, 1 Sep 2004 21:31:01 +0100 > From: Jamie Lokier <jamie@shareable.org> > To: Jeremy Allison <jra@samba.org> > Cc: Trond Myklebust <trond.myklebust@fys.uio.no>, > Alan Cox <alan@lxorguk.ukuu.org.uk>, > Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua>, > Rik van Riel <riel@redhat.com>, Christer Weinigel <christer@weinigel.se>, > Spam <spam@tnonline.net>, Andrew Morton <akpm@osdl.org>, > wichert@wiggy.net, Linus Torvalds <torvalds@osdl.org>, reiser@namesys.com, > hch@lst.de, Linux Filesystem Development <linux-fsdevel@vger.kernel.org>, > Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, flx@namesys.com, > reiserfs-list@namesys.com > Subject: Re: silent semantic changes with reiser4 > > Jeremy Allison wrote: >>> Streams in a Word file? >> >> Yep. >> >>> Are you saying that when I copy a .doc file onto my Linux box and off, >>> I lose part of a Word document? >> >> Right now no, because when Samba refuses the stream open, Word falls >> back into a "tar"-like mode where it linearises the streams into the >> data (it's a legacy mode for storing data on a FAT drive, not an NTFS >> drive). However, the problem is that no currently supported Microsoft >> OS doesn't have streams-capable NTFS support. > > I meant when I copy not using Samba. For example, I copy the .doc > file in Windows NT to an FTP server. > > Does the FTP operation magically linearise the .doc streams on demand? > > Or does FTP lose part of the Word document? > > -- Jamie > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:26 ` Jeremy Allison 2004-09-01 20:31 ` Jamie Lokier @ 2004-09-02 9:48 ` Alan Cox 2004-09-02 11:13 ` Oliver Neukum 1 sibling, 1 reply; 1835+ messages in thread From: Alan Cox @ 2004-09-02 9:48 UTC (permalink / raw) To: Jeremy Allison Cc: Jamie Lokier, Trond Myklebust, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Mer, 2004-09-01 at 21:26, Jeremy Allison wrote: > Right now no, because when Samba refuses the stream open, Word falls > back into a "tar"-like mode where it linearises the streams into the > data (it's a legacy mode for storing data on a FAT drive, not an NTFS > drive). However, the problem is that no currently supported Microsoft > OS doesn't have streams-capable NTFS support. That implies that samba can do the same transform set without kernel help and you'd even get the advantages of being able to transfer the stuff around sanely afterwards. What I don't understand is the tie between Linux having such streams and Windows doing it for Samba to work. Netatalk has always handle this for Macintosh and portably. Presumably any Samba support would need to handle OS's without wacky files for portability too ? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 9:48 ` Alan Cox @ 2004-09-02 11:13 ` Oliver Neukum 2004-09-02 11:01 ` Alan Cox 0 siblings, 1 reply; 1835+ messages in thread From: Oliver Neukum @ 2004-09-02 11:13 UTC (permalink / raw) To: Alan Cox Cc: Jeremy Allison, Jamie Lokier, Trond Myklebust, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list Am Donnerstag, 2. September 2004 11:48 schrieb Alan Cox: > What I don't understand is the tie between Linux having such streams and > Windows doing it for Samba to work. Netatalk has always handle this for > Macintosh and portably. Presumably any Samba support would need to > handle OS's without wacky files for portability too ? Can you do an atomic rename of all streams without kernel support? Regards Oliver ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 11:13 ` Oliver Neukum @ 2004-09-02 11:01 ` Alan Cox 0 siblings, 0 replies; 1835+ messages in thread From: Alan Cox @ 2004-09-02 11:01 UTC (permalink / raw) To: Oliver Neukum Cc: Jeremy Allison, Jamie Lokier, Trond Myklebust, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Iau, 2004-09-02 at 12:13, Oliver Neukum wrote: > Am Donnerstag, 2. September 2004 11:48 schrieb Alan Cox: > > What I don't understand is the tie between Linux having such streams and > > Windows doing it for Samba to work. Netatalk has always handle this for > > Macintosh and portably. Presumably any Samba support would need to > > handle OS's without wacky files for portability too ? > > Can you do an atomic rename of all streams without kernel support? That depends how SAMBA chooses to handle the problem internally and how it chooses to store the data. The netatalk people have atomicity from the view of clients but not from the unix fs internal view. Alan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 14:30 ` Alan Cox 2004-08-29 17:42 ` Jeremy Allison 2004-08-29 18:41 ` Trond Myklebust @ 2004-09-01 20:25 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 20:25 UTC (permalink / raw) To: Alan Cox Cc: Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list Alan Cox wrote: > On Iau, 2004-08-26 at 16:19, Denis Vlasenko wrote: > > > Office suites can store a document with embedded images > > > and spread sheets "easily" by putting the text, the > > > images and spread sheets all in different file streams. > > > > Kinda far reaching. It's hard to assess whether that is good or bad idea. > > Can we start small, like putting metadata (e.g. ACLs) into these streams? > > Openoffice does this in user space and the user space vfs code desktops > use can handle zips so this "just works" already including over NFS, > unlike a kernel proposed method. OpenOffice is very sensible, and I agree that a method of handling document parts that _requires_ kernel support would be foolish. However, please see my message "The argument for fs assistance in handling archives" for how kernel support can improve performance of programs like OpenOffice on large compound documents, while preserving the sensible behaviour and remaining portable. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:00 ` Denis Vlasenko (?) (?) @ 2004-08-26 14:42 ` Felipe Alfaro Solana -1 siblings, 0 replies; 1835+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 14:42 UTC (permalink / raw) To: Denis Vlasenko Cc: Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 16:00, Denis Vlasenko wrote: > > I think Hans is not planning turning old "file is a stream of bytes" > into eight-stream octopus. One stream will remain as a 'main' one, > which contains actual data. Others will keep metadata, etc... > > Note that even today's files aren't simple "streams of bytes" > because they also have names,mode bits etc. This is just exactly how NTFS does work: there's an unnamed stream (which is the one that the majority of userspace programs do manipulate) which contains the user data (the pixels from a PMG image, or text from a text file, etc.), and there are named streams which not all kind of applications do know how to handle. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:49 ` Andrew Morton @ 2004-08-26 13:36 ` Rik van Riel 2004-08-26 13:36 ` Rik van Riel 2004-08-26 17:32 ` Jeremy Allison 2 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 13:36 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Andrew Morton wrote: > All of which can be handled in userspace library code. > > What compelling reason is there for doing this in the kernel? There's a compelling reason to do it in userspace. If an unaware program copies or moves such a file with streams inside, it doesn't break the streams and aware programs will continue to see them. OTOH, if we had the streams in the kernel, unaware applications would continuously break the metadata and streams that the streams aware programs expect ! -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 13:36 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 13:36 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Andrew Morton wrote: > All of which can be handled in userspace library code. > > What compelling reason is there for doing this in the kernel? There's a compelling reason to do it in userspace. If an unaware program copies or moves such a file with streams inside, it doesn't break the streams and aware programs will continue to see them. OTOH, if we had the streams in the kernel, unaware applications would continuously break the metadata and streams that the streams aware programs expect ! -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:36 ` Rik van Riel (?) @ 2004-08-26 13:59 ` Jamie Lokier 2004-08-26 14:07 ` Rik van Riel -1 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 13:59 UTC (permalink / raw) To: Rik van Riel Cc: Andrew Morton, Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > All of which can be handled in userspace library code. > > > > What compelling reason is there for doing this in the kernel? > > There's a compelling reason to do it in userspace. If an > unaware program copies or moves such a file with streams > inside, it doesn't break the streams and aware programs will > continue to see them. > > OTOH, if we had the streams in the kernel, unaware applications > would continuously break the metadata and streams that the > streams aware programs expect ! You appear not to have read any of my mails on this topic. Properly implemented metadata can: (1) operate in both modes simultaneously; (2) work with unaware applications; (3) provide performance enhancements to aware applications; (4) provide storage enhancements to both; (5) provide useful features that work with standard unmodified unix tools, all at once. That includes program copies. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:59 ` Jamie Lokier @ 2004-08-26 14:07 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 14:07 UTC (permalink / raw) To: Jamie Lokier Cc: Andrew Morton, Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > Properly implemented metadata can: [snip world peace] > all at once. That includes program copies. Show me the design for such a properly implemented thing. None of the designs I've seen proposed so far are able to do what you claim. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 14:07 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 14:07 UTC (permalink / raw) To: Jamie Lokier Cc: Andrew Morton, Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > Properly implemented metadata can: [snip world peace] > all at once. That includes program copies. Show me the design for such a properly implemented thing. None of the designs I've seen proposed so far are able to do what you claim. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:36 ` Rik van Riel @ 2004-08-26 23:54 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:54 UTC (permalink / raw) To: Rik van Riel Cc: Andrew Morton, Spam, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: >On Thu, 26 Aug 2004, Andrew Morton wrote: > > > >>All of which can be handled in userspace library code. >> >>What compelling reason is there for doing this in the kernel? >> >> > >There's a compelling reason to do it in userspace. If an >unaware program copies or moves such a file with streams >inside, it doesn't break the streams and aware programs will >continue to see them. > >OTOH, if we had the streams in the kernel, unaware applications >would continuously break the metadata and streams that the >streams aware programs expect ! > > > Well, first off, you don't want streams in the kernel, you want all the little pieces that can be composed together into a stream if you so choose. Streams are ugly, the pieces are all cool. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:54 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:54 UTC (permalink / raw) To: Rik van Riel Cc: Andrew Morton, Spam, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: >On Thu, 26 Aug 2004, Andrew Morton wrote: > > > >>All of which can be handled in userspace library code. >> >>What compelling reason is there for doing this in the kernel? >> >> > >There's a compelling reason to do it in userspace. If an >unaware program copies or moves such a file with streams >inside, it doesn't break the streams and aware programs will >continue to see them. > >OTOH, if we had the streams in the kernel, unaware applications >would continuously break the metadata and streams that the >streams aware programs expect ! > > > Well, first off, you don't want streams in the kernel, you want all the little pieces that can be composed together into a stream if you so choose. Streams are ugly, the pieces are all cool. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:49 ` Andrew Morton 2004-08-26 10:20 ` Spam 2004-08-26 13:36 ` Rik van Riel @ 2004-08-26 17:32 ` Jeremy Allison 2004-08-26 18:17 ` Rik van Riel 2004-08-27 8:13 ` Hans Reiser 2 siblings, 2 replies; 1835+ messages in thread From: Jeremy Allison @ 2004-08-26 17:32 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 02:49:56AM -0700, Andrew Morton wrote: > Spam <spam@tnonline.net> wrote: > > > > Yes, for example documents, image files etc. The multiple data > > streams can contain thumbnails, info about who is editing the file > > (useful for networked files) etc. Could be used for version handling > > and much more. > > All of which can be handled in userspace library code. > > What compelling reason is there for doing this in the kernel? Because without kernel support there is no way someone can publish a new metadata type and have it automatically supported by all application data files (ie. most apps ignore it, and only apps that are aware of it can see it). Without kernel support you have to have all apps agree on a data format. And that's harder to do than getting linux kernel VFS engineers to agree on things :-). Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:32 ` Jeremy Allison @ 2004-08-26 18:17 ` Rik van Riel 2004-08-27 8:13 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 18:17 UTC (permalink / raw) To: Jeremy Allison Cc: Andrew Morton, Spam, wichert, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jeremy Allison wrote: > Because without kernel support there is no way someone can > publish a new metadata type and have it automatically supported > by all application data files (ie. most apps ignore it, and only > apps that are aware of it can see it). So your backup software ignores it, and after a restore you've lost your new metadata ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 18:17 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 18:17 UTC (permalink / raw) To: Jeremy Allison Cc: Andrew Morton, Spam, wichert, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jeremy Allison wrote: > Because without kernel support there is no way someone can > publish a new metadata type and have it automatically supported > by all application data files (ie. most apps ignore it, and only > apps that are aware of it can see it). So your backup software ignores it, and after a restore you've lost your new metadata ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:17 ` Rik van Riel (?) @ 2004-08-26 22:03 ` Felipe Alfaro Solana -1 siblings, 0 replies; 1835+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 22:03 UTC (permalink / raw) To: Rik van Riel Cc: Jeremy Allison, Andrew Morton, Spam, wichert, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 20:17, Rik van Riel wrote: > On Thu, 26 Aug 2004, Jeremy Allison wrote: > > Because without kernel support there is no way someone can > > publish a new metadata type and have it automatically supported > > by all application data files (ie. most apps ignore it, and only > > apps that are aware of it can see it). > > So your backup software ignores it, and after a restore you've > lost your new metadata ? I think there's no easy way to fix this, except fixing the backup software to support xattrs's/streams/forks/whatever. Even on Windows, it has happened this way with older apps that don't support named streams. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:17 ` Rik van Riel (?) (?) @ 2004-08-27 5:28 ` Jeremy Allison 2004-08-27 6:20 ` Hubert Chan 2004-08-27 8:05 ` Hans Reiser -1 siblings, 2 replies; 1835+ messages in thread From: Jeremy Allison @ 2004-08-27 5:28 UTC (permalink / raw) To: Rik van Riel Cc: Jeremy Allison, Andrew Morton, Spam, wichert, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 02:17:04PM -0400, Rik van Riel wrote: > On Thu, 26 Aug 2004, Jeremy Allison wrote: > > > Because without kernel support there is no way someone can > > publish a new metadata type and have it automatically supported > > by all application data files (ie. most apps ignore it, and only > > apps that are aware of it can see it). > > So your backup software ignores it, and after a restore you've > lost your new metadata ? Yes, but then you fix your backup software and everyone lives happily ever after :-)...... It's a better solution than changing *every single application in existance* (the "userspace library" approach). IMHO of course. Jeremy. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 5:28 ` Jeremy Allison @ 2004-08-27 6:20 ` Hubert Chan 2004-08-27 8:05 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-27 6:20 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Jeremy" == Jeremy Allison <jra@samba.org> writes: Jeremy> On Thu, Aug 26, 2004 at 02:17:04PM -0400, Rik van Riel wrote: >> On Thu, 26 Aug 2004, Jeremy Allison wrote: >> > Because without kernel support there is no way someone can publish >> > a new metadata type and have it automatically supported by all >> > application data files (ie. most apps ignore it, and only apps that >> > are aware of it can see it). >> >> So your backup software ignores it, and after a restore you've lost >> your new metadata ? Jeremy> Yes, but then you fix your backup software and everyone lives Jeremy> happily ever after :-)...... Also, once you add support for file-as-dir, you can export any new metadata that you want through that interface, so backup software will be able to handle any new stuff you want to throw at it (ACLs, xattrs, streams, etc.). If anything, I'd say that needing to fix backup software is an argument *for* adding file-as-dir semantics: no need to make any more changes after this one (and it shouldn't be that big of a change, since backup software already knows how to handle files, and how to handle dirs). Jeremy> IMHO of course. ditto. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 6:20 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-27 6:20 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Jeremy" == Jeremy Allison <jra@samba.org> writes: Jeremy> On Thu, Aug 26, 2004 at 02:17:04PM -0400, Rik van Riel wrote: >> On Thu, 26 Aug 2004, Jeremy Allison wrote: >> > Because without kernel support there is no way someone can publish >> > a new metadata type and have it automatically supported by all >> > application data files (ie. most apps ignore it, and only apps that >> > are aware of it can see it). >> >> So your backup software ignores it, and after a restore you've lost >> your new metadata ? Jeremy> Yes, but then you fix your backup software and everyone lives Jeremy> happily ever after :-)...... Also, once you add support for file-as-dir, you can export any new metadata that you want through that interface, so backup software will be able to handle any new stuff you want to throw at it (ACLs, xattrs, streams, etc.). If anything, I'd say that needing to fix backup software is an argument *for* adding file-as-dir semantics: no need to make any more changes after this one (and it shouldn't be that big of a change, since backup software already knows how to handle files, and how to handle dirs). Jeremy> IMHO of course. ditto. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 5:28 ` Jeremy Allison @ 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:05 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:05 UTC (permalink / raw) To: Jeremy Allison Cc: Rik van Riel, Andrew Morton, Spam, wichert, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Well said. Jeremy Allison wrote: > Yes, but then you fix your backup software and everyone lives > >happily ever after :-)...... > >It's a better solution than changing *every single application >in existance* (the "userspace library" approach). > >IMHO of course. > >Jeremy. > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:05 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:05 UTC (permalink / raw) To: Jeremy Allison Cc: Rik van Riel, Andrew Morton, Spam, wichert, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Well said. Jeremy Allison wrote: > Yes, but then you fix your backup software and everyone lives > >happily ever after :-)...... > >It's a better solution than changing *every single application >in existance* (the "userspace library" approach). > >IMHO of course. > >Jeremy. > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:32 ` Jeremy Allison @ 2004-08-27 8:13 ` Hans Reiser 2004-08-27 8:13 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:13 UTC (permalink / raw) To: Jeremy Allison Cc: Andrew Morton, Spam, wichert, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list I think Jeremy is accurate in saying the below. Jeremy Allison wrote: >>What compelling reason is there for doing this in the kernel? >> >> > > >Because without kernel support there is no way someone can >publish a new metadata type and have it automatically supported >by all application data files (ie. most apps ignore it, and only >apps that are aware of it can see it). Without kernel support >you have to have all apps agree on a data format. And >that's harder to do than getting linux kernel VFS engineers >to agree on things :-). > >Jeremy. > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:13 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:13 UTC (permalink / raw) To: Jeremy Allison Cc: Andrew Morton, Spam, wichert, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list I think Jeremy is accurate in saying the below. Jeremy Allison wrote: >>What compelling reason is there for doing this in the kernel? >> >> > > >Because without kernel support there is no way someone can >publish a new metadata type and have it automatically supported >by all application data files (ie. most apps ignore it, and only >apps that are aware of it can see it). Without kernel support >you have to have all apps agree on a data format. And >that's harder to do than getting linux kernel VFS engineers >to agree on things :-). > >Jeremy. > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:32 ` Andrew Morton @ 2004-08-26 9:03 ` Anton Altaparmakov 2004-08-26 9:03 ` Anton Altaparmakov 1 sibling, 0 replies; 1835+ messages in thread From: Anton Altaparmakov @ 2004-08-26 9:03 UTC (permalink / raw) To: Andrew Morton Cc: Spam, Linus Torvalds, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Thu, 2004-08-26 at 00:32, Andrew Morton wrote: > Spam <spam@tnonline.net> wrote: > > > > > In other words, if reiserfs does something special, we should make > > > standard interfaces for doing that special thing, so that everybody can > > > do it without stepping on other peoples toes. > > > > Agreed that would be the best. But how much time and effort will it > > be > > Zero. > > We can add these new features tomorrow, as reiser4-only features, with a > plan in hand to generalise them later. > > -->>__if__<<-- we think these are features which Linux should offer. Please don't forget that if the reiser4 features are merged as they are now, then we will likely be stuck with the API reiser4 chooses. There will be tools that will rely on it springing up no doubt. Moving the reiser4 features to VFS later is fine and good, but what if the VFS doesn't want the same API for those features? Either we would have to allow reiser4 to continue providing the old API even though the VFS now provides a new, shiny API or we would have to break all existing API users on reiser4. Things like "I rebooted into the latest kernel and my computer failed to boot because essential app FOO failed to access the reiser4 API - Help!" spring to mind. Yes, I know I am painting a rather black picture here and I know you might well say "screw apps", its been done plenty of times in Linux kernel development before... Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 9:03 ` Anton Altaparmakov 0 siblings, 0 replies; 1835+ messages in thread From: Anton Altaparmakov @ 2004-08-26 9:03 UTC (permalink / raw) To: Andrew Morton Cc: Spam, Linus Torvalds, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Thu, 2004-08-26 at 00:32, Andrew Morton wrote: > Spam <spam@tnonline.net> wrote: > > > > > In other words, if reiserfs does something special, we should make > > > standard interfaces for doing that special thing, so that everybody can > > > do it without stepping on other peoples toes. > > > > Agreed that would be the best. But how much time and effort will it > > be > > Zero. > > We can add these new features tomorrow, as reiser4-only features, with a > plan in hand to generalise them later. > > -->>__if__<<-- we think these are features which Linux should offer. Please don't forget that if the reiser4 features are merged as they are now, then we will likely be stuck with the API reiser4 chooses. There will be tools that will rely on it springing up no doubt. Moving the reiser4 features to VFS later is fine and good, but what if the VFS doesn't want the same API for those features? Either we would have to allow reiser4 to continue providing the old API even though the VFS now provides a new, shiny API or we would have to break all existing API users on reiser4. Things like "I rebooted into the latest kernel and my computer failed to boot because essential app FOO failed to access the reiser4 API - Help!" spring to mind. Yes, I know I am painting a rather black picture here and I know you might well say "screw apps", its been done plenty of times in Linux kernel development before... Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:03 ` Anton Altaparmakov (?) @ 2004-08-26 10:45 ` Christian Mayrhuber 2004-08-26 10:52 ` Christoph Hellwig -1 siblings, 1 reply; 1835+ messages in thread From: Christian Mayrhuber @ 2004-08-26 10:45 UTC (permalink / raw) To: reiserfs-list; +Cc: Anton Altaparmakov, linux-fsdevel, lkml On Thursday 26 August 2004 11:03, Anton Altaparmakov wrote: > Please don't forget that if the reiser4 features are merged as they are > now, then we will likely be stuck with the API reiser4 chooses. There > will be tools that will rely on it springing up no doubt. > > Moving the reiser4 features to VFS later is fine and good, but what if > the VFS doesn't want the same API for those features? Either we would > have to allow reiser4 to continue providing the old API even though the > VFS now provides a new, shiny API or we would have to break all existing > API users on reiser4. Things like "I rebooted into the latest kernel > and my computer failed to boot because essential app FOO failed to > access the reiser4 API - Help!" spring to mind. Andrew Morton wrote: >b) accept the reiser4-only extensions with a view to turning them into > kernel-wide extensions at some time in the future, so all filesystems > will offer the extensions (as much as poss) or If option b) is chosen Reiser4 can become a playground. There is the reiser4() syscall which you surely don't want to implement for other filesystems. Once there is some experience with this new fancy stuff the dust what is useful/insecure, etc. and what is not will settle and can be condensed into a vfs api. Apps like samba and user scripts will have to be adapted once this is the case, but this should not be to big a problem if this stuff is marked experimental. People which want something stable can continue to use xattrs and a magnitude of filesystems for now. -- lg, Chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:45 ` Christian Mayrhuber @ 2004-08-26 10:52 ` Christoph Hellwig 2004-08-26 12:43 ` Olivier Galibert 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 10:52 UTC (permalink / raw) To: Christian Mayrhuber Cc: reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 12:45:47PM +0200, Christian Mayrhuber wrote: > There is the reiser4() syscall which you surely don't want to implement for > other filesystems. Actually a syscall that goes into filesystem code is the last thing we want. But it's not in the submission, so let's keep that flame^H^H^H^Hdiscussion for later. > Once there is some experience with this new fancy stuff the dust what > is useful/insecure, etc. and what is not will settle and can be condensed > into a vfs api. > Apps like samba and user scripts will have to be adapted once this is > the case, but this should not be to big a problem if this stuff is marked > experimental. > > People which want something stable can continue to use xattrs and a > magnitude of filesystems for now. Sure, no one stops you from playing around with new semantics. But please don't add them to the linux kernel stable series until we have semantics we a) want to stick to for a while and b) actually work. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:52 ` Christoph Hellwig @ 2004-08-26 12:43 ` Olivier Galibert 2004-08-26 12:44 ` Christoph Hellwig 2004-08-26 15:15 ` Adrian Bunk 0 siblings, 2 replies; 1835+ messages in thread From: Olivier Galibert @ 2004-08-26 12:43 UTC (permalink / raw) To: Christoph Hellwig, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote: > Sure, no one stops you from playing around with new semantics. But please > don't add them to the linux kernel stable series until we have semantics we > a) want to stick to for a while and b) actually work. He's not proposing to add it to 2.4, is he? OG. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:43 ` Olivier Galibert @ 2004-08-26 12:44 ` Christoph Hellwig 2004-08-26 12:49 ` Markus Törnqvist 2004-08-26 15:15 ` Adrian Bunk 1 sibling, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:44 UTC (permalink / raw) To: Olivier Galibert, Christoph Hellwig, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: > On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote: > > Sure, no one stops you from playing around with new semantics. But please > > don't add them to the linux kernel stable series until we have semantics we > > a) want to stick to for a while and b) actually work. > > He's not proposing to add it to 2.4, is he? We're talking about 2.6 here. 2.4 is the obsolete kernel series these days. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:44 ` Christoph Hellwig 2004-08-26 12:49 ` Markus Törnqvist @ 2004-08-26 12:49 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-26 12:49 UTC (permalink / raw) To: Christoph Hellwig, Olivier Galibert, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote: >On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: >> He's not proposing to add it to 2.4, is he? >We're talking about 2.6 here. 2.4 is the obsolete kernel series these days. So run it in -mm or fork it to 2.7. 2.7 might not be a bad idea for trying such bleeding edge revolution stuff as this. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 12:49 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: mjt @ 2004-08-26 12:49 UTC (permalink / raw) To: Christoph Hellwig, Olivier Galibert, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote: >On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: >> He's not proposing to add it to 2.4, is he? >We're talking about 2.6 here. 2.4 is the obsolete kernel series these days. So run it in -mm or fork it to 2.7. 2.7 might not be a bad idea for trying such bleeding edge revolution stuff as this. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 12:49 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-26 12:49 UTC (permalink / raw) To: Christoph Hellwig, Olivier Galibert, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote: >On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: >> He's not proposing to add it to 2.4, is he? >We're talking about 2.6 here. 2.4 is the obsolete kernel series these days. So run it in -mm or fork it to 2.7. 2.7 might not be a bad idea for trying such bleeding edge revolution stuff as this. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:49 ` Markus Törnqvist (?) (?) @ 2004-08-26 12:52 ` Christoph Hellwig -1 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:52 UTC (permalink / raw) To: Markus Törnqvist Cc: Olivier Galibert, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 03:49:29PM +0300, Markus Törnqvist wrote: > 2.7 might not be a bad idea for trying such bleeding edge revolution > stuff as this. I already suggested to Linus that 2.7 is probably the right place to experiment with that. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:49 ` Markus Törnqvist @ 2004-08-26 13:35 ` Christian Mayrhuber -1 siblings, 0 replies; 1835+ messages in thread From: Christian Mayrhuber @ 2004-08-26 13:35 UTC (permalink / raw) To: reiserfs-list; +Cc: Markus Törnqvist, Christoph Hellwig, Olivier Galibert On Thursday 26 August 2004 14:49, Markus Törnqvist wrote: > On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote: > >On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: > >> He's not proposing to add it to 2.4, is he? > > > >We're talking about 2.6 here. 2.4 is the obsolete kernel series these > > days. > > So run it in -mm or fork it to 2.7. > > 2.7 might not be a bad idea for trying such bleeding edge revolution > stuff as this. If I remember right, there are now some reasons for a 2.7: 1) udev / dynamic device numbers 2) enhanced filesystem semantics maybe resulting in 3) visible automounter mountpoints -- lg, Chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 13:35 ` Christian Mayrhuber 0 siblings, 0 replies; 1835+ messages in thread From: Christian Mayrhuber @ 2004-08-26 13:35 UTC (permalink / raw) To: reiserfs-list Cc: Markus Törnqvist, Christoph Hellwig, Olivier Galibert, linux-fsdevel, lkml On Thursday 26 August 2004 14:49, Markus Törnqvist wrote: > On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote: > >On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: > >> He's not proposing to add it to 2.4, is he? > > > >We're talking about 2.6 here. 2.4 is the obsolete kernel series these > > days. > > So run it in -mm or fork it to 2.7. > > 2.7 might not be a bad idea for trying such bleeding edge revolution > stuff as this. If I remember right, there are now some reasons for a 2.7: 1) udev / dynamic device numbers 2) enhanced filesystem semantics maybe resulting in 3) visible automounter mountpoints -- lg, Chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:43 ` Olivier Galibert 2004-08-26 12:44 ` Christoph Hellwig @ 2004-08-26 15:15 ` Adrian Bunk 1 sibling, 0 replies; 1835+ messages in thread From: Adrian Bunk @ 2004-08-26 15:15 UTC (permalink / raw) To: Olivier Galibert, Christoph Hellwig, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: > On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote: > > Sure, no one stops you from playing around with new semantics. But please > > don't add them to the linux kernel stable series until we have semantics we > > a) want to stick to for a while and b) actually work. > > He's not proposing to add it to 2.4, is he? 2.6 is a stable series... > OG. cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:51 ` Spam @ 2004-08-26 8:44 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:44 UTC (permalink / raw) To: Spam Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Spam wrote: > >> The fact that one filesystem will >>offer features which other filesystems do not and cannot offer makes me >>queasy for some reason. >> >> > > This last sentence makes me wonder. Where is Linux heading? The idea > that a FS cannot contain features that no other FS has is very > scary. > > I am all for uniformity, but not at the expense of shutting down > advanced progress that Linux is so badly needing. > > This talk about old UNIX seems like people want to still live in the > 70'ies and not look forward. Please wake up! > > ~S > > > > > > well said. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 8:44 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:44 UTC (permalink / raw) To: Spam Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Spam wrote: > >> The fact that one filesystem will >>offer features which other filesystems do not and cannot offer makes me >>queasy for some reason. >> >> > > This last sentence makes me wonder. Where is Linux heading? The idea > that a FS cannot contain features that no other FS has is very > scary. > > I am all for uniformity, but not at the expense of shutting down > advanced progress that Linux is so badly needing. > > This talk about old UNIX seems like people want to still live in the > 70'ies and not look forward. Please wake up! > > ~S > > > > > > well said. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:28 ` Andrew Morton 2004-08-25 22:51 ` Spam @ 2004-08-26 8:31 ` Hans Reiser 2004-08-26 8:45 ` Andrew Morton ` (6 more replies) 1 sibling, 7 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 8:31 UTC (permalink / raw) To: Andrew Morton Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Andrew Morton wrote: >Hans Reiser <reiser@namesys.com> wrote: > > >>I had not intended to respond to this because I have nothing positive to >>say, but Andrew said I needed to respond and suggested I should copy >>Linus. >> >> > >Yes, but I didn't say "flame Christoph and ignore the issues" ;) > > Oh....;-) >There are lots of little things to do with implementation, coding style, >module exports, deadlocks, what code goes where, etc. These are all normal >daily kernel business and we should set them aside for now and concentrate >on the bigger issues. > > Yes, you are right, but I am not sure Viro will go along with that..... ;-) >And as I see it, there are two big issues: > >a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not > anticipate and > >b) it does this within the context of just a single filesystem. > >I see three possible responses: > >a) accept the reiser4-only extensions as-is (possibly with post-review > modifications, of course) or > >b) accept the reiser4-only extensions with a view to turning them into > kernel-wide extensions at some time in the future, so all filesystems > will offer the extensions (as much as poss) or > >c) reject the extensions. > > >My own order of preference is b) c) a). > I don't object to b), though I think b) should wait for 2.7 and reiser4 should not. > The fact that one filesystem will >offer features which other filesystems do not and cannot offer makes me >queasy for some reason. > > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem for Apple, and that means we need to put search engine and database functionality into the filesystem. It takes 11 years of serious research to build a clean storage layer able to handle doing that. Reiser4 has done that, finally. None of the other Linux filesystems have. The next major release of ReiserFS is going to be bursting with semantic enhancements, because the prerequisites for them are in place now. None of the other Linux filesystems have those prerequisites. They won't be able to keep up with the semantic enhancements. This metafiles and file-directories stuff is actually fairly trivial stuff. Look guys, in 1993 I anticipated the battle would be here, and I build the foundation for a defensive tower right at the spot MS and Apple are now maneuvering towards. Help me get the next level on the tower before they get here. It is one hell of a foundation, they won't be able to shake it, their trees are not as powerful. Don't move reiser4 into vfs, use reiser4 as the vfs. Don't write filesystems, write file plugins and disk format plugins and all the other kinds of plugins, and you won't be missing any expressive power that you really want.... Give Saveliev and I some credit. 10 years of hard work at an ivory tower nobody thought mattered. Now the battle leaves the browser and swings our way. Don't duplicate that infrastructure, use it. There is so much we could use help with if talented people like you chose to contribute. >b) means that at some time in the future we need to hoist the reiser4 >extensions (at a conceptual level) (and probably with restrictions) up into >the VFS. This will involve much thought, argument and work. > > >To get us started on this route it would really help me (and, probably, >others) if you could describe what these API extensions are in a very >simple way, without referring to incomprehehsible web pages, > what is not comprehensible....? > and without >using terms which non-reiser4 people don't understand. > > Well, I agree that there is value in defining things in more detail than we have. >It would be best if each extension was addressed in a separate email >thread. > >We also need to discuss what a reiser4 "module" is, what its capabilities >are, and what licensing implications they have. > >Then, we can look at each one and say "yup, that makes sense - we want >Linux to do that" and we can also think about how we would implement it at >the VFS level. > >If we follow the above route I believe we can make progress in a technical >direction and not get deadlocked on personal/political stuff. > > >Now, an alternative to all the above is to just merge reiser4 as-is, after >addressing all the lower-level coding issues. And see what happens. That >may be a thing which Linus wishes to do. I'm easy. > > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser @ 2004-08-26 8:45 ` Andrew Morton 2004-08-26 9:24 ` Hans Reiser 2004-08-26 12:18 ` Christophe Saout 2004-08-26 9:34 ` Christoph Hellwig ` (5 subsequent siblings) 6 siblings, 2 replies; 1835+ messages in thread From: Andrew Morton @ 2004-08-26 8:45 UTC (permalink / raw) To: Hans Reiser Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser <reiser@namesys.com> wrote: > > >To get us started on this route it would really help me (and, probably, > >others) if you could describe what these API extensions are in a very > >simple way, without referring to incomprehehsible web pages, > > > what is not comprehensible....? Pretty much anything at www.namesys.com. The amount of time which is needed to extract the technical info which one is looking for vastly exceeds a gnat-like attention span. As a starting point, please prepare a bullet-point list of userspace-visible changes which the filesystem introduces, or is planned to introduce. And describe the "plugin" system. Why does the filesystem need such a thing (other filesystems get their features via `patch -p1')? And what are the licensing implications of plugins? Are they derived works? Must they be GPL'ed? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:45 ` Andrew Morton @ 2004-08-26 9:24 ` Hans Reiser 2004-08-26 12:18 ` Christophe Saout 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 9:24 UTC (permalink / raw) To: Andrew Morton Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Andrew Morton wrote: >Hans Reiser <reiser@namesys.com> wrote: > > >> >To get us started on this route it would really help me (and, probably, >> >others) if you could describe what these API extensions are in a very >> >simple way, without referring to incomprehehsible web pages, >> > >> what is not comprehensible....? >> >> > >Pretty much anything at www.namesys.com. The amount of time which is >needed to extract the technical info which one is looking for vastly >exceeds a gnat-like attention span. > >As a starting point, please prepare a bullet-point list of >userspace-visible changes which the filesystem introduces, or is planned to >introduce. > >And describe the "plugin" system. Why does the filesystem need such a >thing (other filesystems get their features via `patch -p1')? > > It takes 6 months or more to become competent to change a usual filesystem. Creating a new reiser4 plugin is a weekend programmer fun hack to do. Weekend programmers matter, because they tend to have clever ideas based on understanding a need they have. How many people can easily add new features to ext3 or reiserfs V3? Very few. What happens if you need a disk format change? Well, in V4, you can easily compose a plugin from plugin methods of other plugins, write a little piece of code with the one thing you want different, and add it in. Disk format changes, no big deal, add a new disk format plugin, or a new item plugin, or a new node plugin, etc., and you got your new format. There is a huge difference between code that is designed for reuse, and code that is not. That is the difference between V3 and V4. We were looking at our V3 balancing code, and it special cased each different kind of item (an item is a piece of something, which the balancing code chops objects into as it squeezes them into nodes). It looked like the complexity of the balancing code was going to be N squared, where N was the number of different kinds of items. So, we created item handlers, and wrote balancing code that could slice and dice and merge any item that implemented all of the operations required of an item handler. From there it grew, and we made everything pluggable. Adding features to the new code is far less than the time cost of adding features to the old code. I've seen Nikita complain that something would take 6 weeks to do (changing key assignment algorithms), and then it takes him 3 afternoons, and it was because of the plugins, because when we did something similar in V3 it took 3 man months. >And what are the licensing implications of plugins? Are they derived >works? > Yes. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 9:24 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 9:24 UTC (permalink / raw) To: Andrew Morton Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Andrew Morton wrote: >Hans Reiser <reiser@namesys.com> wrote: > > >> >To get us started on this route it would really help me (and, probably, >> >others) if you could describe what these API extensions are in a very >> >simple way, without referring to incomprehehsible web pages, >> > >> what is not comprehensible....? >> >> > >Pretty much anything at www.namesys.com. The amount of time which is >needed to extract the technical info which one is looking for vastly >exceeds a gnat-like attention span. > >As a starting point, please prepare a bullet-point list of >userspace-visible changes which the filesystem introduces, or is planned to >introduce. > >And describe the "plugin" system. Why does the filesystem need such a >thing (other filesystems get their features via `patch -p1')? > > It takes 6 months or more to become competent to change a usual filesystem. Creating a new reiser4 plugin is a weekend programmer fun hack to do. Weekend programmers matter, because they tend to have clever ideas based on understanding a need they have. How many people can easily add new features to ext3 or reiserfs V3? Very few. What happens if you need a disk format change? Well, in V4, you can easily compose a plugin from plugin methods of other plugins, write a little piece of code with the one thing you want different, and add it in. Disk format changes, no big deal, add a new disk format plugin, or a new item plugin, or a new node plugin, etc., and you got your new format. There is a huge difference between code that is designed for reuse, and code that is not. That is the difference between V3 and V4. We were looking at our V3 balancing code, and it special cased each different kind of item (an item is a piece of something, which the balancing code chops objects into as it squeezes them into nodes). It looked like the complexity of the balancing code was going to be N squared, where N was the number of different kinds of items. So, we created item handlers, and wrote balancing code that could slice and dice and merge any item that implemented all of the operations required of an item handler. From there it grew, and we made everything pluggable. Adding features to the new code is far less than the time cost of adding features to the old code. I've seen Nikita complain that something would take 6 weeks to do (changing key assignment algorithms), and then it takes him 3 afternoons, and it was because of the plugins, because when we did something similar in V3 it took 3 man months. >And what are the licensing implications of plugins? Are they derived >works? > Yes. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:24 ` Hans Reiser @ 2004-08-26 19:26 ` Lee Revell -1 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 19:26 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 05:24, Hans Reiser wrote: > Andrew Morton wrote: > > > >And describe the "plugin" system. Why does the filesystem need such a > >thing (other filesystems get their features via `patch -p1')? > > > > > It takes 6 months or more to become competent to change a usual > filesystem. Creating a new reiser4 plugin is a weekend programmer fun > hack to do. Weekend programmers matter, because they tend to have > clever ideas based on understanding a need they have. How many people > can easily add new features to ext3 or reiserfs V3? Very few. > > What happens if you need a disk format change? > > Well, in V4, you can easily compose a plugin from plugin methods of > other plugins, write a little piece of code with the one thing you want > different, and add it in. Disk format changes, no big deal, add a new > disk format plugin, or a new item plugin, or a new node plugin, etc., > and you got your new format. > OK, real world example. My roommate has an AKAI MPC-2000, a very popular hardware sampler from the 90's. The disk format is known,there are a few utilities to edit the disks on a PC and extract the PCM samples, but there are no tools to mount it on a modern PC. Are you saying that, since I know the MPC disk format, I could write a reiser4 plugin to mount an MPC drive? If so, then Hans has an excellent point. Users do want this kind of thing, and it is worth having to fix tar et al. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 19:26 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 19:26 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 05:24, Hans Reiser wrote: > Andrew Morton wrote: > > > >And describe the "plugin" system. Why does the filesystem need such a > >thing (other filesystems get their features via `patch -p1')? > > > > > It takes 6 months or more to become competent to change a usual > filesystem. Creating a new reiser4 plugin is a weekend programmer fun > hack to do. Weekend programmers matter, because they tend to have > clever ideas based on understanding a need they have. How many people > can easily add new features to ext3 or reiserfs V3? Very few. > > What happens if you need a disk format change? > > Well, in V4, you can easily compose a plugin from plugin methods of > other plugins, write a little piece of code with the one thing you want > different, and add it in. Disk format changes, no big deal, add a new > disk format plugin, or a new item plugin, or a new node plugin, etc., > and you got your new format. > OK, real world example. My roommate has an AKAI MPC-2000, a very popular hardware sampler from the 90's. The disk format is known,there are a few utilities to edit the disks on a PC and extract the PCM samples, but there are no tools to mount it on a modern PC. Are you saying that, since I know the MPC disk format, I could write a reiser4 plugin to mount an MPC drive? If so, then Hans has an excellent point. Users do want this kind of thing, and it is worth having to fix tar et al. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:26 ` Lee Revell (?) @ 2004-08-26 19:33 ` Christophe Saout 2004-08-27 8:05 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 19:33 UTC (permalink / raw) To: Lee Revell Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 923 bytes --] Am Donnerstag, den 26.08.2004, 15:26 -0400 schrieb Lee Revell: > > Well, in V4, you can easily compose a plugin from plugin methods of > > other plugins, write a little piece of code with the one thing you want > > different, and add it in. Disk format changes, no big deal, add a new > > disk format plugin, or a new item plugin, or a new node plugin, etc., > > and you got your new format. > > OK, real world example. My roommate has an AKAI MPC-2000, a very > popular hardware sampler from the 90's. The disk format is known,there > are a few utilities to edit the disks on a PC and extract the PCM > samples, but there are no tools to mount it on a modern PC. Are you > saying that, since I know the MPC disk format, I could write a reiser4 > plugin to mount an MPC drive? No, the underlying storage must be a reiser4-like tree. If you want to mount an MPC drive, write an MPC filesystem. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:33 ` Christophe Saout @ 2004-08-27 8:05 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:05 UTC (permalink / raw) To: Christophe Saout Cc: Lee Revell, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 15:26 -0400 schrieb Lee Revell: > > > >>>Well, in V4, you can easily compose a plugin from plugin methods of >>>other plugins, write a little piece of code with the one thing you want >>>different, and add it in. Disk format changes, no big deal, add a new >>>disk format plugin, or a new item plugin, or a new node plugin, etc., >>>and you got your new format. >>> >>> >>OK, real world example. My roommate has an AKAI MPC-2000, a very >>popular hardware sampler from the 90's. The disk format is known,there >>are a few utilities to edit the disks on a PC and extract the PCM >>samples, but there are no tools to mount it on a modern PC. Are you >>saying that, since I know the MPC disk format, I could write a reiser4 >>plugin to mount an MPC drive? >> >> > >No, the underlying storage must be a reiser4-like tree. > Not necessarily. We just encourage it.... Reiser4 is a body of code that can be sliced and diced as you choose, and it is designed for easy slicing. >If you want to >mount an MPC drive, write an MPC filesystem. > > > However, this last sentence is probably sensible advice. A mount point is probably the right interface from reiser4 for what you want. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:05 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:05 UTC (permalink / raw) To: Christophe Saout Cc: Lee Revell, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 15:26 -0400 schrieb Lee Revell: > > > >>>Well, in V4, you can easily compose a plugin from plugin methods of >>>other plugins, write a little piece of code with the one thing you want >>>different, and add it in. Disk format changes, no big deal, add a new >>>disk format plugin, or a new item plugin, or a new node plugin, etc., >>>and you got your new format. >>> >>> >>OK, real world example. My roommate has an AKAI MPC-2000, a very >>popular hardware sampler from the 90's. The disk format is known,there >>are a few utilities to edit the disks on a PC and extract the PCM >>samples, but there are no tools to mount it on a modern PC. Are you >>saying that, since I know the MPC disk format, I could write a reiser4 >>plugin to mount an MPC drive? >> >> > >No, the underlying storage must be a reiser4-like tree. > Not necessarily. We just encourage it.... Reiser4 is a body of code that can be sliced and diced as you choose, and it is designed for easy slicing. >If you want to >mount an MPC drive, write an MPC filesystem. > > > However, this last sentence is probably sensible advice. A mount point is probably the right interface from reiser4 for what you want. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:05 ` Hans Reiser (?) @ 2004-08-27 9:24 ` Christoph Hellwig -1 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-27 9:24 UTC (permalink / raw) To: Hans Reiser Cc: Christophe Saout, Lee Revell, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Fri, Aug 27, 2004 at 01:05:45AM -0700, Hans Reiser wrote: > Not necessarily. We just encourage it.... Reiser4 is a body of code > that can be sliced and diced as you choose, and it is designed for easy > slicing. So what about slicing it into a simple posix fs as a start so we can review that while still discussing semantic issues. Now that would be a refreshing thing after all that talking, no? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:26 ` Lee Revell (?) (?) @ 2004-08-26 19:34 ` Christoph Hellwig 2004-08-26 19:45 ` Lee Revell -1 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 19:34 UTC (permalink / raw) To: Lee Revell Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote: > OK, real world example. My roommate has an AKAI MPC-2000, a very > popular hardware sampler from the 90's. The disk format is known,there > are a few utilities to edit the disks on a PC and extract the PCM > samples, but there are no tools to mount it on a modern PC. Are you > saying that, since I know the MPC disk format, I could write a reiser4 > plugin to mount an MPC drive? > > If so, then Hans has an excellent point. Users do want this kind of > thing, and it is worth having to fix tar et al. You don't need reiser4 for that, writing read-only linux filesystems is trivial as soon as you have a specification of the ondisk format. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:34 ` Christoph Hellwig @ 2004-08-26 19:45 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 19:45 UTC (permalink / raw) To: Christoph Hellwig Cc: Hans Reiser, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 15:34, Christoph Hellwig wrote: > On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote: > > OK, real world example. My roommate has an AKAI MPC-2000, a very > > popular hardware sampler from the 90's. The disk format is known,there > > are a few utilities to edit the disks on a PC and extract the PCM > > samples, but there are no tools to mount it on a modern PC. Are you > > saying that, since I know the MPC disk format, I could write a reiser4 > > plugin to mount an MPC drive? > > > > If so, then Hans has an excellent point. Users do want this kind of > > thing, and it is worth having to fix tar et al. > > You don't need reiser4 for that, writing read-only linux filesystems is > trivial as soon as you have a specification of the ondisk format. > Of course I could just write an MPC filesystem driver. And, like Andrew said, a filesystem normally gets new features with 'patch -p1'. My question, which was answered by a previous post, was whether the same could be done in a reiser4 plugins. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 19:45 ` Lee Revell 0 siblings, 0 replies; 1835+ messages in thread From: Lee Revell @ 2004-08-26 19:45 UTC (permalink / raw) To: Christoph Hellwig Cc: Hans Reiser, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 15:34, Christoph Hellwig wrote: > On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote: > > OK, real world example. My roommate has an AKAI MPC-2000, a very > > popular hardware sampler from the 90's. The disk format is known,there > > are a few utilities to edit the disks on a PC and extract the PCM > > samples, but there are no tools to mount it on a modern PC. Are you > > saying that, since I know the MPC disk format, I could write a reiser4 > > plugin to mount an MPC drive? > > > > If so, then Hans has an excellent point. Users do want this kind of > > thing, and it is worth having to fix tar et al. > > You don't need reiser4 for that, writing read-only linux filesystems is > trivial as soon as you have a specification of the ondisk format. > Of course I could just write an MPC filesystem driver. And, like Andrew said, a filesystem normally gets new features with 'patch -p1'. My question, which was answered by a previous post, was whether the same could be done in a reiser4 plugins. Lee ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:26 ` Lee Revell ` (2 preceding siblings ...) (?) @ 2004-08-26 20:30 ` Dmitry Baryshkov 2004-08-26 20:38 ` Christophe Saout -1 siblings, 1 reply; 1835+ messages in thread From: Dmitry Baryshkov @ 2004-08-26 20:30 UTC (permalink / raw) To: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hello, On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote: > On Thu, 2004-08-26 at 05:24, Hans Reiser wrote: > > > > Well, in V4, you can easily compose a plugin from plugin methods of > > other plugins, write a little piece of code with the one thing you want > > different, and add it in. Disk format changes, no big deal, add a new > > disk format plugin, or a new item plugin, or a new node plugin, etc., > > and you got your new format. > > > > OK, real world example. My roommate has an AKAI MPC-2000, a very > popular hardware sampler from the 90's. The disk format is known,there > are a few utilities to edit the disks on a PC and extract the PCM > samples, but there are no tools to mount it on a modern PC. Are you > saying that, since I know the MPC disk format, I could write a reiser4 > plugin to mount an MPC drive? > Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4 plugins? From Hans' words it seems so. If this is correct, then maybe reiser4 core should be updated to completely replace current VFS layer? Then it's a good point to create a branch (in old development model it would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and rewrite all (or at least most used) FS as reiser4 plugins. Then everybody will be happy. But this looks too good to be true. Perhaps I misunderstood Hans' words aboud 'new disk format', did I? -- With best wishes Dmitry Baryshkov ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:30 ` Dmitry Baryshkov @ 2004-08-26 20:38 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 20:38 UTC (permalink / raw) To: Dmitry Baryshkov Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 821 bytes --] Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov: > Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4 > plugins? From Hans' words it seems so. If this is correct, then maybe > reiser4 core should be updated to completely replace current VFS layer? > Then it's a good point to create a branch (in old development model it > would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and > rewrite all (or at least most used) FS as reiser4 plugins. Then > everybody will be happy. > > But this looks too good to be true. Perhaps I misunderstood Hans' words > aboud 'new disk format', did I? No. You can change the format the reiser4 storage tree is stored in. As long as other filesystems don't use the same underlying storage tree this is not possible. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 20:38 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 20:38 UTC (permalink / raw) To: Dmitry Baryshkov Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 821 bytes --] Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov: > Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4 > plugins? From Hans' words it seems so. If this is correct, then maybe > reiser4 core should be updated to completely replace current VFS layer? > Then it's a good point to create a branch (in old development model it > would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and > rewrite all (or at least most used) FS as reiser4 plugins. Then > everybody will be happy. > > But this looks too good to be true. Perhaps I misunderstood Hans' words > aboud 'new disk format', did I? No. You can change the format the reiser4 storage tree is stored in. As long as other filesystems don't use the same underlying storage tree this is not possible. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:38 ` Christophe Saout (?) @ 2004-08-26 20:49 ` Dmitry Baryshkov -1 siblings, 0 replies; 1835+ messages in thread From: Dmitry Baryshkov @ 2004-08-26 20:49 UTC (permalink / raw) To: Christophe Saout Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hello, On Thu, Aug 26, 2004 at 10:38:12PM +0200, Christophe Saout wrote: > > No. You can change the format the reiser4 storage tree is stored in. As > long as other filesystems don't use the same underlying storage tree > this is not possible. > Thank you for clarification. As I wrote, it's too good to be true. -- With best wishes Dmitry Baryshkov ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:38 ` Christophe Saout (?) (?) @ 2004-08-26 21:33 ` Nikita Danilov 2004-08-26 21:46 ` Christophe Saout -1 siblings, 1 reply; 1835+ messages in thread From: Nikita Danilov @ 2004-08-26 21:33 UTC (permalink / raw) To: Christophe Saout Cc: Dmitry Baryshkov, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout writes: > Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov: > > > Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4 > > plugins? From Hans' words it seems so. If this is correct, then maybe > > reiser4 core should be updated to completely replace current VFS layer? > > Then it's a good point to create a branch (in old development model it > > would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and > > rewrite all (or at least most used) FS as reiser4 plugins. Then > > everybody will be happy. > > > > But this looks too good to be true. Perhaps I misunderstood Hans' words > > aboud 'new disk format', did I? > > No. You can change the format the reiser4 storage tree is stored in. As > long as other filesystems don't use the same underlying storage tree > this is not possible. Wrong, plugin is called just below entry point from the VFS to the file-system back-end. It can use reiser4 tree, or any storage layer it wants. Or none at all: think about pseudo-files like foo/metas/uid---they are also implemnted by plugins. Programmer dedicated enough can (besides programming Fortran in any language), write reiser4 plugin for any file-system. I will leave description of this activity to people with better command over English language. Nikita. > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:33 ` Nikita Danilov @ 2004-08-26 21:46 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 21:46 UTC (permalink / raw) To: Nikita Danilov Cc: Dmitry Baryshkov, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 594 bytes --] Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov: > Wrong, plugin is called just below entry point from the VFS to the > file-system back-end. It can use reiser4 tree, or any storage layer it > wants. Or none at all: think about pseudo-files like > foo/metas/uid---they are also implemnted by plugins. Yes. But which plugins are we talking about? What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? Of course you can implement another filesystem inside the plugins but they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 21:46 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 21:46 UTC (permalink / raw) To: Nikita Danilov Cc: Dmitry Baryshkov, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 594 bytes --] Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov: > Wrong, plugin is called just below entry point from the VFS to the > file-system back-end. It can use reiser4 tree, or any storage layer it > wants. Or none at all: think about pseudo-files like > foo/metas/uid---they are also implemnted by plugins. Yes. But which plugins are we talking about? What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? Of course you can implement another filesystem inside the plugins but they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:46 ` Christophe Saout (?) @ 2004-08-26 22:03 ` Nikita Danilov 2004-08-26 22:05 ` Christoph Hellwig -1 siblings, 1 reply; 1835+ messages in thread From: Nikita Danilov @ 2004-08-26 22:03 UTC (permalink / raw) To: Christophe Saout Cc: Dmitry Baryshkov, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout writes: > Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov: > > > Wrong, plugin is called just below entry point from the VFS to the > > file-system back-end. It can use reiser4 tree, or any storage layer it > > wants. Or none at all: think about pseudo-files like > > foo/metas/uid---they are also implemnted by plugins. > > Yes. But which plugins are we talking about? > > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? > > Of course you can implement another filesystem inside the plugins but > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? > That was the message of my message. Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:03 ` Nikita Danilov @ 2004-08-26 22:05 ` Christoph Hellwig 2004-08-27 8:12 ` Hans Reiser 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 22:05 UTC (permalink / raw) To: Nikita Danilov Cc: Dmitry Baryshkov, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote: > > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? > > > > Of course you can implement another filesystem inside the plugins but > > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? > > > > That was the message of my message. And I think Christophe and me already agreed in this thread that these upper level plugin facility should go away before merge anyway. We made the mistake of not requesting removel of that one for XFS and now SGI blocks it's removal. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:05 ` Christoph Hellwig @ 2004-08-27 8:12 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:12 UTC (permalink / raw) To: Christoph Hellwig Cc: Nikita Danilov, Dmitry Baryshkov, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: >On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote: > > >> > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? >> > >> > Of course you can implement another filesystem inside the plugins but >> > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? >> > >> >>That was the message of my message. >> >> > >And I think Christophe and me already agreed in this thread that these >upper level plugin facility should go away before merge anyway. We made >the mistake of not requesting removel of that one for XFS and now SGI >blocks it's removal. > >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > Your arrogance is misplaced for a man who has never done a significant piece of research. Christoph, we are doing work, you are in the way, get out of the way. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:12 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:12 UTC (permalink / raw) To: Christoph Hellwig Cc: Nikita Danilov, Dmitry Baryshkov, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: >On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote: > > >> > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? >> > >> > Of course you can implement another filesystem inside the plugins but >> > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? >> > >> >>That was the message of my message. >> >> > >And I think Christophe and me already agreed in this thread that these >upper level plugin facility should go away before merge anyway. We made >the mistake of not requesting removel of that one for XFS and now SGI >blocks it's removal. > >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > Your arrogance is misplaced for a man who has never done a significant piece of research. Christoph, we are doing work, you are in the way, get out of the way. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:12 ` Hans Reiser (?) @ 2004-08-27 8:57 ` Christoph Hellwig -1 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-27 8:57 UTC (permalink / raw) To: Hans Reiser Cc: Nikita Danilov, Dmitry Baryshkov, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Fri, Aug 27, 2004 at 01:12:44AM -0700, Hans Reiser wrote: > Your arrogance is misplaced for a man who has never done a significant > piece of research. > > Christoph, we are doing work, you are in the way, get out of the way. Could you please stop the namecalling. And if you want to check who has done significant work on Linux check the commit logs for yoour and for my shortname. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:45 ` Andrew Morton @ 2004-08-26 12:18 ` Christophe Saout 2004-08-26 12:18 ` Christophe Saout 1 sibling, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 12:18 UTC (permalink / raw) To: Andrew Morton Cc: Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2289 bytes --] Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton: > And describe the "plugin" system. Why does the filesystem need such a > thing (other filesystems get their features via `patch -p1')? Ok, let me try. (Hans, please don't shoot me if I don't get every detail right, I'm trying to simplify and translate) The reiser4 core is just a big and fast storage layer using a single tree. It is bound to a mount point and uses the page cache to manage its memory. The only thing it can do on its own is to flush dirty data to the disk when the VM wants to (memory pressure) or the VFS wants to (unmount, sync) or some "plugin" wants to. Additionally it's completely atomic (with respect to crashes, no isolation like in databases), comparable to data=journal but without the overhead of using a journal for everything. Up to this point there are no users of this "database" and it does not implement any of the VFS methods for a filesystem (except mount and unmount perhaps). All the functionality is provided to the plugins. You can insert, remove, lookup or modify key:object pairs (with an index into the object). The object is a sequence of units. For files a unit would be 1 byte and the index would be the byte offset. For directories the unit would be 1 entry and the index would be the filename. Now there are some plugins that define how the storage layout on the disk is (some kind of "backend" plugins). *And* there are plugins which are users of the "reiser4 client API" and implement the actual VFS methods. There's a UNIX directory plugin and a UNIX file plugin. Directories, inodes and file content are just key:object pairs and the plugins know how to operate on these. There's a new plugin in work that also implements UNIX file semantics but stores the data for that file encrypted and/or compressed. These plugins live between the VFS and the storage layer. Just like the filesystems live between the VFS and the block layer (Hans would say that filesystems are VFS plugins ;-)). > And what are the licensing implications of plugins? Are they derived > works? Must they be GPL'ed? I suppose yes, at least currently, since they can only be linked with reiser4, there's no module infrastructure. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 12:18 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 12:18 UTC (permalink / raw) To: Andrew Morton Cc: Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2289 bytes --] Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton: > And describe the "plugin" system. Why does the filesystem need such a > thing (other filesystems get their features via `patch -p1')? Ok, let me try. (Hans, please don't shoot me if I don't get every detail right, I'm trying to simplify and translate) The reiser4 core is just a big and fast storage layer using a single tree. It is bound to a mount point and uses the page cache to manage its memory. The only thing it can do on its own is to flush dirty data to the disk when the VM wants to (memory pressure) or the VFS wants to (unmount, sync) or some "plugin" wants to. Additionally it's completely atomic (with respect to crashes, no isolation like in databases), comparable to data=journal but without the overhead of using a journal for everything. Up to this point there are no users of this "database" and it does not implement any of the VFS methods for a filesystem (except mount and unmount perhaps). All the functionality is provided to the plugins. You can insert, remove, lookup or modify key:object pairs (with an index into the object). The object is a sequence of units. For files a unit would be 1 byte and the index would be the byte offset. For directories the unit would be 1 entry and the index would be the filename. Now there are some plugins that define how the storage layout on the disk is (some kind of "backend" plugins). *And* there are plugins which are users of the "reiser4 client API" and implement the actual VFS methods. There's a UNIX directory plugin and a UNIX file plugin. Directories, inodes and file content are just key:object pairs and the plugins know how to operate on these. There's a new plugin in work that also implements UNIX file semantics but stores the data for that file encrypted and/or compressed. These plugins live between the VFS and the storage layer. Just like the filesystems live between the VFS and the block layer (Hans would say that filesystems are VFS plugins ;-)). > And what are the licensing implications of plugins? Are they derived > works? Must they be GPL'ed? I suppose yes, at least currently, since they can only be linked with reiser4, there's no module infrastructure. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:18 ` Christophe Saout (?) @ 2004-08-26 12:49 ` Christoph Hellwig 2004-08-26 13:00 ` Christophe Saout -1 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:49 UTC (permalink / raw) To: Christophe Saout Cc: Andrew Morton, Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 02:18:49PM +0200, Christophe Saout wrote: > The reiser4 core is just a big and fast storage layer using a single > tree. It is bound to a mount point and uses the page cache to manage its > memory. The only thing it can do on its own is to flush dirty data to > the disk when the VM wants to (memory pressure) or the VFS wants to > (unmount, sync) or some "plugin" wants to. > Additionally it's completely atomic (with respect to crashes, no > isolation like in databases), comparable to data=journal but without the > overhead of using a journal for everything. > > Up to this point there are no users of this "database" and it does not > implement any of the VFS methods for a filesystem (except mount and > unmount perhaps). Now that part is interesting from the how to sell it to non-Linux users POV, but not for the linux kernel. > All the functionality is provided to the plugins. You can insert, > remove, lookup or modify key:object pairs (with an index into the > object). The object is a sequence of units. For files a unit would be 1 > byte and the index would be the byte offset. For directories the unit > would be 1 entry and the index would be the filename. > > Now there are some plugins that define how the storage layout on the > disk is (some kind of "backend" plugins). > > *And* there are plugins which are users of the "reiser4 client API" and > implement the actual VFS methods. Here comes the problem. All the access checking/race avoidance/loop creation avoiced, in short the posix+extension semantics are implemented in the Linux VFS layer. If you want to allow a second access method (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity inside reiser4. But this is absolutely not what we want, because we moved to this common as all previous version in the different filesystems were buggy, and in fact there are problems that are simply not solveable at the level at all (rename on directories comes to mind). So if you want to provide an additional API you'll have to go through the VFS to get it right - ergo a plugin architecture for upper plugins is worthless. Plugins for the disk-format level are an interesting and workable idea on the other hand. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:49 ` Christoph Hellwig @ 2004-08-26 13:00 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 13:00 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1732 bytes --] Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig: > Now that part is interesting from the how to sell it to non-Linux users > POV, but not for the linux kernel. Why? The question was what these plugins are exactly. This is the answer. > > *And* there are plugins which are users of the "reiser4 client API" and > > implement the actual VFS methods. > > Here comes the problem. All the access checking/race avoidance/loop > creation avoiced, in short the posix+extension semantics are implemented > in the Linux VFS layer. If you want to allow a second access method > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity > inside reiser4. Are you actually listening? If you implement a filesystem there's a place where you have to implement the Linux VFS methods. I'm talking about inode_operations and these things. This has nothing to do with doing anything outside the Linux VFS. And I'm not talking about these metas either. These really don't belong inside the filesystem. > So if you want to provide an additional API you'll have to go through > the VFS to get it right - ergo a plugin architecture for upper plugins > is worthless. See, you didn't listen. ;-) The reiser4 core doesn't know about inodes, directories or files. It's the glue code between the VFS and the storage layer that does. It's implemented as a plugin. This has nothing to do with semantic enhancements yet. These should be removed for now and made a 2.7 topic. Maybe Namesys could provide them as an additional patch from their webpage for those who need it. The plugins that would benefit from this (encrypted and compressed file storage) aren't ready yet anyway. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 13:00 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 13:00 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1732 bytes --] Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig: > Now that part is interesting from the how to sell it to non-Linux users > POV, but not for the linux kernel. Why? The question was what these plugins are exactly. This is the answer. > > *And* there are plugins which are users of the "reiser4 client API" and > > implement the actual VFS methods. > > Here comes the problem. All the access checking/race avoidance/loop > creation avoiced, in short the posix+extension semantics are implemented > in the Linux VFS layer. If you want to allow a second access method > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity > inside reiser4. Are you actually listening? If you implement a filesystem there's a place where you have to implement the Linux VFS methods. I'm talking about inode_operations and these things. This has nothing to do with doing anything outside the Linux VFS. And I'm not talking about these metas either. These really don't belong inside the filesystem. > So if you want to provide an additional API you'll have to go through > the VFS to get it right - ergo a plugin architecture for upper plugins > is worthless. See, you didn't listen. ;-) The reiser4 core doesn't know about inodes, directories or files. It's the glue code between the VFS and the storage layer that does. It's implemented as a plugin. This has nothing to do with semantic enhancements yet. These should be removed for now and made a 2.7 topic. Maybe Namesys could provide them as an additional patch from their webpage for those who need it. The plugins that would benefit from this (encrypted and compressed file storage) aren't ready yet anyway. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:00 ` Christophe Saout (?) @ 2004-08-26 13:07 ` Christoph Hellwig 2004-08-26 13:17 ` Christophe Saout ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 13:07 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 03:00:34PM +0200, Christophe Saout wrote: > Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig: > > > Now that part is interesting from the how to sell it to non-Linux users > > POV, but not for the linux kernel. > > Why? The question was what these plugins are exactly. This is the > answer. I explained it below.. > > > *And* there are plugins which are users of the "reiser4 client API" and > > > implement the actual VFS methods. > > > > Here comes the problem. All the access checking/race avoidance/loop > > creation avoiced, in short the posix+extension semantics are implemented > > in the Linux VFS layer. If you want to allow a second access method > > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity > > inside reiser4. > > Are you actually listening? If you implement a filesystem there's a > place where you have to implement the Linux VFS methods. I'm talking > about inode_operations and these things. This has nothing to do with > doing anything outside the Linux VFS. And I'm not talking about these > metas either. These really don't belong inside the filesystem. as I wrote in this mail this absolutely _does_ belong in the filesystem, it's a major part of it and isn't easily separatable. > > So if you want to provide an additional API you'll have to go through > > the VFS to get it right - ergo a plugin architecture for upper plugins > > is worthless. > > See, you didn't listen. ;-) > > The reiser4 core doesn't know about inodes, directories or files. It's > the glue code between the VFS and the storage layer that does. It's > implemented as a plugin. This has nothing to do with semantic > enhancements yet. These should be removed for now and made a 2.7 topic. Oh yes, it is. As soon as you use different access methods on an overlapping set of objects you see exactly the problems I described. If they don't overlap there's no point for the plugins to start with, you'll better turn the core into a library that can be used by different projects then. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:07 ` Christoph Hellwig @ 2004-08-26 13:17 ` Christophe Saout 2004-08-26 13:27 ` Christophe Saout 2004-08-26 14:44 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 13:17 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1907 bytes --] Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig: > > Why? The question was what these plugins are exactly. This is the > > answer. > > I explained it below.. It's hard to talk to you if you're not even trying to understand what I'm trying to say. Andrew wanted to know what these plugins are. I tried to give an answer and then you come along to tell me that it's not relevant. Sure, Linux doesn't care about what the filesystem does internally but Andrew wanted to know that. > > Are you actually listening? If you implement a filesystem there's a > > place where you have to implement the Linux VFS methods. I'm talking > > about inode_operations and these things. This has nothing to do with > > doing anything outside the Linux VFS. And I'm not talking about these > > metas either. These really don't belong inside the filesystem. > > as I wrote in this mail this absolutely _does_ belong in the filesystem, > it's a major part of it and isn't easily separatable. Huh? Now you're completely confusing me. First you say that that file-as-a-directory is crap then you say that it does belong into the filesystem? I'll try again: inode_operations, etc... are implemented as a "reiser4 plugin". This means that reiser4 is usable as filesystem, you can store files in a directory hierarchy. There's absolute nothing special here. Absolutely nothing. And yes, reiser4 plugins are invisible from the VFS. It looks like a normal filesystem because it behaves like one. The "metas" I was referring in the second part are this file-as-a- directory thing. These don't belong in the filesystem and should therefore be removed. The latter doesn't necessarily have anything to do with plugins. The plugins are plugins for the storage layer, not for some semantic enhancement. Note that I changed the topic to avoid further confusion. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) @ 2004-08-26 13:17 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 13:17 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1907 bytes --] Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig: > > Why? The question was what these plugins are exactly. This is the > > answer. > > I explained it below.. It's hard to talk to you if you're not even trying to understand what I'm trying to say. Andrew wanted to know what these plugins are. I tried to give an answer and then you come along to tell me that it's not relevant. Sure, Linux doesn't care about what the filesystem does internally but Andrew wanted to know that. > > Are you actually listening? If you implement a filesystem there's a > > place where you have to implement the Linux VFS methods. I'm talking > > about inode_operations and these things. This has nothing to do with > > doing anything outside the Linux VFS. And I'm not talking about these > > metas either. These really don't belong inside the filesystem. > > as I wrote in this mail this absolutely _does_ belong in the filesystem, > it's a major part of it and isn't easily separatable. Huh? Now you're completely confusing me. First you say that that file-as-a-directory is crap then you say that it does belong into the filesystem? I'll try again: inode_operations, etc... are implemented as a "reiser4 plugin". This means that reiser4 is usable as filesystem, you can store files in a directory hierarchy. There's absolute nothing special here. Absolutely nothing. And yes, reiser4 plugins are invisible from the VFS. It looks like a normal filesystem because it behaves like one. The "metas" I was referring in the second part are this file-as-a- directory thing. These don't belong in the filesystem and should therefore be removed. The latter doesn't necessarily have anything to do with plugins. The plugins are plugins for the storage layer, not for some semantic enhancement. Note that I changed the topic to avoid further confusion. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:17 ` Christophe Saout @ 2004-08-26 13:24 ` Christoph Hellwig -1 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 13:24 UTC (permalink / raw) To: Christophe Saout Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 2520 bytes --] On Thu, Aug 26, 2004 at 03:17:53PM +0200, Christophe Saout wrote: > It's hard to talk to you if you're not even trying to understand what > I'm trying to say. Andrew wanted to know what these plugins are. I tried > to give an answer and then you come along to tell me that it's not > relevant. Sure, Linux doesn't care about what the filesystem does > internally but Andrew wanted to know that. Huh? I don understand what reiser4 plugins are, and I tell you that a subset of these are _wrong_ for use in the kernel. > > > Are you actually listening? If you implement a filesystem there's a > > > place where you have to implement the Linux VFS methods. I'm talking > > > about inode_operations and these things. This has nothing to do with > > > doing anything outside the Linux VFS. And I'm not talking about these > > > metas either. These really don't belong inside the filesystem. > > > > as I wrote in this mail this absolutely _does_ belong in the filesystem, > > it's a major part of it and isn't easily separatable. > > Huh? > > Now you're completely confusing me. > > First you say that that file-as-a-directory is crap then you say that it > does belong into the filesystem? I think you're talking about something different then me, I'm not talking about the magic meta files but the VFS interface in general. This VFS interface is an integral part of ævery filesystem, and it doesn't make a whole lot to put it into a plugin. If you want your filesystem core portable it does to a certain extent make sense to abstract them out, but as someone who's worked on a few such 'portable' filesystems I can tell you that it doesn't work out as expected. > I'll try again: > > inode_operations, etc... are implemented as a "reiser4 plugin". This > means that reiser4 is usable as filesystem, you can store files in a > directory hierarchy. There's absolute nothing special here. Absolutely > nothing. And yes, reiser4 plugins are invisible from the VFS. It looks > like a normal filesystem because it behaves like one. Now kill the whole plugin mess and let them talk directly to another. My first mail explained to you why it doesn't make sense to have multiple such plugins to work on a common subset of data. > The latter doesn't necessarily have anything to do with plugins. > The plugins are plugins for the storage layer, not for some semantic > enhancement. Again plugins below the pagecache (if that's what you call the "storage layer") _do_ make sense. plugins at the semantic layer don't. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) @ 2004-08-26 13:24 ` Christoph Hellwig 0 siblings, 0 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 13:24 UTC (permalink / raw) To: Christophe Saout Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 2774 bytes --] On Thu, Aug 26, 2004 at 03:17:53PM +0200, Christophe Saout wrote: > It's hard to talk to you if you're not even trying to understand what > I'm trying to say. Andrew wanted to know what these plugins are. I tried > to give an answer and then you come along to tell me that it's not > relevant. Sure, Linux doesn't care about what the filesystem does > internally but Andrew wanted to know that. Huh? I don understand what reiser4 plugins are, and I tell you that a subset of these are _wrong_ for use in the kernel. > > > Are you actually listening? If you implement a filesystem there's a > > > place where you have to implement the Linux VFS methods. I'm talking > > > about inode_operations and these things. This has nothing to do with > > > doing anything outside the Linux VFS. And I'm not talking about these > > > metas either. These really don't belong inside the filesystem. > > > > as I wrote in this mail this absolutely _does_ belong in the filesystem, > > it's a major part of it and isn't easily separatable. > > Huh? > > Now you're completely confusing me. > > First you say that that file-as-a-directory is crap then you say that it > does belong into the filesystem? I think you're talking about something different then me, I'm not talking about the magic meta files but the VFS interface in general. This VFS interface is an integral part of ævery filesystem, and it doesn't make a whole lot to put it into a plugin. If you want your filesystem core portable it does to a certain extent make sense to abstract them out, but as someone who's worked on a few such 'portable' filesystems I can tell you that it doesn't work out as expected. > I'll try again: > > inode_operations, etc... are implemented as a "reiser4 plugin". This > means that reiser4 is usable as filesystem, you can store files in a > directory hierarchy. There's absolute nothing special here. Absolutely > nothing. And yes, reiser4 plugins are invisible from the VFS. It looks > like a normal filesystem because it behaves like one. Now kill the whole plugin mess and let them talk directly to another. My first mail explained to you why it doesn't make sense to have multiple such plugins to work on a common subset of data. > The latter doesn't necessarily have anything to do with plugins. > The plugins are plugins for the storage layer, not for some semantic > enhancement. Again plugins below the pagecache (if that's what you call the "storage layer") _do_ make sense. plugins at the semantic layer don't. - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:24 ` Christoph Hellwig @ 2004-08-26 13:35 ` Christophe Saout -1 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 13:35 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1905 bytes --] Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig: > > First you say that that file-as-a-directory is crap then you say that it > > does belong into the filesystem? > > I think you're talking about something different then me, I'm not > talking about the magic meta files but the VFS interface in general. > > This VFS interface is an integral part of very filesystem, and it > doesn't make a whole lot to put it into a plugin. Right. That's why these plugins are linked in uncoditionally. It doesn't work without them. Hence "plugins" is not a very good name. > If you want your > filesystem core portable it does to a certain extent make sense to > abstract them out, but as someone who's worked on a few such 'portable' > filesystems I can tell you that it doesn't work out as expected. That's your opinion. reiser4 seems to work very well. > Now kill the whole plugin mess and let them talk directly to another. This is how reiserfs v3 did it. And Hans says that it actually complicated things. > My first mail explained to you why it doesn't make sense to have > multiple such plugins to work on a common subset of data. What I understood is that you can select exactly one plugin that e.g. handles the file data. The default plugin is optimized for normal files, another one could implement transparent compression or encryption. Some of these plugins also give the storage layer hints how to group files together to optimized performance. Neither of these things mess with the VFS. > > The latter doesn't necessarily have anything to do with plugins. > > The plugins are plugins for the storage layer, not for some semantic > > enhancement. > > Again plugins below the pagecache (if that's what you call the "storage > layer") _do_ make sense. plugins at the semantic layer don't. Yes, that's what I was trying to say. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) @ 2004-08-26 13:35 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 13:35 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1905 bytes --] Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig: > > First you say that that file-as-a-directory is crap then you say that it > > does belong into the filesystem? > > I think you're talking about something different then me, I'm not > talking about the magic meta files but the VFS interface in general. > > This VFS interface is an integral part of very filesystem, and it > doesn't make a whole lot to put it into a plugin. Right. That's why these plugins are linked in uncoditionally. It doesn't work without them. Hence "plugins" is not a very good name. > If you want your > filesystem core portable it does to a certain extent make sense to > abstract them out, but as someone who's worked on a few such 'portable' > filesystems I can tell you that it doesn't work out as expected. That's your opinion. reiser4 seems to work very well. > Now kill the whole plugin mess and let them talk directly to another. This is how reiserfs v3 did it. And Hans says that it actually complicated things. > My first mail explained to you why it doesn't make sense to have > multiple such plugins to work on a common subset of data. What I understood is that you can select exactly one plugin that e.g. handles the file data. The default plugin is optimized for normal files, another one could implement transparent compression or encryption. Some of these plugins also give the storage layer hints how to group files together to optimized performance. Neither of these things mess with the VFS. > > The latter doesn't necessarily have anything to do with plugins. > > The plugins are plugins for the storage layer, not for some semantic > > enhancement. > > Again plugins below the pagecache (if that's what you call the "storage > layer") _do_ make sense. plugins at the semantic layer don't. Yes, that's what I was trying to say. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:35 ` Christophe Saout (?) @ 2004-08-26 13:40 ` Christoph Hellwig 2004-08-26 13:58 ` Christophe Saout 2004-08-26 23:54 ` Hans Reiser -1 siblings, 2 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 13:40 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 03:35:07PM +0200, Christophe Saout wrote: > > If you want your > > filesystem core portable it does to a certain extent make sense to > > abstract them out, but as someone who's worked on a few such 'portable' > > filesystems I can tell you that it doesn't work out as expected. > > That's your opinion. reiser4 seems to work very well. So how many OSes is it ported to currently? The problem is not that it doesn't work but that it's really hard to maintain. And remember that this maintaince overhead is not just for Hans but for everyone that does VFS-level work. Unless of course we get a blanko permission to break it as soon as fixing the mess becomes non-trivial. > What I understood is that you can select exactly one plugin that e.g. > handles the file data. The default plugin is optimized for normal files, > another one could implement transparent compression or encryption. Some > of these plugins also give the storage layer hints how to group files > together to optimized performance. Neither of these things mess with the > VFS. compression or encryption must sit below the pagecache to work nicely, and this hint things that usually sit at the pagecache level. But let's assume you have a valid use for different file_operations, why don't you simply add in different file_operations instead of adding another internal dispatch layer? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:40 ` Christoph Hellwig @ 2004-08-26 13:58 ` Christophe Saout 2004-08-26 15:37 ` Christoph Hellwig 2004-08-26 23:55 ` Hans Reiser 2004-08-26 23:54 ` Hans Reiser 1 sibling, 2 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 13:58 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2493 bytes --] Am Donnerstag, den 26.08.2004, 15:40 +0200 schrieb Christoph Hellwig: > > That's your opinion. reiser4 seems to work very well. > > So how many OSes is it ported to currently? The problem is not that it > doesn't work but that it's really hard to maintain. And remember that > this maintaince overhead is not just for Hans but for everyone that does > VFS-level work. Where does it add overhead? It's just the interface between the reiser4 storage layer and the rest abstracted out. Ok, there's the code that dispatches the VFS method calls to the plugin call but I would say that's about it. I didn't develop reiser4 so I don't know. /* ->mkdir() VFS method in reiser4 inode_operations */ static int reiser4_mkdir(struct inode *parent /* inode of parent * directory */ , struct dentry *dentry /* dentry of new object to * create */ , int mode /* new object's mode */ ) { reiser4_object_create_data data; reiser4_stat_inc_at(parent->i_sb, vfs_calls.mkdir); data.mode = S_IFDIR | mode; data.id = DIRECTORY_FILE_PLUGIN_ID; return invoke_create_method(parent, dentry, &data); } The invoke_create_method then just calls the create method of the plugin after setting up a context and lookup up the plugin. xfs_iops.c seems to do something similar. > Unless of course we get a blanko permission to break it > as soon as fixing the mess becomes non-trivial. Well, I think so. It's done relatively often. > > What I understood is that you can select exactly one plugin that e.g. > > handles the file data. The default plugin is optimized for normal files, > > another one could implement transparent compression or encryption. Some > > of these plugins also give the storage layer hints how to group files > > together to optimized performance. Neither of these things mess with the > > VFS. > > compression or encryption must sit below the pagecache to work nicely, > and this hint things that usually sit at the pagecache level. But let's > assume you have a valid use for different file_operations, why don't you > simply add in different file_operations instead of adding another > internal dispatch layer? I don't know, ask Hans. How could the VFS know it a filesystem wants to do something specific with a file that is completely transparent to the VFS? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:58 ` Christophe Saout @ 2004-08-26 15:37 ` Christoph Hellwig 2004-08-26 15:48 ` Christophe Saout 2004-08-26 23:55 ` Hans Reiser 1 sibling, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 15:37 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > > compression or encryption must sit below the pagecache to work nicely, > > and this hint things that usually sit at the pagecache level. But let's > > assume you have a valid use for different file_operations, why don't you > > simply add in different file_operations instead of adding another > > internal dispatch layer? > > I don't know, ask Hans. How could the VFS know it a filesystem wants to > do something specific with a file that is completely transparent to the > VFS? The VFS shouldn't, that the whole point. That's why it allows the filesystem to register different method tables for each object. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 15:37 ` Christoph Hellwig @ 2004-08-26 15:48 ` Christophe Saout 2004-08-26 16:06 ` Christoph Hellwig 0 siblings, 1 reply; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 15:48 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1191 bytes --] Am Donnerstag, den 26.08.2004, 17:37 +0200 schrieb Christoph Hellwig: > > > compression or encryption must sit below the pagecache to work nicely, > > > and this hint things that usually sit at the pagecache level. But let's > > > assume you have a valid use for different file_operations, why don't you > > > simply add in different file_operations instead of adding another > > > internal dispatch layer? > > > > I don't know, ask Hans. How could the VFS know it a filesystem wants to > > do something specific with a file that is completely transparent to the > > VFS? > > The VFS shouldn't, that the whole point. That's why it allows the > filesystem to register different method tables for each object. Only the objects it can distinguish. > ops->file = reiser4_file_operations; > ops->symlink = reiser4_symlink_inode_operations; > ops->special = reiser4_special_inode_operations; > ops->dentry = reiser4_dentry_operations; > ops->as = reiser4_as_operations; How could reiser4 register other operations for files that should be stored encrypted or compressed? It's all under reiser4_file_operations. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 15:48 ` Christophe Saout @ 2004-08-26 16:06 ` Christoph Hellwig 2004-08-26 16:14 ` Christophe Saout 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:06 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 05:48:54PM +0200, Christophe Saout wrote: > > > I don't know, ask Hans. How could the VFS know it a filesystem wants to > > > do something specific with a file that is completely transparent to the > > > VFS? > > > > The VFS shouldn't, that the whole point. That's why it allows the > > filesystem to register different method tables for each object. > > Only the objects it can distinguish. Yes, every inode can have different operation vectors. Which is the smallest possible object the VFS knows about. > > ops->file = reiser4_file_operations; > > ops->symlink = reiser4_symlink_inode_operations; > > ops->special = reiser4_special_inode_operations; > > ops->dentry = reiser4_dentry_operations; > > ops->as = reiser4_as_operations; > > How could reiser4 register other operations for files that should be > stored encrypted or compressed? It's all under reiser4_file_operations. Again, your confusing upper and lower plugins. For things happening below the pagecache you could register different address_space operations which sometimes makes sense. But you want e.g. different inode_operations for directories vs symlinks vs files. Please read through some linux filesystem code, okay :) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 16:06 ` Christoph Hellwig @ 2004-08-26 16:14 ` Christophe Saout 2004-08-26 16:25 ` Christoph Hellwig 0 siblings, 1 reply; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 16:14 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 986 bytes --] Am Donnerstag, den 26.08.2004, 18:06 +0200 schrieb Christoph Hellwig: > Again, your confusing upper and lower plugins. For things happening > below the pagecache you could register different address_space > operations which sometimes makes sense. But you want e.g. different > inode_operations for directories vs symlinks vs files. Ok, I see your point. aops. Sorry. Looking at the code this could be done. The wrappers that dispatch the operations are really small and call the plugin that is registered with the inode of the mapping. Instead it could have directly set the corresponding operations. Right. The wrappers are doing a few things before calling the plugin. That could be done the other way round too. But that's more of an implementation issue and could still be changed. > Please read through some linux filesystem code, okay :) I'm partly familiar with that, I just either misunderstood you or wasn't really thinking (probably the latter...). [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 16:14 ` Christophe Saout @ 2004-08-26 16:25 ` Christoph Hellwig 2004-08-26 16:39 ` Christophe Saout 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:25 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 06:14:19PM +0200, Christophe Saout wrote: > Ok, I see your point. aops. Sorry. > Looking at the code this could be done. The wrappers that dispatch the > operations are really small and call the plugin that is registered with > the inode of the mapping. Instead it could have directly set the > corresponding operations. Right. The wrappers are doing a few things > before calling the plugin. That could be done the other way round too. > But that's more of an implementation issue and could still be changed. I agree that it's an implementation issue. But it's also a good proof for how Hans tries to ignore all the existing infrastructure for various reasons. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 16:25 ` Christoph Hellwig @ 2004-08-26 16:39 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 16:39 UTC (permalink / raw) To: Christoph Hellwig Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 964 bytes --] Am Donnerstag, den 26.08.2004, 17:25 +0100 schrieb Christoph Hellwig: > > Looking at the code this could be done. The wrappers that dispatch the > > operations are really small and call the plugin that is registered with > > the inode of the mapping. Instead it could have directly set the > > corresponding operations. Right. The wrappers are doing a few things > > before calling the plugin. That could be done the other way round too. > > But that's more of an implementation issue and could still be changed. > > I agree that it's an implementation issue. But it's also a good proof > for how Hans tries to ignore all the existing infrastructure for various > reasons. I definitely agree with you on that one. It bugged me from the beginning because things like these are very likely to suffer from unnecessary problems that have been solved in the common code or might be solved when correctly done in the common code in the future. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) @ 2004-08-26 16:39 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 16:39 UTC (permalink / raw) To: Christoph Hellwig Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 964 bytes --] Am Donnerstag, den 26.08.2004, 17:25 +0100 schrieb Christoph Hellwig: > > Looking at the code this could be done. The wrappers that dispatch the > > operations are really small and call the plugin that is registered with > > the inode of the mapping. Instead it could have directly set the > > corresponding operations. Right. The wrappers are doing a few things > > before calling the plugin. That could be done the other way round too. > > But that's more of an implementation issue and could still be changed. > > I agree that it's an implementation issue. But it's also a good proof > for how Hans tries to ignore all the existing infrastructure for various > reasons. I definitely agree with you on that one. It bugged me from the beginning because things like these are very likely to suffer from unnecessary problems that have been solved in the common code or might be solved when correctly done in the common code in the future. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-26 13:58 ` Christophe Saout @ 2004-08-26 23:55 ` Hans Reiser 2004-08-26 23:55 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:55 UTC (permalink / raw) To: Christoph Hellwig Cc: Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to >do something specific with a file that is completely transparent to the >VFS? > > > To know what method to use, you must determine the pluginid, and then find the method within that plugin for that vfs operation. As for overhead, well, who eats whose dust in the benchmarks....? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins @ 2004-08-26 23:55 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:55 UTC (permalink / raw) To: Christoph Hellwig Cc: Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to >do something specific with a file that is completely transparent to the >VFS? > > > To know what method to use, you must determine the pluginid, and then find the method within that plugin for that vfs operation. As for overhead, well, who eats whose dust in the benchmarks....? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-26 23:55 ` Hans Reiser (?) @ 2004-08-27 12:04 ` Nikita Danilov 2004-08-27 18:15 ` Hans Reiser 2004-08-28 9:41 ` Redeeman -1 siblings, 2 replies; 1835+ messages in thread From: Nikita Danilov @ 2004-08-27 12:04 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser writes: > Christophe Saout wrote: > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > >do something specific with a file that is completely transparent to the > >VFS? > > > > > > > To know what method to use, you must determine the pluginid, and then > find the method within that plugin for that vfs operation. > > As for overhead, well, who eats whose dust in the benchmarks....? Whoever sponsors the benchmark usually wins. Had you forgotten that mongo setup used by http://www.namesys.com/benchmarks.html was specially `tuned' to reach peak reiser4 performance? Remember why you decided to turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 _second_ stalls with reiser4 under high io loads (large atom is obviously being flushed and everyone waits on it...). In my opinion, it is such things that are of utmost importance for real reiser4 acceptance, not how to name `metas' sub-directory. Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-27 12:04 ` Nikita Danilov @ 2004-08-27 18:15 ` Hans Reiser 2004-08-28 9:41 ` Redeeman 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 18:15 UTC (permalink / raw) To: Nikita Danilov Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Nikita Danilov wrote: >Hans Reiser writes: > > Christophe Saout wrote: > > > > > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > > >do something specific with a file that is completely transparent to the > > >VFS? > > > > > > > > > > > To know what method to use, you must determine the pluginid, and then > > find the method within that plugin for that vfs operation. > > > > As for overhead, well, who eats whose dust in the benchmarks....? > >Whoever sponsors the benchmark usually wins. Had you forgotten that >mongo setup used by http://www.namesys.com/benchmarks.html was specially >`tuned' to reach peak reiser4 performance? Remember why you decided to >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 >_second_ stalls with reiser4 under high io loads (large atom is >obviously being flushed and everyone waits on it...). In my opinion, it >is such things that are of utmost importance for real reiser4 >acceptance, not how to name `metas' sub-directory. > >Nikita. > > > > If you ask real users, they say that reiser4 is fast, and their experience matches our benchmark. You can criticize the benchmark if you want, but then you should run your own and publish it. There are still plenty of things needing fixing though, and you are right that they are more important than the pretext being attempted for keeping us out of the kernel. Probably you could supply them with many superior pretexts.;-) The 90 second stalls, those should be fixed by debugging copy on capture and turning it on, yes? You can hardly claim I failed to push for the completion of copy on capture.... and of course the reason it is not done is simply that there is too much to do.... I (sincerely) hope vs enjoys his vacation, the whole team is working too hard. I am doing 7 day weeks now to try to make payroll for us and keep darpa happy at the same time. fsync performance needs fixing badly, and Chris is right in his diagnosis of what we can do to fix it. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins @ 2004-08-27 18:15 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 18:15 UTC (permalink / raw) To: Nikita Danilov Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Nikita Danilov wrote: >Hans Reiser writes: > > Christophe Saout wrote: > > > > > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > > >do something specific with a file that is completely transparent to the > > >VFS? > > > > > > > > > > > To know what method to use, you must determine the pluginid, and then > > find the method within that plugin for that vfs operation. > > > > As for overhead, well, who eats whose dust in the benchmarks....? > >Whoever sponsors the benchmark usually wins. Had you forgotten that >mongo setup used by http://www.namesys.com/benchmarks.html was specially >`tuned' to reach peak reiser4 performance? Remember why you decided to >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 >_second_ stalls with reiser4 under high io loads (large atom is >obviously being flushed and everyone waits on it...). In my opinion, it >is such things that are of utmost importance for real reiser4 >acceptance, not how to name `metas' sub-directory. > >Nikita. > > > > If you ask real users, they say that reiser4 is fast, and their experience matches our benchmark. You can criticize the benchmark if you want, but then you should run your own and publish it. There are still plenty of things needing fixing though, and you are right that they are more important than the pretext being attempted for keeping us out of the kernel. Probably you could supply them with many superior pretexts.;-) The 90 second stalls, those should be fixed by debugging copy on capture and turning it on, yes? You can hardly claim I failed to push for the completion of copy on capture.... and of course the reason it is not done is simply that there is too much to do.... I (sincerely) hope vs enjoys his vacation, the whole team is working too hard. I am doing 7 day weeks now to try to make payroll for us and keep darpa happy at the same time. fsync performance needs fixing badly, and Chris is right in his diagnosis of what we can do to fix it. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-27 18:15 ` Hans Reiser (?) @ 2004-08-27 18:55 ` Nikita Danilov 2004-08-28 9:53 ` Hans Reiser 2004-08-29 11:17 ` Alex Zarochentsev -1 siblings, 2 replies; 1835+ messages in thread From: Nikita Danilov @ 2004-08-27 18:55 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser writes: > Nikita Danilov wrote: > > >Hans Reiser writes: > > > Christophe Saout wrote: > > > > > > > > > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > > > >do something specific with a file that is completely transparent to the > > > >VFS? > > > > > > > > > > > > > > > To know what method to use, you must determine the pluginid, and then > > > find the method within that plugin for that vfs operation. > > > > > > As for overhead, well, who eats whose dust in the benchmarks....? > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > >mongo setup used by http://www.namesys.com/benchmarks.html was specially > >`tuned' to reach peak reiser4 performance? Remember why you decided to > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 > >_second_ stalls with reiser4 under high io loads (large atom is > >obviously being flushed and everyone waits on it...). In my opinion, it > >is such things that are of utmost importance for real reiser4 > >acceptance, not how to name `metas' sub-directory. > > > >Nikita. > > > > > > > > > If you ask real users, they say that reiser4 is fast, and their > experience matches our benchmark. You can criticize the benchmark if They experience 90 second stalls. And please, do not tell me how fast reiser4 is, I spent a lot of time working with it, and I know very well when it's fast, and when it's deadly slow. > you want, but then you should run your own and publish it. I did, after which you told me to turn OVERWRITE and MODIFY phases off, beause performance was horrible. I not criticizing mongo benchmark per se. I think that it is fundamentally wrong to use results that were deliberatly manipulated to get best appearance to reiser4 (by omitting work-loads where it performs poorly) as an argument. It's not clear who will, according to your colorful expression, `eat dust' as a result of this. Or do you think that users never overwrite of modify files in real life? > The 90 second stalls, those should be fixed by debugging copy on capture > and turning it on, yes? You can hardly claim I failed to push for the No. I described so many times to you, why COC is ineffectual here. > Hans Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-27 18:55 ` Nikita Danilov @ 2004-08-28 9:53 ` Hans Reiser 2004-08-29 11:17 ` Alex Zarochentsev 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 9:53 UTC (permalink / raw) To: Nikita Danilov Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, Alexander Zarochentcev Nikita Danilov wrote: > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially > > >`tuned' to reach peak reiser4 performance? Remember why you decided to > > >turn OVERWRITE and MODIFY phases off? > What I should have done was what I did with fsync performance. With fsync performance I told people that we had not yet tuned for it, please wait a bit and we will tune for it, for now it sucks. Instead what I did was discuss with Zam at the time how it could be fixed, leave it off the website until Zam was given a chance to fix it, and then I managed to forget about it. After release one remembers what all the things that should have been fixed before release were, sigh. Zam and I both know what is needed to fix performance in these phases, and I just spoke with him on the phone. I imagine it is 2 weeks of work for him, but it could be up to 6 weeks. He will comment later. The fix should consist of the following: 1) tweak the relocation policies (zam will say more about this, as he is the maintainer of them) 2) optimize overwrite set so that in the modify phase we behave like a write twice journaling filesystem, which means implement a reserved for the journal only area that exists as long as plenty of disk space is free. 3) finish the repacker (but we won't do that this month....) The modify phase, which picks a random block to modify, is a worst case for a wandering log without a repacker. We could actually do very very well at that kind of activity with a repacker, probably better than a fixed journal, but for now we should try acting like a write twice journal for atoms that small. So, having realized my error thanks to the gracious kindness of how you pointed it out, we will modify the web site to say that we are not yet tuned for and currently suck at modifying random blocks in the middle of files, and appending small amounts to the end of them, and overwriting small files, but that we think we know what is needed to fix those things. I think your characterization of my reasons was unkind and also unfair. I will pass on saying similarly unkind things about you as you are mostly a good person. You never did seem to understand me. You didn't understand the reasons for my technical designs (when they worked it made no sense to you that they did), and you didn't understand me as a person. I wish you well in your new job, and thank you for the hard work you did for me. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins @ 2004-08-28 9:53 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 9:53 UTC (permalink / raw) To: Nikita Danilov Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, Alexander Zarochentcev Nikita Danilov wrote: > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially > > >`tuned' to reach peak reiser4 performance? Remember why you decided to > > >turn OVERWRITE and MODIFY phases off? > What I should have done was what I did with fsync performance. With fsync performance I told people that we had not yet tuned for it, please wait a bit and we will tune for it, for now it sucks. Instead what I did was discuss with Zam at the time how it could be fixed, leave it off the website until Zam was given a chance to fix it, and then I managed to forget about it. After release one remembers what all the things that should have been fixed before release were, sigh. Zam and I both know what is needed to fix performance in these phases, and I just spoke with him on the phone. I imagine it is 2 weeks of work for him, but it could be up to 6 weeks. He will comment later. The fix should consist of the following: 1) tweak the relocation policies (zam will say more about this, as he is the maintainer of them) 2) optimize overwrite set so that in the modify phase we behave like a write twice journaling filesystem, which means implement a reserved for the journal only area that exists as long as plenty of disk space is free. 3) finish the repacker (but we won't do that this month....) The modify phase, which picks a random block to modify, is a worst case for a wandering log without a repacker. We could actually do very very well at that kind of activity with a repacker, probably better than a fixed journal, but for now we should try acting like a write twice journal for atoms that small. So, having realized my error thanks to the gracious kindness of how you pointed it out, we will modify the web site to say that we are not yet tuned for and currently suck at modifying random blocks in the middle of files, and appending small amounts to the end of them, and overwriting small files, but that we think we know what is needed to fix those things. I think your characterization of my reasons was unkind and also unfair. I will pass on saying similarly unkind things about you as you are mostly a good person. You never did seem to understand me. You didn't understand the reasons for my technical designs (when they worked it made no sense to you that they did), and you didn't understand me as a person. I wish you well in your new job, and thank you for the hard work you did for me. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 9:53 ` Hans Reiser (?) @ 2004-08-28 13:47 ` Nikita Danilov 2004-08-28 14:10 ` Konstantin Besch 2004-08-28 23:45 ` Hans Reiser -1 siblings, 2 replies; 1835+ messages in thread From: Nikita Danilov @ 2004-08-28 13:47 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, Alexander Zarochentcev Hans Reiser writes: > Nikita Danilov wrote: > > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially > > > >`tuned' to reach peak reiser4 performance? Remember why you decided to > > > >turn OVERWRITE and MODIFY phases off? > > > What I should have done was what I did with fsync performance. With > fsync performance I told people that we had not yet tuned for it, please > wait a bit and we will tune for it, for now it sucks. Right, and instead of this you are now claiming that these `benchmarks' show superior reiser4 performance. Moreover, you are doing this aggressively and proposing other people to `eat the dust' over what happened to only exist due to your failing to remember something. As some other notable LKML poster put it `inform yourself before posting'. :-) > > Instead what I did was discuss with Zam at the time how it could be > fixed, leave it off the website until Zam was given a chance to fix it, It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were turned off after switching to large keys. > and then I managed to forget about it. After release one remembers > what all the things that should have been fixed before release were, sigh. > [...] > > I think your characterization of my reasons was unkind and also unfair. What fairness and kindness one expects after styling others as `puppies' in public, and commenting on their hair style in purportedly technical argument? [...] > > Hans Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 13:47 ` Nikita Danilov @ 2004-08-28 14:10 ` Konstantin Besch 2004-08-28 22:09 ` Hans Reiser 2004-08-28 23:45 ` Hans Reiser 1 sibling, 1 reply; 1835+ messages in thread From: Konstantin Besch @ 2004-08-28 14:10 UTC (permalink / raw) To: reiserfs-list > I wish you well in your new job, and thank you for the hard work you did > for me. > Hans Oh yeah , Nikita , had to say this : BUT thank you for that hard work you did for us all. It seems it is 'bout time to move out of the company, producing public software, where leader of the project thanks participants for the work done with words "for me" at the end. Always had my doubts that I should thanks Hans Reiser for the FS :) And the facts are that my servers for now get ext3. Good Day in New York ! --Sincerely. On Saturday 28 August 2004 9:47 am, Nikita Danilov wrote: > Hans Reiser writes: > > Nikita Danilov wrote: > > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > > > > >mongo setup used by http://www.namesys.com/benchmarks.html was > > > > > specially `tuned' to reach peak reiser4 performance? Remember why > > > > > you decided to turn OVERWRITE and MODIFY phases off? > > > > What I should have done was what I did with fsync performance. With > > fsync performance I told people that we had not yet tuned for it, please > > wait a bit and we will tune for it, for now it sucks. > > Right, and instead of this you are now claiming that these `benchmarks' > show superior reiser4 performance. Moreover, you are doing this > aggressively and proposing other people to `eat the dust' over what > happened to only exist due to your failing to remember something. As > some other notable LKML poster put it `inform yourself before > posting'. :-) > > > Instead what I did was discuss with Zam at the time how it could be > > fixed, leave it off the website until Zam was given a chance to fix it, > > It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were > turned off after switching to large keys. > > > and then I managed to forget about it. After release one remembers > > what all the things that should have been fixed before release were, > > sigh. > > [...] > > > I think your characterization of my reasons was unkind and also unfair. > > What fairness and kindness one expects after styling others as `puppies' > in public, and commenting on their hair style in purportedly technical > argument? > > [...] > > > Hans > > Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 14:10 ` Konstantin Besch @ 2004-08-28 22:09 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 22:09 UTC (permalink / raw) To: Konstantin Besch; +Cc: reiserfs-list Konstantin Besch wrote: >>I wish you well in your new job, and thank you for the hard work you did >>for me. >> >> > > > >> Hans >> >> > >Oh yeah , Nikita , had to say this : BUT thank you for that hard work you did >for us all. >It seems it is 'bout time to move out of the company, producing public >software, where leader of the project thanks participants for the work done >with words "for me" at the end. > >Always had my doubts that I should thanks Hans Reiser for the FS :) >And the facts are that my servers for now get ext3. >Good Day in New York ! > > --Sincerely. > > > I paid his salary with my credit cards. Under usual english rules and capitalist traditions, that means that saying that he was working for me is socially acceptable english, even if I understood that he worked for more than the money. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 13:47 ` Nikita Danilov 2004-08-28 14:10 ` Konstantin Besch @ 2004-08-28 23:45 ` Hans Reiser 2004-08-29 9:35 ` Nikita Danilov 1 sibling, 1 reply; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 23:45 UTC (permalink / raw) To: Nikita Danilov Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, Alexander Zarochentcev Nikita Danilov wrote: > > > > Instead what I did was discuss with Zam at the time how it could be > > fixed, leave it off the website until Zam was given a chance to fix it, > >It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were >turned off after switching to large keys. > > > The online repacker is the best technical fix for these problems. Tight packing is generally more damaged in its layout by small changes than loose packing, and V4 is the ultimate tight packer. It is not clear to me that making V4 pack more loosely is necessarily a good idea. These phases disturb the original packing at flush time, and the other phases don't. One solution to try is packing less tightly. Maybe once every megabyte skip forward 1 megabyte, and once every 64 megabytes skip to a random disk location, when allocating large new atoms. I think I prefer using a repacker, but we should try it and see. The other solutions of the previous email should also help a lot. The issues of layout in this are similar to the ones improved by the fibration plugin. I remember talking with not just you but zam about how we could fix it, and there was too much in the queue of work, and everyone was complaining to me that we should be debugging not optimizing, and you were the only one who thought it was a big deal. I guess you still think it is a big issue and I still think things are good enough to use without it. I still think Zam understood the issues better than you did (allocation is his code not yours). There are some layout optimizations that a repacker can do best. Still, there are some rough spots in the current allocation policy, and we should look at it. Probably you will have reason to howl if we setup a benchmark which disturbs things with these phases, runs the repacker (I hope to have one in 6 months), and then measures our read performance compared to other filesystems without a repacker....;-) Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 23:45 ` Hans Reiser @ 2004-08-29 9:35 ` Nikita Danilov 0 siblings, 0 replies; 1835+ messages in thread From: Nikita Danilov @ 2004-08-29 9:35 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, Alexander Zarochentcev Hans Reiser writes: [...] > > I remember talking with not just you but zam about how we could fix it, > and there was too much in the queue of work, and everyone was > complaining to me that we should be debugging not optimizing, and you > were the only one who thought it was a big deal. I guess you still You are trying to evade the point. And the point is not how performance in these phases (and their impact on other phases) can be improved, or what excuses did you have for not `doing' it, but the fact that after it was found that large keys behave badly, these phases were turned off, others were switched to lexicographical operation, _and_ resulting benchmark is used as a basis to call other people names. > think it is a big issue and I still think things are good enough to use > without it. I still think Zam understood the issues better than you did > (allocation is his code not yours). > > There are some layout optimizations that a repacker can do best. Still, > there are some rough spots in the current allocation policy, and we > should look at it. > > Probably you will have reason to howl if we setup a benchmark which Surely, as a `big dog' what other sounds can I produce? :) > disturbs things with these phases, runs the repacker (I hope to have one > in 6 months), and then measures our read performance compared to other > filesystems without a repacker....;-) That's ok with me, but it would make sense to remove unfair benchmarks from the site until then. > > Hans Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-27 18:55 ` Nikita Danilov 2004-08-28 9:53 ` Hans Reiser @ 2004-08-29 11:17 ` Alex Zarochentsev 1 sibling, 0 replies; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-29 11:17 UTC (permalink / raw) To: Nikita Danilov Cc: Hans Reiser, Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Fri, Aug 27, 2004 at 10:55:50PM +0400, Nikita Danilov wrote: > Hans Reiser writes: > > Nikita Danilov wrote: > > > > >Hans Reiser writes: > > > > Christophe Saout wrote: > > > > > > > > > > > > > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > > > > >do something specific with a file that is completely transparent to the > > > > >VFS? > > > > > > > > > > > > > > > > > > > To know what method to use, you must determine the pluginid, and then > > > > find the method within that plugin for that vfs operation. > > > > > > > > As for overhead, well, who eats whose dust in the benchmarks....? > > > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially > > >`tuned' to reach peak reiser4 performance? Remember why you decided to > > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 > > >_second_ stalls with reiser4 under high io loads (large atom is > > >obviously being flushed and everyone waits on it...). In my opinion, it > > >is such things that are of utmost importance for real reiser4 > > >acceptance, not how to name `metas' sub-directory. > > > > > >Nikita. > > > > > > > > > > > > > > If you ask real users, they say that reiser4 is fast, and their > > experience matches our benchmark. You can criticize the benchmark if > > They experience 90 second stalls. And please, do not tell me how fast > reiser4 is, I spent a lot of time working with it, and I know very well > when it's fast, and when it's deadly slow. > > > you want, but then you should run your own and publish it. > > I did, after which you told me to turn OVERWRITE and MODIFY phases off, > beause performance was horrible. Hmm, I think not the modify/overwrite phases performance were the problem. The modify and overwrite mongo phases fragment the filesystem too much, it had a bad effect visible in the read phase and even in delete phase. We measured/optimized read performance in assumption that fs is not fragmented, why we should measure bad effect of overwrite instead? other fs do not relocate already allocated blocks, negative effect of file set overwrite is zero for them. We could do the same for reiser4 by tuning flush algorithm and journal blocks allocation. If users want to see read and delete performance as they are, overwrite phase should be excluded, at least until we teach flush algorithm to not break good block allocation. If one wants to measure overwite performance, it is easy to enable overwrite phase in mongo, but read and delete speed will be affected (it was so when I tried to optimize delete). > I not criticizing mongo benchmark per se. I think that it is > fundamentally wrong to use results that were deliberatly manipulated to > get best appearance to reiser4 (by omitting work-loads where it performs > poorly) as an argument. It's not clear who will, according to your > colorful expression, `eat dust' as a result of this. > Or do you think > that users never overwrite of modify files in real life? not whole file set. I mean it is unusual to overwrite all files in the fs. > > The 90 second stalls, those should be fixed by debugging copy on capture > > and turning it on, yes? You can hardly claim I failed to push for the > > No. I described so many times to you, why COC is ineffectual here. There are other ideas :) but, it seems to me, if we will focus on benchmarks, we never complete plugins, metafiles and other useful things. What I want from reiser4 is a possiblity to encrypt and compress ~/Mail/ :). > > > Hans > > Nikita. -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-27 18:15 ` Hans Reiser @ 2004-08-27 22:29 ` Steve Bergman -1 siblings, 0 replies; 1835+ messages in thread From: Steve Bergman @ 2004-08-27 22:29 UTC (permalink / raw) To: Hans Reiser Cc: Nikita Danilov, Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote: > > > If you ask real users, they say that reiser4 is fast, and their > experience matches our benchmark. You can criticize the benchmark if > you want, but then you should run your own and publish it. > As a "real" desktop user who just converted all his partitions from ext3 to reiser4, I have not, as yet, noticed any startling performance increase. Being slightly slightly irked to hear that the benchmark numbers that have been paraded around on Slashdot and the internet in general, at ext3's expense, have had reiser4's "bad" results surgically extracted, I am running my own benchmarks to get the real story on reiser4/ext3 mongo performance on my, rather average, desktop hardware. I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel. Unfortunately, I get an error from mongo.pl that "Done" is not a numeric argument at line 439. I have done this to fix it: --- mongo.pl 2004-08-27 17:07:01.681723313 -0500 +++ mongo_fixed.pl 2004-08-27 17:06:51.369306735 -0500 @@ -429,8 +429,8 @@ if ( -e ${ERR_FILE}) { &DIE ("\nEXITED WITH FAIL\n"); } - my $real = (split ' ', $time_output[0])[1]; - my $cpu = (split ' ', $time_output[2])[1]; + my $real = (split ' ', $time_output[1])[1]; + my $cpu = (split ' ', $time_output[3])[1]; unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) { LOG "@time_output"; What it gets me is the "real" line of the "time" output for "STAT REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME". i.e. only system time is counted. I believe this was the intent of the original code, but want to verify before continuing. Thanks, Steve Bergman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins @ 2004-08-27 22:29 ` Steve Bergman 0 siblings, 0 replies; 1835+ messages in thread From: Steve Bergman @ 2004-08-27 22:29 UTC (permalink / raw) To: Hans Reiser Cc: Nikita Danilov, Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote: > > > If you ask real users, they say that reiser4 is fast, and their > experience matches our benchmark. You can criticize the benchmark if > you want, but then you should run your own and publish it. > As a "real" desktop user who just converted all his partitions from ext3 to reiser4, I have not, as yet, noticed any startling performance increase. Being slightly slightly irked to hear that the benchmark numbers that have been paraded around on Slashdot and the internet in general, at ext3's expense, have had reiser4's "bad" results surgically extracted, I am running my own benchmarks to get the real story on reiser4/ext3 mongo performance on my, rather average, desktop hardware. I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel. Unfortunately, I get an error from mongo.pl that "Done" is not a numeric argument at line 439. I have done this to fix it: --- mongo.pl 2004-08-27 17:07:01.681723313 -0500 +++ mongo_fixed.pl 2004-08-27 17:06:51.369306735 -0500 @@ -429,8 +429,8 @@ if ( -e ${ERR_FILE}) { &DIE ("\nEXITED WITH FAIL\n"); } - my $real = (split ' ', $time_output[0])[1]; - my $cpu = (split ' ', $time_output[2])[1]; + my $real = (split ' ', $time_output[1])[1]; + my $cpu = (split ' ', $time_output[3])[1]; unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) { LOG "@time_output"; What it gets me is the "real" line of the "time" output for "STAT REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME". i.e. only system time is counted. I believe this was the intent of the original code, but want to verify before continuing. Thanks, Steve Bergman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-27 22:29 ` Steve Bergman (?) @ 2004-08-28 6:54 ` Hans Reiser 2004-08-28 10:09 ` Steve Bergman -1 siblings, 1 reply; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 6:54 UTC (permalink / raw) To: Steve Bergman Cc: Nikita Danilov, Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs, Alexander Zarochentcev Steve Bergman wrote: >On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote: > > >>If you ask real users, they say that reiser4 is fast, and their >>experience matches our benchmark. You can criticize the benchmark if >>you want, but then you should run your own and publish it. >> >> >> > > >As a "real" desktop user who just converted all his partitions from ext3 >to reiser4, I have not, as yet, noticed any startling performance >increase. Being slightly slightly irked to hear that the benchmark >numbers that have been paraded around on Slashdot and the internet in >general, at ext3's expense, have had reiser4's "bad" results surgically >extracted, I am running my own benchmarks to get the real story on >reiser4/ext3 mongo performance on my, rather average, desktop hardware. > >I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel. > >Unfortunately, I get an error from mongo.pl that "Done" is not a numeric >argument at line 439. > >I have done this to fix it: > > >--- mongo.pl 2004-08-27 17:07:01.681723313 -0500 >+++ mongo_fixed.pl 2004-08-27 17:06:51.369306735 -0500 >@@ -429,8 +429,8 @@ > if ( -e ${ERR_FILE}) { > &DIE ("\nEXITED WITH FAIL\n"); > } >- my $real = (split ' ', $time_output[0])[1]; >- my $cpu = (split ' ', $time_output[2])[1]; >+ my $real = (split ' ', $time_output[1])[1]; >+ my $cpu = (split ' ', $time_output[3])[1]; > > unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) { > LOG "@time_output"; > > >What it gets me is the "real" line of the "time" output for "STAT >REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME". >i.e. only system time is counted. I believe this was the intent of the >original code, but want to verify before continuing. > >Thanks, >Steve Bergman > > > > > I didn't write this (more precisely, it only vaguely resembles what I wrote in 1996). Are you saying that it reports system time as real time? If yes, then it is an error, we need to go remove a bunch of numbers from our benchmarks, and thanks for finding it. Zam, please comment. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 6:54 ` Hans Reiser @ 2004-08-28 10:09 ` Steve Bergman 2004-08-29 11:52 ` Alex Zarochentsev 2004-08-29 11:55 ` Alex Zarochentsev 0 siblings, 2 replies; 1835+ messages in thread From: Steve Bergman @ 2004-08-28 10:09 UTC (permalink / raw) To: Hans Reiser; +Cc: Alexander Zarochentcev, Nikita Danilov, reiserfs On Fri, 2004-08-27 at 23:54 -0700, Hans Reiser wrote: > > > I didn't write this (more precisely, it only vaguely resembles what I > wrote in 1996). Are you saying that it reports system time as real > time? If yes, then it is an error, we need to go remove a bunch of > numbers from our benchmarks, and thanks for finding it. > > Zam, please comment. > No. I'm saying that on my setup (kernel 2.6.8.1-mm4/perl 5.8.5/bash 3.00) the first line returned from running the test is the: [1]+ Done ... line which throws the array indexes off by 1 and I always get 0 for the real time and, yes, the real time gets reported as the system time, I think. Plus I get a warning about the fact that "Done" is not numeric. This is obviously a problem with my particular setup. After bumping the indexes up by 1, I get the correct real time reported as "STAT REAL_TIME". And the system time is reported as "STAT CPU_TIME". "STAT CPU_UTIL", however is computed in a completely different way based on numbers collected from /proc/stat. If I'm, reading linux/Documentation/filesystems/proc.txt correctly, it is trying to return the (system time) / (total time). The total time agrees with "STAT REAL_TIME". However, the (system time) that it comes up with here is always considerably higher than "STAT CPU_TIME". User error is quite possible, as I am: 1. Just getting to know mongo. 2. Not a perl guy. 3. There is obviously something mongo.pl doesn't like about my setup. -Steve Bergman P.S. In the 2 tests I've run, reiser4 is not doing all that badly against ext3 in OVERWRITE and MODIFY, though ext3 does come in faster. Reiser4 trails badly on APPEND however, ext3 (data=ordered, without htree) being some 2.5 - 4 times faster. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 10:09 ` Steve Bergman @ 2004-08-29 11:52 ` Alex Zarochentsev 2004-08-29 11:55 ` Alex Zarochentsev 1 sibling, 0 replies; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-29 11:52 UTC (permalink / raw) To: Steve Bergman; +Cc: Hans Reiser, Nikita Danilov, reiserfs On Sat, Aug 28, 2004 at 05:09:28AM -0500, Steve Bergman wrote: > On Fri, 2004-08-27 at 23:54 -0700, Hans Reiser wrote: > > > > > I didn't write this (more precisely, it only vaguely resembles what I > > wrote in 1996). Are you saying that it reports system time as real > > time? If yes, then it is an error, we need to go remove a bunch of > > numbers from our benchmarks, and thanks for finding it. > > > > Zam, please comment. > > > > No. I'm saying that on my setup (kernel 2.6.8.1-mm4/perl 5.8.5/bash > 3.00) the first line returned from running the test is the: > > [1]+ Done ... > line which throws the array indexes off by 1 and I always get 0 for the > real time and, yes, the real time gets reported as the system time, I > think. Plus I get a warning about the fact that "Done" is not numeric. > This is obviously a problem with my particular setup. Yes. but that real/sys time parsing isn't reliable. > After bumping the indexes up by 1, I get the correct real time reported > as "STAT REAL_TIME". And the system time is reported as "STAT > CPU_TIME". > > "STAT CPU_UTIL", however is computed in a completely different way based > on numbers collected from /proc/stat. If I'm, reading > linux/Documentation/filesystems/proc.txt correctly, it is trying to > return the (system time) / (total time). The total time agrees with > "STAT REAL_TIME". However, the (system time) that it comes up with here > is always considerably higher than "STAT CPU_TIME". CPU_UTIL counts other background processes too. The foreground process may just wait when all work is done by pdflush. I think CPU_UTIL is more useful than CPU_TIME. > > User error is quite possible, as I am: > > 1. Just getting to know mongo. > 2. Not a perl guy. i don't like perl too much, but seems Perl is perfect for the things you just did (the fixes). > 3. There is obviously something mongo.pl doesn't like about my setup. > > -Steve Bergman > > P.S. In the 2 tests I've run, reiser4 is not doing all that badly > against ext3 in OVERWRITE and MODIFY, though ext3 does come in faster. > Reiser4 trails badly on APPEND however, ext3 (data=ordered, without > htree) being some 2.5 - 4 times faster. -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 10:09 ` Steve Bergman 2004-08-29 11:52 ` Alex Zarochentsev @ 2004-08-29 11:55 ` Alex Zarochentsev 1 sibling, 0 replies; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-29 11:55 UTC (permalink / raw) To: Steve Bergman; +Cc: Hans Reiser, Nikita Danilov, reiserfs, linux-kernel On Sat, Aug 28, 2004 at 05:09:28AM -0500, Steve Bergman wrote: > On Fri, 2004-08-27 at 23:54 -0700, Hans Reiser wrote: > > > > > I didn't write this (more precisely, it only vaguely resembles what I > > wrote in 1996). Are you saying that it reports system time as real > > time? If yes, then it is an error, we need to go remove a bunch of > > numbers from our benchmarks, and thanks for finding it. > > > > Zam, please comment. > > > > No. I'm saying that on my setup (kernel 2.6.8.1-mm4/perl 5.8.5/bash > 3.00) the first line returned from running the test is the: > > [1]+ Done ... > line which throws the array indexes off by 1 and I always get 0 for the > real time and, yes, the real time gets reported as the system time, I > think. Plus I get a warning about the fact that "Done" is not numeric. > This is obviously a problem with my particular setup. Yes. but that real/sys time parsing isn't reliable. > After bumping the indexes up by 1, I get the correct real time reported > as "STAT REAL_TIME". And the system time is reported as "STAT > CPU_TIME". > > "STAT CPU_UTIL", however is computed in a completely different way based > on numbers collected from /proc/stat. If I'm, reading > linux/Documentation/filesystems/proc.txt correctly, it is trying to > return the (system time) / (total time). The total time agrees with > "STAT REAL_TIME". However, the (system time) that it comes up with here > is always considerably higher than "STAT CPU_TIME". CPU_UTIL counts other background processes too. The foreground process may just wait when all work is done by pdflush. I think CPU_UTIL is more useful than CPU_TIME. > > User error is quite possible, as I am: > > 1. Just getting to know mongo. > 2. Not a perl guy. i don't like perl too much, but seems Perl is perfect for the things you just did (the fixes). > 3. There is obviously something mongo.pl doesn't like about my setup. > > -Steve Bergman > > P.S. In the 2 tests I've run, reiser4 is not doing all that badly > against ext3 in OVERWRITE and MODIFY, though ext3 does come in faster. > Reiser4 trails badly on APPEND however, ext3 (data=ordered, without > htree) being some 2.5 - 4 times faster. -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-27 22:29 ` Steve Bergman (?) (?) @ 2004-08-29 11:42 ` Alex Zarochentsev -1 siblings, 0 replies; 1835+ messages in thread From: Alex Zarochentsev @ 2004-08-29 11:42 UTC (permalink / raw) To: Steve Bergman Cc: Hans Reiser, Nikita Danilov, Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs On Fri, Aug 27, 2004 at 05:29:07PM -0500, Steve Bergman wrote: > On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote: > > > > > If you ask real users, they say that reiser4 is fast, and their > > experience matches our benchmark. You can criticize the benchmark if > > you want, but then you should run your own and publish it. > > > > > As a "real" desktop user who just converted all his partitions from ext3 > to reiser4, I have not, as yet, noticed any startling performance > increase. Being slightly slightly irked to hear that the benchmark > numbers that have been paraded around on Slashdot and the internet in > general, at ext3's expense, have had reiser4's "bad" results surgically > extracted, I am running my own benchmarks to get the real story on > reiser4/ext3 mongo performance on my, rather average, desktop hardware. > > I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel. > > Unfortunately, I get an error from mongo.pl that "Done" is not a numeric > argument at line 439. > > I have done this to fix it: > > > --- mongo.pl 2004-08-27 17:07:01.681723313 -0500 > +++ mongo_fixed.pl 2004-08-27 17:06:51.369306735 -0500 > @@ -429,8 +429,8 @@ > if ( -e ${ERR_FILE}) { > &DIE ("\nEXITED WITH FAIL\n"); > } > - my $real = (split ' ', $time_output[0])[1]; > - my $cpu = (split ' ', $time_output[2])[1]; > + my $real = (split ' ', $time_output[1])[1]; > + my $cpu = (split ' ', $time_output[3])[1]; > > unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) { > LOG "@time_output"; > > > What it gets me is the "real" line of the "time" output for "STAT > REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME". > i.e. only system time is counted. I believe this was the intent of the > original code, but want to verify before continuing. it works on our test servers which are SuSE9.0/9.1. I think there is a dependency on system utilities, "Done" is not printed by mongo.pl or any other program from the mongo package. It would be nice to tell us what linux distro you are using and what mongo phase fails. > Thanks, > Steve Bergman Thanks. -- Alex. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-27 12:04 ` Nikita Danilov 2004-08-27 18:15 ` Hans Reiser @ 2004-08-28 9:41 ` Redeeman 1 sibling, 0 replies; 1835+ messages in thread From: Redeeman @ 2004-08-28 9:41 UTC (permalink / raw) To: Reiserfs Mailinglist On Fri, 2004-08-27 at 16:04 +0400, Nikita Danilov wrote: > Hans Reiser writes: > > Christophe Saout wrote: > > > > > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > > >do something specific with a file that is completely transparent to the > > >VFS? > > > > > > > > > > > To know what method to use, you must determine the pluginid, and then > > find the method within that plugin for that vfs operation. > > > > As for overhead, well, who eats whose dust in the benchmarks....? > > Whoever sponsors the benchmark usually wins. Had you forgotten that > mongo setup used by http://www.namesys.com/benchmarks.html was specially > `tuned' to reach peak reiser4 performance? Remember why you decided to > turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 > _second_ stalls with reiser4 under high io loads (large atom is > obviously being flushed and everyone waits on it...). In my opinion, it > is such things that are of utmost importance for real reiser4 > acceptance, not how to name `metas' sub-directory. i am operating alot under heavy loads, often alot compiling, seti and some other applications, and i must say, i never noticed this > > Nikita. > -- Redeeman <redeeman@metanurb.dk> ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-26 13:40 ` Christoph Hellwig @ 2004-08-26 23:54 ` Hans Reiser 2004-08-26 23:54 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:54 UTC (permalink / raw) To: Christoph Hellwig Cc: Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: > > The problem is not that it >doesn't work but that it's really hard to maintain. > You really like to talk about what you know nothing of, yes? Reiser4 is FAR easier to maintain than V3. Ask anyone on reiserfs-dev. It was designed to reduce our software maintenance costs. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins @ 2004-08-26 23:54 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:54 UTC (permalink / raw) To: Christoph Hellwig Cc: Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: > > The problem is not that it >doesn't work but that it's really hard to maintain. > You really like to talk about what you know nothing of, yes? Reiser4 is FAR easier to maintain than V3. Ask anyone on reiserfs-dev. It was designed to reduce our software maintenance costs. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:35 ` Christophe Saout @ 2004-08-26 13:52 ` Rik van Riel -1 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 13:52 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 26 Aug 2004, Christophe Saout wrote: > That's your opinion. reiser4 seems to work very well. Have you tried /bin/cp, or a backup+restore ? What happened to your file streams ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) @ 2004-08-26 13:52 ` Rik van Riel 0 siblings, 0 replies; 1835+ messages in thread From: Rik van Riel @ 2004-08-26 13:52 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 26 Aug 2004, Christophe Saout wrote: > That's your opinion. reiser4 seems to work very well. Have you tried /bin/cp, or a backup+restore ? What happened to your file streams ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:52 ` Rik van Riel (?) @ 2004-08-26 13:58 ` Chris Mason -1 siblings, 0 replies; 1835+ messages in thread From: Chris Mason @ 2004-08-26 13:58 UTC (permalink / raw) To: Rik van Riel Cc: Christophe Saout, Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 09:52, Rik van Riel wrote: > On Thu, 26 Aug 2004, Christophe Saout wrote: > > > That's your opinion. reiser4 seems to work very well. > > Have you tried /bin/cp, or a backup+restore ? > > What happened to your file streams ? Shrug, backup programs can be fixed. The fact that we couldn't backup 16GB files before O_LARGEFILE wasn't a reason not to do it. IMHO it's more important to ask the question 'is this important for linux' and then we can figure out how to make it work. -chris ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:52 ` Rik van Riel @ 2004-08-26 14:02 ` Christophe Saout -1 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 14:02 UTC (permalink / raw) To: Rik van Riel Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 560 bytes --] Am Donnerstag, den 26.08.2004, 09:52 -0400 schrieb Rik van Riel: > > That's your opinion. reiser4 seems to work very well. > > Have you tried /bin/cp, or a backup+restore ? > > What happened to your file streams ? 1. I wasn't talking about reiser4 plugins, not about the file-as-a-directory thing or file streams 2. reiser4 doesn't currently have those 3. I don't really like file streams :-) But I'm using ACLs on a server and had to forward-port an rsync patch to make full backups using rsync possible. That's not really better. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) @ 2004-08-26 14:02 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 14:02 UTC (permalink / raw) To: Rik van Riel Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 560 bytes --] Am Donnerstag, den 26.08.2004, 09:52 -0400 schrieb Rik van Riel: > > That's your opinion. reiser4 seems to work very well. > > Have you tried /bin/cp, or a backup+restore ? > > What happened to your file streams ? 1. I wasn't talking about reiser4 plugins, not about the file-as-a-directory thing or file streams 2. reiser4 doesn't currently have those 3. I don't really like file streams :-) But I'm using ACLs on a server and had to forward-port an rsync patch to make full backups using rsync possible. That's not really better. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:35 ` Christophe Saout ` (2 preceding siblings ...) (?) @ 2004-08-28 11:18 ` Alexander Lyamin 2004-08-28 11:22 ` Christoph Hellwig 2004-08-28 16:11 ` Jan Harkes -1 siblings, 2 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 11:18 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Thu, Aug 26, 2004 at 03:35:07PM +0200, Christophe Saout wrote: > Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig: > > > > First you say that that file-as-a-directory is crap then you say that it > > > does belong into the filesystem? > > > > I think you're talking about something different then me, I'm not > > talking about the magic meta files but the VFS interface in general. > > > > This VFS interface is an integral part of very filesystem, and it > > doesn't make a whole lot to put it into a plugin. > > Right. That's why these plugins are linked in uncoditionally. It doesn't > work without them. Hence "plugins" is not a very good name. its still plugins no matter what. they just emulate "conventional filesystem" behavior for VFS, but at some point you might want just to scrap VFS.. if you'd like to. And I honestly dont understand whats the other Christoph's worries are about. Its got perfomance. Its there. It can emulate "conventional filesystem" behaviour, for legacy apps. Thouse two that are currently crippled by metas, we are so happy arguing about, will get fixed fast. Thats a point where you could happily STOP and live with your happy "conventional filesystem emulation", yet enjoying perfomance aspects (if perfomance hurts you, do not compile reiserfs, just like one big-red distro does, nobody would not complain. except your hard-drive mechanics sick of seeks). But most people would not stop, and its good. Yes, I think it would be nice to have this infrastructure in VFS. Technically. But its not possible, cause of "committee clusterfuck". Socially. Stupidly. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:18 ` Alexander Lyamin @ 2004-08-28 11:22 ` Christoph Hellwig 2004-08-28 11:46 ` Alexander Lyamin 2004-08-28 16:11 ` Jan Harkes 1 sibling, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-28 11:22 UTC (permalink / raw) To: flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > Yes, I think it would be nice to have this infrastructure in VFS. Technically. > But its not possible, cause of "committee clusterfuck". Socially. Stupidly. Please explain your problems. I've not seen a single actually working proposal yet. Current reiser4 implementation apparently oopses and breaks guaranteed fs semantics, it's not going in in either reiser4 or the vfs. Al has started a thread to hash out working semantics, but there's not been a single namesys person involved. Similar all of you have absolutely ignore every single technical question or comment but started a flamefest consisting mostly of personal attacs and "mine is longer" politics. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:22 ` Christoph Hellwig @ 2004-08-28 11:46 ` Alexander Lyamin 2004-08-28 11:51 ` Christoph Hellwig 2004-08-28 11:58 ` Markus Törnqvist 0 siblings, 2 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 11:46 UTC (permalink / raw) To: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 01:22:55PM +0200, Christoph Hellwig wrote: > On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > > Yes, I think it would be nice to have this infrastructure in VFS. Technically. > > But its not possible, cause of "committee clusterfuck". Socially. Stupidly. > > Please explain your problems. I've not seen a single actually working > proposal yet. Current reiser4 implementation apparently oopses and Not true. If true - send bugreports. Its work for me couple of months. there were few hiccups, but they got fixed quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS limitations. Yes, i use it with LVM2 and stuff... de facto its even more stable then reiserfs 3.6, which got issues with CONFIG_PREEMPT. > breaks guaranteed fs semantics, it's not going in in either reiser4 or > the vfs. A > > Al has started a thread to hash out working semantics, but there's not been > a single namesys person involved. Similar all of you have absolutely ignore Al message had a reply. "namesys persons" is Hans Reiser. Sufficient ? > every single technical question or comment but started a flamefest > consisting mostly of personal attacs and "mine is longer" politics. no need to comment on this. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:46 ` Alexander Lyamin @ 2004-08-28 11:51 ` Christoph Hellwig 2004-08-28 12:18 ` Alexander Lyamin 2004-08-28 11:58 ` Markus Törnqvist 1 sibling, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-28 11:51 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote: > Not true. If true - send bugreports. > > Its work for me couple of months. there were few hiccups, but they got fixed > quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS > limitations. Yes, i use it with LVM2 and stuff... See the mails from Christophe in this thread. > > breaks guaranteed fs semantics, it's not going in in either reiser4 or > > the vfs. > A > > > > Al has started a thread to hash out working semantics, but there's not been > > a single namesys person involved. Similar all of you have absolutely ignore > Al message had a reply. > "namesys persons" is Hans Reiser. Sufficient ? http://marc.theaimsgroup.com/?t=109355582600002&r=1&w=2 I can't see Hans anywhere. And honestly Hans has been so out of touch with VFS internals that some person actually understanding the issue might be helpfull. That would probably whoever has taken over Nikita's position. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:51 ` Christoph Hellwig @ 2004-08-28 12:18 ` Alexander Lyamin 0 siblings, 0 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 12:18 UTC (permalink / raw) To: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 01:51:04PM +0200, Christoph Hellwig wrote: > On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote: > > Its work for me couple of months. there were few hiccups, but they got fixed > > quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS > > limitations. Yes, i use it with LVM2 and stuff... > > See the mails from Christophe in this thread. seen. noted. > > > breaks guaranteed fs semantics, it's not going in in either reiser4 or > > > the vfs. > > A > > > > > > Al has started a thread to hash out working semantics, but there's not been > > > a single namesys person involved. Similar all of you have absolutely ignore > > Al message had a reply. > > "namesys persons" is Hans Reiser. Sufficient ? > > http://marc.theaimsgroup.com/?t=109355582600002&r=1&w=2 > > I can't see Hans anywhere. And honestly Hans has been so out of touch > with VFS internals that some person actually understanding the issue > might be helpfull. That would probably whoever has taken over Nikita's > position. I assume Vladimir Saveliev could play this out... -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:46 ` Alexander Lyamin 2004-08-28 11:51 ` Christoph Hellwig @ 2004-08-28 11:58 ` Markus Törnqvist 1 sibling, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-28 11:58 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote: >Not true. If true - send bugreports. Hate to say it, but apparently symlinking out of metas oopses, but that should be trivial to fix? >> Al has started a thread to hash out working semantics, but there's not been >> a single namesys person involved. Similar all of you have absolutely ignore >Al message had a reply. >"namesys persons" is Hans Reiser. Sufficient ? Not to be disrespectful, but Hans has sometimes taken personal shots on the lists and that's not a productive approach. Not that he'd be the only one around to do so. Seems to me, as Christoph said, that some other Namesys guy than Hans should participate in the Al/Linus branch of the thread with ideas. Certainly the VFS may have to be extended right about now, so someone who knows the exact details of Reiser4 should offer ideas and I'm not so sure it's Hans. Looks like he had the ideas but not the implementation. There's also the risk that Reiser4's implementation's don't really bend into even a modified/extended VFS because of the plugins? That's definitely something that has to be approached properly... -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:18 ` Alexander Lyamin 2004-08-28 11:22 ` Christoph Hellwig @ 2004-08-28 16:11 ` Jan Harkes 2004-08-28 16:26 ` Nikita Danilov 2004-08-28 19:41 ` Alexander Lyamin 1 sibling, 2 replies; 1835+ messages in thread From: Jan Harkes @ 2004-08-28 16:11 UTC (permalink / raw) To: flx, Christophe Saout, Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > And I honestly dont understand whats the other Christoph's worries are about. I honestly don't understand what all this 'grassroot' campaining is all about. Very valid technical points have been made and mostly ignored. The whole 'it breaks backups' noise is just that, noise. - It breaks VFS locking rules. - It pretends to provide a uniform way to store streams or metadata. However it only does so files and not for directories, symlinks, fifos, unix domain sockets, or device nodes, which makes it less than uniform. - From what I saw in one part of the discussion, it allows for infinite depth recursion (file/metas/metas/metas/...). Some applications are going to have a lot of fun with that. And finally, - When reiserfs3 got merged, it introduced iget3 and read_inode2 in the VFS layer. Later on when I started to use them for Coda I almost immediately found serious consistency problems, resulting in the iget4_locked implementation in the 2.5 kernels. I don't think anyone ever fixed that race in reiser3. It should hit occasionally on concurrent lookups on SMP or preempt kernels. In 2.4, Coda needed a semaphore to prevent concurrency during the iget3 lookup until the new inode is actually initialized. > Its got perfomance. Its there. It can emulate "conventional > filesystem" behaviour, for legacy apps. So does ramfs. > Yes, I think it would be nice to have this infrastructure in VFS. > Technically. But its not possible, cause of "committee clusterfuck". > Socially. Stupidly. It will get there, just consider it constructive criticism. If we can help resolve or refute the technical issues, all the better. We might even end up with improvements or extensions to the VFS or MM making life easier for everyone. Jan ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 16:11 ` Jan Harkes @ 2004-08-28 16:26 ` Nikita Danilov 2004-08-28 19:41 ` Alexander Lyamin 1 sibling, 0 replies; 1835+ messages in thread From: Nikita Danilov @ 2004-08-28 16:26 UTC (permalink / raw) To: Jan Harkes Cc: flx, Christophe Saout, Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Jan Harkes writes: > On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > > And I honestly dont understand whats the other Christoph's worries are about. > [...] > - When reiserfs3 got merged, it introduced iget3 and read_inode2 in the > VFS layer. Later on when I started to use them for Coda I almost > immediately found serious consistency problems, resulting in the > iget4_locked implementation in the 2.5 kernels. > > I don't think anyone ever fixed that race in reiser3. It should hit Err... it was fixed. > Jan > Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 16:11 ` Jan Harkes 2004-08-28 16:26 ` Nikita Danilov @ 2004-08-28 19:41 ` Alexander Lyamin 1 sibling, 0 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 19:41 UTC (permalink / raw) To: flx, Christophe Saout, Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 12:11:14PM -0400, Jan Harkes wrote: > On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > > It will get there, just consider it constructive criticism. If we can > help resolve or refute the technical issues, all the better. We might > even end up with improvements or extensions to the VFS or MM making life > easier for everyone. That would be nice. Some nice points too (metas/metas in just noice tough). -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:24 ` Christoph Hellwig (?) @ 2004-08-28 10:59 ` Alexander Lyamin -1 siblings, 0 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 10:59 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote: > This VFS interface is an integral part of Фvery filesystem, and it VFS never was "an integral part" of ANY filesystem. my dog knows it. its just unified INTERFACE TO any filesystem (including reiser4). P.S. I imagine, how much flamed it would be if reiser4 made any intensive changes in linux VFS code... -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) @ 2004-08-28 10:59 ` Alexander Lyamin 0 siblings, 0 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 10:59 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote: > This VFS interface is an integral part of Фvery filesystem, and it VFS never was "an integral part" of ANY filesystem. my dog knows it. its just unified INTERFACE TO any filesystem (including reiser4). P.S. I imagine, how much flamed it would be if reiser4 made any intensive changes in linux VFS code... -- "the liberation loophole will make it clear.." lex lyamin - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) @ 2004-08-28 10:59 ` Alexander Lyamin 0 siblings, 0 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 10:59 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote: > This VFS interface is an integral part of Фvery filesystem, and it VFS never was "an integral part" of ANY filesystem. my dog knows it. its just unified INTERFACE TO any filesystem (including reiser4). P.S. I imagine, how much flamed it would be if reiser4 made any intensive changes in linux VFS code... -- "the liberation loophole will make it clear.." lex lyamin - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 10:59 ` Alexander Lyamin (?) (?) @ 2004-08-28 11:12 ` Christoph Hellwig 2004-08-28 12:05 ` Alexander Lyamin -1 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-28 11:12 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote: > Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote: > > This VFS interface is an integral part of ??very filesystem, and it > > VFS never was "an integral part" of ANY filesystem. my dog knows it. > its just unified INTERFACE TO any filesystem (including reiser4). You's misquoting me. IF you quoted the whole context it'd be pretty sure that the part of the filesystem that intefaces with the VFS is meant. But one could even say VFS is integral part of a linux filesystem as it does most of the work a filesystem driver does in other operating systems. > P.S. I imagine, how much flamed it would be if reiser4 made any intensive > changes in linux VFS code... It really depends on how you sent them. If you had a big patch without explanations - sure. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:12 ` Christoph Hellwig @ 2004-08-28 12:05 ` Alexander Lyamin 2004-08-28 13:56 ` Christoph Hellwig 0 siblings, 1 reply; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 12:05 UTC (permalink / raw) To: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 01:12:33PM +0200, Christoph Hellwig wrote: > On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote: > > Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote: > > > This VFS interface is an integral part of ??very filesystem, and it > > > > VFS never was "an integral part" of ANY filesystem. my dog knows it. > > its just unified INTERFACE TO any filesystem (including reiser4). > > You's misquoting me. IF you quoted the whole context it'd be pretty > sure that the part of the filesystem that intefaces with the VFS is > meant. No. Its not me "misquoting", its just someone sound plain ?incoherrent?. Even if I overquoted reply with whole message, its still sound ?incohherent?. > > But one could even say VFS is integral part of a linux filesystem as > it does most of the work a filesystem driver does in other operating > systems. theres no "linux filesystem". there are "linux filesystems". thanks god. But I it would be really grate if you'll elaborate your sentence with example of VFS functionality (lack of it) on said "other operating systems" and if you'll define "most of work". > > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive > > changes in linux VFS code... > > It really depends on how you sent them. If you had a big patch without > explanations - sure. It would work with small tweaks, but you just can take a look at reiser4 code and you'll understand that it just could not be chopped in "set of small patches" altough it could be documented better ofcourse, but its really well commented already. some times, some approaches to some problems just would not work. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 12:05 ` Alexander Lyamin @ 2004-08-28 13:56 ` Christoph Hellwig 2004-08-28 19:23 ` Alexander Lyamin 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-28 13:56 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote: > > But one could even say VFS is integral part of a linux filesystem as > > it does most of the work a filesystem driver does in other operating > > systems. > > theres no "linux filesystem". there are "linux filesystems". > thanks god. a linux filesystem, not the linux filesystem, please read again. > But I it would be really grate if you'll elaborate your sentence with > example of VFS functionality (lack of it) on said "other operating systems" > and if you'll define "most of work". most trivial example is namespace locking, in *BSD, Windows, SVR4 and derivates it's done in the lowlevel filesystem. In plan9, Linux and soon DragonlyBSD it's done in the VFS. > > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive > > > changes in linux VFS code... > > > > It really depends on how you sent them. If you had a big patch without > > explanations - sure. > It would work with small tweaks, but you just can take a look at reiser4 > code and you'll understand that it just could not be chopped in > "set of small patches" altough it could be documented better ofcourse, > but its really well commented already. > > some times, some approaches to some problems just would not work. You still haven't even bother explaining what you want to do. It's hard to argue against vague uncertainity. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 13:56 ` Christoph Hellwig @ 2004-08-28 19:23 ` Alexander Lyamin 2004-08-28 19:27 ` Alexander Lyamin 2004-08-28 22:36 ` Hans Reiser 0 siblings, 2 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 19:23 UTC (permalink / raw) To: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 03:56:55PM +0200, Christoph Hellwig wrote: > On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote: > > > But one could even say VFS is integral part of a linux filesystem as > > > it does most of the work a filesystem driver does in other operating > > > systems. > > > > theres no "linux filesystem". there are "linux filesystems". > > thanks god. > > a linux filesystem, not the linux filesystem, please read again. > > > But I it would be really grate if you'll elaborate your sentence with > > example of VFS functionality (lack of it) on said "other operating systems" > > and if you'll define "most of work". > > most trivial example is namespace locking, in *BSD, Windows, SVR4 and > derivates it's done in the lowlevel filesystem. In plan9, Linux and > soon DragonlyBSD it's done in the VFS. ok. good examples. > > > > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive > > > > changes in linux VFS code... > > > > > > It really depends on how you sent them. If you had a big patch without > > > explanations - sure. > > It would work with small tweaks, but you just can take a look at reiser4 > > code and you'll understand that it just could not be chopped in > > "set of small patches" altough it could be documented better ofcourse, > > but its really well commented already. > > > > some times, some approaches to some problems just would not work. > > You still haven't even bother explaining what you want to do. It's hard > to argue against vague uncertainity. o files as directories - no oppinion on that. o metafiles - AFAIK it was product of Nikita Danilov just playing and fooling. Probably solution that will satisfy you is to have "legacy (dumb?) mode" which is leaving all this fancy stuff out of sight. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 19:23 ` Alexander Lyamin @ 2004-08-28 19:27 ` Alexander Lyamin 2004-08-28 22:36 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 19:27 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 11:23:50PM +0400, Alexander Lyamin wrote: > Sat, Aug 28, 2004 at 03:56:55PM +0200, Christoph Hellwig wrote: > > > > > > some times, some approaches to some problems just would not work. > > > > You still haven't even bother explaining what you want to do. It's hard > > to argue against vague uncertainity. > > o files as directories - no oppinion on that. > > o metafiles - AFAIK it was product of Nikita Danilov just playing and fooling. > > Probably solution that will satisfy you is to have "legacy (dumb?) mode" which > is leaving all this fancy stuff out of sight. and enabled compile time or mount option... -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 19:23 ` Alexander Lyamin @ 2004-08-28 22:36 ` Hans Reiser 2004-08-28 22:36 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 22:36 UTC (permalink / raw) To: flx Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Alexander Lyamin wrote: > > > o metafiles - AFAIK it was product of Nikita Danilov just playing and fooling. > > No, it was not, it was an idea I had for so long I cannot remember when, maybe 1984, and I proposed it to darpa, and then nikita got told about it probably from being told to read the web page on our darpa project, and he took on the task of coding it. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins @ 2004-08-28 22:36 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 22:36 UTC (permalink / raw) To: flx Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Alexander Lyamin wrote: > > > o metafiles - AFAIK it was product of Nikita Danilov just playing and fooling. > > No, it was not, it was an idea I had for so long I cannot remember when, maybe 1984, and I proposed it to darpa, and then nikita got told about it probably from being told to read the web page on our darpa project, and he took on the task of coding it. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 10:59 ` Alexander Lyamin ` (2 preceding siblings ...) (?) @ 2004-08-28 11:14 ` Markus Törnqvist -1 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-28 11:14 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote: >P.S. I imagine, how much flamed it would be if reiser4 made any intensive >changes in linux VFS code... Surely it would be flamefest galore, but thanks to Reiser4 there may be some VFS changes to that direction, maybe completely from Reiser4. Should keep some people at bay if the chances weren't so much replacing VFS than extending it. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 10:59 ` Alexander Lyamin ` (3 preceding siblings ...) (?) @ 2004-08-28 17:18 ` Linus Torvalds 2004-08-28 19:03 ` Alexander Lyamin -1 siblings, 1 reply; 1835+ messages in thread From: Linus Torvalds @ 2004-08-28 17:18 UTC (permalink / raw) To: Alexander Lyamin Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, 28 Aug 2004, Alexander Lyamin wrote: > > VFS never was "an integral part" of ANY filesystem. my dog knows it. That's not really true. Name handling (dentry layer, mounting) is very much an integral part of the filesystem. Almost everything else in the VFS is "helper functions", ie a filesystem can choose to ignore buffer heads, page cache etc, but a filesystem really cannot ignore or override the VFS naming stuff. (Arguably the page cache isn't part of the VFS layer at all, it's really a memory management thing, although it's so intertwined with the VFS helper functions that you can't really draw the line). > its just unified INTERFACE TO any filesystem (including reiser4). True, but that's not the whole truth. It's way more than just an interface. It's a set of rules, and it's in many way the controlling party. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 17:18 ` Linus Torvalds @ 2004-08-28 19:03 ` Alexander Lyamin 2004-08-28 19:09 ` Christoph Hellwig 2004-08-28 19:26 ` reiser4 plugins (was: silent semantic changes with reiser4) Linus Torvalds 0 siblings, 2 replies; 1835+ messages in thread From: Alexander Lyamin @ 2004-08-28 19:03 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Lyamin, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Sat, Aug 28, 2004 at 10:18:44AM -0700, Linus Torvalds wrote: > > > On Sat, 28 Aug 2004, Alexander Lyamin wrote: > > > > VFS never was "an integral part" of ANY filesystem. my dog knows it. > > That's not really true. Theres some truth. And its really arguably, but I think, I see your point. And taking on the situation with reiser4 with "smartmost VFS" approach, time should pass and with time and expirience on hands it will be evident what should go in VFS. Now its may be too early. Considering "amazing PR skills" of Hans Reiser it was the only viable way to get this changes in VFS. Cause, ironically, mr. Hellwig that currently demand it to be scrapped out or go in VFS would instakill Hans Reiser (i know many people would:) if he only touched holy cow of VFS for any reiser4 purpose. As I told before, some technically right things just do not work in this imperfect world. You have to count with social issues, and Hans Reiser wont, he just avoid them whenever its possible. > Name handling (dentry layer, mounting) is very much an integral part of > the filesystem. Almost everything else in the VFS is "helper functions", > ie a filesystem can choose to ignore buffer heads, page cache etc, but a > filesystem really cannot ignore or override the VFS naming stuff. > > (Arguably the page cache isn't part of the VFS layer at all, it's really a > memory management thing, although it's so intertwined with the VFS helper > functions that you can't really draw the line). > > > its just unified INTERFACE TO any filesystem (including reiser4). > > True, but that's not the whole truth. It's way more than just an > interface. It's a set of rules, and it's in many way the controlling > party. > > Linus "True, but thats not the whole truth." -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 19:03 ` Alexander Lyamin @ 2004-08-28 19:09 ` Christoph Hellwig 2004-08-28 21:41 ` Hans Reiser 2004-08-28 19:26 ` reiser4 plugins (was: silent semantic changes with reiser4) Linus Torvalds 1 sibling, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-28 19:09 UTC (permalink / raw) To: flx, Linus Torvalds, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, Aug 28, 2004 at 11:03:50PM +0400, Alexander Lyamin wrote: > And taking on the situation with reiser4 with "smartmost VFS" approach, > time should pass and with time and expirience on hands it will be evident what > should go in VFS. Now its may be too early. > > Considering "amazing PR skills" of Hans Reiser it was the only viable way > to get this changes in VFS. Cause, ironically, mr. Hellwig that currently > demand it to be scrapped out or go in VFS would instakill Hans Reiser > (i know many people would:) if he only touched holy cow of VFS for any > reiser4 purpose. Could you please stop talking such inproven crap? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 19:09 ` Christoph Hellwig @ 2004-08-28 21:41 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 21:41 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, reiserfs-list I think it is reasonable to make the -nopseudos (turns off the metafiles ) mount option mandatory, until the bugs are resolved. Our testing did not find these metafile/VFS bugs because of the reason for all our bugs, we screwed up. There is a distinct difference between some persons and I, which is that some think all of reiser4 should be excluded until metafiles are implemented by VFS some long time from now, and I, in that I merely think buggy optional features should be turned off until they are fixed. I, being renowned for my paranoia and asininity as I am, think these persons find it convenient as an excuse to keep us from competing, and I think that if we were slower there would be less hassle every time we try to get into the kernel. While reiser4 has some significant roughnesses remaining in its performance, I think the average user would find it performs better than other filesystems, and is stable enough for, say, a laptop, and I predict that by the time we have it stable enough for mission critical servers, all the roughness in various important corner cases will be gone. Persons benchmarking it with tarballs, please be sure to use tarballs created on reiser4, not ext2 tarballs, readdir order matters a lot for sorted directory filesystems. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins @ 2004-08-28 21:41 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 21:41 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, reiserfs-list I think it is reasonable to make the -nopseudos (turns off the metafiles ) mount option mandatory, until the bugs are resolved. Our testing did not find these metafile/VFS bugs because of the reason for all our bugs, we screwed up. There is a distinct difference between some persons and I, which is that some think all of reiser4 should be excluded until metafiles are implemented by VFS some long time from now, and I, in that I merely think buggy optional features should be turned off until they are fixed. I, being renowned for my paranoia and asininity as I am, think these persons find it convenient as an excuse to keep us from competing, and I think that if we were slower there would be less hassle every time we try to get into the kernel. While reiser4 has some significant roughnesses remaining in its performance, I think the average user would find it performs better than other filesystems, and is stable enough for, say, a laptop, and I predict that by the time we have it stable enough for mission critical servers, all the roughness in various important corner cases will be gone. Persons benchmarking it with tarballs, please be sure to use tarballs created on reiser4, not ext2 tarballs, readdir order matters a lot for sorted directory filesystems. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-28 21:41 ` Hans Reiser (?) @ 2004-08-30 16:02 ` Herbert Poetzl 2004-08-30 18:55 ` Hans Reiser -1 siblings, 1 reply; 1835+ messages in thread From: Herbert Poetzl @ 2004-08-30 16:02 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, flx, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, Aug 28, 2004 at 02:41:12PM -0700, Hans Reiser wrote: > I think it is reasonable to make the -nopseudos (turns off the metafiles > ) mount option mandatory, until the bugs are resolved. > > Our testing did not find these metafile/VFS bugs because of the reason > for all our bugs, we screwed up. > > There is a distinct difference between some persons and I, which is that > some think all of reiser4 should be excluded until metafiles are > implemented by VFS some long time from now, and I, in that I merely > think buggy optional features should be turned off until they are > fixed. I, being renowned for my paranoia and asininity as I am, think > these persons find it convenient as an excuse to keep us from competing, > and I think that if we were slower there would be less hassle every time > we try to get into the kernel. > > While reiser4 has some significant roughnesses remaining in its > performance, I think the average user would find it performs better than > other filesystems, and is stable enough for, say, a laptop, and I > predict that by the time we have it stable enough for mission critical > servers, all the roughness in various important corner cases will be > gone. > > Persons benchmarking it with tarballs, please be sure to use tarballs > created on reiser4, not ext2 tarballs, readdir order matters a lot for > sorted directory filesystems. hmm, so probably we have to wait until all tar packagers moved to reiser4, so that the available tar files are 'sorted properly' ... best, Herbert > Hans > - > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins 2004-08-30 16:02 ` Herbert Poetzl @ 2004-08-30 18:55 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-30 18:55 UTC (permalink / raw) To: Herbert Poetzl Cc: Linus Torvalds, Christoph Hellwig, flx, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, reiserfs-list Herbert Poetzl wrote: > >hmm, so probably we have to wait until all >tar packagers moved to reiser4, so that the >available tar files are 'sorted properly' ... > >best, >Herbert > > > Or wait for a repacker, which will probably happen sooner. Distros that use reiser4 will probably tar in reiser4 order. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins @ 2004-08-30 18:55 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-30 18:55 UTC (permalink / raw) To: Herbert Poetzl Cc: Linus Torvalds, Christoph Hellwig, flx, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, reiserfs-list Herbert Poetzl wrote: > >hmm, so probably we have to wait until all >tar packagers moved to reiser4, so that the >available tar files are 'sorted properly' ... > >best, >Herbert > > > Or wait for a repacker, which will probably happen sooner. Distros that use reiser4 will probably tar in reiser4 order. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 19:03 ` Alexander Lyamin 2004-08-28 19:09 ` Christoph Hellwig @ 2004-08-28 19:26 ` Linus Torvalds 1 sibling, 0 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-28 19:26 UTC (permalink / raw) To: Alexander Lyamin Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, 28 Aug 2004, Alexander Lyamin wrote: > > Considering "amazing PR skills" of Hans Reiser it was the only viable way > to get this changes in VFS. Cause, ironically, mr. Hellwig that currently > demand it to be scrapped out or go in VFS would instakill Hans Reiser > (i know many people would:) if he only touched holy cow of VFS for any > reiser4 purpose. Hey, you don't have to try to convince me - I haven't been arguing against the reiser4 approach. I think it's a prototyping scheme, and quite frankly I don't think you _can_ get the name lookup right without real VFS support (races, namespaces etc are just not something you can solve in the filesystem). But the fact that name lookup does seriously need VFS support means that I don't think we want to merge reiser4 as-is. No way. That doesn't mean that I think reiser4 was _wrong_. And I certainly agree that there are personality issues, and they are not all just Hans' ;) Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:07 ` Christoph Hellwig @ 2004-08-26 13:27 ` Christophe Saout 2004-08-26 13:27 ` Christophe Saout 2004-08-26 14:44 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 13:27 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1218 bytes --] Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig: > > The reiser4 core doesn't know about inodes, directories or files. It's > > the glue code between the VFS and the storage layer that does. It's > > implemented as a plugin. This has nothing to do with semantic > > enhancements yet. These should be removed for now and made a 2.7 topic. > > Oh yes, it is. As soon as you use different access methods on an > overlapping set of objects you see exactly the problems I described. Right. That shouldn't happen. Since an object is usally tied to exactly one plugin this can't happen I think. At least as long the programmer doesn't mess anything up. > If they don't overlap there's no point for the plugins to start with, > you'll better turn the core into a library that can be used by different > projects then. I think that's what Hans is trying to suggest. The core is theoretically usable for anything. He made a filesystem of it. You can see reiser4 as a library and the UNIX directory and file plugins as the filesystem. That's why they are linked in uncoditionally because it doesn't make sense otherwise. It's mainly the naming that seems to confuse people. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 13:27 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 13:27 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1218 bytes --] Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig: > > The reiser4 core doesn't know about inodes, directories or files. It's > > the glue code between the VFS and the storage layer that does. It's > > implemented as a plugin. This has nothing to do with semantic > > enhancements yet. These should be removed for now and made a 2.7 topic. > > Oh yes, it is. As soon as you use different access methods on an > overlapping set of objects you see exactly the problems I described. Right. That shouldn't happen. Since an object is usally tied to exactly one plugin this can't happen I think. At least as long the programmer doesn't mess anything up. > If they don't overlap there's no point for the plugins to start with, > you'll better turn the core into a library that can be used by different > projects then. I think that's what Hans is trying to suggest. The core is theoretically usable for anything. He made a filesystem of it. You can see reiser4 as a library and the UNIX directory and file plugins as the filesystem. That's why they are linked in uncoditionally because it doesn't make sense otherwise. It's mainly the naming that seems to confuse people. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:07 ` Christoph Hellwig 2004-08-26 13:17 ` Christophe Saout 2004-08-26 13:27 ` Christophe Saout @ 2004-08-26 14:44 ` Jamie Lokier 2004-08-26 16:03 ` Christoph Hellwig 2 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 14:44 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig and Christophe Saout, I think you two are miscommunicating a little. I've never looked at reiser4 code, but I think I understand what you are both saying, so I will explain. Christoph Hellwig wrote: > Here comes the problem. All the access checking/race avoidance/loop > creation avoiced, in short the posix+extension semantics are implemented > in the Linux VFS layer. If you want to allow a second access method > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity > inside reiser4. That would be good. Even better would be if VFS implemented the best semantics that we agree is nice, after some experimenting with reiser4 lets us discover what works well. > > The reiser4 core doesn't know about inodes, directories or files. It's > > the glue code between the VFS and the storage layer that does. It's > > implemented as a plugin. This has nothing to do with semantic > > enhancements yet. These should be removed for now and made a 2.7 topic. > > Oh yes, it is. As soon as you use different access methods on an > overlapping set of objects you see exactly the problems I described. > > If they don't overlap there's no point for the plugins to start with, > you'll better turn the core into a library that can be used by different > projects then. Firstly, the core library _is_ used by different projects. That's the commercial aim of reiser4: to be used by projects other than just as Linux filesystem. For example it might be used by a database, or a portable application which needs to store structured data in a flat file (with random access). A good abstraction makes it more useful than "just" a filesystem. Secondly, the reiser4 design makes the "high-level plugin" interface a good clean abstraction. It's conceptually simple and versatile, so simplifies the code and makes it easier to try new ideas as well. I think the disagreement between you two comes down to the idea that a generally useful API abstraction to file-like objects really should be somewhere in the VFS, and not specific to a single filesystem. Unfortunately, the problem is that reiser4 is the only filesystem which is _technically capable_ of implementing that abstraction in a practical way, apparently. (I'm not sure if this is really true. reiser4's object model is not the same as paths and inodes, but the impedance mismatch doesn't seem huge.) Perhaps in future some other filesystems could do it. Then it would make sense for the API to be common between them. At the same time, some of the features which Hans Reiser would like to offer users -- especially things like metadata indexing -- don't seem to be very practical on top of the VFS abstraction. These are conflicting goals. Hence the argument. VFS is very POSIX, which rules out some interesting things like multiple views of an ordinary file (i.e. viewing a flat file and it's structured form together), coherent files (i.e. metadata which is calculated from the file contents and always guaranteed to match it), and real time indexing (i.e. "grep", "locate" and "local-google" but fancier, always up to date, and still fast). Those features can be implemented poorly on top of POSIX, and hence poorly on top of VFS. We do that today. It works, it's simple, but it sucks. Sometimes it goes wrong (index doesn't match files), it's slow (locate isn't real-time, and if grep kept content indexes it would still be too slow to verify them), and it's not really simple as you have to use special tools for every file format these days, each with their own unique interface. I think we can tidy that up a lot, still using the VFS, but with some extra hooks. But it's not clear what hooks those would be, and how the interesting features would be implemented by other POSIX-only filesystems. It'd be more useful: I want the fancy stuff to work over NFS to a bog standard server, or over an old ext2 filesystem that mustn't be upgraded, not just reiser4, even if I don't have the same performance and it has to create lots of articial "metadata" files to simulate the fancy stuff. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:44 ` Jamie Lokier @ 2004-08-26 16:03 ` Christoph Hellwig 2004-08-26 16:19 ` Jamie Lokier 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:03 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > Firstly, the core library _is_ used by different projects. That's the > commercial aim of reiser4: to be used by projects other than just as > Linux filesystem. For example it might be used by a database, or a > portable application which needs to store structured data in a flat > file (with random access). A good abstraction makes it more useful > than "just" a filesystem. Which is okay as long as it still fits in the kernel. Remember that lots of projects had to adjust so far despite the commerical aims of their creators. > I think the disagreement between you two comes down to the idea that a > generally useful API abstraction to file-like objects really should be > somewhere in the VFS, and not specific to a single filesystem. IT should, and in fact it actually is most of the part. We can have differnt operation vectors per object, but reiser4 is the only filesystem in the tree not making use of that but provinding only a single one and then dispatching internally. Even XFS that carries around and horrible internal dispatching layer as IRIX legacy gets this right. > Unfortunately, the problem is that reiser4 is the only filesystem > which is _technically capable_ of implementing that abstraction in a > practical way, apparently. (I'm not sure if this is really true. > reiser4's object model is not the same as paths and inodes, but the > impedance mismatch doesn't seem huge.) Umm, no. In fact every filesystem does this. There's not too many objects with different namespace semantic: regular files and special files vs directories vs symlinks basically, but as soon as you go to the data patch you can have hundrets of different file_operations on a single filesystems (for special files). ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:03 ` Christoph Hellwig @ 2004-08-26 16:19 ` Jamie Lokier 2004-08-26 16:23 ` Christoph Hellwig 0 siblings, 1 reply; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 16:19 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: > > Unfortunately, the problem is that reiser4 is the only filesystem > > which is _technically capable_ of implementing that abstraction in a > > practical way, apparently. (I'm not sure if this is really true. > > reiser4's object model is not the same as paths and inodes, but the > > impedance mismatch doesn't seem huge.) > > Umm, no. In fact every filesystem does this. There's not too many > objects with different namespace semantic: regular files and special > files vs directories vs symlinks basically, but as soon as you go to the > data patch you can have hundrets of different file_operations on a > single filesystems (for special files). Remember that reiser4 allows you to operate on little pieces of data, glueing and rearranging them inside files (something like that). No other filesystem has that capability, and it's a data model which the fancy features (when they exist) will use. You can map those pieces to underlying directories and files and renames and unlinks, so that the fancy stuff works on other filesystems, but it would be a useless model because those other filesystems wouldn't be recognisably "ordinary" files any more. For reiser4 to expose that model through a VFS interface, and the fancy stuff to use it through the VFS interface, and for the fancy stuff to work (even imperfectly) on other filesystems which don't offer those operations, some kind of fall-back "store metadata and fragment rearrangements in auxiliary files with special names" layer would be requied. That's a big job. I think it's a good job to do (funding, anyone? :), and the right place for that (imperfect but useful) fall-back layer is userspace with perhaps minimal VFS support. Another reason to put the fall-back layer in userspace is that applications which depend on the fancy stuff can still be portable to other OSes, working, but a bit smoother, faster and more "integrated" on reiser4. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:19 ` Jamie Lokier @ 2004-08-26 16:23 ` Christoph Hellwig 2004-08-26 17:08 ` Jamie Lokier 0 siblings, 1 reply; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:23 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 05:19:11PM +0100, Jamie Lokier wrote: > Remember that reiser4 allows you to operate on little pieces of data, > glueing and rearranging them inside files (something like that). > > No other filesystem has that capability, and it's a data model which > the fancy features (when they exist) will use. > > You can map those pieces to underlying directories and files and > renames and unlinks, so that the fancy stuff works on other > filesystems, but it would be a useless model because those other > filesystems wouldn't be recognisably "ordinary" files any more. > > For reiser4 to expose that model through a VFS interface, and the > fancy stuff to use it through the VFS interface, and for the fancy > stuff to work (even imperfectly) on other filesystems which don't > offer those operations, some kind of fall-back "store metadata and > fragment rearrangements in auxiliary files with special names" layer > would be requied. That's a big job. I seem to miss those files in the current implementation. But again the big question is how will the interface look? If the smaller objects interact with normal files (link/rename, contained in normal directories), they will have to obey the same locking protocols. If not a filesystem could experiment with it of course. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:23 ` Christoph Hellwig @ 2004-08-26 17:08 ` Jamie Lokier 0 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 17:08 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: > If the smaller objects interact with normal files (link/rename, > contained in normal directories), they will have to obey the same > locking protocols. If not a filesystem could experiment with it of > course. I think the smaller objects will consist of special files, which are magic attributes like the ones you currently see in reiser4, and ordinary files (which might themselves be containers). Link/rename/locking etc. should work as usual on the ordinary smaller files, but probably aren't allowed at all on the special attribute files, except where there's an appropriate meaning. (For example locking certain attributes might have an appropriate meaning). It would make sense for the special attributes file to not identify as "regular files" (S_IFREG), but there aren't many mode bits to play with. They should also be harder to find than ordinary smaller files. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:18 ` Christophe Saout @ 2004-08-26 23:54 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:54 UTC (permalink / raw) To: Christophe Saout Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton: > > > >>And describe the "plugin" system. Why does the filesystem need such a >>thing (other filesystems get their features via `patch -p1')? >> >> > >Ok, let me try. (Hans, please don't shoot me if I don't get every detail >right, I'm trying to simplify and translate) > >The reiser4 core is just a big and fast storage layer using a single >tree. It is bound to a mount point and uses the page cache to manage its >memory. The only thing it can do on its own is to flush dirty data to >the disk when the VM wants to (memory pressure) or the VFS wants to >(unmount, sync) or some "plugin" wants to. >Additionally it's completely atomic (with respect to crashes, no >isolation like in databases), comparable to data=journal but without the >overhead of using a journal for everything. > >Up to this point there are no users of this "database" and it does not >implement any of the VFS methods for a filesystem (except mount and >unmount perhaps). > >All the functionality is provided to > ^to^via apologies for the correction but it might confuse some > the plugins. You can insert, >remove, lookup or modify key:object pairs (with an index into the >object). The object > ^object^item perform that substitution everywhere in this email > is a sequence of units. For files a unit would be 1 >byte and the index would be the byte offset. For directories the unit >would be 1 entry and the index would be the filename. > >Now there are some plugins that define how the storage layout on the >disk is (some kind of "backend" plugins). > >*And* there are plugins which are users of the "reiser4 client API" and >implement the actual VFS methods. > >There's a UNIX directory plugin and a UNIX file plugin. > >Directories, inodes and file content are just key:object pairs and the >plugins know how to operate on these. > >There's a new plugin in work that also implements UNIX file semantics >but stores the data for that file encrypted and/or compressed. > >These plugins live between the VFS and the storage layer. > >Just like the filesystems live between the VFS and the block layer (Hans >would say that filesystems are VFS plugins ;-)). > > > >>And what are the licensing implications of plugins? Are they derived >>works? Must they be GPL'ed? >> >> > >I suppose yes, at least currently, since they can only be linked with >reiser4, there's no module infrastructure. > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 23:54 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-26 23:54 UTC (permalink / raw) To: Christophe Saout Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton: > > > >>And describe the "plugin" system. Why does the filesystem need such a >>thing (other filesystems get their features via `patch -p1')? >> >> > >Ok, let me try. (Hans, please don't shoot me if I don't get every detail >right, I'm trying to simplify and translate) > >The reiser4 core is just a big and fast storage layer using a single >tree. It is bound to a mount point and uses the page cache to manage its >memory. The only thing it can do on its own is to flush dirty data to >the disk when the VM wants to (memory pressure) or the VFS wants to >(unmount, sync) or some "plugin" wants to. >Additionally it's completely atomic (with respect to crashes, no >isolation like in databases), comparable to data=journal but without the >overhead of using a journal for everything. > >Up to this point there are no users of this "database" and it does not >implement any of the VFS methods for a filesystem (except mount and >unmount perhaps). > >All the functionality is provided to > ^to^via apologies for the correction but it might confuse some > the plugins. You can insert, >remove, lookup or modify key:object pairs (with an index into the >object). The object > ^object^item perform that substitution everywhere in this email > is a sequence of units. For files a unit would be 1 >byte and the index would be the byte offset. For directories the unit >would be 1 entry and the index would be the filename. > >Now there are some plugins that define how the storage layout on the >disk is (some kind of "backend" plugins). > >*And* there are plugins which are users of the "reiser4 client API" and >implement the actual VFS methods. > >There's a UNIX directory plugin and a UNIX file plugin. > >Directories, inodes and file content are just key:object pairs and the >plugins know how to operate on these. > >There's a new plugin in work that also implements UNIX file semantics >but stores the data for that file encrypted and/or compressed. > >These plugins live between the VFS and the storage layer. > >Just like the filesystems live between the VFS and the block layer (Hans >would say that filesystems are VFS plugins ;-)). > > > >>And what are the licensing implications of plugins? Are they derived >>works? Must they be GPL'ed? >> >> > >I suppose yes, at least currently, since they can only be linked with >reiser4, there's no module infrastructure. > > > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser 2004-08-26 8:45 ` Andrew Morton @ 2004-08-26 9:34 ` Christoph Hellwig 2004-08-26 9:37 ` Markus Törnqvist 2004-08-26 18:40 ` Thomas Cataldo 2004-08-26 12:30 ` Redeeman ` (4 subsequent siblings) 6 siblings, 2 replies; 1835+ messages in thread From: Christoph Hellwig @ 2004-08-26 9:34 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 01:31:34AM -0700, Hans Reiser wrote: > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > for Apple, and that means we need to put search engine and database Dou you know a nice thing? We (as in the Linux Community) don't have to compete with anyone. Sure, we're usually trying to be better than anyone else, but unlike companies under maret pressure we can wait until something is ready. Again, if you think you want your work in Linux cooperate with the Linux Developers, and as you have seen in this thread there's a pretty broad consensus that we want things working at the VFS (and actually working of course) ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:34 ` Christoph Hellwig @ 2004-08-26 9:37 ` Markus Törnqvist 2004-08-26 18:40 ` Thomas Cataldo 1 sibling, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-26 9:37 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 11:34:07AM +0200, Christoph Hellwig wrote: > >Dou you know a nice thing? We (as in the Linux Community) don't have to >compete with anyone. Sure, we're usually trying to be better than >anyone else, but unlike companies under maret pressure we can wait until >something is ready. It won't get ready unless it's made ready, yes? And what about the time when people will move away from Linux because Linux has nothing but crappy file systems to offer? Sure, it's not that we'd have to compete, but... >Again, if you think you want your work in Linux cooperate with the Linux >Developers, and as you have seen in this thread there's a pretty broad >consensus that we want things working at the VFS (and actually working >of course) So move it into VFS. Neext! ;) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:34 ` Christoph Hellwig 2004-08-26 9:37 ` Markus Törnqvist @ 2004-08-26 18:40 ` Thomas Cataldo 1 sibling, 0 replies; 1835+ messages in thread From: Thomas Cataldo @ 2004-08-26 18:40 UTC (permalink / raw) To: Christoph Hellwig Cc: Hans Reiser, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 11:34 +0200, Christoph Hellwig wrote: > On Thu, Aug 26, 2004 at 01:31:34AM -0700, Hans Reiser wrote: > > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > > for Apple, and that means we need to put search engine and database > > Dou you know a nice thing? We (as in the Linux Community) don't have to > compete with anyone. Sure, we're usually trying to be better than > anyone else, but unlike companies under maret pressure we can wait until > something is ready. By the way, this kind of features can be achieved easily in userspace with, for example, ext3, beagle (a userspace fulltext indexing daemon) and some (d|i)notify magic. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser 2004-08-26 8:45 ` Andrew Morton 2004-08-26 9:34 ` Christoph Hellwig @ 2004-08-26 12:30 ` Redeeman 2004-08-26 16:32 ` Will Dyson ` (3 subsequent siblings) 6 siblings, 0 replies; 1835+ messages in thread From: Redeeman @ 2004-08-26 12:30 UTC (permalink / raw) To: Reiserfs Mailinglist On Thu, 2004-08-26 at 01:31 -0700, Hans Reiser wrote: > Andrew Morton wrote: > > >Hans Reiser <reiser@namesys.com> wrote: > > > > > >>I had not intended to respond to this because I have nothing positive to > >>say, but Andrew said I needed to respond and suggested I should copy > >>Linus. > >> > >> > > > >Yes, but I didn't say "flame Christoph and ignore the issues" ;) > > > > > Oh....;-) > > >There are lots of little things to do with implementation, coding style, > >module exports, deadlocks, what code goes where, etc. These are all normal > >daily kernel business and we should set them aside for now and concentrate > >on the bigger issues. > > > > > Yes, you are right, but I am not sure Viro will go along with that..... ;-) > > >And as I see it, there are two big issues: > > > >a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not > > anticipate and > > > >b) it does this within the context of just a single filesystem. > > > >I see three possible responses: > > > >a) accept the reiser4-only extensions as-is (possibly with post-review > > modifications, of course) or > > > >b) accept the reiser4-only extensions with a view to turning them into > > kernel-wide extensions at some time in the future, so all filesystems > > will offer the extensions (as much as poss) or > > > >c) reject the extensions. > > > > > >My own order of preference is b) c) a). > > > I don't object to b), though I think b) should wait for 2.7 and reiser4 > should not. > > > The fact that one filesystem will > >offer features which other filesystems do not and cannot offer makes me > >queasy for some reason. > > > > > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > for Apple, and that means we need to put search engine and database > functionality into the filesystem. It takes 11 years of serious > research to build a clean storage layer able to handle doing that. > Reiser4 has done that, finally. None of the other Linux filesystems > have. The next major release of ReiserFS is going to be bursting with > semantic enhancements, because the prerequisites for them are in place > now. None of the other Linux filesystems have those prerequisites. > They won't be able to keep up with the semantic enhancements. This > metafiles and file-directories stuff is actually fairly trivial stuff. > > Look guys, in 1993 I anticipated the battle would be here, and I build > the foundation for a defensive tower right at the spot MS and Apple are > now maneuvering towards. Help me get the next level on the tower before > they get here. It is one hell of a foundation, they won't be able to > shake it, their trees are not as powerful. Don't move reiser4 into vfs, > use reiser4 as the vfs. Don't write filesystems, write file plugins and > disk format plugins and all the other kinds of plugins, and you won't be > missing any expressive power that you really want.... > couldnt be more true > Give Saveliev and I some credit. 10 years of hard work at an ivory > tower nobody thought mattered. Now the battle leaves the browser and > swings our way. Don't duplicate that infrastructure, use it. > > There is so much we could use help with if talented people like you > chose to contribute. > > >b) means that at some time in the future we need to hoist the reiser4 > >extensions (at a conceptual level) (and probably with restrictions) up into > >the VFS. This will involve much thought, argument and work. > > > > > >To get us started on this route it would really help me (and, probably, > >others) if you could describe what these API extensions are in a very > >simple way, without referring to incomprehehsible web pages, > > > what is not comprehensible....? > > > and without > >using terms which non-reiser4 people don't understand. > > > > > Well, I agree that there is value in defining things in more detail than > we have. > > >It would be best if each extension was addressed in a separate email > >thread. > > > >We also need to discuss what a reiser4 "module" is, what its capabilities > >are, and what licensing implications they have. > > > >Then, we can look at each one and say "yup, that makes sense - we want > >Linux to do that" and we can also think about how we would implement it at > >the VFS level. > > > >If we follow the above route I believe we can make progress in a technical > >direction and not get deadlocked on personal/political stuff. > > > > > >Now, an alternative to all the above is to just merge reiser4 as-is, after > >addressing all the lower-level coding issues. And see what happens. That > >may be a thing which Linus wishes to do. I'm easy. > > > > > > > > > > -- Redeeman <redeeman@metanurb.dk> ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser ` (2 preceding siblings ...) 2004-08-26 12:30 ` Redeeman @ 2004-08-26 16:32 ` Will Dyson 2004-08-26 16:44 ` Christophe Saout 2004-08-27 8:08 ` Hans Reiser 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand ` (2 subsequent siblings) 6 siblings, 2 replies; 1835+ messages in thread From: Will Dyson @ 2004-08-26 16:32 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser wrote: > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > for Apple, and that means we need to put search engine and database > functionality into the filesystem. It takes 11 years of serious Hans, I'm very curious about your ideas on how to put search engine and database functionality into the filesystem. Although I've let the Befs filesystem driver largely stagnate over the past two years, from time to time I think about the problem of exporting to userspace the attribute indexes that BeFS keeps on disk. In the original BeOS, they solved the problem by having the filesystem driver itself take a text query string and parse it, returning a list of inodes that match. The whole business of parsing a query string in the kernel (let alone in the filesystem driver) has always seemed ugly to me. However, the best alternative I've come up with is to simply export the index data as a special file (perhaps in sysfs?) and have userspace responsible for searching the index. That would probably work, but it wouldn't help other filesystems that implement even a different index format, much less a different form of extra searchability. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:32 ` Will Dyson @ 2004-08-26 16:44 ` Christophe Saout 2004-08-27 8:08 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 16:44 UTC (permalink / raw) To: Will Dyson Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1007 bytes --] Am Donnerstag, den 26.08.2004, 12:32 -0400 schrieb Will Dyson: > > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > > for Apple, and that means we need to put search engine and database > > functionality into the filesystem. It takes 11 years of serious > > I'm very curious about your ideas on how to put search engine and > database functionality into the filesystem. I could imagine something like this. There are some special folders or files or whatever under every file where some userspace can put some metadata. A keyword perhaps. echo linux > file.bla/keywords/topic The filesystem might then automatically put these keywords into an index and then provide a search mechanism elsewhere where it could ask "find me all dentries with the keyword 'linux'." and it would return a list like locate does. Only that it's in realtime and also works when moving the file around (but not when copying with an unaware program for obvious reasons). [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 16:44 ` Christophe Saout 0 siblings, 0 replies; 1835+ messages in thread From: Christophe Saout @ 2004-08-26 16:44 UTC (permalink / raw) To: Will Dyson Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1007 bytes --] Am Donnerstag, den 26.08.2004, 12:32 -0400 schrieb Will Dyson: > > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > > for Apple, and that means we need to put search engine and database > > functionality into the filesystem. It takes 11 years of serious > > I'm very curious about your ideas on how to put search engine and > database functionality into the filesystem. I could imagine something like this. There are some special folders or files or whatever under every file where some userspace can put some metadata. A keyword perhaps. echo linux > file.bla/keywords/topic The filesystem might then automatically put these keywords into an index and then provide a search mechanism elsewhere where it could ask "find me all dentries with the keyword 'linux'." and it would return a list like locate does. Only that it's in realtime and also works when moving the file around (but not when copying with an unaware program for obvious reasons). [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:44 ` Christophe Saout (?) @ 2004-08-26 17:22 ` Jamie Lokier -1 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-08-26 17:22 UTC (permalink / raw) To: Christophe Saout Cc: Will Dyson, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: > echo linux > file.bla/keywords/topic > > The filesystem might then automatically put these keywords into an index > and then provide a search mechanism elsewhere where it could ask "find > me all dentries with the keyword 'linux'." and it would return a list > like locate does. Only that it's in realtime and also works when moving > the file around (but not when copying with an unaware program for > obvious reasons). It could work even when copying with an unaware program - provided the're a tool which knows to extract metadata from files where that hasn't been done already, or where the file's been modified. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:32 ` Will Dyson @ 2004-08-27 8:08 ` Hans Reiser 2004-08-27 8:08 ` Hans Reiser 1 sibling, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:08 UTC (permalink / raw) To: Will Dyson Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Will Dyson wrote: > > > In the original BeOS, they solved the problem by having the filesystem > driver itself take a text query string and parse it, returning a list > of inodes that match. The whole business of parsing a query string in > the kernel (let alone in the filesystem driver) has always seemed ugly > to me. Why? > However, the best alternative I've come up with is to simply export > the index data as a special file (perhaps in sysfs?) and have > userspace responsible for searching the index. That is the best implementation suggestion I've heard for splitting the filesystem into two parts, one in user space and one in kernel, but I still don't trust it to work well. > That would probably work, but it wouldn't help other filesystems that > implement even a different index format, much less a different form of > extra searchability. > After reiser6 is implemented, we'll have a better idea of what parts need what from what other parts. Until we get there, it needs to be kept together. Frankly, I suspect that naming is not going to split so easily, but when it is done we'll be able to know. The problems of putting too much functionality into the kernel are less than those of partitioning the filesystem code, and defining an inflexible API between the two parts. Kernel interfaces are fairly inflexible because there will always be some moron who sidesteps the fs library to program directly to the kernel interface, gets screwed when the interface changes, has the nerve to complain about it, and works for some powerful distro. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 8:08 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-27 8:08 UTC (permalink / raw) To: Will Dyson Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Will Dyson wrote: > > > In the original BeOS, they solved the problem by having the filesystem > driver itself take a text query string and parse it, returning a list > of inodes that match. The whole business of parsing a query string in > the kernel (let alone in the filesystem driver) has always seemed ugly > to me. Why? > However, the best alternative I've come up with is to simply export > the index data as a special file (perhaps in sysfs?) and have > userspace responsible for searching the index. That is the best implementation suggestion I've heard for splitting the filesystem into two parts, one in user space and one in kernel, but I still don't trust it to work well. > That would probably work, but it wouldn't help other filesystems that > implement even a different index format, much less a different form of > extra searchability. > After reiser6 is implemented, we'll have a better idea of what parts need what from what other parts. Until we get there, it needs to be kept together. Frankly, I suspect that naming is not going to split so easily, but when it is done we'll be able to know. The problems of putting too much functionality into the kernel are less than those of partitioning the filesystem code, and defining an inflexible API between the two parts. Kernel interfaces are fairly inflexible because there will always be some moron who sidesteps the fs library to program directly to the kernel interface, gets screwed when the interface changes, has the nerve to complain about it, and works for some powerful distro. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:08 ` Hans Reiser (?) @ 2004-08-27 18:20 ` Will Dyson 2004-08-27 18:38 ` David Ford ` (2 more replies) -1 siblings, 3 replies; 1835+ messages in thread From: Will Dyson @ 2004-08-27 18:20 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser wrote: > Will Dyson wrote: >> >> In the original BeOS, they solved the problem by having the filesystem >> driver itself take a text query string and parse it, returning a list >> of inodes that match. The whole business of parsing a query string in >> the kernel (let alone in the filesystem driver) has always seemed ugly >> to me. > > Why? Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. String parsing bloats the kernel with code that will be called rarely, and doing it inside the filesystem module makes for duplicate code if more than one filesystem does it. But a good common parser routine (or a kernel api that takes a pre-compiled parse tree) would reduce the bad taste. The real objection I have is that it puts the kernel in the position of doing more work than it has to for the system to operate correctly and efficiently. The user wants to know what files match a certain set of criteria. The filesystem provides special features which can greatly accelerate some searches. Does it make more sense to move the functionality of /usr/bin/find into the kernel, or to export the index information so that an enhanced version of find can make use of it? I don't think having find in the kernel would be worth the cost or complexity. However, now that I've had to think about it while writing the above, it occurs to me that if we wanted to offer something like the beos's live queries, doing it in the kernel might be the way to go. For the 99.9% of people who were not beos users back in the day, the "live query" was a way to register a query string with the kernel, so that you got notifications when the list of files matching the query changed. It was pretty cool, but this is really just blue-skying since I am not stepping up to write such a beast any time soon. >> However, the best alternative I've come up with is to simply export >> the index data as a special file (perhaps in sysfs?) and have >> userspace responsible for searching the index. > > That is the best implementation suggestion I've heard for splitting the > filesystem into two parts, one in user space and one in kernel, but I > still don't trust it to work well. Why not? And if it really splits the filesystem into two parts depends on which of the two following statements you agree with more: "Fast searching is a feature of the filesystem" "Maintaining indexes is a feature of the filesystem" PS. The file-as-directory attribute/stream/whatever model is a much better fit for befs's internals than the xattr api. If the kernel supported it, I would enable support for it in befs. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:20 ` Will Dyson @ 2004-08-27 18:38 ` David Ford 2004-08-28 9:54 ` Hans Reiser 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2 siblings, 0 replies; 1835+ messages in thread From: David Ford @ 2004-08-27 18:38 UTC (permalink / raw) To: Will Dyson Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2800 bytes --] Will Dyson wrote: > Hans Reiser wrote: > >> Will Dyson wrote: >> >>> >>> In the original BeOS, they solved the problem by having the >>> filesystem driver itself take a text query string and parse it, >>> returning a list of inodes that match. The whole business of parsing >>> a query string in the kernel (let alone in the filesystem driver) >>> has always seemed ugly to me. >> >> >> Why? > > > Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. > > String parsing bloats the kernel with code that will be called rarely, > and doing it inside the filesystem module makes for duplicate code if > more than one filesystem does it. But a good common parser routine (or > a kernel api that takes a pre-compiled parse tree) would reduce the > bad taste. Like say.. printk() ? :) > The real objection I have is that it puts the kernel in the position > of doing more work than it has to for the system to operate correctly > and efficiently. The user wants to know what files match a certain set > of criteria. The filesystem provides special features which can > greatly accelerate some searches. Does it make more sense to move the > functionality of /usr/bin/find into the kernel, or to export the index > information so that an enhanced version of find can make use of it? I think it depends largely on where you see file systems going 5, 10, 20 years from now. Are they going to be decadent designs or will they be flush with meta data and other tidbits? Where does the heuristic line sit when it becomes better to put such a function into the kernel v.s. user land? > I don't think having find in the kernel would be worth the cost or > complexity. Probably not. But as I view it, file systems are slowly evolving into relational objects and the margin between what are files and directories versus content within said objects is shrinking. More and more, people are using a relational database to store content that once resided in singular files. Sure, it makes for a much fatter *FS system, but the user doesn't care too much, he simply enjoys the ability to manage, search, and cross reference his content much more easily. And then comes the "it's a user land thing" comment again. And again the response. When does moving this from user land into kernel land become a better choice. With the file systems we have today, it's really not much of a gain compared to the cost of the move. Consider Reiser4 now. More and more meta data is being used in regards to files. That margin is shrinking. BeOS broached it, Microsoft is building it. Just because they did it doesn't mean we have to do it nor that it is right. But it's food for thought. Especially as more and more tidbits get associated with files. -david [-- Attachment #2: david+challenge-response.vcf --] [-- Type: text/x-vcard, Size: 183 bytes --] begin:vcard fn:David Ford n:Ford;David email;internet:david@blue-labs.org title:Industrial Geek tel;home:Ask please tel;cell:(203) 650-3611 x-mozilla-html:TRUE version:2.1 end:vcard ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 18:38 ` David Ford 0 siblings, 0 replies; 1835+ messages in thread From: David Ford @ 2004-08-27 18:38 UTC (permalink / raw) To: Will Dyson Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2800 bytes --] Will Dyson wrote: > Hans Reiser wrote: > >> Will Dyson wrote: >> >>> >>> In the original BeOS, they solved the problem by having the >>> filesystem driver itself take a text query string and parse it, >>> returning a list of inodes that match. The whole business of parsing >>> a query string in the kernel (let alone in the filesystem driver) >>> has always seemed ugly to me. >> >> >> Why? > > > Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. > > String parsing bloats the kernel with code that will be called rarely, > and doing it inside the filesystem module makes for duplicate code if > more than one filesystem does it. But a good common parser routine (or > a kernel api that takes a pre-compiled parse tree) would reduce the > bad taste. Like say.. printk() ? :) > The real objection I have is that it puts the kernel in the position > of doing more work than it has to for the system to operate correctly > and efficiently. The user wants to know what files match a certain set > of criteria. The filesystem provides special features which can > greatly accelerate some searches. Does it make more sense to move the > functionality of /usr/bin/find into the kernel, or to export the index > information so that an enhanced version of find can make use of it? I think it depends largely on where you see file systems going 5, 10, 20 years from now. Are they going to be decadent designs or will they be flush with meta data and other tidbits? Where does the heuristic line sit when it becomes better to put such a function into the kernel v.s. user land? > I don't think having find in the kernel would be worth the cost or > complexity. Probably not. But as I view it, file systems are slowly evolving into relational objects and the margin between what are files and directories versus content within said objects is shrinking. More and more, people are using a relational database to store content that once resided in singular files. Sure, it makes for a much fatter *FS system, but the user doesn't care too much, he simply enjoys the ability to manage, search, and cross reference his content much more easily. And then comes the "it's a user land thing" comment again. And again the response. When does moving this from user land into kernel land become a better choice. With the file systems we have today, it's really not much of a gain compared to the cost of the move. Consider Reiser4 now. More and more meta data is being used in regards to files. That margin is shrinking. BeOS broached it, Microsoft is building it. Just because they did it doesn't mean we have to do it nor that it is right. But it's food for thought. Especially as more and more tidbits get associated with files. -david ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:38 ` David Ford (?) @ 2004-08-28 6:14 ` Hans Reiser -1 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 6:14 UTC (permalink / raw) To: David Ford Cc: Will Dyson, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list David Ford wrote: > Will Dyson wrote: > >> Hans Reiser wrote: >> >>> Will Dyson wrote: >>> >> >> >> String parsing bloats the kernel with code that will be called >> rarely, and doing it inside the filesystem module makes for duplicate >> code if more than one filesystem does it. But a good common parser >> routine (or a kernel api that takes a pre-compiled parse tree) would >> reduce the bad taste. > > > > Like say.. printk() ? :) > or namei()....;-) Seriously, what we are doing is making namei() more powerful. That is all. Currently namei() is very simple. Evolution happens. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:20 ` Will Dyson @ 2004-08-28 9:54 ` Hans Reiser 2004-08-28 9:54 ` Hans Reiser 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 9:54 UTC (permalink / raw) To: Will Dyson Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list I think there are two ways to analyze the code boundary issue. One is "does it belong in the kernel?" Another is, "does it belong in the filesystem. and if so should name resolution in a filesystem be split into two parts, one in kernel, and one in user space." In ten years I might have the knowledge needed to make such a split, but I know for sure that I don't know how to do it today without regretting it tomorrow, and I don't really have confidence that I will ever be able to do it without losing performance. Glad that BeFS finds the new model better.:) Hans Will Dyson wrote: > Hans Reiser wrote: > >> Will Dyson wrote: >> >>> >>> In the original BeOS, they solved the problem by having the >>> filesystem driver itself take a text query string and parse it, >>> returning a list of inodes that match. The whole business of parsing >>> a query string in the kernel (let alone in the filesystem driver) >>> has always seemed ugly to me. >> >> >> Why? > > > Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. > > String parsing bloats the kernel with code that will be called rarely, > and doing it inside the filesystem module makes for duplicate code if > more than one filesystem does it. But a good common parser routine (or > a kernel api that takes a pre-compiled parse tree) would reduce the > bad taste. > > The real objection I have is that it puts the kernel in the position > of doing more work than it has to for the system to operate correctly > and efficiently. The user wants to know what files match a certain set > of criteria. The filesystem provides special features which can > greatly accelerate some searches. Does it make more sense to move the > functionality of /usr/bin/find into the kernel, or to export the index > information so that an enhanced version of find can make use of it? > > I don't think having find in the kernel would be worth the cost or > complexity. > > However, now that I've had to think about it while writing the above, > it occurs to me that if we wanted to offer something like the beos's > live queries, doing it in the kernel might be the way to go. > > For the 99.9% of people who were not beos users back in the day, the > "live query" was a way to register a query string with the kernel, so > that you got notifications when the list of files matching the query > changed. It was pretty cool, but this is really just blue-skying since > I am not stepping up to write such a beast any time soon. > >>> However, the best alternative I've come up with is to simply export >>> the index data as a special file (perhaps in sysfs?) and have >>> userspace responsible for searching the index. >> >> >> That is the best implementation suggestion I've heard for splitting >> the filesystem into two parts, one in user space and one in kernel, >> but I still don't trust it to work well. > > > Why not? And if it really splits the filesystem into two parts depends > on which of the two following statements you agree with more: > > "Fast searching is a feature of the filesystem" > "Maintaining indexes is a feature of the filesystem" > > > PS. > The file-as-directory attribute/stream/whatever model is a much better > fit for befs's internals than the xattr api. If the kernel supported > it, I would enable support for it in befs. > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-28 9:54 ` Hans Reiser 0 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 9:54 UTC (permalink / raw) To: Will Dyson Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list I think there are two ways to analyze the code boundary issue. One is "does it belong in the kernel?" Another is, "does it belong in the filesystem. and if so should name resolution in a filesystem be split into two parts, one in kernel, and one in user space." In ten years I might have the knowledge needed to make such a split, but I know for sure that I don't know how to do it today without regretting it tomorrow, and I don't really have confidence that I will ever be able to do it without losing performance. Glad that BeFS finds the new model better.:) Hans Will Dyson wrote: > Hans Reiser wrote: > >> Will Dyson wrote: >> >>> >>> In the original BeOS, they solved the problem by having the >>> filesystem driver itself take a text query string and parse it, >>> returning a list of inodes that match. The whole business of parsing >>> a query string in the kernel (let alone in the filesystem driver) >>> has always seemed ugly to me. >> >> >> Why? > > > Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. > > String parsing bloats the kernel with code that will be called rarely, > and doing it inside the filesystem module makes for duplicate code if > more than one filesystem does it. But a good common parser routine (or > a kernel api that takes a pre-compiled parse tree) would reduce the > bad taste. > > The real objection I have is that it puts the kernel in the position > of doing more work than it has to for the system to operate correctly > and efficiently. The user wants to know what files match a certain set > of criteria. The filesystem provides special features which can > greatly accelerate some searches. Does it make more sense to move the > functionality of /usr/bin/find into the kernel, or to export the index > information so that an enhanced version of find can make use of it? > > I don't think having find in the kernel would be worth the cost or > complexity. > > However, now that I've had to think about it while writing the above, > it occurs to me that if we wanted to offer something like the beos's > live queries, doing it in the kernel might be the way to go. > > For the 99.9% of people who were not beos users back in the day, the > "live query" was a way to register a query string with the kernel, so > that you got notifications when the list of files matching the query > changed. It was pretty cool, but this is really just blue-skying since > I am not stepping up to write such a beast any time soon. > >>> However, the best alternative I've come up with is to simply export >>> the index data as a special file (perhaps in sysfs?) and have >>> userspace responsible for searching the index. >> >> >> That is the best implementation suggestion I've heard for splitting >> the filesystem into two parts, one in user space and one in kernel, >> but I still don't trust it to work well. > > > Why not? And if it really splits the filesystem into two parts depends > on which of the two following statements you agree with more: > > "Fast searching is a feature of the filesystem" > "Maintaining indexes is a feature of the filesystem" > > > PS. > The file-as-directory attribute/stream/whatever model is a much better > fit for befs's internals than the xattr api. If the kernel supported > it, I would enable support for it in befs. > ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 9:54 ` Hans Reiser (?) @ 2004-08-31 3:14 ` Will Dyson -1 siblings, 0 replies; 1835+ messages in thread From: Will Dyson @ 2004-08-31 3:14 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser wrote: > I think there are two ways to analyze the code boundary issue. One is > "does it belong in the kernel?" Another is, "does it belong in the > filesystem. and if so should name resolution in a filesystem be split > into two parts, one in kernel, and one in user space." In ten years I > might have the knowledge needed to make such a split, but I know for > sure that I don't know how to do it today without regretting it > tomorrow, and I don't really have confidence that I will ever be able to > do it without losing performance. I don't see how exporting a set of indices on file attributes to userspace constitutes putting part of the name resolution into userspace. A file's name (or names, in the case of hardlinks) would still be determined entirely within the filesystem. The more I think about it, the more I am convinced that "the index" is the correct primative to use for exposing any filesystem's fast searching features. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-27 18:20 ` Will Dyson 2004-08-27 18:38 ` David Ford 2004-08-28 9:54 ` Hans Reiser @ 2004-08-28 15:18 ` Alexander G. M. Smith 2004-08-28 22:11 ` Hans Reiser ` (2 more replies) 2 siblings, 3 replies; 1835+ messages in thread From: Alexander G. M. Smith @ 2004-08-28 15:18 UTC (permalink / raw) To: Will Dyson Cc: akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, reiser Will Dyson wrote on Fri, 27 Aug 2004 14:20:29 -0400: > Hans Reiser wrote: > > Will Dyson wrote: > >> In the original BeOS, they solved the problem by having the filesystem > >> driver itself take a text query string and parse it, returning a list > >> of inodes that match. The whole business of parsing a query string in > >> the kernel (let alone in the filesystem driver) has always seemed ugly > >> to me. > > > > Why? > > Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. > > String parsing bloats the kernel with code that will be called rarely, > and doing it inside the filesystem module makes for duplicate code if > more than one filesystem does it. But a good common parser routine (or a > kernel api that takes a pre-compiled parse tree) would reduce the bad taste. Not only bloat, but time wasted reinventing the wheel. For my experimental file system (http://members.rogers.com/agmsbeos/AGMSRAMFileSystem.readme - see the Nifty Features chapter), I had to figure out the BFS query language (it was documented but didn't have every little detail explained - some experimentation was needed), then write a tokenizer and parser for it, to convert it down to a parse tree that could be used during the query's life. Good exercise for the programmer, but duplication of effort. I also added a few new things to the query language, but because the parsing is done by each file system separately, my "improvements" won't show up in other file systems. > For the 99.9% of people who were not beos users back in the day, the > "live query" was a way to register a query string with the kernel, so > that you got notifications when the list of files matching the query > changed. It was pretty cool, but this is really just blue-skying since I > am not stepping up to write such a beast any time soon. Kind of like that. For performance, the query string is converted into a parse tree and then all file operations that touch any attribute mentioned in the query get evaluated against the parse tree to see if the result (in or out of the query result set) has changed. If it changes, then a kernel facility is used to broadcast a message describing the change and file involved to all parties monitoring that query. Of course, before becoming live, the existing files that match the query are found by iterating over an index (choosing the appropriate index is an optimization trick). > >> However, the best alternative I've come up with is to simply export > >> the index data as a special file (perhaps in sysfs?) and have > >> userspace responsible for searching the index. > > > > That is the best implementation suggestion I've heard for splitting the > > filesystem into two parts, one in user space and one in kernel, but I > > still don't trust it to work well. > > Why not? And if it really splits the filesystem into two parts depends > on which of the two following statements you agree with more: > > "Fast searching is a feature of the filesystem" > "Maintaining indexes is a feature of the filesystem" I like that. Put the indexing and a related attribute change update notification mechanism into the kernel. User land libraries can build on that to implement whatever query language you want, thus saving work and having a common language for all file system varieties that support indices. However, one of my (unfinished) experiments was to have magic directories that show query results as their contents. One attribute of the directory (or even its name) would be the query string. That way even old software (like "ls") could use queries! Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level. - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith @ 2004-08-28 22:11 ` Hans Reiser 2004-08-28 22:12 ` Hans Reiser 2004-08-31 1:57 ` Will Dyson 2 siblings, 0 replies; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 22:11 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Will Dyson, akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Alexander G. M. Smith wrote: > > >However, one of my (unfinished) experiments was to have magic directories that show query results as their contents. One attribute of the directory (or even its name) would be the query string. That way even old software (like "ls") could use queries! Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level. > > Beautiful example of the sort of thing that I don't want to be locked out from doing by having the parser in user space. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2004-08-28 22:11 ` Hans Reiser @ 2004-08-28 22:12 ` Hans Reiser 2004-09-01 20:41 ` Jamie Lokier 2004-08-31 1:57 ` Will Dyson 2 siblings, 1 reply; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 22:12 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Will Dyson, akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Alexander G. M. Smith wrote: > > >However, one of my (unfinished) experiments was to have magic directories that show query results as their contents. One attribute of the directory (or even its name) would be the query string. That way even old software (like "ls") could use queries! Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level. > >- Alex > > > > Symlinks also. Symlinks with powerful queries in them would require a parser in the kernel. Thanks for helping me to distill my incoherent reasons for the parser being in the kernel. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-28 22:12 ` Hans Reiser @ 2004-09-01 20:41 ` Jamie Lokier 0 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 20:41 UTC (permalink / raw) To: Hans Reiser Cc: Alexander G. M. Smith, Will Dyson, akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser wrote: > Symlinks also. Symlinks with powerful queries in them would require a > parser in the kernel. No they wouldn't. You can do symlinks with powerful queries in userspace _today_ (as well as directories which list queries). Making queries be up to date in real-time is still a problem, but that has nothing to do with symlinks or directories, or where the parsers live. > Thanks for helping me to distill my incoherent reasons for the > parser being in the kernel. This is one occasion where you're mistaken :) There may be other reasons for parsing query strings in the kernel (though I'm not convinced there are any), but this isn't one. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2004-08-28 22:11 ` Hans Reiser 2004-08-28 22:12 ` Hans Reiser @ 2004-08-31 1:57 ` Will Dyson 2004-08-31 2:33 ` Alexander G. M. Smith 2 siblings, 1 reply; 1835+ messages in thread From: Will Dyson @ 2004-08-31 1:57 UTC (permalink / raw) To: Alexander G. M. Smith Cc: akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, reiser Alexander G. M. Smith wrote: > Not only bloat, but time wasted reinventing the wheel. For my > experimental file system > (http://members.rogers.com/agmsbeos/AGMSRAMFileSystem.readme - see > the Nifty Features chapter), I had to figure out the BFS query I had a look. Cool stuff! > I like that. Put the indexing and a related attribute change update > notification mechanism into the kernel. User land libraries can > build on that to implement whatever query language you want, thus > saving work and having a common language for all file system > varieties that support indices. Yeah, thats basicly what I was thinking. Right now I'm trying to think of any kind of filesystem that could support the general concept of "Fast Searching", but be somehow unable to export this capability as a set of indices on file attributes.... Ok. I give up. It seems like the right abstraction to me. > However, one of my (unfinished) experiments was to have magic > directories that show query results as their contents. One attribute > of the directory (or even its name) would be the query string. That > way even old software (like "ls") could use queries! Implementing > queries-as-directories might require moving some things back into the > kernel, or at least into some plug-in level. That is a pretty awesome idea. It should be done in its own specialized filesystem though. QueryFS. If the change-notification mechanism in the kernel was robust enough (a big if), the rest could be done with a userspace filesystem (via FUSE). Mount a query, cd into it, run grep, write a little bash script... Very unixy. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-31 1:57 ` Will Dyson @ 2004-08-31 2:33 ` Alexander G. M. Smith 2004-08-31 3:22 ` Will Dyson 0 siblings, 1 reply; 1835+ messages in thread From: Alexander G. M. Smith @ 2004-08-31 2:33 UTC (permalink / raw) To: Will Dyson; +Cc: linux-fsdevel, reiserfs-list Will Dyson wrote on Mon, 30 Aug 2004 21:57:17 -0400: > I had a look. Cool stuff! Thanks! > That is a pretty awesome idea. It should be done in its own specialized > filesystem though. QueryFS. If the change-notification mechanism in the > kernel was robust enough (a big if), the rest could be done with a > userspace filesystem (via FUSE). Mount a query, cd into it, run grep, > write a little bash script... Very unixy. That brings up a good point, should the queries be allowed to run across file systems or be restricted to just one? BeOS was restricted to one file system (the query evaluation code was file system specific) and faked it for multiple ones at the user level by repeating the query on each FS. A separate Query FS wouldn't have that limit (or could fake it at the kernel level), if the kernel/FSs exported their indices and change notifications. - Alex ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-31 2:33 ` Alexander G. M. Smith @ 2004-08-31 3:22 ` Will Dyson 0 siblings, 0 replies; 1835+ messages in thread From: Will Dyson @ 2004-08-31 3:22 UTC (permalink / raw) To: Alexander G. M. Smith; +Cc: linux-fsdevel, reiserfs-list Alexander G. M. Smith wrote: > That brings up a good point, should the queries be allowed to run across > file systems or be restricted to just one? BeOS was restricted to one > file system (the query evaluation code was file system specific) and > faked it for multiple ones at the user level by repeating the query on > each FS. A separate Query FS wouldn't have that limit (or could fake > it at the kernel level), if the kernel/FSs exported their indices and > change notifications. I don't see any fundamental reason why filesystem boundaries couldn't be crossed. A separate Query FS could even fake it by scrubbing the disk with /usr/bin/find for filesystems that didn't export indices. Probably a bad idea, of course. But painfully slow execution is not a reason it _couldn't_ be done. Some people even like pain... -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser ` (3 preceding siblings ...) 2004-08-26 16:32 ` Will Dyson @ 2004-08-26 18:12 ` Horst von Brand 2004-08-27 20:32 ` Markus Törnqvist ` (2 more replies) 2004-08-26 20:04 ` Martin J. Bligh 2004-08-27 21:23 ` vfs2 (was Re: silent semantic changes with reiser4) Pavel Machek 6 siblings, 3 replies; 1835+ messages in thread From: Horst von Brand @ 2004-08-26 18:12 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser <reiser@namesys.com> said: > Andrew Morton wrote: [...] > > The fact that one filesystem will offer features which other > > filesystems do not and cannot offer makes me queasy for some reason. Not me. As long as it is clearly experimental stuff that can be removed at the head hacker's whim, that is. 2.7 material. > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > for Apple, Says who? > and that means we need to put search engine and database > functionality into the filesystem. Please don't. Unix works and is extremely popular because it _doesn't_ try to be smart (policy vs mechanism distinction, simple abstractions that can be combined endlessly). If you add this to the filesystems, you either redo _all_ userland to understand _one_ particular way of doing "smart things" (which will turn out to be almost exactly the dumbest possible for some uses, and then you are stuck), or get lots of shards from broken apps (and users, and sysadmins). > It takes 11 years of serious > research to build a clean storage layer able to handle doing that. Great! Build an experimental OS showing how to use it, and through that see if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. > Reiser4 has done that, finally. None of the other Linux filesystems > have. How come nobody before you in the almost 30 years of Unix has ever seen the need for this indispensable feature? > The next major release of ReiserFS is going to be bursting with > semantic enhancements, because the prerequisites for them are in place > now. None of the other Linux filesystems have those prerequisites. To me that would mean that ReiserFS has no place in Linux. > They won't be able to keep up with the semantic enhancements. This > metafiles and file-directories stuff is actually fairly trivial stuff. Maybe. But the question of it being of any use aren't trivially answered "yes". My gut feeling is that the answer is a resounding "no", but that's just me. Direcories are directories, if you want to ship directory-like stuff around, use directories (or tarfiles, or whatever). Just look what happened to the much, much easier stuff of splitting SUID/SGID into capabilities: Nothing at all whatsoever, because they have no decent place to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be able to _define_ APIs to follow, but even so it isn't at all easy to do so. > Look guys, in 1993 I anticipated the battle would be here, and I build > the foundation for a defensive tower right at the spot MS and Apple are > now maneuvering towards. Why place a protective tower around the pit into which they are trying desperately to throw themselves? ;-) > Help me get the next level on the tower before > they get here. It is one hell of a foundation, they won't be able to > shake it, their trees are not as powerful. Don't move reiser4 into vfs, > use reiser4 as the vfs. Don't write filesystems, write file plugins and > disk format plugins and all the other kinds of plugins, and you won't be > missing any expressive power that you really want.... Better write libraries handling whatever weird format you have in mind for each use. Even applications. I don't expect to be able to use emacs to edit a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit a poem. The current trend (which I heartily agree with) is to take stuff _out_ of the kernel (for flexibility, access, development ease, even performance). -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand 2004-08-27 20:32 ` Markus Törnqvist @ 2004-08-27 20:32 ` Markus Törnqvist 2004-09-01 21:43 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-27 20:32 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 02:12:32PM -0400, Horst von Brand wrote: >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> for Apple, >Says who? People will say it when people stop using Linux on servers because they can integrate metadata easier in other operating systems ;) >Please don't. Unix works and is extremely popular because it _doesn't_ try >to be smart (policy vs mechanism distinction, simple abstractions that can >be combined endlessly). If you add this to the filesystems, you either redo Just more moving little pieces to be combined endlessly. I think it looks like Reiser4 semantics should go into VFS and not break legacy posix compatibility. If you don't want to use a filesystem like that, don't. Maybe the reason Linux hasn't developed this yet is the fact that Linux supports n+1 file systems and above and beyond that many userspace ones. So there has been enough work in maintaining and competing the old ones and not coming up with new ones. >_all_ userland to understand _one_ particular way of doing "smart things" >(which will turn out to be almost exactly the dumbest possible for some >uses, and then you are stuck), or get lots of shards from broken apps (and >users, and sysadmins). This is a question if implementation. What if the files-as-dirs have a ..metas/mainstream[1] file and are handled by legacy applications as directories? Nothing breaks, but may look a bit ugly. Nothing one line of shell scripting would not cure. OTOH, the user space progs are quickly patched to support all this. >Great! Build an experimental OS showing how to use it, and through that see >if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. Why should he? Parts of Linux are already here and susceptible to experiments in evolution. (Break out 2.7 for this one, the people want code! :) >How come nobody before you in the almost 30 years of Unix has ever seen the >need for this indispensable feature? Lack of imagination? Lack of need? Lack of drive space? And maybe someone did think among these lines back in the days, it's not like the current dogma of file systems is the only one that has ever been used. >To me that would mean that ReiserFS has no place in Linux. This is the type of argumentation I can not understand. Do you not have the freedom not to use it? And this is not personal, but please say Reiser4, as ReiserFS is coupled with versions 3.x. They are quite different. >Maybe. But the question of it being of any use aren't trivially answered >"yes". My gut feeling is that the answer is a resounding "no", but that's I think it's a pretty big yes, even if the technical details do need to be clear before anything is set in stone. That's the reason for the on-going discussion. >just me. Direcories are directories, if you want to ship directory-like >stuff around, use directories (or tarfiles, or whatever). Just look what Without a common user-space API to deal in them. Oh, joy. Make the common user-space API? Will never happen. >Better write libraries handling whatever weird format you have in mind for >each use. Even applications. I don't expect to be able to use emacs to edit >a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >a poem. The current trend (which I heartily agree with) is to take stuff >_out_ of the kernel (for flexibility, access, development ease, even >performance). I don't think anyone's talking about using emacs to edit a picture, but emacs could be used to edit the picture's description.[2] Maybe with some userspace-hook that exports the exif data via ..metas/ so the kernel won't have to parse it. Linus wrote about why TCP is not in userspace and I think this is a similar situation. Technically. But not everything has to be in kernel space in Reiser4. Or that's what they say and I'm not sure where I stand on an exif parser in the kernel... On the social side, I may be wrong in this, but not many userspace filesystems have the possibility of becoming a de-facto standard. One might say that's because they suck, or they suck because they're implemented in userspace or they are just met with prejudice for the same reason. In the kernel something like this would have to be done right and people would put effort into doing it right. I for one would welcome that. [1] Sorry, guys, just had to use that name.. [2] You can edit a picture in emacs already, it's just prone to break ;) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 20:32 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: mjt @ 2004-08-27 20:32 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 02:12:32PM -0400, Horst von Brand wrote: >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> for Apple, >Says who? People will say it when people stop using Linux on servers because they can integrate metadata easier in other operating systems ;) >Please don't. Unix works and is extremely popular because it _doesn't_ try >to be smart (policy vs mechanism distinction, simple abstractions that can >be combined endlessly). If you add this to the filesystems, you either redo Just more moving little pieces to be combined endlessly. I think it looks like Reiser4 semantics should go into VFS and not break legacy posix compatibility. If you don't want to use a filesystem like that, don't. Maybe the reason Linux hasn't developed this yet is the fact that Linux supports n+1 file systems and above and beyond that many userspace ones. So there has been enough work in maintaining and competing the old ones and not coming up with new ones. >_all_ userland to understand _one_ particular way of doing "smart things" >(which will turn out to be almost exactly the dumbest possible for some >uses, and then you are stuck), or get lots of shards from broken apps (and >users, and sysadmins). This is a question if implementation. What if the files-as-dirs have a ..metas/mainstream[1] file and are handled by legacy applications as directories? Nothing breaks, but may look a bit ugly. Nothing one line of shell scripting would not cure. OTOH, the user space progs are quickly patched to support all this. >Great! Build an experimental OS showing how to use it, and through that see >if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. Why should he? Parts of Linux are already here and susceptible to experiments in evolution. (Break out 2.7 for this one, the people want code! :) >How come nobody before you in the almost 30 years of Unix has ever seen the >need for this indispensable feature? Lack of imagination? Lack of need? Lack of drive space? And maybe someone did think among these lines back in the days, it's not like the current dogma of file systems is the only one that has ever been used. >To me that would mean that ReiserFS has no place in Linux. This is the type of argumentation I can not understand. Do you not have the freedom not to use it? And this is not personal, but please say Reiser4, as ReiserFS is coupled with versions 3.x. They are quite different. >Maybe. But the question of it being of any use aren't trivially answered >"yes". My gut feeling is that the answer is a resounding "no", but that's I think it's a pretty big yes, even if the technical details do need to be clear before anything is set in stone. That's the reason for the on-going discussion. >just me. Direcories are directories, if you want to ship directory-like >stuff around, use directories (or tarfiles, or whatever). Just look what Without a common user-space API to deal in them. Oh, joy. Make the common user-space API? Will never happen. >Better write libraries handling whatever weird format you have in mind for >each use. Even applications. I don't expect to be able to use emacs to edit >a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >a poem. The current trend (which I heartily agree with) is to take stuff >_out_ of the kernel (for flexibility, access, development ease, even >performance). I don't think anyone's talking about using emacs to edit a picture, but emacs could be used to edit the picture's description.[2] Maybe with some userspace-hook that exports the exif data via ..metas/ so the kernel won't have to parse it. Linus wrote about why TCP is not in userspace and I think this is a similar situation. Technically. But not everything has to be in kernel space in Reiser4. Or that's what they say and I'm not sure where I stand on an exif parser in the kernel... On the social side, I may be wrong in this, but not many userspace filesystems have the possibility of becoming a de-facto standard. One might say that's because they suck, or they suck because they're implemented in userspace or they are just met with prejudice for the same reason. In the kernel something like this would have to be done right and people would put effort into doing it right. I for one would welcome that. [1] Sorry, guys, just had to use that name.. [2] You can edit a picture in emacs already, it's just prone to break ;) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 20:32 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-27 20:32 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 02:12:32PM -0400, Horst von Brand wrote: >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> for Apple, >Says who? People will say it when people stop using Linux on servers because they can integrate metadata easier in other operating systems ;) >Please don't. Unix works and is extremely popular because it _doesn't_ try >to be smart (policy vs mechanism distinction, simple abstractions that can >be combined endlessly). If you add this to the filesystems, you either redo Just more moving little pieces to be combined endlessly. I think it looks like Reiser4 semantics should go into VFS and not break legacy posix compatibility. If you don't want to use a filesystem like that, don't. Maybe the reason Linux hasn't developed this yet is the fact that Linux supports n+1 file systems and above and beyond that many userspace ones. So there has been enough work in maintaining and competing the old ones and not coming up with new ones. >_all_ userland to understand _one_ particular way of doing "smart things" >(which will turn out to be almost exactly the dumbest possible for some >uses, and then you are stuck), or get lots of shards from broken apps (and >users, and sysadmins). This is a question if implementation. What if the files-as-dirs have a ..metas/mainstream[1] file and are handled by legacy applications as directories? Nothing breaks, but may look a bit ugly. Nothing one line of shell scripting would not cure. OTOH, the user space progs are quickly patched to support all this. >Great! Build an experimental OS showing how to use it, and through that see >if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. Why should he? Parts of Linux are already here and susceptible to experiments in evolution. (Break out 2.7 for this one, the people want code! :) >How come nobody before you in the almost 30 years of Unix has ever seen the >need for this indispensable feature? Lack of imagination? Lack of need? Lack of drive space? And maybe someone did think among these lines back in the days, it's not like the current dogma of file systems is the only one that has ever been used. >To me that would mean that ReiserFS has no place in Linux. This is the type of argumentation I can not understand. Do you not have the freedom not to use it? And this is not personal, but please say Reiser4, as ReiserFS is coupled with versions 3.x. They are quite different. >Maybe. But the question of it being of any use aren't trivially answered >"yes". My gut feeling is that the answer is a resounding "no", but that's I think it's a pretty big yes, even if the technical details do need to be clear before anything is set in stone. That's the reason for the on-going discussion. >just me. Direcories are directories, if you want to ship directory-like >stuff around, use directories (or tarfiles, or whatever). Just look what Without a common user-space API to deal in them. Oh, joy. Make the common user-space API? Will never happen. >Better write libraries handling whatever weird format you have in mind for >each use. Even applications. I don't expect to be able to use emacs to edit >a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >a poem. The current trend (which I heartily agree with) is to take stuff >_out_ of the kernel (for flexibility, access, development ease, even >performance). I don't think anyone's talking about using emacs to edit a picture, but emacs could be used to edit the picture's description.[2] Maybe with some userspace-hook that exports the exif data via ..metas/ so the kernel won't have to parse it. Linus wrote about why TCP is not in userspace and I think this is a similar situation. Technically. But not everything has to be in kernel space in Reiser4. Or that's what they say and I'm not sure where I stand on an exif parser in the kernel... On the social side, I may be wrong in this, but not many userspace filesystems have the possibility of becoming a de-facto standard. One might say that's because they suck, or they suck because they're implemented in userspace or they are just met with prejudice for the same reason. In the kernel something like this would have to be done right and people would put effort into doing it right. I for one would welcome that. [1] Sorry, guys, just had to use that name.. [2] You can edit a picture in emacs already, it's just prone to break ;) -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 20:32 ` Markus Törnqvist (?) (?) @ 2004-08-27 20:38 ` Linus Torvalds 2004-08-27 20:48 ` Re[2]: " Spam ` (3 more replies) -1 siblings, 4 replies; 1835+ messages in thread From: Linus Torvalds @ 2004-08-27 20:38 UTC (permalink / raw) To: Markus Törnqvist Cc: Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Markus Törnqvist wrote: > > People will say it when people stop using Linux on servers because > they can integrate metadata easier in other operating systems ;) Heh. Considering that WinFS seems to be delayed yet more, I don't think that's a very strong argument. Hell will freeze over before Microsoft does a filesystem right. Besides, WinFS is likely almost in user mode anyway, ie mostly a library, rather like the gnome people are already doing with nome storage. So there's really no point in trying to push your agenda by trying to scare people with MS activities. Linux kernel developers do what's right because it is _right_, not because somebody else does it. Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re[2]: silent semantic changes with reiser4 2004-08-27 20:38 ` Linus Torvalds @ 2004-08-27 20:48 ` Spam 2004-08-27 20:53 ` Markus Törnqvist ` (2 subsequent siblings) 3 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-27 20:48 UTC (permalink / raw) To: Linus Torvalds Cc: Markus Törnqvist, Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Fri, 27 Aug 2004, Markus Törnqvist wrote: >> >> People will say it when people stop using Linux on servers because >> they can integrate metadata easier in other operating systems ;) > Heh. Considering that WinFS seems to be delayed yet more, I don't think > that's a very strong argument. > Hell will freeze over before Microsoft does a filesystem right. Besides, > WinFS is likely almost in user mode anyway, ie mostly a library, rather > like the gnome people are already doing with nome storage. Yes, it is mostly in user land. But, It is still integrated with the Windows API. This means that normal programs that use the Windows API will be able to use most of the new features (like displaying thumbnails, filtered views, etc when you do file operations) Besides, NTFS is not that bad. It does have both encryption and compression and a big set of ACL features that aren't in most Linux filesystems today. > So there's really no point in trying to push your agenda by trying to > scare people with MS activities. Linux kernel developers do what's right > because it is _right_, not because somebody else does it. I agree. Things should be done because they are good, not because you are threatened to. It is a shame really that so many people hate Microsoft (or Apple) so much they fail to see things that actually are good. > Linus ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 20:38 ` Linus Torvalds 2004-08-27 20:48 ` Re[2]: " Spam @ 2004-08-27 20:53 ` Markus Törnqvist 2004-08-27 21:33 ` Bill Huey 2004-08-28 9:55 ` Hans Reiser 3 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-27 20:53 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote: > >Heh. Considering that WinFS seems to be delayed yet more, I don't think >that's a very strong argument. But there's Mac OS and isn't Solaris doing something along these lines? Microsoft isn't alone there ;) >scare people with MS activities. Linux kernel developers do what's right >because it is _right_, not because somebody else does it. Never really doubted that, mind you. I just think this is both right and a competition situation. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 20:53 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: mjt @ 2004-08-27 20:53 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote: > >Heh. Considering that WinFS seems to be delayed yet more, I don't think >that's a very strong argument. But there's Mac OS and isn't Solaris doing something along these lines? Microsoft isn't alone there ;) >scare people with MS activities. Linux kernel developers do what's right >because it is _right_, not because somebody else does it. Never really doubted that, mind you. I just think this is both right and a competition situation. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 20:53 ` Markus Törnqvist 0 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-27 20:53 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote: > >Heh. Considering that WinFS seems to be delayed yet more, I don't think >that's a very strong argument. But there's Mac OS and isn't Solaris doing something along these lines? Microsoft isn't alone there ;) >scare people with MS activities. Linux kernel developers do what's right >because it is _right_, not because somebody else does it. Never really doubted that, mind you. I just think this is both right and a competition situation. -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 20:38 ` Linus Torvalds 2004-08-27 20:48 ` Re[2]: " Spam 2004-08-27 20:53 ` Markus Törnqvist @ 2004-08-27 21:33 ` Bill Huey 2004-08-28 9:55 ` Hans Reiser 3 siblings, 0 replies; 1835+ messages in thread From: Bill Huey @ 2004-08-27 21:33 UTC (permalink / raw) To: Linus Torvalds Cc: Markus T?rnqvist, Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, bhuey On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote: > Heh. Considering that WinFS seems to be delayed yet more, I don't think > that's a very strong argument. Which is another argument, and bragging rights, to integrate something like Reiser FS4. :) > Hell will freeze over before Microsoft does a filesystem right. Besides, > WinFS is likely almost in user mode anyway, ie mostly a library, rather > like the gnome people are already doing with nome storage. They're trying to push their low level storage model into the kernel after they saw how slow their userspace version of that is. It makes sense to have this kind of stuff directly apart of a lower level FS later > So there's really no point in trying to push your agenda by trying to > scare people with MS activities. Linux kernel developers do what's right > because it is _right_, not because somebody else does it. I tried to stay out of this thread, but I couldn't resist after this post. :) MS has other problems, namely, they're a large company trying to basically replicated Apple's (NeXT) OS X stuff, but onl 10x larger and slower than it is. It's really silly what's going on with MS development as well as the greater open source X community. The sooner folks realize that all rendering models are moving in the direction of Apple's OS X, the sooner they can just flat out rip it off. It's going to require both kernel and userspace changes for this to happen. bill ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 20:38 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-27 21:33 ` Bill Huey @ 2004-08-28 9:55 ` Hans Reiser 2004-09-01 22:45 ` Bill Huey 3 siblings, 1 reply; 1835+ messages in thread From: Hans Reiser @ 2004-08-28 9:55 UTC (permalink / raw) To: Linus Torvalds Cc: Markus Törnqvist, Horst von Brand, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: >On Fri, 27 Aug 2004, Markus Törnqvist wrote: > > >>People will say it when people stop using Linux on servers because >>they can integrate metadata easier in other operating systems ;) >> >> > >Heh. Considering that WinFS seems to be delayed yet more, I don't think >that's a very strong argument. > >Hell will freeze over before Microsoft does a filesystem right. Besides, >WinFS is likely almost in user mode anyway, ie mostly a library, rather >like the gnome people are already doing with nome storage. > >So there's really no point in trying to push your agenda by trying to >scare people with MS activities. Linux kernel developers do what's right >because it is _right_, not because somebody else does it. > > Linus > > > > Apple will get it right. I promise it. I have met Dominic, and he is very very sharp. Look at the Tiger demos on their website. Simple interface, looks nice to me.... The one area he might screw up is performance, but I don't care to count on that. WinFS first tried to put it all in the FS, and then it became a user mode library almost certainly because they are making the standard mistakes the database guys make when they try to emulate file systems without changing the core balanced tree algorithms, and their performance sucked and they had to back off. It took 11 years for me to get it right, and they aren't as crazy-err-persistent as I am.;-) We might get lucky and have them produce another NTFS, but then again, when Microsoft focuses on a task, they do much better at it than they do most of the time, and they are focused on WinFS. They have hired very sharp people. We can hope that they don't know how to use them, but when they hire people like Gerard Salton for $1 million a year, there is just possibly a chance that they might try to get their money's worth out of him. You should not be complacent about WinFS being delayed to 2007, because even if I get funding for enhanced ReiserFS semantics tomorrow we also can't get the job done before 2007. This is big science, not writing a device driver. Finally, how much harm will it be if we do it right and it is important and they fail? Suppose I am wrong about them, and we create a powerful unifying namespace for Linux before any other OS does? Is that so bad? Creating a powerful namespace at the heart of Linux is the most important enhancement you can make to the OS. Finally the storage layer is good enough to support putting the relationship between keywords (actually keyobjects in my scheme....) and their documents directly into the FS without losing performance for traditional file system usage patterns, and I get to stop tweaking performance and go have fun with semantics in the next major release. Hans ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 9:55 ` Hans Reiser @ 2004-09-01 22:45 ` Bill Huey 0 siblings, 0 replies; 1835+ messages in thread From: Bill Huey @ 2004-09-01 22:45 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Markus T?rnqvist, Horst von Brand, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, Aug 28, 2004 at 02:55:00AM -0700, Hans Reiser wrote: > mistakes the database guys make when they try to emulate file systems > without changing the core balanced tree algorithms, and their > performance sucked and they had to back off. It took 11 years for me to > get it right, and they aren't as crazy-err-persistent as I am.;-) > > We might get lucky and have them produce another NTFS, but then again, > when Microsoft focuses on a task, they do much better at it than they do > most of the time, and they are focused on WinFS. They have hired very > sharp people. We can hope that they don't know how to use them, but > when they hire people like Gerard Salton for $1 million a year, there is > just possibly a chance that they might try to get their money's worth > out of him. I realize this is a relatively late reply, but... MS's problem is that they hire very smart folks, but the compartimentalize them so they don't influence the overall architecture of the entire system. I guarantee you that even if they did get all of this in place and working, their arrogance in this matter will screw up how this will be properly used and never expressed the power of a system like this. Apple is another story. They're the only other folks that might get this right. Your system is a dream of mine since I was dorking around with persistent objects in Smalltalk systems. But getting other folks to understand something like this and then use it in this way is very difficult and I don't expect MS to be able to do this nor get .NET to compete with Java for server side applications. I don't expect traditional Unix folks to understand why something like this is important also. It's not about a single technology as much as the system as a whole, larger picture... That's were a system starts to show it's superiority. That's my two cents on the matter. bill ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand @ 2004-08-27 20:44 ` Spam 2004-08-27 20:44 ` Spam 2004-09-01 21:43 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-27 20:44 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > Hans Reiser <reiser@namesys.com> said: >> Andrew Morton wrote: > [...] >> > The fact that one filesystem will offer features which other >> > filesystems do not and cannot offer makes me queasy for some reason. > Not me. As long as it is clearly experimental stuff that can be removed at > the head hacker's whim, that is. 2.7 material. >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> for Apple, > Says who? At least I as a user see the benefits largely. The workstation / desktop user has much more need for such features, than perhaps the normal Linux sysadmin. >> and that means we need to put search engine and database >> functionality into the filesystem. > Please don't. Unix works and is extremely popular because it _doesn't_ try > to be smart (policy vs mechanism distinction, simple abstractions that can > be combined endlessly). If you add this to the filesystems, you either redo > _all_ userland to understand _one_ particular way of doing "smart things" > (which will turn out to be almost exactly the dumbest possible for some > uses, and then you are stuck), or get lots of shards from broken apps (and > users, and sysadmins). Indeed. But you forget one thing. Combining lots of small userland tools can actually make things more complex - at least for the user. That said, I do not see that anyone has suggested to remove this way of working with applications. Are the new and old features mutually exclusive? >> It takes 11 years of serious >> research to build a clean storage layer able to handle doing that. > Great! Build an experimental OS showing how to use it, and through that see > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. There are many uses already. It could be encryption, compression, meta-data, file streams, sorting, searching and a multitude of uses and ideas that haven't been discovered yet. >> Reiser4 has done that, finally. None of the other Linux filesystems >> have. > How come nobody before you in the almost 30 years of Unix has ever seen the > need for this indispensable feature? I, as a user, have for a long time. Do not drag everyone over the same edge. Users have different needs. We should try to be as forthcoming as possible to provide the tools and options for them. >> The next major release of ReiserFS is going to be bursting with >> semantic enhancements, because the prerequisites for them are in place >> now. None of the other Linux filesystems have those prerequisites. > To me that would mean that ReiserFS has no place in Linux. Why do you say this? I thought Linux embraced advancements and new technologies. >> They won't be able to keep up with the semantic enhancements. This >> metafiles and file-directories stuff is actually fairly trivial stuff. > Maybe. But the question of it being of any use aren't trivially answered > "yes". My gut feeling is that the answer is a resounding "no", but that's > just me. Direcories are directories, if you want to ship directory-like > stuff around, use directories (or tarfiles, or whatever). Just look what > happened to the much, much easier stuff of splitting SUID/SGID into > capabilities: Nothing at all whatsoever, because they have no decent place > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be > able to _define_ APIs to follow, but even so it isn't at all easy to do so. Exactly. The problem I see in this discussion is the lack of just this - APIs. Because the lack of good APIs we cannot make any larger changes or bring new features because old applications will break. I think that Linux should make new API that will completely separate applications from the storage media. This way we will not as easily break applications when new features are developed. This should be the main priority. Perhaps over a transition period the old ways will be allowed while applications are ported to the new APIs. The change is certainly going to be needed at one point or Linux won't be able to evolve as quickly as its users are demanding it. Application breakage has happened before between virtually every kernel change. It isn't new. Just take devfs and sysfs. >> Look guys, in 1993 I anticipated the battle would be here, and I build >> the foundation for a defensive tower right at the spot MS and Apple are >> now maneuvering towards. > Why place a protective tower around the pit into which they are trying > desperately to throw themselves? ;-) It is easy to say this. But the fact is that users can actually use these new features for good things. >> Help me get the next level on the tower before >> they get here. It is one hell of a foundation, they won't be able to >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> disk format plugins and all the other kinds of plugins, and you won't be >> missing any expressive power that you really want.... > Better write libraries handling whatever weird format you have in mind for > each use. Even applications. I don't expect to be able to use emacs to edit > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit > a poem. The current trend (which I heartily agree with) is to take stuff > _out_ of the kernel (for flexibility, access, development ease, even > performance). If I want to be able to quickly find files by a certain author I could search the meta-data fields (if they are used) with normal tools such as grep. Compression or encryption is another example. Would you rather patch every application there is for Linux to enable support for reading these files and directories? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-27 20:44 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-27 20:44 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > Hans Reiser <reiser@namesys.com> said: >> Andrew Morton wrote: > [...] >> > The fact that one filesystem will offer features which other >> > filesystems do not and cannot offer makes me queasy for some reason. > Not me. As long as it is clearly experimental stuff that can be removed at > the head hacker's whim, that is. 2.7 material. >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> for Apple, > Says who? At least I as a user see the benefits largely. The workstation / desktop user has much more need for such features, than perhaps the normal Linux sysadmin. >> and that means we need to put search engine and database >> functionality into the filesystem. > Please don't. Unix works and is extremely popular because it _doesn't_ try > to be smart (policy vs mechanism distinction, simple abstractions that can > be combined endlessly). If you add this to the filesystems, you either redo > _all_ userland to understand _one_ particular way of doing "smart things" > (which will turn out to be almost exactly the dumbest possible for some > uses, and then you are stuck), or get lots of shards from broken apps (and > users, and sysadmins). Indeed. But you forget one thing. Combining lots of small userland tools can actually make things more complex - at least for the user. That said, I do not see that anyone has suggested to remove this way of working with applications. Are the new and old features mutually exclusive? >> It takes 11 years of serious >> research to build a clean storage layer able to handle doing that. > Great! Build an experimental OS showing how to use it, and through that see > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. There are many uses already. It could be encryption, compression, meta-data, file streams, sorting, searching and a multitude of uses and ideas that haven't been discovered yet. >> Reiser4 has done that, finally. None of the other Linux filesystems >> have. > How come nobody before you in the almost 30 years of Unix has ever seen the > need for this indispensable feature? I, as a user, have for a long time. Do not drag everyone over the same edge. Users have different needs. We should try to be as forthcoming as possible to provide the tools and options for them. >> The next major release of ReiserFS is going to be bursting with >> semantic enhancements, because the prerequisites for them are in place >> now. None of the other Linux filesystems have those prerequisites. > To me that would mean that ReiserFS has no place in Linux. Why do you say this? I thought Linux embraced advancements and new technologies. >> They won't be able to keep up with the semantic enhancements. This >> metafiles and file-directories stuff is actually fairly trivial stuff. > Maybe. But the question of it being of any use aren't trivially answered > "yes". My gut feeling is that the answer is a resounding "no", but that's > just me. Direcories are directories, if you want to ship directory-like > stuff around, use directories (or tarfiles, or whatever). Just look what > happened to the much, much easier stuff of splitting SUID/SGID into > capabilities: Nothing at all whatsoever, because they have no decent place > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be > able to _define_ APIs to follow, but even so it isn't at all easy to do so. Exactly. The problem I see in this discussion is the lack of just this - APIs. Because the lack of good APIs we cannot make any larger changes or bring new features because old applications will break. I think that Linux should make new API that will completely separate applications from the storage media. This way we will not as easily break applications when new features are developed. This should be the main priority. Perhaps over a transition period the old ways will be allowed while applications are ported to the new APIs. The change is certainly going to be needed at one point or Linux won't be able to evolve as quickly as its users are demanding it. Application breakage has happened before between virtually every kernel change. It isn't new. Just take devfs and sysfs. >> Look guys, in 1993 I anticipated the battle would be here, and I build >> the foundation for a defensive tower right at the spot MS and Apple are >> now maneuvering towards. > Why place a protective tower around the pit into which they are trying > desperately to throw themselves? ;-) It is easy to say this. But the fact is that users can actually use these new features for good things. >> Help me get the next level on the tower before >> they get here. It is one hell of a foundation, they won't be able to >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> disk format plugins and all the other kinds of plugins, and you won't be >> missing any expressive power that you really want.... > Better write libraries handling whatever weird format you have in mind for > each use. Even applications. I don't expect to be able to use emacs to edit > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit > a poem. The current trend (which I heartily agree with) is to take stuff > _out_ of the kernel (for flexibility, access, development ease, even > performance). If I want to be able to quickly find files by a certain author I could search the meta-data fields (if they are used) with normal tools such as grep. Compression or encryption is another example. Would you rather patch every application there is for Linux to enable support for reading these files and directories? ^ permalink raw reply [flat|nested] 1835+ messages in thread
[parent not found: <200408291521.i7TFLsQk028363@localhost.localdomain>]
* Re: silent semantic changes with reiser4 [not found] ` <200408291521.i7TFLsQk028363@localhost.localdomain> @ 2004-08-29 16:52 ` Spam 2004-08-29 16:53 ` Spam 2004-08-29 16:54 ` Spam 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-29 16:52 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Horst von Brand <vonbrand@inf.utfsm.cl> said: >> > Hans Reiser <reiser@namesys.com> said: >> >> Andrew Morton wrote: > [...] >> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> >> for Apple, >> >> > Says who? >> >> At least I as a user see the benefits largely. The workstation / >> desktop user has much more need for such features, than perhaps the >> normal Linux sysadmin. > Show benefits, show there are no downsides. Show that the same can't > possibly be done in userland. Then I'll consider buying it. I am not saying it couldn't be done in userland. Most things can. The problem with userland is that it doesn't provide generic access across most applications. For example GnomeVFS is limited to Gnome applications only. The same for KDE. to be able to coordinate application developer on anything is just almost impossible. Still. Why do you oppose plugins, streams and meta files? The could be valuable and easy to use tools for many purposes. One could be would be advanced ACLs defined as a meta-file using XML format. > AFAICS: Smallish benefits for some narrow application niche. Mayor problems > (no decent multiuser way of using it, extremely complex stuff placed in the > kernel). Userland solutions are all around us right now, and work just > fine. No, placing the same gunk in the kernel won't make it magically use > no time or memory, more like exactly the opposite. Besides, all the visions > waved around here have very little in common (except the idea that they > make all restrictions magically dissapear). > Get a list of required features, show concrete ways of using it, outline > possible implementations, discuss resource usage. I.e., do "engineering". > Then we have something to discuss. I am not sure what you refer to as "required features". Do you mean example of usage and contents for streams, plugins and meta files? >> >> and that means we need to put search engine and database >> >> functionality into the filesystem. >> > Please don't. Unix works and is extremely popular because it _doesn't_ try >> > to be smart (policy vs mechanism distinction, simple abstractions that can >> > be combined endlessly). If you add this to the filesystems, you either redo >> > _all_ userland to understand _one_ particular way of doing "smart things" >> > (which will turn out to be almost exactly the dumbest possible for some >> > uses, and then you are stuck), or get lots of shards from broken apps (and >> > users, and sysadmins). >> Indeed. But you forget one thing. Combining lots of small userland >> tools can actually make things more complex - at least for the user. > So? Placing lots of little things in kernel does the same, plus screws over > sysadmins, and makes kernel hacker's job harder. I prefer the former alone. Difference being that it could be transparent in a way otherwise not possible. >> That said, I do not see that anyone has suggested to remove this way >> of working with applications. > Excuse me, if files suddenly become directories, and you can't have some > "reserved" file/directory names, and... you are breaking lots of stuff. >> Are the new and old features mutually exclusive? > Yep. >> >> It takes 11 years of serious >> >> research to build a clean storage layer able to handle doing that. >> >> > Great! Build an experimental OS showing how to use it, and through that see >> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. >> There are many uses already. > Where? >> It could be encryption, compression, >> meta-data, file streams, sorting, searching and a multitude of uses >> and ideas that haven't been discovered yet. > "Could", "multitude ... haven't been discovered yet" is _not_ "many current > uses". >> >> Reiser4 has done that, finally. None of the other Linux filesystems >> >> have. >> > How come nobody before you in the almost 30 years of Unix has ever seen the >> > need for this indispensable feature? >> I, as a user, have for a long time. Do not drag everyone over the >> same edge. Users have different needs. We should try to be as >> forthcoming as possible to provide the tools and options for them. > Why? Breaking the basic paradigms for some user whim is not good OS design > in my book, sorry. Many "bright ideas" have been thrown out over the years > for just the same reasons. >> >> The next major release of ReiserFS is going to be bursting with >> >> semantic enhancements, because the prerequisites for them are in place >> >> now. None of the other Linux filesystems have those prerequisites. >> > To me that would mean that ReiserFS has no place in Linux. >> Why do you say this? I thought Linux embraced advancements and new >> technologies. > If they are right. I think breaking the base of the OS is not a good > idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where > this is supposedly so so badly needed?), but not here. You mean in Windows and MacOS where much of this functionality already exists? >> >> They won't be able to keep up with the semantic enhancements. This >> >> metafiles and file-directories stuff is actually fairly trivial stuff. >> > Maybe. But the question of it being of any use aren't trivially answered >> > "yes". My gut feeling is that the answer is a resounding "no", but that's >> > just me. Direcories are directories, if you want to ship directory-like >> > stuff around, use directories (or tarfiles, or whatever). Just look what >> > happened to the much, much easier stuff of splitting SUID/SGID into >> > capabilities: Nothing at all whatsoever, because they have no decent place >> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be >> > able to _define_ APIs to follow, but even so it isn't at all easy to do so. >> Exactly. The problem I see in this discussion is the lack of just >> this - APIs. Because the lack of good APIs we cannot make any larger >> changes or bring new features because old applications will break. I'll chop this up... > No. There is a complete lack of answers to why it should be done ("I think > it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from > enough) No perhaps not by itself. But it certainly should be considered. These two OS'es have most of the market share. They have great resources trying to make things that work for their users. Users are more important than you think. Linux and any OS can do what it wants, but if it wants to keep users then it has to listen to them. > , on what the alternatives are ("Sorry, it must be in the kernel > because it is impossible to get application writers to agree on how to do > it" This is a perfectly valid reason. If it is necessary to move something into the core of Linux because it couldn't otherwise be done then it should seriously be considered. > will just add another incompatible way of "doing almost the same", this > time in the kernel), People on this list are actually trying to make it to be fully compatible with old legacy applications. If you hadn't noticed, Reiser4 does work. The meta files do work. The discussion is wether parts of Reiser4 should be made generic and fit into the VFS. > and last but not least what the impact is (performance > wise, application breakage, user/sysadmin expectation breakage, > incompatibility with other systems, kernel development impact, to list a > few). Hash this out, and we can talk. You paint up a doomsday picture here. Nothing of this magnitude has happened and won't. besides, what is wrong if sysadmins would have to learn something new? >> I think that Linux should make new API that will completely separate >> applications from the storage media. This way we will not as easily >> break applications when new features are developed. This should be >> the main priority. > The "files + directories" stuff is a nice abstraction, and works today on > all random-access storage media. You don't care much if it is on a CD, a > floppy, a disk, a RAID array, or a flash card, do you? It won't work on > tapes, or printers; but that is to be expected. you didn't actually comment on what I said. I mentioned API's nothing about directories and files. API being the keyword here. If there are good API's that applications can use, for example to access filesystems, then it will be easier and much more possible to introduce new things into the filesystem without breaking existing applications. One example of preventing application breakage could be to have a versioning system. Applications could query which version is supported, or vice versa. From what I understand it is much harder to do anything today because of the fixed ways. This is why we have this discussion. > [...] >> Application breakage has happened before between virtually every >> kernel change. It isn't new. Just take devfs and sysfs. > Oops, now emacs stopped working because devfs was added? gcc needed a mayor > overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_ > breakage, not all-around application breakage. Emacs seem to work fine with the file-as-dir on my Reiser4 partition. And so does actually any other application I have used on it. Be it in Gnome, KDE or in plain shell's. >> >> Look guys, in 1993 I anticipated the battle would be here, and I build >> >> the foundation for a defensive tower right at the spot MS and Apple are >> >> now maneuvering towards. >> >> > Why place a protective tower around the pit into which they are trying >> > desperately to throw themselves? ;-) >> >> It is easy to say this. But the fact is that users can actually use >> these new features for good things. > And can abuse them for bad things, and the features aren't free. Balance > benefits (very minor) against costs (quite large). Abuse depends on permissions and security policies which is your job as admin to look after. It would be possible to limit, for example), the types of plugins that a user can either use or load. Do you give your users access to modprobe or tools like that? >> >> Help me get the next level on the tower before >> >> they get here. It is one hell of a foundation, they won't be able to >> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> >> disk format plugins and all the other kinds of plugins, and you won't be >> >> missing any expressive power that you really want.... >> >> > Better write libraries handling whatever weird format you have in mind for >> > each use. Even applications. I don't expect to be able to use emacs to edit >> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >> > a poem. The current trend (which I heartily agree with) is to take stuff >> > _out_ of the kernel (for flexibility, access, development ease, even >> > performance). >> If I want to be able to quickly find files by a certain author I >> could search the meta-data fields (if they are used) with normal >> tools such as grep. > Iff the metadata is there. It won't (everybody would have to agree on > placing that on the files for it to work), so it is useless. No it is not useless. It just means the file has no meta information. Just as it is today. No change. >> Compression or encryption is another example. >> >> Would you rather patch every application there is for Linux to >> enable support for reading these files and directories? > No. But an encrypted or compressed filessytem doesn't need weird files. > Easiest on applications is just to serve the decrypted/uncompressed files > for use via the traditional APIs. Next easiest is provide programs that > handle cryptography and compression and can be combined with whatever > unaware program you care to use. Funny, that is exactly the way things work > now... So you can setup a directory that has some files compressed and some other encrypted and yet they will be available to use with tar, cp, ls, mc, Nautilus, Emacs etc etc? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 16:52 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-29 16:52 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Horst von Brand <vonbrand@inf.utfsm.cl> said: >> > Hans Reiser <reiser@namesys.com> said: >> >> Andrew Morton wrote: > [...] >> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> >> for Apple, >> >> > Says who? >> >> At least I as a user see the benefits largely. The workstation / >> desktop user has much more need for such features, than perhaps the >> normal Linux sysadmin. > Show benefits, show there are no downsides. Show that the same can't > possibly be done in userland. Then I'll consider buying it. I am not saying it couldn't be done in userland. Most things can. The problem with userland is that it doesn't provide generic access across most applications. For example GnomeVFS is limited to Gnome applications only. The same for KDE. to be able to coordinate application developer on anything is just almost impossible. Still. Why do you oppose plugins, streams and meta files? The could be valuable and easy to use tools for many purposes. One could be would be advanced ACLs defined as a meta-file using XML format. > AFAICS: Smallish benefits for some narrow application niche. Mayor problems > (no decent multiuser way of using it, extremely complex stuff placed in the > kernel). Userland solutions are all around us right now, and work just > fine. No, placing the same gunk in the kernel won't make it magically use > no time or memory, more like exactly the opposite. Besides, all the visions > waved around here have very little in common (except the idea that they > make all restrictions magically dissapear). > Get a list of required features, show concrete ways of using it, outline > possible implementations, discuss resource usage. I.e., do "engineering". > Then we have something to discuss. I am not sure what you refer to as "required features". Do you mean example of usage and contents for streams, plugins and meta files? >> >> and that means we need to put search engine and database >> >> functionality into the filesystem. >> > Please don't. Unix works and is extremely popular because it _doesn't_ try >> > to be smart (policy vs mechanism distinction, simple abstractions that can >> > be combined endlessly). If you add this to the filesystems, you either redo >> > _all_ userland to understand _one_ particular way of doing "smart things" >> > (which will turn out to be almost exactly the dumbest possible for some >> > uses, and then you are stuck), or get lots of shards from broken apps (and >> > users, and sysadmins). >> Indeed. But you forget one thing. Combining lots of small userland >> tools can actually make things more complex - at least for the user. > So? Placing lots of little things in kernel does the same, plus screws over > sysadmins, and makes kernel hacker's job harder. I prefer the former alone. Difference being that it could be transparent in a way otherwise not possible. >> That said, I do not see that anyone has suggested to remove this way >> of working with applications. > Excuse me, if files suddenly become directories, and you can't have some > "reserved" file/directory names, and... you are breaking lots of stuff. >> Are the new and old features mutually exclusive? > Yep. >> >> It takes 11 years of serious >> >> research to build a clean storage layer able to handle doing that. >> >> > Great! Build an experimental OS showing how to use it, and through that see >> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. >> There are many uses already. > Where? >> It could be encryption, compression, >> meta-data, file streams, sorting, searching and a multitude of uses >> and ideas that haven't been discovered yet. > "Could", "multitude ... haven't been discovered yet" is _not_ "many current > uses". >> >> Reiser4 has done that, finally. None of the other Linux filesystems >> >> have. >> > How come nobody before you in the almost 30 years of Unix has ever seen the >> > need for this indispensable feature? >> I, as a user, have for a long time. Do not drag everyone over the >> same edge. Users have different needs. We should try to be as >> forthcoming as possible to provide the tools and options for them. > Why? Breaking the basic paradigms for some user whim is not good OS design > in my book, sorry. Many "bright ideas" have been thrown out over the years > for just the same reasons. >> >> The next major release of ReiserFS is going to be bursting with >> >> semantic enhancements, because the prerequisites for them are in place >> >> now. None of the other Linux filesystems have those prerequisites. >> > To me that would mean that ReiserFS has no place in Linux. >> Why do you say this? I thought Linux embraced advancements and new >> technologies. > If they are right. I think breaking the base of the OS is not a good > idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where > this is supposedly so so badly needed?), but not here. You mean in Windows and MacOS where much of this functionality already exists? >> >> They won't be able to keep up with the semantic enhancements. This >> >> metafiles and file-directories stuff is actually fairly trivial stuff. >> > Maybe. But the question of it being of any use aren't trivially answered >> > "yes". My gut feeling is that the answer is a resounding "no", but that's >> > just me. Direcories are directories, if you want to ship directory-like >> > stuff around, use directories (or tarfiles, or whatever). Just look what >> > happened to the much, much easier stuff of splitting SUID/SGID into >> > capabilities: Nothing at all whatsoever, because they have no decent place >> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be >> > able to _define_ APIs to follow, but even so it isn't at all easy to do so. >> Exactly. The problem I see in this discussion is the lack of just >> this - APIs. Because the lack of good APIs we cannot make any larger >> changes or bring new features because old applications will break. I'll chop this up... > No. There is a complete lack of answers to why it should be done ("I think > it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from > enough) No perhaps not by itself. But it certainly should be considered. These two OS'es have most of the market share. They have great resources trying to make things that work for their users. Users are more important than you think. Linux and any OS can do what it wants, but if it wants to keep users then it has to listen to them. > , on what the alternatives are ("Sorry, it must be in the kernel > because it is impossible to get application writers to agree on how to do > it" This is a perfectly valid reason. If it is necessary to move something into the core of Linux because it couldn't otherwise be done then it should seriously be considered. > will just add another incompatible way of "doing almost the same", this > time in the kernel), People on this list are actually trying to make it to be fully compatible with old legacy applications. If you hadn't noticed, Reiser4 does work. The meta files do work. The discussion is wether parts of Reiser4 should be made generic and fit into the VFS. > and last but not least what the impact is (performance > wise, application breakage, user/sysadmin expectation breakage, > incompatibility with other systems, kernel development impact, to list a > few). Hash this out, and we can talk. You paint up a doomsday picture here. Nothing of this magnitude has happened and won't. besides, what is wrong if sysadmins would have to learn something new? >> I think that Linux should make new API that will completely separate >> applications from the storage media. This way we will not as easily >> break applications when new features are developed. This should be >> the main priority. > The "files + directories" stuff is a nice abstraction, and works today on > all random-access storage media. You don't care much if it is on a CD, a > floppy, a disk, a RAID array, or a flash card, do you? It won't work on > tapes, or printers; but that is to be expected. you didn't actually comment on what I said. I mentioned API's nothing about directories and files. API being the keyword here. If there are good API's that applications can use, for example to access filesystems, then it will be easier and much more possible to introduce new things into the filesystem without breaking existing applications. One example of preventing application breakage could be to have a versioning system. Applications could query which version is supported, or vice versa. From what I understand it is much harder to do anything today because of the fixed ways. This is why we have this discussion. > [...] >> Application breakage has happened before between virtually every >> kernel change. It isn't new. Just take devfs and sysfs. > Oops, now emacs stopped working because devfs was added? gcc needed a mayor > overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_ > breakage, not all-around application breakage. Emacs seem to work fine with the file-as-dir on my Reiser4 partition. And so does actually any other application I have used on it. Be it in Gnome, KDE or in plain shell's. >> >> Look guys, in 1993 I anticipated the battle would be here, and I build >> >> the foundation for a defensive tower right at the spot MS and Apple are >> >> now maneuvering towards. >> >> > Why place a protective tower around the pit into which they are trying >> > desperately to throw themselves? ;-) >> >> It is easy to say this. But the fact is that users can actually use >> these new features for good things. > And can abuse them for bad things, and the features aren't free. Balance > benefits (very minor) against costs (quite large). Abuse depends on permissions and security policies which is your job as admin to look after. It would be possible to limit, for example), the types of plugins that a user can either use or load. Do you give your users access to modprobe or tools like that? >> >> Help me get the next level on the tower before >> >> they get here. It is one hell of a foundation, they won't be able to >> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> >> disk format plugins and all the other kinds of plugins, and you won't be >> >> missing any expressive power that you really want.... >> >> > Better write libraries handling whatever weird format you have in mind for >> > each use. Even applications. I don't expect to be able to use emacs to edit >> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >> > a poem. The current trend (which I heartily agree with) is to take stuff >> > _out_ of the kernel (for flexibility, access, development ease, even >> > performance). >> If I want to be able to quickly find files by a certain author I >> could search the meta-data fields (if they are used) with normal >> tools such as grep. > Iff the metadata is there. It won't (everybody would have to agree on > placing that on the files for it to work), so it is useless. No it is not useless. It just means the file has no meta information. Just as it is today. No change. >> Compression or encryption is another example. >> >> Would you rather patch every application there is for Linux to >> enable support for reading these files and directories? > No. But an encrypted or compressed filessytem doesn't need weird files. > Easiest on applications is just to serve the decrypted/uncompressed files > for use via the traditional APIs. Next easiest is provide programs that > handle cryptography and compression and can be combined with whatever > unaware program you care to use. Funny, that is exactly the way things work > now... So you can setup a directory that has some files compressed and some other encrypted and yet they will be available to use with tar, cp, ls, mc, Nautilus, Emacs etc etc? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 16:52 ` Spam @ 2004-08-29 17:22 ` Nikita Danilov -1 siblings, 0 replies; 1835+ messages in thread From: Nikita Danilov @ 2004-08-29 17:22 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Hans Reiser, Helge Hafting, Rik van Riel, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list Spam writes: > > > Spam <spam@tnonline.net> said: > >> Horst von Brand <vonbrand@inf.utfsm.cl> said: [...] > > > Show benefits, show there are no downsides. Show that the same can't > > possibly be done in userland. Then I'll consider buying it. > > I am not saying it couldn't be done in userland. Most things can. > > The problem with userland is that it doesn't provide generic access > across most applications. For example GnomeVFS is limited to Gnome > applications only. The same for KDE. to be able to coordinate > application developer on anything is just almost impossible. Hmm... drag-and-drop also doesn't work constently over all applications. Let's put it into kernel. Seriously, kernel prformance is critical to the system, and to achieve high performance all kernel code runs in single address space (actually, in portion of address spaces of user processes shared by all processes). All shared system state is located there. This means that bug in a kernel affects whole system. This means that kernel should be kept as simple as possible (and a bit more simple). This is why only things that cannot be done efficiently in the user level are put into kernel. And political agendas of various camps of user-level developers change nothing here. > > Still. Why do you oppose plugins, streams and meta files? The could > be valuable and easy to use tools for many purposes. One could > be would be advanced ACLs defined as a meta-file using XML format. POSIX has standard ACL API in C (well, "eternal draft" only), one can use it to extract ACLs from kernel and convert it to any format to one's heart content. Advantage of this is that when XML goes out of fashion (I hope this wouldn't yet happen when you will read this message), kernel API will remain intact. > Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 17:22 ` Nikita Danilov 0 siblings, 0 replies; 1835+ messages in thread From: Nikita Danilov @ 2004-08-29 17:22 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Hans Reiser, Helge Hafting, Rik van Riel, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list Spam writes: > > > Spam <spam@tnonline.net> said: > >> Horst von Brand <vonbrand@inf.utfsm.cl> said: [...] > > > Show benefits, show there are no downsides. Show that the same can't > > possibly be done in userland. Then I'll consider buying it. > > I am not saying it couldn't be done in userland. Most things can. > > The problem with userland is that it doesn't provide generic access > across most applications. For example GnomeVFS is limited to Gnome > applications only. The same for KDE. to be able to coordinate > application developer on anything is just almost impossible. Hmm... drag-and-drop also doesn't work constently over all applications. Let's put it into kernel. Seriously, kernel prformance is critical to the system, and to achieve high performance all kernel code runs in single address space (actually, in portion of address spaces of user processes shared by all processes). All shared system state is located there. This means that bug in a kernel affects whole system. This means that kernel should be kept as simple as possible (and a bit more simple). This is why only things that cannot be done efficiently in the user level are put into kernel. And political agendas of various camps of user-level developers change nothing here. > > Still. Why do you oppose plugins, streams and meta files? The could > be valuable and easy to use tools for many purposes. One could > be would be advanced ACLs defined as a meta-file using XML format. POSIX has standard ACL API in C (well, "eternal draft" only), one can use it to extract ACLs from kernel and convert it to any format to one's heart content. Advantage of this is that when XML goes out of fashion (I hope this wouldn't yet happen when you will read this message), kernel API will remain intact. > Nikita. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 17:22 ` Nikita Danilov (?) @ 2004-08-29 18:12 ` Markus Törnqvist -1 siblings, 0 replies; 1835+ messages in thread From: Markus Törnqvist @ 2004-08-29 18:12 UTC (permalink / raw) To: Nikita Danilov Cc: Spam, Horst von Brand, Hans Reiser, Helge Hafting, Rik van Riel, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list [ This will probably go down in history as the most pointless message ever, I just told Nik on IRC I'd take it up on the list :) ] On Sun, Aug 29, 2004 at 09:22:08PM +0400, Nikita Danilov wrote: >Spam writes: >Hmm... drag-and-drop also doesn't work constently over all >applications. Let's put it into kernel. What I'm after is that the file system should be able to store any arbitrary metadata. No matter what the format is. Maybe I was a bit harsh on XML in my previous email, but whatever. >Seriously, kernel prformance is critical to the system, and to achieve >high performance all kernel code runs in single address space (actually, >in portion of address spaces of user processes shared by all >processes). All shared system state is located there. This means that >bug in a kernel affects whole system. This means that kernel should be >kept as simple as possible (and a bit more simple). I do agree here. But is it seriously such a big issue to implement the files-as-dirs kind of thing? Be it overloading MAY_READ so it works with chdir or implementing openat or whatever it takes. I don't think this should be much more, if any more, than the capability to store this data. Which Reiser4 does. But moved from the Reiser4 level to the VFS level. Right? >This is why only things that cannot be done efficiently in the user >level are put into kernel. And political agendas of various camps of >user-level developers change nothing here. What I'd want to do is to be able to write a Reiser4 plugin that suits my needs for something. Not that I'd even know how to, but that's a matter of time. Are there other means of doing this than Reiser? I mean Reiser4 plugin functionality by other means. >> Still. Why do you oppose plugins, streams and meta files? The could >> be valuable and easy to use tools for many purposes. One could >> be would be advanced ACLs defined as a meta-file using XML format. >POSIX has standard ACL API in C (well, "eternal draft" only), one can >use it to extract ACLs from kernel and convert it to any format to one's >heart content. Advantage of this is that when XML goes out of fashion (I >hope this wouldn't yet happen when you will read this message), kernel >API will remain intact. Aa, this apparently is an argument against XML parsing in the kernel, not the idea that if someone wanted ACLs, that are in XML format, and could not use them (because the kernel doesn't parse), he should be able to write such a plugin. Besides, let's say the Reiser4 XML-ACL plugin has a parser that actually DOES something. For argument's sake. Would it not be up to the individual user to disable this plugin? I'd assume it's possible to write plugins so that they do not affect anything but themselves, meaning this parser. Of course the problem of copying a file with XML-ACLs to another Reiser4 fs without support for them should be thought of. But I guess this is where you take the shortest route; the XML-ACL files are intact in the file's metadata, but totally unusable? -- mjt ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 [not found] ` <200408291521.i7TFLsQk028363@localhost.localdomain> @ 2004-08-29 16:53 ` Spam 2004-08-29 16:53 ` Spam 2004-08-29 16:54 ` Spam 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-29 16:53 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Horst von Brand <vonbrand@inf.utfsm.cl> said: >> > Hans Reiser <reiser@namesys.com> said: >> >> Andrew Morton wrote: > [...] >> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> >> for Apple, >> >> > Says who? >> >> At least I as a user see the benefits largely. The workstation / >> desktop user has much more need for such features, than perhaps the >> normal Linux sysadmin. > Show benefits, show there are no downsides. Show that the same can't > possibly be done in userland. Then I'll consider buying it. I am not saying it couldn't be done in userland. Most things can. The problem with userland is that it doesn't provide generic access across most applications. For example GnomeVFS is limited to Gnome applications only. The same for KDE. to be able to coordinate application developer on anything is just almost impossible. Still. Why do you oppose plugins, streams and meta files? The could be valuable and easy to use tools for many purposes. One could be would be advanced ACLs defined as a meta-file using XML format. > AFAICS: Smallish benefits for some narrow application niche. Mayor problems > (no decent multiuser way of using it, extremely complex stuff placed in the > kernel). Userland solutions are all around us right now, and work just > fine. No, placing the same gunk in the kernel won't make it magically use > no time or memory, more like exactly the opposite. Besides, all the visions > waved around here have very little in common (except the idea that they > make all restrictions magically dissapear). > Get a list of required features, show concrete ways of using it, outline > possible implementations, discuss resource usage. I.e., do "engineering". > Then we have something to discuss. I am not sure what you refer to as "required features". Do you mean example of usage and contents for streams, plugins and meta files? >> >> and that means we need to put search engine and database >> >> functionality into the filesystem. >> > Please don't. Unix works and is extremely popular because it _doesn't_ try >> > to be smart (policy vs mechanism distinction, simple abstractions that can >> > be combined endlessly). If you add this to the filesystems, you either redo >> > _all_ userland to understand _one_ particular way of doing "smart things" >> > (which will turn out to be almost exactly the dumbest possible for some >> > uses, and then you are stuck), or get lots of shards from broken apps (and >> > users, and sysadmins). >> Indeed. But you forget one thing. Combining lots of small userland >> tools can actually make things more complex - at least for the user. > So? Placing lots of little things in kernel does the same, plus screws over > sysadmins, and makes kernel hacker's job harder. I prefer the former alone. Difference being that it could be transparent in a way otherwise not possible. >> That said, I do not see that anyone has suggested to remove this way >> of working with applications. > Excuse me, if files suddenly become directories, and you can't have some > "reserved" file/directory names, and... you are breaking lots of stuff. >> Are the new and old features mutually exclusive? > Yep. >> >> It takes 11 years of serious >> >> research to build a clean storage layer able to handle doing that. >> >> > Great! Build an experimental OS showing how to use it, and through that see >> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. >> There are many uses already. > Where? >> It could be encryption, compression, >> meta-data, file streams, sorting, searching and a multitude of uses >> and ideas that haven't been discovered yet. > "Could", "multitude ... haven't been discovered yet" is _not_ "many current > uses". >> >> Reiser4 has done that, finally. None of the other Linux filesystems >> >> have. >> > How come nobody before you in the almost 30 years of Unix has ever seen the >> > need for this indispensable feature? >> I, as a user, have for a long time. Do not drag everyone over the >> same edge. Users have different needs. We should try to be as >> forthcoming as possible to provide the tools and options for them. > Why? Breaking the basic paradigms for some user whim is not good OS design > in my book, sorry. Many "bright ideas" have been thrown out over the years > for just the same reasons. >> >> The next major release of ReiserFS is going to be bursting with >> >> semantic enhancements, because the prerequisites for them are in place >> >> now. None of the other Linux filesystems have those prerequisites. >> > To me that would mean that ReiserFS has no place in Linux. >> Why do you say this? I thought Linux embraced advancements and new >> technologies. > If they are right. I think breaking the base of the OS is not a good > idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where > this is supposedly so so badly needed?), but not here. You mean in Windows and MacOS where much of this functionality already exists? >> >> They won't be able to keep up with the semantic enhancements. This >> >> metafiles and file-directories stuff is actually fairly trivial stuff. >> > Maybe. But the question of it being of any use aren't trivially answered >> > "yes". My gut feeling is that the answer is a resounding "no", but that's >> > just me. Direcories are directories, if you want to ship directory-like >> > stuff around, use directories (or tarfiles, or whatever). Just look what >> > happened to the much, much easier stuff of splitting SUID/SGID into >> > capabilities: Nothing at all whatsoever, because they have no decent place >> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be >> > able to _define_ APIs to follow, but even so it isn't at all easy to do so. >> Exactly. The problem I see in this discussion is the lack of just >> this - APIs. Because the lack of good APIs we cannot make any larger >> changes or bring new features because old applications will break. I'll chop this up... > No. There is a complete lack of answers to why it should be done ("I think > it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from > enough) No perhaps not by itself. But it certainly should be considered. These two OS'es have most of the market share. They have great resources trying to make things that work for their users. Users are more important than you think. Linux and any OS can do what it wants, but if it wants to keep users then it has to listen to them. > , on what the alternatives are ("Sorry, it must be in the kernel > because it is impossible to get application writers to agree on how to do > it" This is a perfectly valid reason. If it is necessary to move something into the core of Linux because it couldn't otherwise be done then it should seriously be considered. > will just add another incompatible way of "doing almost the same", this > time in the kernel), People on this list are actually trying to make it to be fully compatible with old legacy applications. If you hadn't noticed, Reiser4 does work. The meta files do work. The discussion is wether parts of Reiser4 should be made generic and fit into the VFS. > and last but not least what the impact is (performance > wise, application breakage, user/sysadmin expectation breakage, > incompatibility with other systems, kernel development impact, to list a > few). Hash this out, and we can talk. You paint up a doomsday picture here. Nothing of this magnitude has happened and won't. besides, what is wrong if sysadmins would have to learn something new? >> I think that Linux should make new API that will completely separate >> applications from the storage media. This way we will not as easily >> break applications when new features are developed. This should be >> the main priority. > The "files + directories" stuff is a nice abstraction, and works today on > all random-access storage media. You don't care much if it is on a CD, a > floppy, a disk, a RAID array, or a flash card, do you? It won't work on > tapes, or printers; but that is to be expected. you didn't actually comment on what I said. I mentioned API's nothing about directories and files. API being the keyword here. If there are good API's that applications can use, for example to access filesystems, then it will be easier and much more possible to introduce new things into the filesystem without breaking existing applications. One example of preventing application breakage could be to have a versioning system. Applications could query which version is supported, or vice versa. From what I understand it is much harder to do anything today because of the fixed ways. This is why we have this discussion. > [...] >> Application breakage has happened before between virtually every >> kernel change. It isn't new. Just take devfs and sysfs. > Oops, now emacs stopped working because devfs was added? gcc needed a mayor > overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_ > breakage, not all-around application breakage. Emacs seem to work fine with the file-as-dir on my Reiser4 partition. And so does actually any other application I have used on it. Be it in Gnome, KDE or in plain shell's. >> >> Look guys, in 1993 I anticipated the battle would be here, and I build >> >> the foundation for a defensive tower right at the spot MS and Apple are >> >> now maneuvering towards. >> >> > Why place a protective tower around the pit into which they are trying >> > desperately to throw themselves? ;-) >> >> It is easy to say this. But the fact is that users can actually use >> these new features for good things. > And can abuse them for bad things, and the features aren't free. Balance > benefits (very minor) against costs (quite large). Abuse depends on permissions and security policies which is your job as admin to look after. It would be possible to limit, for example), the types of plugins that a user can either use or load. Do you give your users access to modprobe or tools like that? >> >> Help me get the next level on the tower before >> >> they get here. It is one hell of a foundation, they won't be able to >> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> >> disk format plugins and all the other kinds of plugins, and you won't be >> >> missing any expressive power that you really want.... >> >> > Better write libraries handling whatever weird format you have in mind for >> > each use. Even applications. I don't expect to be able to use emacs to edit >> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >> > a poem. The current trend (which I heartily agree with) is to take stuff >> > _out_ of the kernel (for flexibility, access, development ease, even >> > performance). >> If I want to be able to quickly find files by a certain author I >> could search the meta-data fields (if they are used) with normal >> tools such as grep. > Iff the metadata is there. It won't (everybody would have to agree on > placing that on the files for it to work), so it is useless. No it is not useless. It just means the file has no meta information. Just as it is today. No change. >> Compression or encryption is another example. >> >> Would you rather patch every application there is for Linux to >> enable support for reading these files and directories? > No. But an encrypted or compressed filessytem doesn't need weird files. > Easiest on applications is just to serve the decrypted/uncompressed files > for use via the traditional APIs. Next easiest is provide programs that > handle cryptography and compression and can be combined with whatever > unaware program you care to use. Funny, that is exactly the way things work > now... So you can setup a directory that has some files compressed and some other encrypted and yet they will be available to use with tar, cp, ls, mc, Nautilus, Emacs etc etc? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 16:53 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-29 16:53 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Horst von Brand <vonbrand@inf.utfsm.cl> said: >> > Hans Reiser <reiser@namesys.com> said: >> >> Andrew Morton wrote: > [...] >> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> >> for Apple, >> >> > Says who? >> >> At least I as a user see the benefits largely. The workstation / >> desktop user has much more need for such features, than perhaps the >> normal Linux sysadmin. > Show benefits, show there are no downsides. Show that the same can't > possibly be done in userland. Then I'll consider buying it. I am not saying it couldn't be done in userland. Most things can. The problem with userland is that it doesn't provide generic access across most applications. For example GnomeVFS is limited to Gnome applications only. The same for KDE. to be able to coordinate application developer on anything is just almost impossible. Still. Why do you oppose plugins, streams and meta files? The could be valuable and easy to use tools for many purposes. One could be would be advanced ACLs defined as a meta-file using XML format. > AFAICS: Smallish benefits for some narrow application niche. Mayor problems > (no decent multiuser way of using it, extremely complex stuff placed in the > kernel). Userland solutions are all around us right now, and work just > fine. No, placing the same gunk in the kernel won't make it magically use > no time or memory, more like exactly the opposite. Besides, all the visions > waved around here have very little in common (except the idea that they > make all restrictions magically dissapear). > Get a list of required features, show concrete ways of using it, outline > possible implementations, discuss resource usage. I.e., do "engineering". > Then we have something to discuss. I am not sure what you refer to as "required features". Do you mean example of usage and contents for streams, plugins and meta files? >> >> and that means we need to put search engine and database >> >> functionality into the filesystem. >> > Please don't. Unix works and is extremely popular because it _doesn't_ try >> > to be smart (policy vs mechanism distinction, simple abstractions that can >> > be combined endlessly). If you add this to the filesystems, you either redo >> > _all_ userland to understand _one_ particular way of doing "smart things" >> > (which will turn out to be almost exactly the dumbest possible for some >> > uses, and then you are stuck), or get lots of shards from broken apps (and >> > users, and sysadmins). >> Indeed. But you forget one thing. Combining lots of small userland >> tools can actually make things more complex - at least for the user. > So? Placing lots of little things in kernel does the same, plus screws over > sysadmins, and makes kernel hacker's job harder. I prefer the former alone. Difference being that it could be transparent in a way otherwise not possible. >> That said, I do not see that anyone has suggested to remove this way >> of working with applications. > Excuse me, if files suddenly become directories, and you can't have some > "reserved" file/directory names, and... you are breaking lots of stuff. >> Are the new and old features mutually exclusive? > Yep. >> >> It takes 11 years of serious >> >> research to build a clean storage layer able to handle doing that. >> >> > Great! Build an experimental OS showing how to use it, and through that see >> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. >> There are many uses already. > Where? >> It could be encryption, compression, >> meta-data, file streams, sorting, searching and a multitude of uses >> and ideas that haven't been discovered yet. > "Could", "multitude ... haven't been discovered yet" is _not_ "many current > uses". >> >> Reiser4 has done that, finally. None of the other Linux filesystems >> >> have. >> > How come nobody before you in the almost 30 years of Unix has ever seen the >> > need for this indispensable feature? >> I, as a user, have for a long time. Do not drag everyone over the >> same edge. Users have different needs. We should try to be as >> forthcoming as possible to provide the tools and options for them. > Why? Breaking the basic paradigms for some user whim is not good OS design > in my book, sorry. Many "bright ideas" have been thrown out over the years > for just the same reasons. >> >> The next major release of ReiserFS is going to be bursting with >> >> semantic enhancements, because the prerequisites for them are in place >> >> now. None of the other Linux filesystems have those prerequisites. >> > To me that would mean that ReiserFS has no place in Linux. >> Why do you say this? I thought Linux embraced advancements and new >> technologies. > If they are right. I think breaking the base of the OS is not a good > idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where > this is supposedly so so badly needed?), but not here. You mean in Windows and MacOS where much of this functionality already exists? >> >> They won't be able to keep up with the semantic enhancements. This >> >> metafiles and file-directories stuff is actually fairly trivial stuff. >> > Maybe. But the question of it being of any use aren't trivially answered >> > "yes". My gut feeling is that the answer is a resounding "no", but that's >> > just me. Direcories are directories, if you want to ship directory-like >> > stuff around, use directories (or tarfiles, or whatever). Just look what >> > happened to the much, much easier stuff of splitting SUID/SGID into >> > capabilities: Nothing at all whatsoever, because they have no decent place >> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be >> > able to _define_ APIs to follow, but even so it isn't at all easy to do so. >> Exactly. The problem I see in this discussion is the lack of just >> this - APIs. Because the lack of good APIs we cannot make any larger >> changes or bring new features because old applications will break. I'll chop this up... > No. There is a complete lack of answers to why it should be done ("I think > it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from > enough) No perhaps not by itself. But it certainly should be considered. These two OS'es have most of the market share. They have great resources trying to make things that work for their users. Users are more important than you think. Linux and any OS can do what it wants, but if it wants to keep users then it has to listen to them. > , on what the alternatives are ("Sorry, it must be in the kernel > because it is impossible to get application writers to agree on how to do > it" This is a perfectly valid reason. If it is necessary to move something into the core of Linux because it couldn't otherwise be done then it should seriously be considered. > will just add another incompatible way of "doing almost the same", this > time in the kernel), People on this list are actually trying to make it to be fully compatible with old legacy applications. If you hadn't noticed, Reiser4 does work. The meta files do work. The discussion is wether parts of Reiser4 should be made generic and fit into the VFS. > and last but not least what the impact is (performance > wise, application breakage, user/sysadmin expectation breakage, > incompatibility with other systems, kernel development impact, to list a > few). Hash this out, and we can talk. You paint up a doomsday picture here. Nothing of this magnitude has happened and won't. besides, what is wrong if sysadmins would have to learn something new? >> I think that Linux should make new API that will completely separate >> applications from the storage media. This way we will not as easily >> break applications when new features are developed. This should be >> the main priority. > The "files + directories" stuff is a nice abstraction, and works today on > all random-access storage media. You don't care much if it is on a CD, a > floppy, a disk, a RAID array, or a flash card, do you? It won't work on > tapes, or printers; but that is to be expected. you didn't actually comment on what I said. I mentioned API's nothing about directories and files. API being the keyword here. If there are good API's that applications can use, for example to access filesystems, then it will be easier and much more possible to introduce new things into the filesystem without breaking existing applications. One example of preventing application breakage could be to have a versioning system. Applications could query which version is supported, or vice versa. From what I understand it is much harder to do anything today because of the fixed ways. This is why we have this discussion. > [...] >> Application breakage has happened before between virtually every >> kernel change. It isn't new. Just take devfs and sysfs. > Oops, now emacs stopped working because devfs was added? gcc needed a mayor > overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_ > breakage, not all-around application breakage. Emacs seem to work fine with the file-as-dir on my Reiser4 partition. And so does actually any other application I have used on it. Be it in Gnome, KDE or in plain shell's. >> >> Look guys, in 1993 I anticipated the battle would be here, and I build >> >> the foundation for a defensive tower right at the spot MS and Apple are >> >> now maneuvering towards. >> >> > Why place a protective tower around the pit into which they are trying >> > desperately to throw themselves? ;-) >> >> It is easy to say this. But the fact is that users can actually use >> these new features for good things. > And can abuse them for bad things, and the features aren't free. Balance > benefits (very minor) against costs (quite large). Abuse depends on permissions and security policies which is your job as admin to look after. It would be possible to limit, for example), the types of plugins that a user can either use or load. Do you give your users access to modprobe or tools like that? >> >> Help me get the next level on the tower before >> >> they get here. It is one hell of a foundation, they won't be able to >> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> >> disk format plugins and all the other kinds of plugins, and you won't be >> >> missing any expressive power that you really want.... >> >> > Better write libraries handling whatever weird format you have in mind for >> > each use. Even applications. I don't expect to be able to use emacs to edit >> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >> > a poem. The current trend (which I heartily agree with) is to take stuff >> > _out_ of the kernel (for flexibility, access, development ease, even >> > performance). >> If I want to be able to quickly find files by a certain author I >> could search the meta-data fields (if they are used) with normal >> tools such as grep. > Iff the metadata is there. It won't (everybody would have to agree on > placing that on the files for it to work), so it is useless. No it is not useless. It just means the file has no meta information. Just as it is today. No change. >> Compression or encryption is another example. >> >> Would you rather patch every application there is for Linux to >> enable support for reading these files and directories? > No. But an encrypted or compressed filessytem doesn't need weird files. > Easiest on applications is just to serve the decrypted/uncompressed files > for use via the traditional APIs. Next easiest is provide programs that > handle cryptography and compression and can be combined with whatever > unaware program you care to use. Funny, that is exactly the way things work > now... So you can setup a directory that has some files compressed and some other encrypted and yet they will be available to use with tar, cp, ls, mc, Nautilus, Emacs etc etc? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 [not found] ` <200408291521.i7TFLsQk028363@localhost.localdomain> @ 2004-08-29 16:54 ` Spam 2004-08-29 16:53 ` Spam 2004-08-29 16:54 ` Spam 2 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-29 16:54 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Horst von Brand <vonbrand@inf.utfsm.cl> said: >> > Hans Reiser <reiser@namesys.com> said: >> >> Andrew Morton wrote: > [...] >> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> >> for Apple, >> >> > Says who? >> >> At least I as a user see the benefits largely. The workstation / >> desktop user has much more need for such features, than perhaps the >> normal Linux sysadmin. > Show benefits, show there are no downsides. Show that the same can't > possibly be done in userland. Then I'll consider buying it. I am not saying it couldn't be done in userland. Most things can. The problem with userland is that it doesn't provide generic access across most applications. For example GnomeVFS is limited to Gnome applications only. The same for KDE. to be able to coordinate application developer on anything is just almost impossible. Still. Why do you oppose plugins, streams and meta files? The could be valuable and easy to use tools for many purposes. One could be would be advanced ACLs defined as a meta-file using XML format. > AFAICS: Smallish benefits for some narrow application niche. Mayor problems > (no decent multiuser way of using it, extremely complex stuff placed in the > kernel). Userland solutions are all around us right now, and work just > fine. No, placing the same gunk in the kernel won't make it magically use > no time or memory, more like exactly the opposite. Besides, all the visions > waved around here have very little in common (except the idea that they > make all restrictions magically dissapear). > Get a list of required features, show concrete ways of using it, outline > possible implementations, discuss resource usage. I.e., do "engineering". > Then we have something to discuss. I am not sure what you refer to as "required features". Do you mean example of usage and contents for streams, plugins and meta files? >> >> and that means we need to put search engine and database >> >> functionality into the filesystem. >> > Please don't. Unix works and is extremely popular because it _doesn't_ try >> > to be smart (policy vs mechanism distinction, simple abstractions that can >> > be combined endlessly). If you add this to the filesystems, you either redo >> > _all_ userland to understand _one_ particular way of doing "smart things" >> > (which will turn out to be almost exactly the dumbest possible for some >> > uses, and then you are stuck), or get lots of shards from broken apps (and >> > users, and sysadmins). >> Indeed. But you forget one thing. Combining lots of small userland >> tools can actually make things more complex - at least for the user. > So? Placing lots of little things in kernel does the same, plus screws over > sysadmins, and makes kernel hacker's job harder. I prefer the former alone. Difference being that it could be transparent in a way otherwise not possible. >> That said, I do not see that anyone has suggested to remove this way >> of working with applications. > Excuse me, if files suddenly become directories, and you can't have some > "reserved" file/directory names, and... you are breaking lots of stuff. >> Are the new and old features mutually exclusive? > Yep. >> >> It takes 11 years of serious >> >> research to build a clean storage layer able to handle doing that. >> >> > Great! Build an experimental OS showing how to use it, and through that see >> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. >> There are many uses already. > Where? >> It could be encryption, compression, >> meta-data, file streams, sorting, searching and a multitude of uses >> and ideas that haven't been discovered yet. > "Could", "multitude ... haven't been discovered yet" is _not_ "many current > uses". >> >> Reiser4 has done that, finally. None of the other Linux filesystems >> >> have. >> > How come nobody before you in the almost 30 years of Unix has ever seen the >> > need for this indispensable feature? >> I, as a user, have for a long time. Do not drag everyone over the >> same edge. Users have different needs. We should try to be as >> forthcoming as possible to provide the tools and options for them. > Why? Breaking the basic paradigms for some user whim is not good OS design > in my book, sorry. Many "bright ideas" have been thrown out over the years > for just the same reasons. >> >> The next major release of ReiserFS is going to be bursting with >> >> semantic enhancements, because the prerequisites for them are in place >> >> now. None of the other Linux filesystems have those prerequisites. >> > To me that would mean that ReiserFS has no place in Linux. >> Why do you say this? I thought Linux embraced advancements and new >> technologies. > If they are right. I think breaking the base of the OS is not a good > idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where > this is supposedly so so badly needed?), but not here. You mean in Windows and MacOS where much of this functionality already exists? >> >> They won't be able to keep up with the semantic enhancements. This >> >> metafiles and file-directories stuff is actually fairly trivial stuff. >> > Maybe. But the question of it being of any use aren't trivially answered >> > "yes". My gut feeling is that the answer is a resounding "no", but that's >> > just me. Direcories are directories, if you want to ship directory-like >> > stuff around, use directories (or tarfiles, or whatever). Just look what >> > happened to the much, much easier stuff of splitting SUID/SGID into >> > capabilities: Nothing at all whatsoever, because they have no decent place >> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be >> > able to _define_ APIs to follow, but even so it isn't at all easy to do so. >> Exactly. The problem I see in this discussion is the lack of just >> this - APIs. Because the lack of good APIs we cannot make any larger >> changes or bring new features because old applications will break. I'll chop this up... > No. There is a complete lack of answers to why it should be done ("I think > it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from > enough) No perhaps not by itself. But it certainly should be considered. These two OS'es have most of the market share. They have great resources trying to make things that work for their users. Users are more important than you think. Linux and any OS can do what it wants, but if it wants to keep users then it has to listen to them. > , on what the alternatives are ("Sorry, it must be in the kernel > because it is impossible to get application writers to agree on how to do > it" This is a perfectly valid reason. If it is necessary to move something into the core of Linux because it couldn't otherwise be done then it should seriously be considered. > will just add another incompatible way of "doing almost the same", this > time in the kernel), People on this list are actually trying to make it to be fully compatible with old legacy applications. If you hadn't noticed, Reiser4 does work. The meta files do work. The discussion is wether parts of Reiser4 should be made generic and fit into the VFS. > and last but not least what the impact is (performance > wise, application breakage, user/sysadmin expectation breakage, > incompatibility with other systems, kernel development impact, to list a > few). Hash this out, and we can talk. You paint up a doomsday picture here. Nothing of this magnitude has happened and won't. besides, what is wrong if sysadmins would have to learn something new? >> I think that Linux should make new API that will completely separate >> applications from the storage media. This way we will not as easily >> break applications when new features are developed. This should be >> the main priority. > The "files + directories" stuff is a nice abstraction, and works today on > all random-access storage media. You don't care much if it is on a CD, a > floppy, a disk, a RAID array, or a flash card, do you? It won't work on > tapes, or printers; but that is to be expected. you didn't actually comment on what I said. I mentioned API's nothing about directories and files. API being the keyword here. If there are good API's that applications can use, for example to access filesystems, then it will be easier and much more possible to introduce new things into the filesystem without breaking existing applications. One example of preventing application breakage could be to have a versioning system. Applications could query which version is supported, or vice versa. From what I understand it is much harder to do anything today because of the fixed ways. This is why we have this discussion. > [...] >> Application breakage has happened before between virtually every >> kernel change. It isn't new. Just take devfs and sysfs. > Oops, now emacs stopped working because devfs was added? gcc needed a mayor > overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_ > breakage, not all-around application breakage. Emacs seem to work fine with the file-as-dir on my Reiser4 partition. And so does actually any other application I have used on it. Be it in Gnome, KDE or in plain shell's. >> >> Look guys, in 1993 I anticipated the battle would be here, and I build >> >> the foundation for a defensive tower right at the spot MS and Apple are >> >> now maneuvering towards. >> >> > Why place a protective tower around the pit into which they are trying >> > desperately to throw themselves? ;-) >> >> It is easy to say this. But the fact is that users can actually use >> these new features for good things. > And can abuse them for bad things, and the features aren't free. Balance > benefits (very minor) against costs (quite large). Abuse depends on permissions and security policies which is your job as admin to look after. It would be possible to limit, for example), the types of plugins that a user can either use or load. Do you give your users access to modprobe or tools like that? >> >> Help me get the next level on the tower before >> >> they get here. It is one hell of a foundation, they won't be able to >> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> >> disk format plugins and all the other kinds of plugins, and you won't be >> >> missing any expressive power that you really want.... >> >> > Better write libraries handling whatever weird format you have in mind for >> > each use. Even applications. I don't expect to be able to use emacs to edit >> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >> > a poem. The current trend (which I heartily agree with) is to take stuff >> > _out_ of the kernel (for flexibility, access, development ease, even >> > performance). >> If I want to be able to quickly find files by a certain author I >> could search the meta-data fields (if they are used) with normal >> tools such as grep. > Iff the metadata is there. It won't (everybody would have to agree on > placing that on the files for it to work), so it is useless. No it is not useless. It just means the file has no meta information. Just as it is today. No change. >> Compression or encryption is another example. >> >> Would you rather patch every application there is for Linux to >> enable support for reading these files and directories? > No. But an encrypted or compressed filessytem doesn't need weird files. > Easiest on applications is just to serve the decrypted/uncompressed files > for use via the traditional APIs. Next easiest is provide programs that > handle cryptography and compression and can be combined with whatever > unaware program you care to use. Funny, that is exactly the way things work > now... So you can setup a directory that has some files compressed and some other encrypted and yet they will be available to use with tar, cp, ls, mc, Nautilus, Emacs etc etc? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-29 16:54 ` Spam 0 siblings, 0 replies; 1835+ messages in thread From: Spam @ 2004-08-29 16:54 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Horst von Brand <vonbrand@inf.utfsm.cl> said: >> > Hans Reiser <reiser@namesys.com> said: >> >> Andrew Morton wrote: > [...] >> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> >> for Apple, >> >> > Says who? >> >> At least I as a user see the benefits largely. The workstation / >> desktop user has much more need for such features, than perhaps the >> normal Linux sysadmin. > Show benefits, show there are no downsides. Show that the same can't > possibly be done in userland. Then I'll consider buying it. I am not saying it couldn't be done in userland. Most things can. The problem with userland is that it doesn't provide generic access across most applications. For example GnomeVFS is limited to Gnome applications only. The same for KDE. to be able to coordinate application developer on anything is just almost impossible. Still. Why do you oppose plugins, streams and meta files? The could be valuable and easy to use tools for many purposes. One could be would be advanced ACLs defined as a meta-file using XML format. > AFAICS: Smallish benefits for some narrow application niche. Mayor problems > (no decent multiuser way of using it, extremely complex stuff placed in the > kernel). Userland solutions are all around us right now, and work just > fine. No, placing the same gunk in the kernel won't make it magically use > no time or memory, more like exactly the opposite. Besides, all the visions > waved around here have very little in common (except the idea that they > make all restrictions magically dissapear). > Get a list of required features, show concrete ways of using it, outline > possible implementations, discuss resource usage. I.e., do "engineering". > Then we have something to discuss. I am not sure what you refer to as "required features". Do you mean example of usage and contents for streams, plugins and meta files? >> >> and that means we need to put search engine and database >> >> functionality into the filesystem. >> > Please don't. Unix works and is extremely popular because it _doesn't_ try >> > to be smart (policy vs mechanism distinction, simple abstractions that can >> > be combined endlessly). If you add this to the filesystems, you either redo >> > _all_ userland to understand _one_ particular way of doing "smart things" >> > (which will turn out to be almost exactly the dumbest possible for some >> > uses, and then you are stuck), or get lots of shards from broken apps (and >> > users, and sysadmins). >> Indeed. But you forget one thing. Combining lots of small userland >> tools can actually make things more complex - at least for the user. > So? Placing lots of little things in kernel does the same, plus screws over > sysadmins, and makes kernel hacker's job harder. I prefer the former alone. Difference being that it could be transparent in a way otherwise not possible. >> That said, I do not see that anyone has suggested to remove this way >> of working with applications. > Excuse me, if files suddenly become directories, and you can't have some > "reserved" file/directory names, and... you are breaking lots of stuff. >> Are the new and old features mutually exclusive? > Yep. >> >> It takes 11 years of serious >> >> research to build a clean storage layer able to handle doing that. >> >> > Great! Build an experimental OS showing how to use it, and through that see >> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. >> There are many uses already. > Where? >> It could be encryption, compression, >> meta-data, file streams, sorting, searching and a multitude of uses >> and ideas that haven't been discovered yet. > "Could", "multitude ... haven't been discovered yet" is _not_ "many current > uses". >> >> Reiser4 has done that, finally. None of the other Linux filesystems >> >> have. >> > How come nobody before you in the almost 30 years of Unix has ever seen the >> > need for this indispensable feature? >> I, as a user, have for a long time. Do not drag everyone over the >> same edge. Users have different needs. We should try to be as >> forthcoming as possible to provide the tools and options for them. > Why? Breaking the basic paradigms for some user whim is not good OS design > in my book, sorry. Many "bright ideas" have been thrown out over the years > for just the same reasons. >> >> The next major release of ReiserFS is going to be bursting with >> >> semantic enhancements, because the prerequisites for them are in place >> >> now. None of the other Linux filesystems have those prerequisites. >> > To me that would mean that ReiserFS has no place in Linux. >> Why do you say this? I thought Linux embraced advancements and new >> technologies. > If they are right. I think breaking the base of the OS is not a good > idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where > this is supposedly so so badly needed?), but not here. You mean in Windows and MacOS where much of this functionality already exists? >> >> They won't be able to keep up with the semantic enhancements. This >> >> metafiles and file-directories stuff is actually fairly trivial stuff. >> > Maybe. But the question of it being of any use aren't trivially answered >> > "yes". My gut feeling is that the answer is a resounding "no", but that's >> > just me. Direcories are directories, if you want to ship directory-like >> > stuff around, use directories (or tarfiles, or whatever). Just look what >> > happened to the much, much easier stuff of splitting SUID/SGID into >> > capabilities: Nothing at all whatsoever, because they have no decent place >> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be >> > able to _define_ APIs to follow, but even so it isn't at all easy to do so. >> Exactly. The problem I see in this discussion is the lack of just >> this - APIs. Because the lack of good APIs we cannot make any larger >> changes or bring new features because old applications will break. I'll chop this up... > No. There is a complete lack of answers to why it should be done ("I think > it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from > enough) No perhaps not by itself. But it certainly should be considered. These two OS'es have most of the market share. They have great resources trying to make things that work for their users. Users are more important than you think. Linux and any OS can do what it wants, but if it wants to keep users then it has to listen to them. > , on what the alternatives are ("Sorry, it must be in the kernel > because it is impossible to get application writers to agree on how to do > it" This is a perfectly valid reason. If it is necessary to move something into the core of Linux because it couldn't otherwise be done then it should seriously be considered. > will just add another incompatible way of "doing almost the same", this > time in the kernel), People on this list are actually trying to make it to be fully compatible with old legacy applications. If you hadn't noticed, Reiser4 does work. The meta files do work. The discussion is wether parts of Reiser4 should be made generic and fit into the VFS. > and last but not least what the impact is (performance > wise, application breakage, user/sysadmin expectation breakage, > incompatibility with other systems, kernel development impact, to list a > few). Hash this out, and we can talk. You paint up a doomsday picture here. Nothing of this magnitude has happened and won't. besides, what is wrong if sysadmins would have to learn something new? >> I think that Linux should make new API that will completely separate >> applications from the storage media. This way we will not as easily >> break applications when new features are developed. This should be >> the main priority. > The "files + directories" stuff is a nice abstraction, and works today on > all random-access storage media. You don't care much if it is on a CD, a > floppy, a disk, a RAID array, or a flash card, do you? It won't work on > tapes, or printers; but that is to be expected. you didn't actually comment on what I said. I mentioned API's nothing about directories and files. API being the keyword here. If there are good API's that applications can use, for example to access filesystems, then it will be easier and much more possible to introduce new things into the filesystem without breaking existing applications. One example of preventing application breakage could be to have a versioning system. Applications could query which version is supported, or vice versa. From what I understand it is much harder to do anything today because of the fixed ways. This is why we have this discussion. > [...] >> Application breakage has happened before between virtually every >> kernel change. It isn't new. Just take devfs and sysfs. > Oops, now emacs stopped working because devfs was added? gcc needed a mayor > overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_ > breakage, not all-around application breakage. Emacs seem to work fine with the file-as-dir on my Reiser4 partition. And so does actually any other application I have used on it. Be it in Gnome, KDE or in plain shell's. >> >> Look guys, in 1993 I anticipated the battle would be here, and I build >> >> the foundation for a defensive tower right at the spot MS and Apple are >> >> now maneuvering towards. >> >> > Why place a protective tower around the pit into which they are trying >> > desperately to throw themselves? ;-) >> >> It is easy to say this. But the fact is that users can actually use >> these new features for good things. > And can abuse them for bad things, and the features aren't free. Balance > benefits (very minor) against costs (quite large). Abuse depends on permissions and security policies which is your job as admin to look after. It would be possible to limit, for example), the types of plugins that a user can either use or load. Do you give your users access to modprobe or tools like that? >> >> Help me get the next level on the tower before >> >> they get here. It is one hell of a foundation, they won't be able to >> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> >> disk format plugins and all the other kinds of plugins, and you won't be >> >> missing any expressive power that you really want.... >> >> > Better write libraries handling whatever weird format you have in mind for >> > each use. Even applications. I don't expect to be able to use emacs to edit >> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >> > a poem. The current trend (which I heartily agree with) is to take stuff >> > _out_ of the kernel (for flexibility, access, development ease, even >> > performance). >> If I want to be able to quickly find files by a certain author I >> could search the meta-data fields (if they are used) with normal >> tools such as grep. > Iff the metadata is there. It won't (everybody would have to agree on > placing that on the files for it to work), so it is useless. No it is not useless. It just means the file has no meta information. Just as it is today. No change. >> Compression or encryption is another example. >> >> Would you rather patch every application there is for Linux to >> enable support for reading these files and directories? > No. But an encrypted or compressed filessytem doesn't need weird files. > Easiest on applications is just to serve the decrypted/uncompressed files > for use via the traditional APIs. Next easiest is provide programs that > handle cryptography and compression and can be combined with whatever > unaware program you care to use. Funny, that is exactly the way things work > now... So you can setup a directory that has some files compressed and some other encrypted and yet they will be available to use with tar, cp, ls, mc, Nautilus, Emacs etc etc? ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand 2004-08-27 20:32 ` Markus Törnqvist 2004-08-27 20:44 ` Spam @ 2004-09-01 21:43 ` Jamie Lokier 2 siblings, 0 replies; 1835+ messages in thread From: Jamie Lokier @ 2004-09-01 21:43 UTC (permalink / raw) To: Horst von Brand; +Cc: linux-kernel, linux-fsdevel, reiserfs-list Horst von Brand wrote: > Build an experimental OS showing how to use it, and through that see > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. Hey, if you can think of a way to build an experimental OS and get it used in real, day-to-day, down-to-earth, practice - i.e. used by a diverse audience of thousands of vocal groups doing different things with it, please let me know. Until you do we have to lobby for some things to be tried in Linux before we know if they really work. -- Jamie ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser ` (4 preceding siblings ...) 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand @ 2004-08-26 20:04 ` Martin J. Bligh 2004-08-26 21:18 ` Hubert Chan 2004-08-27 21:23 ` vfs2 (was Re: silent semantic changes with reiser4) Pavel Machek 6 siblings, 1 reply; 1835+ messages in thread From: Martin J. Bligh @ 2004-08-26 20:04 UTC (permalink / raw) To: Hans Reiser, Andrew Morton Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > Don't move reiser4 into vfs, use reiser4 as the vfs. Ah ... that seems to be a good insight into what you're driving for. That would seem to be good for you ... and bad for other filesystems. So whilst I can see why you'd want it - but I don't think it's a good idea. Competition is good. Andrew wrote: > And what are the licensing implications of plugins? Are they derived > works? Must they be GPL'ed? You replied: >>And what are the licensing implications of plugins? Are they derived >>works? > > Yes. I find it odd that you cut out the last part of his question. So must they be GPL'ed or not? M. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:04 ` Martin J. Bligh @ 2004-08-26 21:18 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-26 21:18 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Martin" == Martin J Bligh <mbligh@aracnet.com> writes: Martin> Andrew wrote: > And what are the licensing implications of plugins? Are they derived >> works? Must they be GPL'ed? Martin> [Hans] replied: >>> And what are the licensing implications of plugins? Are they >>> derived works? >> >> Yes. Martin> I find it odd that you cut out the last part of his question. So Martin> must they be GPL'ed or not? If they are derived works of GPL'ed software, then they obviously must be GPL'ed (or else they are illegal). But if the authors obtain an alternate license from Hans/Namesys, then they are bound by the terms of that license. (Of course, this case would most likely apply only to Reiser4 running under a different operating system.) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* Re: silent semantic changes with reiser4 @ 2004-08-26 21:18 ` Hubert Chan 0 siblings, 0 replies; 1835+ messages in thread From: Hubert Chan @ 2004-08-26 21:18 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel >>>>> "Martin" == Martin J Bligh <mbligh@aracnet.com> writes: Martin> Andrew wrote: > And what are the licensing implications of plugins? Are they derived >> works? Must they be GPL'ed? Martin> [Hans] replied: >>> And what are the licensing implications of plugins? Are they >>> derived works? >> >> Yes. Martin> I find it odd that you cut out the last part of his question. So Martin> must they be GPL'ed or not? If they are derived works of GPL'ed software, then they obviously must be GPL'ed (or else they are illegal). But if the authors obtain an alternate license from Hans/Namesys, then they are bound by the terms of that license. (Of course, this case would most likely apply only to Reiser4 running under a different operating system.) -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1835+ messages in thread
* vfs2 (was Re: silent semantic changes with reiser4) 2004-08-26 8:31 ` Hans Reiser ` (5 preceding siblings ...) 2004-08-26 20:04 ` Martin J. Bligh @ 2004-08-27 21:23 ` Pavel Machek 6 siblings, 0 replies; 1835+ messages in thread From: Pavel Machek @ 2004-08-27 21:23 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hi! > >Yes, but I didn't say "flame Christoph and ignore the issues" ;) > > > > > Oh....;-) ... > enhancements. This metafiles and file-directories stuff is actually > fairly trivial stuff. Now I'm scared. not as powerful. Don't > move reiser4 into vfs, use reiser4 as the vfs. Don't write > filesystems, write file plugins and disk format plugins and all the > other kinds of plugins, and you won't be missing any expressive power > that you really want.... So you want to replace vfs. Fine, but you should have told us before. Also vfs replacement probably should not be hidden in fs/reiserfs4... -- 64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms ^ permalink raw reply [flat|nested] 1835+ messages in thread
end of thread, other threads:[~2004-09-03 3:54 UTC | newest] Thread overview: 1835+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2004-08-24 20:25 silent semantic changes with reiser4 Christoph Hellwig 2004-08-24 20:35 ` Lee Revell 2004-08-24 20:38 ` Christoph Hellwig 2004-08-24 20:42 ` Lee Revell 2004-08-24 21:18 ` Jamie Lokier 2004-08-24 20:38 ` Jeff Garzik 2004-08-24 20:53 ` viro 2004-08-24 21:22 ` viro 2004-08-25 18:28 ` Hans Reiser 2004-08-25 18:45 ` Christoph Hellwig 2004-08-26 9:02 ` Hans Reiser 2004-08-26 18:38 ` William Lee Irwin III 2004-08-26 18:49 ` Linus Torvalds 2004-08-26 18:56 ` William Lee Irwin III 2004-08-27 8:14 ` Hans Reiser 2004-08-27 10:10 ` Giuseppe Bilotta 2004-08-27 15:25 ` William Lee Irwin III [not found] ` <20040827161539.GC28876@khan.acc.umu.se> 2004-08-27 16:18 ` William Lee Irwin III 2004-08-25 19:53 ` Hans Reiser 2004-08-25 19:53 ` Hans Reiser 2004-08-25 20:06 ` Matthew Wilcox 2004-08-26 8:41 ` Hans Reiser 2004-08-26 8:41 ` Hans Reiser 2004-08-25 20:08 ` Christoph Hellwig 2004-08-25 20:19 ` Christoph Hellwig 2004-08-25 20:24 ` Linus Torvalds 2004-08-25 20:25 ` Christoph Hellwig 2004-08-25 20:59 ` viro 2004-08-26 8:43 ` Hans Reiser 2004-08-26 8:43 ` Hans Reiser 2004-08-26 8:42 ` Hans Reiser 2004-08-26 8:42 ` Hans Reiser 2004-08-26 9:24 ` Christoph Hellwig 2004-08-26 18:45 ` Hans Reiser 2004-08-26 18:45 ` Hans Reiser 2004-08-26 13:38 ` Chris Mason 2004-08-26 23:53 ` Hans Reiser 2004-08-26 23:53 ` Hans Reiser 2004-08-27 9:16 ` Christoph Hellwig 2004-08-25 20:22 ` Linus Torvalds 2004-08-25 20:35 ` Christoph Hellwig 2004-08-25 20:41 ` Hans Reiser 2004-08-25 20:41 ` Hans Reiser 2004-08-25 20:51 ` Chris Mason 2004-08-25 20:58 ` Markus Törnqvist 2004-08-25 20:58 ` mjt 2004-08-25 20:58 ` Markus Törnqvist 2004-08-25 21:03 ` Rik van Riel 2004-08-25 21:03 ` Rik van Riel 2004-08-26 9:00 ` Hans Reiser 2004-08-26 9:00 ` Hans Reiser 2004-08-25 20:42 ` viro 2004-08-25 21:00 ` Christophe Saout 2004-08-25 21:00 ` Christophe Saout 2004-08-25 22:59 ` Andrea Arcangeli 2004-08-26 8:35 ` Hans Reiser 2004-08-26 8:35 ` Hans Reiser 2004-08-26 11:28 ` Andrea Arcangeli 2004-08-26 11:44 ` Christophe Saout 2004-08-26 11:44 ` Christophe Saout 2004-08-26 12:16 ` Andrea Arcangeli 2004-08-26 12:23 ` Christophe Saout 2004-08-26 12:23 ` Christophe Saout 2004-08-27 18:38 ` Alex Zarochentsev 2004-08-28 6:17 ` Hans Reiser 2004-08-26 23:53 ` Hans Reiser 2004-08-26 23:53 ` Hans Reiser 2004-08-27 0:18 ` Andrea Arcangeli 2004-08-28 10:49 ` Yury Umanets 2004-08-28 11:11 ` Yury Umanets 2004-08-28 11:11 ` Yury Umanets 2004-08-25 21:00 ` Linus Torvalds 2004-08-25 21:25 ` viro 2004-08-26 0:11 ` Jamie Lokier 2004-08-26 0:30 ` viro 2004-08-26 1:00 ` Jamie Lokier 2004-08-26 3:13 ` viro 2004-08-26 8:49 ` Hans Reiser 2004-08-26 8:49 ` Hans Reiser 2004-08-26 14:05 ` Adrian Bunk 2004-08-26 14:25 ` Christophe Saout 2004-08-26 14:45 ` Matthew Wilcox 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:12 ` Christophe Saout 2004-08-26 16:00 ` Jamie Lokier 2004-08-27 2:17 ` Hans Reiser 2004-08-26 15:41 ` Nikita Danilov 2004-08-29 12:34 ` Alex Zarochentsev 2004-08-29 15:52 ` Shaya Potter 2004-09-01 22:05 ` Jamie Lokier 2004-08-26 15:02 ` Jamie Lokier 2004-08-28 23:14 ` Horst von Brand 2004-09-01 20:08 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Jamie Lokier 2004-09-01 20:50 ` Linus Torvalds 2004-09-02 0:24 ` Jamie Lokier 2004-09-02 1:07 ` Linus Torvalds 2004-09-02 2:30 ` Why do we need openat? (was: The argument for fs assistance in handling archives) Hubert Chan 2004-09-02 2:30 ` Hubert Chan 2004-09-02 3:46 ` The argument for fs assistance in handling archives David Masover 2004-09-02 3:35 ` David Masover 2004-09-02 3:38 ` Linus Torvalds 2004-09-02 4:27 ` David Masover 2004-09-02 6:04 ` Linus Torvalds 2004-09-02 7:15 ` Hans Reiser 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 8:58 ` Hans Reiser 2004-09-02 9:12 ` Oliver Hunt 2004-09-02 9:26 ` Helge Hafting 2004-09-02 9:26 ` Helge Hafting 2004-09-02 9:45 ` Oliver Hunt 2004-09-02 10:02 ` Spam 2004-09-02 10:02 ` Spam 2004-09-02 11:00 ` Giuliano Pochini 2004-09-02 14:38 ` Christian Mayrhuber 2004-09-02 9:52 ` Spam 2004-09-02 9:52 ` Spam 2004-09-02 9:32 ` Alan Cox 2004-09-02 10:27 ` V13 2004-09-02 14:35 ` John Stoffel 2004-09-02 18:42 ` Hubert Chan 2004-09-02 18:42 ` Hubert Chan 2004-09-02 11:09 ` Oliver Neukum 2004-09-02 11:09 ` Oliver Neukum 2004-09-02 11:11 ` Spam 2004-09-02 11:11 ` Spam 2004-09-02 15:40 ` Horst von Brand 2004-09-02 18:21 ` Lee Revell 2004-09-02 18:21 ` Lee Revell 2004-09-02 10:05 ` Bostjan Skufca @ domenca.com 2004-09-02 14:24 ` John Stoffel 2004-09-02 15:36 ` Alan Cox 2004-09-02 15:32 ` Horst von Brand 2004-09-02 17:43 ` Linus Torvalds 2004-09-02 18:22 ` Martin J. Bligh 2004-09-02 18:38 ` Christer Weinigel 2004-09-02 19:02 ` Christian Iversen 2004-09-02 19:52 ` Steve Bergman 2004-09-02 19:52 ` Steve Bergman 2004-09-02 20:06 ` Spam 2004-09-02 20:06 ` Spam 2004-09-02 20:16 ` Martin J. Bligh 2004-09-02 20:35 ` Erik Hensema 2004-09-02 9:04 ` Hans Reiser 2004-09-02 12:50 ` James Bruce 2004-09-02 17:50 ` Linus Torvalds 2004-09-02 7:55 ` Robin Rosenberg 2004-09-02 5:53 ` Hans Reiser 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 16:11 ` Jamie Lokier 2004-09-02 17:41 ` Dave Kleikamp 2004-09-02 17:46 ` Linus Torvalds 2004-09-02 17:50 ` Christoph Hellwig 2004-09-02 18:03 ` Linus Torvalds 2004-09-02 19:04 ` Charles P. Wright 2004-09-02 19:42 ` Linus Torvalds 2004-09-02 20:05 ` Charles P. Wright 2004-09-02 20:30 ` Erez Zadok 2004-09-02 18:46 ` Alan Cox 2004-09-02 20:22 ` Linus Torvalds 2004-09-02 19:50 ` Spam 2004-09-02 19:50 ` Spam 2004-09-02 20:38 ` Frank van Maarseveen 2004-09-02 21:36 ` Dave Kleikamp 2004-09-02 14:06 ` Horst von Brand 2004-09-02 17:32 ` Jamie Lokier 2004-09-02 18:23 ` Christer Weinigel 2004-08-26 18:35 ` silent semantic changes with reiser4 Joel Becker 2004-08-26 19:53 ` Jamie Lokier 2004-08-27 9:19 ` Markus Törnqvist 2004-08-27 9:19 ` mjt 2004-08-27 9:19 ` Markus Törnqvist 2004-08-27 12:26 ` Matthew Wilcox 2004-08-27 12:26 ` Matthew Wilcox 2004-08-26 10:05 ` Chris Wedgwood 2004-08-26 11:02 ` Jamie Lokier 2004-08-26 11:19 ` Spam 2004-08-26 11:19 ` Spam 2004-08-28 0:28 ` Horst von Brand 2004-08-26 15:50 ` Will Dyson 2004-08-26 20:38 ` Lee Revell 2004-08-26 20:38 ` Lee Revell 2004-08-26 20:50 ` Christophe Saout 2004-08-26 20:50 ` Christophe Saout 2004-08-26 20:57 ` Lee Revell 2004-08-26 20:57 ` Lee Revell 2004-08-27 9:21 ` Markus Törnqvist 2004-08-27 9:21 ` mjt 2004-08-27 9:21 ` Markus Törnqvist 2004-08-27 11:34 ` Spam 2004-08-27 11:34 ` Spam 2004-08-26 21:00 ` Jamie Lokier 2004-08-27 21:06 ` Pavel Machek 2004-08-31 1:00 ` David Masover 2004-08-31 8:21 ` Pavel Machek 2004-08-31 18:28 ` Hubert Chan 2004-08-31 19:31 ` Horst von Brand 2004-08-31 20:05 ` Linus Torvalds 2004-08-31 20:29 ` Spam 2004-08-31 20:29 ` Spam 2004-08-31 20:32 ` Pavel Machek 2004-08-31 20:38 ` Linus Torvalds 2004-08-31 20:54 ` Pavel Machek 2004-08-31 20:59 ` Linus Torvalds 2004-08-31 22:07 ` Pavel Machek 2004-08-31 23:19 ` Christer Weinigel 2004-09-01 1:43 ` David Masover 2004-09-01 5:02 ` Pavel Machek 2004-08-31 21:07 ` Hubert Chan 2004-08-31 21:07 ` Hubert Chan 2004-08-31 23:02 ` Christer Weinigel 2004-08-31 23:20 ` Spam 2004-08-31 23:20 ` Spam 2004-09-01 0:00 ` John Newbigin 2004-09-01 12:10 ` Alan Cox 2004-09-01 20:21 ` Hubert Chan 2004-09-01 19:44 ` Markus Törnqvist 2004-09-01 1:14 ` David Masover 2004-09-01 1:14 ` David Masover 2004-09-01 4:59 ` Pavel Machek 2004-09-01 5:23 ` Hubert Chan 2004-09-01 16:14 ` Jamie Lokier 2004-09-01 20:18 ` Pavel Machek 2004-09-01 21:59 ` Jamie Lokier 2004-09-01 22:51 ` Lee Revell 2004-09-01 22:51 ` Lee Revell 2004-09-02 14:25 ` Horst von Brand 2004-09-02 17:47 ` Lee Revell 2004-09-02 17:47 ` Lee Revell 2004-09-02 19:41 ` Spam 2004-09-02 19:41 ` Spam 2004-09-02 18:55 ` Alan Cox 2004-09-02 20:07 ` Spam 2004-09-02 20:07 ` Spam 2004-09-02 19:17 ` Alan Cox 2004-09-02 20:28 ` Spam 2004-09-02 20:28 ` Spam 2004-09-02 20:42 ` Pavel Machek 2004-09-02 19:49 ` Pavel Machek 2004-09-02 20:01 ` Lee Revell 2004-09-02 20:01 ` Lee Revell 2004-09-02 19:16 ` Alan Cox 2004-09-02 20:04 ` Chris Wedgwood 2004-09-02 20:10 ` Spam 2004-09-02 20:10 ` Spam 2004-09-02 20:43 ` Pavel Machek 2004-09-02 20:47 ` Lee Revell 2004-09-02 20:47 ` Lee Revell 2004-09-02 20:49 ` Chris Wedgwood 2004-09-02 20:57 ` Lee Revell 2004-09-02 20:57 ` Lee Revell 2004-09-02 20:58 ` Chris Wedgwood 2004-09-02 20:58 ` Pavel Machek 2004-09-01 5:50 ` Hans Reiser 2004-09-01 6:06 ` Linus Torvalds 2004-09-01 6:32 ` Hubert Chan 2004-09-01 7:22 ` Hans Reiser 2004-09-01 7:22 ` Hans Reiser 2004-09-01 11:51 ` Alexander G. M. Smith 2004-09-01 23:04 ` Hubert Chan 2004-09-01 11:49 ` Alan Cox 2004-09-01 1:10 ` David Masover 2004-09-01 1:10 ` David Masover 2004-08-27 21:01 ` Pavel Machek 2004-08-26 1:13 ` Chris Wright 2004-08-29 9:55 ` Mikaël Cluseau 2004-08-26 8:43 ` Hans Reiser 2004-08-26 8:43 ` Hans Reiser 2004-08-25 21:52 ` Matt Mackall 2004-08-25 22:21 ` Linus Torvalds 2004-08-26 0:18 ` Mikulas Patocka 2004-08-26 0:27 ` Linus Torvalds 2004-08-26 0:51 ` Mikulas Patocka 2004-08-26 8:36 ` Hans Reiser 2004-08-26 8:36 ` Hans Reiser 2004-08-26 10:53 ` Jamie Lokier 2004-08-26 10:59 ` Christoph Hellwig 2004-08-26 11:07 ` Jamie Lokier 2004-08-26 11:17 ` Spam 2004-08-26 11:17 ` Spam 2004-08-26 0:57 ` Rik van Riel 2004-08-26 0:57 ` Rik van Riel 2004-08-26 7:51 ` Helge Hafting 2004-08-26 9:21 ` Paul Jackson 2004-08-26 10:47 ` Jamie Lokier 2004-08-26 11:19 ` Paul Jackson 2004-08-27 12:33 ` Helge Hafting 2004-08-27 12:33 ` Helge Hafting 2004-08-26 8:40 ` Hans Reiser 2004-08-26 8:40 ` Hans Reiser 2004-08-26 14:46 ` Lars Marowsky-Bree 2004-08-26 9:44 ` Paul Jackson 2004-08-26 8:43 ` Hans Reiser 2004-08-26 8:43 ` Hans Reiser 2004-08-26 13:53 ` Michael Halcrow 2004-08-26 22:26 ` Hans Reiser 2004-08-26 22:26 ` Hans Reiser 2004-08-29 14:20 ` Alan Cox 2004-08-25 20:35 ` Alex Zarochentsev 2004-08-25 20:51 ` Christoph Hellwig 2004-08-25 23:54 ` Jamie Lokier 2004-08-26 8:44 ` Hans Reiser 2004-08-26 8:44 ` Hans Reiser 2004-08-26 12:43 ` Christoph Hellwig 2004-08-26 8:43 ` Hans Reiser 2004-08-26 8:43 ` Hans Reiser 2004-08-26 12:41 ` Christoph Hellwig 2004-08-26 13:48 ` Jamie Lokier 2004-08-26 15:57 ` Christoph Hellwig 2004-08-26 16:06 ` Jamie Lokier 2004-08-26 16:13 ` Christoph Hellwig 2004-08-26 16:35 ` Linus Torvalds 2004-08-26 17:20 ` Jamie Lokier 2004-08-26 17:29 ` Linus Torvalds 2004-08-26 17:54 ` viro 2004-08-26 18:16 ` Jamie Lokier 2004-08-26 18:27 ` Linus Torvalds 2004-08-26 19:40 ` Jamie Lokier 2004-08-30 13:25 ` Alex Zarochentsev 2004-08-27 9:18 ` Giuliano Pochini 2004-08-27 10:33 ` Giuseppe Bilotta 2004-08-27 10:33 ` Giuseppe Bilotta 2004-08-27 18:48 ` Hubert Chan 2004-08-27 18:48 ` Hubert Chan 2004-08-28 16:33 ` Giuseppe Bilotta 2004-08-28 16:33 ` Giuseppe Bilotta 2004-08-28 17:27 ` Hubert Chan 2004-08-28 17:27 ` Hubert Chan 2004-08-30 14:10 ` Alex Zarochentsev 2004-08-29 18:43 ` Alex Zarochentsev 2004-08-25 20:20 ` Jeremy Allison 2004-08-26 8:42 ` Hans Reiser 2004-08-26 8:42 ` Hans Reiser 2004-08-26 13:27 ` Rik van Riel 2004-08-26 13:27 ` Rik van Riel 2004-08-26 13:56 ` Markus Törnqvist 2004-08-26 19:58 ` Paul Jackson 2004-08-26 19:58 ` Paul Jackson 2004-08-26 23:53 ` Hans Reiser 2004-08-26 23:53 ` Hans Reiser 2004-08-25 20:22 ` Chris Mason 2004-08-26 8:42 ` Hans Reiser 2004-08-26 8:42 ` Hans Reiser 2004-08-26 9:36 ` Christoph Hellwig 2004-08-25 20:23 ` Chris Friesen 2004-08-25 22:28 ` Andrew Morton 2004-08-25 22:51 ` Spam 2004-08-25 22:51 ` Spam 2004-08-25 22:51 ` Christoph Hellwig 2004-08-25 22:59 ` Linus Torvalds 2004-08-25 23:19 ` Spam 2004-08-25 23:19 ` Spam 2004-08-25 23:32 ` Andrew Morton 2004-08-25 23:37 ` Jeremy Allison 2004-08-25 23:46 ` Wichert Akkerman 2004-08-26 0:01 ` Tierra 2004-08-26 0:05 ` Wichert Akkerman [not found] ` <c7ef32a70408251732442bff16@mail.gmail.com> 2004-08-26 0:33 ` Tierra 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 1:03 ` Jamie Lokier 2004-08-26 1:26 ` Nicholas Miell 2004-08-26 1:53 ` Jamie Lokier 2004-08-26 2:02 ` Nicholas Miell 2004-08-26 8:41 ` Hans Reiser 2004-08-26 8:41 ` Hans Reiser 2004-08-26 19:06 ` Giuseppe Bilotta 2004-08-26 19:06 ` Giuseppe Bilotta 2004-08-26 1:22 ` Jeremy Allison 2004-08-26 11:01 ` Wichert Akkerman [not found] ` <c7ef32a70408260942c8efa54@mail.gmail.com> 2004-08-26 16:44 ` Tierra 2004-08-26 4:44 ` Matt Mackall 2004-08-26 5:09 ` Nicholas Miell 2004-08-26 5:17 ` James Morris 2004-08-26 5:17 ` James Morris 2004-08-26 5:32 ` Matt Mackall 2004-08-26 7:34 ` Denis Vlasenko 2004-08-26 11:42 ` Rik van Riel 2004-08-26 11:42 ` Rik van Riel 2004-08-26 12:00 ` Christophe Saout 2004-08-26 12:00 ` Christophe Saout 2004-08-28 11:00 ` Yury Umanets 2004-08-28 11:00 ` Yury Umanets 2004-08-28 11:12 ` Yury Umanets 2004-08-28 11:12 ` Yury Umanets 2004-08-26 12:12 ` Jamie Lokier 2004-08-26 12:25 ` Wichert Akkerman 2004-08-26 20:35 ` Lee Revell 2004-08-26 20:35 ` Lee Revell 2004-08-27 0:02 ` Hans Reiser 2004-08-26 7:53 ` Markus Törnqvist 2004-08-26 7:53 ` mjt 2004-08-26 7:53 ` Markus Törnqvist 2004-08-26 16:32 ` Jan Harkes 2004-08-26 16:48 ` Linus Torvalds 2004-08-26 18:07 ` Jan Harkes 2004-08-31 3:39 ` Tom Vier 2004-08-31 4:08 ` Linus Torvalds 2004-08-31 4:38 ` Jeff Garzik 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 2004-08-31 5:56 ` Jeff Garzik 2004-08-31 6:50 ` Bill Huey [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> 2004-08-31 20:52 ` Bill Huey 2004-09-02 17:20 ` Diego Calleja 2004-09-02 21:37 ` Bill Huey 2004-08-31 21:23 ` Bill Huey 2004-08-31 9:01 ` Tonnerre 2004-08-31 11:15 ` Userspace file systems Luca Ferroni 2004-08-31 20:53 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen 2004-08-31 19:12 ` Jeff Garzik 2004-09-01 15:37 ` Clay Haapala 2004-08-31 10:57 ` Alan Cox 2004-08-31 17:15 ` Linus Torvalds 2004-08-31 21:13 ` Frank van Maarseveen 2004-08-31 21:25 ` Linus Torvalds 2004-08-31 19:45 ` Toon van der Pas 2004-08-26 17:14 ` Hubert Chan 2004-08-26 17:14 ` Hubert Chan 2004-08-26 8:31 ` Helge Hafting 2004-08-26 8:31 ` Helge Hafting 2004-08-26 18:17 ` Horst von Brand 2004-08-28 16:49 ` Helge Hafting 2004-08-26 9:59 ` Spam 2004-08-26 9:59 ` Spam 2004-08-26 13:05 ` Joseph Fannin 2004-08-26 6:53 ` Paul Jackson 2004-08-26 9:54 ` Spam 2004-08-26 9:54 ` Spam 2004-08-26 5:23 ` Jamie Lokier 2004-08-26 8:16 ` Helge Hafting 2004-08-26 8:16 ` Helge Hafting 2004-08-26 9:51 ` Spam 2004-08-26 9:51 ` Spam 2004-08-26 17:59 ` Shawn Rutledge 2004-08-26 18:17 ` Valdis.Kletnieks 2004-08-27 8:06 ` Hans Reiser 2004-08-26 9:40 ` Spam 2004-08-26 9:40 ` Spam 2004-08-26 9:49 ` Andrew Morton 2004-08-26 10:20 ` Spam 2004-08-26 10:20 ` Spam 2004-08-26 10:24 ` Andrew Morton 2004-08-26 10:51 ` Spam 2004-08-26 10:51 ` Spam 2004-08-26 10:55 ` Andrew Morton 2004-08-26 11:15 ` Spam 2004-08-26 11:15 ` Spam 2004-08-26 11:20 ` Andrew Morton 2004-08-26 11:29 ` Wichert Akkerman 2004-08-28 0:24 ` Horst von Brand 2004-08-26 11:27 ` Wichert Akkerman 2004-08-26 13:39 ` Rik van Riel 2004-08-26 13:39 ` Rik van Riel 2004-08-26 13:59 ` Spam 2004-08-26 13:59 ` Spam 2004-08-26 14:39 ` Felipe Alfaro Solana 2004-08-27 23:58 ` Horst von Brand 2004-08-29 15:17 ` Markus Törnqvist 2004-08-29 16:18 ` Spam 2004-08-29 16:18 ` Spam 2004-08-29 19:10 ` Tonnerre 2004-08-30 0:39 ` Alexander G. M. Smith 2004-08-30 18:39 ` Spam 2004-08-30 18:39 ` Spam 2004-08-30 18:39 ` Spam 2004-08-31 8:15 ` Tonnerre 2004-09-01 20:16 ` Jamie Lokier 2004-09-01 20:28 ` Dave Kleikamp 2004-09-02 3:42 ` Hubert Chan 2004-09-02 3:42 ` Hubert Chan 2004-09-02 3:52 ` David Masover 2004-09-02 5:57 ` Jamie Lokier 2004-09-02 9:50 ` Alan Cox 2004-09-02 20:25 ` Jamie Lokier 2004-09-02 18:16 ` Christer Weinigel 2004-09-02 19:58 ` Jamie Lokier 2004-08-26 11:48 ` Jamie Lokier 2004-08-26 16:07 ` James Bruce 2004-08-26 16:58 ` Jamie Lokier 2004-08-26 21:34 ` Felipe Alfaro Solana 2004-08-26 23:52 ` Hans Reiser 2004-08-26 23:52 ` Hans Reiser 2004-08-26 11:11 ` Bernd Petrovitsch 2004-08-26 23:59 ` Hans Reiser 2004-08-26 23:59 ` Hans Reiser 2004-08-26 10:54 ` Wichert Akkerman 2004-08-26 23:48 ` Hans Reiser 2004-08-26 23:48 ` Hans Reiser 2004-08-26 11:16 ` Christer Weinigel 2004-08-26 11:16 ` Christer Weinigel 2004-08-26 11:35 ` Spam 2004-08-26 11:35 ` Spam 2004-08-26 11:33 ` Wichert Akkerman 2004-08-26 11:43 ` Spam 2004-08-26 11:43 ` Spam 2004-08-27 7:41 ` Bernd Petrovitsch 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:51 ` Christoph Hellwig 2004-08-27 8:49 ` Christoph Hellwig 2004-08-27 9:03 ` Bernd Petrovitsch 2004-08-26 23:56 ` Hans Reiser 2004-08-26 23:56 ` Hans Reiser 2004-08-27 1:05 ` Andreas Dilger 2004-08-27 0:00 ` Stelian Ionescu 2004-08-26 11:58 ` Jamie Lokier 2004-08-26 14:00 ` Denis Vlasenko 2004-08-26 14:00 ` Denis Vlasenko 2004-08-26 14:12 ` Rik van Riel 2004-08-26 14:12 ` Rik van Riel 2004-08-26 14:24 ` Christophe Saout 2004-08-26 15:44 ` Jamie Lokier 2004-08-26 15:54 ` Rik van Riel 2004-08-26 15:54 ` Rik van Riel 2004-08-26 16:04 ` Christophe Saout 2004-08-26 16:04 ` Christophe Saout 2004-08-26 16:07 ` Christoph Hellwig 2004-08-26 16:44 ` Jamie Lokier 2004-08-26 18:40 ` David Masover 2004-08-26 18:40 ` David Masover 2004-08-26 21:17 ` David Lang 2004-08-26 21:05 ` David Lang 2004-08-26 21:19 ` Jamie Lokier 2004-08-26 21:31 ` Linus Torvalds 2004-08-26 22:30 ` Felipe Alfaro Solana 2004-08-26 22:32 ` Chris Dukes 2004-08-26 22:52 ` David Lang 2004-08-26 16:53 ` Jamie Lokier 2004-08-26 19:32 ` John Stoffel 2004-08-26 19:32 ` John Stoffel 2004-08-26 19:51 ` Denis Vlasenko 2004-08-26 20:13 ` Jamie Lokier 2004-08-30 17:37 ` Alex Zarochentsev 2004-08-30 18:51 ` Hans Reiser 2004-08-30 18:51 ` Hans Reiser 2004-08-26 17:05 ` Diego Calleja 2004-08-26 17:16 ` Rik van Riel 2004-08-26 17:16 ` Rik van Riel 2004-08-26 17:20 ` Linus Torvalds 2004-08-26 17:27 ` Jamie Lokier 2004-08-26 17:32 ` Linus Torvalds 2004-08-26 19:09 ` Jamie Lokier 2004-08-26 17:57 ` Rik van Riel 2004-08-26 17:57 ` Rik van Riel 2004-08-26 18:15 ` Linus Torvalds 2004-08-26 18:41 ` Rik van Riel 2004-08-26 18:41 ` Rik van Riel 2004-08-26 18:55 ` Linus Torvalds 2004-08-26 18:59 ` Rik van Riel 2004-08-26 18:59 ` Rik van Riel 2004-08-26 19:14 ` Christophe Saout 2004-08-26 19:14 ` Christophe Saout 2004-08-26 19:23 ` Linus Torvalds 2004-08-26 19:44 ` Rik van Riel 2004-08-26 19:44 ` Rik van Riel 2004-08-26 19:47 ` Christophe Saout 2004-08-26 20:10 ` Rik van Riel 2004-08-26 20:10 ` Rik van Riel 2004-08-26 20:17 ` Christophe Saout 2004-08-26 20:21 ` Jamie Lokier 2004-08-26 20:35 ` Stephen Wille Padnos 2004-08-31 11:01 ` Claus Färber 2004-08-31 11:01 ` Claus Färber 2004-08-31 11:01 ` Claus Färber 2004-09-01 15:13 ` Alan Cox 2004-09-01 15:13 ` Alan Cox 2004-08-26 20:40 ` Linus Torvalds 2004-08-26 22:53 ` Hans Reiser 2004-08-26 22:53 ` Hans Reiser 2004-08-26 23:27 ` Linus Torvalds 2004-08-26 23:47 ` Hans Reiser 2004-08-26 23:47 ` Hans Reiser 2004-08-27 2:03 ` viro 2004-08-27 8:10 ` Hans Reiser 2004-08-27 8:10 ` Hans Reiser 2004-08-26 20:48 ` Jamie Lokier 2004-08-26 20:52 ` Jeremy Allison 2004-08-27 9:19 ` Christoph Hellwig 2004-08-28 3:56 ` Jeremy Allison 2004-08-28 19:18 ` Brad Boyer 2004-08-26 22:20 ` Felipe Alfaro Solana 2004-08-29 0:04 ` Horst von Brand 2004-09-01 22:45 ` Jamie Lokier 2004-09-02 5:23 ` Olaf Hering 2004-09-02 5:23 ` Olaf Hering 2004-09-02 5:23 ` Olaf Hering 2004-08-27 8:06 ` Hans Reiser 2004-08-27 8:06 ` Hans Reiser 2004-08-26 19:29 ` Jamie Lokier 2004-08-26 19:42 ` Christophe Saout 2004-08-26 19:42 ` Christophe Saout 2004-08-27 8:46 ` Herbert Poetzl 2004-08-26 19:02 ` David Masover 2004-08-27 8:06 ` Hans Reiser 2004-08-27 8:06 ` Hans Reiser 2004-08-27 14:12 ` Rik van Riel 2004-08-27 14:12 ` Rik van Riel 2004-08-27 18:27 ` Hans Reiser 2004-08-27 18:44 ` Lee Revell 2004-08-27 19:13 ` Rik van Riel 2004-08-27 19:13 ` Rik van Riel 2004-08-28 6:10 ` Hans Reiser 2004-08-27 20:56 ` Christian Mayrhuber 2004-08-27 14:45 ` Rik van Riel 2004-08-27 14:45 ` Rik van Riel 2004-08-27 15:15 ` Spam 2004-08-27 15:15 ` Spam 2004-08-27 15:15 ` Spam 2004-08-27 15:41 ` [SPAM] " Jamie Lokier 2004-08-27 17:59 ` Hendrik Visage 2004-08-28 0:19 ` Spam 2004-08-28 0:19 ` Spam 2004-08-28 0:19 ` Spam 2004-08-28 1:59 ` Rik van Riel 2004-08-28 1:59 ` Rik van Riel 2004-08-28 2:05 ` Linus Torvalds 2004-08-28 9:53 ` Hans Reiser 2004-08-28 9:53 ` Hans Reiser 2004-08-28 11:21 ` Spam 2004-08-28 11:21 ` Spam 2004-08-28 17:19 ` Hubert Chan 2004-08-28 17:19 ` Hubert Chan 2004-08-28 17:05 ` Helge Hafting 2004-08-28 17:05 ` Helge Hafting 2004-08-28 18:09 ` Linus Torvalds 2004-08-28 18:20 ` Rik van Riel 2004-08-28 18:20 ` Rik van Riel 2004-08-28 18:29 ` viro 2004-08-28 18:44 ` Linus Torvalds 2004-08-28 18:56 ` viro 2004-08-28 19:16 ` Linus Torvalds 2004-08-28 19:41 ` Andries Brouwer 2004-08-28 19:46 ` Linus Torvalds 2004-08-29 13:56 ` Andries Brouwer 2004-08-28 19:45 ` viro 2004-08-29 5:12 ` Linus Torvalds 2004-08-29 5:44 ` Jeremy Allison 2004-08-29 0:46 ` Paul Jakma 2004-08-28 22:29 ` Hans Reiser 2004-08-28 22:29 ` Hans Reiser 2004-08-29 0:40 ` The Necessity of File Types (was silent semantic changes with reiser4) Alexander G. M. Smith 2004-08-31 0:42 ` David Masover 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds 2004-08-29 9:36 ` Hans Reiser 2004-08-29 10:44 ` CaT 2004-08-29 21:05 ` Hubert Chan 2004-08-29 21:05 ` Hubert Chan 2004-08-30 6:31 ` Markus Törnqvist 2004-08-29 17:57 ` Linus Torvalds 2004-08-29 19:53 ` Hans Reiser 2004-08-29 20:14 ` Nicholas Miell 2004-08-29 20:14 ` Nicholas Miell 2004-08-29 21:35 ` Hubert Chan 2004-08-29 21:35 ` Hubert Chan 2004-08-29 21:01 ` Linus Torvalds 2004-08-29 15:12 ` Alexander Lyamin 2004-08-29 16:20 ` viro 2004-08-30 15:00 ` John Stoffel 2004-08-31 0:26 ` Hubert Chan 2004-08-31 1:10 ` Hubert Chan 2004-08-28 23:47 ` Jeremy Allison 2004-08-30 13:56 ` Helge Hafting 2004-08-30 13:56 ` Helge Hafting 2004-08-30 23:56 ` Hubert Chan 2004-08-30 23:56 ` Hubert Chan 2004-08-27 18:40 ` Hubert Chan 2004-08-27 18:40 ` Hubert Chan 2004-08-28 0:02 ` Spam 2004-08-28 0:32 ` Hubert Chan 2004-08-28 0:32 ` Hubert Chan 2004-08-28 1:05 ` Hubert Chan 2004-08-28 1:05 ` Hubert Chan 2004-08-27 18:28 ` Hans Reiser 2004-08-27 18:28 ` Hans Reiser 2004-08-28 6:08 ` Paul Jackson 2004-08-29 15:02 ` Alexander Lyamin 2004-08-29 18:28 ` Hans Reiser 2004-08-29 18:36 ` viro 2004-08-29 18:57 ` viro 2004-08-29 20:06 ` Hans Reiser 2004-08-29 21:27 ` viro 2004-08-29 21:50 ` Linus Torvalds [not found] ` <Pine.LNX.4.60.0408300009001.10533@alpha.polcom.net> 2004-08-29 22:37 ` Linus Torvalds 2004-08-29 23:28 ` viro 2004-08-30 1:43 ` Neil Brown 2004-09-02 10:51 ` Stephan von Krawczynski 2004-09-02 11:26 ` Frank van Maarseveen 2004-09-02 12:58 ` Stephan von Krawczynski 2004-08-29 23:17 ` Trond Myklebust 2004-08-29 23:17 ` Trond Myklebust 2004-08-29 23:25 ` viro 2004-08-29 23:54 ` Linus Torvalds 2004-08-30 1:42 ` Trond Myklebust 2004-08-30 1:42 ` Trond Myklebust 2004-08-30 2:31 ` Linus Torvalds 2004-08-30 3:01 ` viro 2004-08-30 3:55 ` Linus Torvalds 2004-08-30 4:46 ` viro 2004-08-30 6:33 ` viro 2004-08-30 3:12 ` Trond Myklebust 2004-08-30 3:12 ` Trond Myklebust 2004-08-30 5:42 ` Brad Boyer 2004-08-30 0:07 ` Alexander G. M. Smith 2004-08-30 2:18 ` Hans Reiser 2004-08-30 7:42 ` Hans Reiser 2004-08-30 13:17 ` Alexander G. M. Smith 2004-08-30 17:43 ` Paul Stewart 2004-08-30 17:58 ` Linus Torvalds 2004-08-29 15:00 ` Alexander Lyamin 2004-08-31 0:46 ` David Masover 2004-08-31 6:47 ` Hans Reiser 2004-08-31 6:47 ` Hans Reiser 2004-09-01 2:04 ` David Masover 2004-09-01 2:04 ` David Masover 2004-08-26 20:44 ` Martin J. Bligh 2004-08-26 20:54 ` Linus Torvalds 2004-08-26 21:00 ` Martin J. Bligh 2004-08-26 21:09 ` Christophe Saout 2004-08-26 21:09 ` Christophe Saout 2004-08-26 21:13 ` Linus Torvalds 2004-08-26 21:24 ` Jamie Lokier 2004-08-27 9:36 ` David Greaves 2004-08-26 22:44 ` Miquel van Smoorenburg 2004-08-26 21:29 ` Chris Wright 2004-08-27 8:19 ` Anton Altaparmakov 2004-08-27 8:19 ` Anton Altaparmakov 2004-08-26 18:28 ` Denis Vlasenko 2004-08-26 18:28 ` Denis Vlasenko 2004-08-26 18:46 ` Linus Torvalds 2004-08-26 19:13 ` viro 2004-08-26 20:32 ` viro 2004-08-26 20:36 ` Christophe Saout 2004-08-26 20:36 ` Christophe Saout 2004-08-26 20:47 ` Linus Torvalds 2004-08-26 21:28 ` [some sanity for a change] possible design issues for hybrids viro 2004-08-26 22:04 ` Linus Torvalds 2004-08-26 22:36 ` viro 2004-08-26 22:45 ` Linus Torvalds 2004-08-26 22:53 ` viro 2004-08-26 23:24 ` Linus Torvalds 2004-08-26 23:40 ` viro 2004-08-26 23:57 ` Linus Torvalds 2004-08-27 1:01 ` viro 2004-08-27 2:46 ` Linus Torvalds 2004-08-27 3:45 ` viro 2004-08-27 3:57 ` Linus Torvalds 2004-08-27 4:30 ` viro 2004-08-27 4:42 ` Linus Torvalds 2004-08-27 4:46 ` Linus Torvalds 2004-08-27 8:56 ` Anton Altaparmakov 2004-08-27 8:56 ` Anton Altaparmakov 2004-08-27 9:00 ` Christoph Hellwig 2004-08-26 19:36 ` silent semantic changes with reiser4 Jonathan Abbey 2004-08-26 20:16 ` Jamie Lokier 2004-08-26 20:25 ` Christophe Saout 2004-08-26 20:25 ` Christophe Saout 2004-08-26 21:45 ` Nikita Danilov 2004-08-26 21:48 ` Christophe Saout 2004-08-26 21:48 ` Christophe Saout 2004-08-26 22:17 ` Emil Larsson 2004-08-26 22:21 ` Nikita Danilov 2004-08-26 22:29 ` Christophe Saout 2004-08-26 22:29 ` Christophe Saout 2004-08-26 23:35 ` Hans Reiser 2004-08-26 23:35 ` Hans Reiser 2004-08-27 2:00 ` Stephen Wille Padnos 2004-08-27 8:08 ` Hans Reiser 2004-08-26 17:40 ` Diego Calleja 2004-08-26 17:40 ` Diego Calleja 2004-08-26 17:55 ` Christophe Saout 2004-08-26 17:59 ` Rik van Riel 2004-08-26 17:59 ` Rik van Riel 2004-08-26 18:20 ` Jamie Lokier 2004-08-27 8:06 ` Hans Reiser 2004-08-27 8:06 ` Hans Reiser 2004-08-27 2:14 ` Hans Reiser 2004-08-26 15:19 ` Denis Vlasenko 2004-08-26 15:19 ` Denis Vlasenko 2004-08-29 14:30 ` Alan Cox 2004-08-29 17:42 ` Jeremy Allison 2004-08-29 18:41 ` Trond Myklebust 2004-08-29 18:41 ` Trond Myklebust 2004-08-29 19:38 ` Jeremy Allison 2004-09-01 20:19 ` Jamie Lokier 2004-09-01 20:26 ` Jeremy Allison 2004-09-01 20:31 ` Jamie Lokier 2004-09-01 20:35 ` Jeremy Allison 2004-09-01 20:47 ` Jamie Lokier 2004-09-01 20:51 ` Jeremy Allison 2004-09-01 21:35 ` Lee Revell 2004-09-01 21:35 ` Lee Revell 2004-09-01 23:22 ` Oliver Hunt 2004-09-02 12:54 ` Theodore Ts'o 2004-09-02 14:45 ` Stuart Young 2004-09-01 22:05 ` Hubert Chan 2004-09-01 22:05 ` Hubert Chan 2004-09-01 21:09 ` David Lang 2004-09-02 9:48 ` Alan Cox 2004-09-02 11:13 ` Oliver Neukum 2004-09-02 11:01 ` Alan Cox 2004-09-01 20:25 ` Jamie Lokier 2004-08-26 14:42 ` Felipe Alfaro Solana 2004-08-26 13:36 ` Rik van Riel 2004-08-26 13:36 ` Rik van Riel 2004-08-26 13:59 ` Jamie Lokier 2004-08-26 14:07 ` Rik van Riel 2004-08-26 14:07 ` Rik van Riel 2004-08-26 23:54 ` Hans Reiser 2004-08-26 23:54 ` Hans Reiser 2004-08-26 17:32 ` Jeremy Allison 2004-08-26 18:17 ` Rik van Riel 2004-08-26 18:17 ` Rik van Riel 2004-08-26 22:03 ` Felipe Alfaro Solana 2004-08-27 5:28 ` Jeremy Allison 2004-08-27 6:20 ` Hubert Chan 2004-08-27 6:20 ` Hubert Chan 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:13 ` Hans Reiser 2004-08-27 8:13 ` Hans Reiser 2004-08-26 9:03 ` Anton Altaparmakov 2004-08-26 9:03 ` Anton Altaparmakov 2004-08-26 10:45 ` Christian Mayrhuber 2004-08-26 10:52 ` Christoph Hellwig 2004-08-26 12:43 ` Olivier Galibert 2004-08-26 12:44 ` Christoph Hellwig 2004-08-26 12:49 ` Markus Törnqvist 2004-08-26 12:49 ` mjt 2004-08-26 12:49 ` Markus Törnqvist 2004-08-26 12:52 ` Christoph Hellwig 2004-08-26 13:35 ` Christian Mayrhuber 2004-08-26 13:35 ` Christian Mayrhuber 2004-08-26 15:15 ` Adrian Bunk 2004-08-26 8:44 ` Hans Reiser 2004-08-26 8:44 ` Hans Reiser 2004-08-26 8:31 ` Hans Reiser 2004-08-26 8:45 ` Andrew Morton 2004-08-26 9:24 ` Hans Reiser 2004-08-26 9:24 ` Hans Reiser 2004-08-26 19:26 ` Lee Revell 2004-08-26 19:26 ` Lee Revell 2004-08-26 19:33 ` Christophe Saout 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:05 ` Hans Reiser 2004-08-27 9:24 ` Christoph Hellwig 2004-08-26 19:34 ` Christoph Hellwig 2004-08-26 19:45 ` Lee Revell 2004-08-26 19:45 ` Lee Revell 2004-08-26 20:30 ` Dmitry Baryshkov 2004-08-26 20:38 ` Christophe Saout 2004-08-26 20:38 ` Christophe Saout 2004-08-26 20:49 ` Dmitry Baryshkov 2004-08-26 21:33 ` Nikita Danilov 2004-08-26 21:46 ` Christophe Saout 2004-08-26 21:46 ` Christophe Saout 2004-08-26 22:03 ` Nikita Danilov 2004-08-26 22:05 ` Christoph Hellwig 2004-08-27 8:12 ` Hans Reiser 2004-08-27 8:12 ` Hans Reiser 2004-08-27 8:57 ` Christoph Hellwig 2004-08-26 12:18 ` Christophe Saout 2004-08-26 12:18 ` Christophe Saout 2004-08-26 12:49 ` Christoph Hellwig 2004-08-26 13:00 ` Christophe Saout 2004-08-26 13:00 ` Christophe Saout 2004-08-26 13:07 ` Christoph Hellwig 2004-08-26 13:17 ` reiser4 plugins (was: silent semantic changes with reiser4) Christophe Saout 2004-08-26 13:17 ` Christophe Saout 2004-08-26 13:24 ` Christoph Hellwig 2004-08-26 13:24 ` Christoph Hellwig 2004-08-26 13:35 ` Christophe Saout 2004-08-26 13:35 ` Christophe Saout 2004-08-26 13:40 ` Christoph Hellwig 2004-08-26 13:58 ` Christophe Saout 2004-08-26 15:37 ` Christoph Hellwig 2004-08-26 15:48 ` Christophe Saout 2004-08-26 16:06 ` Christoph Hellwig 2004-08-26 16:14 ` Christophe Saout 2004-08-26 16:25 ` Christoph Hellwig 2004-08-26 16:39 ` Christophe Saout 2004-08-26 16:39 ` Christophe Saout 2004-08-26 23:55 ` reiser4 plugins Hans Reiser 2004-08-26 23:55 ` Hans Reiser 2004-08-27 12:04 ` Nikita Danilov 2004-08-27 18:15 ` Hans Reiser 2004-08-27 18:15 ` Hans Reiser 2004-08-27 18:55 ` Nikita Danilov 2004-08-28 9:53 ` Hans Reiser 2004-08-28 9:53 ` Hans Reiser 2004-08-28 13:47 ` Nikita Danilov 2004-08-28 14:10 ` Konstantin Besch 2004-08-28 22:09 ` Hans Reiser 2004-08-28 23:45 ` Hans Reiser 2004-08-29 9:35 ` Nikita Danilov 2004-08-29 11:17 ` Alex Zarochentsev 2004-08-27 22:29 ` Steve Bergman 2004-08-27 22:29 ` Steve Bergman 2004-08-28 6:54 ` Hans Reiser 2004-08-28 10:09 ` Steve Bergman 2004-08-29 11:52 ` Alex Zarochentsev 2004-08-29 11:55 ` Alex Zarochentsev 2004-08-29 11:42 ` Alex Zarochentsev 2004-08-28 9:41 ` Redeeman 2004-08-26 23:54 ` Hans Reiser 2004-08-26 23:54 ` Hans Reiser 2004-08-26 13:52 ` reiser4 plugins (was: silent semantic changes with reiser4) Rik van Riel 2004-08-26 13:52 ` Rik van Riel 2004-08-26 13:58 ` Chris Mason 2004-08-26 14:02 ` Christophe Saout 2004-08-26 14:02 ` Christophe Saout 2004-08-28 11:18 ` Alexander Lyamin 2004-08-28 11:22 ` Christoph Hellwig 2004-08-28 11:46 ` Alexander Lyamin 2004-08-28 11:51 ` Christoph Hellwig 2004-08-28 12:18 ` Alexander Lyamin 2004-08-28 11:58 ` Markus Törnqvist 2004-08-28 16:11 ` Jan Harkes 2004-08-28 16:26 ` Nikita Danilov 2004-08-28 19:41 ` Alexander Lyamin 2004-08-28 10:59 ` Alexander Lyamin 2004-08-28 10:59 ` Alexander Lyamin 2004-08-28 10:59 ` Alexander Lyamin 2004-08-28 11:12 ` Christoph Hellwig 2004-08-28 12:05 ` Alexander Lyamin 2004-08-28 13:56 ` Christoph Hellwig 2004-08-28 19:23 ` Alexander Lyamin 2004-08-28 19:27 ` Alexander Lyamin 2004-08-28 22:36 ` reiser4 plugins Hans Reiser 2004-08-28 22:36 ` Hans Reiser 2004-08-28 11:14 ` reiser4 plugins (was: silent semantic changes with reiser4) Markus Törnqvist 2004-08-28 17:18 ` Linus Torvalds 2004-08-28 19:03 ` Alexander Lyamin 2004-08-28 19:09 ` Christoph Hellwig 2004-08-28 21:41 ` reiser4 plugins Hans Reiser 2004-08-28 21:41 ` Hans Reiser 2004-08-30 16:02 ` Herbert Poetzl 2004-08-30 18:55 ` Hans Reiser 2004-08-30 18:55 ` Hans Reiser 2004-08-28 19:26 ` reiser4 plugins (was: silent semantic changes with reiser4) Linus Torvalds 2004-08-26 13:27 ` silent semantic changes with reiser4 Christophe Saout 2004-08-26 13:27 ` Christophe Saout 2004-08-26 14:44 ` Jamie Lokier 2004-08-26 16:03 ` Christoph Hellwig 2004-08-26 16:19 ` Jamie Lokier 2004-08-26 16:23 ` Christoph Hellwig 2004-08-26 17:08 ` Jamie Lokier 2004-08-26 23:54 ` Hans Reiser 2004-08-26 23:54 ` Hans Reiser 2004-08-26 9:34 ` Christoph Hellwig 2004-08-26 9:37 ` Markus Törnqvist 2004-08-26 18:40 ` Thomas Cataldo 2004-08-26 12:30 ` Redeeman 2004-08-26 16:32 ` Will Dyson 2004-08-26 16:44 ` Christophe Saout 2004-08-26 16:44 ` Christophe Saout 2004-08-26 17:22 ` Jamie Lokier 2004-08-27 8:08 ` Hans Reiser 2004-08-27 8:08 ` Hans Reiser 2004-08-27 18:20 ` Will Dyson 2004-08-27 18:38 ` David Ford 2004-08-27 18:38 ` David Ford 2004-08-28 6:14 ` Hans Reiser 2004-08-28 9:54 ` Hans Reiser 2004-08-28 9:54 ` Hans Reiser 2004-08-31 3:14 ` Will Dyson 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2004-08-28 22:11 ` Hans Reiser 2004-08-28 22:12 ` Hans Reiser 2004-09-01 20:41 ` Jamie Lokier 2004-08-31 1:57 ` Will Dyson 2004-08-31 2:33 ` Alexander G. M. Smith 2004-08-31 3:22 ` Will Dyson 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand 2004-08-27 20:32 ` Markus Törnqvist 2004-08-27 20:32 ` mjt 2004-08-27 20:32 ` Markus Törnqvist 2004-08-27 20:38 ` Linus Torvalds 2004-08-27 20:48 ` Re[2]: " Spam 2004-08-27 20:53 ` Markus Törnqvist 2004-08-27 20:53 ` mjt 2004-08-27 20:53 ` Markus Törnqvist 2004-08-27 21:33 ` Bill Huey 2004-08-28 9:55 ` Hans Reiser 2004-09-01 22:45 ` Bill Huey 2004-08-27 20:44 ` Spam 2004-08-27 20:44 ` Spam [not found] ` <200408291521.i7TFLsQk028363@localhost.localdomain> 2004-08-29 16:52 ` Spam 2004-08-29 16:52 ` Spam 2004-08-29 17:22 ` Nikita Danilov 2004-08-29 17:22 ` Nikita Danilov 2004-08-29 18:12 ` Markus Törnqvist 2004-08-29 16:53 ` Spam 2004-08-29 16:53 ` Spam 2004-08-29 16:54 ` Spam 2004-08-29 16:54 ` Spam 2004-09-01 21:43 ` Jamie Lokier 2004-08-26 20:04 ` Martin J. Bligh 2004-08-26 21:18 ` Hubert Chan 2004-08-26 21:18 ` Hubert Chan 2004-08-27 21:23 ` vfs2 (was Re: silent semantic changes with reiser4) Pavel Machek
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.