silent semantic changes with reiser4

silent semantic changes with reiser4

[Christoph Hellwig]

After looking trough the code and mailinglists I'm quite unhappy with
a bunch of user-visible changes that Hans sneaked in and make reiser4
incompatible with other filesystems and have a slight potential to break
even in the kernel.

 o files as directories
   - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
     .htaccess handling in apache and glibc compilation this also renders
     this flag entirely useless and opens up the races it tries to
     prevent against cmpletely useless
   - meaning of the -x permission.  This one has different meanings on
     directories vs files on UNIX systems.  If we want to support
     directories as files we'll probably have to find a way to work
     around this.
   - dentry aliasing.  I can't find a formal guarantee in the code this
     can't happen
 
 o metafiles - ..metas as a magic name that's just taken out of the
   namespace doesn't sound like a good idea.  If we want this it should
   be a VFS-level option and there should be a translation-layer to
   xattrs.  Not doing this will again confuse applications greatly that
   expect uniform filesystem behaviour.

Given these problems I request that these interfaces are removed from
reiser4 for the kernel merge, and if added later at the proper VFS level
after discussion on linux-kernel and linux-fsdevel, like we did for
xattrs.

Re: silent semantic changes with reiser4

[Lee Revell]

On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote:
>    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
>      .htaccess handling in apache and glibc compilation this also renders
>      this flag entirely useless and opens up the races it tries to
>      prevent against cmpletely useless

So `find -type d' would list every file on the system?

Lee

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Tue, Aug 24, 2004 at 04:35:18PM -0400, Lee Revell wrote:
> On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote:
> >    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
> >      .htaccess handling in apache and glibc compilation this also renders
> >      this flag entirely useless and opens up the races it tries to
> >      prevent against cmpletely useless
> 
> So `find -type d' would list every file on the system?

the find I have here is using lstat and not open with O_DIRECTORY, so
no.

Re: silent semantic changes with reiser4

[Lee Revell]

On Tue, 2004-08-24 at 16:38, Christoph Hellwig wrote:
> On Tue, Aug 24, 2004 at 04:35:18PM -0400, Lee Revell wrote:
> > On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote:
> > >    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
> > >      .htaccess handling in apache and glibc compilation this also renders
> > >      this flag entirely useless and opens up the races it tries to
> > >      prevent against cmpletely useless
> > 
> > So `find -type d' would list every file on the system?
> 
> the find I have here is using lstat and not open with O_DIRECTORY, so
> no.
> 

Ugh, how embarrassing, I completely forgot about stat().

Lee

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > >    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
> > >      .htaccess handling in apache and glibc compilation this also renders
> > >      this flag entirely useless and opens up the races it tries to
> > >      prevent against cmpletely useless
> > 
> > So `find -type d' would list every file on the system?
> 
> the find I have here is using lstat and not open with O_DIRECTORY, so
> no.

The find-like program I use (called treescan) uses O_DIRECTORY as an
optimisation.  It assumes that O_DIRECTORY will only open objects
which are directories and can be read using readdir().

However, if reiser4 returns d_type values, then it won't even attempt
an open on non-DT_DIR objects, and that's a better optimisation.
(reiserfs doesn't return d_type values, unfortunately).

So the list of files that treescan finds depends on whether reiser4
implements d_type.

This is nothing like a POSIX filesystem.  You untar a tree, and then
listing it recursively shows extra things created by reiser4.

I quite like the principle, but because it's not like POSIX and
doesn't match some program's expectations, it's a problem in its
present form.

xattrs aren't a complete solution as you can't store structured data
in an xattr.  For example, with reiser4's model, you can cd into a
.tar, .zip, .mp3 or .xml file and list the internal structure along
with the file's metadata.  You can't do that with xattrs.

Programs exist which quite reasonably assume that when you create a
file, you can't opendir() the file, and recursive listings (like find,
ls -R et al.) won't automatically traverse into every file.

On the other hand, being able to enter a file in a directory-like way
allows structured representations of the contents to be accessed in
the very useful "everything's a file" way -- i.e. ordinary tools.

So here's a semantic proposal:

   1. O_DIRECTORY won't open an ordinary file.
      Corollary: opendir("file") won't open an ordinary file.

   2. An ordinary file path followed by "/" won't open an ordinary file.
      Corollary: opendir("file/") won't open an ordinary file.

      This is because appending a trailing slash is an alternate
      way for userspace to get the same results as O_DIRECTORY.

   3. An ordinary file path followed by "/" _and_ one or more path
      components will open the file as a directory and enter it.
      Corollary: opendir("file/.") will open an ordinary file.

   4. The type of "file/." shall be S_IFDIR, _not_ S_IFREG.
      Corollary: stat("file/.") will return that it's a directory.

      The intention here is that explicit requests to examine the
      metadata or alternate structure representations of a file will
      create such a view, but the view is only available if requested
      explicitly.

      When such a view is created, the results of stat(), O_DIRECTORY
      and opendir() are absolutely consistent.  This will minimise
      confusion.  Programs which recurse over a directory tree won't
      look inside any of the files.  However they can be explicitly
      asked to recurse starting from a path inside a file: then
      they'll recurse over a single file's metadata and structured data.

Regarding the problems of safe locking in the VFS.  The VFS assumes
that directories are not hard linked: i.e. that they cannot appear at
more than one path in a filesystem.  Files-as-directories breaks that.

However, VFS does support directories on multiple paths, using bind mounts.

So it wouldn't be out of the question if entering a file (as described
above) effectively auto-mounted a bind mount at that point.


-- Jamie

Re: silent semantic changes with reiser4

[Jeff Garzik]

Christoph Hellwig wrote:
>  o files as directories
>    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
>      .htaccess handling in apache and glibc compilation this also renders
>      this flag entirely useless and opens up the races it tries to
>      prevent against cmpletely useless


Ouch.

I would definitely classify this as a security hole, since userland 
definitely uses O_DIRECTORY to avoid races.

	Jeff

Re: silent semantic changes with reiser4

[viro]

On Tue, Aug 24, 2004 at 04:38:25PM -0400, Jeff Garzik wrote:
> Christoph Hellwig wrote:
> > o files as directories
> >   - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
> >     .htaccess handling in apache and glibc compilation this also renders
> >     this flag entirely useless and opens up the races it tries to
> >     prevent against cmpletely useless
> 
> 
> Ouch.
> 
> I would definitely classify this as a security hole, since userland 
> definitely uses O_DIRECTORY to avoid races.

Feh.  That's far from the worst parts of the mess introduced by "hybrid"
crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit
higher on the suckitude scale, IMO.

Re: silent semantic changes with reiser4

[viro]

On Tue, Aug 24, 2004 at 09:53:44PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
 
> Feh.  That's far from the worst parts of the mess introduced by "hybrid"
> crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit
> higher on the suckitude scale, IMO.

While we are at it - consider these hybrids vetoed until
	a) sys_link()/sys_link() deadlock is fixed
	b) sys_link()/sys_rename() deadlock is fixed
	c) correctness proof of the locking scheme (in
Documentation/filesystems/directory-locking) is updated to match the
presense of the file/directory hybrids.

Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince
us that there's no more crap of that kind".  IMO a reasonable request, seeing
that the first look at the patches in -mm4 had turned up two exploits in
that area, despite the *YEARS* of warnings about potential trouble and need
to be careful there (actually, I've given Hans too much credit and assumed
that link/link never happens since nobody would be dumb enough to provide
->link() method for non-directory inodes; turns out that somebody is dumb
enough and link/link is as exploitable as link/rename).

Re: silent semantic changes with reiser4

[Hans Reiser]

I allowed myself to get talked out of a final top to bottom code audit, 
and obviously that was a mistake.

It will probably take about 6 weeks.  Apologies for wasting your time 
before that was done.

Hans


viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Tue, Aug 24, 2004 at 09:53:44PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
>  
>
>>Feh.  That's far from the worst parts of the mess introduced by "hybrid"
>>crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit
>>higher on the suckitude scale, IMO.
>>    
>>
>
>While we are at it - consider these hybrids vetoed until
>	a) sys_link()/sys_link() deadlock is fixed
>	b) sys_link()/sys_rename() deadlock is fixed
>	c) correctness proof of the locking scheme (in
>Documentation/filesystems/directory-locking) is updated to match the
>presense of the file/directory hybrids.
>
>Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince
>us that there's no more crap of that kind".  IMO a reasonable request, seeing
>that the first look at the patches in -mm4 had turned up two exploits in
>that area, despite the *YEARS* of warnings about potential trouble and need
>to be careful there (actually, I've given Hans too much credit and assumed
>that link/link never happens since nobody would be dumb enough to provide
>->link() method for non-directory inodes; turns out that somebody is dumb
>enough and link/link is as exploitable as link/rename).
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Wed, Aug 25, 2004 at 11:28:56AM -0700, Hans Reiser wrote:
> I allowed myself to get talked out of a final top to bottom code audit, 
> and obviously that was a mistake.
> 
> It will probably take about 6 weeks.  Apologies for wasting your time 
> before that was done.

I don't think you'll get anywhere with auditing.  We need to write down
the semantics you want, define them at the VFS level and make sure
they're not conflicting with defined userspace semantics or kernel
assumptions.

I think you need to learn the basic distinction between the VFS layer
and a lowlevel filesystem driver.

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>
>
>I don't think you'll get anywhere with auditing.  We need to write down
>the semantics you want, define them at the VFS level and make sure
>they're not conflicting with defined userspace semantics or kernel
>assumptions.
>
>I think you need to learn the basic distinction between the VFS layer
>and a lowlevel filesystem driver.
>
>
>
>  
>
How old are you?  I thought you were the guy at Linux Tag with fashion 
oriented hair who gave a talk on his XFS work?  Did I confuse you with 
someone else?


Hans

Re: silent semantic changes with reiser4

[William Lee Irwin III]

On Thu, Aug 26, 2004 at 02:02:29AM -0700, Hans Reiser wrote:
> How old are you?  I thought you were the guy at Linux Tag with fashion 
> oriented hair who gave a talk on his XFS work?  Did I confuse you with 
> someone else?

Index: mm1-2.6.9-rc1/CREDITS
===================================================================
--- mm1-2.6.9-rc1.orig/CREDITS	2004-08-26 10:37:57.424567967 -0700
+++ mm1-2.6.9-rc1/CREDITS	2004-08-26 11:35:24.977260109 -0700
@@ -2677,6 +2677,14 @@
 S: 79098 Freiburg
 S: Germany
 
+N: Hans Reiser
+E: reiser@namesys.com
+W: http://www.namesys.com/
+D: official Linux kernel hairstyle critic
+S: 6979 Exeter Dr
+S: Oakland, CA 94611
+S: USA
+
 N: Joerg Reuter
 E: jreuter@yaina.de
 W: http://yaina.de/jreuter/

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, William Lee Irwin III wrote:
>  
> +N: Hans Reiser
> +E: reiser@namesys.com
> +W: http://www.namesys.com/
> +D: official Linux kernel hairstyle critic
> +S: 6979 Exeter Dr
> +S: Oakland, CA 94611
> +S: USA
> +

LOL.

In all fairness, how _would_ you have described it?

Besides, I don't think this should go in the CREDITS file, since hair 
styling criticism is clearly an ongoing MAINTAINERS issue, no?

		Linus

Re: silent semantic changes with reiser4

[William Lee Irwin III]

On Thu, 26 Aug 2004, William Lee Irwin III wrote:
>> +N: Hans Reiser
>> +E: reiser@namesys.com
>> +W: http://www.namesys.com/
>> +D: official Linux kernel hairstyle critic
>> +S: 6979 Exeter Dr
>> +S: Oakland, CA 94611
>> +S: USA

On Thu, Aug 26, 2004 at 11:49:19AM -0700, Linus Torvalds wrote:
> LOL.
> In all fairness, how _would_ you have described it?
> Besides, I don't think this should go in the CREDITS file, since hair 
> styling criticism is clearly an ongoing MAINTAINERS issue, no?

As far as maintenance goes, absolutely. As far as describing it goes,
I probably would have said it was short, red, and spiky.


Index: mm1-2.6.9-rc1/MAINTAINERS
===================================================================
--- mm1-2.6.9-rc1.orig/MAINTAINERS	2004-08-26 10:38:09.867927190 -0700
+++ mm1-2.6.9-rc1/MAINTAINERS	2004-08-26 11:58:27.759469733 -0700
@@ -1352,6 +1352,12 @@
 L:	linuxppc64-dev@lists.linuxppc.org
 S:	Supported
 
+LINUX KERNEL HAIRSTYLE CRITICISM
+P: Hans Reiser
+M: reiser@namesys.com
+W: http://www.namesys.com/
+S: Maintained
+
 LINUX SECURITY MODULE (LSM) FRAMEWORK
 P:	Chris Wright
 M:	chrisw@osdl.org

Re: silent semantic changes with reiser4

[Hans Reiser]

William Lee Irwin III wrote:

>
>As far as maintenance goes, absolutely. As far as describing it goes,
>I probably would have said it was short, red, and spiky.
>
>  
>
Ah, so it was him, is he disowning XFS these days or?

Re: silent semantic changes with reiser4

[Giuseppe Bilotta]

William Lee Irwin III wrote:
> On Thu, 26 Aug 2004, William Lee Irwin III wrote:
> >> +N: Hans Reiser
> >> +E: reiser@namesys.com
> >> +W: http://www.namesys.com/
> >> +D: official Linux kernel hairstyle critic
> >> +S: 6979 Exeter Dr
> >> +S: Oakland, CA 94611
> >> +S: USA
> 
> On Thu, Aug 26, 2004 at 11:49:19AM -0700, Linus Torvalds wrote:
> > LOL.
> > In all fairness, how _would_ you have described it?
> > Besides, I don't think this should go in the CREDITS file, since hair 
> > styling criticism is clearly an ongoing MAINTAINERS issue, no?
> 
> As far as maintenance goes, absolutely. As far as describing it goes,
> I probably would have said it was short, red, and spiky.
> 
> 
> Index: mm1-2.6.9-rc1/MAINTAINERS
> ===================================================================
> --- mm1-2.6.9-rc1.orig/MAINTAINERS	2004-08-26 10:38:09.867927190 -0700
> +++ mm1-2.6.9-rc1/MAINTAINERS	2004-08-26 11:58:27.759469733 -0700
> @@ -1352,6 +1352,12 @@
>  L:	linuxppc64-dev@lists.linuxppc.org
>  S:	Supported
>  
> +LINUX KERNEL HAIRSTYLE CRITICISM

Hm. He doesn't critic the Linux Kernel hairstyle, but the Linux 
Kernel Developers' ...

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: silent semantic changes with reiser4

[William Lee Irwin III]

William Lee Irwin III wrote:
>> +LINUX KERNEL HAIRSTYLE CRITICISM

On Fri, Aug 27, 2004 at 12:10:12PM +0200, Giuseppe Bilotta wrote:
> Hm. He doesn't critic the Linux Kernel hairstyle, but the Linux 
> Kernel Developers' ...

Absolutely right.


Index: mm1-2.6.9-rc1/MAINTAINERS
===================================================================
--- mm1-2.6.9-rc1.orig/MAINTAINERS	2004-08-26 10:38:09.000000000 -0700
+++ mm1-2.6.9-rc1/MAINTAINERS	2004-08-27 08:27:42.738139199 -0700
@@ -1352,6 +1352,12 @@
 L:	linuxppc64-dev@lists.linuxppc.org
 S:	Supported
 
+LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM
+P: Hans Reiser
+M: reiser@namesys.com
+W: http://www.namesys.com/
+S: Maintained
+
 LINUX SECURITY MODULE (LSM) FRAMEWORK
 P:	Chris Wright
 M:	chrisw@osdl.org

Re: silent semantic changes with reiser4

[William Lee Irwin III]

On Fri, Aug 27, 2004 at 08:25:39AM -0700, William Lee Irwin III wrote:
>> +LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM

> Aren't there more than one Linux Kernel Developer?  Or is this to
> indicate that he indeed only criticised the hairstyle of one developer?

Best I repair that, then.


Index: mm1-2.6.9-rc1/MAINTAINERS
===================================================================
--- mm1-2.6.9-rc1.orig/MAINTAINERS	2004-08-26 10:38:09.000000000 -0700
+++ mm1-2.6.9-rc1/MAINTAINERS	2004-08-27 09:18:59.234195262 -0700
@@ -1352,6 +1352,12 @@
 L:	linuxppc64-dev@lists.linuxppc.org
 S:	Supported
 
+LINUX KERNEL DEVELOPERS' HAIRSTYLE CRITICISM
+P: Hans Reiser
+M: reiser@namesys.com
+W: http://www.namesys.com/
+S: Maintained
+
 LINUX SECURITY MODULE (LSM) FRAMEWORK
 P:	Chris Wright
 M:	chrisw@osdl.org

Re: silent semantic changes with reiser4

[Hans Reiser]

I had not intended to respond to this because I have nothing positive to 
say, but Andrew said I needed to respond and suggested I should copy 
Linus. Sigh.

Dear Christoph,

Let me see if I can summarize what you and your contingent are saying, 
and if I misconstrue anything, let me know.;-)

You ignored everything I said during the discussion of xattrs about how 
there is no need to have attributes when you can just have files and 
directories, and that xattrs reflected a complete ignorance of name 
space design principles.  When I said we should just add some nice 
optional features to files and directories so that they can do 
everything that attributes can do if they are used that way, you just 
didn't get it.  You instead went for the quick ugly hack called xattrs.  
You then got that ugly hack done first, because quick hacks are, well, 
quick.  I then went about doing it the right way for Reiser4, and got 
DARPA to fund doing it.  I was never silent about it.

Making files into directories caused only two applications out of the 
entire OS to notice the change, and that was because of a bug in what 
error code we returned that we are going to fix.  You think that was a 
disaster; I think it was a triumph.

Now a cleanly architected filesystem with no attributes and just files 
and directories that can do everything attributes are used for exists.  
You don't want it to have the competitive advantage.  Instead, you want 
it to have its clean design excised until you have something that 
duplicates it ready to go, and only then should it be allowed that users 
will use the features of your competitor's filesystem which you 
disdained implementing for so long.

Since you never studied or understood namespace design principles (or 
you would not have created and supported xattrs), you want to rename it 
to be called VFS, rewrite what we have done, and take over as the 
maintainer, mangling its design in a committee clusterfuck as you go. 

We have just implemented very trivial semantic enhancements of the FS 
namespace, nothing like as ambitious as www.namesys.com/whitepaper.html 
or WinFS, and you are already pissing your pants.

Is that a fair summary?

Eat my dust. 

Hans

PS

I should of course qualify what I have said.  The use of files and 
directories in place of attributes is not a finished work.  It has bugs, 
sys_reiser4() does not yet work, and there are little features still 
missing like having files readdir ignores.

Still, except for the bugs, what we have is usable, and there are a lot 
of happy reiser4 users right now even with the bugs. It will need a 
little bit more time, and then all the pieces will be in place.

PPS

If you implement your filesystems as reiser4 plugins, and rename 
reiser4's plugin code to be called "vfs", your filesystems will go 
faster.  Not as fast as reiser4 though, because it has a better layout 
and that affects performance a lot, but faster is faster....  See 
www.namesys.com/benchmarks.html for details.

PPPS

 Since we have such a performance lead, Namesys is about to change its 
focus from the storage layer to semantics, look at 
www.namesys.com/whitepaper.html for details.  Semantic enhancements are 
the important stuff, and finally Namesys is where we have all the 
storage layer prerequisites done right, and the real work can begin.  
The gap between us is about to widen further.



Christoph Hellwig wrote:

>After looking trough the code and mailinglists I'm quite unhappy with
>a bunch of user-visible changes that Hans sneaked in and make reiser4
>incompatible with other filesystems 
>
if we leave you in the dust, run faster.... not my problem....


>Given these problems I request that these interfaces are removed from
>reiser4 for the kernel merge, and if added later at the proper VFS level
>after discussion on linux-kernel and linux-fsdevel, like we did for
>xattrs.
>
>
>  
>
If you can't help fight WinFS, then get out of the way.  Namesys is on 
the march.  Read www.namesys.com/whitepaper.html.

Or, be smart, recognize that reiser4 is faster and more flexible than 
your storage layers because we are older and wiser and worked harder at 
it, join the team, and start contributing plugins that tap into the 
higher performance it offers.

Microsoft tried to build a storage layer that could handle small objects 
without losing performance, failed, and gave up at considerable cost to 
their architecture and pocketbook.

We just broke a hole in the enemy line.  You could come swarming through 
it with us, but it sounds like you prefer complaining to HQ that we are 
getting too far in front of you.

Re: silent semantic changes with reiser4

[Hans Reiser]

I had not intended to respond to this because I have nothing positive to 
say, but Andrew said I needed to respond and suggested I should copy 
Linus. Sigh.

Dear Christoph,

Let me see if I can summarize what you and your contingent are saying, 
and if I misconstrue anything, let me know.;-)

You ignored everything I said during the discussion of xattrs about how 
there is no need to have attributes when you can just have files and 
directories, and that xattrs reflected a complete ignorance of name 
space design principles.  When I said we should just add some nice 
optional features to files and directories so that they can do 
everything that attributes can do if they are used that way, you just 
didn't get it.  You instead went for the quick ugly hack called xattrs.  
You then got that ugly hack done first, because quick hacks are, well, 
quick.  I then went about doing it the right way for Reiser4, and got 
DARPA to fund doing it.  I was never silent about it.

Making files into directories caused only two applications out of the 
entire OS to notice the change, and that was because of a bug in what 
error code we returned that we are going to fix.  You think that was a 
disaster; I think it was a triumph.

Now a cleanly architected filesystem with no attributes and just files 
and directories that can do everything attributes are used for exists.  
You don't want it to have the competitive advantage.  Instead, you want 
it to have its clean design excised until you have something that 
duplicates it ready to go, and only then should it be allowed that users 
will use the features of your competitor's filesystem which you 
disdained implementing for so long.

Since you never studied or understood namespace design principles (or 
you would not have created and supported xattrs), you want to rename it 
to be called VFS, rewrite what we have done, and take over as the 
maintainer, mangling its design in a committee clusterfuck as you go. 

We have just implemented very trivial semantic enhancements of the FS 
namespace, nothing like as ambitious as www.namesys.com/whitepaper.html 
or WinFS, and you are already pissing your pants.

Is that a fair summary?

Eat my dust. 

Hans

PS

I should of course qualify what I have said.  The use of files and 
directories in place of attributes is not a finished work.  It has bugs, 
sys_reiser4() does not yet work, and there are little features still 
missing like having files readdir ignores.

Still, except for the bugs, what we have is usable, and there are a lot 
of happy reiser4 users right now even with the bugs. It will need a 
little bit more time, and then all the pieces will be in place.

PPS

If you implement your filesystems as reiser4 plugins, and rename 
reiser4's plugin code to be called "vfs", your filesystems will go 
faster.  Not as fast as reiser4 though, because it has a better layout 
and that affects performance a lot, but faster is faster....  See 
www.namesys.com/benchmarks.html for details.

PPPS

 Since we have such a performance lead, Namesys is about to change its 
focus from the storage layer to semantics, look at 
www.namesys.com/whitepaper.html for details.  Semantic enhancements are 
the important stuff, and finally Namesys is where we have all the 
storage layer prerequisites done right, and the real work can begin.  
The gap between us is about to widen further.



Christoph Hellwig wrote:

>After looking trough the code and mailinglists I'm quite unhappy with
>a bunch of user-visible changes that Hans sneaked in and make reiser4
>incompatible with other filesystems 
>
if we leave you in the dust, run faster.... not my problem....


>Given these problems I request that these interfaces are removed from
>reiser4 for the kernel merge, and if added later at the proper VFS level
>after discussion on linux-kernel and linux-fsdevel, like we did for
>xattrs.
>
>
>  
>
If you can't help fight WinFS, then get out of the way.  Namesys is on 
the march.  Read www.namesys.com/whitepaper.html.

Or, be smart, recognize that reiser4 is faster and more flexible than 
your storage layers because we are older and wiser and worked harder at 
it, join the team, and start contributing plugins that tap into the 
higher performance it offers.

Microsoft tried to build a storage layer that could handle small objects 
without losing performance, failed, and gave up at considerable cost to 
their architecture and pocketbook.

We just broke a hole in the enemy line.  You could come swarming through 
it with us, but it sounds like you prefer complaining to HQ that we are 
getting too far in front of you.

Re: silent semantic changes with reiser4

[Matthew Wilcox]

That's a nice marketing talk.  Get back to us when you have some technical
contribution to make.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: silent semantic changes with reiser4

[Hans Reiser]

Matthew Wilcox wrote:

>That's a nice marketing talk.  Get back to us when you have some technical
>contribution to make.
>
>  
>
www.namesys.com/download.html

Re: silent semantic changes with reiser4

[Hans Reiser]

Matthew Wilcox wrote:

>That's a nice marketing talk.  Get back to us when you have some technical
>contribution to make.
>
>  
>
www.namesys.com/download.html

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote:
> You ignored everything I said during the discussion of xattrs about how 
> there is no need to have attributes when you can just have files and 
> directories, and that xattrs reflected a complete ignorance of name 
> space design principles.

Actually in most of the discussion you simply didn't participate.  While
xattrs might not be the nicest interface they have the advantag of not
breaking the SuS assumption of what directories vs files are, and they
do not break the Linux O_DIRECTORY semantics that are defined and need
to solve real-world races either.

> When I said we should just add some nice 
> optional features to files and directories so that they can do 
> everything that attributes can do if they are used that way, you just 
> didn't get it.  You instead went for the quick ugly hack called xattrs.  
> You then got that ugly hack done first, because quick hacks are, well, 
> quick.  I then went about doing it the right way for Reiser4, and got 
> DARPA to fund doing it.  I was never silent about it.

For one thing _I_ didn't decide about xattrs anyway.  And I still
haven't seen a design from you on -fsdevel how you try to solve the
problems with files as directories.

> Now a cleanly architected filesystem with no attributes and just files 
> and directories that can do everything attributes are used for exists.  
> You don't want it to have the competitive advantage.  Instead, you want 
> it to have its clean design excised until you have something that 
> duplicates it ready to go, and only then should it be allowed that users 
> will use the features of your competitor's filesystem which you 
> disdained implementing for so long.

My competitors filesystem?  If you look at MAINTAINERS I maintain only
vxfs and sysvfs, neither of which I'd suggest anyone to run their system
on.

> Since you never studied or understood namespace design principles (or 
> you would not have created and supported xattrs), you want to rename it 
> to be called VFS, rewrite what we have done, and take over as the 
> maintainer, mangling its design in a committee clusterfuck as you go. 

Hans, please stop the personal crap or the black helicopters will kidnap
you.   When was the last time you actually worked on kernel namespace
code instead of talking marketing bullshit and ignoring all real world
problems.

> If you implement your filesystems as reiser4 plugins, and rename 
> reiser4's plugin code to be called "vfs", your filesystems will go 
> faster.  Not as fast as reiser4 though, because it has a better layout 
> and that affects performance a lot, but faster is faster....  See 
> www.namesys.com/benchmarks.html for details.

Could you pass on that crack pipe please?

Re: silent semantic changes with reiser4

[Christoph Hellwig]

Btw, I just got reminded you might take my saying as an "piss of you're
idea stinks" or similar things.  So let me clarify again the actual
technical and project managment issues another time before we start
getting really personal :)

Over the last at least five years we've taken as much as possible
semantics out of the filesystems and into the VFS layer, thus having
a separation between the semantical layer (VFS) and the low level
filesystem.  Your attributes are absoultely a VFS thing and as such
should not happen at the filesystem layer, and no, that doesn't mean
they're bad per se, I just think they are a rather bad fit for Linux.

So now go on and try to work together with the other peope doing VFS
level work instead of hiding, or if you think you can't work together
with us search a nice research OS where you can take over the VFS layer,
if your ideas prove to be good I'm sure Linux will pick them up sooner
or later.

	Christoph

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> 
> Over the last at least five years we've taken as much as possible
> semantics out of the filesystems and into the VFS layer, thus having
> a separation between the semantical layer (VFS) and the low level
> filesystem.  Your attributes are absoultely a VFS thing and as such
> should not happen at the filesystem layer, and no, that doesn't mean
> they're bad per se, I just think they are a rather bad fit for Linux.

Now this I agree with, in the sense that I think that if we want to 
support this, it should be supported at a VFS layer. 

On the other hand, I think doing it inside the filesystem with ugly hacks 
is an acceptable way to prototype the idea before it's been proven to 
really be workable. Maybe it has more problems with legacy apps than we'd 
expect..

		Linus

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Wed, Aug 25, 2004 at 01:24:36PM -0700, Linus Torvalds wrote:
> Now this I agree with, in the sense that I think that if we want to 
> support this, it should be supported at a VFS layer. 
> 
> On the other hand, I think doing it inside the filesystem with ugly hacks 
> is an acceptable way to prototype the idea before it's been proven to 
> really be workable. Maybe it has more problems with legacy apps than we'd 
> expect..

Oh, I'm the last person to tell anyone how to prototype things.  I just
don't want such inconsistancies in the mainline kernel.

Re: silent semantic changes with reiser4

[viro]

On Wed, Aug 25, 2004 at 01:24:36PM -0700, Linus Torvalds wrote:
> 
> 
> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> > 
> > Over the last at least five years we've taken as much as possible
> > semantics out of the filesystems and into the VFS layer, thus having
> > a separation between the semantical layer (VFS) and the low level
> > filesystem.  Your attributes are absoultely a VFS thing and as such
> > should not happen at the filesystem layer, and no, that doesn't mean
> > they're bad per se, I just think they are a rather bad fit for Linux.
> 
> Now this I agree with, in the sense that I think that if we want to 
> support this, it should be supported at a VFS layer. 

ACK.  However, I'm still not seeing *ANYTHING* that would look like a workable
scheme in presense of hardlinks.  Show me how to make that deadlock- and
race-free and we might very well do it in VFS.

_That_ is what's missing and it's needed no matter where it's implemented.
You want hybrid objects - you want to solve that one.  So far I've seen
nothing workable.

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Wed, 25 Aug 2004, Christoph Hellwig wrote:
>  
>
>>Over the last at least five years we've taken as much as possible
>>semantics out of the filesystems and into the VFS layer, thus having
>>a separation between the semantical layer (VFS) and the low level
>>filesystem.  Your attributes are absoultely a VFS thing and as such
>>should not happen at the filesystem layer, and no, that doesn't mean
>>they're bad per se, I just think they are a rather bad fit for Linux.
>>    
>>
>
>Now this I agree with, in the sense that I think that if we want to 
>support this, it should be supported at a VFS layer. 
>
>On the other hand, I think doing it inside the filesystem with ugly hacks 
>  
>
what is ugly? ;-/

>is an acceptable way to prototype the idea before it's been proven to 
>really be workable. Maybe it has more problems with legacy apps than we'd 
>expect..
>
>		Linus
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Wed, 25 Aug 2004, Christoph Hellwig wrote:
>  
>
>>Over the last at least five years we've taken as much as possible
>>semantics out of the filesystems and into the VFS layer, thus having
>>a separation between the semantical layer (VFS) and the low level
>>filesystem.  Your attributes are absoultely a VFS thing and as such
>>should not happen at the filesystem layer, and no, that doesn't mean
>>they're bad per se, I just think they are a rather bad fit for Linux.
>>    
>>
>
>Now this I agree with, in the sense that I think that if we want to 
>support this, it should be supported at a VFS layer. 
>
>On the other hand, I think doing it inside the filesystem with ugly hacks 
>  
>
what is ugly? ;-/

>is an acceptable way to prototype the idea before it's been proven to 
>really be workable. Maybe it has more problems with legacy apps than we'd 
>expect..
>
>		Linus
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>
>Over the last at least five years we've taken as much as possible
>semantics out of the filesystems and into the VFS layer, thus having
>a separation between the semantical layer (VFS) and the low level
>filesystem.
>
VFS is the common filesystem layer.  The only reason you think semantics 
belong in the common filesystem layer is that you are not innovating in 
your semantics, and feel content with stasis.

I don't.  I expect that semantics will get radically changed over the 
next few years as we compete with Giampaolo and whatever lesser lights 
are working at Microsoft.

>  Your attributes are absoultely a VFS thing and as such
>should not happen at the filesystem layer, and no, that doesn't mean
>they're bad per se, I just think they are a rather bad fit for Linux.
>
>So now go on and try to work together with the other peope doing VFS
>level work instead of hiding, or if you think you can't work together
>with us search a nice research OS where you can take over the VFS layer,
>if your ideas prove to be good I'm sure Linux will pick them up sooner
>or later.
>
>	Christoph
>
>
>  
>
I tell you what, use xattrs for all the half speed filesystems, and the 
users and I will use metafiles.

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>
>Over the last at least five years we've taken as much as possible
>semantics out of the filesystems and into the VFS layer, thus having
>a separation between the semantical layer (VFS) and the low level
>filesystem.
>
VFS is the common filesystem layer.  The only reason you think semantics 
belong in the common filesystem layer is that you are not innovating in 
your semantics, and feel content with stasis.

I don't.  I expect that semantics will get radically changed over the 
next few years as we compete with Giampaolo and whatever lesser lights 
are working at Microsoft.

>  Your attributes are absoultely a VFS thing and as such
>should not happen at the filesystem layer, and no, that doesn't mean
>they're bad per se, I just think they are a rather bad fit for Linux.
>
>So now go on and try to work together with the other peope doing VFS
>level work instead of hiding, or if you think you can't work together
>with us search a nice research OS where you can take over the VFS layer,
>if your ideas prove to be good I'm sure Linux will pick them up sooner
>or later.
>
>	Christoph
>
>
>  
>
I tell you what, use xattrs for all the half speed filesystems, and the 
users and I will use metafiles.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:42:06AM -0700, Hans Reiser wrote:
> Christoph Hellwig wrote:
> 
> >
> >Over the last at least five years we've taken as much as possible
> >semantics out of the filesystems and into the VFS layer, thus having
> >a separation between the semantical layer (VFS) and the low level
> >filesystem.
> >
> VFS is the common filesystem layer.  The only reason you think semantics 
> belong in the common filesystem layer is that you are not innovating in 
> your semantics, and feel content with stasis.
> 
> I don't.  I expect that semantics will get radically changed over the 
> next few years as we compete with Giampaolo and whatever lesser lights 
> are working at Microsoft.

Hans, please stop the gooddamn personal attack bullshit.

How do you for example suggestion exporting your semantics over the
network if they're not done at the VFS level?  How do you want some
clusterfilesystem support them or tmpfs?

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>
>
>How do you for example suggestion exporting your semantics over the
>network if they're not done at the VFS level?  How do you want some
>clusterfilesystem support them or tmpfs?
>
>
>  
>
by accesses to filename/metas.....

That's the beauty of simplicity, not much is needed.  xattrs on the 
other hand would be a lot of work to integrate into a cluster filesystem.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>
>
>How do you for example suggestion exporting your semantics over the
>network if they're not done at the VFS level?  How do you want some
>clusterfilesystem support them or tmpfs?
>
>
>  
>
by accesses to filename/metas.....

That's the beauty of simplicity, not much is needed.  xattrs on the 
other hand would be a lot of work to integrate into a cluster filesystem.

Hans

Re: silent semantic changes with reiser4

[Chris Mason]

On Thu, 2004-08-26 at 04:42, Hans Reiser wrote:
> Christoph Hellwig wrote:
> 
> >
> >Over the last at least five years we've taken as much as possible
> >semantics out of the filesystems and into the VFS layer, thus having
> >a separation between the semantical layer (VFS) and the low level
> >filesystem.
> >
> VFS is the common filesystem layer.  The only reason you think semantics 
> belong in the common filesystem layer is that you are not innovating in 
> your semantics, and feel content with stasis.

This misunderstanding is the heart of the problem really.  In many ways,
the VFS is the semantic layer, it defines the picture of the directory
tree and the locking rules the applications see.

There's no rule that says you can't innovate in the VFS, in fact it is
where the semantic innovations belong.  Putting them there is what
allows other filesystems to implement your semantics, and this is
exactly what linux needs if these new semantics are going to be
standardized.

In other words, writing a reiser4 plugin is one way to get to the new
semantics, but it must not be the only way.  Keeping them entirely
within reiser4 is good for the research stage, but for actual deployment
they need to be finalized and pushed up.

If the early linux filesystems had taken the same attitude you have
(don't write new filesystems, only write plugins), there would be no
framework allowing the wealth of filesystems we do have, including
reiser4.

-chris

Re: silent semantic changes with reiser4

[Hans Reiser]

Chris Mason wrote:

>
>
>If the early linux filesystems had taken the same attitude you have
>(don't write new filesystems, only write plugins), there would be no
>framework allowing the wealth of filesystems we do have, including
>reiser4.
>
>  
>
Au contraire, with my approach, there would be 10x as many filesystems, 
because people would be easily able to mix-and-match different methods 
and plugins from different filesystems together, and if you wanted the 
MySQL semantics, you would mount mysql, and if you wanted the zope 
semantics you would mount zope.

Re: silent semantic changes with reiser4

[Hans Reiser]

Chris Mason wrote:

>
>
>If the early linux filesystems had taken the same attitude you have
>(don't write new filesystems, only write plugins), there would be no
>framework allowing the wealth of filesystems we do have, including
>reiser4.
>
>  
>
Au contraire, with my approach, there would be 10x as many filesystems, 
because people would be easily able to mix-and-match different methods 
and plugins from different filesystems together, and if you wanted the 
MySQL semantics, you would mount mysql, and if you wanted the zope 
semantics you would mount zope.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 04:53:51PM -0700, Hans Reiser wrote:
> >If the early linux filesystems had taken the same attitude you have
> >(don't write new filesystems, only write plugins), there would be no
> >framework allowing the wealth of filesystems we do have, including
> >reiser4.
> >
> >  
> >
> Au contraire, with my approach, there would be 10x as many filesystems, 
> because people would be easily able to mix-and-match different methods 
> and plugins from different filesystems together, and if you wanted the 
> MySQL semantics, you would mount mysql, and if you wanted the zope 
> semantics you would mount zope.

And if you wanted to make free love you'd make free love you'd make free love
and there would be piece on all earth..

You seem to have lost contact to reality a little.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> 
> For one thing _I_ didn't decide about xattrs anyway.  And I still
> haven't seen a design from you on -fsdevel how you try to solve the
> problems with files as directories.

Hey, files-as-directories are one of my pet things, so I have to side with 
Hans on this one. I think it just makes sense. A hell of a lot more sense 
than xattrs, anyway, since it allows scripts etc standard tools to touch 
the attributes.

It's the UNIX way.

And yes, the semantics can _easily_ be solved in very unixy ways.

One way to solve it is to just realize that a final slash at the end 
implies pretty strongly that you want to treat it as a directory. So what 
you do is:

 - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
 - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)

Problem solved. Very user-friendly, and very intuitive.

Will it potentially break something? Sure. Do we care? Me, I'll take that 
kind of extension _any_ day over xattrs, that are fundamentally flawed in 
my opinion and totally useless. The argument that applications like "tar" 
won't understand the file-as-directory thing is _flawed_, since legacy 
apps won't understand xattrs either.

Oh, add a O_NOXATTRS flag to force a path lookup to only use regular
directories, the same way we have O_NOFOLLOW and friends. That allows
people to see the difference, if they care (ie a file server might decide
that it doesn't want to expose things like this).

I never liked the xattr stuff. It makes little sense, and is totally 
useless for 99.9999% of everything. I still don't see the point of it, 
except for samba. Ugly.

		Linus

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> And yes, the semantics can _easily_ be solved in very unixy ways.
> 
> One way to solve it is to just realize that a final slash at the end 
> implies pretty strongly that you want to treat it as a directory. So what 
> you do is:
> 
>  - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
>  - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
> 
> Problem solved. Very user-friendly, and very intuitive.

That would solve the O_DIRECTORY issue, the dentry aliasing still needs
work though with the semantics for link/unlink/rename.

Maybe Hans & you should start 2.7 to work this out? :)

Re: silent semantic changes with reiser4

I just want to add that I AM capable of working with the other 
filesystem developers in a team-player way, and I am happy to cooperate 
with making portions more reusable where there is serious interest from 
other filesystems in that, but Christoph is a puppy who has never 
written or designed a major filesystem from scratch, and Nikita is a big 
dog who has written stuff very few projects are lucky enough to see the 
likes of, and when Christoph insults Nikita's code, or my design 
guidance for that code, it is not going to bring out my good side.  The 
plugin and metafiles code needs many improvements, but Christoph does 
not have the expertise to understand what those needed improvements are 
because he hasn't invested the work into understanding the code.

Christoph is a bright and clever young fellow, who just hasn't had the 
years of study of the field yet.  I wish him well, and away.;-)

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

I just want to add that I AM capable of working with the other 
filesystem developers in a team-player way, and I am happy to cooperate 
with making portions more reusable where there is serious interest from 
other filesystems in that, but Christoph is a puppy who has never 
written or designed a major filesystem from scratch, and Nikita is a big 
dog who has written stuff very few projects are lucky enough to see the 
likes of, and when Christoph insults Nikita's code, or my design 
guidance for that code, it is not going to bring out my good side.  The 
plugin and metafiles code needs many improvements, but Christoph does 
not have the expertise to understand what those needed improvements are 
because he hasn't invested the work into understanding the code.

Christoph is a bright and clever young fellow, who just hasn't had the 
years of study of the field yet.  I wish him well, and away.;-)

Hans

Re: silent semantic changes with reiser4

[Chris Mason]

On Wed, 2004-08-25 at 16:41, Hans Reiser wrote:
> I just want to add that I AM capable of working with the other 
> filesystem developers in a team-player way, and I am happy to cooperate 
> with making portions more reusable where there is serious interest from 
> other filesystems in that, 

Prove it.  Stop replying for today and come back tomorrow with some
useful discussions.  Christoph suggested that some of the v4 semantics
belong in the VFS and therefore linux as a whole.  He's helping you to
make sure the semantics and fit nicely with the rest of kernel
interfaces and are race free.

Take him up on the offer.

-chris

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Wed, Aug 25, 2004 at 01:41:14PM -0700, Hans Reiser wrote:

>Christoph is a bright and clever young fellow, who just hasn't had the 
>years of study of the field yet.  I wish him well, and away.;-)

I see this is as an opportunity where you can share some of your
experience to Cristoph and many others and work to get the semantics
into VFS.

Please make this work :)

-- 
mjt

Re: silent semantic changes with reiser4

[mjt]

On Wed, Aug 25, 2004 at 01:41:14PM -0700, Hans Reiser wrote:

>Christoph is a bright and clever young fellow, who just hasn't had the 
>years of study of the field yet.  I wish him well, and away.;-)

I see this is as an opportunity where you can share some of your
experience to Cristoph and many others and work to get the semantics
into VFS.

Please make this work :)

-- 
mjt

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Wed, Aug 25, 2004 at 01:41:14PM -0700, Hans Reiser wrote:

>Christoph is a bright and clever young fellow, who just hasn't had the 
>years of study of the field yet.  I wish him well, and away.;-)

I see this is as an opportunity where you can share some of your
experience to Cristoph and many others and work to get the semantics
into VFS.

Please make this work :)

-- 
mjt

Re: silent semantic changes with reiser4

[Rik van Riel]

On Wed, 25 Aug 2004, Hans Reiser wrote:

> other filesystems in that, but Christoph is a puppy who has never 

It's not Christoph who's shown more bark than bite in this thread.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Wed, 25 Aug 2004, Hans Reiser wrote:

> other filesystems in that, but Christoph is a puppy who has never 

It's not Christoph who's shown more bark than bite in this thread.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Wed, 25 Aug 2004, Hans Reiser wrote:
>
>  
>
>>other filesystems in that, but Christoph is a puppy who has never 
>>    
>>
>
>It's not Christoph who's shown more bark than bite in this thread.
>
>  
>
The bite is at www.namesys.com/download.html


Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Wed, 25 Aug 2004, Hans Reiser wrote:
>
>  
>
>>other filesystems in that, but Christoph is a puppy who has never 
>>    
>>
>
>It's not Christoph who's shown more bark than bite in this thread.
>
>  
>
The bite is at www.namesys.com/download.html


Hans

Re: silent semantic changes with reiser4

[viro]

On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> 
> 
> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> > 
> > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > haven't seen a design from you on -fsdevel how you try to solve the
> > problems with files as directories.
> 
> Hey, files-as-directories are one of my pet things, so I have to side with 
> Hans on this one. I think it just makes sense. A hell of a lot more sense 
> than xattrs, anyway, since it allows scripts etc standard tools to touch 
> the attributes.
> 
> It's the UNIX way.

Not if you allow link(2) on them.  And not if you design and market your
stuff as a general-purpose backdoor into kernel.  Note how *EVERY* *DAMN*
*OPERATION* is made possible to override by "plugins".  Which is the reason
for deadlocks in question, BTW.

Don't fool yourself - that's what Hans is selling.  Target market: ISV.
Marketed product: a set of hooks, the wider the better, no matter how
little sense it makes.  The reason for doing that outside of core kernel:
bypassing any review and being able to control the product being sold (see
above).

Shame that it got an actual filesystem mixed in with the marketing plans
and general insanity...

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2418 bytes --]

Am Mittwoch, den 25.08.2004, 21:42 +0100 schrieb :

> > > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > > haven't seen a design from you on -fsdevel how you try to solve the
> > > problems with files as directories.
> > 
> > Hey, files-as-directories are one of my pet things, so I have to side with 
> > Hans on this one. I think it just makes sense. A hell of a lot more sense 
> > than xattrs, anyway, since it allows scripts etc standard tools to touch 
> > the attributes.
> > 
> > It's the UNIX way.
> 
> Not if you allow link(2) on them.

That doesn't make sense anyway. (actually, I tried what happens and the
result was an Oops ;))

It should be completely forbidden to link into a meta-directory or out
of such a directory. You could think of those meta-directory as a sysfs
for that inode. Of course it's not an own filesystem and that means that
there need to be a lot of security precautions in the VFS layer. Where
something like that belongs anyway, if done correctly.

>   And not if you design and market your
> stuff as a general-purpose backdoor into kernel.  Note how *EVERY* *DAMN*
> *OPERATION* is made possible to override by "plugins".  Which is the reason
> for deadlocks in question, BTW.

What do you mean? If you tell that file that you want it to be
compressed or encrypted or modify some attributes (like ACLs) this isn't
necessarily a backdoor.

> Don't fool yourself - that's what Hans is selling.  Target market: ISV.
> Marketed product: a set of hooks, the wider the better, no matter how
> little sense it makes.  The reason for doing that outside of core kernel:
> bypassing any review and being able to control the product being sold (see
> above).

Yes, I don't think it was a good idea either. Probably someone should
remove these features and make it a "normal" filesystem. The people who
need it now can turn it on again and a real solution could be worked out
in Linux 2.7.

I wouldn't use it on a public server anyway now because I'm not
convinced some malicious guy could find a way to exploit that. What if
you changed into a meta directory using ftp and some manage to break
things? This might be very dangerous.

I personally think that the idea of doing something like this (I'm not
speaking of the current implementation which I think is really bad) is
the right way to go in the long term.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2418 bytes --]

Am Mittwoch, den 25.08.2004, 21:42 +0100 schrieb :

> > > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > > haven't seen a design from you on -fsdevel how you try to solve the
> > > problems with files as directories.
> > 
> > Hey, files-as-directories are one of my pet things, so I have to side with 
> > Hans on this one. I think it just makes sense. A hell of a lot more sense 
> > than xattrs, anyway, since it allows scripts etc standard tools to touch 
> > the attributes.
> > 
> > It's the UNIX way.
> 
> Not if you allow link(2) on them.

That doesn't make sense anyway. (actually, I tried what happens and the
result was an Oops ;))

It should be completely forbidden to link into a meta-directory or out
of such a directory. You could think of those meta-directory as a sysfs
for that inode. Of course it's not an own filesystem and that means that
there need to be a lot of security precautions in the VFS layer. Where
something like that belongs anyway, if done correctly.

>   And not if you design and market your
> stuff as a general-purpose backdoor into kernel.  Note how *EVERY* *DAMN*
> *OPERATION* is made possible to override by "plugins".  Which is the reason
> for deadlocks in question, BTW.

What do you mean? If you tell that file that you want it to be
compressed or encrypted or modify some attributes (like ACLs) this isn't
necessarily a backdoor.

> Don't fool yourself - that's what Hans is selling.  Target market: ISV.
> Marketed product: a set of hooks, the wider the better, no matter how
> little sense it makes.  The reason for doing that outside of core kernel:
> bypassing any review and being able to control the product being sold (see
> above).

Yes, I don't think it was a good idea either. Probably someone should
remove these features and make it a "normal" filesystem. The people who
need it now can turn it on again and a real solution could be worked out
in Linux 2.7.

I wouldn't use it on a public server anyway now because I'm not
convinced some malicious guy could find a way to exploit that. What if
you changed into a meta directory using ftp and some manage to break
things? This might be very dangerous.

I personally think that the idea of doing something like this (I'm not
speaking of the current implementation which I think is really bad) is
the right way to go in the long term.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Andrea Arcangeli]

On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote:
> It should be completely forbidden to link into a meta-directory or out
> of such a directory. [..]

agreed.

> Yes, I don't think it was a good idea either. Probably someone should

I personally would like plugins only if the API they use wouldn't allow
them to corrupt the underlying fs, I'm not sure if this is the case with
reiserfs4.

About the backwards compatibility, another option is to add a O_FILEDIR
and have bash learn about it when you cd into a file. No magic with the
slashes then.

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrea Arcangeli wrote:

>On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote:
>  
>
>>It should be completely forbidden to link into a meta-directory or out
>>of such a directory. [..]
>>    
>>
>
>agreed.
>
>  
>
>>Yes, I don't think it was a good idea either. Probably someone should
>>    
>>
>
>I personally would like plugins only if the API they use wouldn't allow
>them to corrupt the underlying fs, I'm not sure if this is the case with
>reiserfs4.
>  
>
You compile crap into the kernel, you are screwed.  We have not and will 
not change that.

Reiser4 plugins are not for end users to download from amazon.com, they 
are for weekend hackers to send me a cool plugin for me to review, 
assign a plugin id to, and send to Linus in the next release.  Sometimes 
being less ambitious works better, and dynamically loaded plugins would 
have made the infrastructure too bulky to be fun and fast.

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrea Arcangeli wrote:

>On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote:
>  
>
>>It should be completely forbidden to link into a meta-directory or out
>>of such a directory. [..]
>>    
>>
>
>agreed.
>
>  
>
>>Yes, I don't think it was a good idea either. Probably someone should
>>    
>>
>
>I personally would like plugins only if the API they use wouldn't allow
>them to corrupt the underlying fs, I'm not sure if this is the case with
>reiserfs4.
>  
>
You compile crap into the kernel, you are screwed.  We have not and will 
not change that.

Reiser4 plugins are not for end users to download from amazon.com, they 
are for weekend hackers to send me a cool plugin for me to review, 
assign a plugin id to, and send to Linus in the next release.  Sometimes 
being less ambitious works better, and dynamically loaded plugins would 
have made the infrastructure too bulky to be fun and fast.

Re: silent semantic changes with reiser4

[Andrea Arcangeli]

On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
> Reiser4 plugins are not for end users to download from amazon.com, they 
> are for weekend hackers to send me a cool plugin for me to review, 
> assign a plugin id to, and send to Linus in the next release.  Sometimes 

then what's the difference in having the plugin fixed in stone into
reiserfs? That's my whole point. Get the patch from the weekend hacker,
check it, send the patch to Linus to add the new feature to reiser4,
just call it "feature" not plugin. That's how it works normally for
everything. Many fs have many features, many of them optional.  you
wouldn't need to build any hook infrastructure either that way. Hooks
would be needed if this wasn't open source me thinks. Or if you want
people to fetch the module from amazon without your review.

Another reason I could see the modularization/hooking useful is if those
feature would take lots of kernel space, but this sure isn't the case
for reiserfs, infact having modules would waste _more_ ram due the half
wasted page allocation for the module text. The only single reason we
use modules is to avoid wasting tons of ram by loading every possible
device driver on earth, it's not that we use modules because they're
more flexible, if something they're more fragile. I never use modules in
my test kernel just to go fast (because I self compile them). The last
good thing of the modules is during development you don't need to reboot
the machine to test a kernel change in a driver, but you don't need that
with reiserfs since you can make the fs itself a module (just change the
name of the fs, AFIK you do that all the time already).

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

Am Donnerstag, den 26.08.2004, 13:28 +0200 schrieb Andrea Arcangeli:

> then what's the difference in having the plugin fixed in stone into
> reiserfs? That's my whole point.

I think you're not exactly understanding what a reiser4 "plugin" is.

Currently plugins are not modules. It just means that there's a defined
interface between the reiser4 core and the plugins. Just like you could
see filesystems as "VFS plugins".

In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort
of glue code between the Reiser4 storage tree and the Linux VFS)


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

Am Donnerstag, den 26.08.2004, 13:28 +0200 schrieb Andrea Arcangeli:

> then what's the difference in having the plugin fixed in stone into
> reiserfs? That's my whole point.

I think you're not exactly understanding what a reiser4 "plugin" is.

Currently plugins are not modules. It just means that there's a defined
interface between the reiser4 core and the plugins. Just like you could
see filesystems as "VFS plugins".

In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort
of glue code between the Reiser4 storage tree and the Linux VFS)


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Andrea Arcangeli]

On Thu, Aug 26, 2004 at 01:44:59PM +0200, Christophe Saout wrote:
> I think you're not exactly understanding what a reiser4 "plugin" is.
> 
> Currently plugins are not modules. It just means that there's a defined
> interface between the reiser4 core and the plugins. Just like you could
> see filesystems as "VFS plugins".

but filesystems are exactly always modules (in precompiled kernels at
least).

Anyways I don't see any visible EXPORT_SYMBOL in reiser4.

> 
> In fact, reiser4 plugins are
> - users of the reiser4 core API
> - some of them are implementing Linux VFS methods (thus being some sort
> of glue code between the Reiser4 storage tree and the Linux VFS)

then it seems a bit misleading to call those things plugins.

As you wrote those are "users of the reiser4 core API", with plugins I
always meant dynamically loadable things, like plugins for web browers
(hence kernel modules in kernel space). While here apparently you can't
plugin anything at runtime, it's just code that uses the reiser4 core
API that can be modified with a kernel patch as usual.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

Am Donnerstag, den 26.08.2004, 14:16 +0200 schrieb Andrea Arcangeli:

> > Currently plugins are not modules. It just means that there's a defined
> > interface between the reiser4 core and the plugins. Just like you could
> > see filesystems as "VFS plugins".
> 
> but filesystems are exactly always modules (in precompiled kernels at
> least).

Yes, of cousre.

> Anyways I don't see any visible EXPORT_SYMBOL in reiser4.

Yes, there's no module infrastructure there. These "plugins" need to be
linked at compile time. Some plugins need to be anyway since the the
filesystem is useless without them. Others whould be optional features.

> > In fact, reiser4 plugins are
> > - users of the reiser4 core API
> > - some of them are implementing Linux VFS methods (thus being some sort
> > of glue code between the Reiser4 storage tree and the Linux VFS)
> 
> then it seems a bit misleading to call those things plugins.
> 
> As you wrote those are "users of the reiser4 core API", with plugins I
> always meant dynamically loadable things, like plugins for web browers
> (hence kernel modules in kernel space). While here apparently you can't
> plugin anything at runtime, it's just code that uses the reiser4 core
> API that can be modified with a kernel patch as usual.

Right. The name is probably at bit misleading. But assuming the
compression "plugin" whould be dynamically loadable at some later point
it would be a plugin then?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

Am Donnerstag, den 26.08.2004, 14:16 +0200 schrieb Andrea Arcangeli:

> > Currently plugins are not modules. It just means that there's a defined
> > interface between the reiser4 core and the plugins. Just like you could
> > see filesystems as "VFS plugins".
> 
> but filesystems are exactly always modules (in precompiled kernels at
> least).

Yes, of cousre.

> Anyways I don't see any visible EXPORT_SYMBOL in reiser4.

Yes, there's no module infrastructure there. These "plugins" need to be
linked at compile time. Some plugins need to be anyway since the the
filesystem is useless without them. Others whould be optional features.

> > In fact, reiser4 plugins are
> > - users of the reiser4 core API
> > - some of them are implementing Linux VFS methods (thus being some sort
> > of glue code between the Reiser4 storage tree and the Linux VFS)
> 
> then it seems a bit misleading to call those things plugins.
> 
> As you wrote those are "users of the reiser4 core API", with plugins I
> always meant dynamically loadable things, like plugins for web browers
> (hence kernel modules in kernel space). While here apparently you can't
> plugin anything at runtime, it's just code that uses the reiser4 core
> API that can be modified with a kernel patch as usual.

Right. The name is probably at bit misleading. But assuming the
compression "plugin" whould be dynamically loadable at some later point
it would be a plugin then?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 02:16:30PM +0200, Andrea Arcangeli wrote:
> On Thu, Aug 26, 2004 at 01:44:59PM +0200, Christophe Saout wrote:
> > I think you're not exactly understanding what a reiser4 "plugin" is.
> > 
> > Currently plugins are not modules. It just means that there's a defined
> > interface between the reiser4 core and the plugins. Just like you could
> > see filesystems as "VFS plugins".
> 
> but filesystems are exactly always modules (in precompiled kernels at
> least).
> 
> Anyways I don't see any visible EXPORT_SYMBOL in reiser4.

they would be there, I think.
> 
> > 
> > In fact, reiser4 plugins are
> > - users of the reiser4 core API
> > - some of them are implementing Linux VFS methods (thus being some sort
> > of glue code between the Reiser4 storage tree and the Linux VFS)
> 
> then it seems a bit misleading to call those things plugins.

There should be infracture to change plugin on fly, and choosing object's
plugins at object's creation time.  Currently only certain object's plugins
can be changed through <object>/metas/plugin interface. 

> 
> As you wrote those are "users of the reiser4 core API", with plugins I
> always meant dynamically loadable things, like plugins for web browers
> (hence kernel modules in kernel space). While here apparently you can't
> plugin anything at runtime, it's just code that uses the reiser4 core
> API that can be modified with a kernel patch as usual.

-- 
Alex.

Re: silent semantic changes with reiser4

[Hans Reiser]

Alex Zarochentsev wrote:

>
>
>There should be infracture to change plugin on fly, and choosing object's
>plugins at object's creation time.  Currently only certain object's plugins
>can be changed through <object>/metas/plugin interface. 
>  
>
To be more clear for those not familiar with our project, there is much 
more work to be done, and one of the areas that (extremely) needs more 
work is changing plugins on the fly and clean interfaces for specifying 
them at creation time.

Re: silent semantic changes with reiser4

Andrea Arcangeli wrote:

>On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
>  
>
>>Reiser4 plugins are not for end users to download from amazon.com, they 
>>are for weekend hackers to send me a cool plugin for me to review, 
>>assign a plugin id to, and send to Linus in the next release.  Sometimes 
>>    
>>
>
>then what's the difference in having the plugin fixed in stone into
>reiserfs? That's my whole point. Get the patch from the weekend hacker,
>check it, send the patch to Linus to add the new feature to reiser4,
>just call it "feature" not plugin. 
>
Think of it as being like VFS, where you can plugin new filesystems.  
Only in this case, you can plugin new kinds of files, and everything you 
need to implement those new kinds of files (items, nodes, keys, 
processing before flushing to disk, etc.) also gets implemented as a 
plugin.  It is an Uber-VFS.

> The only single reason we
>use modules is to avoid wasting tons of ram by loading every possible
>device driver on earth,
>
I am not concerned about ram in this design, I want nifty new kinds of 
files easy crafted over a weekend by sysadmins working in Canada and 
Guatemala.  Software engineering cost is what matters to me ( I turned 
40, so I think different now....;-) )

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrea Arcangeli wrote:

>On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
>  
>
>>Reiser4 plugins are not for end users to download from amazon.com, they 
>>are for weekend hackers to send me a cool plugin for me to review, 
>>assign a plugin id to, and send to Linus in the next release.  Sometimes 
>>    
>>
>
>then what's the difference in having the plugin fixed in stone into
>reiserfs? That's my whole point. Get the patch from the weekend hacker,
>check it, send the patch to Linus to add the new feature to reiser4,
>just call it "feature" not plugin. 
>
Think of it as being like VFS, where you can plugin new filesystems.  
Only in this case, you can plugin new kinds of files, and everything you 
need to implement those new kinds of files (items, nodes, keys, 
processing before flushing to disk, etc.) also gets implemented as a 
plugin.  It is an Uber-VFS.

> The only single reason we
>use modules is to avoid wasting tons of ram by loading every possible
>device driver on earth,
>
I am not concerned about ram in this design, I want nifty new kinds of 
files easy crafted over a weekend by sysadmins working in Canada and 
Guatemala.  Software engineering cost is what matters to me ( I turned 
40, so I think different now....;-) )

Re: silent semantic changes with reiser4

[Andrea Arcangeli]

On Thu, Aug 26, 2004 at 04:53:23PM -0700, Hans Reiser wrote:
> Think of it as being like VFS, where you can plugin new filesystems.  
> Only in this case, you can plugin new kinds of files, and everything you 
> need to implement those new kinds of files (items, nodes, keys, 
> processing before flushing to disk, etc.) also gets implemented as a 
> plugin.  It is an Uber-VFS.

I'm just confused by the naming, following your comparison with it being
like VFS, I don't call reiser4 a "plugin" of the VFS.

The vfs offers an interface for the fs to register into
it. Like the scsi highlevel code also offers an interface for the
lowlevel driver to register into it, but we don't call scsi lowlevel
drivers "plugins".

the ext3 directory code calling ext3_getblk would be also a plugin in
your vocabulary, and the ext3_getblk call would be plugin-hook.

> I am not concerned about ram in this design, I want nifty new kinds of 
> files easy crafted over a weekend by sysadmins working in Canada and 
> Guatemala.  Software engineering cost is what matters to me ( I turned 
> 40, so I think different now....;-) )

Adding proper layering and abstractions is sure a good thing, if you can
turn the street into a crowd of reiser4 kernel hackers because your API
is so clean that any sysadmin can craft new kind of files over the
weekend you're welcome to go ahead ;). (I just doubt the "plugin"
terminology can make a difference by the time people start scratching
their head into the code, the design and the clean abstraction of the
code will make a difference instead)

Re: silent semantic changes with reiser4

[Yury Umanets]

Andrea Arcangeli wrote:

>On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
>  
>
>>Reiser4 plugins are not for end users to download from amazon.com, they 
>>are for weekend hackers to send me a cool plugin for me to review, 
>>assign a plugin id to, and send to Linus in the next release.  Sometimes 
>>    
>>
>
>then what's the difference in having the plugin fixed in stone into
>reiserfs? That's my whole point. Get the patch from the weekend hacker,
>check it, send the patch to Linus to add the new feature to reiser4,
>just call it "feature" not plugin. That's how it works normally for
>everything. Many fs have many features, many of them optional.  you
>wouldn't need to build any hook infrastructure either that way. Hooks
>would be needed if this wasn't open source me thinks. Or if you want
>people to fetch the module from amazon without your review.
>
>Another reason I could see the modularization/hooking useful is if those
>feature would take lots of kernel space, but this sure isn't the case
>for reiserfs, infact having modules would waste _more_ ram due the half
>wasted page allocation for the module text. The only single reason we
>use modules is to avoid wasting tons of ram by loading every possible
>device driver on earth, it's not that we use modules because they're
>more flexible, if something they're more fragile. I never use modules in
>my test kernel just to go fast (because I self compile them). The last
>good thing of the modules is during development you don't need to reboot
>the machine to test a kernel change in a driver, but you don't need that
>with reiserfs since you can make the fs itself a module (just change the
>name of the fs, AFIK you do that all the time already).
>  
>
There lots of good things about plugins as they are in reiser4. And 
probably the best one (as for me) is that, reiser4 core code (quite 
complex thing) does not need to be changed after adding new plugins 
(read new features).

It is written once and provides some basic functionality (balancing, 
atoms, etc.), which works well with all kinds of plugins if they 
implement required functions from plugin interface.

As for me, this looks like Linux's VFS. Sure, if I want to add some very 
new filesystem with features which were not foreseen,  and VFS does not 
provide needed interface, it will need some changes. But in simple case, 
when I just want to develop once more disk filesystem, which say will be 
different from another disk filesystem in only that respect it manages 
its metadata in different manner and aims to achieve better performance 
this way. In this case I will not need to chnage VFS at all. The same 
about reiser4 core and new plugins.

>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>


-- 
umka

Re: silent semantic changes with reiser4

[Yury Umanets]

Andrea Arcangeli wrote:

>On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
>  
>
>>Reiser4 plugins are not for end users to download from amazon.com, they 
>>are for weekend hackers to send me a cool plugin for me to review, 
>>assign a plugin id to, and send to Linus in the next release.  Sometimes 
>>    
>>
>
>then what's the difference in having the plugin fixed in stone into
>reiserfs? That's my whole point. Get the patch from the weekend hacker,
>check it, send the patch to Linus to add the new feature to reiser4,
>just call it "feature" not plugin. That's how it works normally for
>everything. Many fs have many features, many of them optional.  you
>wouldn't need to build any hook infrastructure either that way. Hooks
>would be needed if this wasn't open source me thinks. Or if you want
>people to fetch the module from amazon without your review.
>
>Another reason I could see the modularization/hooking useful is if those
>feature would take lots of kernel space, but this sure isn't the case
>for reiserfs, infact having modules would waste _more_ ram due the half
>wasted page allocation for the module text. The only single reason we
>use modules is to avoid wasting tons of ram by loading every possible
>device driver on earth, it's not that we use modules because they're
>more flexible, if something they're more fragile. I never use modules in
>my test kernel just to go fast (because I self compile them). The last
>good thing of the modules is during development you don't need to reboot
>the machine to test a kernel change in a driver, but you don't need that
>with reiserfs since you can make the fs itself a module (just change the
>name of the fs, AFIK you do that all the time already).
>  
>
There lots of good things about plugins as they are in reiser4. And
probably the best one (as for me) is that, reiser4 core code (quite
complex thing) does not need to be changed after adding new plugins
(read new features).

It is written once and provides some basic functionality (balancing,
atoms, etc.), which works well with all kinds of plugins if they
implement required functions from plugin interface.

As for me, this looks like Linux's VFS. Sure, if I want to add some very
new filesystem with features which were not foreseen,  and VFS does not
provide needed interface, it will need some changes. But in simple case,
when I just want to develop once more disk filesystem, which say will be
different from another disk filesystem in only that respect it manages
its metadata in different manner and aims to achieve better performance
this way. In this case I will not need to chnage VFS at all. The same
about reiser4 core and new plugins.

>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>


-- 
umka

Re: silent semantic changes with reiser4

[Yury Umanets]

Andrea Arcangeli wrote:

>On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
>  
>
>>Reiser4 plugins are not for end users to download from amazon.com, they 
>>are for weekend hackers to send me a cool plugin for me to review, 
>>assign a plugin id to, and send to Linus in the next release.  Sometimes 
>>    
>>
>
>then what's the difference in having the plugin fixed in stone into
>reiserfs? That's my whole point. Get the patch from the weekend hacker,
>check it, send the patch to Linus to add the new feature to reiser4,
>just call it "feature" not plugin. That's how it works normally for
>everything. Many fs have many features, many of them optional.  you
>wouldn't need to build any hook infrastructure either that way. Hooks
>would be needed if this wasn't open source me thinks. Or if you want
>people to fetch the module from amazon without your review.
>
>Another reason I could see the modularization/hooking useful is if those
>feature would take lots of kernel space, but this sure isn't the case
>for reiserfs, infact having modules would waste _more_ ram due the half
>wasted page allocation for the module text. The only single reason we
>use modules is to avoid wasting tons of ram by loading every possible
>device driver on earth, it's not that we use modules because they're
>more flexible, if something they're more fragile. I never use modules in
>my test kernel just to go fast (because I self compile them). The last
>good thing of the modules is during development you don't need to reboot
>the machine to test a kernel change in a driver, but you don't need that
>with reiserfs since you can make the fs itself a module (just change the
>name of the fs, AFIK you do that all the time already).
>  
>
There lots of good things about plugins as they are in reiser4. And
probably the best one (as for me) is that, reiser4 core code (quite
complex thing) does not need to be changed after adding new plugins
(read new features).

It is written once and provides some basic functionality (balancing,
atoms, etc.), which works well with all kinds of plugins if they
implement required functions from plugin interface.

As for me, this looks like Linux's VFS. Sure, if I want to add some very
new filesystem with features which were not foreseen,  and VFS does not
provide needed interface, it will need some changes. But in simple case,
when I just want to develop once more disk filesystem, which say will be
different from another disk filesystem in only that respect it manages
its metadata in different manner and aims to achieve better performance
this way. In this case I will not need to chnage VFS at all. The same
about reiser4 core and new plugins.

>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>


-- 
umka

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Wed, 25 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
>
> On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> > 
> > It's the UNIX way.
> 
> Not if you allow link(2) on them.

Heh. I don't think that's a very strong argument against being "unixy", 
considering how traditional unix _used_ to handle directories.

mkdir/rmdir/rename only came later. Now, obviously they did come later for 
a good reason, but still..

The interesting part is that thanks to the dcache, we should be perfectly
able to actually _see_ circular links etc, so some of the problems with 
linking directories should actually be quite solvable - something that is 
_not_ true for a traditional UNIX VFS layer.

Of course, the dcache introduces some new problems of its own wrt
directory aliasing, but I don't actually think that should be fundamental
either. Treating them more as a "static mountpoint" from an FS angle and
less as a traditional Unix hardlink should be doable, I'd have thought.

(Also, it's entirely possible that the filesystem may not support some of
the more esoteric linking/renaming operations. For example, in a
traditional xattrs setup where the xattr is linked on-disk with the file
it is associated with, you simply _can't_ link it somewhere else, or
rename it to any other directory. That's not a VFS layer issue, obviously,
but I thought I'd bring up the point that file-as-dir cases may have
limitations that normal files don't have).

>  And not if you design and market your stuff as a general-purpose
> backdoor into kernel.

Now that's a separate argument, and not one I'm personally interested in
arguing at least right now. I haven't actually looked at the reiser4 code,
so I'm really _only_ arguing against special-case attributes.

		Linus

Re: silent semantic changes with reiser4

[viro]

On Wed, Aug 25, 2004 at 02:00:01PM -0700, Linus Torvalds wrote:
> Of course, the dcache introduces some new problems of its own wrt
> directory aliasing, but I don't actually think that should be fundamental
> either. Treating them more as a "static mountpoint" from an FS angle and
> less as a traditional Unix hardlink should be doable, I'd have thought.

Yeah, if we ditch the "mountpoints are busy and untouchable" stuff.  Which
I'd love to, but it's a hell of a visible (and admin-visible) change.

FWIW, current deadlocks are unrelated to actual operation succeeding.
Look: we have sys_link() making sure that parent of target is a directory
(PATH_LOOKUP, in a "it has ->lookup()" sense), then locking target's parent,
then checking that it has ->link() (everyone on reiser4 does) and then
checking that source (old link to file) is *not* a directory (in S_ISDIR
sense).  Then we lock source.

Note that currently it's OK - we get "all non-directories are always locked
after all directories".  With filesystem that provides hybrid objects with
non-NULL ->link() it's not true and we are in deadlock country.  Before
we get anywhere near fs code.

I'm not saying that this particular instance is hard to fix, but it wasn't
even looked at.  All it would take is checking the description of current
locking scheme and looking through the proof of correctness (present in the
tree).  That's the first point where said proof breaks if we have hybrids.
And it's what, about 4 screenfuls of text?

I have no problems with discussing such stuff and no problems with having it
merged if it actually works.  But let's start with something better than
"let's hope nothing breaks if we just add such objects and do nothing else,
'cause hybridi files/directories are good, mmmkay?"

Re: silent semantic changes with reiser4

[Jamie Lokier]

This message suggests a way to extend the VFS safe locking rules to
include files-as-directories.

viro@parcelfarce.linux.theplanet.co.uk wrote:
> Note that currently it's OK - we get "all non-directories are always locked
> after all directories".  With filesystem that provides hybrid objects with
> non-NULL ->link() it's not true and we are in deadlock country.  Before
> we get anywhere near fs code.

Is this a problem if we treat entering a file-as-directory as crossing
a mount point (i.e. like auto-mounting)?

Simply doing a path walk would lock the file and then cross the mount
point to a directory.

A way to ensure that preserves the lock order is to require that the
metadata is in a different filesystem to its file (i.e. not crossing a
bind mount to the same filesystem).

That has the side effect of preventing hard links between metadata
files and non-metadata, which in my opinion is fine.

Path walking will lock the file, and then lock the directory on a
different filesystem.  Lock order is still safe, provided a strict
order is maintained between the two filesystems.

The strict order is ensured by preventing bind mounts which create a
path cycle containing a file->metadata edge.  One way to ensure that
is to prevent mounts on the metadata filesystems, but the rule doesn't
have to be that strict.  This condition only needs to be checked in
the mount() syscall.

-- Jamie

Re: silent semantic changes with reiser4

[viro]

On Thu, Aug 26, 2004 at 01:11:52AM +0100, Jamie Lokier wrote:
> Is this a problem if we treat entering a file-as-directory as crossing
> a mount point (i.e. like auto-mounting)?

Yes - mountpoints can't be e.g. unlinked.  Moreover, having directory
mounted on non-directory is also an interesting situation.

> Simply doing a path walk would lock the file and then cross the mount
> point to a directory.

*Ugh*

What would happen if you open that directory or chdir there?  If it's
"underlying file stays locked" - we are in even more obvious deadlocks.

> A way to ensure that preserves the lock order is to require that the
> metadata is in a different filesystem to its file (i.e. not crossing a
> bind mount to the same filesystem).
> 
> That has the side effect of preventing hard links between metadata
> files and non-metadata, which in my opinion is fine.

We don't actually need a different fs - different vfsmount will do just fine.

> The strict order is ensured by preventing bind mounts which create a
> path cycle containing a file->metadata edge.  One way to ensure that
> is to prevent mounts on the metadata filesystems, but the rule doesn't
> have to be that strict.  This condition only needs to be checked in
> the mount() syscall.

You really don't want to lock mountpoint on path lookup, so I don't see
how that would be relevant - it's a hell to clean up, for one thing
(I've crossed ten mountpoints on the way, when do I unlock them and
how do I prevent deadlocks from that?)  Besides, different namespaces
can have completely different mount trees, so tracking down all that
stuff would be hell in its own right.

The main issue I see with all schemes in that direction (and something
like that could be made workable) is the semantics of unlink() on
mountpoints.  *Especially* with users being able to see attributes of
files they do not own (e.g. reiser4 mode/uid/gid stuff).  Ability to
pin down any damn file on the system and make it impossible to replace
is not something you want to give to any user.

Re: silent semantic changes with reiser4

[Jamie Lokier]

viro@parcelfarce.linux.theplanet.co.uk wrote:
> > Is this a problem if we treat entering a file-as-directory as crossing
> > a mount point (i.e. like auto-mounting)?
> 
> Yes - mountpoints can't be e.g. unlinked.  Moreover, having directory
> mounted on non-directory is also an interesting situation.

Ok, so can we make it so mountpoints can be unlinked? :)

The mount would continue to exist, but with no name, until its last
user disappears.

> > Simply doing a path walk would lock the file and then cross the mount
> > point to a directory.
> 
> *Ugh*
> 
> What would happen if you open that directory or chdir there?  If it's
> "underlying file stays locked" - we are in even more obvious deadlocks.

I think the underlying file does not stay locked, and once you've
entered it as a directory, it can be unlinked.

If you have the directory open or chdir into it, then it _may_ have
the effect of keeping the file's storage allocated when you unlink it
-- just like when a file is unlinked while opened.  As that is not a
user-visible property, it's a filesystem-specific implementation
detail as to whether it keeps the file's data in existence while the
metadata directories and/or files are open.

> > A way to ensure that preserves the lock order is to require that the
> > metadata is in a different filesystem to its file (i.e. not crossing a
> > bind mount to the same filesystem).
> > 
> > That has the side effect of preventing hard links between metadata
> > files and non-metadata, which in my opinion is fine.
> 
> We don't actually need a different fs - different vfsmount will do just fine.

> > The strict order is ensured by preventing bind mounts which create a
> > path cycle containing a file->metadata edge.  One way to ensure that
> > is to prevent mounts on the metadata filesystems, but the rule doesn't
> > have to be that strict.  This condition only needs to be checked in
> > the mount() syscall.
> 
> You really don't want to lock mountpoint on path lookup, so I don't see
> how that would be relevant - it's a hell to clean up, for one thing
> (I've crossed ten mountpoints on the way, when do I unlock them and
> how do I prevent deadlocks from that?)  Besides, different namespaces
> can have completely different mount trees, so tracking down all that
> stuff would be hell in its own right.

I didn't mean locking a chain of mountpoints, I meant the temporary
state where two dentries and/or inodes are locked, parent and child,
during a path walk.  However I'm not very familiar with that part of
the VFS and I see that the current RCU dcache might not lock that much
during a path walk.

> The main issue I see with all schemes in that direction (and something
> like that could be made workable) is the semantics of unlink() on
> mountpoints.  *Especially* with users being able to see attributes of
> files they do not own (e.g. reiser4 mode/uid/gid stuff).  Ability to
> pin down any damn file on the system and make it impossible to replace
> is not something you want to give to any user.

I agree, users shouldn't be able to pin down a file.

I think unlink() should succeed on a file while something is visiting
inside its metadata directory.

It's a filesystem quality-of-implementation feature whether that
actually releases the file's data.  It's a desirable feature because
one user shouldn't be able to pin another user's quota'd data if they
don't have permission to open the file, but if it's not implemented by
a filesystem then it doesn't break anything fundamental.

It's a semantics question whether unlinking a file makes the metadata
(i.e. "uid", "mode", "content-type" etc.) disappear at the same time,
or if the metadata stays around until the last visitor leaves it.  A
filesystem might be able to keep the metadata in existence even if it
deletes the file's storage on unlink(), but it would be nice for the
VFS to declare which semantic is preferred.

One of the big potential uses for file-as-directory is to go inside
archive files, ELF files, .iso files and so on in a convenient way.
In those cases, if you open one of the virtually generated "archive
content" files, then you might expect the data to continue to exist
after the underlying file is unlinked.  I think that's reasonable:
being inside an archive is very similar to having it open.  There is
no quota pinning problem with this, because "archive content" files
should inherit permission restrictions from the underlying file.  If
you can't read the file, then you can't read it's unpacked contents.

(reiser4 doesn't offer that last feature, but any of the myriad
userspace filesystem hooks could offer it if the VFS has approprate
auto-mounting file-as-directory hooks).

-- Jamie

Re: silent semantic changes with reiser4

[viro]

On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote:
> viro@parcelfarce.linux.theplanet.co.uk wrote:
> > > Is this a problem if we treat entering a file-as-directory as crossing
> > > a mount point (i.e. like auto-mounting)?
> > 
> > Yes - mountpoints can't be e.g. unlinked.  Moreover, having directory
> > mounted on non-directory is also an interesting situation.
> 
> Ok, so can we make it so mountpoints can be unlinked? :)

User-visible change of behaviour and IIRC a SuS violation on top of that.
 
> I think the underlying file does not stay locked, and once you've
> entered it as a directory, it can be unlinked.

So why lock it at all in that case?

> I didn't mean locking a chain of mountpoints, I meant the temporary
> state where two dentries and/or inodes are locked, parent and child,
> during a path walk.  However I'm not very familiar with that part of
> the VFS and I see that the current RCU dcache might not lock that much
> during a path walk.

Never had been needed on crossing mountpoints, actually.

> I agree, users shouldn't be able to pin down a file.
> 
> I think unlink() should succeed on a file while something is visiting
> inside its metadata directory.

See above.  Again, the fundamental problem with that is allowing unlink
and friends on a mountpoint.  I would love to do that, but it always
generated -EBUSY on all Unices.  Linux got a bit more users and userland
code than Plan 9 - they can afford such changes, but...

And yes, from the kernel POV it's trivial to do - witness the MNT_DETACH
codepath in umount - it's much simpler than "normal" umount exactly because
it doesn't try to emulate old "it's busy, can't umount" behaviour.

With umount we could introduce "don't bother with that shit" flag.  With
unlink() we would have to make that default behaviour to be useful.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>
>
>One of the big potential uses for file-as-directory is to go inside
>archive files, ELF files, .iso files and so on in a convenient way.
>  
>
Yes, this was part of the plan, tar file-directory plugins would be cute.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>
>
>One of the big potential uses for file-as-directory is to go inside
>archive files, ELF files, .iso files and so on in a convenient way.
>  
>
Yes, this was part of the plan, tar file-directory plugins would be cute.

Re: silent semantic changes with reiser4

[Adrian Bunk]

On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote:
> Jamie Lokier wrote:
> 
> >One of the big potential uses for file-as-directory is to go inside
> >archive files, ELF files, .iso files and so on in a convenient way.
> > 
> >
> Yes, this was part of the plan, tar file-directory plugins would be cute.

Silly question:

GNU Midnight Commander allows for ages to go into e.g. tar files, so I 
know the benefits of this. Additionally, in GNU Midnight Commander, this 
works no matter which file system I use (e.g. it works on iso9660), and 
it even works the same way on other OS's like e.g. Solaris and NetBSD.

What is the technical reason why a tar plugin should be reiser4 
specific, instead of a generic VFS or userspace solution that would 
allow the same also on other fs like e.g. iso9660?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

Am Donnerstag, den 26.08.2004, 16:05 +0200 schrieb Adrian Bunk:

> > Yes, this was part of the plan, tar file-directory plugins would be cute.
> 
> Silly question:
> 
> GNU Midnight Commander allows for ages to go into e.g. tar files, so I 
> know the benefits of this. Additionally, in GNU Midnight Commander, this 
> works no matter which file system I use (e.g. it works on iso9660), and 
> it even works the same way on other OS's like e.g. Solaris and NetBSD.
> 
> What is the technical reason why a tar plugin should be reiser4 
> specific, instead of a generic VFS or userspace solution that would 
> allow the same also on other fs like e.g. iso9660?

I don't think that this should be done inside the kernel. tar is a
perfectly valid userspace application.

What reiser4 can do, but the VFS can't is to insert or remove data in
the middle of a file. Adding this above the page cache would probably be
almost impossible (truncate seems already complicated enough).

But below page-cache and thus invisible from the VFS or the applications
it is extremely useful for the compression plugin. When changing some
data in the middle of the file the compressed data might grow in size.
reiser4 can handle this inside the storage layer and doesn't mess with
the rest of the filesystem.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Matthew Wilcox]

On Thu, Aug 26, 2004 at 04:25:13PM +0200, Christophe Saout wrote:
> What reiser4 can do, but the VFS can't is to insert or remove data in
> the middle of a file. Adding this above the page cache would probably be
> almost impossible (truncate seems already complicated enough).
> 
> But below page-cache and thus invisible from the VFS or the applications
> it is extremely useful for the compression plugin. When changing some
> data in the middle of the file the compressed data might grow in size.
> reiser4 can handle this inside the storage layer and doesn't mess with
> the rest of the filesystem.

Never say never ... regular readers of -fsdevel already know what I'm
going to say so can skip this post.

When compressing data on an ext2-like filesystem, you can use the
property of "holes" to deallocate disc blocks in the middle of a file.
It works like this:

Split the file into 64k chunks
Compress each chunk individually, then write them to disc on 64k boundaries,
leaving gaps.
Now you can quickly seek to the appropriate 64k block for the data you're
looking for and decompress just that 64k block.

This is how RISCiX managed to fit an 80MB distro onto a 50MB disc.
Someone has written an equivalent filesystem for Linux called SquashFS.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> What reiser4 can do, but the VFS can't is to insert or remove data in
> the middle of a file. Adding this above the page cache would probably be
> almost impossible (truncate seems already complicated enough).

That would be one of those "special features" that a
VFS-plus-userspace implementation of archive views could take
advantage of on reiser4, while using a slower method (sometimes much
slower) on all other filesystems.

By the way, can reiser4 share parts of files between different files?

-- Jamie

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

Am Donnerstag, den 26.08.2004, 16:04 +0100 schrieb Jamie Lokier:

> Christophe Saout wrote:
> > What reiser4 can do, but the VFS can't is to insert or remove data in
> > the middle of a file. Adding this above the page cache would probably be
> > almost impossible (truncate seems already complicated enough).
> 
> That would be one of those "special features" that a
> VFS-plus-userspace implementation of archive views could take
> advantage of on reiser4, while using a slower method (sometimes much
> slower) on all other filesystems.

I'm just thinking about something. While you can't cut bytes out of unix
iles a lot of filesystems can do this (holes). Most of them only on a
block boundary, reiser4 on a byte boundary. If the filesystems could
export this functionality using an enhanced API we could implement a
compression plugin and other things on the VFS level that works with
every filesystem supporting the required mechanisms, not only reiser4.
And those features would take advantage of reiser4's storage mechanisms.
I think Hans made the plugins reiser4-only because only reiser4 has a
similar API at that time (and obviously because he didn't even think
about doing it otherwise).

> By the way, can reiser4 share parts of files between different files?

At the moment a file is exactly one object. But someone could write
another file plugin that spans a file across multiple objects, then yes,
multiple files could share parts.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> I'm just thinking about something. While you can't cut bytes out of unix
> iles a lot of filesystems can do this (holes). Most of them only on a
> block boundary, reiser4 on a byte boundary. If the filesystems could
> export this functionality using an enhanced API we could implement a
> compression plugin and other things on the VFS level that works with
> every filesystem supporting the required mechanisms, not only reiser4.
> And those features would take advantage of reiser4's storage mechanisms.
> I think Hans made the plugins reiser4-only because only reiser4 has a
> similar API at that time (and obviously because he didn't even think
> about doing it otherwise).

Yes.  In some cases the compression or re-archiving algorithm would
choose different parameters, depending on the alignment capabilities
of the underlying filesystem.  In some cases, that alignment is
"offset 0 is the only one you can have", which is a severe special
case where truncate and rewrite is required for changes.

> > By the way, can reiser4 share parts of files between different files?
> 
> At the moment a file is exactly one object. But someone could write
> another file plugin that spans a file across multiple objects, then yes,
> multiple files could share parts.

Nice.  That's useful for files which appear as themselves and also as
part of a container.

And also for revision managed filesystems :)

-- Jamie

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>
>
>  
>
>>>By the way, can reiser4 share parts of files between different files?
>>>      
>>>
>>At the moment a file is exactly one object. But someone could write
>>another file plugin that spans a file across multiple objects, then yes,
>>multiple files could share parts.
>>    
>>
>
>Nice.  That's useful for files which appear as themselves and also as
>part of a container.
>
>And also for revision managed filesystems :)
>
>-- Jamie
>
>
>
>  
>
Inheritance is one of the unimplemented reiser4 features that Vitaly is 
working on.

Re: silent semantic changes with reiser4

[Nikita Danilov]

Jamie Lokier writes:
 > Christophe Saout wrote:
 > > What reiser4 can do, but the VFS can't is to insert or remove data in
 > > the middle of a file. Adding this above the page cache would probably be
 > > almost impossible (truncate seems already complicated enough).
 > 
 > That would be one of those "special features" that a
 > VFS-plus-userspace implementation of archive views could take
 > advantage of on reiser4, while using a slower method (sometimes much
 > slower) on all other filesystems.
 > 
 > By the way, can reiser4 share parts of files between different files?

It can, but fsck is supposed to cope with this. :)

 > 
 > -- Jamie

Nikita.

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 04:04:34PM +0100, Jamie Lokier wrote:
> Christophe Saout wrote:
> > What reiser4 can do, but the VFS can't is to insert or remove data in
> > the middle of a file. Adding this above the page cache would probably be
> > almost impossible (truncate seems already complicated enough).
> 
> That would be one of those "special features" that a
> VFS-plus-userspace implementation of archive views could take
> advantage of on reiser4, while using a slower method (sometimes much
> slower) on all other filesystems.
> 
> By the way, can reiser4 share parts of files between different files?

no, those file plugins are not written yet :)

Do you mean COW files or some another thing?  For COW files, the reiser4
support is not enough, we need to teach cp(1) or another utility to inform the
fs layer that copying can be done by creation of COW files.

> -- Jamie

-- 
Alex.

Re: silent semantic changes with reiser4

[Shaya Potter]

On Sun, 2004-08-29 at 16:34 +0400, Alex Zarochentsev wrote:

> Do you mean COW files or some another thing?  For COW files, the reiser4
> support is not enough, we need to teach cp(1) or another utility to inform the
> fs layer that copying can be done by creation of COW files.

it's easy enough to add that via an fs ioctl(), I did the equivalent
last semester for some research I'm doing while using ext3cow (which
unfortunately was put on hold for the summer), it's just a take off of
sys_link(), a sys_cowpy() of sorts.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Alex Zarochentsev wrote:
> > > What reiser4 can do, but the VFS can't is to insert or remove data in
> > > the middle of a file. Adding this above the page cache would probably be
> > > almost impossible (truncate seems already complicated enough).
> > 
> > That would be one of those "special features" that a
> > VFS-plus-userspace implementation of archive views could take
> > advantage of on reiser4, while using a slower method (sometimes much
> > slower) on all other filesystems.
> > 
> > By the way, can reiser4 share parts of files between different files?
> 
> no, those file plugins are not written yet :)
> 
> Do you mean COW files or some another thing?

COW files would be nice as well, but I meant another thing: for files
which are parts of an uncompressed archive, it makes sense, if the fs
offers it efficiently, to share the storage.

Doing it properly requires that memory is also shared, for example
when COW files are used to isolate trees in virtual server jails, you
still want the mapped executables to share memory.

You can't do that just by writing a plugin.

> For COW files, the reiser4 support is not enough, we need to teach
> cp(1) or another utility to inform the fs layer that copying can be
> done by creation of COW files.

Actually we don't.  The VM and VFS layers could deduce it with no
changes to cp(1), by tracking which pages are identical due to read(3)
and write(3).  However, changing cp(1) is easier.

reiser4 support is obviously the first prerequisite...

-- Jamie

Re: silent semantic changes with reiser4

[Jamie Lokier]

Adrian Bunk wrote:
> What is the technical reason why a tar plugin should be reiser4 
> specific, instead of a generic VFS or userspace solution that would 
> allow the same also on other fs like e.g. iso9660?

It should be a generic VFS plugin, not reiser4 or userspace.  The VFS
plugin should call out to userspace for most actions (except handling
cached data), and it should take advantage of special reiser4 features
for storage and performance optimisations.  But it should still work
over a standard filesystem, when those special features aren't
available.  I guess FUSE and many earlier projects are heading in this
direction.

A generic userspace solution doesn't let you "cd" into a tar file from
all programs like you can inside Midnight Commander.

Gnome and KDE take the approach that every userspace system call
should be intercepted and filtered, to create the illusion of virtual
data.  As a result, different programs see different virtual data: you
can't just cut and paste a path from Gnome or KDE into any other
program.  It's not just a "social problem of libraries" thing:
sometimes I have programs which don't link to libc.  Sometimes I have
programs which mustn't link with anything that calls malloc().  It'd
be silly for them to have a different view of the filesystem just
because they can't link with some userspace library.

The Gnome/KDE/Midnight Commander pure userspace solution is silly: if
_every_ program in the system should get the same view, it makes much
more sense for the kernel to filter the system calls and redirect the
virtual accesses to a userspace daemon, while keeping the real
accesses at full speed.

Furthermore is makes much more sense for the kernel's page cache to
hold those uncompressed pages, than for every userspace application to
try and cooperatively manage a cache of uncompressed fragments in the
most inefficient way.

There's another problem with the Midnight Commander approach.  If I
"cd" into a tar file, and then a program writes to the tar file, I
don't always see the changes straight away. The two views aren't
coherent.  This isn't an easy problem to solve, but it should be
solved.

When a simple "cd" into .tar.gz or .iso is implemented properly, it
will have _no_ performance penalty after you have first looked in the
file, so long as it remains in the on-disk cache.  And, the filesystem
will manage that cache intelligently.

Imagine: for looking at source files and such, you probably won't
bother untarring in future, and you won't bother keeping untarred
source trees in your home directory for easy access to things you look
at often.  Why waste the space?  You could install whole applications
as a .tar and run them from within it, with no performance penalty.

Similarly, the filesystem will be able to archive directories
automatically that haven't been touched in a long time, with no
visible change except increased storage space.  "grep" will be a bit
slower, but you'll have a useful search tool by then (using coherent
indexes) which will be more useful than grep, and much faster.

-- Jamie

Re: silent semantic changes with reiser4

[Horst von Brand]

Jamie Lokier <jamie@shareable.org> said:

[...]

> When a simple "cd" into .tar.gz or .iso is implemented properly, it
> will have _no_ performance penalty after you have first looked in the
> file, so long as it remains in the on-disk cache.  And, the filesystem
> will manage that cache intelligently.

Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or
un-tarred in memory (or on disk cache) for this to be true, and that takes
quite a long time. Each time you want to peek anew at linux/Makefile, the
whole tarfile will have to be read and stored somewhere, and that is just
too slow for my taste. The .tar format is optimized for compact storage,
the on-disk format of a filesystem is optimized for fast access and
modifiability. Now go ahead and enlarge a file on your .iso/.tar a bit...it
will take ages to rebuild the whole thing. There is a _reason_ why there
are filesystems and archives, and they use different formats. If it weren't
so, everybody and Aunt Tillie would just carry .ext3's around, and would
wonder what the heck all this fuss is about.

> Imagine: for looking at source files and such, you probably won't
> bother untarring in future, and you won't bother keeping untarred
> source trees in your home directory for easy access to things you look
> at often.  Why waste the space?  You could install whole applications
> as a .tar and run them from within it, with no performance penalty.

Perpetuum mobile is a nice idea too.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jamie Lokier]

I'm going to explain why filesystem support for .tar.gz or other
"document container"-like formats is useful.  This does _not_ mean tar
in the kernel (I know someone who can't read will think that if I
don't say it isn't); it does mean hooks in the kernel for maintaining
coherency between different views and filesystem support for cacheing.

The vision I'm going for here is:

  1. OpenOffice and similar programs store compound documents in
     standard file formats, such as .tar.gz, compressed XML and such.

     Fs support can reduce CPU time handling these sorts of files, as
     I explain below, while still working with the standard file formats.

     With appropriate userspace support, programs can be written which
     have access to the capabilities on all platforms, but reduced CPU
     time occurs only on platforms with the fs support.

  2. Real-time indexing and local search engine tools.  This isn't
     just things like local Google; it's also your MP3 player scanning
     for titles & artists, your email program scanning for subject
     lines to display the summary fast, your blog server caching built
     pages, your programming environment scanning for tags, and your
     file transfer program scanning for shared deltas to reduce bandwidth.

     I won't explain how these work as it would make this mail too
     long.  It should be clear to anyone who thinks about it why the
     coherency mechanism is essential for real-time, and a consistent
     interface to container internals helps with performance.

Horst von Brand wrote:
> Jamie Lokier <jamie@shareable.org> said:
> > When a simple "cd" into .tar.gz or .iso is implemented properly, it
> > will have _no_ performance penalty after you have first looked in the
> > file, so long as it remains in the on-disk cache.  And, the filesystem
> > will manage that cache intelligently.
> 
> Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or
> un-tarred in memory (or on disk cache) for this to be true, and that takes
> quite a long time. Each time you want to peek anew at linux/Makefile, the
> whole tarfile will have to be read and stored somewhere,

Wrong.  "So long as it remains in the on-disk cache" means each time
you peek at linux/Makefile, the tarfile is _not_ read.

For a tarfile it's slow the first time, and when it falls out of the
on-disk cache, otherwise, for component files you are using regularly
(even over a long time) it's as fast as reading a plain file.

You obviously know this, as you mentioned on-disk cache in the reply,
so I infer from the rest of your mail that what you're trying to say
is more about modifications than reading archives.  That it would be
silly to keep working data in .tar.gz files, because working inside
them regularly would be slow.

Which means you must be assuming, incorrectly, that these .tar.gz
files are really kept up to date on disk with every component file
modification.

Which is silly.  .tar.gz files are suitable for *transport* and
*archival*, not regular random access; it's almost rude of you to
suggest I didn't know that.

The proposal is that .tar.gz files (and others) are analysed on demand
and content cached on disk as it is read.  Then subsequent reads will
be as fast as if you had unpacked the archives by hand, manually using
the tar command.  This is obviously exactly the same as you do now,
with a small bit of added convenience.

The other part of the proposal is that when you modify a component,
the modifications are stored on disk in the same way as ordinary
files, using the regular high performance random access disk
structures.  Nothing is done to recreate the archive at this point; I
think this is where you misunderstood and thus flamed.

_If_ after modifying components, you then read the .tar.gz as a file,
then (and only then) is it recreated, taking in the worst case the
same time as running the tar command.

The _only_ times when that occurs are precisely those times when you
would have run the tar command manually: because you only read the
.tar.gz file when you need the flat file for some purpose, such as
attaching it to an email, transferring by FTP or HTTP, or reading it
into a program that needs it in that format.

If there is anything about that strategy that doesn't make sense, then
I suggest I have failed to explain it properly, and you're welcome to
demand a clearer explanation.

> the .tar format is optimized for compact storage, the on-disk format
> of a filesystem is optimized for fast access and modifiability.

Actually no, .tar is not compact at all.  It's also not optimised for
random read access, but after an index is build it is very fast for that.

.tar.gz is compact.  Although that is not especially fast for random
read access, you can build a "compression dictionary index" which
optimises random read access even in a .tar.gz, without ever unpacking
the whole thing.

Some formats like .iso, .zip and .jar are optimised for compact
storage _and_ fast random access.  They come with an index, and don't
need one to be built and cached.

These are not filesystem-like formats, obviously, but they are the
formats you need to pack and unpack when exchanging data with other
people.  That's the _only_ reason they're on your disk (virtually;
they may not really exist some of the time).

> Now go ahead and enlarge a file on your .iso/.tar a bit...it
> will take ages to rebuild the whole thing. There is a _reason_ why there
> are filesystems and archives, and they use different formats. If it weren't
> so, everybody and Aunt Tillie would just carry .ext3's around, and would
> wonder what the heck all this fuss is about.

If you enlarge a file in your .iso/.tar subdirectory a bit... nothing
happens.  Why would a smart programmer do anything so silly as rebuild
the archive at that point?

_If_ you subsequently read the .iso/.tar _file_, then and only then
does it rebuild.  Once, after lots of component writes.  The only time
you would ever do that is if you are specifically reading the archive
file, which means you actually want to use the repacked file at that
point, for example to FTP it somewhere or use as an email attachment.

If the filesystem does not do that on demand, then you would have run
the tar command manually at that point, precisely because that's a
point where you need the repacked archive.  So in case where the
filesystem repacks the archive, it takes exactly the same time as you
would have taken anyway; it's just automatic instead of manual.  (As a
handy side effect, the automatic method offers lower latency for
transmissions).

Now, why would we bother with all this?

I see three reasons: convenience, time efficiency, and storage efficiency.

Convenience is simply that it is handy to be able to look inside
archive files, in those situations where we _currently_ use them, without
having to manually untar when we need to, and without having to
remember to clean up old directories when we discover we aren't using
those often any more.  This is _not_ an argument for using .tar.gz
files in place of ordinary directories!  Convenience applies to doing
the things you do now.

Time efficiency has two angles.  A simple one is that accessing
.tar.gz contents through any kind of filesystem interface, even pure
userspace, can be faster than unpacking whole files, simply because
there are ways to decode parts without unpacking it all.

However, the main time efficiency that I see comes from the increasing
number of applications where the "Open" and "Save" operations store
data in *.gz files (e.g. OpenOffice compressed XML documents), or
*.tar.gz files (some compound document formats), or other things like
that.  (If you think about it, quite a lot of things are like that).

With these, every "Open" currently has to decompress and maybe unpack
an archive format.  Every "Save" currently has to pack and then
compress.  This is done so the user sees a single flat file containing
a complex document, but it is a waste of CPU time until the user
actually transports the flat file.

The lazy proposal, as described earlier in this mail, _removes_ these
decompression, unpacking, packing and compression CPU-intensive steps
when they are unnecessary.  The experience of a single file containing
a complex document is maintained, but the CPU time is reduced in many
typical operations.  "Open" gets faster after you first look at a
file, "Save" gets a lot faster for large documents, and the equivalent
of grep (or later, real-time local search engine) gets a lot faster
too.  There is no operation where CPU time is overall increased.

This is what I've meant throughout this thread, when I say containers:
document files of the kind used to hold text, figures, etc. that are
typically transported as a unit, and edited as a unit, but nonetheless
at the moment they're stored in somewhat CPU intensive formats, for
compactness.  That's fine for a 1 page letter, but think of the
OpenOffice 500-page book containing a large number of diagrams.

However, even simple programs that read & write compressed XML benefit.

The proposal allows that sort of thing to be handled more time
efficiently that it is today, and in a way that is very practical to use.

(It's unthinkable that OpenOffice and similar programs would have a
lot of code which stored data in a special way just for Linux, just
for these performance benefits which are otherwise user-invisible, but
it's thinkable that a general purpose userspace library which is
portable to all platforms could be written, which takes advantage of
the facility when it's available and does the equivalent of today's
"compress on save" when the filesystem facility isn't available).

Finally, storage efficiency comes from simply allowing the filesystem
and supporting tools to decide when it is best to store data in
unpacked, packed & compressed, both at the same time, or another
archival form.  The filesystem has comparativaly good knowledge of
which data to archive and when, but it can only maintain the illusion
if there's a mechanism to make archived forms and unpacked forms
coherent.

Now, I'm sure there is a way to implement this on top of a neat and
simple kernel feature involving weird bind mounts, leases, dnotifies
and FUSE.  But those kernel offering are quite a mess at the moment
and don't fit together in a way which can usefully create this effect.

Auto-mounting uservfs directories over file-as-directory, using
moveable bind mounts _nearly_ offers the kernel primitives we need to
build this in userspace and get the all the efficiencies.  But not quite.

(We could obviously do it all in userspace by putting _everything_ in
a userspace filesystem, but that would be silly as it would throw away
all of the performance of having a threaded filesystem in the kernel.
It might do as a proof of concept though).

-- Jamie

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Wed, 1 Sep 2004, Jamie Lokier wrote:
>
> I'm going to explain why filesystem support for .tar.gz or other
> "document container"-like formats is useful.  This does _not_ mean tar
> in the kernel (I know someone who can't read will think that if I
> don't say it isn't); it does mean hooks in the kernel for maintaining
> coherency between different views and filesystem support for cacheing.

I think that's a valid thing, but there are some fundamental problems with 
it if you expect it to work on a normal filesystem (ie something that 
isn't fundamentally designed as a database).

For example, _what_ kind of coherency do you think is acceptable? Quite
frankly, using standard UNIX interfaces, absolute coherency just isn't an
option, because it's just not possible to try to atomically update a view
at the same time somebody else is writing to the "main file". "mmap()" is
the most obvious example of this, but even the _basic_ notion of multiple
"read" calls is not atomic without locking that is _way_ too expensive.

A "read()" on a file is not atomic even on the _plain_ file: if somebody 
does a concurrent "write()", the reader may see a partial update. This 
becomes a million times more confusing if the reader is seeing a 
structured view of the file the writer is modifying.

Also, it's likely impossible to write() to the view-file, again unless you 
expect all the underlying filesystems to be something really special.

So from a _practical_ standpoint, I suspect that the best you can really 
do pretty cheaply (and which gets you 90% of what you probably want) is:

 - open-close consistency: the "validity" of the cache is checked at 
   _open_ time, and no guarantees are given about the cache being 
   updated afterwards.
 - read-only access to the cache (ie you can only read the view, not write 
   to it).

and quite frankly, I think you can do the above pretty much totally in
user space with a small library and a daemon (in fact, ignoring security
issues you probably don't even need the daemon). And if you can prototype
it like that, and people actually find it useful, I suspect kernel support
for better performance might be possible.

Suggested interface:

	int open_cached_view(int base_fd, char *type, char *subname);

where "type" would be the type of the view (ie "tar" for a tar-file view,
"idtag" for a mp3 ID tag, or NULL for "autodetect default view") and
"subname" would be the cache entry name (ie the tar-file filename, or the
tag type to open).

I bet you could write a small library to test this out for a few types.  
See if it's useful to you. And only if it's useful (and would make a huge
performance difference) would it be worth putting in the kernel.

Implementation of the _user_space_ library would be something like this:

	#define MAXNAME 1024
	int open_cached_view(int base_fd, char *type, char *subname)
	{
	        struct stat st;
	        char filename[PATH_MAX];
	        char name[MAXNAME];
	        int len, cachefd;
	
	        if (fstat(base_fd, &st) < 0)
	                return -1;
	        sprintf(name, "/proc/self/fd/%d", base_fd);
	        len = readlink(name, filename, sizeof(filename)-1);
	        if (len < 0)
	                return -1;
	        filename[len] = 0;
	
	        /* FIXME! Replace '/' with '#' in "type" and "subname" */
	        len = snprintf(name, sizeof(name),
	                        "%04llx/%04llx/%s/%s/%s",
	                        (unsigned long long) st.st_dev,
	                        (unsigned long long) st.st_ino,
	                        type ? : "default",
	                        subname,
	                        filename);
	        errno = ENAMETOOLONG;
	        if (len >= sizeof(name))
	                return -1;
	        cachefd = open(name, O_RDONLY);
	        if (cachefd >= 0) {
	                /* Check mtime here - maybe we could have kernel support */
	                return cachefd;
	        }
	        if (errno != ENOENT)
	                return -1;
	        /*
	        .. try to generate cache file here ..
	         */

see what I'm aiming at? You start out with a generic "attribute cache" 
library that does some hacky things (like depending on "mtime" for 
coherency) and then if that works out you can see if it's useful.

			Linus

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jamie Lokier]

Linus Torvalds wrote:
> > I'm going to explain why filesystem support for .tar.gz or other
> > "document container"-like formats is useful.  This does _not_ mean tar
> > in the kernel (I know someone who can't read will think that if I
> > don't say it isn't); it does mean hooks in the kernel for maintaining
> > coherency between different views and filesystem support for cacheing.

Thanks for your reply.

> I think that's a valid thing, but there are some fundamental problems with 
> it if you expect it to work on a normal filesystem (ie something that 
> isn't fundamentally designed as a database).
> 
> For example, _what_ kind of coherency do you think is acceptable? Quite
> frankly, using standard UNIX interfaces, absolute coherency just isn't an
> option, because it's just not possible to try to atomically update a view
> at the same time somebody else is writing to the "main file". "mmap()" is
> the most obvious example of this, but even the _basic_ notion of multiple
> "read" calls is not atomic without locking that is _way_ too expensive.
>
> A "read()" on a file is not atomic even on the _plain_ file: if somebody 
> does a concurrent "write()", the reader may see a partial update. This 
> becomes a million times more confusing if the reader is seeing a 
> structured view of the file the writer is modifying.

I agree.  Coherency should be about equivalent to what writing flat
files offers.

(For now.  Eventually I think we'll see transactional I/O in Linux, to
solve the sort of updater-to-server glitches which are currently
solved by putting even trivial data behind clunky database servers).

The coherency I'd like would also ensure that you'll never see an
invalid file structure in the long term, even if during writing
operations you might see temporarily invalid structure.

That is easy if you only have read-only views.

> Also, it's likely impossible to write() to the view-file, again unless you 
> expect all the underlying filesystems to be something really special.

Right.  I wonder if you read the part of my message which deals with
lazy updates of container files.

The idea is that write() to a view-file doesn't repack the container
file until the container file is read.

Practically, that means the view-file's write handler, which is
probably in userspace, grabs a kind of mandatory lock (similar to
F_SETLEASE) on the container file and then truncates it.  After a
time, or when a program tries to read the container (whichever comes
first), the view-file's handler is notified, it regenerates the
container file, and releases the lock.

This is the form of coherency needed to make writes work properly.

(You can take it more fine grained, locking at the page level, but
that's just to improve performance some more.  It's not fundamental.)

Notice that it doesn't need a special filesystem.  View-file writes
will work with any ordinary filesystem.  A special filesystem would
make it perform better (much better), by allowing the truncated state
to persist across reboots, with an xattr to say what's needed to
recreate the data -- but it's not fundamentally necessary.

It does need kernel support for that kind of lock, which F_SETLEASE
isn't, and notifications of mount events, and notifications of
file writes.

> So from a _practical_ standpoint, I suspect that the best you can really 
> do pretty cheaply (and which gets you 90% of what you probably want) is:
> 
>  - open-close consistency: the "validity" of the cache is checked at 
>    _open_ time, and no guarantees are given about the cache being 
>    updated afterwards.
>  - read-only access to the cache (ie you can only read the view, not write 
>    to it).

I think that gets 90% of the useful features, but leaves out some
applications which would raise eyebrows when they actually work --
i.e. the component editing applications.

> and quite frankly, I think you can do the above pretty much totally in
> user space with a small library and a daemon (in fact, ignoring security
> issues you probably don't even need the daemon). And if you can prototype
> it like that, and people actually find it useful, I suspect kernel support
> for better performance might be possible.

Right now it's stupidly impossible for a daemon to monitor a file for
changes and not be tricked by mtime faults, because someone can link
to the file and modify it on another path.  That's obviously a bug in
dnotify which I keep meaning to fix.  There are other dnotify
problems, some of which are fundamental when it's over a network.

I do care about security issues, for things like an "md5" or
"compiled" attribute especially, but for a prototype that's just
something to live with.

> Suggested interface:
> 
> 	int open_cached_view(int base_fd, char *type, char *subname);

Something userspace is useful anyway, so that a fully userspace
alternative is available, so that people writing apps will take up the
approach and take advantage of fs-level support where available while
still being portable elsewhere.

> see what I'm aiming at? You start out with a generic "attribute cache" 
> library that does some hacky things (like depending on "mtime" for 
> coherency) and then if that works out you can see if it's useful.

Ok but I don't think that form of it is useful!  It's the sort of
interface that specific attribute-using programs will have to use if
they're to be portable, but it doesn't provide any special advantages
to any other programs.

Gnome VFS, KDE etc. provide much of that kind of interface already.
Evidently some people find that useful.  But I don't, precisely
because it's so tied into one set of programs or another.

This is where the file-as-directory metadata stuff is so potentially
interesting.  It's actually a nice interface that every program can see.

It's the nice interface when running on Linux that'll make it deliciously
worthwhile for portable applications, such as MP3 retaggers and PDF
image extractors, to be written in the form of a tool plus a plugin
for a common view-file library, instead of all in the tool as they are now.

Without the feature of a nice interface on Linux, there's no reason
why application writers would bother to learn and use a view-file library.

To be fair, the whole thing could be prototyped (with glitches, but
enough to demonstrate) in userspace, running everything through
something like uservfs or even NFS, especially with file-as-directory
being blessed at the VFS level.

That is exactly the right way to go about it.  I started some work on
that back in '99, around the same time Pavel was exerimenting with
podfuk (which become uservfs), but haven't and don't have the time or
money(*) to take it further.

-- Jamie

(*) That's a hint to potential employers who want to support this work,
by the way.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Thu, 2 Sep 2004, Jamie Lokier wrote:
> 
> This is where the file-as-directory metadata stuff is so potentially
> interesting.  It's actually a nice interface that every program can see.

.. but you can _still_ do exactly this in a user-space library and 
prototype it. Just take over "open" (and opendir/readdir), and you can do 
so.

> To be fair, the whole thing could be prototyped (with glitches, but
> enough to demonstrate) in userspace, running everything through
> something like uservfs or even NFS, especially with file-as-directory
> being blessed at the VFS level.

I do believe that you can prototype it even without any kernel support.

This is why I want to have stronger reasons for real VFS support. I see
the Samba argument, and I buy that one, if only because it has such a high
consistency requirement (let's face it, people expect more guarantees of
their file servers than of most other things). But on the other hand, the
Samba guys apparently _are_ happy with "openat()", so they don't
necessarily need any normal namespace stuff.

And I suspect that a user-level prototype would show that what you 
_really_ want may be something else altogether..

		Linus

Why do we need openat? (was: The argument for fs assistance in handling archives)

[Hubert Chan]

>>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes:

[...]

Linus> This is why I want to have stronger reasons for real VFS
Linus> support. I see the Samba argument, and I buy that one, if only
Linus> because it has such a high consistency requirement (let's face
Linus> it, people expect more guarantees of their file servers than of
Linus> most other things). But on the other hand, the Samba guys
Linus> apparently _are_ happy with "openat()", so they don't necessarily
Linus> need any normal namespace stuff.

Maybe things have got lost in this thread (or these many threads), so
bear with me, but what exactly does openat() gain us that we can't get
from exporting things from the normal file namespace?  The only issue
that I've seen raised is that you need to be able to distinguish between
the contents of a directory and its substreams.  And I've responded that
streams should be stuck in a specially-named subdirectory (like
..streams) (which should be done even if only files could contain
substreams), to which I have seen no objections raised (not even as much
as a "you're stupid"), and so can only assume nobody (who cares) has any
objections.

So as far as I can see, openat() has no advantages, and exporting
things through the normal namespace has several advantages, such as:
- can manipulate substreams using legacy programs -- no modifications
  needed.  And don't tell me "runat" until someone shows me how to deal
  with my GIMP/emacs problem.
- useful for exporting different things (streams, acl's or other access
  control schemes, xattrs, version control, automatic untar, ...)
- only need to modify tar (and backup programs) once -- any new
  filesystem extensions, no matter how complex, can be exported through
  the filesystem interface, and tar will be able to catch them.
- referring to the substreams is simpler ("foo.txt/streams/bar"
  instead of "the bar stream of foo.txt"), which leads to simpler user
  interface.  This is even more marked if we, for some strange reason,
  discover that it's a good idea for streams to be allowed to have
  substreams of their own ("foo.txt/streams/bar/streams/baz" vs. "the
  baz stream of the bar stream of foo.txt")
- I think I'm forgetting some...

If this is indeed the case, then it seems to me that the logical choice
is to export streams through the normal file namespace.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Why do we need openat? (was: The argument for fs assistance in handling archives)

[Hubert Chan]

>>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes:

[...]

Linus> This is why I want to have stronger reasons for real VFS
Linus> support. I see the Samba argument, and I buy that one, if only
Linus> because it has such a high consistency requirement (let's face
Linus> it, people expect more guarantees of their file servers than of
Linus> most other things). But on the other hand, the Samba guys
Linus> apparently _are_ happy with "openat()", so they don't necessarily
Linus> need any normal namespace stuff.

Maybe things have got lost in this thread (or these many threads), so
bear with me, but what exactly does openat() gain us that we can't get
from exporting things from the normal file namespace?  The only issue
that I've seen raised is that you need to be able to distinguish between
the contents of a directory and its substreams.  And I've responded that
streams should be stuck in a specially-named subdirectory (like
..streams) (which should be done even if only files could contain
substreams), to which I have seen no objections raised (not even as much
as a "you're stupid"), and so can only assume nobody (who cares) has any
objections.

So as far as I can see, openat() has no advantages, and exporting
things through the normal namespace has several advantages, such as:
- can manipulate substreams using legacy programs -- no modifications
  needed.  And don't tell me "runat" until someone shows me how to deal
  with my GIMP/emacs problem.
- useful for exporting different things (streams, acl's or other access
  control schemes, xattrs, version control, automatic untar, ...)
- only need to modify tar (and backup programs) once -- any new
  filesystem extensions, no matter how complex, can be exported through
  the filesystem interface, and tar will be able to catch them.
- referring to the substreams is simpler ("foo.txt/streams/bar"
  instead of "the bar stream of foo.txt"), which leads to simpler user
  interface.  This is even more marked if we, for some strange reason,
  discover that it's a good idea for streams to be allowed to have
  substreams of their own ("foo.txt/streams/bar/streams/baz" vs. "the
  baz stream of the bar stream of foo.txt")
- I think I'm forgetting some...

If this is indeed the case, then it seems to me that the logical choice
is to export streams through the normal file namespace.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Linus Torvalds wrote:
[...]
| This is why I want to have stronger reasons for real VFS support. I see
| the Samba argument, and I buy that one, if only because it has such a high
| consistency requirement (let's face it, people expect more guarantees of
| their file servers than of most other things). But on the other hand, the
| Samba guys apparently _are_ happy with "openat()", so they don't
| necessarily need any normal namespace stuff.

I see your Samba and I raise you an NFS.

For that matter, how about every distributed FS to come.

All it needs to do is support transactions (which most try to anyway)
and files-as-directories to support _all_ future FS features transparently.

It might ask the fs for a non-recursive serialization/dump of a file
rather than just reading it directly -- this makes things like named
pipes and device nodes sane.  This would be optional and only supported
when the client and server both have support for the same fs -- other,
more hack-ish things would be needed to support Samba.

It would also use the best caching mechanism discussed earlier.  (I'm
not yet arrogant enough to insist that mine is the best.)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTaXs3gHNmZLgCUhAQIG5g/+MEHPsNK41goKI8yJPwKdigbRbrfGf3sj
xpFGR15/DAW4b4qxr1BIt5hyAQoFLq13POM/lCV12hKp1RSo9atuKim6DyUhOX1c
0xUK6efY0ew+mvYHKaJUgwrD8BV7XYaTS+5y/4wbpgdOlPHsx06lHVh90KtAUAE7
vdsFWRAgHT/g34rSRBJfHAaom/7VtDce25roLrB3CNI2v+9tERYee+xkBPoy2+Y1
5tLgoInDi05z7bUu0lCJ+Rq2LCHuKGmgInI86gul5BfHlO3j2zYnJ5VYT8QR6onM
iThiPzFrrpoXEjQRRo7E4WHQCD/2BGXzQdS78JzwTt1B8oNrIkZR9/tcv6keNzr6
fyUhtQef5TZJtwpON1gZPqraBUiTDUdWHJSXGLB5+78vMy+bS7gHqXN5DR049+7b
urQFtnZBhi3zY8cpT0hTPnW2PcGnDuGMt9qUiFBn8Hx9Tkti6y+7Jl42a40jgBAW
z4hSHPKVzu8d/qeBapVHtqIkaLTdim1JnmFY+SCiiuCPEMOELDxw+o/rcwTIls9s
3TBJ5lqL/+84W8JU6t1CpzEgF+dRhPSBNyrG5msVixT7cmB2GO87rLBfchM1cotq
lBMyuDwmwO6RanwkEvOV//3wsG8POMw5f2Gx8eljqU9uzSpYcKry+Is2ZVmHKJ/1
24eqEXIB24Y=
=GSMf
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jamie Lokier wrote:
[...]
|>A "read()" on a file is not atomic even on the _plain_ file: if somebody
|>does a concurrent "write()", the reader may see a partial update. This
|>becomes a million times more confusing if the reader is seeing a
|>structured view of the file the writer is modifying.
|
|
| I agree.  Coherency should be about equivalent to what writing flat
| files offers.
|
| (For now.  Eventually I think we'll see transactional I/O in Linux, to

Eventually?  This was supposed to be in reiser4, maybe it's been put off
until 4.1...

And that is the right solution.  Not the only one, but the right one.
Caching isn't the only thing sorely in need of transaction support right
now.  Actually, I find it hard to think of anything on Linux which
shouldn't have transactions -- why should /etc/fstab or
/home/david/homework be more fragile than /var/lib/mysql?

|>Also, it's likely impossible to write() to the view-file, again unless
you
|>expect all the underlying filesystems to be something really special.
|
|
| Right.  I wonder if you read the part of my message which deals with
| lazy updates of container files.
|
| The idea is that write() to a view-file doesn't repack the container
| file until the container file is read.
|
| Practically, that means the view-file's write handler, which is
| probably in userspace, grabs a kind of mandatory lock (similar to

In userspace?  I don't buy that -- this is exactly what a reiser4 plugin
is for, just in OOP technology.  Instead of "write handler", it's
probably called "write method".

| F_SETLEASE) on the container file and then truncates it.  After a
| time, or when a program tries to read the container (whichever comes
| first), the view-file's handler is notified, it regenerates the
| container file, and releases the lock.

Why "after a time"?  Should be "when a program tries to read the
container" -- and with transactions or even some clever locking, it can
be just as consistant as the flatfile write.

[...]
| (You can take it more fine grained, locking at the page level, but
| that's just to improve performance some more.  It's not fundamental.)

Actually, "locking at the page level" rarely makes more sense than "use
two files".  And with transactions, I think it works like this:

Process A opens file.tar.gz/contents/foo.c and starts writing.  When
it's maybe halfway through, process B opens file.tar.gz for reading.  So
it gets the version of foo.c that existed before A started writing.  B
finishes before A does, and then process C opens file.tar.gz for writing
- -- it has to wait for A to complete.

| Notice that it doesn't need a special filesystem.  View-file writes
| will work with any ordinary filesystem.  A special filesystem would
| make it perform better (much better), by allowing the truncated state
| to persist across reboots, with an xattr to say what's needed to
| recreate the data -- but it's not fundamentally necessary.

I am hoping that someone will implement a VFS-based storage layer plugin
for reiser4.  This means that one can use reiser4 interface features
(such as reading file.tar.gz as a directory) without needing to
reformat.  It also means that one could conceivably port reiser4 to a
userland library, even.  I think.

I wonder if it's as much of a performance hit as uservfs.

[...]
|>Suggested interface:
|>
|>	int open_cached_view(int base_fd, char *type, char *subname);

Oh, God no.  Alright, it's more useful than no caching at all, but still...

Alright, let's say I have /usr/src/linux as a hardlink farm to
/usr/src/linux.tar.bz2/contents/.  With kernel support, I can actually
do a make from inside /usr/src/linux and have it behave sanely.

Do you want to patch make?  If yes, you may as well patch everything.
In fact, you may as well patch glibc to replace open with your
open_cached_file.  Not to mention handling things like readdir, stat,
and so on...

Not to mention the overhead of checking if a file is part of a cache for
every single open, ever...

Putting it in the filesystem simplifies things.  There's already some
sort of read/write method for the file -- overload that for files which
need to be fetched and cached.  Files which are already in the cache, or
have nothing to do with the cache, remain exactly as fast as they are now.

I'm not sure about the best way to make this portable between
filesystems.  I'm not sure if it's really that useful to do so.  I'm
beginning to see why Hans would like reiser4 to be the new VFS.

| Something userspace is useful anyway, so that a fully userspace
| alternative is available, so that people writing apps will take up the
| approach and take advantage of fs-level support where available while
| still being portable elsewhere.

Maybe.  But here, I'd suggest that the kernel implementation is fine,
too.  AFAIK, Reiser4 was designed to be portable -- specifically because
certain apps -- not OSes, but apps -- might embed it for a fee.

|>see what I'm aiming at? You start out with a generic "attribute cache"
|>library that does some hacky things (like depending on "mtime" for
|>coherency) and then if that works out you can see if it's useful.
|
|
| Ok but I don't think that form of it is useful!  It's the sort of
| interface that specific attribute-using programs will have to use if
| they're to be portable, but it doesn't provide any special advantages
| to any other programs.
|
| Gnome VFS, KDE etc. provide much of that kind of interface already.
| Evidently some people find that useful.  But I don't, precisely
| because it's so tied into one set of programs or another.
|
| This is where the file-as-directory metadata stuff is so potentially
| interesting.  It's actually a nice interface that every program can see.

Exactly!  Every program should be able to see the interface.  The
interface should also be transparent sometimes.  Example --
reimplementing InterMezzo/Lustre.  Files that are cached are exactly as
fast to write as local files.  To invalidate the cache, you delete the file.

| Without the feature of a nice interface on Linux, there's no reason
| why application writers would bother to learn and use a view-file library.

That's not as compelling a reason.  Application writers don't care at
all whether it's in the kernel or userland.  All putting it in the
kernel does for adoption is make everyone sit up and notice.

[...]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTaU5XgHNmZLgCUhAQJsBQ//WBWU28IrQc8mFokLyeFtIAIErqUE65oj
TDcx0/Dg+2JD4zAMht6idhMDfYU5kvBj5C9srMvdklprgPJ/dIQSS/MS8o6qnZl4
kRvjUwI/vBV9a2gwxzt32NrLK+ZFuWwOiRJ1ia6zOyxqOE/jx8z1VDR7R17lbakh
EbFG/2taoHyRke4O+rxig93OxTUBJ72KBKZ+LGrDMwzm7uNP069UupWO0ARpBlMM
6T0o8bm4QK7dgqXgowDGx22hFo5ZPLc3undsHepnu+i8cZMPpOMigbgwpiG/VNnu
t00WOrJGigNkeOrCB4y4LO6LQFL7DjLRRI3T9Y3t3YUFegYDdG6A1L2ZQcrxqtlj
alSx6YYYigH2+MBaN+ONIPlZVe9ybdTpAWl+hkS1nvKaNVc5xYUTFBhBAzIr/4Ol
Zuu5sE8/Q7+5oOCqp9zW51g4oBIlDE0CY3EoZ1P5UrQXT5KphVWr8AUXcP+9CL5c
vsIbYWTmbU/XgxIb+lyB2anTGnMzy1TxUHVxAR4ijXPSDuAbojJhtg1VuDMGhahq
NfW34jRbDJSPDoar/eN2jxF8a4bG7xt8w10LO0lnnjznNJdsurk4HmsHFhPZ0DJj
OJTaChsEr+KUldLyNila8A+6Qg4LvSvNSPArXxCTkGizN+jeyFlNSgqFOQajS/Pk
fptJFhA1x3Q=
=dUCH
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Wed, 1 Sep 2004, David Masover wrote:
> 
> And that is the right solution.  Not the only one, but the right one.
> Caching isn't the only thing sorely in need of transaction support right
> now.  Actually, I find it hard to think of anything on Linux which
> shouldn't have transactions -- why should /etc/fstab or
> /home/david/homework be more fragile than /var/lib/mysql?

It's easy to talk big. 

It's damn hard to _implement_ a complex system, and make it stable and 
bug-free, and support legacy applications.

There is a reason why we do only what _must_ be done in kernel space.

		Linus

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Linus Torvalds wrote:
|
| On Wed, 1 Sep 2004, David Masover wrote:
|
|>And that is the right solution.  Not the only one, but the right one.
|>Caching isn't the only thing sorely in need of transaction support right
|>now.  Actually, I find it hard to think of anything on Linux which
|>shouldn't have transactions -- why should /etc/fstab or
|>/home/david/homework be more fragile than /var/lib/mysql?
|
|
| It's easy to talk big.
|
| It's damn hard to _implement_ a complex system, and make it stable and
| bug-free, and support legacy applications.
|
| There is a reason why we do only what _must_ be done in kernel space.
|
| 		Linus

I think transactions should be done in kernel space.  It's the only way
to "enforce" them sanely.

Besides, I'm not the one who first suggested this transaction idea.
That was Hans, and it was in the whitepaper.  I don't know if it's done
yet -- I think it got put off for v4.1.

But if it is going to be done anyway, doesn't it solve the consistency
issue?

I can understand wanting caching to work now, but which is the priority
- -- caching or transactions?  I vote transactions.  The consistancy of
fstab is far more important to me than the consistancy of foo.tgz/bar.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTahTXgHNmZLgCUhAQIumQ//SqEViCUbBcFJ3xsiQfEEUSe2gU+6dyJZ
7i3OeexnLRKrnxIsij5SmYTmGdYXhpaQWMrrjie8uFX0ocA0HTMsA1BzB4ZhtKnO
sYc/jAZusGpAjnyy0t0Q/hwjoMRIsCRVwqnD17Qw8rcuwYT5yaABMboXPiOPjQqu
JTkCeb+ATq5hF9VYL9TbcqaOycV7BMh+N7VXGPS/PIgeyYXyNwfIBWHrN8sjVQDA
3h9a9IEjEyReMdqlNbv73/wggGNqsmprDLIMFhMSPP4LXS5/xt1AhXW1R+mNDXjA
ckNq1lLjD1iUIQ7gIPHjR1Vi5op5GaXqVtdJLXtTyvNuc3nM/yFzM0T7+hgO4rSj
F0VkFFV8FbKRSVv1MsPv1O8EpFNpAs1uX+1g/vZ4ZHwPuOaMeXKT6rVTsr9WesfU
NdPMi0bXF6fAHbpjw3k6v+Jg95Mg+w8bgYOEDptz6zRuex8xC+LPrXE9ZDrYX+nm
+xG2wtMZyMr2y8wMp71RLz6Sj+Vq0cp4HNqccsMLfZ1OEWu2IzneBAmCSGQjLrxL
9coj0G3WpFwspTClh1DGBfuBJPS5O33iEsJ2w80bMhy/wKK5kZG/NyMAGQKazxAr
+7Se6t2RGHIxX5/EEoPI/e0AKLxFuvPi9ilq4bJNNDBMUO2wOkB8V0GLGzFA/0F2
9ALDAaO/dg8=
=pvxM
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Wed, 1 Sep 2004, David Masover wrote:
> 
> I think transactions should be done in kernel space.  It's the only way
> to "enforce" them sanely.

Well, they clearly can't, since it depends on the filesystem. Also, 
transactions have a tendency to be expensive, and people have a tendency 
to ask for more than you give them. Doing transactions on one file is 
only the beginning - you'll find people who want transactions across file 
boundaries etc.

That's a basic fact when it comes to pretty much anything: you can always 
find people who want something better. You can't be all things to all 
people, so what you actually _want_ to do is to expose the _minimal_ set 
of capabilities that people can build on.

So I definitely don't want this discussion to degenerate (any more than it
lng since has ;) into what people _wish_ for. No, it should be a "what is
the _least_ we can absolutely do" that solves real problems.

		Linus

Re: The argument for fs assistance in handling archives

[Hans Reiser]

Linus Torvalds wrote:

>
>. Doing transactions on one file is 
>only the beginning - you'll find people who want transactions across file 
>boundaries etc.
>  
>
Yup, that's what reiser4 is aiming at and where things get exciting for 
version control systems and the like.  We are just missing the api code 
for it at the moment, all the infrastructure is there, and the api is in 
progress in sys_reiser4.

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Thu, 2 Sep 2004, Hans Reiser wrote:
> Linus Torvalds wrote:
> >
> >. Doing transactions on one file is 
> >only the beginning - you'll find people who want transactions across file 
> >boundaries etc.
>
> Yup, that's what reiser4 is aiming at and where things get exciting for 
> version control systems and the like.  We are just missing the api code 
> for it at the moment, all the infrastructure is there, and the api is in 
> progress in sys_reiser4.

But _my_ point is, no user program is going to take _advantage_ of
anything that only one filesystem on one system offers.

So there's no point.

It's much saner (from _any_ app standpoint) to roll their own database in 
user space - that way it just works.

In other words, nobody is really ever going to take advantage of this. 
This is _not_ how technical advanncement happens. The way you get people 
to take advantage of something is to have a nice gradual ramp-up, not a 
sudden new feature that they can't realistically use.

So before you do transactions in the filesystem, you have to be able to do 
them in user space - and then make the filesystem version be _compatible_ 
with that user space library. That way you can get people to use the 
library ("hey, it works anywhere") and then you can get them to use your 
filesystem ("hey, if you use our super-duper filesystem, you can do what 
you are already doing five times faster").

See? You're starting at the wrong end. Give me a portable interface to use 
_first_. Then do transactions in the filesystem.

(Now, as an academic exercise and as a learnign experience and prototyping 
work it migt make sense to do the filesystem first. But then you have to 
really _consider_ it a prototype, and realize that to make it useful you'd 
still have to do the user thing before you can "sell" people on the idea).

		Linus

Re: The argument for fs assistance in handling archives

[Hans Reiser]

Linus Torvalds wrote:

> But _my_ point is, no user program is going to take _advantage_ of
>
>anything that only one filesystem on one system offers.
>  
>
Apple does not have this problem....

and yes, the apps will take advantage of it, which is different from 
depending on it.  If you use the wrong fs you will lose some of the 
features of the app.

For 30 years nothing much has happened in Unix filesystem semantics 
because of sheer cowardice (excepting Clearcase, which priced itself 
into a niche market).   It is 25 years past time for someone to change 
things.  That someone will have first mover advantage, and the more 
little semantic features possessed the more lure there will be to use it 
which will increase market share which will lure more apps into 
depending on it and in a few years the other filesystems will 
(deservedly) have only a small market share because the apps won't all 
work on them.

Besides, there are enhancements which are simply compelling.  You can 
write a dramatically better performance version control system with a 
much simpler design if the FS is atomic.    Our transaction manager 
first draft was written by a version control guy, and he would probably 
be happy to tell you how  lack of atomicity other than rename makes 
version control software design hideous.

We have the performance lead.  By next year we will be stable enough for 
mission critical servers, and then we start the serious semantic 
enhancements.

If you don't embrace progress, then you doom Linux to following behind, 
because the guys at Apple are pretty aggressive now that Jobs is back, 
and they WILL change the semantics, and they will do so in compelling 
ways, and Linux will be reduced to aping them when it should be leading 
them.

Hans

Re: The argument for fs assistance in handling archives

[Oliver Hunt]

How would we go about finding out how many data forks were in a file? 
Because in order to be able to retrieve data from a fork we would need
to know that the fork were there.  Currently this would imply that we
go looking through mtab or some such to find out what fs we're running
on, which seems ugly.

Alternatively we go through the _exciting_ task of making every other
fs (with the exceptions of ntfs, and whatever it is that macs use,
which would need there own custom code) and add code that effectively
goes

getNumForks(fileref){ return 1;} 

or add a new open call that can take a fork number...

either way we have to add a new syscall, that doesn't conform to any
real standard(though i suppose it would be possible to use
macOS/windows style fork iopening interface)

I personally like the concept of having multiple forks in a file, but
in this case I'm inclined towards usermode first, then if it takes off
add kernel level support.

--Oliver Hunt

On Thu, 02 Sep 2004 01:58:30 -0700, Hans Reiser <reiser@namesys.com> wrote:
> Linus Torvalds wrote:
> 
> > But _my_ point is, no user program is going to take _advantage_ of
> >
> >anything that only one filesystem on one system offers.
> >
> >
> Apple does not have this problem....
> 
> and yes, the apps will take advantage of it, which is different from
> depending on it.  If you use the wrong fs you will lose some of the
> features of the app.
> 
> For 30 years nothing much has happened in Unix filesystem semantics
> because of sheer cowardice (excepting Clearcase, which priced itself
> into a niche market).   It is 25 years past time for someone to change
> things.  That someone will have first mover advantage, and the more
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into
> depending on it and in a few years the other filesystems will
> (deservedly) have only a small market share because the apps won't all
> work on them.
> 
> Besides, there are enhancements which are simply compelling.  You can
> write a dramatically better performance version control system with a
> much simpler design if the FS is atomic.    Our transaction manager
> first draft was written by a version control guy, and he would probably
> be happy to tell you how  lack of atomicity other than rename makes
> version control software design hideous.
> 
> We have the performance lead.  By next year we will be stable enough for
> mission critical servers, and then we start the serious semantic
> enhancements.
> 
> If you don't embrace progress, then you doom Linux to following behind,
> because the guys at Apple are pretty aggressive now that Jobs is back,
> and they WILL change the semantics, and they will do so in compelling
> ways, and Linux will be reduced to aping them when it should be leading
> them.
> 
> Hans
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Oliver Hunt wrote:

>How would we go about finding out how many data forks were in a file? 
>Because in order to be able to retrieve data from a fork we would need
>to know that the fork were there.  Currently this would imply that we
>go looking through mtab or some such to find out what fs we're running
>on, which seems ugly.
>
>  
>
Depends on how the forks eventually get implemented.
With the file-as-directory concept, all you need is to
look at the file's directory part to see what is there.  (The forks,
implemented as files in a subdirectory.)  It is done the same way
as for an ordinary directory, so nothing "new".

>Alternatively we go through the _exciting_ task of making every other
>fs (with the exceptions of ntfs, and whatever it is that macs use,
>which would need there own custom code) and add code that effectively
>goes
>
>getNumForks(fileref){ return 1;} 
>
>  
>
Necessary if some other mechanism is used, sure.
Helge Hafting

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Oliver Hunt wrote:

>How would we go about finding out how many data forks were in a file? 
>Because in order to be able to retrieve data from a fork we would need
>to know that the fork were there.  Currently this would imply that we
>go looking through mtab or some such to find out what fs we're running
>on, which seems ugly.
>
>  
>
Depends on how the forks eventually get implemented.
With the file-as-directory concept, all you need is to
look at the file's directory part to see what is there.  (The forks,
implemented as files in a subdirectory.)  It is done the same way
as for an ordinary directory, so nothing "new".

>Alternatively we go through the _exciting_ task of making every other
>fs (with the exceptions of ntfs, and whatever it is that macs use,
>which would need there own custom code) and add code that effectively
>goes
>
>getNumForks(fileref){ return 1;} 
>
>  
>
Necessary if some other mechanism is used, sure.
Helge Hafting

Re: The argument for fs assistance in handling archives

[Oliver Hunt]

> Depends on how the forks eventually get implemented.
> With the file-as-directory concept, all you need is to
> look at the file's directory part to see what is there.  (The forks,
> implemented as files in a subdirectory.)  It is done the same way
> as for an ordinary directory, so nothing "new".
> 
This still doesn't solve the problem of how we distinguish between a
multi-fork file
and a directory, for those old programs(ie. almost all that currently
exist), to be able to access meaningful data we would need to either
return just the primary fork, a serialized version of all forks in the
file, or make the file look more or less identical to a directory.

The first option could cause problems when transfering files between
different filesystems,
the second results in programs getting metadata they can't handle, and
the third option results in few of the advantages over, well,
directories...  And even those applications that could handle the fork
information nicely would need to fs type to find out whether they were
handling a directory or a multi-forked file...

As I say I like the idea, but I can't see anyway of implementing it in
a way that is useful without first putting considerable effort into at
least the VFS if not all the actual fs drivers.

--Oliver Hunt

Re: The argument for fs assistance in handling archives

[Spam]

  

>> Depends on how the forks eventually get implemented.
>> With the file-as-directory concept, all you need is to
>> look at the file's directory part to see what is there.  (The forks,
>> implemented as files in a subdirectory.)  It is done the same way
>> as for an ordinary directory, so nothing "new".
>> 
> This still doesn't solve the problem of how we distinguish between a
> multi-fork file
> and a directory, for those old programs(ie. almost all that currently
> exist), to be able to access meaningful data we would need to either
> return just the primary fork, a serialized version of all forks in the
> file, or make the file look more or less identical to a directory.

  Well. wasn't the idea that unless programs specifically tried to
  open the file-as-dir as a directory it would look like a file?

  ls -F would show it as file. Or have I understood wrong?

> The first option could cause problems when transfering files between
> different filesystems,

  The meta-data should be deleted if it the file is copied or moved to
  a medium that doesn't support it. However a _warning_ may be shown
  to the user if there is risk to loose data.

  I do not think we should not implement something like this because
  most other filesystems in Linux doesn't support it. Think of other
  limitations there is in various ones. Large file support. Limited
  amount of files, length of file names, etc.

> the second results in programs getting metadata they can't handle, and
> the third option results in few of the advantages over, well,
> directories...  And even those applications that could handle the fork
> information nicely would need to fs type to find out whether they were
> handling a directory or a multi-forked file...

  No, if an application supported the file-as-dir then it could simply
  try to open the file as a directory. It would fail on systems that
  didn't support it.

  The meta-data and file streams would be seen as ordinary files. If
  applications support the contents or not isn't really relevant, is
  it?.

> As I say I like the idea, but I can't see anyway of implementing it in
> a way that is useful without first putting considerable effort into at
> least the VFS if not all the actual fs drivers.

  Indeed. It is important that something like this gets implemented as
  a transparent way as possible. If it could be done in a general way
  so other filesystems like ext3/4 can eventually support it then that
  would be wonderful. I do not, however, think that we should block it
  in reiser4 because no other filesystems support it.

  ~S

> --Oliver Hunt

Re: The argument for fs assistance in handling archives

[Spam]

  

>> Depends on how the forks eventually get implemented.
>> With the file-as-directory concept, all you need is to
>> look at the file's directory part to see what is there.  (The forks,
>> implemented as files in a subdirectory.)  It is done the same way
>> as for an ordinary directory, so nothing "new".
>> 
> This still doesn't solve the problem of how we distinguish between a
> multi-fork file
> and a directory, for those old programs(ie. almost all that currently
> exist), to be able to access meaningful data we would need to either
> return just the primary fork, a serialized version of all forks in the
> file, or make the file look more or less identical to a directory.

  Well. wasn't the idea that unless programs specifically tried to
  open the file-as-dir as a directory it would look like a file?

  ls -F would show it as file. Or have I understood wrong?

> The first option could cause problems when transfering files between
> different filesystems,

  The meta-data should be deleted if it the file is copied or moved to
  a medium that doesn't support it. However a _warning_ may be shown
  to the user if there is risk to loose data.

  I do not think we should not implement something like this because
  most other filesystems in Linux doesn't support it. Think of other
  limitations there is in various ones. Large file support. Limited
  amount of files, length of file names, etc.

> the second results in programs getting metadata they can't handle, and
> the third option results in few of the advantages over, well,
> directories...  And even those applications that could handle the fork
> information nicely would need to fs type to find out whether they were
> handling a directory or a multi-forked file...

  No, if an application supported the file-as-dir then it could simply
  try to open the file as a directory. It would fail on systems that
  didn't support it.

  The meta-data and file streams would be seen as ordinary files. If
  applications support the contents or not isn't really relevant, is
  it?.

> As I say I like the idea, but I can't see anyway of implementing it in
> a way that is useful without first putting considerable effort into at
> least the VFS if not all the actual fs drivers.

  Indeed. It is important that something like this gets implemented as
  a transparent way as possible. If it could be done in a general way
  so other filesystems like ext3/4 can eventually support it then that
  would be wonderful. I do not, however, think that we should block it
  in reiser4 because no other filesystems support it.

  ~S

> --Oliver Hunt

Re: The argument for fs assistance in handling archives

[Giuliano Pochini]

On Thu, 2 Sep 2004, Spam wrote:

>   Well. wasn't the idea that unless programs specifically tried to
>   open the file-as-dir as a directory it would look like a file?
>
>   ls -F would show it as file. Or have I understood wrong?

Yes, otherwise apps that do

if (S_ISDIR()) {
	..
} else if (S_ISREG()) {
	..
}

would behave differently from apps that check the file in
different order, and they would probably break because in
a regular fs that order is not important. The goal is to
keep the semantic changes as hidden as possible for apps
that don't know about the new features.


--
Giuliano.

Re: The argument for fs assistance in handling archives

[Christian Mayrhuber]

On Thursday 02 September 2004 12:02, Spam wrote:
> > As I say I like the idea, but I can't see anyway of implementing it in
> > a way that is useful without first putting considerable effort into at
> > least the VFS if not all the actual fs drivers.
> 
>   Indeed. It is important that something like this gets implemented as
>   a transparent way as possible. If it could be done in a general way
>   so other filesystems like ext3/4 can eventually support it then that
>   would be wonderful. I do not, however, think that we should block it
>   in reiser4 because no other filesystems support it.

What about extending the namespace with leading ".." and "...".

In Unix names starting with a  "." already have the meaning of being a 
hidden/config entry.

A name starting ".." means streams/metainformation, etc. something belonging 
to the file/directory that should get backed up, copied, etc.
There could be a "..streams", "..metas" or a "..acl" entry.

If it starts with "..." it means some system specific information, like the
name of the hash algorithm used for the current directory. It's basically 
information that is not portable and not required by applications, like
"...fsplugins".

There is still the big problem of how to copy files with their associated 
streams or meta information onto a standard unix filesystem as the 
file/directory duality cannot be expressed. (It's forbidden to have
a directory + file with the same name in the same directory)
Maybe copy could create something like a ".#filename" directory for this kind
of information if the advanced features are not supported on the target 
filesystem. This is neither nice nor clean, but at least you don't loose 
information. I do not suggest that the kernel should simulate the advanced 
features with  ".#filename" directories, it's more a backup/restore thing 
that could work if it's written down properly.

-- 
lg, Chris

Re: The argument for fs assistance in handling archives

  

> Linus Torvalds wrote:

>> But _my_ point is, no user program is going to take _advantage_ of
>>
>>anything that only one filesystem on one system offers.
>>  
>>
> Apple does not have this problem....

> and yes, the apps will take advantage of it, which is different from
> depending on it.  If you use the wrong fs you will lose some of the 
> features of the app.

> For 30 years nothing much has happened in Unix filesystem semantics 
> because of sheer cowardice (excepting Clearcase, which priced itself
> into a niche market).   It is 25 years past time for someone to change
> things.  That someone will have first mover advantage, and the more 
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into 
> depending on it and in a few years the other filesystems will 
> (deservedly) have only a small market share because the apps won't all
> work on them.

> Besides, there are enhancements which are simply compelling.  You can
> write a dramatically better performance version control system with a
> much simpler design if the FS is atomic.    Our transaction manager 
> first draft was written by a version control guy, and he would probably
> be happy to tell you how  lack of atomicity other than rename makes 
> version control software design hideous.

  Btw, version control for ordinary files would be a great feature. I
  think something like it is available through Windows 2000/3 server.
  Isn't it called "Shadow Copies". It works over network shares. :)

  It allows you to restore previous versions of the file even if you
  delete or overwrite it.

  Features like this do make a good point and helps protecting data -
  something that is important IMHO.
  
  http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp


> We have the performance lead.  By next year we will be stable enough for
> mission critical servers, and then we start the serious semantic 
> enhancements.

> If you don't embrace progress, then you doom Linux to following behind,
> because the guys at Apple are pretty aggressive now that Jobs is back,
> and they WILL change the semantics, and they will do so in compelling
> ways, and Linux will be reduced to aping them when it should be leading
> them.

> Hans

Re: The argument for fs assistance in handling archives

[Spam]

  

> Linus Torvalds wrote:

>> But _my_ point is, no user program is going to take _advantage_ of
>>
>>anything that only one filesystem on one system offers.
>>  
>>
> Apple does not have this problem....

> and yes, the apps will take advantage of it, which is different from
> depending on it.  If you use the wrong fs you will lose some of the 
> features of the app.

> For 30 years nothing much has happened in Unix filesystem semantics 
> because of sheer cowardice (excepting Clearcase, which priced itself
> into a niche market).   It is 25 years past time for someone to change
> things.  That someone will have first mover advantage, and the more 
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into 
> depending on it and in a few years the other filesystems will 
> (deservedly) have only a small market share because the apps won't all
> work on them.

> Besides, there are enhancements which are simply compelling.  You can
> write a dramatically better performance version control system with a
> much simpler design if the FS is atomic.    Our transaction manager 
> first draft was written by a version control guy, and he would probably
> be happy to tell you how  lack of atomicity other than rename makes 
> version control software design hideous.

  Btw, version control for ordinary files would be a great feature. I
  think something like it is available through Windows 2000/3 server.
  Isn't it called "Shadow Copies". It works over network shares. :)

  It allows you to restore previous versions of the file even if you
  delete or overwrite it.

  Features like this do make a good point and helps protecting data -
  something that is important IMHO.
  
  http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp


> We have the performance lead.  By next year we will be stable enough for
> mission critical servers, and then we start the serious semantic 
> enhancements.

> If you don't embrace progress, then you doom Linux to following behind,
> because the guys at Apple are pretty aggressive now that Jobs is back,
> and they WILL change the semantics, and they will do so in compelling
> ways, and Linux will be reduced to aping them when it should be leading
> them.

> Hans

Re: The argument for fs assistance in handling archives

[Alan Cox]

On Iau, 2004-09-02 at 10:52, Spam wrote:
>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)

Netapps have done this for years with NFS, Subversion does it with DAV,
clearcase does it with NFS.

Re: The argument for fs assistance in handling archives

[V13]

On Thursday 02 September 2004 12:52, Spam wrote:
> > Linus Torvalds wrote:
> > Besides, there are enhancements which are simply compelling.  You can
> > write a dramatically better performance version control system with a
> > much simpler design if the FS is atomic.    Our transaction manager
> > first draft was written by a version control guy, and he would probably
> > be happy to tell you how  lack of atomicity other than rename makes
> > version control software design hideous.
>
>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)
>
>   It allows you to restore previous versions of the file even if you
>   delete or overwrite it.
>
>   Features like this do make a good point and helps protecting data -
>   something that is important IMHO.

I believe you mean something simillar to: 

file1.txt;1
file1.txt;2
file1.txt;3 (yeap, it's VMS) 

where you'll have to cleanup old versions when you don't need them any more... 
AFAIK that this is older than HDDs

<<V13>>

Re: The argument for fs assistance in handling archives

[John Stoffel]

V13> I believe you mean something simillar to: 

V13> file1.txt;1
V13> file1.txt;2
V13> file1.txt;3 (yeap, it's VMS) 

Or TOPS-20, a precursor to VMS in some ways.  It was a nice feature.

V13> where you'll have to cleanup old versions when you don't need
V13> them any more...  AFAIK that this is older than HDDs

It was usually an automatic cleanup past a certain point or if you
went over disk quota.  

In any case, while I do like this feature, I'm not sure how we would
cleanly implement this inside the unix namespace, or if inside a new
namespace, how that new namespace would be joined with a standard Unix
one.

John

Re: The argument for fs assistance in handling archives

[Hubert Chan]

>>>>> "John" == John Stoffel <stoffel@lucent.com> writes:

V13> I believe you mean something simillar to:
V13> file1.txt;1
V13> file1.txt;2
V13> file1.txt;3 (yeap, it's VMS)

[...]

John> In any case, while I do like this feature, I'm not sure how we
John> would cleanly implement this inside the unix namespace, or if
John> inside a new namespace, how that new namespace would be joined
John> with a standard Unix one.

  file1.txt/..version/1

or, for those version control systems like svn and arch where the
version number refers to the version of the whole project tree:

  project/..version/1/file1.txt

You could even export strange things like:

  project/..by-date/2004-08-24/file1.txt

to see what the file looked like on August 24.  Or

  project/..diff/3-1/file1.txt

to see a diff between version 3 and 1.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: The argument for fs assistance in handling archives

[Hubert Chan]

>>>>> "John" == John Stoffel <stoffel@lucent.com> writes:

V13> I believe you mean something simillar to:
V13> file1.txt;1
V13> file1.txt;2
V13> file1.txt;3 (yeap, it's VMS)

[...]

John> In any case, while I do like this feature, I'm not sure how we
John> would cleanly implement this inside the unix namespace, or if
John> inside a new namespace, how that new namespace would be joined
John> with a standard Unix one.

  file1.txt/..version/1

or, for those version control systems like svn and arch where the
version number refers to the version of the whole project tree:

  project/..version/1/file1.txt

You could even export strange things like:

  project/..by-date/2004-08-24/file1.txt

to see what the file looked like on August 24.  Or

  project/..diff/3-1/file1.txt

to see a diff between version 3 and 1.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: The argument for fs assistance in handling archives

[Oliver Neukum]

Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)
> 
>   It allows you to restore previous versions of the file even if you
>   delete or overwrite it.

There's no need to do that in kernel, unless you want to be able
to force it unto users.

	Regards
		Oliver

Re: The argument for fs assistance in handling archives

[Oliver Neukum]

Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)
> 
>   It allows you to restore previous versions of the file even if you
>   delete or overwrite it.

There's no need to do that in kernel, unless you want to be able
to force it unto users.

	Regards
		Oliver

Re: The argument for fs assistance in handling archives

[Spam]

  

> Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
>>   Btw, version control for ordinary files would be a great feature. I
>>   think something like it is available through Windows 2000/3 server.
>>   Isn't it called "Shadow Copies". It works over network shares. :)
>> 
>>   It allows you to restore previous versions of the file even if you
>>   delete or overwrite it.

> There's no need to do that in kernel, unless you want to be able
> to force it unto users.

  Exactly ;)

  Difference with having it in just certain applications like Gnome-VFS
  etc is that it would work with all applications, over SMB shares
  etc. This is a great advantage and benefit for many desktop users
  but also in a corporate environment. Even though there are backups
  of users documents it is a pain to restore them every time they
  loose the original file. Besides, the backup is never as fresh
  either.
  

> 	Regards
> 		Oliver

Re: The argument for fs assistance in handling archives

[Spam]

  

> Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
>>   Btw, version control for ordinary files would be a great feature. I
>>   think something like it is available through Windows 2000/3 server.
>>   Isn't it called "Shadow Copies". It works over network shares. :)
>> 
>>   It allows you to restore previous versions of the file even if you
>>   delete or overwrite it.

> There's no need to do that in kernel, unless you want to be able
> to force it unto users.

  Exactly ;)

  Difference with having it in just certain applications like Gnome-VFS
  etc is that it would work with all applications, over SMB shares
  etc. This is a great advantage and benefit for many desktop users
  but also in a corporate environment. Even though there are backups
  of users documents it is a pain to restore them every time they
  loose the original file. Besides, the backup is never as fresh
  either.
  

> 	Regards
> 		Oliver

Re: The argument for fs assistance in handling archives

[Horst von Brand]

Spam <spam@tnonline.net> said:

[...]

>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)

Right. And when space runs low, the old copies get the axe. And then you
need the old version... and are royally screwed because you trusted the
system to save it for you while willy-nilly fooling around.

I never liked the MS-DOS "undelete" feature, because I saw too many people
burned. It is _less_ data security than "deleted file is gone for good", in
the end sum.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives

[Lee Revell]

On Thu, 2004-09-02 at 05:52, Spam wrote:

>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)
> 
>   It allows you to restore previous versions of the file even if you
>   delete or overwrite it.
> 
>   Features like this do make a good point and helps protecting data -
>   something that is important IMHO.
>   
>   http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp
> 

Netapp's WAFL has had this for ages, they are called snapshots, and they
are one of the most useful file system features ever.  But Microsoft
certainly did not invent them.

Lee

Re: The argument for fs assistance in handling archives

[Lee Revell]

On Thu, 2004-09-02 at 05:52, Spam wrote:

>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)
> 
>   It allows you to restore previous versions of the file even if you
>   delete or overwrite it.
> 
>   Features like this do make a good point and helps protecting data -
>   something that is important IMHO.
>   
>   http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp
> 

Netapp's WAFL has had this for ages, they are called snapshots, and they
are one of the most useful file system features ever.  But Microsoft
certainly did not invent them.

Lee

Re: The argument for fs assistance in handling archives

[Bostjan Skufca @ domenca.com]

I think Hans' idea about further evolution is right.

(2.7 which will lead into 3.0 maybe?)


Regards,
Bostjan


On Thursday 02 of September 2004 10:58, Hans Reiser wrote:
> Linus Torvalds wrote:
> > But _my_ point is, no user program is going to take _advantage_ of
> >
> >anything that only one filesystem on one system offers.
>
> Apple does not have this problem....
>
> and yes, the apps will take advantage of it, which is different from
> depending on it.  If you use the wrong fs you will lose some of the
> features of the app.
>
> For 30 years nothing much has happened in Unix filesystem semantics
> because of sheer cowardice (excepting Clearcase, which priced itself
> into a niche market).   It is 25 years past time for someone to change
> things.  That someone will have first mover advantage, and the more
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into
> depending on it and in a few years the other filesystems will
> (deservedly) have only a small market share because the apps won't all
> work on them.
>
> Besides, there are enhancements which are simply compelling.  You can
> write a dramatically better performance version control system with a
> much simpler design if the FS is atomic.    Our transaction manager
> first draft was written by a version control guy, and he would probably
> be happy to tell you how  lack of atomicity other than rename makes
> version control software design hideous.
>
> We have the performance lead.  By next year we will be stable enough for
> mission critical servers, and then we start the serious semantic
> enhancements.
>
> If you don't embrace progress, then you doom Linux to following behind,
> because the guys at Apple are pretty aggressive now that Jobs is back,
> and they WILL change the semantics, and they will do so in compelling
> ways, and Linux will be reduced to aping them when it should be leading
> them.
>
> Hans

Re: The argument for fs assistance in handling archives

[John Stoffel]

Hans> For 30 years nothing much has happened in Unix filesystem
Hans> semantics because of sheer cowardice (excepting Clearcase, which
Hans> priced itself into a niche market).

Keeping a consistent namespace is cowardice?  And ClearCase is an
example of an application which sets up it's own namespace and grafts
it onto the standard Unix one.  Most standard Unix tools work just
fine inside clearcase Views, but to manage the metadata (xattrs, or
whatever you want to call them), you need to use the 'cleartool'
command.  Hmmm... sounds like 'runat' to me.

Hans> It is 25 years past time for someone to change things.  That
Hans> someone will have first mover advantage, and the more little
Hans> semantic features possessed the more lure there will be to use
Hans> it which will increase market share which will lure more apps
Hans> into depending on it and in a few years the other filesystems
Hans> will (deservedly) have only a small market share because the
Hans> apps won't all work on them.

This is all pure marketing speak and economic theory.  Show us the
*technical* advantages, not just wishful thinking.  

Hans> Besides, there are enhancements which are simply compelling.
Hans> You can write a dramatically better performance version control
Hans> system with a much simpler design if the FS is atomic. 

Define atomic please, with state diagrams and clear examples.  

Hans> We have the performance lead.  By next year we will be stable
Hans> enough for mission critical servers, and then we start the
Hans> serious semantic enhancements.

I think you've got it backwards.  Make your serious semantic
enhancements first, then make it stable, then make it fast.  Because
when you change the semantics, you break all kinds of things and then
it doesn't matter how fast you are.

Hans> If you don't embrace progress, then you doom Linux to following
Hans> behind, because the guys at Apple are pretty aggressive now that
Hans> Jobs is back, and they WILL change the semantics, and they will
Hans> do so in compelling ways, and Linux will be reduced to aping
Hans> them when it should be leading them.

Monkey see, Monkey do then.  


I'd like to point out another successful company which has extended
the standard Unix namespace and that's 'Network Appliance' with it's
.snapshot directory structure.  It's a great idea and allows my users
to restore files from snapshots without me having to think about it.  

But it still causes problems since when I use rsync to move data, I
need to put in stuff like:

  rsync -az --exclude ".snapshot" --delete --delete-excluded <src> <dest>

to make sure it doesn't descend into that directory of previous
versions and try to copy them over as well.  And of course now now
other apps can use the .snapshot name.  But what if other vendors aped
(ook ook) this decided to use their own names?  Who decides which gets
priority?  

I think you need to go back and re-read Pike's paper on namespaces
that you pointed to before and mull it over.  And look at how
simplicity is inherently powerful.  If the design is too complicated,
you're probably doing it wrong.

John

Re: The argument for fs assistance in handling archives

[Alan Cox]

On Iau, 2004-09-02 at 15:24, John Stoffel wrote:
> This is all pure marketing speak and economic theory. 

I wouldn't grace it with the label economic theory. Lets be honest about
this.. its pure -research-. Research is good but don't screw around with
Linux semantics until you've done the research, figured out how to also
make it useful to real users and proved its a good idea despite the past
evidence from things like MacOS


Alan

Re: The argument for fs assistance in handling archives

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> Linus Torvalds wrote:

> > But _my_ point is, no user program is going to take _advantage_ of
> >anything that only one filesystem on one system offers.

> Apple does not have this problem....

... because Apple customers are a captive lot...

> and yes, the apps will take advantage of it, which is different from 
> depending on it.  If you use the wrong fs you will lose some of the 
> features of the app.

... which is undesirable to say the very least for cross-platform Unix
(even worse, Unix + Windows) applications, so applications won't ever use
it (until Linux + super-duper filesystem is more than, say, 80% of the
market). I.e., it won't happen soon enough for this to make sense right
now. Perhaps in 13 more years...

> For 30 years nothing much has happened in Unix filesystem semantics 
> because of sheer cowardice

Careful threading there... calling respected people names without any proof
isn't exactly great PR.

>                            (excepting Clearcase, which priced itself 
> into a niche market).  It is 25 years past time for someone to change
> things.

Your opinion. Some people disagree.

>          That someone will have first mover advantage, and the more
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into depending
> on it and in a few years the other filesystems will (deservedly) have
> only a small market share because the apps won't all work on them.

You seem to be blinded by the "success effect": You only ever see the ones
who succeed, you don't see the masses who tried to be "first movers" and
failed utterly. 

> Besides, there are enhancements which are simply compelling.  You can 
> write a dramatically better performance version control system with a 
> much simpler design if the FS is atomic.

Iff the performance of the FS is better _at doing what the system wants
than the system itself_. That the CVS is simpler/faster is no use
whatsoever if the underlying OS is more complex/slower. Besides, what is
ideal for CVS isn't necesarily good for word processing or databases, and
probably irrelevant or at least overkill for a lot of other applications,
which all end up paying the costs.

Concentrate on giving what _most_ applications want, make it possible for
applications with weird requirements to do their job.

>                                             Our transaction manager 
> first draft was written by a version control guy, and he would probably 
> be happy to tell you how  lack of atomicity other than rename makes 
> version control software design hideous.

Right. But it doesn't make the design of text editing, or word processing,
or mail handling, or... hideous, does it? Having a strong background in an
area does put blinds on one's eyes.

> We have the performance lead.

I recently saw (preliminary) numbers directly contradicting this somewhere.

>                                By next year we will be stable enough for 
> mission critical servers, and then we start the serious semantic 
> enhancements.

By which the performance lead (if any) will do down the drain. There ain't
no free lunch.

> If you don't embrace progress, then you doom Linux to following behind, 
> because the guys at Apple are pretty aggressive now that Jobs is back, 
> and they WILL change the semantics, and they will do so in compelling 
> ways, and Linux will be reduced to aping them when it should be leading 
> them.

Why on earth should Linux copy a niche system? By now Linux's marketshare
is _much_ larger than Apple's... I'd say it is the other way around: Apple
tries desperately to differentiate itself from MSFT (and traditional Unix +
Linux) in order not to dissapear (and keep their grip on their users). If
they are doing it right, only time will tell. But that is no reason to
screw up Linux' strengths.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Thu, 2 Sep 2004, Hans Reiser wrote:
>
> Apple does not have this problem....

Deep, Hans. Deep.

You're ignoring the issue. You're flaming about not "embracing progress", 
when I was talking about how progress happens. You don't seem to 
understand _how_ people embrace progress, Hans. Railing at windmills isn't 
real smart.

		Linus

Re: The argument for fs assistance in handling archives

[Martin J. Bligh]

> For 30 years nothing much has happened in Unix filesystem semantics 
> because of sheer cowardice 

Or because it works fine, and isn't broken.

M.

Re: The argument for fs assistance in handling archives

[Christer Weinigel]

"Martin J. Bligh" <mbligh@aracnet.com> writes:

> > For 30 years nothing much has happened in Unix filesystem semantics 
> > because of sheer cowardice 
> 
> Or because it works fine, and isn't broken.

I've heard the same argument a lot of times.  People complaining that
Unix is so seventies because it sticks to the old boring philosophy of
everything is a file and that a file is a stream of bytes, nothing
more.  Modern operating systems such as VMS with basic database
handling in the OS itself, or MacOS or NT with named streams is so
much more modern.  Why don't we get with the times?

It may be because just because of the simplicity it's fairly easy to
use, harder to break och does one thing well.  If you want structured
storage, use a database, on top of the low level primitives, or use
multiple files in a directory.  Why complicate things?

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: The argument for fs assistance in handling archives

[Christian Iversen]

On Thursday 02 September 2004 20:38, Christer Weinigel wrote:
> "Martin J. Bligh" <mbligh@aracnet.com> writes:
> > > For 30 years nothing much has happened in Unix filesystem semantics
> > > because of sheer cowardice
> >
> > Or because it works fine, and isn't broken.
>
> I've heard the same argument a lot of times.  People complaining that
> Unix is so seventies because it sticks to the old boring philosophy of
> everything is a file and that a file is a stream of bytes, nothing
> more.  Modern operating systems such as VMS with basic database
> handling in the OS itself, or MacOS or NT with named streams is so
> much more modern.  Why don't we get with the times?
>
> It may be because just because of the simplicity it's fairly easy to
> use, harder to break och does one thing well.  If you want structured
> storage, use a database, on top of the low level primitives, or use
> multiple files in a directory.  Why complicate things?

Sounds very much like the reason for using only triangle primitives on modern 
3D-graphics hardware. And that works :)

-- 
Regards,
Christian Iversen

Re: The argument for fs assistance in handling archives

[Steve Bergman]

On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote:
> > For 30 years nothing much has happened in Unix filesystem semantics 
> > because of sheer cowardice 
> 
> Or because it works fine, and isn't broken.

OK.  I'm not a kernel hacker.  I'm not a crack C programmer.  Nor am I a
designer of filesystems.  I'm just a guy that puts together and supports
Linux systems for my customers.

In following this thread, I may be missing huge chunks of concept.

However, a few things are becoming clear to me:

1. The file as directory thing adds complexity that the administrator
has to deal with.  Symlinks are useful, but it's still aggravating to
tar off a directory structure, take it somewhere, and then realize that
all you have is links to something not in the archive because you didn't
get your tar switches just right.  Now we're talking about adding
another set of "files which are not really files" to the semantics.
More complexity.  I'll take simplicity over some ivory tower ideal of
"unified name space" any day.

2. The use of multiple streams within files by Linux apps would make
Linux as cross-platform unfriendly as MS is trying to be.  Say these
features start getting used and you copy an OO.org document from a Linux
box to a BSD box.  It's broken.  Of course, OO.org wouldn't use the
streams in the first place because it would destroy their cross platform
portability.  So what's the point?  No one who cares about cross
platform portability can use it.  Everyone who doesn't care about cross
platform portability please raise your hand.

3. MS does require attributes and multiple streams, which makes these
features important (even essential) to Samba, and Samba alone.  Samba is
important to Linux, so this can't be ignored. (Here I am implicitly
assuming that Samba will need kernel support for this to do it right.)

So it seems to me that the only real consideration is giving Samba what
it needs without making the semantics one bit more complex than
absolutely necessary.  It might even be wise to discourage use of these
ambiguous new objects by the casual application programmer.

Then again, maybe I just have tunnel vision... 


-Steve Bergman

Re: The argument for fs assistance in handling archives

[Steve Bergman]

On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote:
> > For 30 years nothing much has happened in Unix filesystem semantics 
> > because of sheer cowardice 
> 
> Or because it works fine, and isn't broken.

OK.  I'm not a kernel hacker.  I'm not a crack C programmer.  Nor am I a
designer of filesystems.  I'm just a guy that puts together and supports
Linux systems for my customers.

In following this thread, I may be missing huge chunks of concept.

However, a few things are becoming clear to me:

1. The file as directory thing adds complexity that the administrator
has to deal with.  Symlinks are useful, but it's still aggravating to
tar off a directory structure, take it somewhere, and then realize that
all you have is links to something not in the archive because you didn't
get your tar switches just right.  Now we're talking about adding
another set of "files which are not really files" to the semantics.
More complexity.  I'll take simplicity over some ivory tower ideal of
"unified name space" any day.

2. The use of multiple streams within files by Linux apps would make
Linux as cross-platform unfriendly as MS is trying to be.  Say these
features start getting used and you copy an OO.org document from a Linux
box to a BSD box.  It's broken.  Of course, OO.org wouldn't use the
streams in the first place because it would destroy their cross platform
portability.  So what's the point?  No one who cares about cross
platform portability can use it.  Everyone who doesn't care about cross
platform portability please raise your hand.

3. MS does require attributes and multiple streams, which makes these
features important (even essential) to Samba, and Samba alone.  Samba is
important to Linux, so this can't be ignored. (Here I am implicitly
assuming that Samba will need kernel support for this to do it right.)

So it seems to me that the only real consideration is giving Samba what
it needs without making the semantics one bit more complex than
absolutely necessary.  It might even be wise to discourage use of these
ambiguous new objects by the casual application programmer.

Then again, maybe I just have tunnel vision... 


-Steve Bergman

Re: The argument for fs assistance in handling archives

[Spam]

  

> On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote:
>> > For 30 years nothing much has happened in Unix filesystem semantics
>> > because of sheer cowardice 
>> 
>> Or because it works fine, and isn't broken.

> OK.  I'm not a kernel hacker.  I'm not a crack C programmer.  Nor am I a
> designer of filesystems.  I'm just a guy that puts together and supports
> Linux systems for my customers.

> In following this thread, I may be missing huge chunks of concept.

> However, a few things are becoming clear to me:

> 1. The file as directory thing adds complexity that the administrator
> has to deal with.  Symlinks are useful, but it's still aggravating to
> tar off a directory structure, take it somewhere, and then realize that
> all you have is links to something not in the archive because you didn't
> get your tar switches just right.  Now we're talking about adding
> another set of "files which are not really files" to the semantics.
> More complexity.  I'll take simplicity over some ivory tower ideal of
> "unified name space" any day.

  Are you afraid to learn something new? ;) Just joking. But really,
  it doesn't have to be very difficult. The extra streams etc would
  just be saved as files. If tar is patched then it would be no
  problem and no extra stuff but perhaps a switch --save-metas.

> 2. The use of multiple streams within files by Linux apps would make
> Linux as cross-platform unfriendly as MS is trying to be.  Say these
> features start getting used and you copy an OO.org document from a Linux
> box to a BSD box.  It's broken.  Of course, OO.org wouldn't use the
> streams in the first place because it would destroy their cross platform
> portability.  So what's the point?  No one who cares about cross
> platform portability can use it.  Everyone who doesn't care about cross
> platform portability please raise your hand.

  Actually MacOS and Windows support multiple streams, only Linux
  doesn't. But of course there are BSD's etc too. I'd say to leave
  them behind.

  The file streams would make my day a lot easier. The idea to
  split up contents of OO.org files into streams is bad. But that
  doesn't make the file streams bad. I see many uses that would make
  my every day life easier.

  It isn't about cross plat form compatibility, but to add features
  that are useful and meta-data and file streams are.

  Also. No one forces you to use either meta-data or streams, just as
  no one forces you to use ACLs or other things.

> 3. MS does require attributes and multiple streams, which makes these
> features important (even essential) to Samba, and Samba alone.  Samba is
> important to Linux, so this can't be ignored. (Here I am implicitly
> assuming that Samba will need kernel support for this to do it right.)

  I do not think the Samba would really require the streams support
  but it would certainly make life easier for Samba. Not to mention
  that these files would also be natively viewed on the Linux host.

> So it seems to me that the only real consideration is giving Samba what
> it needs without making the semantics one bit more complex than
> absolutely necessary.  It might even be wise to discourage use of these
> ambiguous new objects by the casual application programmer.

> Then again, maybe I just have tunnel vision... 

  I would agree with tunnel vision. The kernel should provide the
  tools and options. Users and developers can then invent new things
  to use them. :)

  ~S

> -Steve Bergman

Re: The argument for fs assistance in handling archives

[Spam]

  

> On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote:
>> > For 30 years nothing much has happened in Unix filesystem semantics
>> > because of sheer cowardice 
>> 
>> Or because it works fine, and isn't broken.

> OK.  I'm not a kernel hacker.  I'm not a crack C programmer.  Nor am I a
> designer of filesystems.  I'm just a guy that puts together and supports
> Linux systems for my customers.

> In following this thread, I may be missing huge chunks of concept.

> However, a few things are becoming clear to me:

> 1. The file as directory thing adds complexity that the administrator
> has to deal with.  Symlinks are useful, but it's still aggravating to
> tar off a directory structure, take it somewhere, and then realize that
> all you have is links to something not in the archive because you didn't
> get your tar switches just right.  Now we're talking about adding
> another set of "files which are not really files" to the semantics.
> More complexity.  I'll take simplicity over some ivory tower ideal of
> "unified name space" any day.

  Are you afraid to learn something new? ;) Just joking. But really,
  it doesn't have to be very difficult. The extra streams etc would
  just be saved as files. If tar is patched then it would be no
  problem and no extra stuff but perhaps a switch --save-metas.

> 2. The use of multiple streams within files by Linux apps would make
> Linux as cross-platform unfriendly as MS is trying to be.  Say these
> features start getting used and you copy an OO.org document from a Linux
> box to a BSD box.  It's broken.  Of course, OO.org wouldn't use the
> streams in the first place because it would destroy their cross platform
> portability.  So what's the point?  No one who cares about cross
> platform portability can use it.  Everyone who doesn't care about cross
> platform portability please raise your hand.

  Actually MacOS and Windows support multiple streams, only Linux
  doesn't. But of course there are BSD's etc too. I'd say to leave
  them behind.

  The file streams would make my day a lot easier. The idea to
  split up contents of OO.org files into streams is bad. But that
  doesn't make the file streams bad. I see many uses that would make
  my every day life easier.

  It isn't about cross plat form compatibility, but to add features
  that are useful and meta-data and file streams are.

  Also. No one forces you to use either meta-data or streams, just as
  no one forces you to use ACLs or other things.

> 3. MS does require attributes and multiple streams, which makes these
> features important (even essential) to Samba, and Samba alone.  Samba is
> important to Linux, so this can't be ignored. (Here I am implicitly
> assuming that Samba will need kernel support for this to do it right.)

  I do not think the Samba would really require the streams support
  but it would certainly make life easier for Samba. Not to mention
  that these files would also be natively viewed on the Linux host.

> So it seems to me that the only real consideration is giving Samba what
> it needs without making the semantics one bit more complex than
> absolutely necessary.  It might even be wise to discourage use of these
> ambiguous new objects by the casual application programmer.

> Then again, maybe I just have tunnel vision... 

  I would agree with tunnel vision. The kernel should provide the
  tools and options. Users and developers can then invent new things
  to use them. :)

  ~S

> -Steve Bergman

Re: The argument for fs assistance in handling archives

[Martin J. Bligh]

>> 1. The file as directory thing adds complexity that the administrator
>> has to deal with.  Symlinks are useful, but it's still aggravating to
>> tar off a directory structure, take it somewhere, and then realize that
>> all you have is links to something not in the archive because you didn't
>> get your tar switches just right.  Now we're talking about adding
>> another set of "files which are not really files" to the semantics.
>> More complexity.  I'll take simplicity over some ivory tower ideal of
>> "unified name space" any day.
> 
>   Are you afraid to learn something new? ;) Just joking. But really,
>   it doesn't have to be very difficult. The extra streams etc would
>   just be saved as files. If tar is patched then it would be no
>   problem and no extra stuff but perhaps a switch --save-metas.

If they're saved as files, and the app has to be changed to use them
anyway, then what's the point? Just change the app to use new files
instead.

>   I would agree with tunnel vision. The kernel should provide the
>   tools and options. Users and developers can then invent new things
>   to use them. :)

Ugh. Change for changes sake is not a good thing. There's enough real
problems in the world without inventing random features. More complexity
without gain is a Bad Thing (tm).

I'm not saying streams is bad ... just that there don't seem to have
been very many convincing (to me) arguments raised for it yet. The 
versioning stuff would be nice, IMHO, because the stuff mainly using it 
wouldn't need to be modified in many instances ... "vi /etc/configfile" 
would keep old copies for you (only the recovery tool would need to
understand it). Saving icons seems easy enough to do with "foo.icon"
as another file.

M.

Re: The argument for fs assistance in handling archives

[Erik Hensema]

Steve Bergman (steve@rueb.com) wrote:
> However, a few things are becoming clear to me:
>
> 1. The file as directory thing adds complexity that the administrator
> has to deal with.  Symlinks are useful, but it's still aggravating to
> tar off a directory structure, take it somewhere, and then realize that
> all you have is links to something not in the archive because you didn't
> get your tar switches just right.  Now we're talking about adding
> another set of "files which are not really files" to the semantics.
> More complexity.  I'll take simplicity over some ivory tower ideal of
> "unified name space" any day.

As with any addition of complexity we have to ask ourselves: do
the advantages outweigh the disadvantages?

Adding complexity into the kernel may simplify userspace. Adding
the complexity of an IP layer into the kernel simplifies
networking applications, for example (they won't have to talk IP
to the kernel).

In this case it's hard to imagine what advantages a multistream
file will bring, simply because we aren't used to working with
them. Or maybe there aren't any advantages at all. Point is: we
don't know yet. Maybe we have to toy with the idea for some years
before we really learn why multistream files are so great.

I do realize that this isn't how Linux development is done. First
we need a problem, then somebody sits down and codes a solution.
With reiser4 we've got a solution but we haven't figured out what
the problem is yet.

> 2. The use of multiple streams within files by Linux apps would make
> Linux as cross-platform unfriendly as MS is trying to be.  Say these
> features start getting used and you copy an OO.org document from a Linux
> box to a BSD box.  It's broken.  Of course, OO.org wouldn't use the
> streams in the first place because it would destroy their cross platform
> portability.  So what's the point?  No one who cares about cross
> platform portability can use it.  Everyone who doesn't care about cross
> platform portability please raise your hand.

But do we really want to be the lowest common denominator? If you
want to lead, you can't be compatible (in some cases).
Also, if the reiser4 ideas catch on, they may spark the
development of other, similar, filesystems. Maybe we'll end up
with a ext4 in the 2.8 kernel...

> 3. MS does require attributes and multiple streams, which makes these
> features important (even essential) to Samba, and Samba alone.  Samba is
> important to Linux, so this can't be ignored. (Here I am implicitly
> assuming that Samba will need kernel support for this to do it right.)
>
> So it seems to me that the only real consideration is giving Samba what
> it needs without making the semantics one bit more complex than
> absolutely necessary.  It might even be wise to discourage use of these
> ambiguous new objects by the casual application programmer.
>
> Then again, maybe I just have tunnel vision... 

Not tunnel vision. You're just looking at what we need *now*, and
that happens to be exactly what we *have* now.

One possible use for multistream files that I see, is storage of
'not so important' data alongside of the real data of a file.
Think: the mimetype of a file; some metadata like author,
keywords, etc.

When you copy the file and lose all metadata except the file
itself, no real harm is done.

This could be used by the new mime system Gnome uses, for
example. On regular filesystems, the mimetype of a file is
determined by its extension. On reiser4(-like filesystems), the
mimetype is determined by the 'mimetype' stream.
This would clearly simplify the gnome userspace, as there is no
need anymore for a seperate extension<->mimetype conversion
anymore. It would also be compatible with other tools or desktop
environements, we only have to agree on what stream we're going
to use for what metadata.

IMHO reiser4 is a toy we should give the Linux users to play
with. Let's see what they can come up with. With the talented
hackers in the community, it should be great.

-- 
Erik Hensema <erik@hensema.net>
(attempting to write english while drinking beer)

Re: The argument for fs assistance in handling archives

[Hans Reiser]

Linus Torvalds wrote:

>
>But _my_ point is, no user program is going to take _advantage_ of
>anything that only one filesystem on one system offers.
>
>So there's no point.
>
>I
>
Filesystem diversity will only be crippling for Linux if you choose for 
it to be so.

Re: The argument for fs assistance in handling archives

[James Bruce]

Linus Torvalds wrote:

>...
>But _my_ point is, no user program is going to take _advantage_ of
>anything that only one filesystem on one system offers.
>
>So there's no point.
>
>It's much saner (from _any_ app standpoint) to roll their own database in 
>user space - that way it just works.
>
>In other words, nobody is really ever going to take advantage of this. 
>This is _not_ how technical advanncement happens. The way you get people 
>to take advantage of something is to have a nice gradual ramp-up, not a 
>sudden new feature that they can't realistically use.
>...
>

Sure, but there are plenty of existing interfaces that you could 
emulate.  One could make a small library to use a transactional 
filesystem to implement the Berkely DB interface (libdb) for example.  
So on filesystems without such support your app could use the regular 
userspace database, but on a transactional filesystem it'd just use 
regular files, which would simplify database management and likely 
increase performance over the userspace-only version (libdb is pretty 
slow).  In terms of functionality it'd just be a drop-in replacement, 
just like math libraries that use MMX/SSE when available.

    - Jim Bruce

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Thu, 2 Sep 2004, James Bruce wrote:
> Linus Torvalds wrote:
>
> >In other words, nobody is really ever going to take advantage of this. 
> >This is _not_ how technical advanncement happens. The way you get people 
> >to take advantage of something is to have a nice gradual ramp-up, not a 
> >sudden new feature that they can't realistically use.
> 
> Sure, but there are plenty of existing interfaces that you could 
> emulate. 

Absolutely. We should look at them, and whether they solve any issues. 

I'm not saying that Hans would have to make up a new interface. Quite the
reverse. One interface I suggested to use for attributes (and everybody
hates ;) was the existing one from Solaris. Similarly, there are probably 
perfectly fine interfaces for the issue of transactions..

		Linus

Re: The argument for fs assistance in handling archives

[Robin Rosenberg]

On Thursday 02 September 2004 09.15, Hans Reiser wrote:
> Linus Torvalds wrote:
> >. Doing transactions on one file is
> >only the beginning - you'll find people who want transactions across file
> >boundaries etc.
>
> Yup, that's what reiser4 is aiming at and where things get exciting for
> version control systems and the like.  We are just missing the api code
> for it at the moment, all the infrastructure is there, and the api is in
> progress in sys_reiser4.

Hope that is the same as or implied XA-support. Having to syncrhonized file 
updates with database transactions is not an uncommon problem. 

-- robin

Re: The argument for fs assistance in handling archives

[Hans Reiser]

Linus Torvalds wrote:

>On Wed, 1 Sep 2004, David Masover wrote:
>  
>
>>And that is the right solution.  Not the only one, but the right one.
>>Caching isn't the only thing sorely in need of transaction support right
>>now.  Actually, I find it hard to think of anything on Linux which
>>shouldn't have transactions -- why should /etc/fstab or
>>/home/david/homework be more fragile than /var/lib/mysql?
>>    
>>
>
>It's easy to talk big. 
>
>It's damn hard to _implement_ a complex system, and make it stable and 
>bug-free, and support legacy applications.
>
>There is a reason why we do only what _must_ be done in kernel space.
>
>		Linus
>
>
>  
>
Filesystems need transactions. Given that they need it, it was very 
tempting to build an infrastructure offering some of the functionality 
to user space, and so we did in reiser4. What we have is very 
rudimentary, but I think I am prepared to say that important aspects of 
high performance transactions are better done in the kernel than in user 
space. It will be interesting to see what aspects of transaction 
functionality belong in user space as our transactions API comes into 
existence. Clearly there are some things about isolation that are better 
implemented by user space because user space can make optimizations the 
kernel doesn't know enough to make. Still, it will be interesting to see 
how low we can make the overhead of isolation. Right now reiser4 
supports atomicity but not isolation, which is to say we offer to 
guarantee that a set of operations will either all survive a crash or 
none of them will, but we don't let you roll them back without a crash. 
(Originally I called this a transcrash rather than atomicity, but nobody 
liked the term.) Being modest in our ambition was important to 
performance, but maybe in time we will find a high performance way to be 
more ambitious.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Alan Cox]

On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> and quite frankly, I think you can do the above pretty much totally in
> user space with a small library and a daemon (in fact, ignoring security
> issues you probably don't even need the daemon). And if you can prototype
> it like that, and people actually find it useful, I suspect kernel support
> for better performance might be possible.

Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
it since the late 1990's.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jamie Lokier]

Alan Cox wrote:
> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> > and quite frankly, I think you can do the above pretty much totally in
> > user space with a small library and a daemon (in fact, ignoring security
> > issues you probably don't even need the daemon). And if you can prototype
> > it like that, and people actually find it useful, I suspect kernel support
> > for better performance might be possible.
> 
> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> it since the late 1990's.

Firstly, if I have to do it from a Gnome program, about the only
program where looking in a tar file is visibly useful is Nautilus.
Ironically, clicking on a tar file in Nautilus doesn't work, despite
having a dependency on gnome-vfs2. :/

Secondly, no, Gnome and MC don't support entering a container file,
letting you make changes in it, and remembering those changes to
_lazily_ regenerate the container file when you need it linearized,
possibly months later or never, by some unrelated program.

Thirdly, you must be referring to the Gnome versions of Bash, Make,
GCC, coreutils and Perl which I haven't found.  Perhaps we have a
different idea of what "supports this" means :)

uservfs, which is based on gnome-vfs and getting a bit rusty due to
disuse, does try to solve the last problem.  Unfortunately it needs
further work to have a nicer interface, and the second problem is
still not solved by gnome-vfs.

-- Jamie

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Dave Kleikamp]

On Thu, 2004-09-02 at 11:11, Jamie Lokier wrote:

> Firstly, if I have to do it from a Gnome program, about the only
> program where looking in a tar file is visibly useful is Nautilus.
> Ironically, clicking on a tar file in Nautilus doesn't work, despite
> having a dependency on gnome-vfs2. :/

This should be fixed in Nautilus, not the kernel.

> Secondly, no, Gnome and MC don't support entering a container file,
> letting you make changes in it, and remembering those changes to
> _lazily_ regenerate the container file when you need it linearized,
> possibly months later or never, by some unrelated program.

Why do this in a tar file?  tar = "tape archive".  It isn't designed to
be a file system.  Sure, it's nice to have tools that make it easier to
access files in a tar file, but to this isn't a job for the kernel.

> Thirdly, you must be referring to the Gnome versions of Bash, Make,
> GCC, coreutils and Perl which I haven't found.  Perhaps we have a
> different idea of what "supports this" means :)

Please don't tell me that we have expectations to run make from within a
tar file.  This is getting silly.  tar does a pretty good job of
extracting files into real directories, and putting them back into an
archive.  I don't see a need to teach the kernel how to deal with
compound files when user space can do it very easily.

Shaggy

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Thu, 2 Sep 2004, Alan Cox wrote:
>
> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> > and quite frankly, I think you can do the above pretty much totally in
> > user space with a small library and a daemon (in fact, ignoring security
> > issues you probably don't even need the daemon). And if you can prototype
> > it like that, and people actually find it useful, I suspect kernel support
> > for better performance might be possible.
> 
> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> it since the late 1990's.

And nobody has asked for kernel support that I know of.

So either "it just works" in user space, or people haven't figured out the 
kernel could help them. Or decided it's not worth it, exactly because 
they'd still have to support systems/filesystems that can't be converted.

		Linus

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Thu, Sep 02, 2004 at 10:46:32AM -0700, Linus Torvalds wrote:
> > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> > > and quite frankly, I think you can do the above pretty much totally in
> > > user space with a small library and a daemon (in fact, ignoring security
> > > issues you probably don't even need the daemon). And if you can prototype
> > > it like that, and people actually find it useful, I suspect kernel support
> > > for better performance might be possible.
> > 
> > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> > it since the late 1990's.
> 
> And nobody has asked for kernel support that I know of.
> 
> So either "it just works" in user space, or people haven't figured out the 
> kernel could help them. Or decided it's not worth it, exactly because 
> they'd still have to support systems/filesystems that can't be converted.

http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto
a kernel filesystem.  The code is horrible, but it shows that it can
be done.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Thu, 2 Sep 2004, Christoph Hellwig wrote:
> 
> http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto
> a kernel filesystem.  The code is horrible, but it shows that it can
> be done.

I do like the setup where the extended features are done as a "view" on 
top of some other filesystem, so that you can choose to _either_ access 
the raw (and supposedly stable, simply by virtue of simplicity) or the 
"fancy" interface. Without having to reformat the disk to a filesystem you 
don't trust, or you have other reasons you can't use (disk sharing with 
other systems, whatever).

It doesn't have to be "user", btw, in the sense that a lot of the normal 
code could be in kernel mode. Same way as Tux handling all the regular 
static requests entirely in kernel mode, but having the ability for 
calling down to apache..

		Linus

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Charles P. Wright]

On Thu, 2004-09-02 at 14:03, Linus Torvalds wrote:
> On Thu, 2 Sep 2004, Christoph Hellwig wrote:
> > 
> > http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto
> > a kernel filesystem.  The code is horrible, but it shows that it can
> > be done.
> 
> I do like the setup where the extended features are done as a "view" on 
> top of some other filesystem, so that you can choose to _either_ access 
> the raw (and supposedly stable, simply by virtue of simplicity) or the 
> "fancy" interface. Without having to reformat the disk to a filesystem you 
> don't trust, or you have other reasons you can't use (disk sharing with 
> other systems, whatever).
> 
> It doesn't have to be "user", btw, in the sense that a lot of the normal 
> code could be in kernel mode. Same way as Tux handling all the regular 
> static requests entirely in kernel mode, but having the ability for 
> calling down to apache..
Stackable file systems are a good fit for doing things like this.  They
layer on top of an existing directory and intercept all of the vfs_*
calls.  You can do lots of neat things like encryption versioning,
compression, and more.

At Stony Brook's Filesystems and Storage Laboratory
(http://www.fsl.cs.sunysb.edu), we maintain a toolkit (FiST) that makes
developing these file systems easier by providing stackable file system
templates for Linux and FreeBSD.

There are papers and the FiST toolkit available at
http://www.filesystems.org/

Charles

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Thu, 2 Sep 2004, Charles P. Wright wrote:
>
> Stackable file systems are a good fit for doing things like this.  They
> layer on top of an existing directory and intercept all of the vfs_*
> calls.  You can do lots of neat things like encryption versioning,
> compression, and more.

Stackable filesystems are really hard, and ambiguous on where things go, 
and what happens when you remove an entry that hides another one. Sure, 
you can make up any amount of semantics, but the fact is, all the 
semantics _are_ going to be pretty arbitrary, and fairly complex.

I'd frankly _much_ prefer not having stackable filesystems supported by 
the VFS layer.

That said, I think you can make stackable filesystems as a _separate_
filesystem (aka "view" or whatever), which is slightly different from
supporting them in the VFS layer itself: make it a filesystem of its own,
rather than supporting it on a VFS layer as a merge of two different
filesystems. That way you still have totally arbitrary semantics for how
the stacking works, but at least it's not codified by the VFS layer - it's
codified by nothing more special than one more random filesystem
implementation, and you could in fact have _different_ semantics depending
on what the user wants.

Same end result, different "mindset". I think this is what FiST actually 
does, but I could be wrong.

What's the difference? If the VFS layer supports it (like the VFS supports 
regular "opaque" overmounting), you'd just do

	mount -o stack /dir /otherdir

the same way you do "bind" mounts. In contrast, if you do it as a 
third-party "view" filesystem, it would be more like

	mount -t stackfs -o under=/usr,over=/usr/local /mnt

where /usr and /usr/local would be the same old underlying filesystems, 
and you'd see a totally new filesystem at /mnt.

Btw, this kind of non-VFS-supported filesystem would not necessarily have 
coherency guarantees: if you changed /usr or /usr/local directly (rather 
than through the stackable /mnt), you could confuse stackfs. Kind of the 
same way you can confuse a NFSv2 client by operating directly on the 
server (ie I'm not saying "confuse" in a "blow up the kernel" kind of 
way, but in a "would return -ESTALE or similar").

		Linus

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Charles P. Wright]

On Thu, 2004-09-02 at 15:42, Linus Torvalds wrote:
> On Thu, 2 Sep 2004, Charles P. Wright wrote:
> >
> > Stackable file systems are a good fit for doing things like this.  They
> > layer on top of an existing directory and intercept all of the vfs_*
> > calls.  You can do lots of neat things like encryption versioning,
> > compression, and more.
> 
> Stackable filesystems are really hard, and ambiguous on where things go, 
> and what happens when you remove an entry that hides another one. Sure, 
> you can make up any amount of semantics, but the fact is, all the 
> semantics _are_ going to be pretty arbitrary, and fairly complex.
I think by stackable you are thinking what we call a Union.

By stackable, I mean that you just layer some functionality on an
existing file system, without having to modify it.

I agree that Unioning semantics are difficult.  We've actually
implemented a stackable unioning file system, and dealing with an
arbitrary number of branches, especially in the face of some operations
failing in the middle, is quite complex.  Rename is especially hard.

I think for the type of problem, where you have the file system jump
down into archives or do other like-minded operations, you will also end
up with having to make some arbitrary decisions with complex semantics.

> I'd frankly _much_ prefer not having stackable filesystems supported by 
> the VFS layer.

> That said, I think you can make stackable filesystems as a _separate_
> filesystem (aka "view" or whatever), which is slightly different from
> supporting them in the VFS layer itself: make it a filesystem of its own,
> rather than supporting it on a VFS layer as a merge of two different
> filesystems. That way you still have totally arbitrary semantics for how
> the stacking works, but at least it's not codified by the VFS layer - it's
> codified by nothing more special than one more random filesystem
> implementation, and you could in fact have _different_ semantics depending
> on what the user wants.
> 
> Same end result, different "mindset". I think this is what FiST actually 
> does, but I could be wrong.
Yes, to the VFS FiST behaves as a file system.  To the file system it is
stacked on, it behaves like the VFS.

> What's the difference? If the VFS layer supports it (like the VFS supports 
> regular "opaque" overmounting), you'd just do
> 
> 	mount -o stack /dir /otherdir
> 
> the same way you do "bind" mounts. In contrast, if you do it as a 
> third-party "view" filesystem, it would be more like
> 
> 	mount -t stackfs -o under=/usr,over=/usr/local /mnt
> 
> where /usr and /usr/local would be the same old underlying filesystems, 
> and you'd see a totally new filesystem at /mnt.
That is very close to how a FiST file system works.  However, I think
you are more closely describing a Union mount that would provide a
merged view of "/user" and "/usr/local" (with /usr/local taking
precedence over /user).

More concretely, something like:
mount -t gzipfs -o dir=/compressed none /mnt

Would transparently compress data written to /mnt/a, and physically
store it in /compressed/a.  When reading /mnt/a, the data is
decompressed and returned to the user-space.

This a real example that is actually included as part of FiST.

> Btw, this kind of non-VFS-supported filesystem would not necessarily have 
> coherency guarantees: if you changed /usr or /usr/local directly (rather 
> than through the stackable /mnt), you could confuse stackfs. Kind of the 
> same way you can confuse a NFSv2 client by operating directly on the 
> server (ie I'm not saying "confuse" in a "blow up the kernel" kind of 
> way, but in a "would return -ESTALE or similar").
With a simple stacking file system, one way to avoid this is an overlay
mount.

mount -t cryptfs -o dir=/home/cpwright none /home/cpwright

There are some disadvantages to hiding the lower level (e.g., with
Cryptfs you want to backup the encrypted data, not the decrypted data).

Charles

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Erez Zadok]

In message <Pine.LNX.4.58.0409021230450.2295@ppc970.osdl.org>, Linus Torvalds writes:
[...]
> That said, I think you can make stackable filesystems as a _separate_
> filesystem (aka "view" or whatever), which is slightly different from
> supporting them in the VFS layer itself: make it a filesystem of its own,
> rather than supporting it on a VFS layer as a merge of two different
> filesystems. That way you still have totally arbitrary semantics for how
> the stacking works, but at least it's not codified by the VFS layer - it's
> codified by nothing more special than one more random filesystem
> implementation, and you could in fact have _different_ semantics depending
> on what the user wants.
> 
> Same end result, different "mindset". I think this is what FiST actually 
> does, but I could be wrong.
> 
> What's the difference? If the VFS layer supports it (like the VFS supports 
> regular "opaque" overmounting), you'd just do
> 
> 	mount -o stack /dir /otherdir
> 
> the same way you do "bind" mounts. In contrast, if you do it as a 
> third-party "view" filesystem, it would be more like
> 
> 	mount -t stackfs -o under=/usr,over=/usr/local /mnt
> 
> where /usr and /usr/local would be the same old underlying filesystems, 
> and you'd see a totally new filesystem at /mnt.
[...]

Yes, that's exactly how our stacked f/s's work: not touching or modifying
VFS or lower f/s's, but of course we use the VFS the way other f/s's use it
(calling vfs_* functions and such).  Each of our stackable f/s is a separate
f/s in its own.

And they're a nice way to augment the functionality of any other f/s w/o the
hassles of changing a lower-level f/s format or the VFS.  We recently did an
example f/s that adds EA+ACL functionality on top of anything else, using
the lower file system's EA/ACL support if it's there, and for f/s's that
don't support EA/ACL natively, we support the ea/acl f/s calls and store the
actual ea/acl data in a small database.  We've even demonstrated that it
worked on v/fat (yes, we know, vfat+ACLs is an an oxymoron :-)

We also found out over the years that our stackable file systems are an
interesting way to prototype features that might eventually move up to the
VFS itself; this is b/c each stacked f/s has to appear to the lower f/s as
the VFS itself, essentially emulating most of the VFS's functionality while
at the same time appearing to the VFS as a lower-level f/s.  It's not easy
maintaining this split personality. :-)

Erez.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Alan Cox]

On Iau, 2004-09-02 at 18:46, Linus Torvalds wrote:
> > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> > it since the late 1990's.
> 
> And nobody has asked for kernel support that I know of.

I asked our desktop people. They want something like inotify because
dontify doesn't cut it. They have zero interest in the multiple streams
and hiding icons in streams type stuff.

Alan

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Thu, 2 Sep 2004, Alan Cox wrote:
> 
> I asked our desktop people. They want something like inotify because
> dontify doesn't cut it.

Well, dnotify() really _is_ inotify(), since it does actually work on 
inodes, not dentries.

I think what they are really complaining about is that dnotify() only 
notifies the _directory_ when a file is changed, and they'd like it to 
notify the file itself too. Which is a one-liner, really.

Does the following make sense? (Totally untested, use-at-your-own-risk, 
I've-never-actually-used-dnotify-in-user-space, whatever).

		Linus

===== fs/dnotify.c 1.17 vs edited =====
--- 1.17/fs/dnotify.c	2004-08-09 18:45:22 -07:00
+++ edited/fs/dnotify.c	2004-09-02 13:21:26 -07:00
@@ -160,6 +160,8 @@
 	if (!dir_notify_enable)
 		return;
 
+	__inode_dir_notify(dentry->d_inode, event);
+
 	spin_lock(&dentry->d_lock);
 	parent = dentry->d_parent;
 	if (parent->d_inode->i_dnotify_mask & event) {

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  



> On Thu, 2 Sep 2004, Alan Cox wrote:
>>
>> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
>> > and quite frankly, I think you can do the above pretty much totally in
>> > user space with a small library and a daemon (in fact, ignoring security
>> > issues you probably don't even need the daemon). And if you can prototype
>> > it like that, and people actually find it useful, I suspect kernel support
>> > for better performance might be possible.
>> 
>> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
>> it since the late 1990's.

> And nobody has asked for kernel support that I know of.

  Actually. Doesn't matter if it is in kernel or not for the users as
  long as it works.

  The problem is that I do not see either Gnome or KDE to ever get
  along to form one standard that everyone will use. Their libraries
  are huge and memory hogging which so many Linux users just do not
  like. What if a user doesn't want KDE or Gnome? Would all files
  created with either be broken?

  I doubt that something like file streams and meta-data can
  successfully be implemented purely in user-space and get the same
  support (ie be used by many programs) if this change doesn't come
  from the kernel. I just do not see it happen.

> So either "it just works" in user space, or people haven't figured out the
> kernel could help them. Or decided it's not worth it, exactly because
> they'd still have to support systems/filesystems that can't be converted.

> 		Linus

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  



> On Thu, 2 Sep 2004, Alan Cox wrote:
>>
>> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
>> > and quite frankly, I think you can do the above pretty much totally in
>> > user space with a small library and a daemon (in fact, ignoring security
>> > issues you probably don't even need the daemon). And if you can prototype
>> > it like that, and people actually find it useful, I suspect kernel support
>> > for better performance might be possible.
>> 
>> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
>> it since the late 1990's.

> And nobody has asked for kernel support that I know of.

  Actually. Doesn't matter if it is in kernel or not for the users as
  long as it works.

  The problem is that I do not see either Gnome or KDE to ever get
  along to form one standard that everyone will use. Their libraries
  are huge and memory hogging which so many Linux users just do not
  like. What if a user doesn't want KDE or Gnome? Would all files
  created with either be broken?

  I doubt that something like file streams and meta-data can
  successfully be implemented purely in user-space and get the same
  support (ie be used by many programs) if this change doesn't come
  from the kernel. I just do not see it happen.

> So either "it just works" in user space, or people haven't figured out the
> kernel could help them. Or decided it's not worth it, exactly because
> they'd still have to support systems/filesystems that can't be converted.

> 		Linus

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 10:46:05AM +0100, Alan Cox wrote:
> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> > and quite frankly, I think you can do the above pretty much totally in
> > user space with a small library and a daemon (in fact, ignoring security
> > issues you probably don't even need the daemon). And if you can prototype
> > it like that, and people actually find it useful, I suspect kernel support
> > for better performance might be possible.
> 
> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> it since the late 1990's.

Can it do this:

	cd FC2-i386-disc1.iso
	ls

or this:

	cd /dev/sda1
	ls
	cd /dev/floppy
	ls
	cd /dev/cdrom
	ls

?

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Dave Kleikamp]

On Thu, 2004-09-02 at 15:38, Frank van Maarseveen wrote:
> Can it do this:
> 
> 	cd FC2-i386-disc1.iso
> 	ls
> 
> or this:
> 
> 	cd /dev/sda1
> 	ls
> 	cd /dev/floppy
> 	ls
> 	cd /dev/cdrom
> 	ls
> 
> ?

We have the mount command for that.  :^)
-- 
David Kleikamp
IBM Linux Technology Center

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Horst von Brand]

Jamie Lokier <jamie@shareable.org> said:
> I'm going to explain why filesystem support for .tar.gz or other
> "document container"-like formats is useful.

Nobody disagrees there (I think), the disagreement is on whether the
usefullness is worth the hassle.

>                                               This does _not_ mean tar
> in the kernel (I know someone who can't read will think that if I
> don't say it isn't); it does mean hooks in the kernel for maintaining
> coherency between different views and filesystem support for cacheing.

"Coherency" and "different views" implies atomic transactions, and being
able to tell that an access to the file requieres updating random junk
about it. It requires being able to guess if it is worth updating now
(given that the file might be modified a dozen times more before the junk
being checked).

> The vision I'm going for here is:
> 
>   1. OpenOffice and similar programs store compound documents in
>      standard file formats, such as .tar.gz, compressed XML and such.

And they are doing fine AFAICS. Besides, they won't exactly jump on the
possibility of leaving behind all other OSes on which they run to become a
Linux-only format.

>      Fs support can reduce CPU time handling these sorts of files, as
>      I explain below, while still working with the standard file formats.

I don't buy this one. A tar.gz must be uncompressed and unpacked, and
whatever you could save is surely dwarfed by those costs.

>      With appropriate userspace support, programs can be written which
>      have access to the capabilities on all platforms, but reduced CPU
>      time occurs only on platforms with the fs support.

Userspace support isn't there on any of the platforms right now, if ever it
will be a strange-Linux-installation thing for quite some time to come. Not
exactly attractive for application writers.

>   2. Real-time indexing and local search engine tools.

Sure! Gimme the CPU power and disk throughput for that, pretty please. [No,
I won't tell I have better use for those right now...]

>                                                         This isn't
>      just things like local Google; it's also your MP3 player scanning
>      for titles & artists, your email program scanning for subject
>      lines to display the summary fast, your blog server caching built
>      pages, your programming environment scanning for tags, and your
>      file transfer program scanning for shared deltas to reduce bandwidth.

With no description on how this is supposed to work, this is pure science
fiction/wet dreams.

>      I won't explain how these work as it would make this mail too
>      long.  It should be clear to anyone who thinks about it why the
>      coherency mechanism is essential for real-time, and a consistent
>      interface to container internals helps with performance.

Coherency is essential, but it isn't free. Far from it. The easiest way of
getting coherency is having _one_ authoritative source. That way you don't
need coherency, and don't pay for it. Anything in this class must by force
be just hints, to be recomputed at a moment's notice. I.e., let the
application who might use it check and recompute as needed.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jamie Lokier]

Horst von Brand wrote:
> >                                               This does _not_ mean tar
> > in the kernel (I know someone who can't read will think that if I
> > don't say it isn't); it does mean hooks in the kernel for maintaining
> > coherency between different views and filesystem support for cacheing.
> 
> "Coherency" and "different views" implies atomic transactions,

Wrong.  They imply ordering.

> and being able to tell that an access to the file requieres updating
> random junk about it.

Correct.

> It requires being able to guess if it is worth updating now
> (given that the file might be modified a dozen times more before the junk
> being checked).

Wrong.  Lazy and deterministic are compatible.

> 
> > The vision I'm going for here is:
> > 
> >   1. OpenOffice and similar programs store compound documents in
> >      standard file formats, such as .tar.gz, compressed XML and such.
> 
> And they are doing fine AFAICS. Besides, they won't exactly jump on the
> possibility of leaving behind all other OSes on which they run to become a
> Linux-only format.

You replied before reading the rest of the mail again, didn't you?
The whole purpose of the idea is to work with formats which are _not_
Linux-only.

> >      Fs support can reduce CPU time handling these sorts of files, as
> >      I explain below, while still working with the standard file formats.
> 
> I don't buy this one. A tar.gz must be uncompressed and unpacked, and
> whatever you could save is surely dwarfed by those costs.

Didn't they explain what "lazy" means in your computer science class?
Google for "lazy evaluation".

In this context it means _unnecessary_ uncomressions and compressions,
i.e. repeated ones, or ones where you never really needed the
compressed form, are eliminated.  Necessary ones are of course done.

Those are things which are currently wasting your CPU because apps
cannot eliminate them: to do so implies synchronisation between
different apps running at different times.  Surprise surprise, that's
something filesystems and kernels are good for.

> >      With appropriate userspace support, programs can be written which
> >      have access to the capabilities on all platforms, but reduced CPU
> >      time occurs only on platforms with the fs support.
> 
> Userspace support isn't there on any of the platforms right now, if ever it
> will be a strange-Linux-installation thing for quite some time to come. Not
> exactly attractive for application writers.

Again, you have a strange vision of what userspace support means.  I
say it means a portable and simple library for accessing components
inside a compressed container file in a standard format.

Something I believe OpenOffice et al. already has - so to say it
doesn't exist is both incorrect and missing the point.  The point is
to change that library which already exists, so it uses the filesystem
facility when available.  And to make the library better in standalone
form, so that other tools which manipulate container-like formats are
inclined to work with it as plugins, instead of creating their own
interface as cli tools.

_At no point is portability an issue_ - that library is already
portable to everything - and it's certainly not a strange-Linux thing
nor meant to become one.


> >   2. Real-time indexing and local search engine tools.
> 
> Sure! Gimme the CPU power and disk throughput for that, pretty please. [No,
> I won't tell I have better use for those right now...]

You still haven't grasped the idea of an algorithm which is more
complex but reduces CPU time and disk throughput, have you?

*Today*, when I open Rhythmbox I have to wait 5 minutes while it wastes
my CPU power and disk throughput scanning all the files in my Music
directory.

The entire point of these indexing schemes is so that programs like
Rhythmbox will display the same data without having to scan the disk
for 5 minutes every time they start up.  That's a near infinite
_saving_ of CPU power and time.

If you don't see that I'm just going to have to suggest you google for
"cache" and learn a bit about them.

> >      This isn't
> >      just things like local Google; it's also your MP3 player scanning
> >      for titles & artists, your email program scanning for subject
> >      lines to display the summary fast, your blog server caching built
> >      pages, your programming environment scanning for tags, and your
> >      file transfer program scanning for shared deltas to reduce bandwidth.
> 
> With no description on how this is supposed to work, this is pure science
> fiction/wet dreams.

Sigh.  If you must continue to hurl your blunt instruments around,
here is a description.  I didn't want to write this becuase it is off
topic, and you insist on not understanding the basic of algorithm
complexity and CPU usage, so you'll probably not understand this
either, but I'll give it a go.

    1. Local Google (by which I mean a search engine on your local machine),
       Real-time (by which I mean the results are always up to date):

       Every file modified since last search is known to the query engine.
       This is a reality: BeOS does it; WinFS is expected to do it.

       We know that it's possible to update free text indexes with
       small amounts of known changes quickly, at the time of a query.

       Thus we have real-time local free text search engine, and other
       features like searching inside files and for file names.  The
       point is the real-time nature of it: the results you get
       correspond to exactly the contents of the filesystem at the time of
       the query (writes which occur _during_ a query are obviously
       not coherent with this, but writes which complete before the
       query, even immediately before, appear in the results).

       Note that file write timing need not be affected - all the work
       can happen during queries (although it is best to have a
       nightly re-index or whatever so that the delay at query times
       is kept moderate).

    2. MP3 player scanning artists & titles:

       Easy.  MP3 player does what it does, shows the extracted ID
       tags from all .mp3 files in your home directory.  They do this
       already!  The difference is that with fs coherency hooks, they
       can _store_ that ID information, retrieve it later without
       having to scan all the .mp3s again (see Rhythmbox earlier), and
       keep their lists updated on the screen as soon as any .mp3 is
       changed or even any new ones created anywhere.

       Technically it's a simpler subset of real-time queries in 1.

    3. Email program scanning for subject lines fast:

       See Evolution; the only difference is stat() on a thousand
       files won't be required, it'll be exposed through a standard
       query instead of an Evolution-only cache method so other mail
       programs may use it as well as shell commands, and you can
       pretend you have mbox instead of maildir (mbox is a container
       format...).

    4. Blog server caching built pages:

       It's a reality already, obviously.  The difference is it'll
       make sense to built the pages through an indexed query on files
       instead of a database, e.g. one file per article, and
       (independent of the first point) the process of building can
       keep track of all the prerequisite files and scripts and
       templates used to produce the page, and actually expect to
       know, _with coherent accuracy_, if any of those prerequisites
       is different the next time the page is needed.

       In other words, complex script-generated web pages, cached
       output, and with no overhead when serving each page to check
       prerequisites (i.e. no 100 stat() calls), because we know the
       filesystem would have told us if any prerequisite had changed
       prior to the moment we begin serving the cached output.

       You can do something close to this already with dnotify (if you
       ignore that it doesn't tell you about new hard links, which is
       a dnotify bug), although doing the query part is unrealistic
       with dnotify and stat() prerequisites unless the directory
       names are structured thoughtfully.

    5. Programming environment scanning for tags:

       By now this should be obvious.  No need to run "exuberant
       ctags" or whatever class-hierarchy-extraction and
       documentation-extraction program every so often after making
       changes, and yet your IDE's clickable list of tags and notes
       stays up to date with file modifications in real time.  I think
       some IDEs do this already with moderate size trees, using
       stat(), but it's not realistic when you have tens of thousands
       of source files.

    6. File transfer program scanning for shared deltas.

       This is nothing more than searching all files for common
       subsequences which match strong hashes which are expected to be
       common during file transfer operations.  E.g. the GPL header at
       the start of many source files would have one such hash.  A
       hash of every whole file is another one, along with the name of
       the file (it is a key which indicates a "likely to match the
       corresponding hash" condition), as is a hash of every aligned
       64k subsequence, or whatever is appropriate to reduce disk I/O.

       Having an index of these can speed up some file transfer
       protocols, in the manner of rsync but comparing among a large
       group of files instead of just two.  The point is that kind of
       protocol can be more efficient than rsync (sometimes
       dramatically so), but it's only a net gain to use that kind of
       algorithm if you have a handy _and reliable_ index of likely
       common subsequences and whole-file hashes, otherwise it uses
       far too much disk I/O to check the group.  The index needs
       filesystem coherency support, otherwise it is not reliable
       enough for this bandwidth optimisation.

> >      I won't explain how these work as it would make this mail too
> >      long.  It should be clear to anyone who thinks about it why the
> >      coherency mechanism is essential for real-time, and a consistent
> >      interface to container internals helps with performance.
> 
> Coherency is essential, but it isn't free. Far from it. The easiest way of
> getting coherency is having _one_ authoritative source. That way you don't
> need coherency, and don't pay for it. Anything in this class must by force
> be just hints, to be recomputed at a moment's notice. I.e., let the
> application who might use it check and recompute as needed.

(a) Without some kind of minimal hook, there is no way for the application
to check without re-reading all the files it has used for a
computation, starting at the moment where it's about to potentially
recompute the result.

Sometimes you can get away with stat() on the files (e.g. Evolution
and fontconfig do that), but often you can't because that's not
trustworthy enough (security checks, protocol optimisations,
transparent "as if I didn't cache anything" semantics), and sometimes
stat() calls are far too slow anyway (anything that involves a lot of files).

(The trustworthiness problem is likely to be solved, for some
applications, by an xattr or such which is guaranteed to be deleted or
changed when a file is modified.  That is really a very minimal hook,
but it is one).

(b) Even with all the misgivings of stat(), you can't realistically
update real time displays, e.g. the lists of artists & albums on the
screen in your MP3 player whenever you modify anything in your music
collection using another program.  At best, you have to expect the
display to not notice the change for a while (minutes in the mp3
case), or until you click an "Explicit Refresh" button.  Neither
should be necessary.


It's true that the simplest coherency is one authoratitive source
which you have to check all the time (although even that doesn't work
for some things).  But checking all the time does rule out, due to its
high algorithmic complexity, a lot of very interesting applications
and even some interesting simple unixish tools which you might like,
such as real-time "locate".

-- Jamie

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Christer Weinigel]

Jamie Lokier <jamie@shareable.org> writes:

>     1. Local Google (by which I mean a search engine on your local machine),
>        Real-time (by which I mean the results are always up to date):
> 
>        Every file modified since last search is known to the query engine.
>        This is a reality: BeOS does it; WinFS is expected to do it.
> 
>        Thus we have real-time local free text search engine, and other
>        features like searching inside files and for file names.  The
>        point is the real-time nature of it: the results you get
>        correspond to exactly the contents of the filesystem at the time of
>        the query (writes which occur _during_ a query are obviously
>        not coherent with this, but writes which complete before the
>        query, even immediately before, appear in the results).

Can be done with dnotify/inotify and a cache daemon keeping track of
mtime.  Yes, this will need a kernel change to make sure mtime always
changed when the file changes, but it does not require anything else.

>     2. MP3 player scanning artists & titles:

Same.

>     3. Email program scanning for subject lines fast:

Same here.

>     4. Blog server caching built pages:
>     5. Programming environment scanning for tags:
>     6. File transfer program scanning for shared deltas.

And so on.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Joel Becker]

On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote:
> Yes, this was part of the plan, tar file-directory plugins would be cute.

	Question:  Is "cat /foo/bar/baz.tar.gz/metas" the attribute
directory or a directory in the tarball named "metas"?

Joel

-- 

"I'm so tired of being tired,
 Sure as night will follow day.
 Most things I worry about
 Never happen anyway."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: silent semantic changes with reiser4

[Jamie Lokier]

Joel Becker wrote:
> On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote:
> > Yes, this was part of the plan, tar file-directory plugins would be cute.
> 
> 	Question:  Is "cat /foo/bar/baz.tar.gz/metas" the attribute
> directory or a directory in the tarball named "metas"?

This needs to be designed.

Perhaps /foo/bar/baz.tar.gz/tar/metas is the directory in the tarball
named "metas".

Or perhaps /foo/bar/baz.tar.gz/x/metas is: it's independent of archive
format, and I personally tend to extract things into a directory
called "x". [*]

Or perhaps /foo/bar/baz.tar.gz/metas is, and the attribute directory
is /foo/bar/baz.tar.gz/../metas, to be perverse ;)

I prefer the second one, ("x/metas"), but not with any conviction.

-- Jamie


[*] Actually I prefer:

      /foo/bar/baz.tar.gz/content/metas
      /foo/bar/baz-0.01.tar.gz/content/baz-0.01/metas

           Archives always in "content".  One layer of decompression
           always tried for .tar files and other uncompressed archive
           formats.

      /foo/bar/baz.tar.gz/x -> content/
      /foo/bar/baz-0.01.tar.gz/x -> content/baz-0.01/

           If the root of the archive contains a single directory, "x"
           is a symlink to it.  Otherwise "x" is a symlink to the root
           directory of the archive.  This is comfortable with the
           common practice by which archives are distributed, without
           making a mess when someone forgets to put everything in a
           top-level directory.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 11:35:32AM -0700, Joel Becker wrote:
>	Question:  Is "cat /foo/bar/baz.tar.gz/metas" the attribute
>directory or a directory in the tarball named "metas"?

This has been fought over on the reiserfs-list ad nauseaum, but
it's a valid point.

That's why I tend to rename metas/ into ..metas/ to avoid
name clashes, even if I've never had a directory named metas/
apart from what Reiser4 ships.

It is then debatable if it should be renamed before it's too
late, have it renamable in the kernel configs (and the name
exported via /sys or something) or just leave it be.

-- 
mjt

Re: silent semantic changes with reiser4

[mjt]

On Thu, Aug 26, 2004 at 11:35:32AM -0700, Joel Becker wrote:
>	Question:  Is "cat /foo/bar/baz.tar.gz/metas" the attribute
>directory or a directory in the tarball named "metas"?

This has been fought over on the reiserfs-list ad nauseaum, but
it's a valid point.

That's why I tend to rename metas/ into ..metas/ to avoid
name clashes, even if I've never had a directory named metas/
apart from what Reiser4 ships.

It is then debatable if it should be renamed before it's too
late, have it renamable in the kernel configs (and the name
exported via /sys or something) or just leave it be.

-- 
mjt

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 11:35:32AM -0700, Joel Becker wrote:
>	Question:  Is "cat /foo/bar/baz.tar.gz/metas" the attribute
>directory or a directory in the tarball named "metas"?

This has been fought over on the reiserfs-list ad nauseaum, but
it's a valid point.

That's why I tend to rename metas/ into ..metas/ to avoid
name clashes, even if I've never had a directory named metas/
apart from what Reiser4 ships.

It is then debatable if it should be renamed before it's too
late, have it renamable in the kernel configs (and the name
exported via /sys or something) or just leave it be.

-- 
mjt

Re: silent semantic changes with reiser4

[Matthew Wilcox]

On Fri, Aug 27, 2004 at 12:19:54PM +0300, Markus   Törnqvist wrote:
> That's why I tend to rename metas/ into ..metas/ to avoid
> name clashes, even if I've never had a directory named metas/
> apart from what Reiser4 ships.

http://packages.debian.org/cgi-bin/search_contents.pl?word=metas&searchmode=searchfilesanddirs&case=insensitive&version=unstable&arch=i386

OK, those are capital METAS rather than junior metas, but it does show
this is not a unique word to reiser4.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: silent semantic changes with reiser4

[Matthew Wilcox]

On Fri, Aug 27, 2004 at 12:19:54PM +0300, Markus   Törnqvist wrote:
> That's why I tend to rename metas/ into ..metas/ to avoid
> name clashes, even if I've never had a directory named metas/
> apart from what Reiser4 ships.

http://packages.debian.org/cgi-bin/search_contents.pl?word=metas&searchmode=searchfilesanddirs&case=insensitive&version=unstable&arch=i386

OK, those are capital METAS rather than junior metas, but it does show
this is not a unique word to reiser4.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[Chris Wedgwood]

On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote:

> One of the big potential uses for file-as-directory is to go inside
> archive files, ELF files, .iso files and so on in a convenient way.

Arguably this belongs in userspace --- and people have put it there.


  --cw

Re: silent semantic changes with reiser4

[Jamie Lokier]

Chris Wedgwood wrote:
> On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote:
> 
> > One of the big potential uses for file-as-directory is to go inside
> > archive files, ELF files, .iso files and so on in a convenient way.
> 
> Arguably this belongs in userspace --- and people have put it there.

I agree that these belong in userspace, and that there's plenty* of
userspace code doing a similar thing already.  I don't think there's
any argument over it.

However, as far as I know it's not accessible in a file-as-directory
form as yet.  In my opinion that is the most natural form and it would
be very intuitive to use.  I hope we can pick a useful semantics for
them, and also provide filesystem-independent plugins with GNU
Hurd-like per-user extensibility.

-- Jamie

* plenty == too much.
  Gnome, KDE, Emacs and Bash all see different virtual filesystems.
  (All but Bash implement their own virtual filesystem extensions).
  That makes them much less useful than they could be.

Re: silent semantic changes with reiser4

[Spam]

> Chris Wedgwood wrote:
>> On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote:
>> 
>> > One of the big potential uses for file-as-directory is to go inside
>> > archive files, ELF files, .iso files and so on in a convenient way.
>> 
>> Arguably this belongs in userspace --- and people have put it there.

> I agree that these belong in userspace, and that there's plenty* of
> userspace code doing a similar thing already.  I don't think there's
> any argument over it.

> However, as far as I know it's not accessible in a file-as-directory
> form as yet.  In my opinion that is the most natural form and it would
> be very intuitive to use.  I hope we can pick a useful semantics for
> them, and also provide filesystem-independent plugins with GNU
> Hurd-like per-user extensibility.

> -- Jamie

> * plenty == too much.
>   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
>   (All but Bash implement their own virtual filesystem extensions).
>   That makes them much less useful than they could be.

  Exactly,  and  I  doubt  they  have the ability to join together and
  create  a  common  uniform standard out of it either. They have just
  far to different views on Linux and how they want things to be done.

Re: silent semantic changes with reiser4

[Spam]

> Chris Wedgwood wrote:
>> On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote:
>> 
>> > One of the big potential uses for file-as-directory is to go inside
>> > archive files, ELF files, .iso files and so on in a convenient way.
>> 
>> Arguably this belongs in userspace --- and people have put it there.

> I agree that these belong in userspace, and that there's plenty* of
> userspace code doing a similar thing already.  I don't think there's
> any argument over it.

> However, as far as I know it's not accessible in a file-as-directory
> form as yet.  In my opinion that is the most natural form and it would
> be very intuitive to use.  I hope we can pick a useful semantics for
> them, and also provide filesystem-independent plugins with GNU
> Hurd-like per-user extensibility.

> -- Jamie

> * plenty == too much.
>   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
>   (All but Bash implement their own virtual filesystem extensions).
>   That makes them much less useful than they could be.

  Exactly,  and  I  doubt  they  have the ability to join together and
  create  a  common  uniform standard out of it either. They have just
  far to different views on Linux and how they want things to be done.

Re: silent semantic changes with reiser4

[Horst von Brand]

Spam <spam@tnonline.net> said:
> > Chris Wedgwood wrote:

[...]

> >   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
> >   (All but Bash implement their own virtual filesystem extensions).
> >   That makes them much less useful than they could be.

>   Exactly,  and  I  doubt  they  have the ability to join together and
>   create  a  common  uniform standard out of it either. They have just
>   far to different views on Linux and how they want things to be done.

And the solution is... inventing yet another, incompatible way of doing
things that nobody outside its followers will ever use. Inside the kernel.
Should have thought of that before.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Will Dyson]

Jamie Lokier wrote:

> However, as far as I know it's not accessible in a file-as-directory
> form as yet.  In my opinion that is the most natural form and it would
> be very intuitive to use.  I hope we can pick a useful semantics for
> them, and also provide filesystem-independent plugins with GNU
> Hurd-like per-user extensibility.
> 
> -- Jamie
> 
> * plenty == too much.
>   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
>   (All but Bash implement their own virtual filesystem extensions).
>   That makes them much less useful than they could be.

It has always bugged me that Gnome and KDE implement their own VFS layers.

It seems to me that having a standard userspace filesystem layer like 
FUSE (http://sourceforge.net/projects/avf) would provide a lot of the 
benefit that HURD users (all 5 of them) see from filesystem 
"translators". Now, safely allowing unprivileged users to run arbitrary 
userspace filesystems would be a real trick. But, if it were possible, 
it could be combined with files-as-directories in some rather neat ways.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 11:50, Will Dyson wrote:
> Jamie Lokier wrote:
> 
> > However, as far as I know it's not accessible in a file-as-directory
> > form as yet.  In my opinion that is the most natural form and it would
> > be very intuitive to use.  I hope we can pick a useful semantics for
> > them, and also provide filesystem-independent plugins with GNU
> > Hurd-like per-user extensibility.
> > 
> > -- Jamie
> > 
> > * plenty == too much.
> >   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
> >   (All but Bash implement their own virtual filesystem extensions).
> >   That makes them much less useful than they could be.
> 
> It has always bugged me that Gnome and KDE implement their own VFS layers.
> 

Same here.  This always seemed like something the kernel should be able
to handle.  It seems to me that if reiser4 had been available at the
time the Gnome and KDE developers would not have needed to do this.

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 11:50, Will Dyson wrote:
> Jamie Lokier wrote:
> 
> > However, as far as I know it's not accessible in a file-as-directory
> > form as yet.  In my opinion that is the most natural form and it would
> > be very intuitive to use.  I hope we can pick a useful semantics for
> > them, and also provide filesystem-independent plugins with GNU
> > Hurd-like per-user extensibility.
> > 
> > -- Jamie
> > 
> > * plenty == too much.
> >   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
> >   (All but Bash implement their own virtual filesystem extensions).
> >   That makes them much less useful than they could be.
> 
> It has always bugged me that Gnome and KDE implement their own VFS layers.
> 

Same here.  This always seemed like something the kernel should be able
to handle.  It seems to me that if reiser4 had been available at the
time the Gnome and KDE developers would not have needed to do this.

Lee

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:

> > It has always bugged me that Gnome and KDE implement their own VFS layers. 
> 
> Same here.  This always seemed like something the kernel should be able
> to handle.  It seems to me that if reiser4 had been available at the
> time the Gnome and KDE developers would not have needed to do this.

Well, the kernel doesn't have a filesystem that speaks http, scp and
those things. GnomeVFS is URL-based. It has some pseudo-protocols that
extract a pseudo directory-tree for all installed applications + the
changes the used made, created on the fly from a set of XML files that
are read-only and system-wide and the user-overridden changes. I don't
know if all of these things would really make sense inside the kernel.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:

> > It has always bugged me that Gnome and KDE implement their own VFS layers. 
> 
> Same here.  This always seemed like something the kernel should be able
> to handle.  It seems to me that if reiser4 had been available at the
> time the Gnome and KDE developers would not have needed to do this.

Well, the kernel doesn't have a filesystem that speaks http, scp and
those things. GnomeVFS is URL-based. It has some pseudo-protocols that
extract a pseudo directory-tree for all installed applications + the
changes the used made, created on the fly from a set of XML files that
are read-only and system-wide and the user-overridden changes. I don't
know if all of these things would really make sense inside the kernel.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
> Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:
> 
> > > It has always bugged me that Gnome and KDE implement their own VFS layers. 
> > 
> > Same here.  This always seemed like something the kernel should be able
> > to handle.  It seems to me that if reiser4 had been available at the
> > time the Gnome and KDE developers would not have needed to do this.
> 
> Well, the kernel doesn't have a filesystem that speaks http, scp and
> those things. GnomeVFS is URL-based. It has some pseudo-protocols that
> extract a pseudo directory-tree for all installed applications + the
> changes the used made, created on the fly from a set of XML files that
> are read-only and system-wide and the user-overridden changes. I don't
> know if all of these things would really make sense inside the kernel.
> 

True.  FWIW, I never use most of those features.  It's just too damn
slow.  Windows seems to implement all of the useful features of
GnomeVFS, and they are 10x faster.

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
> Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:
> 
> > > It has always bugged me that Gnome and KDE implement their own VFS layers. 
> > 
> > Same here.  This always seemed like something the kernel should be able
> > to handle.  It seems to me that if reiser4 had been available at the
> > time the Gnome and KDE developers would not have needed to do this.
> 
> Well, the kernel doesn't have a filesystem that speaks http, scp and
> those things. GnomeVFS is URL-based. It has some pseudo-protocols that
> extract a pseudo directory-tree for all installed applications + the
> changes the used made, created on the fly from a set of XML files that
> are read-only and system-wide and the user-overridden changes. I don't
> know if all of these things would really make sense inside the kernel.
> 

True.  FWIW, I never use most of those features.  It's just too damn
slow.  Windows seems to implement all of the useful features of
GnomeVFS, and they are 10x faster.

Lee

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote:
>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
>> are read-only and system-wide and the user-overridden changes. I don't
>> know if all of these things would really make sense inside the kernel.
>True.  FWIW, I never use most of those features.  It's just too damn
>slow.  Windows seems to implement all of the useful features of
>GnomeVFS, and they are 10x faster.

Are they in the kernel in Windows?

-- 
mjt

Re: silent semantic changes with reiser4

[mjt]

On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote:
>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
>> are read-only and system-wide and the user-overridden changes. I don't
>> know if all of these things would really make sense inside the kernel.
>True.  FWIW, I never use most of those features.  It's just too damn
>slow.  Windows seems to implement all of the useful features of
>GnomeVFS, and they are 10x faster.

Are they in the kernel in Windows?

-- 
mjt

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote:
>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
>> are read-only and system-wide and the user-overridden changes. I don't
>> know if all of these things would really make sense inside the kernel.
>True.  FWIW, I never use most of those features.  It's just too damn
>slow.  Windows seems to implement all of the useful features of
>GnomeVFS, and they are 10x faster.

Are they in the kernel in Windows?

-- 
mjt

Re: silent semantic changes with reiser4

[Spam]

Hello Markus,

Friday, August 27, 2004, 11:21:31 AM, you wrote:

> On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote:
>>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
>>> are read-only and system-wide and the user-overridden changes. I don't
>>> know if all of these things would really make sense inside the kernel.
>>True.  FWIW, I never use most of those features.  It's just too damn
>>slow.  Windows seems to implement all of the useful features of
>>GnomeVFS, and they are 10x faster.

> Are they in the kernel in Windows?

  Some are, some aren't :)

  Many  of the features are between the FS and the application layers.
  Therefore  they will work with all programs that use the Windows API
  to  access  files  etc.  I  do not think that many programs actually
  access   disks  directly  at the FS or below level. Exceptions would
  perhaps be file-recovery tools etc.

  Others are simply plugins to Windows Explorer.

  I  don't  know  if  all,  any,  or none, are installed at the kernel
  level. I doubt they are.

  But  the  problem  with  Linux  seem to be that programs actually do
  access  the  FS  directly.  Therefore  everything  will break if you
  change the FS structures to much.

  Like I said, this is IMHO something that Linux should somehow solve.
  Applications  should  really  not  access  the filesystems directly.
  Perhaps Grub and stuff like it should, but not many other.

  ~S

Re: silent semantic changes with reiser4

[Spam]

Hello Markus,

Friday, August 27, 2004, 11:21:31 AM, you wrote:

> On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote:
>>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
>>> are read-only and system-wide and the user-overridden changes. I don't
>>> know if all of these things would really make sense inside the kernel.
>>True.  FWIW, I never use most of those features.  It's just too damn
>>slow.  Windows seems to implement all of the useful features of
>>GnomeVFS, and they are 10x faster.

> Are they in the kernel in Windows?

  Some are, some aren't :)

  Many  of the features are between the FS and the application layers.
  Therefore  they will work with all programs that use the Windows API
  to  access  files  etc.  I  do not think that many programs actually
  access   disks  directly  at the FS or below level. Exceptions would
  perhaps be file-recovery tools etc.

  Others are simply plugins to Windows Explorer.

  I  don't  know  if  all,  any,  or none, are installed at the kernel
  level. I doubt they are.

  But  the  problem  with  Linux  seem to be that programs actually do
  access  the  FS  directly.  Therefore  everything  will break if you
  change the FS structures to much.

  Like I said, this is IMHO something that Linux should somehow solve.
  Applications  should  really  not  access  the filesystems directly.
  Perhaps Grub and stuff like it should, but not many other.

  ~S

Re: silent semantic changes with reiser4

[Jamie Lokier]

Lee Revell wrote:
> Same here.  This always seemed like something the kernel should be able
> to handle.  It seems to me that if reiser4 had been available at the
> time the Gnome and KDE developers would not have needed to do this.

reiser4 has prompted the discussion again, but it doesn't provide the
virtual filesystem capabilities that we're talking about, which Gnome
and KDE implement in userspace.  The FUSE project is much more
relevant for that.

We had file-as-directory discussions years ago, and 5 years ago I was
experimenting with doing virtual filesystems with that capability over NFS.

podfuk offered something close to Gnome and KDE but in kernel space:
virtual filesytems accessible through the kernel, but it didn't get
too popular.  It was based on Midnight Commander's VFS (which has
evolved into Gnome's VFS), using the CODA filesystem interface to
provide the kernel hooks.  It had some problems, and was a bit fiddly
to use.  We still don't have something that's particularly nice to use
for ordinary users.

-- Jamie

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > One of the big potential uses for file-as-directory is to go inside
> > > archive files, ELF files, .iso files and so on in a convenient way.
> > 
> > Arguably this belongs in userspace --- and people have put it there.
> 
> I agree that these belong in userspace, and that there's plenty* of
> userspace code doing a similar thing already.  I don't think there's
> any argument over it.
> 
> However, as far as I know it's not accessible in a file-as-directory
> form as yet.  In my opinion that is the most natural form and it would
> be very intuitive to use.  

It does not work. .deb file is ar archive. It is also debian
package. How do you know if I want to see it as a package or as a archive?
How do you identify file types, anyway?

Same happens for .tar.gz.

uservfs does

cd foo.deb#uar
vs.
cd foo.deb#udeb

and

cd foo.tgz#utar
vs.
cat foo.tgz#ugz


-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pavel Machek wrote:
[...]
| uservfs does
|
| cd foo.deb#uar
cd foo.deb/ar
| vs.
| cd foo.deb#udeb
cd foo.deb/deb

and why would you want that, instead of just:
cd foo.deb	# for the ar
dpkg -i foo.deb	# for the deb

|
| and
|
| cd foo.tgz#utar
cd foo.tgz
| vs.
| cat foo.tgz#ugz
cat foo.tgz

Just want to extract the tar file?  Maybe something like
cat foo.tgz/gunzip
In which case (of course) foo.tgz/gunzip has exactly the same directory
contents as foo.tgz

Looks different, that's all.

In fact, for just about any syntax anyone could suggest, I can't really
see why you can't just replace all weird symbols with a slash and a
symbol.  Instead of
	foo.tgz#utar
you have
	foo.tgz/#/utar
Only difference is, some things which used to require special tools can
now be serviced by less than what's in busybox.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTPNpngHNmZLgCUhAQJ0PA//RmPgHL4MgiQn7Li5b52DI/0p+Zp9JUs7
OaA+k4wHeXEIdBProtBQ+3541noEg086bVriYRnajsA9xgd/P9ncE7ci/d9Kagdl
8KmH4eWnCjNkL4x8GnDeR+EM5EfcgoSxR7ezO1KO9uMMcPFnCvmeB1yB5U+Fr9Q1
HVPHfGsh+ZWU2CFJHoJx27Q07UK+bFvzGr4JkhLbRMbPeEROrkHJHp3wj/N2yXXb
5icLgFa0g/b6wM8SuBAaO5xVM8jkmECI/P5Jo3n/d/CTUumb/BoKMIgbMurR61CG
hrZ40mizpCMFAnLrgm9FIvjKtZUsigG+oM/2xTtku2Z2nSM16/ChKtJOAj6MoMGT
xvWzLdTY6kEL30VDaHtcVvIyUFtUe5oYxEasnrcKseV7hJtccgjTYKB21PnWyZFz
/KIdh8sBg6i1nuYOtHncL6agD1M2bg1vBYwbMvIRfa1YbAFUbW1A6hDTftxJIhLn
Beso6d68SyfNMdQ8yWloR7sIufmbDut2fx1SHS3wPt2Z5W1e0XBexcpSqdD8KPns
lvmC0ggPDuQzXjPCIsS17Uk5vrCtVEdrsCCaj62a8UE2d1mCqu4CyLwMYCFv3Sve
SwZ36iOv4fzdvgjPOUg27tMwbpa/WiDAcZJ/31BfZfTDl4E9LbkQjWYT/YgsJuaQ
wp3Hi/Cgh/4=
=8YVp
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> | uservfs does
> |
> | cd foo.deb#uar
> cd foo.deb/ar
> | vs.
> | cd foo.deb#udeb
> cd foo.deb/deb
> 
> and why would you want that, instead of just:
> cd foo.deb	# for the ar
> dpkg -i foo.deb	# for the deb

Because I want to see contents of that .deb package, nicely parsed?

> Just want to extract the tar file?  Maybe something like
> cat foo.tgz/gunzip
> In which case (of course) foo.tgz/gunzip has exactly the same directory
> contents as foo.tgz

Yes, that would work.

> In fact, for just about any syntax anyone could suggest, I can't really
> see why you can't just replace all weird symbols with a slash and a
> symbol.  Instead of
> 	foo.tgz#utar
> you have
> 	foo.tgz/#/utar
> Only difference is, some things which used to require special tools can
> now be serviced by less than what's in busybox.

That would work, too. I do not get your comment about busybox.
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Pavel" == Pavel Machek <pavel@ucw.cz> writes:

Pavel> Hi!
>> | uservfs does
>> |
>> | cd foo.deb#uar cd foo.deb/ar | vs.  | cd foo.deb#udeb cd
>> foo.deb/deb
>> 
>> and why would you want that, instead of just: cd foo.deb # for the ar
>> dpkg -i foo.deb # for the deb

Pavel> Because I want to see contents of that .deb package, nicely
Pavel> parsed?

What nicely-parsed information do you want to see?  A deb package is an
ar archive of three files:
- debian-binary, which contains the deb format version
- control.tar.gz, which contains the control data and related things
- data.tar.gz, which contains the actual files to be installed

So to get at the format version, you can "cat foo.deb/debian-binary".
To look at the control data, you can "ls foo.deb/control.tar.gz".  To
browse the files, "cd foo.deb/data.tar.gz".

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Horst von Brand]

Pavel Machek <pavel@ucw.cz> said:
> David Masover <ninja@slaphack.com> said:

[...]

> > Just want to extract the tar file?  Maybe something like
> > cat foo.tgz/gunzip
> > In which case (of course) foo.tgz/gunzip has exactly the same directory
> > contents as foo.tgz
> 
> Yes, that would work.
> 
> > In fact, for just about any syntax anyone could suggest, I can't really
> > see why you can't just replace all weird symbols with a slash and a
> > symbol.  Instead of
> > 	foo.tgz#utar
> > you have
> > 	foo.tgz/#/utar
> > Only difference is, some things which used to require special tools can
> > now be serviced by less than what's in busybox.
> 
> That would work, too. I do not get your comment about busybox.

You do need extra tools anyway, placing them in the kernel is cheating (and
absolutely pointless, IMHO).
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Horst von Brand wrote:
> 
> You do need extra tools anyway, placing them in the kernel is cheating (and
> absolutely pointless, IMHO).

I agree.

There's no point to having the kernel export information that is already 
inherent in the main stream.

I've seen all these examples of exposing MP3 ID information as a "side 
stream", and that's TOTALLY POINTLESS! The information is already there, 
it's in a standard format, and exporting it as a stream buys you 
absolutely nothing.

Where named attributes make sense is when they are _independent_ data.  
Data that is only tangentially related to the main stream itself, and
which the main stream _cannot_ encompass because of some real technical
issue.

In a graphical environment, the "icon" stream is a good example of this.  
It literally has _nothing_ to do with the data in the main stream. The
only linkage is a totally non-technical one, where the user wanted to
associate a secondary stream with the main stream _without_ altering the
main one. THAT is where named streams make sense.

But if you want to look at one particular file inside a tar-file, do so in 
user space. There are zero advantages to exposing it as a side-stream, and 
there are absolutely _tons_ of disadvantages.

In short, named streams only make sense if:
 - they are tied to the file some way that is _independent_ of the file 
   contents (since if it's dependent on the file contents, you're just a 
   ton better off regenerating it with a caching server)
 - there are serious reasons to keep the lookup synchronized (since if 
   there isn't such a reason, you're just better off with a separate 
   shadow tree in user space)

And realize that the "separate shadow tree" actually works very well.  
That's how version control systems like CVS have always worked. It's
certainly how you can make icon information work too. If you use a tool
for accessing the data, the tool can maintain coherency and you'll never
care about the side stream.

Which means that normally we really don't _want_ named streams. In 99% of
all cases we can use equally good - and _much_ simpler - tool-based
solutions.

Which means that the only _real_ technical issue for supporting named
streams really ends up being things like samba, which want named streams
just because the work they do fundamentally is about them, for externally
dictated reasons. Doing named streams for any other reason is likely just
being stupid.

Once you do decide that you have to do named streams, you might then 
decide to use them for convenient things like icons. But it should very 
much be a secondary issue at that point.

		Linus

Re: silent semantic changes with reiser4

[Spam]

> On Tue, 31 Aug 2004, Horst von Brand wrote:
>> 
>> You do need extra tools anyway, placing them in the kernel is cheating (and
>> absolutely pointless, IMHO).

> I agree.

> There's no point to having the kernel export information that is already
> inherent in the main stream.

  There are two things you are saying in the same sentence here. One I
  do  not  agree  with.  There is point to export information from the
  main  stream  to  secondary file streams. Just it doesn't have to be
  done by the kernel.

  Just  do  not limit things to a few defined types of things that can
  be  stored in a stream. IMHO a file stream should be able to contain
  just  about  anything  the user wishes. That said, the kernel should
  not have to know about the contents.

> I've seen all these examples of exposing MP3 ID information as a "side
> stream", and that's TOTALLY POINTLESS! The information is already there,
> it's in a standard format, and exporting it as a stream buys you 
> absolutely nothing.

  True.  But this is the case for this specific example. I have stated
  previously  about what I think would be legitimate uses as examples.
  Of  course  these  apply  to  me  and  what  I work with. One of the
  examples   were  to  add  streams  that  contained  descriptions  or
  keywords.

> Where named attributes make sense is when they are _independent_ data.
> Data that is only tangentially related to the main stream itself, and
> which the main stream _cannot_ encompass because of some real technical
> issue.

> In a graphical environment, the "icon" stream is a good example of this.
> It literally has _nothing_ to do with the data in the main stream. The
> only linkage is a totally non-technical one, where the user wanted to
> associate a secondary stream with the main stream _without_ altering the
> main one. THAT is where named streams make sense.

> But if you want to look at one particular file inside a tar-file, do so in
> user space. There are zero advantages to exposing it as a side-stream, and
> there are absolutely _tons_ of disadvantages.

  The  tar archive support seem (is) to be useless because tar is also
  so supported and implemented in every type of application out there.
  It is also easy to script and otherwise work with.

  But  the  idea to be able to extend the filesystem to show/represent
  data  as  file streams and meta directores is very good.

  Someone might come up with an idea to allow browsing and changing of
  data  inside  XML files. Of course this may already be done with sed
  and other tools.

> In short, named streams only make sense if:
>  - they are tied to the file some way that is _independent_ of the file
>    contents (since if it's dependent on the file contents, you're just a
>    ton better off regenerating it with a caching server)
>  - there are serious reasons to keep the lookup synchronized (since if
>    there isn't such a reason, you're just better off with a separate
>    shadow tree in user space)

> And realize that the "separate shadow tree" actually works very well.
> That's how version control systems like CVS have always worked. It's
> certainly how you can make icon information work too. If you use a tool
> for accessing the data, the tool can maintain coherency and you'll never
> care about the side stream.

  Just  that  with  shadow  trees  (I assume they are stored simply in
  another  folder  beside  the  file)  you  won't be able to "cp filea
  /mnt/backup" and keep the extra stuff.

  Sure  icons  could  also easily be stores as file.icon, just like on
  the Amiga :). It just isn't as nice.

> Which means that normally we really don't _want_ named streams. In 99% of
> all cases we can use equally good - and _much_ simpler - tool-based
> solutions.

> Which means that the only _real_ technical issue for supporting named
> streams really ends up being things like samba, which want named streams
> just because the work they do fundamentally is about them, for externally
> dictated reasons. Doing named streams for any other reason is likely just
> being stupid.

  I  doubt  it would be stupid. Perhaps today just not needed for many
  things.

> Once you do decide that you have to do named streams, you might then
> decide to use them for convenient things like icons. But it should very
> much be a secondary issue at that point.

  I think we focus to much on the actual contents of these streams. Do
  we  really  have  to  know  what will in the future be stored inside
  these streams?

  For  example.  Applications may use file streams for locking instead
  of having a .lock file. The future isn't known, and we shouldn't try
  to  limit  what  we can put in the streams. You said 99% of the uses
  won't  need  them.  True.  The  same is with xattr, ACLs and so many
  other things.

  Even  if 0.1% of the users today would use it I think we should have
  the support for named streams.

  ~S

>     Linus

Re: silent semantic changes with reiser4

[Spam]

> On Tue, 31 Aug 2004, Horst von Brand wrote:
>> 
>> You do need extra tools anyway, placing them in the kernel is cheating (and
>> absolutely pointless, IMHO).

> I agree.

> There's no point to having the kernel export information that is already
> inherent in the main stream.

  There are two things you are saying in the same sentence here. One I
  do  not  agree  with.  There is point to export information from the
  main  stream  to  secondary file streams. Just it doesn't have to be
  done by the kernel.

  Just  do  not limit things to a few defined types of things that can
  be  stored in a stream. IMHO a file stream should be able to contain
  just  about  anything  the user wishes. That said, the kernel should
  not have to know about the contents.

> I've seen all these examples of exposing MP3 ID information as a "side
> stream", and that's TOTALLY POINTLESS! The information is already there,
> it's in a standard format, and exporting it as a stream buys you 
> absolutely nothing.

  True.  But this is the case for this specific example. I have stated
  previously  about what I think would be legitimate uses as examples.
  Of  course  these  apply  to  me  and  what  I work with. One of the
  examples   were  to  add  streams  that  contained  descriptions  or
  keywords.

> Where named attributes make sense is when they are _independent_ data.
> Data that is only tangentially related to the main stream itself, and
> which the main stream _cannot_ encompass because of some real technical
> issue.

> In a graphical environment, the "icon" stream is a good example of this.
> It literally has _nothing_ to do with the data in the main stream. The
> only linkage is a totally non-technical one, where the user wanted to
> associate a secondary stream with the main stream _without_ altering the
> main one. THAT is where named streams make sense.

> But if you want to look at one particular file inside a tar-file, do so in
> user space. There are zero advantages to exposing it as a side-stream, and
> there are absolutely _tons_ of disadvantages.

  The  tar archive support seem (is) to be useless because tar is also
  so supported and implemented in every type of application out there.
  It is also easy to script and otherwise work with.

  But  the  idea to be able to extend the filesystem to show/represent
  data  as  file streams and meta directores is very good.

  Someone might come up with an idea to allow browsing and changing of
  data  inside  XML files. Of course this may already be done with sed
  and other tools.

> In short, named streams only make sense if:
>  - they are tied to the file some way that is _independent_ of the file
>    contents (since if it's dependent on the file contents, you're just a
>    ton better off regenerating it with a caching server)
>  - there are serious reasons to keep the lookup synchronized (since if
>    there isn't such a reason, you're just better off with a separate
>    shadow tree in user space)

> And realize that the "separate shadow tree" actually works very well.
> That's how version control systems like CVS have always worked. It's
> certainly how you can make icon information work too. If you use a tool
> for accessing the data, the tool can maintain coherency and you'll never
> care about the side stream.

  Just  that  with  shadow  trees  (I assume they are stored simply in
  another  folder  beside  the  file)  you  won't be able to "cp filea
  /mnt/backup" and keep the extra stuff.

  Sure  icons  could  also easily be stores as file.icon, just like on
  the Amiga :). It just isn't as nice.

> Which means that normally we really don't _want_ named streams. In 99% of
> all cases we can use equally good - and _much_ simpler - tool-based
> solutions.

> Which means that the only _real_ technical issue for supporting named
> streams really ends up being things like samba, which want named streams
> just because the work they do fundamentally is about them, for externally
> dictated reasons. Doing named streams for any other reason is likely just
> being stupid.

  I  doubt  it would be stupid. Perhaps today just not needed for many
  things.

> Once you do decide that you have to do named streams, you might then
> decide to use them for convenient things like icons. But it should very
> much be a secondary issue at that point.

  I think we focus to much on the actual contents of these streams. Do
  we  really  have  to  know  what will in the future be stored inside
  these streams?

  For  example.  Applications may use file streams for locking instead
  of having a .lock file. The future isn't known, and we shouldn't try
  to  limit  what  we can put in the streams. You said 99% of the uses
  won't  need  them.  True.  The  same is with xattr, ACLs and so many
  other things.

  Even  if 0.1% of the users today would use it I think we should have
  the support for named streams.

  ~S

>     Linus

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > You do need extra tools anyway, placing them in the kernel is cheating (and
> > absolutely pointless, IMHO).
> 
> I agree.
> 
> There's no point to having the kernel export information that is already 
> inherent in the main stream.
> 
> I've seen all these examples of exposing MP3 ID information as a "side 
> stream", and that's TOTALLY POINTLESS! The information is already there, 
> it's in a standard format, and exporting it as a stream buys you 
> absolutely nothing.

It buys me caching. I do quite often

bzcat patch.2.6.8.bz2 | less (read the patch)
(sometimes repeat that few times because I hit ^c when I should not
have etc).
cd ...clean; bzcat patch.2.6.8.bz2 | patch -Esp1
cd ...linux; bzcat patch.2.6.8.bz2 | patch -Esp1

Now... that's total waste of cpu. bzip2 decompression takes quite some
time. 

I could do

bzcat patch.2.6.8.bz2 > /tmp/delete.me.when.you.are.done

...but I'd probably forget to delete that one and anyway, it requires
me to think about it. Nicest way would be

cat patch.2.6.8.bz2/ubz | less
cd ...clean; cat patch.2.6.8.bz2/ubz | patch -Esp1
cd ...linux; cat patch.2.6.8.bz2/ubz | patch -Esp1

with kernel intelligently caching uncompressed data. I believe this
can not be done completely in userspace.
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Pavel Machek wrote:
> 
> It buys me caching.

I'll actually buy into that. If only because I consider caching to be one 
of the more important things that the kernel does (caches are a _classic_ 
case of "shared data that needs synchronization").

However, that said, user space can trivially cache things in the 
filesystem, so while this may be a convenient feature, I think you should 
look at perhaps doing it in the _shell_ instead..

		Linus

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > It buys me caching.
> 
> I'll actually buy into that. If only because I consider caching to be one 
> of the more important things that the kernel does (caches are a _classic_ 
> case of "shared data that needs synchronization").
> 
> However, that said, user space can trivially cache things in the 
> filesystem, so while this may be a convenient feature, I think you should 
> look at perhaps doing it in the _shell_ instead..

That cache should disappear as soon as I need disk
space. I.e. userspace should never see -ENOSPC because of this kind of
caching. This need some kernel support. Ouch and cached file should
atomically go away as soon as main file changes, otherwise I do not
see how multiple processes could cooperate on caching...

chattr +kill-this-file-when-low-on-disk-space patch.bz2...ubz 

would solve first problem. Not sure how to do the second one.

								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Pavel Machek wrote:
> > 
> > However, that said, user space can trivially cache things in the 
> > filesystem, so while this may be a convenient feature, I think you should 
> > look at perhaps doing it in the _shell_ instead..
> 
> That cache should disappear as soon as I need disk
> space. I.e. userspace should never see -ENOSPC because of this kind of
> caching. This need some kernel support. Ouch and cached file should
> atomically go away as soon as main file changes, otherwise I do not
> see how multiple processes could cooperate on caching...

Well, what other projects have done is to just reserve a certain amount of 
disk for caching. See "ccache", which solves both of the above problems 
(it doesn't shrink the cache on ENOSPC, but reserving diskspace is 
accepted practice for things like this..)

		Linus

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > However, that said, user space can trivially cache things in the 
> > > filesystem, so while this may be a convenient feature, I think you should 
> > > look at perhaps doing it in the _shell_ instead..
> > 
> > That cache should disappear as soon as I need disk
> > space. I.e. userspace should never see -ENOSPC because of this kind of
> > caching. This need some kernel support. Ouch and cached file should
> > atomically go away as soon as main file changes, otherwise I do not
> > see how multiple processes could cooperate on caching...
> 
> Well, what other projects have done is to just reserve a certain amount of 
> disk for caching. See "ccache", which solves both of the above problems 
> (it doesn't shrink the cache on ENOSPC, but reserving diskspace is 
> accepted practice for things like this..)

Okay, that does work, it just is not really nice. Just as reserving
fixed ammount of space for disk cache is bad, reserving fixed ammount
of space for ccache (and similar) is bad. When there are few of such
caches, balancing between them starts to matter...
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: silent semantic changes with reiser4

[Christer Weinigel]

Pavel Machek <pavel@ucw.cz> writes:

> Okay, that does work, it just is not really nice. Just as reserving
> fixed ammount of space for disk cache is bad, reserving fixed ammount
> of space for ccache (and similar) is bad. When there are few of such
> caches, balancing between them starts to matter...

So try to convince them to use the same cache daemon or the same
shared cache manager library then.  It isn't all that different from
implementing a kernel interface that everyone is supposed to use.

A cache manager daemon could sit and watch the free space on the disk
every other second and start deleting the cached files (according to
some LCU heuristics or whatever) whenever free space is getting low.

To see if the original file is newer than the cached file, good old
mtime can be used.  

I can't remember who right now, but someone mentioned a few problems
with timestamps:

1. The resolutions is too low in current kernels, but that just means
   that someone(tm) needs to implement nanosecond timestamps in the
   kernel and for the filesystems.

2. Even nanosecond timestams may become to coarse in the future.  One
   way of getting around this is to keep track of the latest timestamp
   written and make sure the next timestamp is one higher than the
   latest one.  This way timestamps are always increasing, and I doubt
   that we can have modifications each nanosecond for a long time with
   the hardware available for the next ten years.

3. If time goes backwards beacuse somebody changes the clock, the
   caching (or makefile rules) can get confued.  This is a hard one,
   but I actually we can use the doctors response "don't do that
   then".  With NTP we can guarantee a monotonically increasing clock.

I belive the kernel could give some assistance to make it easier to
see if a file has been modified, I remember that a few suggestions
were thrown around the last time Samba and dcache aliases were
discussed on l-k.  I definitely belive that kind of infrastructure
belongs in the kernel.  But the cache manager itself, no.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christer Weinigel wrote:
| Pavel Machek <pavel@ucw.cz> writes:
|
|
|>Okay, that does work, it just is not really nice. Just as reserving
|>fixed ammount of space for disk cache is bad, reserving fixed ammount
|>of space for ccache (and similar) is bad. When there are few of such
|>caches, balancing between them starts to matter...
|
|
| So try to convince them to use the same cache daemon or the same
| shared cache manager library then.  It isn't all that different from
| implementing a kernel interface that everyone is supposed to use.

You're right, it's not.  Is there some huge performance loss by having a
kernel interface that then turns around and calls a userland daemon?  If
yes, I can understand the argument.  If no, then the kernel interface
should ultimately be able to replace gnome-vfs -- and it should rely
heavily on userland utilities.

Caching is very generic.  It's so generic that there should be exactly
one cache manager, the way there is exactly one VM manager for Linux.
If there's a choice, choose it at boot or compile-time.

Abiword uses gzipped files.  So does gnumeric, gnucash, cube (a cute
little game engine), and many others.  There's no way every single one
will suddenly use a new library/daemon.  But with plugins and
intelligent caching, any kind of compression could be supported without
the app needing to support it natively -- either it goes in a compressed
folder or the user browses to it and changes an attribute.


| A cache manager daemon could sit and watch the free space on the disk
| every other second and start deleting the cached files (according to
| some LCU heuristics or whatever) whenever free space is getting low.

Agreed.  The daemon should go in userspace.  But the interface should be
in the kernel.  Just as no one suggested putting tar in the kernel, I
don't think anyone would really insist that the daemon needs to go in
the kernel.

That is, unless critical system files start actually being cached
streams of the actual package from the distro...  But this way leads to
madness.  Any serious issues here are better solved by an initrd.

[...]
| I belive the kernel could give some assistance to make it easier to
| see if a file has been modified, I remember that a few suggestions
| were thrown around the last time Samba and dcache aliases were
| discussed on l-k.  I definitely belive that kind of infrastructure
| belongs in the kernel.  But the cache manager itself, no.

Isn't there already dnotify?

Dnotify is wrong here, because I don't think it would be sane in a
sufficiently large setup.  But wouldn't it be nice if the kernel not
only told you whether a file has been modified, but told you immediately
after it was modified?

Yet another reason why there has to be a common cache and significant
kernel management -- if I've got a drive of less than, say, 5 gigs, I am
simply not willing to reserve any large amount of it for cache.  Take
the example of a source tarball being automatically extracted and the
contents cached.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUpPHgHNmZLgCUhAQLhIg/9G87W4OCt8hOvTK+XniPgWGn+YC7t4fKy
mi4QowX1eLmnFEovprjSUzPgKRMeDSChZ8M1pvXXWJQ/iULE44WH8vNiH8H974zw
h5O8dNI/AczuzaCalQ+BYXEGM8pasaCGCI/sLrr+WBy97NOpM2VUUKY+qKxSFWFt
x7u7/XCGGWSHVG3t3KzpkqD2K8u3IGSrV/vHIR2PLatEXu3G8K8sUAhfXMgFsTFp
gQAuQN/YzQKGUphnMzFWLDMYyEYjAA4VACZFqFz9QJIYeWzP+YmGdLfZ5mzN91JF
g5ACGHUxbYdjZsjTh6RbalVr1HpAAy8mvGNydbwUvuOxaXBvH44vUfRLnvIUoZs9
BB0Tq4xPqAwAtP0Z7b5Qem1ef3gqGXo1hfsEJWJZPv1lssYzn1+Jpn99p6N6VkXI
fbAHnypbHeYbf30lC5lvID0yngb+s5EUaxzxOYqFA8QpEJD/bl3S9Qx6bpom/9AL
U8kdK8QjNbhHSMrOpl8mg77VRLXdINoV7R5iSXXUMdIcA1j+4R1AIPh62JAMwQDF
/Vzek8KNyxvJDgF/y78LiGETZcG/l2W4szEsJFFF4iIKLri5+OFGiLFz8jVNViKu
Lr7Q7u7TgWSSykdTrHFcR/WRupAHeL5RB5DA9HBZErqgAgJWz/Rjv743G8PYFKs8
oPhRyePnFkE=
=cQLE
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> I belive the kernel could give some assistance to make it easier to
> see if a file has been modified, I remember that a few suggestions
> were thrown around the last time Samba and dcache aliases were
> discussed on l-k.  I definitely belive that kind of infrastructure
> belongs in the kernel.  But the cache manager itself, no.

Well, callback "filesystem full" would be nice, too; but I guess that
is "support user cache", too...
								Pavel
-- 
When do you have heart between your knees?

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes:

[...]

Linus> I've seen all these examples of exposing MP3 ID information as a
Linus> "side stream", and that's TOTALLY POINTLESS! The information is
Linus> already there, it's in a standard format, and exporting it as a
Linus> stream buys you absolutely nothing.

Exporting it through the filesystem would mean that I could use my
favourite editor to edit MP3/OGG tags, instead of having to use some
other program that has a piece-of-crap interface.  It's a
user-interface issue, and not a technical issue.

Someone might then tell me to write my own MP3-tag editor that has an
interface that I like.  How may MP3 tag editors do we need?  Besides, I
have better things to do with my time.

Actually, I would argue that things like ID3 tags for MP3 files, and
EXIF for JPEG exist only because filesystems in general can't support
associating metadata/attributes with files.  If filesystems were built
the "right way" to begin with, all those tagging schemes would never
have come into existence.  Of course, that's just a dream, but one can
dream, can't they?

[...]

Linus> And realize that the "separate shadow tree" actually works very
Linus> well.  That's how version control systems like CVS have always
Linus> worked.

VMS (or some other OS -- too lazy to look it up) exposed version control
through the filesystem.  Their naming sucked (or so says Pike), but
that's what they did.

Yes, CVS works, but I wouldn't say that it works very well.  If I want
to get a specific version of a file, I need to drop to the command line
and type in some cvs command that I don't remember and then open the
file.  I can't just do "C-x C-f foo.c/version/2.1" in emacs.

Yes, emacs has version-control support, so that I can open up a
specific version.  But if v-c isn't exposed through the filesystem, we
would have to add support to emacs for every v-c system out there.  And
do the same for vi, gedit, nano, bluefish, conglomerate, Mozilla
Composer, glade, etc.

Linus> It's certainly how you can make icon information work too. If you
Linus> use a tool for accessing the data, the tool can maintain
Linus> coherency and you'll never care about the side stream.

The point is that I shouldn't need to use a special tool.  I should be
able to use my favourite text/image/sound/etc. editor.

Using a special tool may work for you, but it isn't the optimal user
interface in general.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes:

[...]

Linus> I've seen all these examples of exposing MP3 ID information as a
Linus> "side stream", and that's TOTALLY POINTLESS! The information is
Linus> already there, it's in a standard format, and exporting it as a
Linus> stream buys you absolutely nothing.

Exporting it through the filesystem would mean that I could use my
favourite editor to edit MP3/OGG tags, instead of having to use some
other program that has a piece-of-crap interface.  It's a
user-interface issue, and not a technical issue.

Someone might then tell me to write my own MP3-tag editor that has an
interface that I like.  How may MP3 tag editors do we need?  Besides, I
have better things to do with my time.

Actually, I would argue that things like ID3 tags for MP3 files, and
EXIF for JPEG exist only because filesystems in general can't support
associating metadata/attributes with files.  If filesystems were built
the "right way" to begin with, all those tagging schemes would never
have come into existence.  Of course, that's just a dream, but one can
dream, can't they?

[...]

Linus> And realize that the "separate shadow tree" actually works very
Linus> well.  That's how version control systems like CVS have always
Linus> worked.

VMS (or some other OS -- too lazy to look it up) exposed version control
through the filesystem.  Their naming sucked (or so says Pike), but
that's what they did.

Yes, CVS works, but I wouldn't say that it works very well.  If I want
to get a specific version of a file, I need to drop to the command line
and type in some cvs command that I don't remember and then open the
file.  I can't just do "C-x C-f foo.c/version/2.1" in emacs.

Yes, emacs has version-control support, so that I can open up a
specific version.  But if v-c isn't exposed through the filesystem, we
would have to add support to emacs for every v-c system out there.  And
do the same for vi, gedit, nano, bluefish, conglomerate, Mozilla
Composer, glade, etc.

Linus> It's certainly how you can make icon information work too. If you
Linus> use a tool for accessing the data, the tool can maintain
Linus> coherency and you'll never care about the side stream.

The point is that I shouldn't need to use a special tool.  I should be
able to use my favourite text/image/sound/etc. editor.

Using a special tool may work for you, but it isn't the optimal user
interface in general.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Christer Weinigel]

Linus Torvalds <torvalds@osdl.org> writes:

> In a graphical environment, the "icon" stream is a good example of this.  
> It literally has _nothing_ to do with the data in the main stream. The
> only linkage is a totally non-technical one, where the user wanted to
> associate a secondary stream with the main stream _without_ altering the
> main one. THAT is where named streams make sense.

I think that the "icon" argument for named streams is a silly
argument, since different users may want to have different icons for
the same file.  Say that I want /usr/bin/emacs to have the enterprise
icon and someone else wants the gnu head icon.  And besides, root owns
the file anyways, so neither of us mortal users should be able to add
a stream to it.

Another reason for named streams that usually crops up is the ability
set a "preferred application" for a certain file, so that when I
double click on a document I want to open it with antiword instead of
openoffice.  But the same contra-argument applies here, different
users have different preferences.

I can see the argument for having the equivalent of Content-type or
Content-transfer-encoding as a named stream though.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Spam]

  

> Linus Torvalds <torvalds@osdl.org> writes:

>> In a graphical environment, the "icon" stream is a good example of this.
>> It literally has _nothing_ to do with the data in the main stream. The
>> only linkage is a totally non-technical one, where the user wanted to
>> associate a secondary stream with the main stream _without_ altering the
>> main one. THAT is where named streams make sense.

> I think that the "icon" argument for named streams is a silly
> argument, since different users may want to have different icons for
> the same file.  Say that I want /usr/bin/emacs to have the enterprise
> icon and someone else wants the gnu head icon.  And besides, root owns
> the file anyways, so neither of us mortal users should be able to add
> a stream to it.

  Yet again are we thinking in blocking ways. Firstly this was an
  example. Usually, though, most users accept the default icon for a
  file. If they do not they can still change the icon for the link
  they make on their start-menu/home folder/etc.

> Another reason for named streams that usually crops up is the ability
> set a "preferred application" for a certain file, so that when I
> double click on a document I want to open it with antiword instead of
> openoffice.  But the same contra-argument applies here, different
> users have different preferences.

  I can make the same argument as for the icons.

> I can see the argument for having the equivalent of Content-type or
> Content-transfer-encoding as a named stream though.

  That would be a nice thing.

>   /Christer

Re: silent semantic changes with reiser4

[Spam]

  

> Linus Torvalds <torvalds@osdl.org> writes:

>> In a graphical environment, the "icon" stream is a good example of this.
>> It literally has _nothing_ to do with the data in the main stream. The
>> only linkage is a totally non-technical one, where the user wanted to
>> associate a secondary stream with the main stream _without_ altering the
>> main one. THAT is where named streams make sense.

> I think that the "icon" argument for named streams is a silly
> argument, since different users may want to have different icons for
> the same file.  Say that I want /usr/bin/emacs to have the enterprise
> icon and someone else wants the gnu head icon.  And besides, root owns
> the file anyways, so neither of us mortal users should be able to add
> a stream to it.

  Yet again are we thinking in blocking ways. Firstly this was an
  example. Usually, though, most users accept the default icon for a
  file. If they do not they can still change the icon for the link
  they make on their start-menu/home folder/etc.

> Another reason for named streams that usually crops up is the ability
> set a "preferred application" for a certain file, so that when I
> double click on a document I want to open it with antiword instead of
> openoffice.  But the same contra-argument applies here, different
> users have different preferences.

  I can make the same argument as for the icons.

> I can see the argument for having the equivalent of Content-type or
> Content-transfer-encoding as a named stream though.

  That would be a nice thing.

>   /Christer

Re: silent semantic changes with reiser4

[John Newbigin]

Christer Weinigel wrote:

> 
> I think that the "icon" argument for named streams is a silly
> argument, since different users may want to have different icons for
> the same file.  Say that I want /usr/bin/emacs to have the enterprise
> icon and someone else wants the gnu head icon.  And besides, root owns
> the file anyways, so neither of us mortal users should be able to add
> a stream to it.
> 
> Another reason for named streams that usually crops up is the ability
> set a "preferred application" for a certain file, so that when I
> double click on a document I want to open it with antiword instead of
> openoffice.  But the same contra-argument applies here, different
> users have different preferences.
> 
> I can see the argument for having the equivalent of Content-type or
> Content-transfer-encoding as a named stream though.
You still have the same problem with that.  What if I want to override 
the content-type for some reason.  (I have had to do this is the past, 
but not for root owned files.  In the end though I had to edit 
/etc/mailcap to do what I wanted :( )

You still have problems with icons.  What format should they be stored 
in?  If all this effort is to make samba work then would this mean a 
windows icon?  That is not going to make your X11 app happy.

This could probably all be solved with magic plugins in the filesystem 
(reiserfs style) but it is probably better to return to simple approach 
of "everything is a file".  The executable /usr/bin/emacs is a file. 
The way it looks on my desktop is a file ~/.desktop/emacs etc.  This is 
what most apps do currently and it works quite well.

Just my 2c


-- 
John Newbigin - Computer Systems Officer
School of Information Technology
Swinburne University of Technology
Melbourne, Australia
http://www.it.swin.edu.au/staff/jnewbigin

Re: silent semantic changes with reiser4

[Alan Cox]

On Maw, 2004-08-31 at 21:05, Linus Torvalds wrote:
> In a graphical environment, the "icon" stream is a good example of this.  
> It literally has _nothing_ to do with the data in the main stream. The
> only linkage is a totally non-technical one, where the user wanted to
> associate a secondary stream with the main stream _without_ altering the
> main one. THAT is where named streams make sense.

The icon doesn't belong in the document, that was a catastrophic
disaster in early MacOS (although they only use an index). Users want to
manage their icon choices, tags, tooltip notes and attached labels, and
you cannot do that in the file if you don't own the file. 

Also the icon is *not* unrelated to the file in a modern GUI, eg rox and
nautilus uses scaled versions of the content for many media types and
will show you pictures, frames from a movie etc to help you remember the
content. On top they then add user specific annotations.

The things that are more independant are:

"This file was created by OpenOffice 1.2"
"This is a text/plain file in UTF-8"
"This file has a UUID of ...."

The type has dragons because you get heirarchical typing within
documents (consider XML containing namespaces)

(UUIDs being one really useful thing we don't tag everywhere onto files
that would be a godsend on the desktop providing they moved with the
file on rename)

Alan

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Alan" == Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

Alan> The icon doesn't belong in the document, that was a catastrophic
Alan> disaster in early MacOS (although they only use an index). Users
Alan> want to manage their icon choices, tags, tooltip notes and
Alan> attached labels, and you cannot do that in the file if you don't
Alan> own the file.

Most desktop environments provide defaults, but let the user customize
their own settings.  The substream *could* (I'm not saying if it's a
good idea.  It's just a possibility.) be used to provide a default
icon.

Alan> Also the icon is *not* unrelated to the file in a modern GUI, eg
Alan> rox and nautilus uses scaled versions of the content for many
Alan> media types and will show you pictures, frames from a movie etc to
Alan> help you remember the content.

For documents, the icon is not completely unrelated, but it isn't
something that can be directly extracted.  Generating thumbnails
usually takes time, so the filemanagers need to cache it somewhere.
Right now, they use ~/.thumbnails (at least that's what freedesktop.org
recommends), but then you run into consistency issues -- when you delete
the file, the thumbnail should be deleted too, and when you rename/move
the file, the thumbnail should be attached to the new filename, and not
have to be regenerated.

Alan> On top they then add user specific annotations.

And I think that would be a good use of substreams/attributes/metas
too.  You don't really want to store it in some separate location,
because you run into the same consistency issues as I mentioned
before.  Sure, if you move a file using Nautilus, it may be smart
enough to move the annotations, but I'd bet that mv doesn't know what
to do.

Alan> The things that are more independant are:

Alan> "This file was created by OpenOffice 1.2" "This is a text/plain
Alan> file in UTF-8" "This file has a UUID of ...."

Alan> The type has dragons because you get heirarchical typing within
Alan> documents (consider XML containing namespaces)

Well, I think the web guys are going to have to figure something out,
and we can just follow what they do ...

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Tue, Aug 31, 2004 at 01:05:40PM -0700, Linus Torvalds wrote:
>
>There's no point to having the kernel export information that is already 
>inherent in the main stream.

As said before, it can be bounced :)

You have support for streams, someone could read the inherent information
and write it to the stream. At the risk of these two streams being
unsynced temporarily, requiring some syncer.

>I've seen all these examples of exposing MP3 ID information as a "side 
>stream", and that's TOTALLY POINTLESS! The information is already there, 

No it's not pointless :)

>it's in a standard format, and exporting it as a stream buys you 
>absolutely nothing.

There is no nice standard way of finding MP3s by the ID information.

find music/ -name *mp3/..metas/id3/ | xargs grep -i "dream theater"

or something. Which reveals the metadata to find. 
The kernel doesn't have to parse the ID3 tags, that can be handled in 
userspace by a daemon, updated based on mtimes or somesuch, yes?

>Which means that normally we really don't _want_ named streams. In 99% of
>all cases we can use equally good - and _much_ simpler - tool-based
>solutions.

Sure, when someone goes through every application from ls to nautilus
and rams a tool-based patch down the app maintainer's throat and then
we're still dependent on the filesystem layout, which is easier to break
than the filesystem itself :)

>Which means that the only _real_ technical issue for supporting named
>streams really ends up being things like samba, which want named streams

So they should be implemented for samba and other guys can come up with
alternative means for using them ;)

-- 
mjt

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
| Pavel Machek <pavel@ucw.cz> said:
|
|>David Masover <ninja@slaphack.com> said:
|
|
| [...]
[...]
| You do need extra tools anyway, placing them in the kernel is cheating
(and
| absolutely pointless, IMHO).

Repeat after me:  plugins in kernel does NOT equal tools in kernel.

It's not hard to, for instance, imagine a generic plugin for archive
manipulation which talks to a userspace daemon/library.  The kernel
doesn't know anything other than (maybe) a list of extensions that are
archives.  All else is handled in userspace -- the idea that "this is a
zipfile" and "zipfiles can be extracted with the 'zip' command" are all
in userspace.

It's not about the kernel, it's about the interface.  And see my other mail:
	cat foo.zip/README
	less foo.zip/contents/bar.c
is a lot easier than
	lynx http://google.com/search?q=zip
	emerge zip
	man zip
	unzip foo.zip
	cat bar.c
which already assumes quite a lot of expertise.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUieXgHNmZLgCUhAQLj8Q//VEuRWg9RB/PVg/M1t36SraKIRdbrgGq6
0DYLZwwVWkfz+UKGhUkACNGQzhHG3ySn/lRjqXCsJhzMOzfgmte2CwgGxunHYqdp
W69OSZ1vv0ZLpdpzEkVdMfRhhg8RBL54Ewj6fQu7G1ZzJ0J0HUFCEZIsVoL4zJfk
+GFl4M+vvc+152HipEDsvrrPasc0dSxmryyzPbyABrfPRi3l2E3Okfewn/4MCC1E
H/L04nmvJQ2OSZYu1eSMSpnLrgQdZJbYefRuSinpqfKhjpwQXjj2rnfcPm+xc3hh
FDMlAUPA/DAlthTuhxd9+8iDPEARY57/FAMFTalYVhBEv0NWieMMn9V6R8BbnhZW
8y9EWu0Nvoowg9qElkVjWEt5M1Rbeuq4TDviyVyPYg/KcTKDGpjLuyQQbWv6hMVM
WAx6THtRK8Ji/XLCj4Kh6qDOEYP1nuwqLYSWv13BtNAJ3bVeKryOJ4Er4AmuJomA
uydBPQVsrtzTNmk7Eul2ZjrxXdESpUnUIAbu7H1Gbn51uP4rQTdlXkdPTv6s/wvG
OSOMWKs28GBFsG3+IJLnoN2wxrffpgTBEAaPwbh9TRFKLIH+pkCFSx2cIkTj93nr
RjETFObjYdJGhGygrq0xOmN5e7iprdxhxd3/+fvjQxnZGxXT3fZLUwhVX9/zhaXG
cxau2Onr3jc=
=MrR9
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
| Pavel Machek <pavel@ucw.cz> said:
|
|>David Masover <ninja@slaphack.com> said:
|
|
| [...]
[...]
| You do need extra tools anyway, placing them in the kernel is cheating
(and
| absolutely pointless, IMHO).

Repeat after me:  plugins in kernel does NOT equal tools in kernel.

It's not hard to, for instance, imagine a generic plugin for archive
manipulation which talks to a userspace daemon/library.  The kernel
doesn't know anything other than (maybe) a list of extensions that are
archives.  All else is handled in userspace -- the idea that "this is a
zipfile" and "zipfiles can be extracted with the 'zip' command" are all
in userspace.

It's not about the kernel, it's about the interface.  And see my other mail:
	cat foo.zip/README
	less foo.zip/contents/bar.c
is a lot easier than
	lynx http://google.com/search?q=zip
	emerge zip
	man zip
	unzip foo.zip
	cat bar.c
which already assumes quite a lot of expertise.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUieXgHNmZLgCUhAQLj8Q//VEuRWg9RB/PVg/M1t36SraKIRdbrgGq6
0DYLZwwVWkfz+UKGhUkACNGQzhHG3ySn/lRjqXCsJhzMOzfgmte2CwgGxunHYqdp
W69OSZ1vv0ZLpdpzEkVdMfRhhg8RBL54Ewj6fQu7G1ZzJ0J0HUFCEZIsVoL4zJfk
+GFl4M+vvc+152HipEDsvrrPasc0dSxmryyzPbyABrfPRi3l2E3Okfewn/4MCC1E
H/L04nmvJQ2OSZYu1eSMSpnLrgQdZJbYefRuSinpqfKhjpwQXjj2rnfcPm+xc3hh
FDMlAUPA/DAlthTuhxd9+8iDPEARY57/FAMFTalYVhBEv0NWieMMn9V6R8BbnhZW
8y9EWu0Nvoowg9qElkVjWEt5M1Rbeuq4TDviyVyPYg/KcTKDGpjLuyQQbWv6hMVM
WAx6THtRK8Ji/XLCj4Kh6qDOEYP1nuwqLYSWv13BtNAJ3bVeKryOJ4Er4AmuJomA
uydBPQVsrtzTNmk7Eul2ZjrxXdESpUnUIAbu7H1Gbn51uP4rQTdlXkdPTv6s/wvG
OSOMWKs28GBFsG3+IJLnoN2wxrffpgTBEAaPwbh9TRFKLIH+pkCFSx2cIkTj93nr
RjETFObjYdJGhGygrq0xOmN5e7iprdxhxd3/+fvjQxnZGxXT3fZLUwhVX9/zhaXG
cxau2Onr3jc=
=MrR9
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> | You do need extra tools anyway, placing them in the kernel is cheating
> (and
> | absolutely pointless, IMHO).
> 
> Repeat after me:  plugins in kernel does NOT equal tools in kernel.
> 
> It's not hard to, for instance, imagine a generic plugin for archive
> manipulation which talks to a userspace daemon/library.  The kernel
> doesn't know anything other than (maybe) a list of extensions that are
> archives.  All else is handled in userspace -- the idea that "this is a
> zipfile" and "zipfiles can be extracted with the 'zip' command" are all
> in userspace.
> 
> It's not about the kernel, it's about the interface.  And see my other mail:
> 	cat foo.zip/README
> 	less foo.zip/contents/bar.c
> is a lot easier than
> 	lynx http://google.com/search?q=zip
> 	emerge zip
> 	man zip
> 	unzip foo.zip
> 	cat bar.c
> which already assumes quite a lot of expertise.

If you only want nice user interface, you can have that today. Its
done using coda, and hosted at uservfs.sf.net.
								Pavel
-- 
When do you have heart between your knees?

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Pavel" == Pavel Machek <pavel@ucw.cz> writes:

Pavel> If you only want nice user interface, you can have that
Pavel> today. Its done using coda, and hosted at uservfs.sf.net.

Hmmm.  Last activity seems to be about three years ago.  Any idea if it
works with Linux 2.6?  Is there any chance at all that it will be
merged into the kernel?

Can anyone comment on the speed?  I don't have the ability right now to
try out any new kernel stuff.

The user interface isn't that nice either.  There's no browsability for
the methods.  (Hmm.  I have this strange .deb file.  How do I look at
the contents?  #utar? #ugz? ...)  And honestly, I think that "#" hack
is pretty ugly.  There's also possible name conflicts -- although it's
highly unlikely --  if I have a file that just happens to be called
foo.tar.gz#utar, I can't browse foo.tar.gz.

There may be some nice ideas in there, but from a quick lookover, it
seems to be lacking in execution.  I'll probably give it a try anyways,
during my next round of kernel compiling/testing.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Pavel Machek wrote:
> > It's not about the kernel, it's about the interface.  And see my other mail:
> > 	cat foo.zip/README
> > 	less foo.zip/contents/bar.c
> > is a lot easier than
> > 	lynx http://google.com/search?q=zip
> > 	emerge zip
> > 	man zip
> > 	unzip foo.zip
> > 	cat bar.c
> > which already assumes quite a lot of expertise.
> 
> If you only want nice user interface, you can have that today. Its
> done using coda, and hosted at uservfs.sf.net.

Can I do these already using uservfs?

   cd
   less foo.zip/contents/bar.c
   less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c
   grep -R obscure_label ~/RPMS

No I can't.

Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can
only do the above with convenient paths, completion etc. if I mount my
home directory and distfiles over uservfs, so that all I/O in those
directories including ordinary accesses is through uservfs, and have
the _real_ storage elsewhere.  That's quite illogical, cumbersome to
set up, and slows down normal I/O significantly.

I can do some of those things in a similar way if I do them inside
Midnight Commander, Gnome, KDE, or Emacs -- but even then a path that
works in one of those doesn't work in another, and the extensions
don't even work in all programs, plugins or scripts for those systems.

As we're adding file-as-directories to VFS, I propose it would be good
to have *hooks* in place, and non-metadata namespace reserved inside
the directories, so that uservfs-like userspace filesystems can
be auto-mounted at those places and synchronised with the file.

In other words, something that makes the next version of uservfs a
whole lot more useful.

The good news is that a lot of the hooks are falling into place already.

Kernel support is needed to handle the interface, cacheing to trigger
deletions, page cacheing so that I/O is a normal speed and mmap()
works, moveable mounts so that renames work and the cached data
followed them, and coherency so that if the underlying archive is
changed, it invalidates cached data with the appropriate degree of
synchronisation.  Note that the current dnotify cannot provide the
latter with any strength; it's too weak an interface (so is inotify).
(You can omit a lot of these things, but then you get a flaky interface).

The FUSE project looks like it will provide an efficient interface to
the page cache, avoiding round trips where possible and supporting
mmap.  If it doesn't, it should.  (Similar VFS facilities are useful
for distributed filesystems).  The presently discussed file-as-directory
semantics will provide a nice view, and force the necessary minimal
tool changes, so _all_ tools will be fine with the new uservfs.

So all I am asking for is a facility to auto-mount with
file-as-directory, and the ability for a userspace daemon to be
notified of regular file modifications synchronously.  Both can be
added later, once file-as-directory and moveable mounts are
established.  (fcntl leases and dnotify _almost_ provide this, but
they don't.  Looks like incoherent hacks keep getting added all over
the place for Samba... :).

That's all.  Kernel space needs to provide very little that it doesn't
already provide, for this rather nice and universally usable feature to work.

-- Jamie

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > It's not about the kernel, it's about the interface.  And see my other mail:
> > > 	cat foo.zip/README
> > > 	less foo.zip/contents/bar.c
> > > is a lot easier than
> > > 	lynx http://google.com/search?q=zip
> > > 	emerge zip
> > > 	man zip
> > > 	unzip foo.zip
> > > 	cat bar.c
> > > which already assumes quite a lot of expertise.
> > 
> > If you only want nice user interface, you can have that today. Its
> > done using coda, and hosted at uservfs.sf.net.
> 
> Can I do these already using uservfs?
> 
>    cd
>    less foo.zip/contents/bar.c

less foo.zip#uzip/contents/bar.c

>    less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c

less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c

>    grep -R obscure_label ~/RPMS

I do not think you'd want this. How would you tell grep obscure label
without entering archives, then? 

> No I can't.

> Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can
> only do the above with convenient paths, completion etc. if I mount
> my

Well, you can... or at least I had version somewhere that
automagically prepended /overlay when file was not found. That way,
normal requests were fast and uservfs were used whenever it was
needed... At the price of little uglyness and slowing down -ENOENTs.

> As we're adding file-as-directories to VFS, I propose it would be good
> to have *hooks* in place, and non-metadata namespace reserved inside
> the directories, so that uservfs-like userspace filesystems can
> be auto-mounted at those places and synchronised with the file.
> 
> In other words, something that makes the next version of uservfs a
> whole lot more useful.

Yep, something less hacky would be nice.

> So all I am asking for is a facility to auto-mount with
> file-as-directory, and the ability for a userspace daemon to be
> notified of regular file modifications synchronously.  Both can be
> added later, once file-as-directory and moveable mounts are

If the userspace daemon is synchronously notified of file
modification... will not that lead to very ugly deadlocks?
								Pavel

-- 
Horseback riding is like software...
...vgf orggre jura vgf serr.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Pavel Machek wrote:
> > Can I do these already using uservfs?
> > 
> >    cd
> >    less foo.zip/contents/bar.c
> 
> less foo.zip#uzip/contents/bar.c
> 
> >    less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c
> 
> less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c
>
> >    grep -R obscure_label ~/RPMS
> 
> I do not think you'd want this. How would you tell grep obscure label
> without entering archives, then? 

Granted, this is where we need "grep --recurse-into-files" or, more
generally useful, a find option.

> > No I can't.
> 
> > Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can
> > only do the above with convenient paths, completion etc. if I mount
> > my
> 
> Well, you can... or at least I had version somewhere that
> automagically prepended /overlay when file was not found. That way,
> normal requests were fast and uservfs were used whenever it was
> needed... At the price of little uglyness and slowing down -ENOENTs.

There you go... a kernel hook to prepend /overlay works :)
These hook I keep mentioning, they are really very simple :)

> > So all I am asking for is a facility to auto-mount with
> > file-as-directory, and the ability for a userspace daemon to be
> > notified of regular file modifications synchronously.  Both can be
> > added later, once file-as-directory and moveable mounts are
> 
> If the userspace daemon is synchronously notified of file
> modification... will not that lead to very ugly deadlocks?

Not if it's written properly.

-- Jamie

Re: silent semantic changes with reiser4

[Lee Revell]

On Wed, 2004-09-01 at 17:59, Jamie Lokier wrote:
> Pavel Machek wrote:
> > > Can I do these already using uservfs?
> > > 
> > >    cd
> > >    less foo.zip/contents/bar.c
> > 
> > less foo.zip#uzip/contents/bar.c
> > 
> > >    less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c
> > 
> > less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c
> >
> > >    grep -R obscure_label ~/RPMS
> > 
> > I do not think you'd want this. How would you tell grep obscure label
> > without entering archives, then? 
> 
> Granted, this is where we need "grep --recurse-into-files" or, more
> generally useful, a find option.
> 

FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
option, enabled by default, to look inside archives.  If you tell it to
look for a driver in a given directory it will also look inside .cab 
and .zip files.  It's extremely useful, I would imagine someone who uses
XP a lot will come to expect this feature.

Of course, no idea how it's implemented, but users like it.

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Wed, 2004-09-01 at 17:59, Jamie Lokier wrote:
> Pavel Machek wrote:
> > > Can I do these already using uservfs?
> > > 
> > >    cd
> > >    less foo.zip/contents/bar.c
> > 
> > less foo.zip#uzip/contents/bar.c
> > 
> > >    less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c
> > 
> > less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c
> >
> > >    grep -R obscure_label ~/RPMS
> > 
> > I do not think you'd want this. How would you tell grep obscure label
> > without entering archives, then? 
> 
> Granted, this is where we need "grep --recurse-into-files" or, more
> generally useful, a find option.
> 

FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
option, enabled by default, to look inside archives.  If you tell it to
look for a driver in a given directory it will also look inside .cab 
and .zip files.  It's extremely useful, I would imagine someone who uses
XP a lot will come to expect this feature.

Of course, no idea how it's implemented, but users like it.

Lee

Re: silent semantic changes with reiser4

[Horst von Brand]

Lee Revell <rlrevell@joe-job.com> said:

[...]

> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> option, enabled by default, to look inside archives.  If you tell it to
> look for a driver in a given directory it will also look inside .cab 
> and .zip files.  It's extremely useful, I would imagine someone who uses
> XP a lot will come to expect this feature.

It is trivial to implement this by looking inside the files. I.e., the way
mc has done this for ages.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 10:25, Horst von Brand wrote:
> Lee Revell <rlrevell@joe-job.com> said:
> 
> [...]
> 
> > FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > option, enabled by default, to look inside archives.  If you tell it to
> > look for a driver in a given directory it will also look inside .cab 
> > and .zip files.  It's extremely useful, I would imagine someone who uses
> > XP a lot will come to expect this feature.
> 
> It is trivial to implement this by looking inside the files. I.e., the way
> mc has done this for ages.

This requires a separate, MC-specific namespace.  The point is to unify
the namespace, not fragment it.

If Hans had a comprehensible web page, maybe more people would
understand this aspect of his argument.  Would you put digressions about
chaos theory and King Arthur in a man page?

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 10:25, Horst von Brand wrote:
> Lee Revell <rlrevell@joe-job.com> said:
> 
> [...]
> 
> > FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > option, enabled by default, to look inside archives.  If you tell it to
> > look for a driver in a given directory it will also look inside .cab 
> > and .zip files.  It's extremely useful, I would imagine someone who uses
> > XP a lot will come to expect this feature.
> 
> It is trivial to implement this by looking inside the files. I.e., the way
> mc has done this for ages.

This requires a separate, MC-specific namespace.  The point is to unify
the namespace, not fragment it.

If Hans had a comprehensible web page, maybe more people would
understand this aspect of his argument.  Would you put digressions about
chaos theory and King Arthur in a man page?

Lee

Re: silent semantic changes with reiser4

[Spam]

  

> Lee Revell <rlrevell@joe-job.com> said:

> [...]

>> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
>> option, enabled by default, to look inside archives.  If you tell it to
>> look for a driver in a given directory it will also look inside .cab
>> and .zip files.  It's extremely useful, I would imagine someone who uses
>> XP a lot will come to expect this feature.

> It is trivial to implement this by looking inside the files. I.e., the way
> mc has done this for ages.

  Difference is that you can't do "locate" or "find" or "Search".. You
  would have to open the files in an archive-supporting application
  such as mc.

Re: silent semantic changes with reiser4

[Spam]

  

> Lee Revell <rlrevell@joe-job.com> said:

> [...]

>> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
>> option, enabled by default, to look inside archives.  If you tell it to
>> look for a driver in a given directory it will also look inside .cab
>> and .zip files.  It's extremely useful, I would imagine someone who uses
>> XP a lot will come to expect this feature.

> It is trivial to implement this by looking inside the files. I.e., the way
> mc has done this for ages.

  Difference is that you can't do "locate" or "find" or "Search".. You
  would have to open the files in an archive-supporting application
  such as mc.

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-09-02 at 20:41, Spam wrote:
> > It is trivial to implement this by looking inside the files. I.e., the way
> > mc has done this for ages.
> 
>   Difference is that you can't do "locate" or "find" or "Search".. You
>   would have to open the files in an archive-supporting application
>   such as mc.

And would you rather that logic was running swappable in shared library
space or privileged and unswappable in kernel ?

Alan

Re: silent semantic changes with reiser4

[Spam]

  

> On Iau, 2004-09-02 at 20:41, Spam wrote:
>> > It is trivial to implement this by looking inside the files. I.e., the way
>> > mc has done this for ages.
>> 
>>   Difference is that you can't do "locate" or "find" or "Search".. You
>>   would have to open the files in an archive-supporting application
>>   such as mc.

> And would you rather that logic was running swappable in shared library
> space or privileged and unswappable in kernel ?

  I would rather have it as a filesystem/vfs plugin that would allow
  all my programs to use the features the plugin gives, even if it
  would mean being totally in kernel space.

> Alan

Re: silent semantic changes with reiser4

[Spam]

  

> On Iau, 2004-09-02 at 20:41, Spam wrote:
>> > It is trivial to implement this by looking inside the files. I.e., the way
>> > mc has done this for ages.
>> 
>>   Difference is that you can't do "locate" or "find" or "Search".. You
>>   would have to open the files in an archive-supporting application
>>   such as mc.

> And would you rather that logic was running swappable in shared library
> space or privileged and unswappable in kernel ?

  I would rather have it as a filesystem/vfs plugin that would allow
  all my programs to use the features the plugin gives, even if it
  would mean being totally in kernel space.

> Alan

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-09-02 at 21:07, Spam wrote:
> > And would you rather that logic was running swappable in shared library
> > space or privileged and unswappable in kernel ?
> 
>   I would rather have it as a filesystem/vfs plugin that would allow
>   all my programs to use the features the plugin gives, even if it
>   would mean being totally in kernel space.

Thats not what I asked. 

Re: silent semantic changes with reiser4

[Spam]

> On Iau, 2004-09-02 at 21:07, Spam wrote:
>> > And would you rather that logic was running swappable in shared library
>> > space or privileged and unswappable in kernel ?
>> 
>>   I would rather have it as a filesystem/vfs plugin that would allow
>>   all my programs to use the features the plugin gives, even if it
>>   would mean being totally in kernel space.

> Thats not what I asked.

  I failed to understand your question then. :(

  ~S

Re: silent semantic changes with reiser4

[Spam]

> On Iau, 2004-09-02 at 21:07, Spam wrote:
>> > And would you rather that logic was running swappable in shared library
>> > space or privileged and unswappable in kernel ?
>> 
>>   I would rather have it as a filesystem/vfs plugin that would allow
>>   all my programs to use the features the plugin gives, even if it
>>   would mean being totally in kernel space.

> Thats not what I asked.

  I failed to understand your question then. :(

  ~S

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > It is trivial to implement this by looking inside the files. I.e., the way
> > > mc has done this for ages.
> > 
> >   Difference is that you can't do "locate" or "find" or "Search".. You
> >   would have to open the files in an archive-supporting application
> >   such as mc.
> 
> And would you rather that logic was running swappable in shared library
> space or privileged and unswappable in kernel ?

Well, having it in kernel mean that cache gets actually shared between
different processes -- and that's usefull. With coda and similar tools
you can put most of the logic into userspace (kernel impact is 20
lines). You *will* want it to run priviledged, because otherwise
caches are useless.
								Pavel

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> >> option, enabled by default, to look inside archives.  If you tell it to
> >> look for a driver in a given directory it will also look inside .cab
> >> and .zip files.  It's extremely useful, I would imagine someone who uses
> >> XP a lot will come to expect this feature.
> 
> > It is trivial to implement this by looking inside the files. I.e., the way
> > mc has done this for ages.
> 
>   Difference is that you can't do "locate" or "find" or "Search".. You
>   would have to open the files in an archive-supporting application
>   such as mc.

You really need archive support in find. At the very least you need
option "enter archives" vs. "do not enter archives". Entering archives
automagically is seriously wrong.
								Pavel

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 15:49, Pavel Machek wrote:
> Hi!
> 
> > >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > >> option, enabled by default, to look inside archives.  If you tell it to
> > >> look for a driver in a given directory it will also look inside .cab
> > >> and .zip files.  It's extremely useful, I would imagine someone who uses
> > >> XP a lot will come to expect this feature.
> > 
> > > It is trivial to implement this by looking inside the files. I.e., the way
> > > mc has done this for ages.
> > 
> >   Difference is that you can't do "locate" or "find" or "Search".. You
> >   would have to open the files in an archive-supporting application
> >   such as mc.
> 
> You really need archive support in find. At the very least you need
> option "enter archives" vs. "do not enter archives". Entering archives
> automagically is seriously wrong.

But is it efficient to make every application that reads files have to
know how to get inside a tar file, just to read its contents?  That
seems like a massive duplication of effort.  Better to have the contents
accessible via a separate stream, in the same namespace.  Fix it once in
the kernel vs. fix it in umpteen apps.

The key point here is that the expressive power of the system is greatly
reduced by having a fragmented namespace.  Of course there are any
number of ways for an app to find out what is in a tar file, but
exporting all of that information in a unified namespace is nontrivial
and much more interesting.

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 15:49, Pavel Machek wrote:
> Hi!
> 
> > >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > >> option, enabled by default, to look inside archives.  If you tell it to
> > >> look for a driver in a given directory it will also look inside .cab
> > >> and .zip files.  It's extremely useful, I would imagine someone who uses
> > >> XP a lot will come to expect this feature.
> > 
> > > It is trivial to implement this by looking inside the files. I.e., the way
> > > mc has done this for ages.
> > 
> >   Difference is that you can't do "locate" or "find" or "Search".. You
> >   would have to open the files in an archive-supporting application
> >   such as mc.
> 
> You really need archive support in find. At the very least you need
> option "enter archives" vs. "do not enter archives". Entering archives
> automagically is seriously wrong.

But is it efficient to make every application that reads files have to
know how to get inside a tar file, just to read its contents?  That
seems like a massive duplication of effort.  Better to have the contents
accessible via a separate stream, in the same namespace.  Fix it once in
the kernel vs. fix it in umpteen apps.

The key point here is that the expressive power of the system is greatly
reduced by having a fragmented namespace.  Of course there are any
number of ways for an app to find out what is in a tar file, but
exporting all of that information in a unified namespace is nontrivial
and much more interesting.

Lee

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-09-02 at 21:01, Lee Revell wrote:
> But is it efficient to make every application that reads files have to
> know how to get inside a tar file, just to read its contents?  That
> seems like a massive duplication of effort.  Better to have the contents
> accessible via a separate stream, in the same namespace.  Fix it once in
> the kernel vs. fix it in umpteen apps.

Thats how you get yourself a non useful OS. Fix it in a library and
share it between the apps that care. Like say.. gnome-vfs2

Re: silent semantic changes with reiser4

[Chris Wedgwood]

On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote:

> Better to have the contents accessible via a separate stream, in the
> same namespace.  Fix it once in the kernel vs. fix it in umpteen
> apps.

This is ridiculous.  We have shared libraries, 99% of applications
manage to use libc for example --- this isn't that different.


  --cw

Re: silent semantic changes with reiser4

[Spam]

> On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote:

>> Better to have the contents accessible via a separate stream, in the
>> same namespace.  Fix it once in the kernel vs. fix it in umpteen
>> apps.

> This is ridiculous.  We have shared libraries, 99% of applications
> manage to use libc for example --- this isn't that different.

  Yes if that was the case then programs wouldn't brake if libc and
  its API was fixed. Sadly, I think it isn't.

  ~S


>   --cw

Re: silent semantic changes with reiser4

[Spam]

> On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote:

>> Better to have the contents accessible via a separate stream, in the
>> same namespace.  Fix it once in the kernel vs. fix it in umpteen
>> apps.

> This is ridiculous.  We have shared libraries, 99% of applications
> manage to use libc for example --- this isn't that different.

  Yes if that was the case then programs wouldn't brake if libc and
  its API was fixed. Sadly, I think it isn't.

  ~S


>   --cw

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > > >> option, enabled by default, to look inside archives.  If you tell it to
> > > >> look for a driver in a given directory it will also look inside .cab
> > > >> and .zip files.  It's extremely useful, I would imagine someone who uses
> > > >> XP a lot will come to expect this feature.
> > > 
> > > > It is trivial to implement this by looking inside the files. I.e., the way
> > > > mc has done this for ages.
> > > 
> > >   Difference is that you can't do "locate" or "find" or "Search".. You
> > >   would have to open the files in an archive-supporting application
> > >   such as mc.
> > 
> > You really need archive support in find. At the very least you need
> > option "enter archives" vs. "do not enter archives". Entering archives
> > automagically is seriously wrong.
> 
> But is it efficient to make every application that reads files have to
> know how to get inside a tar file, just to read its contents?  That

Application does not have to know how to handle tar/zip/etc, but it
has to make distinction between "enter archives" and "do not enter
archives". See uservfs.sf.net.
								Pavel

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 16:43, Pavel Machek wrote:
> Hi!
> 
> > > > >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > > > >> option, enabled by default, to look inside archives.  If you tell it to
> > > > >> look for a driver in a given directory it will also look inside .cab
> > > > >> and .zip files.  It's extremely useful, I would imagine someone who uses
> > > > >> XP a lot will come to expect this feature.
> > > > 
> > > > > It is trivial to implement this by looking inside the files. I.e., the way
> > > > > mc has done this for ages.
> > > > 
> > > >   Difference is that you can't do "locate" or "find" or "Search".. You
> > > >   would have to open the files in an archive-supporting application
> > > >   such as mc.
> > > 
> > > You really need archive support in find. At the very least you need
> > > option "enter archives" vs. "do not enter archives". Entering archives
> > > automagically is seriously wrong.
> > 
> > But is it efficient to make every application that reads files have to
> > know how to get inside a tar file, just to read its contents?  That
> 
> Application does not have to know how to handle tar/zip/etc, but it
> has to make distinction between "enter archives" and "do not enter
> archives". See uservfs.sf.net.

But how do you cache the information you had to look in the archive for
in a way that other apps can use it?  How do you synchronize access to
the cache and maintain cache coherency in userspace?

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 16:43, Pavel Machek wrote:
> Hi!
> 
> > > > >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > > > >> option, enabled by default, to look inside archives.  If you tell it to
> > > > >> look for a driver in a given directory it will also look inside .cab
> > > > >> and .zip files.  It's extremely useful, I would imagine someone who uses
> > > > >> XP a lot will come to expect this feature.
> > > > 
> > > > > It is trivial to implement this by looking inside the files. I.e., the way
> > > > > mc has done this for ages.
> > > > 
> > > >   Difference is that you can't do "locate" or "find" or "Search".. You
> > > >   would have to open the files in an archive-supporting application
> > > >   such as mc.
> > > 
> > > You really need archive support in find. At the very least you need
> > > option "enter archives" vs. "do not enter archives". Entering archives
> > > automagically is seriously wrong.
> > 
> > But is it efficient to make every application that reads files have to
> > know how to get inside a tar file, just to read its contents?  That
> 
> Application does not have to know how to handle tar/zip/etc, but it
> has to make distinction between "enter archives" and "do not enter
> archives". See uservfs.sf.net.

But how do you cache the information you had to look in the archive for
in a way that other apps can use it?  How do you synchronize access to
the cache and maintain cache coherency in userspace?

Lee

Re: silent semantic changes with reiser4

[Chris Wedgwood]

On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote:

> But how do you cache the information you had to look in the archive
> for in a way that other apps can use it?

~/.object-cache/ or whatever

> How do you synchronize access to the cache and maintain cache
> coherency in userspace?

coherency doesn't exist for normal files for the most part anyhow

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 16:49, Chris Wedgwood wrote:
> On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote:
> 
> > But how do you cache the information you had to look in the archive
> > for in a way that other apps can use it?
> 
> ~/.object-cache/ or whatever
> 

How are permissions handled?  If root lists the contents of a tar file
that is world readable, then joeuser comes along and does the same, can
joeuser sees the cached listing?  Would it depend on root's umask?

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 16:49, Chris Wedgwood wrote:
> On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote:
> 
> > But how do you cache the information you had to look in the archive
> > for in a way that other apps can use it?
> 
> ~/.object-cache/ or whatever
> 

How are permissions handled?  If root lists the contents of a tar file
that is world readable, then joeuser comes along and does the same, can
joeuser sees the cached listing?  Would it depend on root's umask?

Lee

Re: silent semantic changes with reiser4

[Chris Wedgwood]

On Thu, Sep 02, 2004 at 04:57:06PM -0400, Lee Revell wrote:

> How are permissions handled?  If root lists the contents of a tar
> file that is world readable, then joeuser comes along and does the
> same, can joeuser sees the cached listing?

everyone has their own cache i guess, works well enough elsewhere

also, am i the only person scared by the code complexity when it comes
to root and setuid code here?  sounds like a disaster waiting to
happen

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > Application does not have to know how to handle tar/zip/etc, but it
> > has to make distinction between "enter archives" and "do not enter
> > archives". See uservfs.sf.net.
> 
> But how do you cache the information you had to look in the archive for
> in a way that other apps can use it?  How do you synchronize access to
> the cache and maintain cache coherency in userspace?

Uservfs.sf.net.

Unlike alan, I do not think that "do it all in library" is good
idea. I put it in the userspace as "codafs" server, and let
applications see it as a regular filesystem.
							Pavel

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus, you are looking at this like a lieutenant instead of an HQ 
staffer, which is unusual of you.

You are saying, 1-2% simpler and better, no biggie, why work so hard to 
get it?

And we are saying, 1-2% simpler and better, times thousands of 
applications, wow! That's a lot!

Yes, changing cat to use openat() is no big deal. 1-2% additional coding 
cost for cat, who cares? But if you add 1-2% coding cost to every 
application which might access an attribute/stream/whatever, well, that 
totals more than the effort of authoring emacs. Can you see that?

Namespace simplifications and empowerments are force multipliers. They 
don't add to Linux like writing a new app adds to Linux, they add to 
Linux like adding percentage improvements to every app in Linux adds to 
Linux. HQ staffer types know that if you collect enough itty bitty 
little force multipliers, you win the war. Whether the troops have to 
spend five minutes a day polishing their shoes because their type of 
shoe needs polishing, that matters more than losing a tank battalion, 
when you are a major power. Linux is a major power.

This is a software engineering issue. We are discussing improvements 
that because they are diffused throughout the OS in their impact, seem 
like no big deal. But they are a big deal. One of the major determinants 
of an organization's efficiency is whether the management can recognize 
widely diffused inefficiencies as well as it can recognize focused 
inefficiencies of the same magnitude.

David is so very right about the usability issues. Usability is all 
about trivia. Usability battles feature no dragons, they feature armies 
of spiders. If you add one step to what a user needs to do to get to the 
data with an app, that matters a lot. That's doubling the spider army. 
Making a 2 step process to access data into a one step process to access 
the data halves the time cost of accessing the data. Lifestyle 
efficiency is mostly about reducing the cost of trivia, because trivia 
is most of what we do. Most people have as much interest in reading the 
man page for tar as you have in learning how to turn the hand crank to 
start your car. Just the look of "tar -xf" turns them away. Crypto-Geek 
gobbledy-gook is what it is. Let's value their time, there are a lot of 
them.

Hans

David Masover wrote:

>
>
> It's not about the kernel, it's about the interface. And see my other 
> mail:
> cat foo.zip/README
> less foo.zip/contents/bar.c
> is a lot easier than
> lynx http://google.com/search?q=zip
> emerge zip
> man zip
> unzip foo.zip
> cat bar.c
> which already assumes quite a lot of expertise.
>
>

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Hans Reiser wrote:
> 
> You are saying, 1-2% simpler and better, no biggie, why work so hard to 
> get it?
> 
> And we are saying, 1-2% simpler and better, times thousands of 
> applications, wow! That's a lot!

But would thousands care? Seriously?

For example, you could make just _one_ program support "openat()", and 
you'd get most of the advantages, with no possibility of _breaking_ any of 
thousands of applications..

I know, you've ignored the "runat" program (which is just a wrapper around
the openat() system call), but it _has_ been mentioned several times in 
this thread. Yes, you'd type a bit more to do

	runat file ls -l

instead of

	ls -l file/

but at least the openat/runat approach also works for directories, which 
does actually make it a lot more _generic_ than the "show in the regular 
filespace" approach. No special cases.

So your comparison isn't valid, because you're ignoring the people who
shout "runat" at you. You've also not ever actually answered about the
problems about directories with attributes.

		Linus

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes:

[...]

Linus> I know, you've ignored the "runat" program (which is just a
Linus> wrapper around the openat() system call), but it _has_ been
Linus> mentioned several times in this thread. Yes, you'd type a bit
Linus> more to do

Linus> 	runat file ls -l

Linus> instead of

Linus> 	ls -l file/

And how do I get GIMP (or insert your favourite image editing program
here) to edit xterm/icon.png without having to open a new instance of
GIMP?  runat isn't going to help here.  You'll have to modify every
editing program if you want that type of functionality.

Linus> but at least the openat/runat approach also works for
Linus> directories, which does actually make it a lot more _generic_
Linus> than the "show in the regular filespace" approach. No special
Linus> cases.

Linus> So your comparison isn't valid, because you're ignoring the
Linus> people who shout "runat" at you. You've also not ever actually
Linus> answered about the problems about directories with attributes.

All attributes should be stuffed in a specially-named subdirectory, like
Reiser4 puts all metadata in a metas subdirectory.  That's one solution.
The only trick here is picking a suitable name for the subdirectory that
is both user-friendly and highly unlikely to generate clashes.

Both of these I've mentioned before in this thread, and nobody has
brought up any objections yet.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Tue, 31 Aug 2004, Hans Reiser wrote:
>  
>
>>You are saying, 1-2% simpler and better, no biggie, why work so hard to 
>>get it?
>>
>>And we are saying, 1-2% simpler and better, times thousands of 
>>applications, wow! That's a lot!
>>    
>>
>
>But would thousands care? Seriously?
>  
>
>For example, you could make just _one_ program support "openat()", and 
>you'd get most of the advantages, with no possibility of _breaking_ any of 
>thousands of applications..
>
>I know, you've ignored the "runat" program (which is just a wrapper around
>the openat() system call), but it _has_ been mentioned several times in 
>this thread. Yes, you'd type a bit more to do
>
>	runat file ls -l
>
>instead of
>
>	ls -l file/
>
>but at least the openat/runat approach also works for directories, which 
>does actually make it a lot more _generic_ than the "show in the regular 
>filespace" approach. No special cases.
>  
>
So you are saying I can do runat emacs my-compound-document/stream-name? 
and in place of CTRL-X CTRL-F to open a file in emacs I type what to 
open an attribute?

I think it does not work Linus....


>So your comparison isn't valid, because you're ignoring the people who
>shout "runat" at you. You've also not ever actually answered about the
>problems about directories with attributes.
>  
>
Which problem specifically? I will list them and hope I don't miss what 
you want addressed. I won't claim to be sure I have the right answers 
for all of these (because VFS has not been my interest in the past), but 
these look feasible to me.;-)

Cycle detection:

We should either 1) make hard links only link to the file aspect of the 
file-directory duality, and persons who want to link to the directory 
aspect must use symlinks (best short term answer), or 2) ask Alexander 
Smith to help us with applying his cycle detection algorithm and gain 
the benefit of being able to hard link to directories (if it works well, 
best long term answer).

Ambiguity:

Linus said:

 - how do you handle the ambiguity of
	//attr/usr/bin/emacs/icon
   (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the 
   "emacs/icon" attribute on "/usr/bin").

If you want to avoid crowding the namespace with some piece of meta 
information that you want to be applicable to both directories and the 
files in them, and also widely used enough that you care about crowding 
the namespace, and unambiguous as to what level in the hierarchy it 
metas, then you put it into the metas subdirectory. There is no 
principled difference between a file that is a child of a directory and 
an attribute. The ambiguity is inherent in hiearchical semantics I 
think, and whether or not you have attributes you have this type of 
ambiguity.

Thus, you have either:

/usr/bin/metas/emacs/icon

or /usr/bin/emacs/metas/icon

and these are easily distinguished and unambiguous. This makes metas a 
reserved keyword, and there was a long discussion about whether to use 
"metas" or "..metas" as the reserved subdirectory name which I can 
summarize for you on request. There is also "pseudos", which contains 
files implementing special methods of special plugins. Feel free to 
invent more reserved subdirectories as needed, or to argue for fewer.


Seeing two trees, one with attrs and one without:

Linus said:

If you open a filename in some "secondary" tree (be it /proc or //attr or 
whatever) based on the filename in the primary one, you have two issues 
that you need to work out:

 - how do you handle a name change in the primary tree at the same time as 
   lookup
 - how do you handle the ambiguity of
	//attr/usr/bin/emacs/icon
   (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the 
   "emacs/icon" attribute on "/usr/bin").

The ambiguity can be handled by saying that attributes only have one
component (ie only the _last_ component of a lookup is the attribute
name). But the race between primary tree and secondary tree cannot be
handled in a normal name-space.

Are you sure that you have to have two trees in order to see two trees? 
That is, can you not refuse to follow or see a link depending on whether 
some flag is set and what the type of the link is?

Locking: 

Linus said:

let's say that you have filename "a" hard-linked to filename "b", 
and you have a directory structure of streams under there. So you have

	a/file1
	a/dir1/file2
	a/dir2/file3

and (through the hard-link with "b") you have aliases of all these same 
names available as "b/file1", "b/dir1/file2" etc).

Now, imagine that you have two processes doing

	mv a/dir1 a/dir2/newdir

and

	mv b/dir2 b/dir1/newdir

at the same time. Both of them MUST NOT SUCCEED, for pretty obvious 
reasons (you'd have moved two directories within each other, and now 
neither would be accessible any more).

How do you handle locking for this situation?

  

(Here is where my knowledge of VFS is shakiest.)  Making "hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks" seems to solve this.  If we want to use the Alexander Smith solution to cycle detection, and allow hard links to directories, then we must make sure that for the hard linked entity, there is a single lock for it somewhere (e.g. the directory inode) that can be taken.  If you create two dentries for the same object, then flag the dentries as needing to lock a lock in their shared inode and not in the dentry.  If you don't want to read in the inode from disk to do that, then create a pointer to a lock shared by them both.

Linus, is this a start on solving some of the issues?  

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Tue, 31 Aug 2004, Hans Reiser wrote:
>  
>
>>You are saying, 1-2% simpler and better, no biggie, why work so hard to 
>>get it?
>>
>>And we are saying, 1-2% simpler and better, times thousands of 
>>applications, wow! That's a lot!
>>    
>>
>
>But would thousands care? Seriously?
>  
>
>For example, you could make just _one_ program support "openat()", and 
>you'd get most of the advantages, with no possibility of _breaking_ any of 
>thousands of applications..
>
>I know, you've ignored the "runat" program (which is just a wrapper around
>the openat() system call), but it _has_ been mentioned several times in 
>this thread. Yes, you'd type a bit more to do
>
>	runat file ls -l
>
>instead of
>
>	ls -l file/
>
>but at least the openat/runat approach also works for directories, which 
>does actually make it a lot more _generic_ than the "show in the regular 
>filespace" approach. No special cases.
>  
>
So you are saying I can do runat emacs my-compound-document/stream-name? 
and in place of CTRL-X CTRL-F to open a file in emacs I type what to 
open an attribute?

I think it does not work Linus....


>So your comparison isn't valid, because you're ignoring the people who
>shout "runat" at you. You've also not ever actually answered about the
>problems about directories with attributes.
>  
>
Which problem specifically? I will list them and hope I don't miss what 
you want addressed. I won't claim to be sure I have the right answers 
for all of these (because VFS has not been my interest in the past), but 
these look feasible to me.;-)

Cycle detection:

We should either 1) make hard links only link to the file aspect of the 
file-directory duality, and persons who want to link to the directory 
aspect must use symlinks (best short term answer), or 2) ask Alexander 
Smith to help us with applying his cycle detection algorithm and gain 
the benefit of being able to hard link to directories (if it works well, 
best long term answer).

Ambiguity:

Linus said:

 - how do you handle the ambiguity of
	//attr/usr/bin/emacs/icon
   (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the 
   "emacs/icon" attribute on "/usr/bin").

If you want to avoid crowding the namespace with some piece of meta 
information that you want to be applicable to both directories and the 
files in them, and also widely used enough that you care about crowding 
the namespace, and unambiguous as to what level in the hierarchy it 
metas, then you put it into the metas subdirectory. There is no 
principled difference between a file that is a child of a directory and 
an attribute. The ambiguity is inherent in hiearchical semantics I 
think, and whether or not you have attributes you have this type of 
ambiguity.

Thus, you have either:

/usr/bin/metas/emacs/icon

or /usr/bin/emacs/metas/icon

and these are easily distinguished and unambiguous. This makes metas a 
reserved keyword, and there was a long discussion about whether to use 
"metas" or "..metas" as the reserved subdirectory name which I can 
summarize for you on request. There is also "pseudos", which contains 
files implementing special methods of special plugins. Feel free to 
invent more reserved subdirectories as needed, or to argue for fewer.


Seeing two trees, one with attrs and one without:

Linus said:

If you open a filename in some "secondary" tree (be it /proc or //attr or 
whatever) based on the filename in the primary one, you have two issues 
that you need to work out:

 - how do you handle a name change in the primary tree at the same time as 
   lookup
 - how do you handle the ambiguity of
	//attr/usr/bin/emacs/icon
   (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the 
   "emacs/icon" attribute on "/usr/bin").

The ambiguity can be handled by saying that attributes only have one
component (ie only the _last_ component of a lookup is the attribute
name). But the race between primary tree and secondary tree cannot be
handled in a normal name-space.

Are you sure that you have to have two trees in order to see two trees? 
That is, can you not refuse to follow or see a link depending on whether 
some flag is set and what the type of the link is?

Locking: 

Linus said:

let's say that you have filename "a" hard-linked to filename "b", 
and you have a directory structure of streams under there. So you have

	a/file1
	a/dir1/file2
	a/dir2/file3

and (through the hard-link with "b") you have aliases of all these same 
names available as "b/file1", "b/dir1/file2" etc).

Now, imagine that you have two processes doing

	mv a/dir1 a/dir2/newdir

and

	mv b/dir2 b/dir1/newdir

at the same time. Both of them MUST NOT SUCCEED, for pretty obvious 
reasons (you'd have moved two directories within each other, and now 
neither would be accessible any more).

How do you handle locking for this situation?

  

(Here is where my knowledge of VFS is shakiest.)  Making "hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks" seems to solve this.  If we want to use the Alexander Smith solution to cycle detection, and allow hard links to directories, then we must make sure that for the hard linked entity, there is a single lock for it somewhere (e.g. the directory inode) that can be taken.  If you create two dentries for the same object, then flag the dentries as needing to lock a lock in their shared inode and not in the dentry.  If you don't want to read in the inode from disk to do that, then create a pointer to a lock shared by them both.

Linus, is this a start on solving some of the issues?  

Re: silent semantic changes with reiser4

[Alexander G. M. Smith]

Hans Reiser wrote on Wed, 01 Sep 2004 00:22:49 -0700:
> Making "hard links only link to the file aspect of the file-directory
> duality, and persons who want to link to the directory aspect must use
> symlinks" seems to solve this.  If we want to use the Alexander Smith
> solution to cycle detection, and allow hard links to directories, then
> we must make sure that for the hard linked entity, there is a single
> lock for it somewhere (e.g. the directory inode) that can be taken.

Oddly enough that's what I did in my RAM file system.  No matter how
the file/directory node is found (the path can lead through different
parents), it has a unique ID and a corresponding single lock for it.

Rename/delete goes and locks all the nodes that might change parents
and/or children, then it does its operations (though it has to watch
out for deadlocks, which are unavoidable when two rename/deletes both
sneak up on the same file by different paths).

- Alex

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes:

Linus> For example, you could make just _one_ program support
Linus> "openat()", and you'd get most of the advantages, with no
Linus> possibility of _breaking_ any of thousands of applications..

I'm starting to think that even if we had openat, it wouldn't solve
much.  Maybe I'm just not imaginative enough, but I still see problems.
Let's pretend, just for the sake of argument, that every program all of
a sudden magically was modified to support openat, which is a big
assumption.  But we'll make that assumption so that we can ignore _that_
problem for a second.

The question still remains, how do I tell the thing to open a stream?
If cat knows about openat, how do I tell it to cat foo.txt, stream
bar?  I obviously can't do it using a normal filename, because if I
could do that, we could just forget about the openat business.  The
only way I can think of is to give cat a flag that tells it what stream
to open: "cat --stream bar foo.txt".  Ugly.  It also makes it
impossible to do redirection, because you can't pass flags to "<".

And before someone yells "runat" at me (which, btw, won't work with
redirection either), consider larger programs like GIMP or emacs.  Let's
look at GIMP first.  I've already got an instance of GIMP open, and I
don't want to waste memory with another instance.  How do I tell it to
open the icon.png stream of the file xterm?  When I go to File | Open,
up pops a dialog that only shows real files.  Trying to design a dialog
box that will show real files and substreams, and not confuse the users
is not going to be fun.  Please don't ask the GTK/GNOME developers to
design another file selection dialog.  There were enough people whining
the last round.  If they need to add more complexity, we'll have even
more whining.

Now let's look at emacs.  I want to open up the description.txt stream
of the file xterm.  So I do "C-x C-f", and it prompts me on the status
bar for the file name.  What do I type?  Well, description.txt doesn't
live in the normal namespace, so the only way around that is for emacs
to ask for the filename, and then ask for the stream name.  That's
extra-annoying when I usually just want to open just the normal stream,
and that second prompt just makes more work for me.

Linus> I know, you've ignored the "runat" program (which is just a
Linus> wrapper around the openat() system call), but it _has_ been
Linus> mentioned several times in this thread. Yes, you'd type a bit
Linus> more to do

Linus> 	runat file ls -l

Linus> instead of

Linus> 	ls -l file/

Might not be just a "bit more" typing, because you lose handy things
like tab completion.  Say I want to "cat" file foo.txt, stream bar.  Or
was that baz?  Instead of being able to type
  cat foo/ba<tab>
and have the filename magically come up, I need to
  runat foo ls ba*
  runat foo cat baz
(which also requires more thinking, since I need to think about how to
formulate the query, read the output, and write the new command, and so
is much much slower than being able to mindlessly hit the <tab> key.)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Alan Cox]

On Mer, 2004-09-01 at 06:50, Hans Reiser wrote:
> Yes, changing cat to use openat() is no big deal. 1-2% additional coding 
> cost for cat, who cares? 

Large cost on its own. The FSF would almost certainly reject such a
change until someone had written portable emulation of the feature for 
every other platform which would mean vendor patches which would
essentially mean it wouldn't happen.

I know I rarely agree with Hans but I think he's right on this one, it
has to work with the existing open() API.

Alan

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pavel Machek wrote:
| Hi!
|
|
|>| uservfs does
|>|
|>| cd foo.deb#uar
|>cd foo.deb/ar
|>| vs.
|>| cd foo.deb#udeb
|>cd foo.deb/deb
|>
|>and why would you want that, instead of just:
|>cd foo.deb	# for the ar
|>dpkg -i foo.deb	# for the deb
|
|
| Because I want to see contents of that .deb package, nicely parsed?
|
|
|>Just want to extract the tar file?  Maybe something like
|>cat foo.tgz/gunzip
|>In which case (of course) foo.tgz/gunzip has exactly the same directory
|>contents as foo.tgz
|
|
| Yes, that would work.
|
|
|>In fact, for just about any syntax anyone could suggest, I can't really
|>see why you can't just replace all weird symbols with a slash and a
|>symbol.  Instead of
|>	foo.tgz#utar
|>you have
|>	foo.tgz/#/utar
|>Only difference is, some things which used to require special tools can
|>now be serviced by less than what's in busybox.
|
|
| That would work, too. I do not get your comment about busybox.

Busybox is a bare minimum of unix tools, yet still includes things like
tar/bz2.  With such a system, you can use the most basic (sub-GNU)
versions of tools like cp, cat, sh, ls, and so on, to do much more
complex operations.  What's more, you can see everything you can do to a
file just by looking at it.  I can certainly imagine doing
	cat foo.tgz/metas/README
rather than a google search for tgz, followed by a "man tar" and "man
gzip". Assuming I know nothing about tar/gzip files, it's a lot easier
as a new user to have one common interface for everything -- including
docs.  Fewer commands to memorize, faster to learn new stuff.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUha3gHNmZLgCUhAQKRJw/7BUEm41gKZGoVj7gOkcUVe/SRdTs9WtTL
jkjCX8Zzr/1nnAKTHLqNJEiAJ68Ndf+dRhDfL5y6dDHhOy/Dx+38SzsWz+dQnMJ0
Fiq5fFDO/DTd0UPecXYPhBrtVeCdfjyyENhgRgJud6D8F39qvLWrZFUsIkjAA3lj
nzZ/48jRPDWT/mF6JJmv0RfVvNtMWh9HUcW/i0+n4WZU1yFp6jyiF5ExCOP84z3x
yXQ4Q9hnjgt52MDJTrGsCDvfCSQscQilxvXbl0iaozquIhG9Jz1XNYpij2XUygFi
RDc1LA1k74VvbWahlkdfbTI+3P4qeMYGBblmfwXVDRwW0i4NH7X0/6xcnr1QjaK/
f/7EHBz0wnkIG4OpcGWzu9ik4WOR0WXapKrd938XC/mG+WzIluJlQsxG1A2HoRqg
R66E/mVMVvHJWnJjrW6s10WIemPiKnuXgeIVPifsw81DLXlZ3wxjlZU3iuo7xked
O6wNDFQBWQR8kv+EMDIt3gtbUdB+5KpxGR+Yli0AjxEdpdk3oZTfe5XQiNdJhcl5
xrcomVKXGvR96IlNUwHy1EsQk5LySMl101vJIS4z6i7lAIM5CQ1d6eOZrPp/DBOP
Rw0KoQaiPd3SKNMgoyWSKQL6h85i1PRUN62WH0T2NRU1fZ3EazpiRZr+fuaFqFT9
Wk2lkDvm5yw=
=ShWo
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pavel Machek wrote:
| Hi!
|
|
|>| uservfs does
|>|
|>| cd foo.deb#uar
|>cd foo.deb/ar
|>| vs.
|>| cd foo.deb#udeb
|>cd foo.deb/deb
|>
|>and why would you want that, instead of just:
|>cd foo.deb	# for the ar
|>dpkg -i foo.deb	# for the deb
|
|
| Because I want to see contents of that .deb package, nicely parsed?
|
|
|>Just want to extract the tar file?  Maybe something like
|>cat foo.tgz/gunzip
|>In which case (of course) foo.tgz/gunzip has exactly the same directory
|>contents as foo.tgz
|
|
| Yes, that would work.
|
|
|>In fact, for just about any syntax anyone could suggest, I can't really
|>see why you can't just replace all weird symbols with a slash and a
|>symbol.  Instead of
|>	foo.tgz#utar
|>you have
|>	foo.tgz/#/utar
|>Only difference is, some things which used to require special tools can
|>now be serviced by less than what's in busybox.
|
|
| That would work, too. I do not get your comment about busybox.

Busybox is a bare minimum of unix tools, yet still includes things like
tar/bz2.  With such a system, you can use the most basic (sub-GNU)
versions of tools like cp, cat, sh, ls, and so on, to do much more
complex operations.  What's more, you can see everything you can do to a
file just by looking at it.  I can certainly imagine doing
	cat foo.tgz/metas/README
rather than a google search for tgz, followed by a "man tar" and "man
gzip". Assuming I know nothing about tar/gzip files, it's a lot easier
as a new user to have one common interface for everything -- including
docs.  Fewer commands to memorize, faster to learn new stuff.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUha3gHNmZLgCUhAQKRJw/7BUEm41gKZGoVj7gOkcUVe/SRdTs9WtTL
jkjCX8Zzr/1nnAKTHLqNJEiAJ68Ndf+dRhDfL5y6dDHhOy/Dx+38SzsWz+dQnMJ0
Fiq5fFDO/DTd0UPecXYPhBrtVeCdfjyyENhgRgJud6D8F39qvLWrZFUsIkjAA3lj
nzZ/48jRPDWT/mF6JJmv0RfVvNtMWh9HUcW/i0+n4WZU1yFp6jyiF5ExCOP84z3x
yXQ4Q9hnjgt52MDJTrGsCDvfCSQscQilxvXbl0iaozquIhG9Jz1XNYpij2XUygFi
RDc1LA1k74VvbWahlkdfbTI+3P4qeMYGBblmfwXVDRwW0i4NH7X0/6xcnr1QjaK/
f/7EHBz0wnkIG4OpcGWzu9ik4WOR0WXapKrd938XC/mG+WzIluJlQsxG1A2HoRqg
R66E/mVMVvHJWnJjrW6s10WIemPiKnuXgeIVPifsw81DLXlZ3wxjlZU3iuo7xked
O6wNDFQBWQR8kv+EMDIt3gtbUdB+5KpxGR+Yli0AjxEdpdk3oZTfe5XQiNdJhcl5
xrcomVKXGvR96IlNUwHy1EsQk5LySMl101vJIS4z6i7lAIM5CQ1d6eOZrPp/DBOP
Rw0KoQaiPd3SKNMgoyWSKQL6h85i1PRUN62WH0T2NRU1fZ3EazpiRZr+fuaFqFT9
Wk2lkDvm5yw=
=ShWo
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > One of the big potential uses for file-as-directory is to go inside
> > archive files, ELF files, .iso files and so on in a convenient way.
> 
> Arguably this belongs in userspace --- and people have put it there.
> 

uservfs.sf.net, using existing coda hooks. There are 2+
very similar projects.


Advantage is we can do ftpfs, too. Having ftp done by

> /ftp
cd /ftp/sunsite

would be too ugly for reiser4 plugins... Or not?
-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: silent semantic changes with reiser4

[Chris Wright]

* viro@parcelfarce.linux.theplanet.co.uk (viro@parcelfarce.linux.theplanet.co.uk) wrote:
> On Thu, Aug 26, 2004 at 01:11:52AM +0100, Jamie Lokier wrote:
> > Is this a problem if we treat entering a file-as-directory as crossing
> > a mount point (i.e. like auto-mounting)?
> 
> Yes - mountpoints can't be e.g. unlinked.

Could it be essentially MNT_DETACH'd?

Re: silent semantic changes with reiser4

[Mikaël Cluseau]

Hello,

This thread is very very very very big. I think lots of people spend 
hours talking here. Isn't it time for a summary of all this please ?

(for people like me who are quite busy and try to follow... but also for 
the health of the debate)

Thank!

Mikael.

Re: silent semantic changes with reiser4

[Hans Reiser]

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
>  
>
>>On Wed, 25 Aug 2004, Christoph Hellwig wrote:
>>    
>>
>>>For one thing _I_ didn't decide about xattrs anyway.  And I still
>>>haven't seen a design from you on -fsdevel how you try to solve the
>>>problems with files as directories.
>>>      
>>>
>>Hey, files-as-directories are one of my pet things, so I have to side with 
>>Hans on this one. I think it just makes sense. A hell of a lot more sense 
>>than xattrs, anyway, since it allows scripts etc standard tools to touch 
>>the attributes.
>>
>>It's the UNIX way.
>>    
>>
>
>Not if you allow link(2) on them.  And not if you design and market your
>stuff as a general-purpose backdoor into kernel. 
>
What backdoor?  Please spell it out.  Plugins are not dynamically 
loadable. 

> Note how *EVERY* *DAMN*
>*OPERATION* is made possible to override by "plugins".  Which is the reason
>for deadlocks in question, BTW.
>
>Don't fool yourself - that's what Hans is selling.  Target market: ISV.
>  
>
Hunh?  No, target market is hackers who like to spend a weekend dreaming 
up funky new kinds of files.  One guy (Jason Holt, clever guy) came up 
with an idea for write only files for which even root cannot read the 
parts of the file written prior to the time root was achieved, because 
the encryption key is changed in a forward computable only direction 
with every write, and the start key is kept on another computer.  Lots 
of folks will have plugins I would never dream of.   Think of photoshop 
plugins, and what plugins did for photoshop.  Same thing will happen to 
reiser4 next year (or earlier).

>Marketed product: a set of hooks, the wider the better, no matter how
>little sense it makes.  The reason for doing that outside of core kernel:
>bypassing any review and being able to control the product being sold (see
>above).
>
>Shame that it got an actual filesystem mixed in with the marketing plans
>and general insanity...
>
>
>  
>
I am one of those free software hippy-child anarchists who thinks that 
random people should come up with ideas and contribute them.  You 
understand me well.;-)

Re: silent semantic changes with reiser4

[Hans Reiser]

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
>  
>
>>On Wed, 25 Aug 2004, Christoph Hellwig wrote:
>>    
>>
>>>For one thing _I_ didn't decide about xattrs anyway.  And I still
>>>haven't seen a design from you on -fsdevel how you try to solve the
>>>problems with files as directories.
>>>      
>>>
>>Hey, files-as-directories are one of my pet things, so I have to side with 
>>Hans on this one. I think it just makes sense. A hell of a lot more sense 
>>than xattrs, anyway, since it allows scripts etc standard tools to touch 
>>the attributes.
>>
>>It's the UNIX way.
>>    
>>
>
>Not if you allow link(2) on them.  And not if you design and market your
>stuff as a general-purpose backdoor into kernel. 
>
What backdoor?  Please spell it out.  Plugins are not dynamically 
loadable. 

> Note how *EVERY* *DAMN*
>*OPERATION* is made possible to override by "plugins".  Which is the reason
>for deadlocks in question, BTW.
>
>Don't fool yourself - that's what Hans is selling.  Target market: ISV.
>  
>
Hunh?  No, target market is hackers who like to spend a weekend dreaming 
up funky new kinds of files.  One guy (Jason Holt, clever guy) came up 
with an idea for write only files for which even root cannot read the 
parts of the file written prior to the time root was achieved, because 
the encryption key is changed in a forward computable only direction 
with every write, and the start key is kept on another computer.  Lots 
of folks will have plugins I would never dream of.   Think of photoshop 
plugins, and what plugins did for photoshop.  Same thing will happen to 
reiser4 next year (or earlier).

>Marketed product: a set of hooks, the wider the better, no matter how
>little sense it makes.  The reason for doing that outside of core kernel:
>bypassing any review and being able to control the product being sold (see
>above).
>
>Shame that it got an actual filesystem mixed in with the marketing plans
>and general insanity...
>
>
>  
>
I am one of those free software hippy-child anarchists who thinks that 
random people should come up with ideas and contribute them.  You 
understand me well.;-)

Re: silent semantic changes with reiser4

[Matt Mackall]

On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> 
> 
> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> > 
> > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > haven't seen a design from you on -fsdevel how you try to solve the
> > problems with files as directories.
> 
> Hey, files-as-directories are one of my pet things, so I have to side with 
> Hans on this one. I think it just makes sense. A hell of a lot more sense 
> than xattrs, anyway, since it allows scripts etc standard tools to touch 
> the attributes.
> 
> It's the UNIX way.

I thought the UNIX way is "everything's a file", not "everything's a
directory".

> Will it potentially break something? Sure. Do we care? Me, I'll take that 
> kind of extension _any_ day over xattrs, that are fundamentally flawed in 
> my opinion and totally useless.

There's always the option that they're both broken.

-- 
Mathematics is the supreme nostalgia of our time.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Wed, 25 Aug 2004, Matt Mackall wrote:
> > 
> > It's the UNIX way.
> 
> I thought the UNIX way is "everything's a file", not "everything's a
> directory".

It really was. Directories were historically largely just files too, 
although with the special "lookup" operation. 

Historic unix didn't have readdir/rmdir/mkdir/rename or really much _any_
special directory handling. Directories were just files, and you read them 
like files.

Of course, even in that early unix, "directories" were very much a 
reality even apart from the fact that they happened to be implemented 
pretty much like files. Nobody has ever claimed that the UNIX way is 
"everything is _one_ file", after all ;)

> > Will it potentially break something? Sure. Do we care? Me, I'll take that 
> > kind of extension _any_ day over xattrs, that are fundamentally flawed in 
> > my opinion and totally useless.
> 
> There's always the option that they're both broken.

Yes. Highly likely. However, something like that _does_ end up what a 
Windows fileserver wants. IOW, even if it's broken, _something_ is likely 
forced on us by that nasty thing we call "real users". Damn them.

		Linus

Re: silent semantic changes with reiser4

[Mikulas Patocka]

On Wed, 25 Aug 2004, Linus Torvalds wrote:

> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> >
> > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > haven't seen a design from you on -fsdevel how you try to solve the
> > problems with files as directories.
>
> Hey, files-as-directories are one of my pet things, so I have to side with
> Hans on this one. I think it just makes sense. A hell of a lot more sense
> than xattrs, anyway, since it allows scripts etc standard tools to touch
> the attributes.
>
> It's the UNIX way.
>
> And yes, the semantics can _easily_ be solved in very unixy ways.
>
> One way to solve it is to just realize that a final slash at the end
> implies pretty strongly that you want to treat it as a directory. So what
> you do is:
>
>  - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
>  - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
>
> Problem solved. Very user-friendly, and very intuitive.

Stupid question: who will use it? And why?

Anyone can write an userspace library, that implements function
set_attribute(char *file, char *attribute, char *value), that creates
directory ".attr/file" in file's directory and stores attribute there.
(and you can get list of attributes from shell too:
ls `echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/'`
). There's no need to add extra functionality to kernel and filesystem.

Advantage:
- you don't add bloat to kernel or filesystem
- you don't need to teach tar/cp -a/mc about attributes
- you won't lose attributes after editing file in vim (it creates another
file and renames it over original one)

> Will it potentially break something? Sure. Do we care? Me, I'll take that
> kind of extension _any_ day over xattrs, that are fundamentally flawed in
> my opinion and totally useless. The argument that applications like "tar"
> won't understand the file-as-directory thing is _flawed_, since legacy
> apps won't understand xattrs either.

The only way xattrs are useful is that backup/restore software doesn't
have to know about every filesystem with it's specific attributes and
every magic ioctl for setting them. Instead it can save/restore
filesystem-specific attributes without understanding what do they mean.
However there's no need why application should use them. And no
application does.

I can't imagine anyone shipping an application with "this app requires
reiser4" prerequisite. Why should anyone use it if he can store attributes
in ".attr" directory or whereever and make the application work on any OS
and any filesystem?

Mikulas

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Mikulas Patocka wrote:
> 
> Stupid question: who will use it? And why?
> 
> Anyone can write an userspace library, that implements function
> set_attribute(char *file, char *attribute, char *value), that creates
> directory ".attr/file" in file's directory and stores attribute there.
> (and you can get list of attributes from shell too:
> ls `echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/'`
> ). There's no need to add extra functionality to kernel and filesystem.

..and the above is, roughly, what I understand samba etc falls back on.

The problem ends up being that the above isn't in any way safe from people 
moving files around (oops, where did those attributes go?) nor does it 
have any consistency guarantees. So it only works well if _one_ 
application does this, and that application follows all the locking rules.

Is it enough? It may have to be.

> The only way xattrs are useful is that backup/restore software doesn't
> have to know about every filesystem with it's specific attributes and
> every magic ioctl for setting them. Instead it can save/restore
> filesystem-specific attributes without understanding what do they mean.
> However there's no need why application should use them. And no
> application does.

If no application does, then why back them up? Why implement them in the 
first place?

In other words - some apps obviously do want to use the. Sadly.

		Linus

Re: silent semantic changes with reiser4

[Mikulas Patocka]

> > The only way xattrs are useful is that backup/restore software doesn't
> > have to know about every filesystem with it's specific attributes and
> > every magic ioctl for setting them. Instead it can save/restore
> > filesystem-specific attributes without understanding what do they mean.
> > However there's no need why application should use them. And no
> > application does.
>
> If no application does, then why back them up? Why implement them in the
> first place?
>
> In other words - some apps obviously do want to use the. Sadly.

You can add more functionality to filesystem and use xattrs to control it.
For example:
- acls
- compress file
- encrypt file (copy user's password into task_struct and use it to
encrypt his files)
- preallocate file in 4MB contignuous chunks, becuase it needs real time
multimedia access
- sync/append-only/immutable
etc.
However there's no need why an application should care whether the file is
compressed, whether it has acls, or so. And applications don't.

And I think this is the only legitimate use for xattrs. Who else uses them
except samba? I don't see how reiser4's hybrids would help.

Mikulas

Re: silent semantic changes with reiser4

[Hans Reiser]

Streams are quite ugly.  However, if you decompose streams into all of 
the little pieces that are needed to emulate them, the pieces are quite 
nice.

For instance, inheriting stat data from a common parent is nice, and 
inheritance is nice, and being able to cat dirname/pseudos/cat and get a 
concatenation of all of the files is nice, and being able to cat 
dirname/pseudos/tar and get an archive of the directory is nice, and, 
well, if you decompose all of the features of streams into little 
features you get a bunch of fun little features much nicer than streams.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Streams are quite ugly.  However, if you decompose streams into all of 
the little pieces that are needed to emulate them, the pieces are quite 
nice.

For instance, inheriting stat data from a common parent is nice, and 
inheritance is nice, and being able to cat dirname/pseudos/cat and get a 
concatenation of all of the files is nice, and being able to cat 
dirname/pseudos/tar and get an archive of the directory is nice, and, 
well, if you decompose all of the features of streams into little 
features you get a bunch of fun little features much nicer than streams.

Hans

Re: silent semantic changes with reiser4

[Jamie Lokier]

Hans Reiser wrote:
> being able to cat dirname/pseudos/cat and get a 
> concatenation of all of the files is nice, and being able to cat 
> dirname/pseudos/tar and get an archive of the directory is nice

Yes.  Being able to cd into filename.tar.gz and filename.iso is also
nice, but all of these features should be supported by the VFS
generically, not in any specific filesystem, and there should be a
hook to invoke the various fun filesystem-independent handlers by name.

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote:
> Hans Reiser wrote:
> > being able to cat dirname/pseudos/cat and get a 
> > concatenation of all of the files is nice, and being able to cat 
> > dirname/pseudos/tar and get an archive of the directory is nice
> 
> Yes.  Being able to cd into filename.tar.gz and filename.iso is also
> nice, but all of these features should be supported by the VFS
> generically, not in any specific filesystem, and there should be a
> hook to invoke the various fun filesystem-independent handlers by name.

It doesn't belong into the kernel at all.  If at all it belongs into a
userspace filesystems, but even in that case the magic detection of
which one to use is kinda hard.  You absoutely don't want to hardcode
file formats in the kernel.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > > being able to cat dirname/pseudos/cat and get a 
> > > concatenation of all of the files is nice, and being able to cat 
> > > dirname/pseudos/tar and get an archive of the directory is nice
> > 
> > Yes.  Being able to cd into filename.tar.gz and filename.iso is also
> > nice, but all of these features should be supported by the VFS
> > generically, not in any specific filesystem, and there should be a
> > hook to invoke the various fun filesystem-independent handlers by name.
> 
> It doesn't belong into the kernel at all.  If at all it belongs into a
> userspace filesystems, but even in that case the magic detection of
> which one to use is kinda hard.  You absoutely don't want to hardcode
> file formats in the kernel.

Oh, I agree userspace should be involved.

-- Jamie

Re: silent semantic changes with reiser4

[Spam]

> On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote:
>> Hans Reiser wrote:
>> > being able to cat dirname/pseudos/cat and get a 
>> > concatenation of all of the files is nice, and being able to cat 
>> > dirname/pseudos/tar and get an archive of the directory is nice
>> 
>> Yes.  Being able to cd into filename.tar.gz and filename.iso is also
>> nice, but all of these features should be supported by the VFS
>> generically, not in any specific filesystem, and there should be a
>> hook to invoke the various fun filesystem-independent handlers by name.

> It doesn't belong into the kernel at all.  If at all it belongs into a
> userspace filesystems, but even in that case the magic detection of
> which one to use is kinda hard.  You absoutely don't want to hardcode
> file formats in the kernel.

  Do  you  mean  user-level  file  system  as  a  VFS  handled by user
  applications,  or  a  intermediate  file  system  layer  between any
  application  and  the  real  file  system?  The latter would be good
  enough as it would still be transparent to the applications.

  ~S
  

Re: silent semantic changes with reiser4

[Spam]

> On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote:
>> Hans Reiser wrote:
>> > being able to cat dirname/pseudos/cat and get a 
>> > concatenation of all of the files is nice, and being able to cat 
>> > dirname/pseudos/tar and get an archive of the directory is nice
>> 
>> Yes.  Being able to cd into filename.tar.gz and filename.iso is also
>> nice, but all of these features should be supported by the VFS
>> generically, not in any specific filesystem, and there should be a
>> hook to invoke the various fun filesystem-independent handlers by name.

> It doesn't belong into the kernel at all.  If at all it belongs into a
> userspace filesystems, but even in that case the magic detection of
> which one to use is kinda hard.  You absoutely don't want to hardcode
> file formats in the kernel.

  Do  you  mean  user-level  file  system  as  a  VFS  handled by user
  applications,  or  a  intermediate  file  system  layer  between any
  application  and  the  real  file  system?  The latter would be good
  enough as it would still be transparent to the applications.

  ~S
  

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Mikulas Patocka wrote:
> On Wed, 25 Aug 2004, Linus Torvalds wrote:

> > One way to solve it is to just realize that a final slash at the end
> > implies pretty strongly that you want to treat it as a directory. So what
> > you do is:
> 
> Stupid question: who will use it? And why?

I've got a stupid question too.  How do you back up these
things ?

If your backup program reads them as a file and restores
them as a file, you might lose your directory-inside-the-file
magic.

If your backup program dives into the file despite stat()
saying it's a file and you restore your backup, how are the
"file is a file" semantics preserved ?

Obviously this is something that needs to be sorted out at
the VFS layer.  A filesystem specific backup and restore
program isn't desirable, if only because then there'd be
no way for Hans's users to switch to reiser5 in 2010 ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Mikulas Patocka wrote:
> On Wed, 25 Aug 2004, Linus Torvalds wrote:

> > One way to solve it is to just realize that a final slash at the end
> > implies pretty strongly that you want to treat it as a directory. So what
> > you do is:
> 
> Stupid question: who will use it? And why?

I've got a stupid question too.  How do you back up these
things ?

If your backup program reads them as a file and restores
them as a file, you might lose your directory-inside-the-file
magic.

If your backup program dives into the file despite stat()
saying it's a file and you restore your backup, how are the
"file is a file" semantics preserved ?

Obviously this is something that needs to be sorted out at
the VFS layer.  A filesystem specific backup and restore
program isn't desirable, if only because then there'd be
no way for Hans's users to switch to reiser5 in 2010 ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Helge Hafting]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Mikulas Patocka wrote:
>  
>
>>On Wed, 25 Aug 2004, Linus Torvalds wrote:
>>    
>>
>
>  
>
>>>One way to solve it is to just realize that a final slash at the end
>>>implies pretty strongly that you want to treat it as a directory. So what
>>>you do is:
>>>      
>>>
>>Stupid question: who will use it? And why?
>>    
>>
>
>I've got a stupid question too.  How do you back up these
>things ?
>
>If your backup program reads them as a file and restores
>them as a file, you might lose your directory-inside-the-file
>magic.
>
>If your backup program dives into the file despite stat()
>saying it's a file and you restore your backup, how are the
>"file is a file" semantics preserved ?
>
>  
>
>Obviously this is something that needs to be sorted out at
>the VFS layer.  A filesystem specific backup and restore
>program isn't desirable, if only because then there'd be
>no way for Hans's users to switch to reiser5 in 2010 ;)
>
>  
>
Sure, this sort of thing must be sorted out at the VFS layer.
And a backup program working on such a filesystem
will need to know that something can be a file, a directory - or both.

So an old "tar" won't get this right as it will assume that an object
is either file or directory.  The change to get it right won't be
that big - just notice that an object is both, then backup the
ordinary file contents as usual, before recursing into the
directory it also provides and backup stuff there as usual.

The resulting .tar can of course only be unpacked properly
on a fs supporting file-as-directory, similiar to how a .tar of
a fs with links only will unpack properly on a fs supporting links.

I don't see much problems for userland.  Old apps will keep working,
as the new features is a superset.  Those who care about
file-as-directory extras will provide patches for "tar" and friends,
after that the extras become useable.

Helge Hafting

Re: silent semantic changes with reiser4

[Paul Jackson]

Helge wrote:
> So an old "tar" won't get this right as it will assume that an object
> is either file or directory.

There are many backup apps, not just one.  I've written a few myself,
none of which will ever be worthy of notice.  The sourceforge
Topic.System.Archiving.Backup lists 335 projects at present.

I find the idea that most backup tools and scripts will silently
stop working correctly to be pretty scary.  

And then there's archiving, installation, distribution, administration,
emulation, file system and partition managers, and on and on.

===

I wonder if we can make this "modal" somehow.

The one consistency I see is that apps that want the "enhanced" view
need to ask for it, somehow.  It is the new views of the data that are
being added - let the app announce to the kernel (usually via
specialized code in some shared library that the app is using to get the
alternate views) that either per-task, or per-file descriptor, it is to
see the "enhanced" view, as a side affect of trying to access it.

Old stuff, or even new stuff that is content to work with the "classic"
view that a file is a single data stream, and that directories only
have pathnames, not data, would by default see that view, and see
_all_ the data, presented somehow in that view, perhaps as additional
files with magic names.

This still leaves the breakage that such tools don't know, and don't
preserve, the magic linkage between such magic files.  But that is
much less of an issue, in my view.  Programs such as backups that are
manipulating the files of apps they know nothing about already have
to presume that all the files are important in inscrutable ways, and
just be careful to preserve or copy or backup all of them.

Yeah - I realize that there will be a few followups denouncing modal
architectures.  I might even agree with some of them.

If this were easy, it would have been done years ago.

The onus should be on the new stuff to request the enhanced view,
rather than on the old dogs to learn new tricks.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Jamie Lokier]

Paul Jackson wrote:
> I find the idea that most backup tools and scripts will silently
> stop working correctly to be pretty scary.  

Too late.  We have xattrs already; many programs don't store them.

-- Jamie

Re: silent semantic changes with reiser4

[Paul Jackson]

> Too late.  We have xattrs already; many programs don't store them.

If by your "too late" you mean we can stop worrying about any more
breakage of file system utilities, because there exists an example
in which some were already broken, then you are absolutely wrong.

Just because we caused some breakage doesn't give us license to cause
even more.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Helge Hafting]

Paul Jackson wrote:

>Helge wrote:
>  
>
>>So an old "tar" won't get this right as it will assume that an object
>>is either file or directory.
>>    
>>
>
>There are many backup apps, not just one.  I've written a few myself,
>none of which will ever be worthy of notice.  The sourceforge
>Topic.System.Archiving.Backup lists 335 projects at present.
>
>I find the idea that most backup tools and scripts will silently
>stop working correctly to be pretty scary.  
>  
>
They won't stop working, they will merely not support
the new features.  That is only a problem if
you actually use those features.  If all you have
is plain files and plain directories - no problem.  Not
if files-as-directories are implemented right.

Helge Hafting

Re: silent semantic changes with reiser4

[Helge Hafting]

Paul Jackson wrote:

>Helge wrote:
>  
>
>>So an old "tar" won't get this right as it will assume that an object
>>is either file or directory.
>>    
>>
>
>There are many backup apps, not just one.  I've written a few myself,
>none of which will ever be worthy of notice.  The sourceforge
>Topic.System.Archiving.Backup lists 335 projects at present.
>
>I find the idea that most backup tools and scripts will silently
>stop working correctly to be pretty scary.  
>  
>
They won't stop working, they will merely not support
the new features.  That is only a problem if
you actually use those features.  If all you have
is plain files and plain directories - no problem.  Not
if files-as-directories are implemented right.

Helge Hafting

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>
>If your backup program dives into the file despite stat()
>saying it's a file and you restore your backup, how are the
>"file is a file" semantics preserved ?
>
>Obviously this is something that needs to be sorted out at
>the VFS layer.
>
It needs to be sorted out, whether it is sorted out at the VFS layer is 
unimportant.

>  A filesystem specific backup and restore
>program isn't desirable, if only because then there'd be
>no way for Hans's users to switch to reiser5 in 2010 ;)
>
>  
>
It might be that we need a filenameA/metas/backup method for all of our 
file plugins, which if cat'd gives a set of instructions which if 
executed are adequate for restoring filenameA.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>
>If your backup program dives into the file despite stat()
>saying it's a file and you restore your backup, how are the
>"file is a file" semantics preserved ?
>
>Obviously this is something that needs to be sorted out at
>the VFS layer.
>
It needs to be sorted out, whether it is sorted out at the VFS layer is 
unimportant.

>  A filesystem specific backup and restore
>program isn't desirable, if only because then there'd be
>no way for Hans's users to switch to reiser5 in 2010 ;)
>
>  
>
It might be that we need a filenameA/metas/backup method for all of our 
file plugins, which if cat'd gives a set of instructions which if 
executed are adequate for restoring filenameA.

Hans

Re: silent semantic changes with reiser4

[Lars Marowsky-Bree]

On 2004-08-26T01:40:29,
   Hans Reiser <reiser@namesys.com> said:

> >Obviously this is something that needs to be sorted out at
> >the VFS layer.
> >
> It needs to be sorted out, whether it is sorted out at the VFS layer is 
> unimportant.

So what exactly is wrong with sorting it out at the VFS layer, and why
do you _insist_ on sorting it in the reiserfs4 core? I'm missing
something, please fill me in on the details.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering	   \\\  /// 
SUSE Labs, Research and Development \honk/ 
SUSE LINUX AG - A Novell company     \\// 

Re: silent semantic changes with reiser4

[Paul Jackson]

Rik van Riel writes:
> If your backup program reads them as a file and restores
> them as a file, you might lose your directory-inside-the-file
> magic.

Encode the magic in the names, by stealing a bit of the existing
filename space to encode it.

Such works pretty well as part of the magic to map long filenames
into DOS 8.3 names on my FAT partitions.

Apps linked with the appropriate Windows library see nice fancy
long names.

The rest of the world, including DOS apps and my Unix backup
scripts, see the primitive 8.3 names, including one or a few
extra files per directory, which are nothing special to them.

So long as these other apps don't presume to know that they can
keep some of the files in an apps directory, and drop others, then
it works well enough.  And no self-respecting general purpose
backup program is going to presume such knowledge anyway.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Wed, 25 Aug 2004, Christoph Hellwig wrote:
>  
>
>>For one thing _I_ didn't decide about xattrs anyway.  And I still
>>haven't seen a design from you on -fsdevel how you try to solve the
>>problems with files as directories.
>>    
>>
>
>Hey, files-as-directories are one of my pet things, so I have to side with 
>Hans on this one. I think it just makes sense. A hell of a lot more sense 
>than xattrs, anyway, since it allows scripts etc standard tools to touch 
>the attributes.
>
>It's the UNIX way.
>
>And yes, the semantics can _easily_ be solved in very unixy ways.
>
>One way to solve it is to just realize that a final slash at the end 
>implies pretty strongly that you want to treat it as a directory. So what 
>you do is:
>
> - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
> - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
>
>Problem solved. Very user-friendly, and very intuitive.
>
>Will it potentially break something? Sure. Do we care? Me, I'll take that 
>kind of extension _any_ day over xattrs, that are fundamentally flawed in 
>my opinion and totally useless. The argument that applications like "tar" 
>won't understand the file-as-directory thing is _flawed_, since legacy 
>apps won't understand xattrs either.
>
>Oh, add a O_NOXATTRS flag to force a path lookup to only use regular
>directories, the same way we have O_NOFOLLOW and friends. That allows
>people to see the difference, if they care (ie a file server might decide
>that it doesn't want to expose things like this).
>  
>
I think we should require people to care enough to supply an O_NOMETAS 
flag to see the difference.

>I never liked the xattr stuff. It makes little sense, and is totally 
>useless for 99.9999% of everything. I still don't see the point of it, 
>except for samba. Ugly.
>
>		Linus
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Wed, 25 Aug 2004, Christoph Hellwig wrote:
>  
>
>>For one thing _I_ didn't decide about xattrs anyway.  And I still
>>haven't seen a design from you on -fsdevel how you try to solve the
>>problems with files as directories.
>>    
>>
>
>Hey, files-as-directories are one of my pet things, so I have to side with 
>Hans on this one. I think it just makes sense. A hell of a lot more sense 
>than xattrs, anyway, since it allows scripts etc standard tools to touch 
>the attributes.
>
>It's the UNIX way.
>
>And yes, the semantics can _easily_ be solved in very unixy ways.
>
>One way to solve it is to just realize that a final slash at the end 
>implies pretty strongly that you want to treat it as a directory. So what 
>you do is:
>
> - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
> - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
>
>Problem solved. Very user-friendly, and very intuitive.
>
>Will it potentially break something? Sure. Do we care? Me, I'll take that 
>kind of extension _any_ day over xattrs, that are fundamentally flawed in 
>my opinion and totally useless. The argument that applications like "tar" 
>won't understand the file-as-directory thing is _flawed_, since legacy 
>apps won't understand xattrs either.
>
>Oh, add a O_NOXATTRS flag to force a path lookup to only use regular
>directories, the same way we have O_NOFOLLOW and friends. That allows
>people to see the difference, if they care (ie a file server might decide
>that it doesn't want to expose things like this).
>  
>
I think we should require people to care enough to supply an O_NOMETAS 
flag to see the difference.

>I never liked the xattr stuff. It makes little sense, and is totally 
>useless for 99.9999% of everything. I still don't see the point of it, 
>except for samba. Ugly.
>
>		Linus
>
>
>  
>

Re: silent semantic changes with reiser4

[Michael Halcrow]

[-- Attachment #1: Type: text/plain, Size: 5340 bytes --]

If I may chime in here...

On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> > 
> > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > haven't seen a design from you on -fsdevel how you try to solve
> > the problems with files as directories.
> 
> Hey, files-as-directories are one of my pet things, so I have to
> side with Hans on this one. I think it just makes sense. A hell of a
> lot more sense than xattrs, anyway, since it allows scripts etc
> standard tools to touch the attributes.
> 
> It's the UNIX way.

This is an issue that directly affects work I am doing in extended
cryptfs:

http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55
http://halcrow.us/~mhalcrow/ols2004.pdf
http://halcrow.us/~mhalcrow/ols_cryptfs.sxi

The basic idea is that the cryptographic context for every file is
correlated with the individual file via xattr's.  A file is a unit of
data that should, as it stands, contain all the information requisite
for the encrypting filesystem layer to transparently decrypt (and
encrypt, when the file is written to).  This allows for a key->file
granularity, as opposed to a key->block device (dm-crypt) or a
key->mount point (CFS) granularity.

My grand vision is to have a policy that determines whether or not the
encrypted version of the file or the decrypted version of the file is
read, dependent on whether or not the file is leaving the security
domain (the storage device under the control of the currently running
kernel).  For example, if the ``cp'' command is copying a file from a
filesystem mounted from /dev/hda1 to a filesystem mounted from
/dev/fd0, then the policy would indicate that (unless otherwise noted
in the .cryptfsrc file in the root of the filesystem mounted from
/dev/fd0, which might also contain the default security context for
that filesystem or directory - like whose public keys should be used
to encrypt the symmetric key for data) the file is leaving the
security domain, and the encrypted contents of the file should be
given to cp.  Same with mutt reading an email attachment (as opposed
to, say, .muttrc, where, more likely than not, the unencrypted version
is wanted).

The goal is to enable an ``encrypted by default'' policy, in which
files on the storage devices are independent encrypted units that
remain encrypted until an application that actually needs to see the
decrypted contents opens them.  Then the encryption and decryption is
done transparently by the fs layer, as long as the user has the right
keys.  Extended attributes seem like a natural way to store this
context.

Once you consider that you can have a crypto context for each file,
you can start doing other neat tricks, like keyed hashes over extents
within the file, to allow for dynamic integrity verification during
the read.  If an offset of 1.5 gigabytes into a 2-gigabyte has been
tampered with, then that tampering will be caught when that portion of
the file is read; you don't have to verify the hash of the entire
2-gigabyte file at the time of the open.  Of course, this would very
rapidly overrun the available xattr storage size.  And so to
realistically implement something like this, some new underlying file
format is in order.

In any case, the issue of userspace applications supporting extended
attributes is key to the viability of this approach.  If cp, uuencode,
tar, or what not do not preserve the extended attributes, then the
crypto context is lost, and the file is unreadable.  So the $64,000
question is, just how committed is the community to this whole concept
of extended attributes?  From this point, should I assume that good
xattr support is forthcoming, or should I abandon the idea of using
xattr's for this altogether?

One solution I've been kicking around is to make cryptfs
GnuPG-compatible.  Not only would this eliminate the need to store
some of the crypto context in the xattr set, but it would also
preserve the crypto context with apps that don't know about xattr's,
and it would be possible for users who are not running cryptfs to read
the files with gpg.  Keyed hashes over extents would be doable if
GnuPG allowed for opaque data blobs in the file that gpg would just
ignore when decrypting the file (gnupg-dev list had technical issues
last time I tried to post these ideas to it - any gpg guys around that
can comment on this?).

> I never liked the xattr stuff. It makes little sense, and is totally 
> useless for 99.9999% of everything. I still don't see the point of it, 
> except for samba. Ugly.

If xattr's wind up getting supported by a certain critical mass of
applications, then they are somewhat useful for me, although, as
currently implemented, are insufficient for what I really need (keyed
hashes over extents require too much space).

BTW, early this week I migrated cryptfs over to use David Howell's new
keyring, which is working out nicely.

Mike
.___________________________________________________________________.
                         Michael A. Halcrow                          
       Security Software Engineer, IBM Linux Technology Center       
GnuPG Fingerprint: 05B5 08A8 713A 64C1 D35D  2371 2D3C FDDA 3EB6 601D

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Hans Reiser]

Michael Halcrow wrote:

> This is an issue that directly affects work I am doing in extended
>
>cryptfs:
>
>http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55
>http://halcrow.us/~mhalcrow/ols2004.pdf
>http://halcrow.us/~mhalcrow/ols_cryptfs.sxi
>
>  
>
Reiser4 has an encryption plugin that will ship sometime this year.  You 
might want to talk to edward@namesys.com about it.

Re: silent semantic changes with reiser4

[Hans Reiser]

Michael Halcrow wrote:

> This is an issue that directly affects work I am doing in extended
>
>cryptfs:
>
>http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55
>http://halcrow.us/~mhalcrow/ols2004.pdf
>http://halcrow.us/~mhalcrow/ols_cryptfs.sxi
>
>  
>
Reiser4 has an encryption plugin that will ship sometime this year.  You 
might want to talk to edward@namesys.com about it.

Re: silent semantic changes with reiser4

[Alan Cox]

On Mer, 2004-08-25 at 21:22, Linus Torvalds wrote:
>  - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
>  - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
> 
> Problem solved. Very user-friendly, and very intuitive.

Breaks bash and almost all other file completion 8)

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Wed, Aug 25, 2004 at 10:08:59PM +0200, Christoph Hellwig wrote:
> On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote:
> > You ignored everything I said during the discussion of xattrs about how 
> > there is no need to have attributes when you can just have files and 
> > directories, and that xattrs reflected a complete ignorance of name 
> > space design principles.
> 
> Actually in most of the discussion you simply didn't participate.  While
> xattrs might not be the nicest interface they have the advantag of not
> breaking the SuS assumption of what directories vs files are, and they
> do not break the Linux O_DIRECTORY semantics that are defined and need
> to solve real-world races either.

Reiser4 may have a mount option for whoose who like or have to use traditional
O_DIRECTORY semantics.  There would be no /metas under non-directories, if user
wants that.

[ ... ]

-- 
Alex.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote:
> Reiser4 may have a mount option for whoose who like or have to use traditional
> O_DIRECTORY semantics.  There would be no /metas under non-directories, if user
> wants that.

Again, O_DIRECTORY was added to solve a real-world race, not just for
the sake of it.  If you really want to add that option I'll research the
CAN number for you so you can named it after that - or just call it -o
insecure directly.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > Reiser4 may have a mount option for whoose who like or have to use
> > traditional O_DIRECTORY semantics.  There would be no /metas under
> > non-directories, if user wants that.
> 
> Again, O_DIRECTORY was added to solve a real-world race, not just for
> the sake of it.  If you really want to add that option I'll research the
> CAN number for you so you can named it after that - or just call it -o
> insecure directly.

man open(2) explains that O_DIRECTORY is used by opendir() to prevent
blocking when opening pipes and certain devices*, and should only by
used by opendir (of course it isn't only used by opendir, as it's a
handy optimisation).

In fact O_DIRECTORY is also used by Glibc to optimise away stat()
before and fstat() after calls.

An O_NODEVICE flag would be equally secure, and more generally useful.

It's important that device nodes cannot be opened when O_DIRECTORY is
set.  This is compatible with reiser4 file-as-directory semantics, but
I don't know if reiser4 actually implements this.  If it does (and it
should) then there is no device blocking problem.

That leaves only the optimisation of fstat() in opendir().

But that begs the question: do we want opendir() to succeed on a reiser4 file?

It's up to us to decide if we like that semantic, or prefer a
different one such as the one I described before (path search enters
file-as-directory, but opendir() directly on it fails), or the Sun
syscalls someone mentioned.

-- Jamie


* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on
many systems.  It's best if you can avoid opening devices at all though.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>Christoph Hellwig wrote:
>  
>
>>>Reiser4 may have a mount option for whoose who like or have to use
>>>traditional O_DIRECTORY semantics.  There would be no /metas under
>>>non-directories, if user wants that.
>>>      
>>>
>>Again, O_DIRECTORY was added to solve a real-world race, not just for
>>the sake of it.  If you really want to add that option I'll research the
>>CAN number for you so you can named it after that - or just call it -o
>>insecure directly.
>>    
>>
>
>man open(2) explains that O_DIRECTORY is used by opendir() to prevent
>blocking when opening pipes and certain devices*, and should only by
>used by opendir (of course it isn't only used by opendir, as it's a
>handy optimisation).
>
>In fact O_DIRECTORY is also used by Glibc to optimise away stat()
>before and fstat() after calls.
>
>An O_NODEVICE flag would be equally secure, and more generally useful.
>
>It's important that device nodes cannot be opened when O_DIRECTORY is
>set.  This is compatible with reiser4 file-as-directory semantics, but
>I don't know if reiser4 actually implements this.  If it does (and it
>should) then there is no device blocking problem.
>
>That leaves only the optimisation of fstat() in opendir().
>
>But that begs the question: do we want opendir() to succeed on a reiser4 file?
>  
>
Yes, we do.  Why not?  Not is more complex.....

>It's up to us to decide if we like that semantic, or prefer a
>different one such as the one I described before (path search enters
>file-as-directory, but opendir() directly on it fails), or the Sun
>syscalls someone mentioned.
>
>-- Jamie
>
>
>* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on
>many systems.  It's best if you can avoid opening devices at all though.
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>Christoph Hellwig wrote:
>  
>
>>>Reiser4 may have a mount option for whoose who like or have to use
>>>traditional O_DIRECTORY semantics.  There would be no /metas under
>>>non-directories, if user wants that.
>>>      
>>>
>>Again, O_DIRECTORY was added to solve a real-world race, not just for
>>the sake of it.  If you really want to add that option I'll research the
>>CAN number for you so you can named it after that - or just call it -o
>>insecure directly.
>>    
>>
>
>man open(2) explains that O_DIRECTORY is used by opendir() to prevent
>blocking when opening pipes and certain devices*, and should only by
>used by opendir (of course it isn't only used by opendir, as it's a
>handy optimisation).
>
>In fact O_DIRECTORY is also used by Glibc to optimise away stat()
>before and fstat() after calls.
>
>An O_NODEVICE flag would be equally secure, and more generally useful.
>
>It's important that device nodes cannot be opened when O_DIRECTORY is
>set.  This is compatible with reiser4 file-as-directory semantics, but
>I don't know if reiser4 actually implements this.  If it does (and it
>should) then there is no device blocking problem.
>
>That leaves only the optimisation of fstat() in opendir().
>
>But that begs the question: do we want opendir() to succeed on a reiser4 file?
>  
>
Yes, we do.  Why not?  Not is more complex.....

>It's up to us to decide if we like that semantic, or prefer a
>different one such as the one I described before (path search enters
>file-as-directory, but opendir() directly on it fails), or the Sun
>syscalls someone mentioned.
>
>-- Jamie
>
>
>* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on
>many systems.  It's best if you can avoid opening devices at all though.
>
>
>  
>

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 12:54:09AM +0100, Jamie Lokier wrote:
> Christoph Hellwig wrote:
> > > Reiser4 may have a mount option for whoose who like or have to use
> > > traditional O_DIRECTORY semantics.  There would be no /metas under
> > > non-directories, if user wants that.
> > 
> > Again, O_DIRECTORY was added to solve a real-world race, not just for
> > the sake of it.  If you really want to add that option I'll research the
> > CAN number for you so you can named it after that - or just call it -o
> > insecure directly.
> 
> man open(2) explains that O_DIRECTORY is used by opendir() to prevent
> blocking when opening pipes and certain devices*, and should only by
> used by opendir (of course it isn't only used by opendir, as it's a
> handy optimisation).
> 
> In fact O_DIRECTORY is also used by Glibc to optimise away stat()
> before and fstat() after calls.
> 
> An O_NODEVICE flag would be equally secure, and more generally useful.

It would prevent the open device issue but you'd need another syscall.
Anyways, this is an existing API and chaging it now would suddenly
making everything that relies on it for that purpose insecure, so it's
completely out of option.

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote:
>  
>
>>Reiser4 may have a mount option for whoose who like or have to use traditional
>>O_DIRECTORY semantics.  There would be no /metas under non-directories, if user
>>wants that.
>>    
>>
>
>Again, O_DIRECTORY was added to solve a real-world race, not just for
>the sake of it. 
>
Can you supply more details and we will try to reply concretely?  Thanks.

> If you really want to add that option I'll research the
>CAN number for you so you can named it after that - or just call it -o
>insecure directly.
>
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote:
>  
>
>>Reiser4 may have a mount option for whoose who like or have to use traditional
>>O_DIRECTORY semantics.  There would be no /metas under non-directories, if user
>>wants that.
>>    
>>
>
>Again, O_DIRECTORY was added to solve a real-world race, not just for
>the sake of it. 
>
Can you supply more details and we will try to reply concretely?  Thanks.

> If you really want to add that option I'll research the
>CAN number for you so you can named it after that - or just call it -o
>insecure directly.
>
>
>
>  
>

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:43:59AM -0700, Hans Reiser wrote:
> >Again, O_DIRECTORY was added to solve a real-world race, not just for
> >the sake of it. 
> >
> Can you supply more details and we will try to reply concretely?  Thanks.

It's really userland programming 101 :)

 you need to fstat after open to really check that no one swtiched the
 path below you to a fifo or device, O_DIRECTORY gets that directly in
 the open call, with the additional benefit of making sure you don't
 get any of the sideeffects at all that would happen if you're opening
 a device file.

 the current reiser4 semantics break that and as soon as you're having a
 world-writeable (e.g. /tmp) dir on it and someone is doing an opendir
 on it he's lost.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
>  you need to fstat after open to really check that no one swtiched the
>  path below you to a fifo or device, O_DIRECTORY gets that directly in
>  the open call, with the additional benefit of making sure you don't
>  get any of the sideeffects at all that would happen if you're opening
>  a device file.
> 
>  the current reiser4 semantics break that and as soon as you're having a
>  world-writeable (e.g. /tmp) dir on it and someone is doing an opendir
>  on it he's lost.

How does the current reiser4 semantics break that?

In a reiser4 filesystem, a file _is_ a directory.
opendir() is supposed to succeed on it.

There's bound to be some security issue, but I'm not sure what you're
getting at with /tmp.  What sort of sort of security problem arises
with a world-writeable directory such as /tmp, that cannot arise with
the standard fs semantics?

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 02:48:12PM +0100, Jamie Lokier wrote:
> >  the current reiser4 semantics break that and as soon as you're having a
> >  world-writeable (e.g. /tmp) dir on it and someone is doing an opendir
> >  on it he's lost.
> 
> How does the current reiser4 semantics break that?
> 
> In a reiser4 filesystem, a file _is_ a directory.
> opendir() is supposed to succeed on it.
> 
> There's bound to be some security issue, but I'm not sure what you're
> getting at with /tmp.  What sort of sort of security problem arises
> with a world-writeable directory such as /tmp, that cannot arise with
> the standard fs semantics?

Actually you are right on that issue because it would open the
device/fifo as directory and not device/fifo (in fact I'd had to look at
the code again to see whether they actually do this only for files or
also for special files)

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > There's bound to be some security issue, but I'm not sure what you're
> > getting at with /tmp.  What sort of sort of security problem arises
> > with a world-writeable directory such as /tmp, that cannot arise with
> > the standard fs semantics?
> 
> Actually you are right on that issue because it would open the
> device/fifo as directory and not device/fifo (in fact I'd had to look at
> the code again to see whether they actually do this only for files or
> also for special files)

Are you saying that with reiser4, you can open a device or fifo with
O_DIRECTORY?

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 05:06:38PM +0100, Jamie Lokier wrote:
> Christoph Hellwig wrote:
> > > There's bound to be some security issue, but I'm not sure what you're
> > > getting at with /tmp.  What sort of sort of security problem arises
> > > with a world-writeable directory such as /tmp, that cannot arise with
> > > the standard fs semantics?
> > 
> > Actually you are right on that issue because it would open the
> > device/fifo as directory and not device/fifo (in fact I'd had to look at
> > the code again to see whether they actually do this only for files or
> > also for special files)
> 
> Are you saying that with reiser4, you can open a device or fifo with
> O_DIRECTORY?

That's what I thought, but as far as I can follow the code this is not
actually true.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Christoph Hellwig wrote:
>
> > Are you saying that with reiser4, you can open a device or fifo with
> > O_DIRECTORY?
> 
> That's what I thought, but as far as I can follow the code this is not
> actually true.

It should be possible to do, though. There's nothing really different in 
making the "default" (unnamed) fork be a special device or a fifo.

And it would be perfectly ok for O_DIRECTORY to open such a file, as long
as it opens the directory branch, not the special device.

I advocated (long ago) something like this for /dev handling, just because
I think it would make sense to have

	/dev/hda	<- special file
	/dev/hda/part1	<- partition 1 (aka /dev/hda1)

which just seems like a very obvious and intuitive interface to me. Of 
course, we have so much legacy in /dev that there's no real point to doing 
this, but it's still an appealing approach, I think.

But I do take Al's concerns seriously. I like the notion of supporting
"containers", but there are undoubtedly serious issues in the notion. I
don't strictly know exactly _how_ to implement it sanely (I can talk about
using the vfsmnt structure all I like, but the fact is, it's a different
thing from a normal mount, and there may be serious problems indeed
there).

Still, I really do like the idea of merging the notion of file and
directory into one notion of "container". I absolutely _detest_ files with
internal structure that tools have to know about (ie I hate seeing all
those embedded formats that I can't use "grep" on - MIME being one case).
I'd much rather see a "group of files"  and a "file with a grouping of
information".

(Now, flattening that "group of files" is obviously needed for serial 
protocols, so I think MIME/tar/xxxx are fine for _transporting_ data, but 
I'm saying that outside of transport I really prefer a "collection of 
files" approach).

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> And it would be perfectly ok for O_DIRECTORY to open such a file, as long
> as it opens the directory branch, not the special device.

What about:

    cd /dev/into_directory_branch/
    run_setuid_program

        -> calls pwd
           pwd opens("."), (".."), ("../..") etc.
 
        -> the setuid program thus ends up opening a device or fifo,
           when it does pwd's path walk.  Yes it could use the getcwd
           syscall, but some programs do their own path walk.

> I advocated (long ago) something like this for /dev handling, just because
> I think it would make sense to have
> 
> 	/dev/hda	<- special file
> 	/dev/hda/part1	<- partition 1 (aka /dev/hda1)
> 
> which just seems like a very obvious and intuitive interface to me. Of 
> course, we have so much legacy in /dev that there's no real point to doing 
> this, but it's still an appealing approach, I think.

It also fits the container idea very well:

        /dev/hda/part1/ <- the filesystem inside partition 1

That's not a contrivance, it's what I'd expect to happen if
/dev/hda is an ordinary file containing a disk image:

 	image.bin	 <- hard disk image (a regular file)
 	image.bin/part1	 <- partition 1
 	image.bin/part1/ <- the filesystem inside partition 1

That's assuming there's a format handler which recognises that image.

-- Jamie

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
>
>     run_setuid_program
> 
>         -> calls pwd
>            pwd opens("."), (".."), ("../..") etc.

Ehh.. Not only does pwd not do that..

(hint: there's a getcwd() system call)

>         -> the setuid program thus ends up opening a device or fifo,
>            when it does pwd's path walk.  Yes it could use the getcwd
>            syscall, but some programs do their own path walk.

.. but even if it did that, it should use O_DIRECTORY when it did so. If 
it doesn't, it's broken.

So no, it would _not_ open the device or fifo when it did so.

The fact is, anything that expects to open a directory should already be 
opening it with O_DIRECTORY.

That said, ".." and "." are special already inside the kernel, and it 
migth be worth making them automatically imply O_DIRECTORY, since nothing 
else makes sense anyway. That would fix the case where somebody uses ".." 
_without_ using O_DIRECTORY. 

> It also fits the container idea very well:
> 
>         /dev/hda/part1/ <- the filesystem inside partition 1

I don't think you can do that. The kernel has no idea how to mount the
filesystem.

If it's already mounted somewhere else, that's a different issue.  
Although it might be mounted in several places (as a bind mount) with
different writability, I guess, so even then it might be "interesting".

		Linus

Re: silent semantic changes with reiser4

[viro]

On Thu, Aug 26, 2004 at 10:29:45AM -0700, Linus Torvalds wrote:
> >         /dev/hda/part1/ <- the filesystem inside partition 1
> 
> I don't think you can do that. The kernel has no idea how to mount the
> filesystem.

... and what's more, fundamental thing for mount is fs type, what to
do with device(s) (if any) and any other arguments is the filesystem's
business.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> >         -> the setuid program thus ends up opening a device or fifo,
> >            when it does pwd's path walk.  Yes it could use the getcwd
> >            syscall, but some programs do their own path walk.
> 
> .. but even if it did that, it should use O_DIRECTORY when it did so. If 
> it doesn't, it's broken.

Didn't someone just say that O_DIRECTORY will succeed on a device,
precisely because opendir() is supposed to succeed on the device?

I'm not sure.  Let me refresh my memory.

Linus Torvalds wrote:
> And it would be perfectly ok for O_DIRECTORY to open such a file, as long
> as it opens the directory branch, not the special device.

Oh, I see.  You're right.  ".." sees the path of directory branches.

> > It also fits the container idea very well:
> > 
> >         /dev/hda/part1/ <- the filesystem inside partition 1
> 
> I don't think you can do that. The kernel has no idea how to mount the
> filesystem.

It is not the kernel which decides.  The filesystem containing
/dev/hda/part1 opens "the directory branch".  If the filesystem is
suitably configured, it's exactly the same as opening a .zip or ELF
file: it asks userspace for help.  Userspace helps by either mounting
it, or creating a container virtual directory -- in exactly the same
way it helps with .zip and ELF files by creating a container virtual
directory for those.  It would mount it if the kernel supports the fs,
and create a virtual directory if the kernel doesn't support the fs
and userspace does.

This is exactly the same as when you cd into a filesystem image: the
helper mounts (this time with a lookback file mount) if the kernel
supports the image fs, otherwise the helper may be able to decode it
in userspace.

By the way, these mechanisms could allow some of the old filesystems
that nobody uses any more to be removed from the kernel altogether and
kept in an archive of userspace-only supported filesystems.  What do
you think of that idea?

> If it's already mounted somewhere else, that's a different issue.  
> Although it might be mounted in several places (as a bind mount) with
> different writability, I guess, so even then it might be "interesting".

The obvious implementation has the userspace helper just mounting it,
end of story.  If the mount command fails, it fails.  Much like autofs.

-- Jamie

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
>
> > .. but even if it did that, it should use O_DIRECTORY when it did so. If 
> > it doesn't, it's broken.
> 
> Didn't someone just say that O_DIRECTORY will succeed on a device,
> precisely because opendir() is supposed to succeed on the device?

It will succeed on the _name_ of the device, but it won't open the device 
node. It would open the _directory_ node associated with that name.

Think of it this way: a pathname always points to one "container". That 
container is either a directory or a single node - and O_DIRECTORY ends up 
being the thing that chooses between them.

> > I don't think you can do that. The kernel has no idea how to mount the
> > filesystem.
> 
> It is not the kernel which decides.  The filesystem containing
> /dev/hda/part1 opens "the directory branch".

But that filesystem cannot know what the _other_ filesystem configurations 
are. And that's what you'd have to have to mount.

> The obvious implementation has the userspace helper just mounting it,
> end of story.  If the mount command fails, it fails.  Much like autofs.

Yes, that would work, but it's of questionable use. If you want autofs, 
then just _use_ autofs.

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> > It is not the kernel which decides.  The filesystem containing
> > /dev/hda/part1 opens "the directory branch".
> 
> But that filesystem cannot know what the _other_ filesystem configurations 
> are. And that's what you'd have to have to mount.

At which point, userspace comes in.  Who knows what userspace chooses.

> > The obvious implementation has the userspace helper just mounting it,
> > end of story.  If the mount command fails, it fails.  Much like autofs.
> 
> Yes, that would work, but it's of questionable use. If you want autofs, 
> then just _use_ autofs.

I don't mean to imply it's useful behaviour: only that it seems to
happen by default if you don't do anything special to prevent it, if
you see what I mean.

It does seem useful to have _files_ mounted automagically (using
loopback devices): for example, cd into a .iso file and it would be
nice to see the contents.  At this point, deciding on the preferred
-EBUSY or not behaviour becomes relevant, as does deciding whether the
.iso in this example is viewed using the kernel's isofs code, or using
a userspace helper in the same way as we'd use one to browse a .zip file.

-- Jamie

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 10:29:45AM -0700, Linus Torvalds wrote:
> 
> 
> On Thu, 26 Aug 2004, Jamie Lokier wrote:
> >
> >     run_setuid_program
> > 
> >         -> calls pwd
> >            pwd opens("."), (".."), ("../..") etc.
> 
> Ehh.. Not only does pwd not do that..
> 
> (hint: there's a getcwd() system call)
> 
> >         -> the setuid program thus ends up opening a device or fifo,
> >            when it does pwd's path walk.  Yes it could use the getcwd
> >            syscall, but some programs do their own path walk.
> 
> .. but even if it did that, it should use O_DIRECTORY when it did so. If 
> it doesn't, it's broken.
> 
> So no, it would _not_ open the device or fifo when it did so.
> 
> The fact is, anything that expects to open a directory should already be 
> opening it with O_DIRECTORY.

reiser4 files-as-dirs do not depend on that.

A file behaves as directory if one calls i_op->lookup() or f_op->readdir().  I
am not sure that O_DIRECTORY should be a switch. 

> That said, ".." and "." are special already inside the kernel, and it 
> migth be worth making them automatically imply O_DIRECTORY, since nothing 
> else makes sense anyway. That would fix the case where somebody uses ".." 
> _without_ using O_DIRECTORY. 
> 
> > It also fits the container idea very well:
> > 
> >         /dev/hda/part1/ <- the filesystem inside partition 1
> 
> I don't think you can do that. The kernel has no idea how to mount the
> filesystem.
> 
> If it's already mounted somewhere else, that's a different issue.  
> Although it might be mounted in several places (as a bind mount) with
> different writability, I guess, so even then it might be "interesting".
> 
> 		Linus

-- 
Alex.

Re: silent semantic changes with reiser4

[Giuliano Pochini]

On 26-Aug-2004 Linus Torvalds wrote:

> I advocated (long ago) something like this for /dev handling, just because
> I think it would make sense to have
>
>       /dev/hda        <- special file
>       /dev/hda/part1  <- partition 1 (aka /dev/hda1)

This breaks the r4 semantics if I understood it correctly. Because
/partN are not simply associated to the file, they are part of the
file. ie. when I modify /dev/hda I also change /dev/hda/partN and
vice-versa. I don't see any pratical problem, though.


> Still, I really do like the idea of merging the notion of file and
> directory into one notion of "container". I absolutely _detest_ files with
> internal structure that tools have to know about (ie I hate seeing all
> those embedded formats that I can't use "grep" on - MIME being one case).
> I'd much rather see a "group of files"  and a "file with a grouping of
> information".

You're actually looking for a database with a legacy fs-like interface :)


--
Giuliano.

Re: silent semantic changes with reiser4

[Giuseppe Bilotta]

Giuliano Pochini wrote:
> On 26-Aug-2004 Linus Torvalds wrote:
> 
> > I advocated (long ago) something like this for /dev handling, just because
> > I think it would make sense to have
> >
> >       /dev/hda        <- special file
> >       /dev/hda/part1  <- partition 1 (aka /dev/hda1)
> 
> This breaks the r4 semantics if I understood it correctly. Because
> /partN are not simply associated to the file, they are part of the
> file. ie. when I modify /dev/hda I also change /dev/hda/partN and
> vice-versa. I don't see any pratical problem, though.
> 
> 
> > Still, I really do like the idea of merging the notion of file and
> > directory into one notion of "container". I absolutely _detest_ files with
> > internal structure that tools have to know about (ie I hate seeing all
> > those embedded formats that I can't use "grep" on - MIME being one case).
> > I'd much rather see a "group of files"  and a "file with a grouping of
> > information".
> 
> You're actually looking for a database with a legacy fs-like interface :)

Hm. Isn't this exactly what MS has in store for Longhorn?

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: silent semantic changes with reiser4

[Giuseppe Bilotta]

Giuliano Pochini wrote:
> On 26-Aug-2004 Linus Torvalds wrote:
> 
> > I advocated (long ago) something like this for /dev handling, just because
> > I think it would make sense to have
> >
> >       /dev/hda        <- special file
> >       /dev/hda/part1  <- partition 1 (aka /dev/hda1)
> 
> This breaks the r4 semantics if I understood it correctly. Because
> /partN are not simply associated to the file, they are part of the
> file. ie. when I modify /dev/hda I also change /dev/hda/partN and
> vice-versa. I don't see any pratical problem, though.
> 
> 
> > Still, I really do like the idea of merging the notion of file and
> > directory into one notion of "container". I absolutely _detest_ files with
> > internal structure that tools have to know about (ie I hate seeing all
> > those embedded formats that I can't use "grep" on - MIME being one case).
> > I'd much rather see a "group of files"  and a "file with a grouping of
> > information".
> 
> You're actually looking for a database with a legacy fs-like interface :)

Hm. Isn't this exactly what MS has in store for Longhorn?

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes:

Giuseppe> Giuliano Pochini wrote:

[...]

>> You're actually looking for a database with a legacy fs-like
>> interface :)

Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn?

No.  To access the database-like functionality in WinFS, you need to use
a special API.

Hans as been planning this for a long time before WinFS was ever
announced.  See http://www.namesys.com/whitepaper.html

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes:

Giuseppe> Giuliano Pochini wrote:

[...]

>> You're actually looking for a database with a legacy fs-like
>> interface :)

Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn?

No.  To access the database-like functionality in WinFS, you need to use
a special API.

Hans as been planning this for a long time before WinFS was ever
announced.  See http://www.namesys.com/whitepaper.html

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Giuseppe Bilotta]

Hubert Chan wrote:
> >>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes:
> 
> Giuseppe> Giuliano Pochini wrote:
> 
> [...]
> 
> >> You're actually looking for a database with a legacy fs-like
> >> interface :)
> 
> Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn?
> 
> No.  To access the database-like functionality in WinFS, you need to use
> a special API.

I see, thanks. Might be interesting to know what they plan to 
make accessible of WinFS to legacy apps via the old API, then 
(especially since, at according to a /. article, they're 
planning to retrofit WinFS to WinXP ...)

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: silent semantic changes with reiser4

[Giuseppe Bilotta]

Hubert Chan wrote:
> >>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes:
> 
> Giuseppe> Giuliano Pochini wrote:
> 
> [...]
> 
> >> You're actually looking for a database with a legacy fs-like
> >> interface :)
> 
> Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn?
> 
> No.  To access the database-like functionality in WinFS, you need to use
> a special API.

I see, thanks. Might be interesting to know what they plan to 
make accessible of WinFS to legacy apps via the old API, then 
(especially since, at according to a /. article, they're 
planning to retrofit WinFS to WinXP ...)

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes:

Giuseppe> Hubert Chan wrote:

>> No.  To access the database-like functionality in WinFS, you need to
>> use a special API.

Giuseppe> I see, thanks. Might be interesting to know what they plan to
Giuseppe> make accessible of WinFS to legacy apps via the old API, then
Giuseppe> (especially since, at according to a /. article, they're
Giuseppe> planning to retrofit WinFS to WinXP ...)

AFAIK, all files are still accessible through normal NTFS.  You just
need the special API to do querying and such.  So old apps can access
all the files normally.  They just can't take advantage of all the new
whizbang features.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Giuseppe" == Giuseppe Bilotta <bilotta78@hotpop.com> writes:

Giuseppe> Hubert Chan wrote:

>> No.  To access the database-like functionality in WinFS, you need to
>> use a special API.

Giuseppe> I see, thanks. Might be interesting to know what they plan to
Giuseppe> make accessible of WinFS to legacy apps via the old API, then
Giuseppe> (especially since, at according to a /. article, they're
Giuseppe> planning to retrofit WinFS to WinXP ...)

AFAIK, all files are still accessible through normal NTFS.  You just
need the special API to do querying and such.  So old apps can access
all the files normally.  They just can't take advantage of all the new
whizbang features.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Fri, Aug 27, 2004 at 11:18:55AM +0200, Giuliano Pochini wrote:
> 
> On 26-Aug-2004 Linus Torvalds wrote:
> 
> > I advocated (long ago) something like this for /dev handling, just because
> > I think it would make sense to have
> >
> >       /dev/hda        <- special file
> >       /dev/hda/part1  <- partition 1 (aka /dev/hda1)
> 
> This breaks the r4 semantics if I understood it correctly. 

Reiser4 currently links /metas/ directory under each reiser4 fs object. 
Reiser4 "silently" allows regular file to be a directory for that.

That is only an example how an fs object may link its internal hierarchical
structure to the fs tree.  /dev/hda/-as-a-dir is another example, but, I am
afraid, it is not a reiser4 business to look into the devices.  Devfs or its
successor may do it. 

> Because
> /partN are not simply associated to the file, they are part of the
> file. ie. when I modify /dev/hda I also change /dev/hda/partN and
> vice-versa. I don't see any pratical problem, though.
> 
> 
> > Still, I really do like the idea of merging the notion of file and
> > directory into one notion of "container". I absolutely _detest_ files with
> > internal structure that tools have to know about (ie I hate seeing all
> > those embedded formats that I can't use "grep" on - MIME being one case).
> > I'd much rather see a "group of files"  and a "file with a grouping of
> > information".
> 
> You're actually looking for a database with a legacy fs-like interface :)
> 
> 
> --
> Giuliano.

-- 
Alex.

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 06:13:03PM +0200, Christoph Hellwig wrote:
> On Thu, Aug 26, 2004 at 05:06:38PM +0100, Jamie Lokier wrote:
> > Christoph Hellwig wrote:
> > > > There's bound to be some security issue, but I'm not sure what you're
> > > > getting at with /tmp.  What sort of sort of security problem arises
> > > > with a world-writeable directory such as /tmp, that cannot arise with
> > > > the standard fs semantics?
> > > 
> > > Actually you are right on that issue because it would open the
> > > device/fifo as directory and not device/fifo (in fact I'd had to look at
> > > the code again to see whether they actually do this only for files or
> > > also for special files)
> > 
> > Are you saying that with reiser4, you can open a device or fifo with
> > O_DIRECTORY?
> 
> That's what I thought, but as far as I can follow the code this is not
> actually true.

All reiser4 inodes have i_ops->lookup != NULL, so open(..., O_DIRECTORY) would
succeed on them (thanks Nikita for reminding me that).

It may be better to pass the control of that to ->i_op->permission() through 
explicit MAY_LOOKUP flag.  It can be possible to eliminate the race by some
more complex logic (strict ownership check, for example) in ->permission().

-- 
Alex.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote:
> 
> You ignored everything I said during the discussion of xattrs about how 
> there is no need to have attributes when you can just have files and 
> directories, and that xattrs reflected a complete ignorance of name 
> space design principles.  When I said we should just add some nice 
> optional features to files and directories so that they can do 
> everything that attributes can do if they are used that way, you just 
> didn't get it.  You instead went for the quick ugly hack called xattrs.  
> You then got that ugly hack done first, because quick hacks are, well, 
> quick.  I then went about doing it the right way for Reiser4, and got 
> DARPA to fund doing it.  I was never silent about it.

I don't want to comment on any of the technical issues about VFS etc. as
I would be completely out of my depth, however I do want to say 2 things. Firstly,
this is a feature that Samba users have been needing for many years to maintain
compatibility with NTFS and Windows clients. Microsoft no longer sell any servers
or clients without support for multiple data streams per file, and their latest
XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad
that Hans and Namesys have created something we can use to match this
functionality - soon we will need it in order to be able to exist in
a Microsoft client-dominated world.

My second point is the following. Hans - did you *really* have to reinvent
the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001
for Solaris) ?

http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5
http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C
http://bama.ua.edu/cgi-bin/man-cgi?openat+2

I'm complaining here as someone who will have to write portable code
to try and work on all these "files with streams" systems.

Jeremy.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jeremy Allison wrote:

>On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote:
>  
>
>>You ignored everything I said during the discussion of xattrs about how 
>>there is no need to have attributes when you can just have files and 
>>directories, and that xattrs reflected a complete ignorance of name 
>>space design principles.  When I said we should just add some nice 
>>optional features to files and directories so that they can do 
>>everything that attributes can do if they are used that way, you just 
>>didn't get it.  You instead went for the quick ugly hack called xattrs.  
>>You then got that ugly hack done first, because quick hacks are, well, 
>>quick.  I then went about doing it the right way for Reiser4, and got 
>>DARPA to fund doing it.  I was never silent about it.
>>    
>>
>
>I don't want to comment on any of the technical issues about VFS etc. as
>I would be completely out of my depth, however I do want to say 2 things. Firstly,
>this is a feature that Samba users have been needing for many years to maintain
>compatibility with NTFS and Windows clients. Microsoft no longer sell any servers
>or clients without support for multiple data streams per file, and their latest
>XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad
>that Hans and Namesys have created something we can use to match this
>functionality - soon we will need it in order to be able to exist in
>a Microsoft client-dominated world.
>  
>
I agree that your work is important without agreeing that MS client 
domination will last.;-)  It is indeed my desire to give you every 
single feature you need to emulate MS streams within files, but doing it 
using directories that are files.  I would like to support you in 
emulating windows faster than windows.

>My second point is the following. Hans - did you *really* have to reinvent
>the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001
>for Solaris) ?
>  
>
I interviewed for the file system architect job at Sun in, I think, 
1999, and they offered me the job conditional on my giving up on my 
Linux work.  (After much trying and failing to convince them that it 
would be okay for me to work on Linux also, I declined the job, much to 
my fiscal loss and work satisfaction.)

They do not do a pure job of implementing attributes in the file 
namespace though.  There are far more distinctions between files and 
attributes than are necessary that are described in these man pages 
below, and those distinctions cause a loss of closure.  I can say more 
on that if asked.

>http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5
>http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C
>http://bama.ua.edu/cgi-bin/man-cgi?openat+2
>
>I'm complaining here as someone who will have to write portable code
>to try and work on all these "files with streams" systems.
>
>Jeremy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Jeremy Allison wrote:

>On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote:
>  
>
>>You ignored everything I said during the discussion of xattrs about how 
>>there is no need to have attributes when you can just have files and 
>>directories, and that xattrs reflected a complete ignorance of name 
>>space design principles.  When I said we should just add some nice 
>>optional features to files and directories so that they can do 
>>everything that attributes can do if they are used that way, you just 
>>didn't get it.  You instead went for the quick ugly hack called xattrs.  
>>You then got that ugly hack done first, because quick hacks are, well, 
>>quick.  I then went about doing it the right way for Reiser4, and got 
>>DARPA to fund doing it.  I was never silent about it.
>>    
>>
>
>I don't want to comment on any of the technical issues about VFS etc. as
>I would be completely out of my depth, however I do want to say 2 things. Firstly,
>this is a feature that Samba users have been needing for many years to maintain
>compatibility with NTFS and Windows clients. Microsoft no longer sell any servers
>or clients without support for multiple data streams per file, and their latest
>XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad
>that Hans and Namesys have created something we can use to match this
>functionality - soon we will need it in order to be able to exist in
>a Microsoft client-dominated world.
>  
>
I agree that your work is important without agreeing that MS client 
domination will last.;-)  It is indeed my desire to give you every 
single feature you need to emulate MS streams within files, but doing it 
using directories that are files.  I would like to support you in 
emulating windows faster than windows.

>My second point is the following. Hans - did you *really* have to reinvent
>the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001
>for Solaris) ?
>  
>
I interviewed for the file system architect job at Sun in, I think, 
1999, and they offered me the job conditional on my giving up on my 
Linux work.  (After much trying and failing to convince them that it 
would be okay for me to work on Linux also, I declined the job, much to 
my fiscal loss and work satisfaction.)

They do not do a pure job of implementing attributes in the file 
namespace though.  There are far more distinctions between files and 
attributes than are necessary that are described in these man pages 
below, and those distinctions cause a loss of closure.  I can say more 
on that if asked.

>http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5
>http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C
>http://bama.ua.edu/cgi-bin/man-cgi?openat+2
>
>I'm complaining here as someone who will have to write portable code
>to try and work on all these "files with streams" systems.
>
>Jeremy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Hans Reiser wrote:

> I agree that your work is important without agreeing that MS client 
> domination will last.;-)  It is indeed my desire to give you every 
> single feature you need to emulate MS streams within files, but doing it 
> using directories that are files.  I would like to support you in 
> emulating windows faster than windows.

1) how do you back up and restore files with streams inside ?

2) how do standard unix utilities handle them ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Hans Reiser wrote:

> I agree that your work is important without agreeing that MS client 
> domination will last.;-)  It is indeed my desire to give you every 
> single feature you need to emulate MS streams within files, but doing it 
> using directories that are files.  I would like to support you in 
> emulating windows faster than windows.

1) how do you back up and restore files with streams inside ?

2) how do standard unix utilities handle them ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 09:27:52AM -0400, Rik van Riel wrote:
>
>1) how do you back up and restore files with streams inside ?

With a program that supports files with streams inside.

>2) how do standard unix utilities handle them ?

Most likely they don't ;) That is, until they are fixed or replaced.
I've heard of people who want xattrs to be backed up so they use star, 
not gnu tar, already.

-- 
mjt

Re: silent semantic changes with reiser4

[Paul Jackson]

Markus wrote:
> >2) how do standard unix utilities handle them ?
> 
> Most likely they don't ;) That is, until they are fixed or replaced.

Not good, in my view.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Paul Jackson]

Markus wrote:
> >2) how do standard unix utilities handle them ?
> 
> Most likely they don't ;) That is, until they are fixed or replaced.

Not good, in my view.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Hans Reiser wrote:
>
>  
>
>>I agree that your work is important without agreeing that MS client 
>>domination will last.;-)  It is indeed my desire to give you every 
>>single feature you need to emulate MS streams within files, but doing it 
>>using directories that are files.  I would like to support you in 
>>emulating windows faster than windows.
>>    
>>
>
>1) how do you back up and restore files with streams inside ?
>
>2) how do standard unix utilities handle them ?
>
>  
>
To repeat, I think it would be nice to implement a 
filename/pseudos/backup method for all the plugins.

Guys, we just have the beginnings in place.  One plugin method at a time 
it will all fall into place.

What we have now is useful now.  The more methods come into existence, 
the more compelling it becomes --- standard network economic theory 
applies here.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Hans Reiser wrote:
>
>  
>
>>I agree that your work is important without agreeing that MS client 
>>domination will last.;-)  It is indeed my desire to give you every 
>>single feature you need to emulate MS streams within files, but doing it 
>>using directories that are files.  I would like to support you in 
>>emulating windows faster than windows.
>>    
>>
>
>1) how do you back up and restore files with streams inside ?
>
>2) how do standard unix utilities handle them ?
>
>  
>
To repeat, I think it would be nice to implement a 
filename/pseudos/backup method for all the plugins.

Guys, we just have the beginnings in place.  One plugin method at a time 
it will all fall into place.

What we have now is useful now.  The more methods come into existence, 
the more compelling it becomes --- standard network economic theory 
applies here.

Hans

Re: silent semantic changes with reiser4

[Chris Mason]

On Wed, 2004-08-25 at 15:53, Hans Reiser wrote:
> I had not intended to respond to this because I have nothing positive to 
> say, but Andrew said I needed to respond and suggested I should copy 
> Linus. Sigh.
> 
> Dear Christoph,
> 
> Let me see if I can summarize what you and your contingent are saying, 
> and if I misconstrue anything, let me know.;-)
> 
Just for fun why don't we look at the way things are today:

1) reiser4 has semantics that do belong at the VFS level.  They weren't
implemented at the VFS level for a variety of reasons, none of which
really matter right now.

2) new kernel patches that fragment the application developers between
apis are a bad thing.  There does need to be one interface here, and it
is in Hans' best interest to unify his work by working with people to
introduce new kernel wide apis.

This starts with exactly what Christoph described in writing a short
summary of how you want things to work today.  Since we can't resist,
we'll also go ahead and rehash all the old flame wars over this, but try
to include some new ideas about where you want to see the reiser4
interfaces in 6 months as well.

-chris

Re: silent semantic changes with reiser4

[Hans Reiser]

Chris Mason wrote:

>
>
>This starts with exactly what Christoph described in writing a short
>summary of how you want things to work today.  Since we can't resist,
>we'll also go ahead and rehash all the old flame wars over this, but try
>to include some new ideas about where you want to see the reiser4
>interfaces in 6 months as well.
>
>-chris
>
>
>
>
>
>  
>
Did Christoph read the www.namesys.com URL that has for several years 
been hiding from him? ;-)

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Chris Mason wrote:

>
>
>This starts with exactly what Christoph described in writing a short
>summary of how you want things to work today.  Since we can't resist,
>we'll also go ahead and rehash all the old flame wars over this, but try
>to include some new ideas about where you want to see the reiser4
>interfaces in 6 months as well.
>
>-chris
>
>
>
>
>
>  
>
Did Christoph read the www.namesys.com URL that has for several years 
been hiding from him? ;-)

Hans

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:42:50AM -0700, Hans Reiser wrote:
> Did Christoph read the www.namesys.com URL that has for several years 
> been hiding from him? ;-)

Yes, and I'm sick of your marketing bullshit Hans.

Re: silent semantic changes with reiser4

[Chris Friesen]

Hans Reiser wrote:

> Making files into directories caused only two applications out of the
> entire OS to notice the change, and that was because of a bug in what
> error code we returned that we are going to fix.  You think that was a
> disaster; I think it was a triumph.

>  Since we have such a performance lead, Namesys is about to change its
> focus from the storage layer to semantics, look at
> www.namesys.com/whitepaper.html for details.  Semantic enhancements are
> the important stuff, and finally Namesys is where we have all the
> storage layer prerequisites done right, and the real work can begin. 

Just curious about your comments on Jamie Lokier's suggestions for enabling 
files-as-directories semantics without breaking existing apps.

Chris

Re: silent semantic changes with reiser4

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
> I had not intended to respond to this because I have nothing positive to 
> say, but Andrew said I needed to respond and suggested I should copy 
> Linus.

Yes, but I didn't say "flame Christoph and ignore the issues" ;)

There are lots of little things to do with implementation, coding style,
module exports, deadlocks, what code goes where, etc.  These are all normal
daily kernel business and we should set them aside for now and concentrate
on the bigger issues.

And as I see it, there are two big issues:

a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
   anticipate and 

b) it does this within the context of just a single filesystem.

I see three possible responses:

a) accept the reiser4-only extensions as-is (possibly with post-review
   modifications, of course) or

b) accept the reiser4-only extensions with a view to turning them into
   kernel-wide extensions at some time in the future, so all filesystems
   will offer the extensions (as much as poss) or

c) reject the extensions.


My own order of preference is b) c) a).  The fact that one filesystem will
offer features which other filesystems do not and cannot offer makes me
queasy for some reason.

b) means that at some time in the future we need to hoist the reiser4
extensions (at a conceptual level) (and probably with restrictions) up into
the VFS.  This will involve much thought, argument and work.


To get us started on this route it would really help me (and, probably,
others) if you could describe what these API extensions are in a very
simple way, without referring to incomprehehsible web pages, and without
using terms which non-reiser4 people don't understand.

It would be best if each extension was addressed in a separate email
thread.

We also need to discuss what a reiser4 "module" is, what its capabilities
are, and what licensing implications they have.

Then, we can look at each one and say "yup, that makes sense - we want
Linux to do that" and we can also think about how we would implement it at
the VFS level.

If we follow the above route I believe we can make progress in a technical
direction and not get deadlocked on personal/political stuff.


Now, an alternative to all the above is to just merge reiser4 as-is, after
addressing all the lower-level coding issues.  And see what happens.  That
may be a thing which Linus wishes to do.  I'm easy.

Re: silent semantic changes with reiser4

[Spam]

> And as I see it, there are two big issues:

> a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
>    anticipate and 

> b) it does this within the context of just a single filesystem.

> I see three possible responses:

> a) accept the reiser4-only extensions as-is (possibly with post-review
>    modifications, of course) or

> b) accept the reiser4-only extensions with a view to turning them into
>    kernel-wide extensions at some time in the future, so all filesystems
>    will offer the extensions (as much as poss) or

> c) reject the extensions.


> My own order of preference is b) c) a).  The fact that one filesystem will
> offer features which other filesystems do not and cannot offer makes me
> queasy for some reason.

  This last sentence makes me wonder. Where is Linux heading? The idea
  that   a  FS  cannot  contain  features that no other FS has is very
  scary.

  I  am  all  for  uniformity, but not at the expense of shutting down
  advanced progress that Linux is so badly needing.

  This talk about old UNIX seems like people want to still live in the
  70'ies and not look forward. Please wake up!

  ~S

Re: silent semantic changes with reiser4

[Spam]

> And as I see it, there are two big issues:

> a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
>    anticipate and 

> b) it does this within the context of just a single filesystem.

> I see three possible responses:

> a) accept the reiser4-only extensions as-is (possibly with post-review
>    modifications, of course) or

> b) accept the reiser4-only extensions with a view to turning them into
>    kernel-wide extensions at some time in the future, so all filesystems
>    will offer the extensions (as much as poss) or

> c) reject the extensions.


> My own order of preference is b) c) a).  The fact that one filesystem will
> offer features which other filesystems do not and cannot offer makes me
> queasy for some reason.

  This last sentence makes me wonder. Where is Linux heading? The idea
  that   a  FS  cannot  contain  features that no other FS has is very
  scary.

  I  am  all  for  uniformity, but not at the expense of shutting down
  advanced progress that Linux is so badly needing.

  This talk about old UNIX seems like people want to still live in the
  70'ies and not look forward. Please wake up!

  ~S

Re: silent semantic changes with reiser4

[Christoph Hellwig]

>   This last sentence makes me wonder. Where is Linux heading? The idea
>   that   a  FS  cannot  contain  features that no other FS has is very
>   scary.
> 
>   I  am  all  for  uniformity, but not at the expense of shutting down
>   advanced progress that Linux is so badly needing.
> 
>   This talk about old UNIX seems like people want to still live in the
>   70'ies and not look forward. Please wake up!

Just because semantics are at the VFS layer doesn't mean every
filesystem has to implement them.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Spam wrote:
> 
> > My own order of preference is b) c) a).  The fact that one filesystem will
> > offer features which other filesystems do not and cannot offer makes me
> > queasy for some reason.
> 
>   This last sentence makes me wonder. Where is Linux heading? The idea
>   that   a  FS  cannot  contain  features that no other FS has is very
>   scary.

That's not what Andrew said or meant.

Note the "cannot offer". As in "there is no way to offer them even if the 
filesystem could support it otherwise". 

We have tons of filesystems that do things other filesystems cannot do.  
Most filesystems support writing to a file - despite the fact that some
filesystems (iso9600 being an obvious one) cannot. The infrastructure is
there in the VFS layer, and it becomes a _choice_ for the filesystem
whether it offers certain capabilities.

So look at what Andrew said, again: his top choice would be (b). Let's see 
what that was again, shall we?

> b) accept the reiser4-only extensions with a view to turning them into
>    kernel-wide extensions at some time in the future, so all filesystems
>    will offer the extensions (as much as poss) or

In other words, if reiserfs does something special, we should make 
standard interfaces for doing that special thing, so that everybody can 
do it without stepping on other peoples toes.

That doesn't mean that we'd _force_ everybody to do it. The same way we 
don't force iso9660 to write to a CD-ROM.

		Linus

Re: silent semantic changes with reiser4

[Spam]

> On Thu, 26 Aug 2004, Spam wrote:
>> 
>> > My own order of preference is b) c) a).  The fact that one filesystem will
>> > offer features which other filesystems do not and cannot offer makes me
>> > queasy for some reason.
>> 
>>   This last sentence makes me wonder. Where is Linux heading? The idea
>>   that   a  FS  cannot  contain  features that no other FS has is very
>>   scary.

> That's not what Andrew said or meant.

> Note the "cannot offer". As in "there is no way to offer them even if the
> filesystem could support it otherwise". 

> We have tons of filesystems that do things other filesystems cannot do.
> Most filesystems support writing to a file - despite the fact that some
> filesystems (iso9600 being an obvious one) cannot. The infrastructure is
> there in the VFS layer, and it becomes a _choice_ for the filesystem
> whether it offers certain capabilities.

> So look at what Andrew said, again: his top choice would be (b). Let's see
> what that was again, shall we?

>> b) accept the reiser4-only extensions with a view to turning them into
>>    kernel-wide extensions at some time in the future, so all filesystems
>>    will offer the extensions (as much as poss) or

> In other words, if reiserfs does something special, we should make 
> standard interfaces for doing that special thing, so that everybody can
> do it without stepping on other peoples toes.

  Agreed  that would be the best. But how much time and effort will it
  be,  and  how much of the original ideas would be lost on the way to
  implement  them  in  the  VFS? Especially with new and very advanced
  FS's like Reiser4.

  Isn't  the  line  between the actual file system and the virtual one
  very hair thin? Where would the separation lay in Reiser4?
  
> That doesn't mean that we'd _force_ everybody to do it. The same way we
> don't force iso9660 to write to a CD-ROM.

>       Linus

  I got caught in the moment of flame war. My appologies.

  ~S

Re: silent semantic changes with reiser4

[Spam]

> On Thu, 26 Aug 2004, Spam wrote:
>> 
>> > My own order of preference is b) c) a).  The fact that one filesystem will
>> > offer features which other filesystems do not and cannot offer makes me
>> > queasy for some reason.
>> 
>>   This last sentence makes me wonder. Where is Linux heading? The idea
>>   that   a  FS  cannot  contain  features that no other FS has is very
>>   scary.

> That's not what Andrew said or meant.

> Note the "cannot offer". As in "there is no way to offer them even if the
> filesystem could support it otherwise". 

> We have tons of filesystems that do things other filesystems cannot do.
> Most filesystems support writing to a file - despite the fact that some
> filesystems (iso9600 being an obvious one) cannot. The infrastructure is
> there in the VFS layer, and it becomes a _choice_ for the filesystem
> whether it offers certain capabilities.

> So look at what Andrew said, again: his top choice would be (b). Let's see
> what that was again, shall we?

>> b) accept the reiser4-only extensions with a view to turning them into
>>    kernel-wide extensions at some time in the future, so all filesystems
>>    will offer the extensions (as much as poss) or

> In other words, if reiserfs does something special, we should make 
> standard interfaces for doing that special thing, so that everybody can
> do it without stepping on other peoples toes.

  Agreed  that would be the best. But how much time and effort will it
  be,  and  how much of the original ideas would be lost on the way to
  implement  them  in  the  VFS? Especially with new and very advanced
  FS's like Reiser4.

  Isn't  the  line  between the actual file system and the virtual one
  very hair thin? Where would the separation lay in Reiser4?
  
> That doesn't mean that we'd _force_ everybody to do it. The same way we
> don't force iso9660 to write to a CD-ROM.

>       Linus

  I got caught in the moment of flame war. My appologies.

  ~S

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
> > In other words, if reiserfs does something special, we should make 
> > standard interfaces for doing that special thing, so that everybody can
> > do it without stepping on other peoples toes.
> 
>   Agreed  that would be the best. But how much time and effort will it
>   be

Zero.

We can add these new features tomorrow, as reiser4-only features, with a
plan in hand to generalise them later.

-->>__if__<<-- we think these are features which Linux should offer.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Aug 25, 2004 at 04:32:25PM -0700, Andrew Morton wrote:
> 
> We can add these new features tomorrow, as reiser4-only features, with a
> plan in hand to generalise them later.
> 
> -->>__if__<<-- we think these are features which Linux should offer.

Multiple-data-stream files are something we should offer, definately (IMHO).
I don't care how we do it, but I know it's something we need as application
developers.

Jeremy.

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Jeremy Allison wrote:
> Multiple-data-stream files are something we should offer, definately (IMHO).
> I don't care how we do it, but I know it's something we need as application
> developers.

Aside from samba, is there any other application that has a use for
them? 

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Tierra]

On Thu, 26 Aug 2004 01:46:29 +0200, Wichert Akkerman <wichert@wiggy.net> wrote:
> Previously Jeremy Allison wrote:
> > Multiple-data-stream files are something we should offer, definately (IMHO).
> > I don't care how we do it, but I know it's something we need as application
> > developers.
> 
> Aside from samba, is there any other application that has a use for
> them?
> 
> Wichert.
> 
> --
> Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
> http://www.wiggy.net/                   It is hard to make things simple.
> 

It's not so much about samba having a use for them as it is about
other applications using them between platforms in the same way. I'm
glad to see Jeremy is on top of this because it's not something I had
even thought about having available on the linux platform in the sense
that my work deals with data streams on NTFS, HFS, and whatever other
filesystems join the venture.

NTFS had very poor support for them for a long time. At one point I
was convinced it was going to be taken out eventually, but SP2 just
showed with even better support for them, and more MS software has
started using them. They can't turn around at this point and take them
out, so they must be planning on using them to a wider extent.

Bryan

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Tierra wrote:
> It's not so much about samba having a use for them as it is about
> other applications using them between platforms in the same way.

But what for? Personally I still haven't seen a good use for them.
Apple's resource fork comes close though.

> NTFS had very poor support for them for a long time. At one point I
> was convinced it was going to be taken out eventually, but SP2 just
> showed with even better support for them, and more MS software has
> started using them.

I wonder how they intend to shoehorn that into distribution across
other media such as emails or zip files.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Tierra]

On Thu, 26 Aug 2004 02:05:40 +0200, Wichert Akkerman <wichert@wiggy.net> wrote:
> But what for? Personally I still haven't seen a good use for them.
> Apple's resource fork comes close though.

There's one way where it can really come in handy, and that's with
specific application file types. Certain media could be saved in a
better format and read more efficiently. The reason you haven't seen
this done yet though is because multiple data streams isn't a standard
and isn't available in most enviroments. There also isn't any file
transfer protocols that will handle transfer of multiple data streams
except through Windows File Sharing.

Take into account though that I have absolutely no experience or
knowledge of how it all works in the backend, and don't know the
implications involved in the implementation.

As for applications that already use data streams, you mentioned
Apple's resource fork already, so I assume you know about file types
there. IIS currently uses ADS for image thumbnailing and it's also
possible to hide an alternate website right inline with a public one.
I've heard of a couple other applications on Windows that uses them.
But you are right, none of them _appropriately_ use data streams for
any specific advantage over regular files except to keep system
controlled information hidden.

Bryan

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
> Previously Jeremy Allison wrote:
> > Multiple-data-stream files are something we should offer, definately (IMHO).
> > I don't care how we do it, but I know it's something we need as application
> > developers.
> 
> Aside from samba, is there any other application that has a use for
> them? 
> 

Anything that currently stores a file's metadata in another file really
wants this right now. Things like image thumbnails, document summaries,
digital signatures, etc.

As to how to do it, I think the Solaris interface is reasonably decent.
The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view

(An important detail for those who want to access their
multiple-data-streams from non-MDS aware apps is the runat shell
command, which basically does a chdir into the specified file's
attribute directory and then runs a command. i.e. 'runat ~/blah ls' will
list the ~/blah's attributes.)

The only real problem I have with their design is the calling them
attributes and using "at" everywhere. 

"Attributes",  because it will get confused with the current Linux xattr
implementation (which is still useful for things that actually are file
attributes, like security labels, ACLs, weird attributess that
FAT/NTFS/whatever have, etc.).

I don't like "at" because the API changes don't have anything to do with
the actual attributes. It's a general set of changes to allow paths
relative to a fd instead of the cwd, and doesn't really have anything
specifically to do with attributes (with the exception of the O_XATTR
flag). 

Replace "at" with "rel" and O_XATTR with O_FORK or O_MULTI or something,
and it's all good.

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
> Previously Jeremy Allison wrote:
> > Multiple-data-stream files are something we should offer, definately (IMHO).
> > I don't care how we do it, but I know it's something we need as application
> > developers.
> 
> Aside from samba, is there any other application that has a use for
> them? 
> 

Anything that currently stores a file's metadata in another file really
wants this right now. Things like image thumbnails, document summaries,
digital signatures, etc.

As to how to do it, I think the Solaris interface is reasonably decent.
The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view

(An important detail for those who want to access their
multiple-data-streams from non-MDS aware apps is the runat shell
command, which basically does a chdir into the specified file's
attribute directory and then runs a command. i.e. 'runat ~/blah ls' will
list the ~/blah's attributes.)

The only real problem I have with their design is the calling them
attributes and using "at" everywhere. 

"Attributes",  because it will get confused with the current Linux xattr
implementation (which is still useful for things that actually are file
attributes, like security labels, ACLs, weird attributess that
FAT/NTFS/whatever have, etc.).

I don't like "at" because the API changes don't have anything to do with
the actual attributes. It's a general set of changes to allow paths
relative to a fd instead of the cwd, and doesn't really have anything
specifically to do with attributes (with the exception of the O_XATTR
flag). 

Replace "at" with "rel" and O_XATTR with O_FORK or O_MULTI or something,
and it's all good.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Nicholas Miell wrote:
> Anything that currently stores a file's metadata in another file really
> wants this right now. Things like image thumbnails, document summaries,
> digital signatures, etc.

Additionally, all of those things you describe should be deleted if
the file is modified -- to indicate that they're no longer valid and
should be regenerated if needed.

Whereas there are some other kinds of metadata which should not be
deleted if the file is modified.

-- Jamie

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Wed, 2004-08-25 at 18:03, Jamie Lokier wrote:
> Nicholas Miell wrote:
> > Anything that currently stores a file's metadata in another file really
> > wants this right now. Things like image thumbnails, document summaries,
> > digital signatures, etc.
> 
> Additionally, all of those things you describe should be deleted if
> the file is modified -- to indicate that they're no longer valid and
> should be regenerated if needed.
> 
> Whereas there are some other kinds of metadata which should not be
> deleted if the file is modified.
> 
> -- Jamie

Presumably the app which uses the metadata will be smart enough to
compare the st_mtime of the MDS/stream/attribute/whatever (can we choose
a name for these things now?) to the st_mtime of the file and do the
right thing.

thumbnail - regenerate it
summary - keep it, it's relatively independent of the file's exact
contents
signature - always verify it
etc.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Nicholas Miell wrote:
> > Additionally, all of those things you describe should be deleted if
> > the file is modified -- to indicate that they're no longer valid and
> > should be regenerated if needed.
> > 
> > Whereas there are some other kinds of metadata which should not be
> > deleted if the file is modified.
> > 
> > -- Jamie
> 
> Presumably the app which uses the metadata will be smart enough to
> compare the st_mtime of the MDS/stream/attribute/whatever (can we choose
> a name for these things now?) to the st_mtime of the file and do the
> right thing.

[This is straying off the topic of files-as-directories].

st_mtime tests are weak.  They break sometimes, and are not suitable
for strong data models such as transparently caching generated data
from a file's contents.

They're especially breakable if you change a file and then read it
within a second.  Sometimes, more than a second.

As has been raised before, nanosecond timestamps (a) don't solve this
unless they're stored in the filesystem; (b) even then they will fail
one day, on a sufficiently fast box; (c) don't work when there are
changes to wall clock time.

They're fine where you don't care if the generated data is wrong sometimes.

If you want the _equivalent_ behaviour to opening and parsing the
file, but fast, then they're no good.

Modification serial numbers would work, though.

-- Jamie

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Wed, 2004-08-25 at 18:53, Jamie Lokier wrote:
> Nicholas Miell wrote:
> > > Additionally, all of those things you describe should be deleted if
> > > the file is modified -- to indicate that they're no longer valid and
> > > should be regenerated if needed.
> > > 
> > > Whereas there are some other kinds of metadata which should not be
> > > deleted if the file is modified.
> > > 
> > > -- Jamie
> > 
> > Presumably the app which uses the metadata will be smart enough to
> > compare the st_mtime of the MDS/stream/attribute/whatever (can we choose
> > a name for these things now?) to the st_mtime of the file and do the
> > right thing.
> 
> [This is straying off the topic of files-as-directories].
> 
> st_mtime tests are weak.  They break sometimes, and are not suitable
> for strong data models such as transparently caching generated data
> from a file's contents.
> 
> They're especially breakable if you change a file and then read it
> within a second.  Sometimes, more than a second.
> 
> As has been raised before, nanosecond timestamps (a) don't solve this
> unless they're stored in the filesystem; (b) even then they will fail
> one day, on a sufficiently fast box; (c) don't work when there are
> changes to wall clock time.
> 
> They're fine where you don't care if the generated data is wrong sometimes.
> 
> If you want the _equivalent_ behaviour to opening and parsing the
> file, but fast, then they're no good.
> 
> Modification serial numbers would work, though.
> 
> -- Jamie

Remember that the Solaris attribute model is just another filesystem
tree rooted in a hidden directory associated with a regular file. If you
can come up with semantics that work in general for all files, that's
fine.

inodes that delete themselves when other inodes are changed creep me
out.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>Nicholas Miell wrote:
>  
>
>>Anything that currently stores a file's metadata in another file really
>>wants this right now. Things like image thumbnails, document summaries,
>>digital signatures, etc.
>>    
>>
>
>Additionally, all of those things you describe should be deleted if
>the file is modified -- to indicate that they're no longer valid and
>should be regenerated if needed.
>
>Whereas there are some other kinds of metadata which should not be
>deleted if the file is modified.
>
>-- Jamie
>
>
>  
>
Yes, I agree.

Actually we plan to have a whole link taxonomy, and one expected feature 
is that some links don't count towards the refcount needed to keep an 
object in existence (For instance, the links between key words and text 
documents, you don't want to have to explicitly unlink every keyword in 
order to delete a document).

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>Nicholas Miell wrote:
>  
>
>>Anything that currently stores a file's metadata in another file really
>>wants this right now. Things like image thumbnails, document summaries,
>>digital signatures, etc.
>>    
>>
>
>Additionally, all of those things you describe should be deleted if
>the file is modified -- to indicate that they're no longer valid and
>should be regenerated if needed.
>
>Whereas there are some other kinds of metadata which should not be
>deleted if the file is modified.
>
>-- Jamie
>
>
>  
>
Yes, I agree.

Actually we plan to have a whole link taxonomy, and one expected feature 
is that some links don't count towards the refcount needed to keep an 
object in existence (For instance, the links between key words and text 
documents, you don't want to have to explicitly unlink every keyword in 
order to delete a document).

Re: silent semantic changes with reiser4

[Giuseppe Bilotta]

Jamie Lokier wrote:
> Nicholas Miell wrote:
> > Anything that currently stores a file's metadata in another file really
> > wants this right now. Things like image thumbnails, document summaries,
> > digital signatures, etc.
> 
> Additionally, all of those things you describe should be deleted if
> the file is modified -- to indicate that they're no longer valid and
> should be regenerated if needed.

In principle, not all of them. For example, a document summary 
for a text document or a long (textual) description of a video 
clip might remain the same when the user is only working on the 
finishing details.

Maybe the metadata needs an attribute to determine how 
'immutable' it should be wrt changes on the file? (Can you 
spell meta-meta-data <g>)

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: silent semantic changes with reiser4

[Giuseppe Bilotta]

Jamie Lokier wrote:
> Nicholas Miell wrote:
> > Anything that currently stores a file's metadata in another file really
> > wants this right now. Things like image thumbnails, document summaries,
> > digital signatures, etc.
> 
> Additionally, all of those things you describe should be deleted if
> the file is modified -- to indicate that they're no longer valid and
> should be regenerated if needed.

In principle, not all of them. For example, a document summary 
for a text document or a long (textual) description of a video 
clip might remain the same when the user is only working on the 
finishing details.

Maybe the metadata needs an attribute to determine how 
'immutable' it should be wrt changes on the file? (Can you 
spell meta-meta-data <g>)

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
> 
> Anything that currently stores a file's metadata in another file really
> wants this right now. Things like image thumbnails, document summaries,
> digital signatures, etc.

Which is exactly how Windows apps have started to use streams.

> As to how to do it, I think the Solaris interface is reasonably decent.
> The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view

I agree. This is an interface Samba can live with I think. I
was thinking of implementing it anyway, just so I could piss
off the Linux kernel developers by saying "oh if you need proper
Windows semantics on Samba you have to use an advanced OS like
Solaris, Linux doesn't cut it" :-) :-).

> The only real problem I have with their design is the calling them
> attributes and using "at" everywhere. 

Yep - they're different from xattrs. The easiest way to remember
this is that file streams are seekable and get a fd, xattrs aren't
and don't :-).

Jeremy.

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Jeremy Allison wrote:
> Which is exactly how Windows apps have started to use streams.

Interesting I can not find a way using the .NET framework, which is
supposed to be The Future, to use streams.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Tierra]

On Thu, 26 Aug 2004 13:01:41 +0200, Wichert Akkerman <wichert@wiggy.net> wrote:
> Previously Jeremy Allison wrote:
> > Which is exactly how Windows apps have started to use streams.
>
> Interesting I can not find a way using the .NET framework, which is
> supposed to be The Future, to use streams.

That's because you don't even need .NET...
http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/q105/7/63.asp&NoWebContent=1&NoWebContent=1

http://tinyurl.com/45mfa

Bryan

Re: silent semantic changes with reiser4

[Matt Mackall]

On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
> On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
> > Previously Jeremy Allison wrote:
> > > Multiple-data-stream files are something we should offer, definately (IMHO).
> > > I don't care how we do it, but I know it's something we need as application
> > > developers.
> > 
> > Aside from samba, is there any other application that has a use for
> > them? 
> > 
> 
> Anything that currently stores a file's metadata in another file really
> wants this right now. Things like image thumbnails, document summaries,
> digital signatures, etc.

That is _highly_ debatable. I would much rather have my cp and grep
and cat and tar and such continue to work than have to rewrite every
tool because we've thrown the file-is-a-stream-of-bytes concept out
the window. Never mind that I've got thumbnails, document summaries,
and digital signatures already.

While the number of annoying properties of files with forks is
practically endless, the biggest has got to be utter lack of
portability. How do you stick the thing in an attachment or on an ftp
site? Well you can't because it's NOT A FILE. 

A file is a stream of bytes.

-- 
Mathematics is the supreme nostalgia of our time.

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Wed, 2004-08-25 at 21:44, Matt Mackall wrote:
> On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
> > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
> > > Previously Jeremy Allison wrote:
> > > > Multiple-data-stream files are something we should offer, definately (IMHO).
> > > > I don't care how we do it, but I know it's something we need as application
> > > > developers.
> > > 
> > > Aside from samba, is there any other application that has a use for
> > > them? 
> > > 
> > 
> > Anything that currently stores a file's metadata in another file really
> > wants this right now. Things like image thumbnails, document summaries,
> > digital signatures, etc.
> 
> That is _highly_ debatable. I would much rather have my cp and grep
> and cat and tar and such continue to work than have to rewrite every
> tool because we've thrown the file-is-a-stream-of-bytes concept out
> the window. Never mind that I've got thumbnails, document summaries,
> and digital signatures already.
> 
> While the number of annoying properties of files with forks is
> practically endless, the biggest has got to be utter lack of
> portability. How do you stick the thing in an attachment or on an ftp
> site? Well you can't because it's NOT A FILE. 
> 
> A file is a stream of bytes.

"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all
the time and are fixed. It's called progress.

cp, grep, cat, and tar will continue to work just fine on files with
multiple streams.

tar and cp will lose the extra streams until somebody fixes them, but
they lose ACLs and xattrs right now, and I don't hear anybody suggesting
that ACLs and xattrs be removed from the kernel because of this.

Fixing programs that do recursive filesystem traversals is a matter of a
glibc patch to nftw(3) and a modification to their option processing.

Holding back a useful feature because you don't want to upgrade
coreutils is just plain dumb.

(BTW, for email, multipart/parallel is a start, but a specific multipart
content type for multi-stream file attachments would probably be more
appropriate.)

Re: silent semantic changes with reiser4

[James Morris]

On Wed, 25 Aug 2004, Nicholas Miell wrote:

> (BTW, for email, multipart/parallel is a start, but a specific multipart
> content type for multi-stream file attachments would probably be more
> appropriate.)

I don't understand why something like this is so important that it needs 
underlying VFS support.  Multipart messages seem to have survived fine up 
until now.  Where is the justification?


- James
-- 
James Morris
<jmorris@redhat.com>

Re: silent semantic changes with reiser4

[James Morris]

On Wed, 25 Aug 2004, Nicholas Miell wrote:

> (BTW, for email, multipart/parallel is a start, but a specific multipart
> content type for multi-stream file attachments would probably be more
> appropriate.)

I don't understand why something like this is so important that it needs 
underlying VFS support.  Multipart messages seem to have survived fine up 
until now.  Where is the justification?


- James
-- 
James Morris
<jmorris@redhat.com>

Re: silent semantic changes with reiser4

[Matt Mackall]

On Wed, Aug 25, 2004 at 10:09:08PM -0700, Nicholas Miell wrote:
> On Wed, 2004-08-25 at 21:44, Matt Mackall wrote:
> > On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
> > > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
> > > > Previously Jeremy Allison wrote:
> > > > > Multiple-data-stream files are something we should offer, definately (IMHO).
> > > > > I don't care how we do it, but I know it's something we need as application
> > > > > developers.
> > > > 
> > > > Aside from samba, is there any other application that has a use for
> > > > them? 
> > > > 
> > > 
> > > Anything that currently stores a file's metadata in another file really
> > > wants this right now. Things like image thumbnails, document summaries,
> > > digital signatures, etc.
> > 
> > That is _highly_ debatable. I would much rather have my cp and grep
> > and cat and tar and such continue to work than have to rewrite every
> > tool because we've thrown the file-is-a-stream-of-bytes concept out
> > the window. Never mind that I've got thumbnails, document summaries,
> > and digital signatures already.
> > 
> > While the number of annoying properties of files with forks is
> > practically endless, the biggest has got to be utter lack of
> > portability. How do you stick the thing in an attachment or on an ftp
> > site? Well you can't because it's NOT A FILE. 
> > 
> > A file is a stream of bytes.
> 
> "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all
> the time and are fixed. It's called progress.

What it breaks is the concept of a file. In ways that are ill-defined,
not portable, hard to work with, and needlessly complex. Along the
way, it breaks every single application that ever thought it knew what
a file was.

Progress? No, this has been done before. Various dead operating
systems have done it or similar and regretted it. Most recently MacOS,
which jumped through major hurdles to begin purging themselves of
resource forks when they went to OS X. They're still there, but
heavily deprecated.

> cp, grep, cat, and tar will continue to work just fine on files with
> multiple streams.

Find some silly person with an iBook and open a shell on OS X. Use cp
to copy a file with a resource fork. Oh look, the Finder has no idea
what the new file is, even though it looks exactly identical in the
shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
fork. How is that ever going to work?

I like cat < a > b. You can keep your progress.

-- 
Mathematics is the supreme nostalgia of our time.

Re: silent semantic changes with reiser4

[Denis Vlasenko]

> > > A file is a stream of bytes.
> >
> > "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all
> > the time and are fixed. It's called progress.
>
> What it breaks is the concept of a file. In ways that are ill-defined,
> not portable, hard to work with, and needlessly complex. Along the
> way, it breaks every single application that ever thought it knew what
> a file was.
>
> Progress? No, this has been done before. Various dead operating
> systems have done it or similar and regretted it. Most recently MacOS,
> which jumped through major hurdles to begin purging themselves of
> resource forks when they went to OS X. They're still there, but
> heavily deprecated.
>
> > cp, grep, cat, and tar will continue to work just fine on files with
> > multiple streams.
>
> Find some silly person with an iBook and open a shell on OS X. Use cp
> to copy a file with a resource fork. Oh look, the Finder has no idea
> what the new file is, even though it looks exactly identical in the
> shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
> fork. How is that ever going to work?
>
> I like cat < a > b. You can keep your progress.

cat <a >b does not preserve following file properties even on standard
UNIX filesystems: name,owner,group,permissions.
-- 
vda

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Denis Vlasenko wrote:

> > I like cat < a > b. You can keep your progress.
> 
> cat <a >b does not preserve following file properties even on standard
> UNIX filesystems: name,owner,group,permissions.

Losing permissions is one thing.  Annoying, mostly.

However, actual losing file data during such a copy is
nothing short of a disaster, IMHO.  

In my opinion we shouldn't merge file-as-a-directory
semantics into the kernel until we figure out how to
fix the backup/restore problem and keep standard unix
utilities work.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Denis Vlasenko wrote:

> > I like cat < a > b. You can keep your progress.
> 
> cat <a >b does not preserve following file properties even on standard
> UNIX filesystems: name,owner,group,permissions.

Losing permissions is one thing.  Annoying, mostly.

However, actual losing file data during such a copy is
nothing short of a disaster, IMHO.  

In my opinion we shouldn't merge file-as-a-directory
semantics into the kernel until we figure out how to
fix the backup/restore problem and keep standard unix
utilities work.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 3556 bytes --]

Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:

> > > I like cat < a > b. You can keep your progress.
> > 
> > cat <a >b does not preserve following file properties even on standard
> > UNIX filesystems: name,owner,group,permissions.
> 
> Losing permissions is one thing.  Annoying, mostly.
> 
> However, actual losing file data during such a copy is
> nothing short of a disaster, IMHO.  
> 
> In my opinion we shouldn't merge file-as-a-directory
> semantics into the kernel until we figure out how to
> fix the backup/restore problem and keep standard unix
> utilities work.

Well, again, what about xattrs and ACLs...?

Actually, reiser4 doesn't currently implement storage of arbitrary user
data under a file.

It's just some sysfs-like information that can be retrieved, some
properties can be changed. If you copy a file the worst thing that can
happen right now is that you lose the information whether that file
should be encrypted or compressed. You don't lose data.

In case some people are wondering, this is what you can find in the
reiser4 metas:

leto:/home/chtephan/.muttrc/metas > la
insgesamt 1
dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
-rwx------  1 chtephan users 290 12. Jan 2004  ..
-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
--w-------  1 chtephan users   0 26. Aug 13:52 new
-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid

Some of them are reiser4 specific things (items, key, locality, oid)
which can only be queried anyway.

The rest is what you can also get using other VFS calls.

leto:/home/chtephan/.muttrc/metas/plugin > la
insgesamt 0
dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd

Here you can change reiser4 specific things. Here you can change some
properties how the file is stored in the reiser4 tree. For example you
can activate compression, encryption or authentication or set some hints
to optimize speed.

You can't create or remove these pseudo-files, just like in sysfs.

I don't know, but if someone wants to store user data I think you would
have to implement a directory "userdata" or something where you can
store it.

Not all of the information does make a lot sense inside a modified tar.
Who is interested in the bmap information of the file or some other
read-only information?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 3556 bytes --]

Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:

> > > I like cat < a > b. You can keep your progress.
> > 
> > cat <a >b does not preserve following file properties even on standard
> > UNIX filesystems: name,owner,group,permissions.
> 
> Losing permissions is one thing.  Annoying, mostly.
> 
> However, actual losing file data during such a copy is
> nothing short of a disaster, IMHO.  
> 
> In my opinion we shouldn't merge file-as-a-directory
> semantics into the kernel until we figure out how to
> fix the backup/restore problem and keep standard unix
> utilities work.

Well, again, what about xattrs and ACLs...?

Actually, reiser4 doesn't currently implement storage of arbitrary user
data under a file.

It's just some sysfs-like information that can be retrieved, some
properties can be changed. If you copy a file the worst thing that can
happen right now is that you lose the information whether that file
should be encrypted or compressed. You don't lose data.

In case some people are wondering, this is what you can find in the
reiser4 metas:

leto:/home/chtephan/.muttrc/metas > la
insgesamt 1
dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
-rwx------  1 chtephan users 290 12. Jan 2004  ..
-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
--w-------  1 chtephan users   0 26. Aug 13:52 new
-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid

Some of them are reiser4 specific things (items, key, locality, oid)
which can only be queried anyway.

The rest is what you can also get using other VFS calls.

leto:/home/chtephan/.muttrc/metas/plugin > la
insgesamt 0
dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd

Here you can change reiser4 specific things. Here you can change some
properties how the file is stored in the reiser4 tree. For example you
can activate compression, encryption or authentication or set some hints
to optimize speed.

You can't create or remove these pseudo-files, just like in sysfs.

I don't know, but if someone wants to store user data I think you would
have to implement a directory "userdata" or something where you can
store it.

Not all of the information does make a lot sense inside a modified tar.
Who is interested in the bmap information of the file or some other
read-only information?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Yury Umanets]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:
>
>  
>
>>>>I like cat < a > b. You can keep your progress.
>>>>        
>>>>
>>>cat <a >b does not preserve following file properties even on standard
>>>UNIX filesystems: name,owner,group,permissions.
>>>      
>>>
>>Losing permissions is one thing.  Annoying, mostly.
>>
>>However, actual losing file data during such a copy is
>>nothing short of a disaster, IMHO.  
>>
>>In my opinion we shouldn't merge file-as-a-directory
>>semantics into the kernel until we figure out how to
>>fix the backup/restore problem and keep standard unix
>>utilities work.
>>    
>>
>
>Well, again, what about xattrs and ACLs...?
>
>Actually, reiser4 doesn't currently implement storage of arbitrary user
>data under a file.
>
>It's just some sysfs-like information that can be retrieved, some
>properties can be changed. If you copy a file the worst thing that can
>happen right now is that you lose the information whether that file
>should be encrypted or compressed. You don't lose data.
>
>In case some people are wondering, this is what you can find in the
>reiser4 metas:
>
>leto:/home/chtephan/.muttrc/metas > la
>insgesamt 1
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
>-rwx------  1 chtephan users 290 12. Jan 2004  ..
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
>--w-------  1 chtephan users   0 26. Aug 13:52 new
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid
>
>Some of them are reiser4 specific things (items, key, locality, oid)
>which can only be queried anyway.
>
>The rest is what you can also get using other VFS calls.
>
>leto:/home/chtephan/.muttrc/metas/plugin > la
>insgesamt 0
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd
>
>Here you can change reiser4 specific things. Here you can change some
>properties how the file is stored in the reiser4 tree. For example you
>can activate compression, encryption or authentication or set some hints
>to optimize speed.
>  
>
I suppose that you are talking about the prinsiple ability to change 
these things. Because in real life it will also need some kind of 
convertion from one look (say with old plugin) to another one. Just like 
tail conversion rebuilds file's body after tail policy plugin gets changed.





-- 
umka

Re: silent semantic changes with reiser4

[Yury Umanets]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:
>
>  
>
>>>>I like cat < a > b. You can keep your progress.
>>>>        
>>>>
>>>cat <a >b does not preserve following file properties even on standard
>>>UNIX filesystems: name,owner,group,permissions.
>>>      
>>>
>>Losing permissions is one thing.  Annoying, mostly.
>>
>>However, actual losing file data during such a copy is
>>nothing short of a disaster, IMHO.  
>>
>>In my opinion we shouldn't merge file-as-a-directory
>>semantics into the kernel until we figure out how to
>>fix the backup/restore problem and keep standard unix
>>utilities work.
>>    
>>
>
>Well, again, what about xattrs and ACLs...?
>
>Actually, reiser4 doesn't currently implement storage of arbitrary user
>data under a file.
>
>It's just some sysfs-like information that can be retrieved, some
>properties can be changed. If you copy a file the worst thing that can
>happen right now is that you lose the information whether that file
>should be encrypted or compressed. You don't lose data.
>
>In case some people are wondering, this is what you can find in the
>reiser4 metas:
>
>leto:/home/chtephan/.muttrc/metas > la
>insgesamt 1
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
>-rwx------  1 chtephan users 290 12. Jan 2004  ..
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
>--w-------  1 chtephan users   0 26. Aug 13:52 new
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid
>
>Some of them are reiser4 specific things (items, key, locality, oid)
>which can only be queried anyway.
>
>The rest is what you can also get using other VFS calls.
>
>leto:/home/chtephan/.muttrc/metas/plugin > la
>insgesamt 0
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd
>
>Here you can change reiser4 specific things. Here you can change some
>properties how the file is stored in the reiser4 tree. For example you
>can activate compression, encryption or authentication or set some hints
>to optimize speed.
>  
>
I suppose that you are talking about the prinsiple ability to change 
these things. Because in real life it will also need some kind of 
convertion from one look (say with old plugin) to another one. Just like 
tail conversion rebuilds file's body after tail policy plugin gets changed.





-- 
umka

Re: silent semantic changes with reiser4

[Yury Umanets]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:
>
>  
>
>>>>I like cat < a > b. You can keep your progress.
>>>>        
>>>>
>>>cat <a >b does not preserve following file properties even on standard
>>>UNIX filesystems: name,owner,group,permissions.
>>>      
>>>
>>Losing permissions is one thing.  Annoying, mostly.
>>
>>However, actual losing file data during such a copy is
>>nothing short of a disaster, IMHO.  
>>
>>In my opinion we shouldn't merge file-as-a-directory
>>semantics into the kernel until we figure out how to
>>fix the backup/restore problem and keep standard unix
>>utilities work.
>>    
>>
>
>Well, again, what about xattrs and ACLs...?
>
>Actually, reiser4 doesn't currently implement storage of arbitrary user
>data under a file.
>
>It's just some sysfs-like information that can be retrieved, some
>properties can be changed. If you copy a file the worst thing that can
>happen right now is that you lose the information whether that file
>should be encrypted or compressed. You don't lose data.
>
>In case some people are wondering, this is what you can find in the
>reiser4 metas:
>
>leto:/home/chtephan/.muttrc/metas > la
>insgesamt 1
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
>-rwx------  1 chtephan users 290 12. Jan 2004  ..
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
>--w-------  1 chtephan users   0 26. Aug 13:52 new
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid
>
>Some of them are reiser4 specific things (items, key, locality, oid)
>which can only be queried anyway.
>
>The rest is what you can also get using other VFS calls.
>
>leto:/home/chtephan/.muttrc/metas/plugin > la
>insgesamt 0
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd
>
>Here you can change reiser4 specific things. Here you can change some
>properties how the file is stored in the reiser4 tree. For example you
>can activate compression, encryption or authentication or set some hints
>to optimize speed.
>  
>
I suppose that you are talking about the principle ability to change
these things. Because in real life it will also need some kind of
conversion from one look (say with old plugin) to another one. Just like
tail conversion rebuilds file's body after tail policy plugin gets changed.





-- 
umka

Re: silent semantic changes with reiser4

[Yury Umanets]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:
>
>  
>
>>>>I like cat < a > b. You can keep your progress.
>>>>        
>>>>
>>>cat <a >b does not preserve following file properties even on standard
>>>UNIX filesystems: name,owner,group,permissions.
>>>      
>>>
>>Losing permissions is one thing.  Annoying, mostly.
>>
>>However, actual losing file data during such a copy is
>>nothing short of a disaster, IMHO.  
>>
>>In my opinion we shouldn't merge file-as-a-directory
>>semantics into the kernel until we figure out how to
>>fix the backup/restore problem and keep standard unix
>>utilities work.
>>    
>>
>
>Well, again, what about xattrs and ACLs...?
>
>Actually, reiser4 doesn't currently implement storage of arbitrary user
>data under a file.
>
>It's just some sysfs-like information that can be retrieved, some
>properties can be changed. If you copy a file the worst thing that can
>happen right now is that you lose the information whether that file
>should be encrypted or compressed. You don't lose data.
>
>In case some people are wondering, this is what you can find in the
>reiser4 metas:
>
>leto:/home/chtephan/.muttrc/metas > la
>insgesamt 1
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
>-rwx------  1 chtephan users 290 12. Jan 2004  ..
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
>--w-------  1 chtephan users   0 26. Aug 13:52 new
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid
>
>Some of them are reiser4 specific things (items, key, locality, oid)
>which can only be queried anyway.
>
>The rest is what you can also get using other VFS calls.
>
>leto:/home/chtephan/.muttrc/metas/plugin > la
>insgesamt 0
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd
>
>Here you can change reiser4 specific things. Here you can change some
>properties how the file is stored in the reiser4 tree. For example you
>can activate compression, encryption or authentication or set some hints
>to optimize speed.
>  
>
I suppose that you are talking about the principle ability to change
these things. Because in real life it will also need some kind of
conversion from one look (say with old plugin) to another one. Just like
tail conversion rebuilds file's body after tail policy plugin gets changed.





-- 
umka

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> > cat <a >b does not preserve following file properties even on standard
> > UNIX filesystems: name,owner,group,permissions.
> 
> Losing permissions is one thing.  Annoying, mostly.

I find losing mtime annoying.  That's why I don't use Mozilla's
download manager (and Navigator's before it) because they presumed
that mtime isn't worth copying.  ncftp and wget do the right thing,
but it's annoying that I can't use the fancy tools and get the
metadata _I_ want transferred by them.

> However, actual losing file data during such a copy is
> nothing short of a disaster, IMHO.  

Note that file-as-directory doesn't imply that you can store just
anything into those directories.

Is it a problem to decree that "file data MUST NOT be stored in a
metadata directory; only non-essential metadata and data computed from
the file data may be stored"?

> In my opinion we shouldn't merge file-as-a-directory
> semantics into the kernel until we figure out how to
> fix the backup/restore problem and keep standard unix
> utilities work.

Same as xattrs, for data that should be stored.

But, unlike xattrs, while permissions and C-source-encoding should be
stored, some of this metadata should _not_ be stored when making an
archive and perhaps not when making a backup either.

Personally I wouldn't want an archive containing a file to save and
restore recomputable metadata like cached, extracted-on-demand MP3 id
tags and email headers-as-metadata.  I'd rather they were recalculated
when needed, implying a stronger guarantee about their relationship to
the data from which they're extracted - and implying that a user
program might not be allowed to set them to inconsistent values.

-- Jamie

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Jamie Lokier wrote:
> I find losing mtime annoying.  That's why I don't use Mozilla's
> download manager (and Navigator's before it) because they presumed
> that mtime isn't worth copying.

It's free software, you could fix that :)

> Is it a problem to decree that "file data MUST NOT be stored in a
> metadata directory; only non-essential metadata and data computed from
> the file data may be stored"?

That would put us right back at the samba-and-other-apps-want-streams
point in the discussion, but it certainly would not be a bad starting
point. From there we could always add streams later on if the API is
right.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 07:42, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Denis Vlasenko wrote:
> 
> > > I like cat < a > b. You can keep your progress.
> > 
> > cat <a >b does not preserve following file properties even on standard
> > UNIX filesystems: name,owner,group,permissions.
> 
> Losing permissions is one thing.  Annoying, mostly.
> 

cat <a >b does preserve permissions - the permissions of b.  I always
considered this a feature.

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 07:42, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Denis Vlasenko wrote:
> 
> > > I like cat < a > b. You can keep your progress.
> > 
> > cat <a >b does not preserve following file properties even on standard
> > UNIX filesystems: name,owner,group,permissions.
> 
> Losing permissions is one thing.  Annoying, mostly.
> 

cat <a >b does preserve permissions - the permissions of b.  I always
considered this a feature.

Lee

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>
>In my opinion we shouldn't merge file-as-a-directory
>semantics into the kernel until we figure out how to
>fix the backup/restore problem and keep standard unix
>utilities work.
>
>  
>
One incremental piece at a time our solution will fall into place.  
Pieces should go in when they are individually ready.  Warning users 
that some pieces might be changed out from under them up until when 2.8 
ships is ok with me though.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:
>
>What it breaks is the concept of a file. In ways that are ill-defined,
>not portable, hard to work with, and needlessly complex. Along the
>way, it breaks every single application that ever thought it knew what
>a file was.

It breaks the concept of a file. In ways that offer more versatility,
challenge the imagination to make even better progress and keeps
Linux competing with competitors who are implementing this stuff
as we speak.

I for one would truly welcome the coming of thumbnails and descriptions
in picture files, because I have a real-life project going on where
that would be extremely handy to have in the actual file.
Were I any richer, I'd pay Namesys to have this work for me :)

>Find some silly person with an iBook and open a shell on OS X. Use cp
>to copy a file with a resource fork. Oh look, the Finder has no idea
>what the new file is, even though it looks exactly identical in the
>shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
>fork. How is that ever going to work?

Then I guess OS X ships a broken implementation of cp, yes?

On the cat example, what if cat < a > b simply copies the "main stream"
and not the metadata, as a feature. The key being, "as a feature"

The metadata streams could get file descriptors of their own OR
another program, streamcat or something, could be written to compensate.

>I like cat < a > b. You can keep your progress.

With all due respect, I hope not too many people agree with you :)

-- 
mjt

Re: silent semantic changes with reiser4

[mjt]

On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:
>
>What it breaks is the concept of a file. In ways that are ill-defined,
>not portable, hard to work with, and needlessly complex. Along the
>way, it breaks every single application that ever thought it knew what
>a file was.

It breaks the concept of a file. In ways that offer more versatility,
challenge the imagination to make even better progress and keeps
Linux competing with competitors who are implementing this stuff
as we speak.

I for one would truly welcome the coming of thumbnails and descriptions
in picture files, because I have a real-life project going on where
that would be extremely handy to have in the actual file.
Were I any richer, I'd pay Namesys to have this work for me :)

>Find some silly person with an iBook and open a shell on OS X. Use cp
>to copy a file with a resource fork. Oh look, the Finder has no idea
>what the new file is, even though it looks exactly identical in the
>shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
>fork. How is that ever going to work?

Then I guess OS X ships a broken implementation of cp, yes?

On the cat example, what if cat < a > b simply copies the "main stream"
and not the metadata, as a feature. The key being, "as a feature"

The metadata streams could get file descriptors of their own OR
another program, streamcat or something, could be written to compensate.

>I like cat < a > b. You can keep your progress.

With all due respect, I hope not too many people agree with you :)

-- 
mjt

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:
>
>What it breaks is the concept of a file. In ways that are ill-defined,
>not portable, hard to work with, and needlessly complex. Along the
>way, it breaks every single application that ever thought it knew what
>a file was.

It breaks the concept of a file. In ways that offer more versatility,
challenge the imagination to make even better progress and keeps
Linux competing with competitors who are implementing this stuff
as we speak.

I for one would truly welcome the coming of thumbnails and descriptions
in picture files, because I have a real-life project going on where
that would be extremely handy to have in the actual file.
Were I any richer, I'd pay Namesys to have this work for me :)

>Find some silly person with an iBook and open a shell on OS X. Use cp
>to copy a file with a resource fork. Oh look, the Finder has no idea
>what the new file is, even though it looks exactly identical in the
>shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
>fork. How is that ever going to work?

Then I guess OS X ships a broken implementation of cp, yes?

On the cat example, what if cat < a > b simply copies the "main stream"
and not the metadata, as a feature. The key being, "as a feature"

The metadata streams could get file descriptors of their own OR
another program, streamcat or something, could be written to compensate.

>I like cat < a > b. You can keep your progress.

With all due respect, I hope not too many people agree with you :)

-- 
mjt

Re: silent semantic changes with reiser4

[Jan Harkes]

On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus   Törnqvist wrote:
> On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:
> >What it breaks is the concept of a file. In ways that are ill-defined,
> >not portable, hard to work with, and needlessly complex. Along the
> >way, it breaks every single application that ever thought it knew what
> >a file was.
> 
> It breaks the concept of a file. In ways that offer more versatility,
> challenge the imagination to make even better progress and keeps
> Linux competing with competitors who are implementing this stuff
> as we speak.

Sigh, it really isn't all that groundbreaking. The only special feature
is the fact that it introduces directory hardlinks.

Just turn the phrase 'file-as-dir' around and you might see the point.
As far as userspace is concerned the implementation would behave 99%
identical if we provided a 'dir-as-file' semantics. i.e. opening a
directory without 'O_DIRECTORY' would return a filehandle to a default
file within the directory. Something like open("/path/to/dir", O_RDONLY)
would be translated to open('/path/to/dir/_contents', O_RDONLY).

So I could simply replace all my files with directories and move the
original file contents into a '_contents' file. Applications can still
open them by the same name, although the attributes would look a bit
different. Poof, we have the same file-as-dir concept, but now we play
fair with the existing VFS and don't introduced races/deadlocks and
other issues. Also existing backup tools will be able to back up and
restore these 'streams'.

(btw. the same could be implemented completely in userspace, for
instance in glibc. Whenever an open call gets an EISDIR error, simply
retry the open, but this time with /_contents appended).

> I for one would truly welcome the coming of thumbnails and descriptions
> in picture files, because I have a real-life project going on where
> that would be extremely handy to have in the actual file.

Ehhh, that already exists and doesn't require meta-data streams,

    http://www.exif.org/

> >Find some silly person with an iBook and open a shell on OS X. Use cp
> >to copy a file with a resource fork. Oh look, the Finder has no idea
> >what the new file is, even though it looks exactly identical in the
> >shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
> >fork. How is that ever going to work?
> 
> Then I guess OS X ships a broken implementation of cp, yes?

No cp is working just fine, there is no copy syscall in the VFS, so any
file system specific features such meta data streams will have to be
handled by the application.

So at some point 'cp' will have to be taught about macos resource forks,
posix acls, xattrs, afs acls, reiserfs file-as-dir, freebsd attributes,
samba/ntfs streams. And possibly even learn the semantical differences
between these so that it can copy metadata between the various file
systems types.

Jan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jan Harkes wrote:
> 
> (btw. the same could be implemented completely in userspace, for
> instance in glibc. Whenever an open call gets an EISDIR error, simply
> retry the open, but this time with /_contents appended).

Yes, and no - just to make it obvious before people jump on this, a lot of 
things can be prototyped in user space with things like this, but once you 
have to deal with races and mixed tool environments, user space suddenly 
doesn't work so well any more.

I think Jan understands this distinction, I just wanted to make sure 
everybody else is aware of the _one_ thing that kernel land does well:

 - safely synchronize globally visible data structures

That's quite fundamental. 99% of what a kernel does is exactly that. TCP
would be in user space too, if it wasn't for _exactly_ this issue. A lot
of people think that kernels are about hardware access, and yes, that's
the other 99% of the picture (I see the _big_ picture, remember?), but the 
"safe access to common data" is really very fundamental.

The kernel is literally the thing that makes sure that you don't have -
and _cannot_ have - user programs that confuse each other by modifying 
data unsynchronized.

For example, a filesystem is really nothing but a way to access a disk in 
a controlled manner - it's not so much about hardware access, as it is 
about maintaining a coherent view of how some shared data (disk or 
whatever) is maintained. 

Same goes for caches. We could cache things in user space, but if you want 
to _share_ your caches (so that you don't have to re-load them for every 
new application), you need some entity that manages those shared data 
structures in a secure manner. In other words, you need the kernel.

The same goes for something like a "container file". Whether you see it as
"dir-as-file" or "file-as-dir" (and I agree with Jan that the two are
totally equivalent), the point of having the capability in the kernel is
not that the operations cannot be done in user space - the point is that
they cannot be done in user space _safely_. The kernel is kind of the
thing that guarantees that everybody follows the rules.

Imagine the security problems if a set-uid program were to (unwittingly) 
depend on a user-space library that implements what Jan's prototype 
library would do? Races galore, since a user-space implementation wouldn't 
have _any_ way to do tests like the above atomically. 

		Linus

Re: silent semantic changes with reiser4

[Jan Harkes]

I trimmed/adjusted the CC list somewhat in the hope of getting a more
productive discussion going.

On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
> On Thu, 26 Aug 2004, Jan Harkes wrote:
> > 
> > (btw. the same could be implemented completely in userspace, for
> > instance in glibc. Whenever an open call gets an EISDIR error, simply
> > retry the open, but this time with /_contents appended).
> 
> Yes, and no - just to make it obvious before people jump on this, a lot of 
> things can be prototyped in user space with things like this, but once you 
> have to deal with races and mixed tool environments, user space suddenly 
> doesn't work so well any more.

Agreed.

> The same goes for something like a "container file". Whether you see it as
> "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are
> totally equivalent), the point of having the capability in the kernel is

Only to some point. I was mostly trying to make the point that similar
functionality is already achievable within the current VFS framework.

However, files and directories still have some semantical differences,
and these are probably the points that we would have to address to get
Hans's efforts work well in cooperation with the Linux VFS.

We cannot atomically unlink a dir-as-file, but can unlink a file-as-dir.
In this case, the dcache seems to have no problems unlinking/unhashing a
non-empty directory, so I'm assuming that the unlink problem is mostly
enforced to avoid unreachable, but still allocated files in a file
system. (I'm sure that Viro can probably come up with other reasons why
unlinking a non-empty directory can be evil).

Also, we cannot create hardlinks to a directory (-as-file), but I guess
reiserfs4 allows hardlinks to files (-as-directory). Luckily as far as
hardlinked directories are concerned the file-as-dir implementation
should be a relatively benign subset. I assume that we are not allowed
to rename or link arbitrary objects from the regular fs into it, Coda
returns EXDEV in such situations. As a result there will be no cycles
and this simple premise should get rid of at least a couple of the
possible deadlock scenarios that I've seen mentioned.

Jan

Re: silent semantic changes with reiser4

[Tom Vier]

On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
>  - safely synchronize globally visible data structures
> That's quite fundamental. 99% of what a kernel does is exactly that. TCP
> would be in user space too, if it wasn't for _exactly_ this issue. A lot

What about microkernels? They do tcp in userspace. So did winsock, iirc. As
long as a trusted process keeps data such as free ports, what's the problem?
Perhaps a process could even simply hand off proto:port handles and let a
library do the rest, but i know little about exokernels. (There's probably
some catches.)

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Mon, 30 Aug 2004, Tom Vier wrote:

> On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
> >  - safely synchronize globally visible data structures
> > That's quite fundamental. 99% of what a kernel does is exactly that. TCP
> > would be in user space too, if it wasn't for _exactly_ this issue. A lot
> 
> What about microkernels? They do tcp in userspace.

No they don't. They do TCP in a separate address space from user space, 
that just also happens to be separate from the "microkernel address 
space".

So a microkernel will have _more_ address spaces, and they won't be "user
space". They'll be "server deamon space" or something. Now, that's also
why they tend to have performance problems - because you need to copy the
data between different address spaces, and switch the CPU context etc
around.

Not user space. They may be "ring 3" from a CPU standpoint, but they 
aren't user space from a _user_ standpoint - it's still very much a 
separate address space, with domain protection.

> So did winsock, iirc.

Now that is a different case. Things like the PalmOS TCP stack (and, I
believe, Winsock) are true "user space" TCP stacks, in that they really
do live as libraries in the same address space as the user app.

It sucks. Exactly because now the data structures are _not_ protected, and
they are _not_ shared. So your library basically ends up being a "single
client" library, with no protection between clients (or no sharing: you
can have "protected" single-stream TCP, but then you won't share the TCP
state that needs to be shared like listen queues etc).

This works in an environment like Palm or Win-3.0, which are really just
single-client _anyway_, without any protection. But notice how Windows
started doing TCP in the kernel, and notice how you can actually use it as
a server these days?

In short: you _need_ to have a separate address space (either kernel, or
"TCP server" or whatever) if you want to have reliable, secure and
generally usable TCP.


> As long as a trusted process keeps data such as free ports, what's the
> problem?

None - because it's not user space any more. 

Well, performance might still suck, of course. And it does. 

		Linus

Re: silent semantic changes with reiser4

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Mon, 30 Aug 2004, Tom Vier wrote:
> 
> 
>>On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
>>
>>> - safely synchronize globally visible data structures
>>>That's quite fundamental. 99% of what a kernel does is exactly that. TCP
>>>would be in user space too, if it wasn't for _exactly_ this issue. A lot
>>
>>What about microkernels? They do tcp in userspace.
> 
> 
> No they don't. They do TCP in a separate address space from user space, 
> that just also happens to be separate from the "microkernel address 
> space".
> 
> So a microkernel will have _more_ address spaces, and they won't be "user
> space". They'll be "server deamon space" or something. Now, that's also
> why they tend to have performance problems - because you need to copy the
> data between different address spaces, and switch the CPU context etc
> around.


Man, this thread came a long way.  I peek in and see a pet topic...


_My_ own definition of microkernel, which differs from traditional CS, 
is where I would love to see Linux go:  move as much as humanly possible 
to userspace, such that, the kernel only contains pagecache operations, 
driver fast paths, and security checks.  Move slow paths, including ACPI 
probing, PCI bus walking, driver init/reset paths, some of the ioctl 
handling, to userspace.  Be willing to accept extra context switches as 
a cost in all but the fast paths.

What you have left after you move all the slow paths to userspace is my 
version of a microkernel.  The kernel is still monolithic and a single 
address space, but a lot smaller.

Now this (a) is likely just a pipe dream and (b) will increase 
complexity because each driver will _require_ a userspace component, but 
hey... you gotta have goals in life.

	Jeff

Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 12:38:14AM -0400, Jeff Garzik wrote:
> _My_ own definition of microkernel, which differs from traditional CS, 
> is where I would love to see Linux go:  move as much as humanly possible 
> to userspace, such that, the kernel only contains pagecache operations, 
> driver fast paths, and security checks.  Move slow paths, including ACPI 
> probing, PCI bus walking, driver init/reset paths, some of the ioctl 
> handling, to userspace.  Be willing to accept extra context switches as 
> a cost in all but the fast paths.

	http://www.dragonflybsd.org

DragonFly BSD, the only remotely functional open source BSD project on this
planet, has plans in place to push certain kernel components like their VFS
layer into userspace for easier debugging, testing and other things that go
with developing file systems easily. If they back it with something like C++
for doing constructor style type conversion on top of overloaded operators
to back VFS data structure translation, could easily import stuff like most
Linux file systems without major restructuring, say, if they had their
translation library written. In this case, userspace kernel systems have
some serious programming advantages over traditional kernels.

They're also pushing an async syscall model to support a non-1:1 threading
system for userspace unlike what Linux has done with futexes. It'll allow
for them to plug in various userspace schedulers at will for whatever
scheduling policy you want dynamically, even while the program is running.

This isn't mentioning their single system image clustering goals that have
abstracted various kernel systems like their VFS stuff to a messaging model
which should get them proccess migration as an intrinsic part of their OS
design. They've made a lot of progress over the last year and it's largely
because of Matt Dillon's ability to create a non-overly political group that
creates trust. He's a true leader, not some random loud mouth(s) unlike
other BSD efforts. It's much like the Linux community in this regard with
Linus.

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Jeff Garzik]

Bill Huey (hui) wrote:
> DragonFly BSD, the only remotely functional open source BSD project on this
> planet, has plans in place to push certain kernel components like their VFS
> layer into userspace for easier debugging, testing and other things that go

That violates Jeff's First Rule, put the fast path stuff in the kernel.


> with developing file systems easily. If they back it with something like C++
> for doing constructor style type conversion on top of overloaded operators
> to back VFS data structure translation, could easily import stuff like most
> Linux file systems without major restructuring, say, if they had their
> translation library written. In this case, userspace kernel systems have
> some serious programming advantages over traditional kernels.

Sounds like security would be a pain in the ass :)


> They're also pushing an async syscall model to support a non-1:1 threading
> system for userspace unlike what Linux has done with futexes. It'll allow

messaging passing is also known as "really slow C function calls"

My PCI bus passes messages all the time.  A message is in the eye of the 
beholder.

	Jeff

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 01:56:45AM -0400, Jeff Garzik wrote:
> Bill Huey (hui) wrote:
> >DragonFly BSD, the only remotely functional open source BSD project on this
> >planet, has plans in place to push certain kernel components like their VFS
> >layer into userspace for easier debugging, testing and other things that go
> 
> That violates Jeff's First Rule, put the fast path stuff in the kernel.

It's not for native file system, but one that are coming around. I kind
of misspoke when I said that the VFS layer is going to be pushed into
userspace. It's not. Access to the VFS layer is going to be exported to
userspace. The system is interesting because of its potential programming
flexibility.

> >with developing file systems easily. If they back it with something like 
> >C++
> >for doing constructor style type conversion on top of overloaded operators
> >to back VFS data structure translation, could easily import stuff like most
> >Linux file systems without major restructuring, say, if they had their
> >translation library written. In this case, userspace kernel systems have
> >some serious programming advantages over traditional kernels.
> 
> Sounds like security would be a pain in the ass :)

Don't know. But if you can get away from the typical C programming errors, then
it could be a win.

> >They're also pushing an async syscall model to support a non-1:1 threading
> >system for userspace unlike what Linux has done with futexes. It'll allow
> 
> messaging passing is also known as "really slow C function calls"
> 
> My PCI bus passes messages all the time.  A message is in the eye of the 
> beholder.

Message passing is used as a method of concurrency in that system outside
of the typical use of function calles. It makes access to kernel facilities,
remote or local, transparent. They also don't use a mutex for their locking,
but something called a token which is per-CPU kind of locking primitive. It's
definitely an interesting system with a lot of potential.

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 03:56:13PM +0200, Diego Calleja wrote:
> El Tue, 31 Aug 2004 01:56:45 -0400 Jeff Garzik <jgarzik@pobox.com> escribi?:
> > messaging passing is also known as "really slow C function calls"
> > 
> > My PCI bus passes messages all the time.  A message is in the eye of the 
> > beholder.
> 
> There're some numbers for that:
> 
> http://lists.freebsd.org/pipermail/freebsd-hackers/2003-August/002426.html

As you can see the numbers are very fast for a general purpose system
like that. Add that with their XIO framework for data propagation and
you have a really amazingly fast system that combines both aggressive
design and optimized low level facilities for both remote and local
commuication.

Matt's one of the very few open source engineers on this planet that
can do both well.

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Diego Calleja]

El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <bhuey@lnxw.com> escribió:

> As you can see the numbers are very fast for a general purpose system
> like that. Add that with their XIO framework for data propagation and

For the syscall case, I remember that they would be able to "batch" syscalls,
so this framework could have even better performance in some cases.
The good thing is that they didn't do it just for the sake of doing it
(like microkernel people) but to achieve their goals (SSI, etc)

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Thu, Sep 02, 2004 at 07:20:58PM +0200, Diego Calleja wrote:
> El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <bhuey@lnxw.com> escribi?:
> 
> > As you can see the numbers are very fast for a general purpose system
> > like that. Add that with their XIO framework for data propagation and
> 
> For the syscall case, I remember that they would be able to "batch" syscalls,
> so this framework could have even better performance in some cases.
> The good thing is that they didn't do it just for the sake of doing it
> (like microkernel people) but to achieve their goals (SSI, etc)

For what they're doing, they have minimize the overhead cross boundaries
as much as possible for usage in both user and kernel spaces. If dfBSD
achieves what they intend to do, they'll have on of the best SSI systems
on the planet and it'll be able to use the VFS layer in ways that other
systems can't even touch with iSCSI drivers, etc...

They've got process suspend and restore already in order to support
serialization across machines. Having it VFS driven, VM objects, etc...
makes the concept of migration a function of the VFS layer. It's quite
clever and I'm sure that it's driven by Matt's experience with distributed
databases, specifically Backplane.

All locking in his system is CPU local, no mutexes of any sort, so
migration as a function of messaging, scheduling and other things are
done in per-CPU chunks. It's quite clever make handling in per-process
resources chunks.

	http://twiki.im.ufba.br/pub/MAT154/WebHome/duality78.pdf

Talks about part of this and how "procedural" is isomorphic to "message-passing"
concurrency system. One can be transformed to another.

It's a project that's move much faster in the area of SMP than FreeBSD 5.x
(Solaris style SMP) and really the only BSD project on this planet that has
the talent and social chemistry to make it happen.

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 03:56:13PM +0200, Diego Calleja wrote:
> There're some numbers for that:
> 
> http://lists.freebsd.org/pipermail/freebsd-hackers/2003-August/002426.html

Link about XIO and things hijacked from FreeBSD-current.

http://leaf.dragonflybsd.org/mailarchive/kernel/2004-03/msg00469.html

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

Salut,

On Mon, Aug 30, 2004 at 10:30:55PM -0700, Bill Huey wrote:
> DragonFly BSD, the only remotely functional open source BSD project on this
> planet, has plans in place to push certain kernel components like their VFS
> layer into userspace for easier debugging, testing and other things that go
> with developing file systems easily. If they back it with something like C++
> for doing constructor style type conversion on top of overloaded operators
> to back VFS data structure translation, could easily import stuff like most
> Linux file systems without major restructuring, say, if they had their
> translation library written. In this case, userspace kernel systems have
> some serious programming advantages over traditional kernels.

Uh, what about enhancing lufs code in this regard and porting all fs'es to it?

				Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Userspace file systems

[Luca Ferroni]

Il Tue, 31 Aug 2004 11:01:58 +0200,  Tonnerre <tonnerre@thundrix.ch> ha scritto:

> Uh, what about enhancing lufs code in this regard and porting all fs'es to it?
> 

I think that porting all file systems to LUFS is a mad idea :), 
but providing a facility in the kernel for user-space file system 
implementations could be great.

I am working on PackageFS (http://packagefs.sourceforge.net)
which is an user-space file system to manage packages:
it relies on LUFS, but I plan to port it soon to FUSE ( http://sourceforge.net/projects/avf)
that, at present, is maintained by Miklos Szeredi and works with latest kernels.

PackageFS relies also on existent package managers, it will be transparent to all
package formats. I. e.: you can install a package by copying, 
and see package informations by a simple cat.
Now it works only in Debian distro, and it is far from complete (it is 0.09 version :) ).

So, what do you think about including a kernel module to make a user able 
to develop, install and use an user-space file system ?

Maybe Miklos Szeredi will write more details about FUSE which is the interesting part
for kernel inclusion.

Bye
Luca

-- 
-----------------------------------------------
They'll never stole us our.... FREEDOM!!!
Luca Ferroni
ICQ #317977679
www.cs.unibo.it/~fferroni/
-----------------------------------------------

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 11:01:58AM +0200, Tonnerre wrote:
> Salut,
> 
> On Mon, Aug 30, 2004 at 10:30:55PM -0700, Bill Huey wrote:
...
> > translation library written. In this case, userspace kernel systems have
> > some serious programming advantages over traditional kernels.
> 
> Uh, what about enhancing lufs code in this regard and porting all fs'es to it?

I wasn't aware that your project has gotten this far. I'll tell Hiten
about this to see if some of this can be used in their project.

Thanks

bill

Re: silent semantic changes with reiser4

[Ryan Breen]

On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote:
> 
> Man, this thread came a long way.
> 

You said it -- from Reiser to microkernel.  If we can only figure out
a way to get a BitKeeper discussion going, we'll have the Grand
Unified Flamewar.

Re: silent semantic changes with reiser4

[Jeff Garzik]

Ryan Breen wrote:
> On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote:
> 
>>Man, this thread came a long way.
>>
> 
> 
> You said it -- from Reiser to microkernel.  If we can only figure out
> a way to get a BitKeeper discussion going, we'll have the Grand
> Unified Flamewar.


If you can find a link between the reiserfs thread and the webcam driver 
thread, you've gone all the way from filesystems to Nazis, thus invoking 
Godwin's Law ;-)

	Jeff


[1] http://en.wikipedia.org/wiki/Godwin's_law

Re: silent semantic changes with reiser4

[Clay Haapala]

On Tue, 31 Aug 2004, Ryan Breen verbalised:
> On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote:
>> 
>> Man, this thread came a long way.
>> 
> 
> You said it -- from Reiser to microkernel.  If we can only figure out
> a way to get a BitKeeper discussion going, we'll have the Grand
> Unified Flamewar.

OK, I'll byte: consider file versioning of an SCM implemented as a
ReiserFS 4 plug-in, as in:

diff -u ultrastore.c/delta/3.2.3 ultrastore.c/delta/3.2.4

(Brrr!  Just had a VMS flash-back shiver.)
-- 
Clay Haapala (chaapala@cisco.com) Cisco Systems Storage +1 763-398-1056
   6450 Wedgwood Rd, Suite 130 Maple Grove MN 55311 PGP: C89240AD
 The mysterious blob seen in satellite photos off the east coast of Florida
 during the passing of Hurricane Charley has now been positively identified
 as a large cloud of Democrat absentee-ballot applications and voter lists.

Re: silent semantic changes with reiser4

[Alan Cox]

On Maw, 2004-08-31 at 05:08, Linus Torvalds wrote:
> > What about microkernels? They do tcp in userspace.
> 
> No they don't. They do TCP in a separate address space from user space, 
> that just also happens to be separate from the "microkernel address 
> space".

Several do TCP in user space. The only thing you need in kernel for
TCP/IP is enough decode to decide who gets the packet. Even some non
microkernel embedded OS's do this in order to keep kernel size down.

Alan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Alan Cox wrote:
> 
> Several do TCP in user space. The only thing you need in kernel for
> TCP/IP is enough decode to decide who gets the packet.

Only thing? I don't think so. 

You also want to make sure that regular users cannot send "impossible" 
packets. Think about the old "ping of death" kind of thing, where a normal 
mis-behaving (and I'm not saying intentionally so: it might be a small bug 
that just overwrites some data) program should _not_ be able to cause 
problems on the network.

Admins absolutely _hate_ that. They will ban an OS if it sends out packets
that cause troublem. You should remember that - we used to do strange
things on the net (long long time ago), and we brought down servers by
mistake, and nobody ever considered it a server bug: it was a Linux bug
that it wouldn't do the right thing.

Things like not sending FIN-packets when a program suddenly goes away is 
NOT acceptable behaviour! Neither is it acceptable behaviour to allow user 
programs to make up their own packets.

> Even some non microkernel embedded OS's do this in order to keep kernel
> size down.

..and I'm not disagreeing that it doesn't happen. I explicitly mentioned 
PalmOS, I bet it happens in other cases too. But I'd strongly argue that 
it's a bug, not a feature.

It's a bug that people tend to accept in a "single-client" environment. 

NOTE! This is totally ignoring the fact that you can't be called "UNIX" 
any more. You _need_ to have sequence numbers etc be shared between 
multiple programs that all write to the stream. Again, that _does_ mean 
that you have another protection domain (aka "kernel" or "TCP deamon") 
that keeps track of the sequence number. 

		Linus

Re: silent semantic changes with reiser4

[Frank van Maarseveen]

On Tue, Aug 31, 2004 at 10:15:25AM -0700, Linus Torvalds wrote:
> 
> Admins absolutely _hate_ that. They will ban an OS if it sends out packets
> that cause troublem. You should remember that - we used to do strange
> things on the net (long long time ago), and we brought down servers by
> mistake, and nobody ever considered it a server bug: it was a Linux bug
> that it wouldn't do the right thing.
> 
> Things like not sending FIN-packets when a program suddenly goes away is 
> NOT acceptable behaviour! Neither is it acceptable behaviour to allow user 
> programs to make up their own packets.

The user/kernel distinction is not always (heck, maybe almost never)
present in the embedded world (a large world). The notion of a "regular
user" does not apply at all in such a case. This is not a bug but merely
the state of technology. It will slowly go away I think because embedded
software becomes more complex and hardware becomes cheaper.

To implement multiple TCP clients _and_ the TCP/IP stack in one space is
perfectly possible and it's actually done in practice. It has advantages
(speed, when done properly) and disadvantages (complexity/bug-prone).

> 
> NOTE! This is totally ignoring the fact that you can't be called "UNIX" 
> any more. You _need_ to have sequence numbers etc be shared between 
> multiple programs that all write to the stream. Again, that _does_ mean 
> that you have another protection domain (aka "kernel" or "TCP deamon") 
> that keeps track of the sequence number. 

There is nothing in the networking or UNIX standards that prescibe another
protection domain for this. Would be insane to leave that out in a hosted
environment but it _can_ be done without.

-- 
Frank

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Frank van Maarseveen wrote:
> 
> There is nothing in the networking or UNIX standards that prescibe another
> protection domain for this. Would be insane to leave that out in a hosted
> environment but it _can_ be done without.

My point is that TCP _does_ have a lot of state that needs to be handled 
in a safe manner by a proper operating system.

The fact that there are OS's out there that are crap doesn't change that 
matter. There are lots of embedded OS's out there that still do 
multitasking in a purely cooperative way. I don't think it's a valid model 
for anything but toys. Same goes for putting TCP in user space. It's 
doable, but it's not an "OS". It's a program loader.

		Linus

Re: silent semantic changes with reiser4

[Toon van der Pas]

On Mon, Aug 30, 2004 at 09:08:07PM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 30 Aug 2004, Tom Vier wrote:
> 
> > On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
> > >  - safely synchronize globally visible data structures
> > > That's quite fundamental. 99% of what a kernel does is exactly
> > > that. TCP would be in user space too, if it wasn't for
> > > _exactly_ this issue. A lot
> > 
> > What about microkernels? They do tcp in userspace.
> 
> No they don't. They do TCP in a separate address space from user
> space, that just also happens to be separate from the "microkernel
> address space".

The HP NonStop Server (formerly called Tandem) has an MPP architecture
and a message passing OS (the NonStop Kernel). On this OS the TCP/IP
stack is implemented as a library against which the application
software is linked. The ethernet controllers/drivers maintain a
routing table for routing incoming traffic to the right process.
So the TCP/IP stack is implemented in userspace on the NonStop
Server, I would say.

Regards,
Toon.
-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Jan" == Jan Harkes <jaharkes@cs.cmu.edu> writes:

Jan> On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus   Törnqvist wrote:

[...]

>> I for one would truly welcome the coming of thumbnails and
>> descriptions in picture files, because I have a real-life project
>> going on where that would be extremely handy to have in the actual
>> file.

Jan> Ehhh, that already exists and doesn't require meta-data streams,

Jan>     http://www.exif.org/

but then you need an extra library to access the exif data.  You can't
just do something like "display foo.jpg/exif/thumbnail".  Or even
better, "echo 'this is a comment' > foo.jpg/comment" to change the jpeg
comment without an external program.

And I don't think exif tags work with all image formats.

>> On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:
>> >Find some silly person with an iBook and open a shell on OS X. Use
>> >cp to copy a file with a resource fork. Oh look, the Finder has no
>> >idea what the new file is, even though it looks exactly identical in
>> >the shell. Isn't that _wonderful_? Now try cat < a > b on a file
>> >with a fork. How is that ever going to work?

Under file-as-dir, it would not look the same under the shell if cp
didn't copy everything -- the directory contents would be different.
To copy the metadata as well, you would do something like "cp -r".
Similarly with "cat < a > b".

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Jan" == Jan Harkes <jaharkes@cs.cmu.edu> writes:

Jan> On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus   Törnqvist wrote:

[...]

>> I for one would truly welcome the coming of thumbnails and
>> descriptions in picture files, because I have a real-life project
>> going on where that would be extremely handy to have in the actual
>> file.

Jan> Ehhh, that already exists and doesn't require meta-data streams,

Jan>     http://www.exif.org/

but then you need an extra library to access the exif data.  You can't
just do something like "display foo.jpg/exif/thumbnail".  Or even
better, "echo 'this is a comment' > foo.jpg/comment" to change the jpeg
comment without an external program.

And I don't think exif tags work with all image formats.

>> On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:
>> >Find some silly person with an iBook and open a shell on OS X. Use
>> >cp to copy a file with a resource fork. Oh look, the Finder has no
>> >idea what the new file is, even though it looks exactly identical in
>> >the shell. Isn't that _wonderful_? Now try cat < a > b on a file
>> >with a fork. How is that ever going to work?

Under file-as-dir, it would not look the same under the shell if cp
didn't copy everything -- the directory contents would be different.
To copy the metadata as well, you would do something like "cp -r".
Similarly with "cat < a > b".

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Helge Hafting]

Matt Mackall wrote:
[...]

>What it breaks is the concept of a file. In ways that are ill-defined,
>not portable, hard to work with, and needlessly complex. Along the
>way, it breaks every single application that ever thought it knew what
>a file was.
>
>Progress? No, this has been done before. Various dead operating
>systems have done it or similar and regretted it. Most recently MacOS,
>which jumped through major hurdles to begin purging themselves of
>resource forks when they went to OS X. They're still there, but
>heavily deprecated.
>
>  
>
>>cp, grep, cat, and tar will continue to work just fine on files with
>>multiple streams.
>>    
>>
>
>Find some silly person with an iBook and open a shell on OS X. Use cp
>to copy a file with a resource fork. Oh look, the Finder has no idea
>what the new file is, even though it looks exactly identical in the
>shell. Isn't that _wonderful_? 
>
It is what I'd expect.  Now, use cp -R to copy  the file
_with its directory_, and see if that fares better.  If not - bad
implementation of fs and/or cp.  The way I see file-as -directory
is that _file_ operations (like the reads issued by cat) only
work on the file part.  You want the directory part?  Use
directory operations such as those "cp -R" use.

>Now try cat < a > b on a file with a
>fork. How is that ever going to work?
>  
>
It is going to copy the _file_ part, of course.  I wouldn't
expect anything else - "cat" doesn't deal with directories.

This also raise the question of when to use file-as-directory.
A usage where you need everything to follow the file, even
when using "cat" calls out for an application that puts everything
into one file.  Directory-as-file is the wrong tool for that job, so don't
worry about such problems.

Sticking thumbnails in a file-as-dir is another story though.  Move the
file (with "mv a b", not  "cat a>b;rm a") moves the image file _and_ the
thumbnail.  No time wasted on regenerating thumbnails, no disk space
or cleanup time wasted on dangling thumbs.  Use the image file
for its intended purposes (view it, mail it off, serve it on the web,
edit it, print it) and the thumbnail doesn't get in the way
because it isn't embedded in the file format.  Embedding it
in the file might work for jpeg which support generic embedded data,
it surely won't work for every image format out there.

This is my idea of how file-as-directory should work.

Helge Hafting


Helge Hafting

Re: silent semantic changes with reiser4

[Helge Hafting]

Matt Mackall wrote:
[...]

>What it breaks is the concept of a file. In ways that are ill-defined,
>not portable, hard to work with, and needlessly complex. Along the
>way, it breaks every single application that ever thought it knew what
>a file was.
>
>Progress? No, this has been done before. Various dead operating
>systems have done it or similar and regretted it. Most recently MacOS,
>which jumped through major hurdles to begin purging themselves of
>resource forks when they went to OS X. They're still there, but
>heavily deprecated.
>
>  
>
>>cp, grep, cat, and tar will continue to work just fine on files with
>>multiple streams.
>>    
>>
>
>Find some silly person with an iBook and open a shell on OS X. Use cp
>to copy a file with a resource fork. Oh look, the Finder has no idea
>what the new file is, even though it looks exactly identical in the
>shell. Isn't that _wonderful_? 
>
It is what I'd expect.  Now, use cp -R to copy  the file
_with its directory_, and see if that fares better.  If not - bad
implementation of fs and/or cp.  The way I see file-as -directory
is that _file_ operations (like the reads issued by cat) only
work on the file part.  You want the directory part?  Use
directory operations such as those "cp -R" use.

>Now try cat < a > b on a file with a
>fork. How is that ever going to work?
>  
>
It is going to copy the _file_ part, of course.  I wouldn't
expect anything else - "cat" doesn't deal with directories.

This also raise the question of when to use file-as-directory.
A usage where you need everything to follow the file, even
when using "cat" calls out for an application that puts everything
into one file.  Directory-as-file is the wrong tool for that job, so don't
worry about such problems.

Sticking thumbnails in a file-as-dir is another story though.  Move the
file (with "mv a b", not  "cat a>b;rm a") moves the image file _and_ the
thumbnail.  No time wasted on regenerating thumbnails, no disk space
or cleanup time wasted on dangling thumbs.  Use the image file
for its intended purposes (view it, mail it off, serve it on the web,
edit it, print it) and the thumbnail doesn't get in the way
because it isn't embedded in the file format.  Embedding it
in the file might work for jpeg which support generic embedded data,
it surely won't work for every image format out there.

This is my idea of how file-as-directory should work.

Helge Hafting


Helge Hafting

Re: silent semantic changes with reiser4

[Horst von Brand]

Helge Hafting <helge.hafting@hist.no> said:
> Matt Mackall wrote:

[...]

> >Find some silly person with an iBook and open a shell on OS X. Use cp
> >to copy a file with a resource fork. Oh look, the Finder has no idea
> >what the new file is, even though it looks exactly identical in the
> >shell. Isn't that _wonderful_? 

> It is what I'd expect.

Great. That means that all the tools of the trade stop working. Sounds like
show-killer feature to me.

>                         Now, use cp -R to copy  the file
> _with its directory_,

Either it is a file or a directory. Make up your mind. If you have no clear
distinction, you'll only get messed up. Badly.

>                       and see if that fares better.  If not - bad
> implementation of fs and/or cp.  The way I see file-as -directory
> is that _file_ operations (like the reads issued by cat) only
> work on the file part.  You want the directory part?  Use
> directory operations such as those "cp -R" use.

Excuse me, I must grab my sickness bag here.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Helge Hafting]

On Thu, Aug 26, 2004 at 02:17:03PM -0400, Horst von Brand wrote:
> Helge Hafting <helge.hafting@hist.no> said:
> > Matt Mackall wrote:
> 
> [...]
> 
> > >Find some silly person with an iBook and open a shell on OS X. Use cp
> > >to copy a file with a resource fork. Oh look, the Finder has no idea
> > >what the new file is, even though it looks exactly identical in the
> > >shell. Isn't that _wonderful_? 
> 
> > It is what I'd expect.
> 
> Great. That means that all the tools of the trade stop working. Sounds like
> show-killer feature to me.
>
All wrong.  Feel free to dislike the concept all you want, but don't provide 
false arguments.  I am sure you can think up some _good_ ones, if
you feel so strongly about this.

_No_ tool "stop working" with such a fs.  Of course the new
feature (file-as-directory) may not be supported - which is a problem
only for files that use the new feature.  Stick a lot of ordinary
files and directories in such a fs, and the tools works as before
with no breakage.  People obviously won't deploy new features before
they are useable - the fs have to come first.  Then the tools,
and widespread use only after that.  This is similiar to other extensions 
like ACLs which many tools don't understand.  They're still useable though.

Of course file-as-directory can have pilot uses like the often
mentioned thumbnails - because it doesn't matter if those aren't
backed up or get lost at random during the experimental phase.

 
> >                         Now, use cp -R to copy  the file
> > _with its directory_,
> 
> Either it is a file or a directory. Make up your mind. 

Wrong again.  The file-as-directory concept means that
a filename refers to something that is a file, or a directory,
or both.  That's the point of it.  Being both isn't
possible with current filesystems (except reiser4) of course, 
but file-as-directory is a proposed _extension_.  

> If you have no clear
> distinction, you'll only get messed up. Badly.
>
The distinction is clear enough - three options instead of two.
 
> Excuse me, I must grab my sickness bag here.

Oh, very impressive.  Please take a look at what we're talking
about before attacking.   

Helge Hafting

Re: silent semantic changes with reiser4

[Spam]

>> cp, grep, cat, and tar will continue to work just fine on files with
>> multiple streams.

> Find some silly person with an iBook and open a shell on OS X. Use cp
> to copy a file with a resource fork. Oh look, the Finder has no idea
> what the new file is, even though it looks exactly identical in the
> shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
> fork. How is that ever going to work?

  It  should  work  on  the main file, the extra streams and meta data
  won't be copied.  Just as it is now with special attributes etc. Cat
  reads the content of a file stream, not the file container.

> I like cat < a > b. You can keep your progress.

Re: silent semantic changes with reiser4

[Spam]

>> cp, grep, cat, and tar will continue to work just fine on files with
>> multiple streams.

> Find some silly person with an iBook and open a shell on OS X. Use cp
> to copy a file with a resource fork. Oh look, the Finder has no idea
> what the new file is, even though it looks exactly identical in the
> shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
> fork. How is that ever going to work?

  It  should  work  on  the main file, the extra streams and meta data
  won't be copied.  Just as it is now with special attributes etc. Cat
  reads the content of a file stream, not the file container.

> I like cat < a > b. You can keep your progress.

Re: silent semantic changes with reiser4

[Joseph Fannin]

[-- Attachment #1: Type: text/plain, Size: 2248 bytes --]

On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:

> What it breaks is the concept of a file. In ways that are ill-defined,
> not portable, hard to work with, and needlessly complex. Along the
> way, it breaks every single application that ever thought it knew what
> a file was.
> 
> Progress? No, this has been done before. Various dead operating
> systems have done it or similar and regretted it. Most recently MacOS,
> which jumped through major hurdles to begin purging themselves of
> resource forks when they went to OS X. They're still there, but
> heavily deprecated.
> 
> > cp, grep, cat, and tar will continue to work just fine on files with
> > multiple streams.
> 
> Find some silly person with an iBook and open a shell on OS X. Use cp
> to copy a file with a resource fork. Oh look, the Finder has no idea
> what the new file is, even though it looks exactly identical in the
> shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
> fork. How is that ever going to work?

   I've been known to be silly now and then, for sure.  Apple says
this about the next version of OS X, due next year:

"Tiger provides a standard, Darwin-level API for managing resource
forks, filesystem metadata, security information, properties and other
attributes in a consistent, cross-platform manner. For example, common
UNIX utilities such as cp, tar and rsync can properly handle HFS+
resource forks."

    That's marketing copy, but I think it's worth noting.  It seems to
be that resource forks aren't deprecated across the board,
just... changed.

    Apple has found a way to make this work, at least partially.  I
don't know if it's a good way, but there it is.

    I'd like to be able to keep my Macintosh stuff on my Linux
fileserver, without losing the resource forks.  That would require
that the network fs support it of course, and whether I could use tar
to back up my linux box would depend on how streams might be
implemented there.

> I like cat < a > b. You can keep your progress.

    Can't we have both?

-- 
Joseph Fannin
jhf@rivenstone.net

"Bull in pure form is rare; there is usually some contamination by data."
    -- William Graves Perry Jr.

[-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --]

Re: silent semantic changes with reiser4

[Paul Jackson]

> "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all
> the time and are fixed. It's called progress.

Yes - we break things all the time.  But we take notice, when we are
breaking things, of how long standing and deeply embedded they are.

The deeper the roots, the more respect we show it, and the harder
it will be to change.

Heaping scorn on someone who is reluctant to change something as
deeply embedded as "a file is a byte stream" does not further the
discussion.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Spam]

> cp, grep, cat, and tar will continue to work just fine on files with
> multiple streams.

> tar and cp will lose the extra streams until somebody fixes them, but
> they lose ACLs and xattrs right now, and I don't hear anybody suggesting
> that ACLs and xattrs be removed from the kernel because of this.

  I  thought  that  the  underlaying  FS/VFS would make sure the extra
  streams  would  be  preserved. Normal applications that doesn't have
  support  for  them  should  not  destroy the extra streams (and meta
  data,attributes etc).

  Of course tar is different, but any probgram that uses the normal OS
  API to copy, write and read files should not affect the extra info.

  ~S

> Fixing programs that do recursive filesystem traversals is a matter of a
> glibc patch to nftw(3) and a modification to their option processing.

> Holding back a useful feature because you don't want to upgrade
> coreutils is just plain dumb.

> (BTW, for email, multipart/parallel is a start, but a specific multipart
> content type for multi-stream file attachments would probably be more
> appropriate.)

Re: silent semantic changes with reiser4

[Spam]

> cp, grep, cat, and tar will continue to work just fine on files with
> multiple streams.

> tar and cp will lose the extra streams until somebody fixes them, but
> they lose ACLs and xattrs right now, and I don't hear anybody suggesting
> that ACLs and xattrs be removed from the kernel because of this.

  I  thought  that  the  underlaying  FS/VFS would make sure the extra
  streams  would  be  preserved. Normal applications that doesn't have
  support  for  them  should  not  destroy the extra streams (and meta
  data,attributes etc).

  Of course tar is different, but any probgram that uses the normal OS
  API to copy, write and read files should not affect the extra info.

  ~S

> Fixing programs that do recursive filesystem traversals is a matter of a
> glibc patch to nftw(3) and a modification to their option processing.

> Holding back a useful feature because you don't want to upgrade
> coreutils is just plain dumb.

> (BTW, for email, multipart/parallel is a start, but a specific multipart
> content type for multi-stream file attachments would probably be more
> appropriate.)

Re: silent semantic changes with reiser4

[Jamie Lokier]

Matt Mackall wrote:
> > Anything that currently stores a file's metadata in another file really
> > wants this right now. Things like image thumbnails, document summaries,
> > digital signatures, etc.
> 
> That is _highly_ debatable. I would much rather have my cp and grep
> and cat and tar and such continue to work than have to rewrite every
> tool because we've thrown the file-is-a-stream-of-bytes concept out
> the window. Never mind that I've got thumbnails, document summaries,
> and digital signatures already.
> 
> While the number of annoying properties of files with forks is
> practically endless, the biggest has got to be utter lack of
> portability. How do you stick the thing in an attachment or on an ftp
> site? Well you can't because it's NOT A FILE. 
> 
> A file is a stream of bytes.

I couldn't agree more.  Metadata which is easily lost is a terrible
place to put a "document summary".

However, it's not a bad place to put thumbnails which are generated
from the file contents, if they can be regenerated after transporting.
It's slightly better than a ".thumbnails" directory because the latter
won't follow renames and things like that.

It's a very good place to put metadata which is semantically bound to
the file in special ways not intended for transport, such as security
attributes, auto-invalidated digests of the file's contents, an
auto-unpacked view of an archive, a virtual tree of an XML or other
structured file, or an auto-generated textual representation of a
binary file.

The key concept is "not intended for transport".

Generally that means permissions metadata, and things which are
re-generated from the file's contents on demand.*

It's a very bad place to put an abstract of a text document, or the
names of authors, or the MIME content type, or the character encoding
of the document, or the name of the shell to run the file as a script,
unless these things are also deducible from the file's contents.
However, sometimes the content doesn't give any clue, and then
something like the character encoding of a text file is usefully
stored in the metadata simply because that's better than nothing.

Of course the facility can be abused.  What does Windows XP do?  Does
it only store the appropriate kind of metadata in the alternate
streams?  Even Windows XP users expect files transfered over HTTP or
WebDAV to work, don't they?

-- Jamie

* - By my definition, modification time arguably shouldn't be metadata.
    Personally I like downloads to be stamped with their source's
    modification time, which is why I don't use Mozilla's download
    manager but cut-and-paste URLs to "wget -N" instead.

Re: silent semantic changes with reiser4

[Helge Hafting]

Matt Mackall wrote:

>On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
>  
>
>>On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
>>    
>>
>>>Previously Jeremy Allison wrote:
>>>      
>>>
>>>>Multiple-data-stream files are something we should offer, definately (IMHO).
>>>>I don't care how we do it, but I know it's something we need as application
>>>>developers.
>>>>        
>>>>
>>>Aside from samba, is there any other application that has a use for
>>>them? 
>>>
>>>      
>>>
>>Anything that currently stores a file's metadata in another file really
>>wants this right now. Things like image thumbnails, document summaries,
>>digital signatures, etc.
>>    
>>
>
>That is _highly_ debatable. I would much rather have my cp and grep
>and cat and tar and such continue to work than have to rewrite every
>tool because we've thrown the file-is-a-stream-of-bytes concept out
>the window. Never mind that I've got thumbnails, document summaries,
>and digital signatures already.
>  
>
Utilities that works on files only, such as cat, should keep
working.  No problem there.  If you cat a file that also is
a directory, then the file contents is all you get - by design.

Utilities that _only_ traverse the directory tree, such as find,
should keep working too.  Perhaps with a very minor update
so they don't mistake a file-directory for a file only.  I.e.
find must recurse into anything that support directory semantics.

Something that both recurse and operate on files (cp -a, tar, grep, ...
will need minor updating.  They already knows how to handle
files and directories, now they will need an update so
they're open for objects that are both.  I.e. let grep scan the file
as usual, then recurse into its directory part in the usual way too.
Performance problems should be avoided by not supporting
directory operations on files with no directory content, which I
believe will be many of them.

The "file-as-directory" thing will not be that useful before the tools
gets these relatively simple updates.  Till then it'll be a toy, which 
shouldn't
stop it from getting into the VFS.  Updating the tools will be a task
for the file-as-dir fans.

(I don't know wether reiser4 does things this way - it is certainly
they way I would want file-as-directory though.)

>While the number of annoying properties of files with forks is
>practically endless, the biggest has got to be utter lack of
>portability. How do you stick the thing in an attachment or on an ftp
>site? Well you can't because it's NOT A FILE. 
>  
>
Stick the file in an attachment and you get the file only. 
No problem, it is designed that way.  An app that really
wants everything in a single file should use a file structured
for that, not file-as-dir.  File-as-dir attach stuff to a file in a
more loose way.

If you want to attach the directory contents too, do what you usually do
when you want to mail someone a directory tree.  You can't stick a 
directory
in an attachment because it is not a file.  So you either attach every 
file in
the tree, or use tar.  In this case, an updated tar.

The ftp server shouldn't be a problem.  It supports both files and
directories already.  It may need a minor update in order to
not mistake directory for file or vice-versa when someone
request an operation.

ftp> get filename  #Will get you the contents of the file part only - by 
design.
ftp> cd filename    #Will change into the directory (if the file indeed 
provides one.)

>A file is a stream of bytes.
>  
>
Sure.  And a file with a directory support both directory and file 
operations.
You can get the stream of bytes as usual - that's the file part.  Or you
can cd into the directory as usual. There isn't much overlap between
file operations and directory operations, so there is little conflict 
the way
I see it.  Merely letting the tools know that being a file no longer rule
out the possibilities of directory operations.  Getting the VFS right
is another matter of course, but I don't worry about userland tools.

Helge Hafting

Re: silent semantic changes with reiser4

[Helge Hafting]

Matt Mackall wrote:

>On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
>  
>
>>On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
>>    
>>
>>>Previously Jeremy Allison wrote:
>>>      
>>>
>>>>Multiple-data-stream files are something we should offer, definately (IMHO).
>>>>I don't care how we do it, but I know it's something we need as application
>>>>developers.
>>>>        
>>>>
>>>Aside from samba, is there any other application that has a use for
>>>them? 
>>>
>>>      
>>>
>>Anything that currently stores a file's metadata in another file really
>>wants this right now. Things like image thumbnails, document summaries,
>>digital signatures, etc.
>>    
>>
>
>That is _highly_ debatable. I would much rather have my cp and grep
>and cat and tar and such continue to work than have to rewrite every
>tool because we've thrown the file-is-a-stream-of-bytes concept out
>the window. Never mind that I've got thumbnails, document summaries,
>and digital signatures already.
>  
>
Utilities that works on files only, such as cat, should keep
working.  No problem there.  If you cat a file that also is
a directory, then the file contents is all you get - by design.

Utilities that _only_ traverse the directory tree, such as find,
should keep working too.  Perhaps with a very minor update
so they don't mistake a file-directory for a file only.  I.e.
find must recurse into anything that support directory semantics.

Something that both recurse and operate on files (cp -a, tar, grep, ...
will need minor updating.  They already knows how to handle
files and directories, now they will need an update so
they're open for objects that are both.  I.e. let grep scan the file
as usual, then recurse into its directory part in the usual way too.
Performance problems should be avoided by not supporting
directory operations on files with no directory content, which I
believe will be many of them.

The "file-as-directory" thing will not be that useful before the tools
gets these relatively simple updates.  Till then it'll be a toy, which 
shouldn't
stop it from getting into the VFS.  Updating the tools will be a task
for the file-as-dir fans.

(I don't know wether reiser4 does things this way - it is certainly
they way I would want file-as-directory though.)

>While the number of annoying properties of files with forks is
>practically endless, the biggest has got to be utter lack of
>portability. How do you stick the thing in an attachment or on an ftp
>site? Well you can't because it's NOT A FILE. 
>  
>
Stick the file in an attachment and you get the file only. 
No problem, it is designed that way.  An app that really
wants everything in a single file should use a file structured
for that, not file-as-dir.  File-as-dir attach stuff to a file in a
more loose way.

If you want to attach the directory contents too, do what you usually do
when you want to mail someone a directory tree.  You can't stick a 
directory
in an attachment because it is not a file.  So you either attach every 
file in
the tree, or use tar.  In this case, an updated tar.

The ftp server shouldn't be a problem.  It supports both files and
directories already.  It may need a minor update in order to
not mistake directory for file or vice-versa when someone
request an operation.

ftp> get filename  #Will get you the contents of the file part only - by 
design.
ftp> cd filename    #Will change into the directory (if the file indeed 
provides one.)

>A file is a stream of bytes.
>  
>
Sure.  And a file with a directory support both directory and file 
operations.
You can get the stream of bytes as usual - that's the file part.  Or you
can cd into the directory as usual. There isn't much overlap between
file operations and directory operations, so there is little conflict 
the way
I see it.  Merely letting the tools know that being a file no longer rule
out the possibilities of directory operations.  Getting the VFS right
is another matter of course, but I don't worry about userland tools.

Helge Hafting

Re: silent semantic changes with reiser4

[Spam]

> On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
>> On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
>> > Previously Jeremy Allison wrote:
>> > > Multiple-data-stream files are something we should offer, definately (IMHO).
>> > > I don't care how we do it, but I know it's something we need as application
>> > > developers.
>> > 
>> > Aside from samba, is there any other application that has a use for
>> > them? 
>> > 
>> 
>> Anything that currently stores a file's metadata in another file really
>> wants this right now. Things like image thumbnails, document summaries,
>> digital signatures, etc.

> That is _highly_ debatable. I would much rather have my cp and grep
> and cat and tar and such continue to work than have to rewrite every
> tool because we've thrown the file-is-a-stream-of-bytes concept out
> the window. Never mind that I've got thumbnails, document summaries,
> and digital signatures already.

  In  Windows,  the  extra file streams are not lost or removed if you
  use  a  program that doesn't support them. They are only lost if you
  move the file to a file system that doesn't support the streams.

  Even RAR support the NTFS file streams.

> While the number of annoying properties of files with forks is
> practically endless, the biggest has got to be utter lack of
> portability. How do you stick the thing in an attachment or on an ftp
> site? Well you can't because it's NOT A FILE. 

> A file is a stream of bytes.

Re: silent semantic changes with reiser4

[Spam]

> On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
>> On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
>> > Previously Jeremy Allison wrote:
>> > > Multiple-data-stream files are something we should offer, definately (IMHO).
>> > > I don't care how we do it, but I know it's something we need as application
>> > > developers.
>> > 
>> > Aside from samba, is there any other application that has a use for
>> > them? 
>> > 
>> 
>> Anything that currently stores a file's metadata in another file really
>> wants this right now. Things like image thumbnails, document summaries,
>> digital signatures, etc.

> That is _highly_ debatable. I would much rather have my cp and grep
> and cat and tar and such continue to work than have to rewrite every
> tool because we've thrown the file-is-a-stream-of-bytes concept out
> the window. Never mind that I've got thumbnails, document summaries,
> and digital signatures already.

  In  Windows,  the  extra file streams are not lost or removed if you
  use  a  program that doesn't support them. They are only lost if you
  move the file to a file system that doesn't support the streams.

  Even RAR support the NTFS file streams.

> While the number of annoying properties of files with forks is
> practically endless, the biggest has got to be utter lack of
> portability. How do you stick the thing in an attachment or on an ftp
> site? Well you can't because it's NOT A FILE. 

> A file is a stream of bytes.

Re: silent semantic changes with reiser4

[Shawn Rutledge]

--- Nicholas Miell <nmiell@gmail.com> wrote:
> Anything that currently stores a file's metadata in another file
> really
> wants this right now. Things like image thumbnails, document
> summaries,
> digital signatures, etc.

As well as cases where the metadata is currently _in_ the file, right? 
Like MP3ID tags, EXIF tags, etc.  And so many MP3 jukebox-server
programs currently use a database to store metadata, just because they
don't trust the MP3ID tags or need to store supplemental info.  Being
able to put this stuff in the filesystem along with the files would be
so much cleaner.  So I think there are plenty of uses for it.

IMO the Mac resource fork was always a good idea too, and I haven't a
clue why it got left out of OSX (maybe Woz "got it" but Jobs doesn't).

Somebody said "we don't have to teach tar about it"; does that mean
that tar in its present form would already successfully find the
alternate streams?


=====
. _______  Shawn T. Rutledge / KB7PWD ecloud@bigfoot.com
 (_  | |_)    http://ecloud.org/  kb7pwd@kb7pwd.ampr.org
 __) | | \______________________________________________

Re: silent semantic changes with reiser4

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 352 bytes --]

On Thu, 26 Aug 2004 10:59:52 PDT, Shawn Rutledge said:

> Somebody said "we don't have to teach tar about it"; does that mean
> that tar in its present form would already successfully find the
> alternate streams?

Look at history - did anybody teach tar about xattrs or acls?  If nobody
cared about those, why will reiser4 metadata be any different?


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: silent semantic changes with reiser4

[Hans Reiser]

Valdis.Kletnieks@vt.edu wrote:

> Look at history - did anybody teach tar about xattrs or acls? If nobody
>
>cared about those, why will reiser4 metadata be any different?
>
>  
>
My funding proposals often contain budget for modifying the most 
important user space apps to know how to use the new features.  Somebody 
should fund me....

Re: silent semantic changes with reiser4

[Spam]

> Previously Jeremy Allison wrote:
>> Multiple-data-stream files are something we should offer, definately (IMHO).
>> I don't care how we do it, but I know it's something we need as application
>> developers.

> Aside from samba, is there any other application that has a use for
> them?

  Yes,  for  example  documents,  image  files  etc. The multiple data
  streams  can  contain thumbnails, info about who is editing the file
  (useful for networked files) etc. Could be used for version handling
  and much more.

  Also, just because we do not see all the benefits right now, doesn't
  mean there won't be any.

  ~S

> Wichert.

Re: silent semantic changes with reiser4

[Spam]

> Previously Jeremy Allison wrote:
>> Multiple-data-stream files are something we should offer, definately (IMHO).
>> I don't care how we do it, but I know it's something we need as application
>> developers.

> Aside from samba, is there any other application that has a use for
> them?

  Yes,  for  example  documents,  image  files  etc. The multiple data
  streams  can  contain thumbnails, info about who is editing the file
  (useful for networked files) etc. Could be used for version handling
  and much more.

  Also, just because we do not see all the benefits right now, doesn't
  mean there won't be any.

  ~S

> Wichert.

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
>    Yes,  for  example  documents,  image  files  etc. The multiple data
>    streams  can  contain thumbnails, info about who is editing the file
>    (useful for networked files) etc. Could be used for version handling
>    and much more.

All of which can be handled in userspace library code.

What compelling reason is there for doing this in the kernel?

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> wrote:
>>
>>    Yes,  for  example  documents,  image  files  etc. The multiple data
>>    streams  can  contain thumbnails, info about who is editing the file
>>    (useful for networked files) etc. Could be used for version handling
>>    and much more.

> All of which can be handled in userspace library code.

> What compelling reason is there for doing this in the kernel?


  Because  having user space tools and code will make it not work with
  everything. Keeping stuff in the kernel should make the new features
  transparent to the applications.

  Applications  that support the new features will benefit, all others
  will continue to work without destroying data.

  

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> wrote:
>>
>>    Yes,  for  example  documents,  image  files  etc. The multiple data
>>    streams  can  contain thumbnails, info about who is editing the file
>>    (useful for networked files) etc. Could be used for version handling
>>    and much more.

> All of which can be handled in userspace library code.

> What compelling reason is there for doing this in the kernel?


  Because  having user space tools and code will make it not work with
  everything. Keeping stuff in the kernel should make the new features
  transparent to the applications.

  Applications  that support the new features will benefit, all others
  will continue to work without destroying data.

  

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
> 
> 
> > Spam <spam@tnonline.net> wrote:
> >>
> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
> >>    streams  can  contain thumbnails, info about who is editing the file
> >>    (useful for networked files) etc. Could be used for version handling
> >>    and much more.
> 
> > All of which can be handled in userspace library code.
> 
> > What compelling reason is there for doing this in the kernel?
> 
> 
>   Because  having user space tools and code will make it not work with
>   everything. Keeping stuff in the kernel should make the new features
>   transparent to the applications.
> 
>   Applications  that support the new features will benefit, all others
>   will continue to work without destroying data.

Sorry, but that all sounds a bit fluffy.   Please provide some examples.

(Generally, getting all of userspace to agree on a particular library is
socially hard [*], but I don't see that as a reason for putting the
functionality into the kernel)

[*] Example: where's the library to manipulate /etc/whatever.conf? [**]

[**] yes, I know about gconf.

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> wrote:
>>
>> 
>> 
>> > Spam <spam@tnonline.net> wrote:
>> >>
>> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
>> >>    streams  can  contain thumbnails, info about who is editing the file
>> >>    (useful for networked files) etc. Could be used for version handling
>> >>    and much more.
>> 
>> > All of which can be handled in userspace library code.
>> 
>> > What compelling reason is there for doing this in the kernel?
>> 
>> 
>>   Because  having user space tools and code will make it not work with
>>   everything. Keeping stuff in the kernel should make the new features
>>   transparent to the applications.
>> 
>>   Applications  that support the new features will benefit, all others
>>   will continue to work without destroying data.

> Sorry, but that all sounds a bit fluffy.   Please provide some examples.

  We  already had the examples with cp and mv. Both should continue to
  work and the files will still be copied. The same with Konqueror and
  Nautilus.  Files  and  their  meta-files/streams/attributes  will be
  retained as long as applications are using the OS API.

  However,  if  things  are  to  implemented  in  user-space, then old
  applications  will not work correctly and there is risk that all the
  extra information will be lost or corrupted.

  You  said  it  would be socially hard. I think it would be very much
  close to impossible to get it right. Imagine that Gnome and Nautilus
  would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
  app-xyz  would  implement  this anytime soon and in the meantime the
  data is at risk.

  If  we  want  things  to  work  from  the  start, including with old
  applications  then  these  features  should  be at below-application
  level.  If  it  is  in VFS or FS doesn't matter to me, as long as it
  doesn't hinder or delay progress to long.

> (Generally, getting all of userspace to agree on a particular library is
> socially hard [*], but I don't see that as a reason for putting the
> functionality into the kernel)

> [*] Example: where's the library to manipulate /etc/whatever.conf? [**]

> [**] yes, I know about gconf.

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> wrote:
>>
>> 
>> 
>> > Spam <spam@tnonline.net> wrote:
>> >>
>> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
>> >>    streams  can  contain thumbnails, info about who is editing the file
>> >>    (useful for networked files) etc. Could be used for version handling
>> >>    and much more.
>> 
>> > All of which can be handled in userspace library code.
>> 
>> > What compelling reason is there for doing this in the kernel?
>> 
>> 
>>   Because  having user space tools and code will make it not work with
>>   everything. Keeping stuff in the kernel should make the new features
>>   transparent to the applications.
>> 
>>   Applications  that support the new features will benefit, all others
>>   will continue to work without destroying data.

> Sorry, but that all sounds a bit fluffy.   Please provide some examples.

  We  already had the examples with cp and mv. Both should continue to
  work and the files will still be copied. The same with Konqueror and
  Nautilus.  Files  and  their  meta-files/streams/attributes  will be
  retained as long as applications are using the OS API.

  However,  if  things  are  to  implemented  in  user-space, then old
  applications  will not work correctly and there is risk that all the
  extra information will be lost or corrupted.

  You  said  it  would be socially hard. I think it would be very much
  close to impossible to get it right. Imagine that Gnome and Nautilus
  would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
  app-xyz  would  implement  this anytime soon and in the meantime the
  data is at risk.

  If  we  want  things  to  work  from  the  start, including with old
  applications  then  these  features  should  be at below-application
  level.  If  it  is  in VFS or FS doesn't matter to me, as long as it
  doesn't hinder or delay progress to long.

> (Generally, getting all of userspace to agree on a particular library is
> socially hard [*], but I don't see that as a reason for putting the
> functionality into the kernel)

> [*] Example: where's the library to manipulate /etc/whatever.conf? [**]

> [**] yes, I know about gconf.

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
> 
> 
> > Spam <spam@tnonline.net> wrote:
> >>
> >> 
> >> 
> >> > Spam <spam@tnonline.net> wrote:
> >> >>
> >> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
> >> >>    streams  can  contain thumbnails, info about who is editing the file
> >> >>    (useful for networked files) etc. Could be used for version handling
> >> >>    and much more.
> >> 
> >> > All of which can be handled in userspace library code.
> >> 
> >> > What compelling reason is there for doing this in the kernel?
> >> 
> >> 
> >>   Because  having user space tools and code will make it not work with
> >>   everything. Keeping stuff in the kernel should make the new features
> >>   transparent to the applications.
> >> 
> >>   Applications  that support the new features will benefit, all others
> >>   will continue to work without destroying data.
> 
> > Sorry, but that all sounds a bit fluffy.   Please provide some examples.
> 
>   We  already had the examples with cp and mv. Both should continue to
>   work and the files will still be copied. The same with Konqueror and
>   Nautilus.  Files  and  their  meta-files/streams/attributes  will be
>   retained as long as applications are using the OS API.
> 
>   However,  if  things  are  to  implemented  in  user-space, then old
>   applications  will not work correctly and there is risk that all the
>   extra information will be lost or corrupted.
> 
>   You  said  it  would be socially hard. I think it would be very much
>   close to impossible to get it right. Imagine that Gnome and Nautilus
>   would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
>   app-xyz  would  implement  this anytime soon and in the meantime the
>   data is at risk.
> 

No.  All of the applications which you initially identified can be
implemented by putting the various bits of data into a single file and
getting applications to agree on the format of that file.

For example, some image file formats already support embedded metadata, do
they not?

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> wrote:
>>
>> 
>> 
>> > Spam <spam@tnonline.net> wrote:
>> >>
>> >> 
>> >> 
>> >> > Spam <spam@tnonline.net> wrote:
>> >> >>
>> >> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
>> >> >>    streams  can  contain thumbnails, info about who is editing the file
>> >> >>    (useful for networked files) etc. Could be used for version handling
>> >> >>    and much more.
>> >> 
>> >> > All of which can be handled in userspace library code.
>> >> 
>> >> > What compelling reason is there for doing this in the kernel?
>> >> 
>> >> 
>> >>   Because  having user space tools and code will make it not work with
>> >>   everything. Keeping stuff in the kernel should make the new features
>> >>   transparent to the applications.
>> >> 
>> >>   Applications  that support the new features will benefit, all others
>> >>   will continue to work without destroying data.
>> 
>> > Sorry, but that all sounds a bit fluffy.   Please provide some examples.
>> 
>>   We  already had the examples with cp and mv. Both should continue to
>>   work and the files will still be copied. The same with Konqueror and
>>   Nautilus.  Files  and  their  meta-files/streams/attributes  will be
>>   retained as long as applications are using the OS API.
>> 
>>   However,  if  things  are  to  implemented  in  user-space, then old
>>   applications  will not work correctly and there is risk that all the
>>   extra information will be lost or corrupted.
>> 
>>   You  said  it  would be socially hard. I think it would be very much
>>   close to impossible to get it right. Imagine that Gnome and Nautilus
>>   would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
>>   app-xyz  would  implement  this anytime soon and in the meantime the
>>   data is at risk.
>> 

> No.  All of the applications which you initially identified can be
> implemented by putting the various bits of data into a single file and
> getting applications to agree on the format of that file.

> For example, some image file formats already support embedded metadata, do
> they not?

  Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
  these  with  an application that doesn't support the extensions then
  you will loose them.

  Also,  you  are thinking _very_ narrowly now. There are thousands of
  file  formats.  Implementing  the  support  for  meta-data/ streams/
  attributes  in  the  kernel  will  make  it  possible  to  use  this
  generically for all files.

  I  use  this  in  Windows  quite much. I put information description
  fields  for  lots  of  different  files. These descriptions are then
  searchable etc. I could use the command prompt to copy the files and
  the descriptions would still be there.

  Secondly, do you expect file managers like Nautilus and Konqueror to
  support  every piece of file format on the planet so they could read
  information directly from the documents?

  ~S

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> wrote:
>>
>> 
>> 
>> > Spam <spam@tnonline.net> wrote:
>> >>
>> >> 
>> >> 
>> >> > Spam <spam@tnonline.net> wrote:
>> >> >>
>> >> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
>> >> >>    streams  can  contain thumbnails, info about who is editing the file
>> >> >>    (useful for networked files) etc. Could be used for version handling
>> >> >>    and much more.
>> >> 
>> >> > All of which can be handled in userspace library code.
>> >> 
>> >> > What compelling reason is there for doing this in the kernel?
>> >> 
>> >> 
>> >>   Because  having user space tools and code will make it not work with
>> >>   everything. Keeping stuff in the kernel should make the new features
>> >>   transparent to the applications.
>> >> 
>> >>   Applications  that support the new features will benefit, all others
>> >>   will continue to work without destroying data.
>> 
>> > Sorry, but that all sounds a bit fluffy.   Please provide some examples.
>> 
>>   We  already had the examples with cp and mv. Both should continue to
>>   work and the files will still be copied. The same with Konqueror and
>>   Nautilus.  Files  and  their  meta-files/streams/attributes  will be
>>   retained as long as applications are using the OS API.
>> 
>>   However,  if  things  are  to  implemented  in  user-space, then old
>>   applications  will not work correctly and there is risk that all the
>>   extra information will be lost or corrupted.
>> 
>>   You  said  it  would be socially hard. I think it would be very much
>>   close to impossible to get it right. Imagine that Gnome and Nautilus
>>   would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
>>   app-xyz  would  implement  this anytime soon and in the meantime the
>>   data is at risk.
>> 

> No.  All of the applications which you initially identified can be
> implemented by putting the various bits of data into a single file and
> getting applications to agree on the format of that file.

> For example, some image file formats already support embedded metadata, do
> they not?

  Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
  these  with  an application that doesn't support the extensions then
  you will loose them.

  Also,  you  are thinking _very_ narrowly now. There are thousands of
  file  formats.  Implementing  the  support  for  meta-data/ streams/
  attributes  in  the  kernel  will  make  it  possible  to  use  this
  generically for all files.

  I  use  this  in  Windows  quite much. I put information description
  fields  for  lots  of  different  files. These descriptions are then
  searchable etc. I could use the command prompt to copy the files and
  the descriptions would still be there.

  Secondly, do you expect file managers like Nautilus and Konqueror to
  support  every piece of file format on the planet so they could read
  information directly from the documents?

  ~S

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
>    Secondly, do you expect file managers like Nautilus and Konqueror to
>    support  every piece of file format on the planet so they could read
>    information directly from the documents?

Sure.  You're proposing that we implement a single, golden multi-stream file
format in the kernel.

We could just as well do that in libmultistreamfileformat.so.

But I'll grant that one cannot go adding new metadata to, say, C files this
way.  I don't know how useful such a thing is though.

Remember that my main point is that there's a lack of coordination in
userspace.  Hell, there's none.  Putting it in-kernel forces that
coordination, and may be the way to go, but it's pretty sad.

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Andrew Morton wrote:
> But I'll grant that one cannot go adding new metadata to, say, C files this
> way.  I don't know how useful such a thing is though.

That is actually one of the few places where a bit of metadata would be
very useful. Right now there is no way to indicate in what encoding a
source is written: ascii, utf-8, ucs16, etc. are all possible. But a
compiler or interpreter has no good way to figure that out.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Horst von Brand]

Andrew Morton <akpm@osdl.org> said:
> Spam <spam@tnonline.net> wrote:
> >
> >    Secondly, do you expect file managers like Nautilus and Konqueror to
> >    support  every piece of file format on the planet so they could read
> >    information directly from the documents?
> 
> Sure.  You're proposing that we implement a single, golden multi-stream file
> format in the kernel.

Which won't be able to cater for all needs, I'm afraid. Just look at /etc,
and contemplate why there are so many different configuration file formats.
Sure, some are just random coder preferences set in stone, but others do
make sense being different. The same will happen with file formats. Heck,
it is already that way, with file formats for C source code, ELF shared
libraries, various graphics and audio and video formats, ...

> We could just as well do that in libmultistreamfileformat.so.

Right.

> But I'll grant that one cannot go adding new metadata to, say, C files this
> way.  I don't know how useful such a thing is though.

You can never know. Perhaps annotations in swahili? Alternative versions
for different architectures? In object format(s)? Copyright info? 
Changelog?

> Remember that my main point is that there's a lack of coordination in
> userspace.  Hell, there's none.  Putting it in-kernel forces that
> coordination, and may be the way to go, but it's pretty sad.

I think that forcing coordination is wrong.  For example, I do value having
ar/tar/cpio/zip doing essentially the same thing (and being able to define
and use my own way of doing it if it pleases me), and I'd strongly oppose
the kernel forcing one way of doing it down my throat.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Spam wrote:
>   Also,  you  are thinking _very_ narrowly now. There are thousands of
>   file  formats.  Implementing  the  support  for  meta-data/ streams/
>   attributes  in  the  kernel  will  make  it  possible  to  use  this
>   generically for all files.

And how many different formats will be used for thumbnails? How many
different names to indicate a summary for a document?

Having the ability to add attributes standardizes a way to access that
data, but not the format and naming of those attributes. It could be
a step closer to a more flexible way of handling storage, but you have
to realize that a lot (if not most) of the work still needs to be done
in userspace.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Spam wrote:

>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>   these  with  an application that doesn't support the extensions then
>   you will loose them.

OK, so we've got a choice.

Either we will lose the extensions when modifying the file
with an unaware program, or you lose the extensions when
copying (or restoring from backup) using an unaware program.

Personally I'd prefer to keep the file intact when not
modifying it...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Spam wrote:

>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>   these  with  an application that doesn't support the extensions then
>   you will loose them.

OK, so we've got a choice.

Either we will lose the extensions when modifying the file
with an unaware program, or you lose the extensions when
copying (or restoring from backup) using an unaware program.

Personally I'd prefer to keep the file intact when not
modifying it...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Spam]

> On Thu, 26 Aug 2004, Spam wrote:

>>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>>   these  with  an application that doesn't support the extensions then
>>   you will loose them.

> OK, so we've got a choice.

> Either we will lose the extensions when modifying the file
> with an unaware program, or you lose the extensions when
> copying (or restoring from backup) using an unaware program.

> Personally I'd prefer to keep the file intact when not
> modifying it...

  Backup to me is a special case, not simply a copy of files, but also
  retaining all the extra data, info, attributes, etc, that comes with
  the file.

  Enabling  support  to  have  all  of  this  extra  stuff  below  the
  application  level  it  will  be  possible to retain everything even
  though applications do not support them.

  Backup tools, however, must know about these new features to be able
  to  backup  them.  And so will every tool that directly accesses the
  filesystem instead of using the OS API for it.

  And  from  I  learned  in an earlier message, tools like cp actually
  work  on  the  fs level and themselves move the data to the new file
  instead   of  letting  the OS handle it. This seem to me as it could
  be  dangerous and could also prevent any kind of enhancements to the
  FS unless every tool like cp is patched.
  

Re: silent semantic changes with reiser4

[Spam]

> On Thu, 26 Aug 2004, Spam wrote:

>>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>>   these  with  an application that doesn't support the extensions then
>>   you will loose them.

> OK, so we've got a choice.

> Either we will lose the extensions when modifying the file
> with an unaware program, or you lose the extensions when
> copying (or restoring from backup) using an unaware program.

> Personally I'd prefer to keep the file intact when not
> modifying it...

  Backup to me is a special case, not simply a copy of files, but also
  retaining all the extra data, info, attributes, etc, that comes with
  the file.

  Enabling  support  to  have  all  of  this  extra  stuff  below  the
  application  level  it  will  be  possible to retain everything even
  though applications do not support them.

  Backup tools, however, must know about these new features to be able
  to  backup  them.  And so will every tool that directly accesses the
  filesystem instead of using the OS API for it.

  And  from  I  learned  in an earlier message, tools like cp actually
  work  on  the  fs level and themselves move the data to the new file
  instead   of  letting  the OS handle it. This seem to me as it could
  be  dangerous and could also prevent any kind of enhancements to the
  FS unless every tool like cp is patched.
  

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 15:59, Spam wrote:

>   Backup to me is a special case, not simply a copy of files, but also
>   retaining all the extra data, info, attributes, etc, that comes with
>   the file.
>
>   Enabling  support  to  have  all  of  this  extra  stuff  below  the
>   application  level  it  will  be  possible to retain everything even
>   though applications do not support them.
>
>   Backup tools, however, must know about these new features to be able
>   to  backup  them.  And so will every tool that directly accesses the
>   filesystem instead of using the OS API for it.

Thus, this kind of metadata API must be made accessible through the VFS layer, 
if we want to make it FS-independent (not tied to Reiser4). If the real FS 
does not support metadata operations, just ignore them (much like in ISO9660 
ignoring write operations).

>   And  from  I  learned  in an earlier message, tools like cp actually
>   work  on  the  fs level and themselves move the data to the new file
>   instead   of  letting  the OS handle it. This seem to me as it could
>   be  dangerous and could also prevent any kind of enhancements to the
>   FS unless every tool like cp is patched.

I still don't understand why a copy_file() syscall is still missing.

Re: silent semantic changes with reiser4

[Horst von Brand]

Spam <spam@tnonline.net> said:
> Andrew Morton <akpm@osdl.org> said:

[...]

> > For example, some image file formats already support embedded metadata, do
> > they not?

>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>   these  with  an application that doesn't support the extensions then
>   you will loose them.

As you will each time you muck around with your files-are-directories.
Nothing new there.

>   Also,  you  are thinking _very_ narrowly now. There are thousands of
>   file  formats.  Implementing  the  support  for  meta-data/ streams/
>   attributes  in  the  kernel  will  make  it  possible  to  use  this
>   generically for all files.

So the _kernel_ has to know about thousands of formats, just in case it
some blue day it comes across a strange file? Better leave that to the
applications.

You will _not_ be able to define a single format for extra data about the
file, there will be differing extra data for different users (do you
suggest a root-only file with special writable pouches for "graphical icon
for the file" for each user on the system?!), so the idea in itself is
doomed from the start.

>   I  use  this  in  Windows  quite much.

Then use it and be happy. No need to screw up Linux for that.

>                                          I put information description
>   fields  for  lots  of  different  files. These descriptions are then
>   searchable etc. I could use the command prompt to copy the files and
>   the descriptions would still be there.

The descriptions might make sense to _you_, _now_. No guarantee they make
any sense (or are in the least useful) for other users on your system, and
I might want them in arabic or some such. The descriptions might make no
sense to you in a couple of years.

>   Secondly, do you expect file managers like Nautilus and Konqueror to
>   support  every piece of file format on the planet so they could read
>   information directly from the documents?

That's their (self-selected) job, yes.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
>
>So the _kernel_ has to know about thousands of formats, just in case it
>some blue day it comes across a strange file? Better leave that to the
>applications.

Or come up with a way to export some interface to userspace.

Or make a standard description format in something like XML that can be
used for arbitrary files and have userspace interface with that metadata
stream that is the XML.
This is, however, dangerously close to what we have now and does. not. work,
so the only added bonus we might have is the fact that it's endorsed by the
kernel.
I don't like this idea except as a last fallback solution...

>>   I  use  this  in  Windows  quite much.
>Then use it and be happy. No need to screw up Linux for that.

Linux does not need to be screwed up by that.
It can be done right.

>The descriptions might make sense to _you_, _now_. No guarantee they make
>any sense (or are in the least useful) for other users on your system, and

They also make perfect sense to me, and I would like this to have
had this yesterday. or last year ;)

>I might want them in arabic or some such. The descriptions might make no
>sense to you in a couple of years.

So what do you do if you use someone else's system and he doesn't have
Arabic man pages but you want them in Arabic?

And a description that doesn't make sense in many years is badly formed.

>>   Secondly, do you expect file managers like Nautilus and Konqueror to
>>   support  every piece of file format on the planet so they could read
>>   information directly from the documents?
>That's their (self-selected) job, yes.

Sure, but they store it differently in different places. Having them in
the actual file makes it universal.

I think the biggest issue is that you are arguing against this out of
some principle. There is a certain want for these new features among
the users, there are people who are willing to implement them, there
is Linus who seems to agree and accept.

What do you, and people who agree with you, lose if this gets implemented?
Drive space when you have the slightly-grown kernel sources downloaded?

-- 
mjt

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Andrew Morton <akpm@osdl.org> said:

> [...]

>> > For example, some image file formats already support embedded metadata, do
>> > they not?

>>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>>   these  with  an application that doesn't support the extensions then
>>   you will loose them.

> As you will each time you muck around with your files-are-directories.
> Nothing new there.

>>   Also,  you  are thinking _very_ narrowly now. There are thousands of
>>   file  formats.  Implementing  the  support  for  meta-data/ streams/
>>   attributes  in  the  kernel  will  make  it  possible  to  use  this
>>   generically for all files.

> So the _kernel_ has to know about thousands of formats, just in case it
> some blue day it comes across a strange file? Better leave that to the
> applications.

  No,  not  at all. The only think the kernel would need to support is
  the actual meta files and streams - not the contents of the streams.
  That is up to the applications to know and use.

  The  idea  with  plugins  to reiser4 would allow someone to create a
  plugin that would export the thumbnails from TIFF and JPEG images as
  separate streams.

> You will _not_ be able to define a single format for extra data about the
> file, there will be differing extra data for different users (do you
> suggest a root-only file with special writable pouches for "graphical icon
> for the file" for each user on the system?!), so the idea in itself is
> doomed from the start.

  We  do  not need to define a single format. You are locking yourself
  to  the contents of streams and meta files.

  If  someone  likes  the  idea of a plugin that would allow for icons
  then  that  may  be  it. It would be up to application developers to
  know  about  this and use it. Perhaps they could agree on a standard
  even.
>>   I  use  this  in  Windows  quite much.

> Then use it and be happy. No need to screw up Linux for that.

  This  wouldn't  screw anything up besides allowing for extending the
  functionality.  There  is noone that forces you to enable and instll
  plugins,  use  extra attributes or meta info. All of this always was
  optional (as I see it).

>>                                          I put information description
>>   fields  for  lots  of  different  files. These descriptions are then
>>   searchable etc. I could use the command prompt to copy the files and
>>   the descriptions would still be there.

> The descriptions might make sense to _you_, _now_. No guarantee they make
> any sense (or are in the least useful) for other users on your system, and
> I might want them in arabic or some such. The descriptions might make no
> sense to you in a couple of years.

  So? The same may be for your own word documents etc. It doesn't mean
  I do not find it useful now, or that anyone else might not. Shall we
  not  include  features  because "someone" on the same system may not
  use it? Seems very backward to me.

  If   I   include   a filestream, description.xml, then that wouldn't
  affect  anyone. I would know how to read the file, and that would be
  enough.

>>   Secondly, do you expect file managers like Nautilus and Konqueror to
>>   support  every piece of file format on the planet so they could read
>>   information directly from the documents?

> That's their (self-selected) job, yes.

  Yes  and  no.  I  used  them  as example but there are lots of other
  programs  that  can access files. Midnight commander, cp, tar, email
  programs and lots of others.

  besides, I think it would be easier for them to 1) write plugins and
  2)  just  use normal file access method to provide the extra data to
  the users.

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Andrew Morton <akpm@osdl.org> said:

> [...]

>> > For example, some image file formats already support embedded metadata, do
>> > they not?

>>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>>   these  with  an application that doesn't support the extensions then
>>   you will loose them.

> As you will each time you muck around with your files-are-directories.
> Nothing new there.

>>   Also,  you  are thinking _very_ narrowly now. There are thousands of
>>   file  formats.  Implementing  the  support  for  meta-data/ streams/
>>   attributes  in  the  kernel  will  make  it  possible  to  use  this
>>   generically for all files.

> So the _kernel_ has to know about thousands of formats, just in case it
> some blue day it comes across a strange file? Better leave that to the
> applications.

  No,  not  at all. The only think the kernel would need to support is
  the actual meta files and streams - not the contents of the streams.
  That is up to the applications to know and use.

  The  idea  with  plugins  to reiser4 would allow someone to create a
  plugin that would export the thumbnails from TIFF and JPEG images as
  separate streams.

> You will _not_ be able to define a single format for extra data about the
> file, there will be differing extra data for different users (do you
> suggest a root-only file with special writable pouches for "graphical icon
> for the file" for each user on the system?!), so the idea in itself is
> doomed from the start.

  We  do  not need to define a single format. You are locking yourself
  to  the contents of streams and meta files.

  If  someone  likes  the  idea of a plugin that would allow for icons
  then  that  may  be  it. It would be up to application developers to
  know  about  this and use it. Perhaps they could agree on a standard
  even.
>>   I  use  this  in  Windows  quite much.

> Then use it and be happy. No need to screw up Linux for that.

  This  wouldn't  screw anything up besides allowing for extending the
  functionality.  There  is noone that forces you to enable and instll
  plugins,  use  extra attributes or meta info. All of this always was
  optional (as I see it).

>>                                          I put information description
>>   fields  for  lots  of  different  files. These descriptions are then
>>   searchable etc. I could use the command prompt to copy the files and
>>   the descriptions would still be there.

> The descriptions might make sense to _you_, _now_. No guarantee they make
> any sense (or are in the least useful) for other users on your system, and
> I might want them in arabic or some such. The descriptions might make no
> sense to you in a couple of years.

  So? The same may be for your own word documents etc. It doesn't mean
  I do not find it useful now, or that anyone else might not. Shall we
  not  include  features  because "someone" on the same system may not
  use it? Seems very backward to me.

  If   I   include   a filestream, description.xml, then that wouldn't
  affect  anyone. I would know how to read the file, and that would be
  enough.

>>   Secondly, do you expect file managers like Nautilus and Konqueror to
>>   support  every piece of file format on the planet so they could read
>>   information directly from the documents?

> That's their (self-selected) job, yes.

  Yes  and  no.  I  used  them  as example but there are lots of other
  programs  that  can access files. Midnight commander, cp, tar, email
  programs and lots of others.

  besides, I think it would be easier for them to 1) write plugins and
  2)  just  use normal file access method to provide the extra data to
  the users.

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]

Salut,

On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
> >   Also,  you  are thinking _very_ narrowly now. There are thousands of
> >   file  formats.  Implementing  the  support  for  meta-data/ streams/
> >   attributes  in  the  kernel  will  make  it  possible  to  use  this
> >   generically for all files.
> 
> So the _kernel_ has to know about thousands of formats, just in case it
> some blue day it comes across a strange file? Better leave that to the
> applications.

I'd  do  that  quite  differently:  Collect metadata  about  files  in
extended attributes (which are  rather universally useable), and do it
from  an userspace daemon/application/file  manager plugin  which uses
dynamically   loadable   plugins   for   the  different   file   types
differentiated by libmagic.

This has nothing to do in the kernel.

			    Tonnerre


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Alexander G. M. Smith]

Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200:
> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
> > So the _kernel_ has to know about thousands of formats, just in case it
> > some blue day it comes across a strange file? Better leave that to the
> > applications.
> 
> I'd  do  that  quite  differently:  Collect metadata  about  files  in
> extended attributes (which are  rather universally useable), and do it
> from  an userspace daemon/application/file  manager plugin  which uses
> dynamically   loadable   plugins   for   the  different   file   types
> differentiated by libmagic.
> 
> This has nothing to do in the kernel.

I mostly agree, like I was saying earlier, file types are needed!  The kernel doesn't have to know about all of them, just some of them.  It should be possible to attach a file type to everything so you know what kind of thing it is, not just block or character device or file, but something like a MIME type.  If space is a concern, add to the inode a unique 8 byte character code for each popular type and a global translation table to get back the MIME string.

The kernel (includes file systems and plug-ins) would have to know some standard file types and how to compare them for indexing purposes.  For example, an integer attribute like the year a music album was released would be stored as a 4 byte attribute "file" with a type like "primitive/int32", which tells the kernel to use 32 bit integer number comparison for indexing order.  Another type the kernel could be aware of are the file types used for security attributes.  A third kind it might need to know about are the computed attributes, which get set whenever related attributes they are computed from change.  For any unknown types, the kernel just stores the data and doesn't do any extra processing.

Optionally the kernel could also maintain the global list of all file types and their properties (such as which ones are indexed, which are computed), though that could also be done in userland if you aren't doing indexing or computed attributes or other fancy operations.

The determination of the types would be in userland (smart programs would write the type when they created the file).  Like you said, a daemon could classify unknown files in the background (BeOS does this).  Most of the uses of the file type would also be in userland, such as deciding which program can handle a given file, or what icon to display, or to hint that a file/directory/attribute thing should be interpreted as an old style directory to the user, or to mark a file thing as being worth backing up (don't need to back up the computed ones).

- Alex

Re: silent semantic changes with reiser4

[Spam]

  

> Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200:
>> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
>> > So the _kernel_ has to know about thousands of formats, just in case it
>> > some blue day it comes across a strange file? Better leave that to the
>> > applications.
>> 
>> I'd  do  that  quite  differently:  Collect metadata  about  files  in
>> extended attributes (which are  rather universally useable), and do it
>> from  an userspace daemon/application/file  manager plugin  which uses
>> dynamically   loadable   plugins   for   the  different   file  types
>> differentiated by libmagic.
>> 
>> This has nothing to do in the kernel.

> I mostly agree, like I was saying earlier, file types are needed!
> The kernel doesn't have to know about all of them, just some of
> them.  It should be possible to attach a file type to everything so
> you know what kind of thing it is, not just block or character
> device or file, but something like a MIME type.  If space is a
> concern, add to the inode a unique 8 byte character code for each
> popular type and a global translation table to get back the MIME
> string.

  The  kernel  doesn't  have anything to know about normal files.It is
  up to applications to determine and know the contents. Isn't it?

  Exception  would  be  of  course  special  meta  files  that  define
  attributes and permissions for the filesystem.


> The kernel (includes file systems and plug-ins) would have to
> know some standard file types and how to compare them for indexing
> purposes.  For example, an integer attribute like the year a music
> album was released would be stored as a 4 byte attribute "file" with
> a type like "primitive/int32", which tells the kernel to use 32 bit
> integer number comparison for indexing order.  Another type the
> kernel could be aware of are the file types used for security
> attributes.  A third kind it might need to know about are the
> computed attributes, which get set whenever related attributes they
> are computed from change.  For any unknown types, the kernel just
> stores the data and doesn't do any extra processing.

> Optionally the kernel could also maintain the global list of all
> file types and their properties (such as which ones are indexed,
> which are computed), though that could also be done in userland if
> you aren't doing indexing or computed attributes or other fancy
> operations.

> The determination of the types would be in userland (smart
> programs would write the type when they created the file).  Like you
> said, a daemon could classify unknown files in the background (BeOS
> does this).  Most of the uses of the file type would also be in
> userland, such as deciding which program can handle a given file, or
> what icon to display, or to hint that a file/directory/attribute
> thing should be interpreted as an old style directory to the user,
> or to mark a file thing as being worth backing up (don't need to
> back up the computed ones).

> - Alex

Re: silent semantic changes with reiser4

[Spam]

  

> Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200:
>> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
>> > So the _kernel_ has to know about thousands of formats, just in case it
>> > some blue day it comes across a strange file? Better leave that to the
>> > applications.
>> 
>> I'd  do  that  quite  differently:  Collect metadata  about  files  in
>> extended attributes (which are  rather universally useable), and do it
>> from  an userspace daemon/application/file  manager plugin  which uses
>> dynamically   loadable   plugins   for   the  different   file  types
>> differentiated by libmagic.
>> 
>> This has nothing to do in the kernel.

> I mostly agree, like I was saying earlier, file types are needed!
> The kernel doesn't have to know about all of them, just some of
> them.  It should be possible to attach a file type to everything so
> you know what kind of thing it is, not just block or character
> device or file, but something like a MIME type.  If space is a
> concern, add to the inode a unique 8 byte character code for each
> popular type and a global translation table to get back the MIME
> string.

  The  kernel  doesn't  have anything to know about normal files.It is
  up to applications to determine and know the contents. Isn't it?

  Exception  would  be  of  course  special  meta  files  that  define
  attributes and permissions for the filesystem.


> The kernel (includes file systems and plug-ins) would have to
> know some standard file types and how to compare them for indexing
> purposes.  For example, an integer attribute like the year a music
> album was released would be stored as a 4 byte attribute "file" with
> a type like "primitive/int32", which tells the kernel to use 32 bit
> integer number comparison for indexing order.  Another type the
> kernel could be aware of are the file types used for security
> attributes.  A third kind it might need to know about are the
> computed attributes, which get set whenever related attributes they
> are computed from change.  For any unknown types, the kernel just
> stores the data and doesn't do any extra processing.

> Optionally the kernel could also maintain the global list of all
> file types and their properties (such as which ones are indexed,
> which are computed), though that could also be done in userland if
> you aren't doing indexing or computed attributes or other fancy
> operations.

> The determination of the types would be in userland (smart
> programs would write the type when they created the file).  Like you
> said, a daemon could classify unknown files in the background (BeOS
> does this).  Most of the uses of the file type would also be in
> userland, such as deciding which program can handle a given file, or
> what icon to display, or to hint that a file/directory/attribute
> thing should be interpreted as an old style directory to the user,
> or to mark a file thing as being worth backing up (don't need to
> back up the computed ones).

> - Alex

Re: silent semantic changes with reiser4

[Spam]

  

> Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200:
>> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
>> > So the _kernel_ has to know about thousands of formats, just in case it
>> > some blue day it comes across a strange file? Better leave that to the
>> > applications.
>> 
>> I'd  do  that  quite  differently:  Collect metadata  about  files  in
>> extended attributes (which are  rather universally useable), and do it
>> from  an userspace daemon/application/file  manager plugin  which uses
>> dynamically   loadable   plugins   for   the  different   file  types
>> differentiated by libmagic.
>> 
>> This has nothing to do in the kernel.

> I mostly agree, like I was saying earlier, file types are needed!
> The kernel doesn't have to know about all of them, just some of
> them.  It should be possible to attach a file type to everything so
> you know what kind of thing it is, not just block or character
> device or file, but something like a MIME type.  If space is a
> concern, add to the inode a unique 8 byte character code for each
> popular type and a global translation table to get back the MIME
> string.

  The  kernel  doesn't  have anything to know about normal files.It is
  up to applications to determine and know the contents. Isn't it?

  Exception  would  be  of  course  special  meta  files  that  define
  attributes and permissions for the filesystem.


> The kernel (includes file systems and plug-ins) would have to
> know some standard file types and how to compare them for indexing
> purposes.  For example, an integer attribute like the year a music
> album was released would be stored as a 4 byte attribute "file" with
> a type like "primitive/int32", which tells the kernel to use 32 bit
> integer number comparison for indexing order.  Another type the
> kernel could be aware of are the file types used for security
> attributes.  A third kind it might need to know about are the
> computed attributes, which get set whenever related attributes they
> are computed from change.  For any unknown types, the kernel just
> stores the data and doesn't do any extra processing.

> Optionally the kernel could also maintain the global list of all
> file types and their properties (such as which ones are indexed,
> which are computed), though that could also be done in userland if
> you aren't doing indexing or computed attributes or other fancy
> operations.

> The determination of the types would be in userland (smart
> programs would write the type when they created the file).  Like you
> said, a daemon could classify unknown files in the background (BeOS
> does this).  Most of the uses of the file type would also be in
> userland, such as deciding which program can handle a given file, or
> what icon to display, or to hint that a file/directory/attribute
> thing should be interpreted as an old style directory to the user,
> or to mark a file thing as being worth backing up (don't need to
> back up the computed ones).

> - Alex

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 1351 bytes --]

Salut,

On Sun, Aug 29, 2004 at 08:39:25PM -0400, Alexander G. M. Smith wrote:
> I mostly  agree, like I was  saying earlier, file  types are needed!
> The kernel  doesn't have  to know  about all of  them, just  some of
> them.  It should be possible to  attach a file type to everything so
> you  know what  kind of  thing it  is, not  just block  or character
> device or file, but something like a MIME type.

Then store the value computed  by libmagic into an extended attribute,
if you  like. I still  don't see why  the kernel should even  care. It
shouldn't read things from files anyway, as a kernel shall be bootable
on a busybox  system (and *Step) as  well, so we don't rely  on the fs
layout in any way.

> Optionally the  kernel could  also maintain the  global list  of all
> file types  and their  properties (such as  which ones  are indexed,
> which are computed),  though that could also be  done in userland if
> you  aren't doing  indexing or  computed attributes  or  other fancy
> operations.

Quel horreur!

Do it in userland, really.

If I  get the time,  I'll write you  a small daemon based  on libmagic
which  stores  the  file  attributes  in xattrs,  or  if  they're  not
supported, in some MacOS/Xish per-directory files. Even a file manager
("finder") can do that, there's not even the need for a daemon.

				Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Tonnerre wrote:
> Quel horreur!
> Do it in userland, really.

I'm amazed that after all this discussion where all the realistic
implementations are done in userland with kernel support for calling
out to it, there are people who think the kernel is supposed to decode
MP3 files or whatever.

Nobody is advocating that!

> If I  get the time,  I'll write you  a small daemon based  on libmagic
> which  stores  the  file  attributes  in xattrs,  or  if  they're  not
> supported, in some MacOS/Xish per-directory files. Even a file manager
> ("finder") can do that, there's not even the need for a daemon.

How are you going to do the part where the xattr changes when the file
is modified?

(For example, if I edit an HTML file which is encoded in iso-8859-1,
change it to utf-8 and indicate that in a META element, and save it
under the same name, the full content-type should change from
"text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

I don't see how you can do that without kernel support.

Don't say dnotify or inotify, because neither would work.

-- Jamie

Re: silent semantic changes with reiser4

[Dave Kleikamp]

On Wed, 2004-09-01 at 15:16, Jamie Lokier wrote:
> (For example, if I edit an HTML file which is encoded in iso-8859-1,
> change it to utf-8 and indicate that in a META element, and save it
> under the same name, the full content-type should change from
> "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
> 
> I don't see how you can do that without kernel support.

Your html editor should do that.
-- 
David Kleikamp
IBM Linux Technology Center

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Dave" == Dave Kleikamp <shaggy@austin.ibm.com> writes:

Dave> On Wed, 2004-09-01 at 15:16, Jamie Lokier wrote:
>> (For example, if I edit an HTML file which is encoded in iso-8859-1,
>> change it to utf-8 and indicate that in a META element, and save it
>> under the same name, the full content-type should change from
>> "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
>> 
>> I don't see how you can do that without kernel support.

Dave> Your html editor should do that.

And if it doesn't, you'll need to patch your editor.  Every editor will
need to be patched to make sure that it is always consistent.  Even
sed.  And you'll have to do that for every dynamically-changing
attribute that people want.  Sure, you can go that route if you want,
but I wish you luck.  You'll need it.

(Just for the record, I don't particularly think that the example given
by Jamie is a good one.  The "meta" element to me is a hack necessitated
by not being able to attach attributes to a file in a consistent manner.
The "proper" way to do it would be to set the content-type as an
attribute, and let the webserver read that and send it in the HTTP
header, where it's supposed to be.

Another example could be thumbnails.  There are probably more, but I'm
not feeling very creative right now.)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Dave" == Dave Kleikamp <shaggy@austin.ibm.com> writes:

Dave> On Wed, 2004-09-01 at 15:16, Jamie Lokier wrote:
>> (For example, if I edit an HTML file which is encoded in iso-8859-1,
>> change it to utf-8 and indicate that in a META element, and save it
>> under the same name, the full content-type should change from
>> "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
>> 
>> I don't see how you can do that without kernel support.

Dave> Your html editor should do that.

And if it doesn't, you'll need to patch your editor.  Every editor will
need to be patched to make sure that it is always consistent.  Even
sed.  And you'll have to do that for every dynamically-changing
attribute that people want.  Sure, you can go that route if you want,
but I wish you luck.  You'll need it.

(Just for the record, I don't particularly think that the example given
by Jamie is a good one.  The "meta" element to me is a hack necessitated
by not being able to attach attributes to a file in a consistent manner.
The "proper" way to do it would be to set the content-type as an
attribute, and let the webserver read that and send it in the HTTP
header, where it's supposed to be.

Another example could be thumbnails.  There are probably more, but I'm
not feeling very creative right now.)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave Kleikamp wrote:
| On Wed, 2004-09-01 at 15:16, Jamie Lokier wrote:
|
|>(For example, if I edit an HTML file which is encoded in iso-8859-1,
|>change it to utf-8 and indicate that in a META element, and save it
|>under the same name, the full content-type should change from
|>"text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
|>
|>I don't see how you can do that without kernel support.
|
|
| Your html editor should do that.

Isn't the kernel faced with the same issues here as userland programs
without meta attribute support?  How do you read the META element before
you know the encoding?

Is there property attached to each file descriptor that defines what
encoding it's using?  I know there is in perl, but I doubt it's at the
OS level.  If it is at the OS level, the kernel should use that.

Otherwise I agree -- the html editor should do that.  Although I'd hate
to see Vim patched to do that.  Maybe save it as file/utf8?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTaZEHgHNmZLgCUhAQLmHxAAif/DzI3HDPPixHEJURTW8LMSBYi+RhgC
Rsldit8G6N9Sx1wKReSZqoTI27OYUodVcXjgtSYbzr9w0dw2ifuLaU87550nCcwl
p/3ivEiPww3Pqc6PgN2PpAhYT2b/xnZrWJSU3g9siGp9D4APBZ4yxz8GO9N8pAgg
a/juHtLCNCtPKD8/IdjCYRn9W7Ka7uWF7cEHp58Yyj0R70EG4ljIRbJRhqKGT2Ew
6EIFcieStmuveA2Kna4Od4gAXIE5CrKEO8whRvplOsvHAlsZMp0zkgUGiKzBzQU0
K7ICkddo3Bnos1pc7BQ/7F595tNCNREqnlvCkWGVifLPdcyujULZ/U/OAVdD7zSJ
AvhZs5A1/hPFYTEqrzSNsYteDsTSc0TJsiWS8gx+qyGzOzaVMAOIuf86jQGKzBT7
WSYutU4itrE1QYvdJ9XXeess8m9aSY1A9G1EkhtWmaRDq/SuCpSf01cNjZZVu52v
Q1h6faCC9JTHM1pDPMBlMjeQOnhrhv/AuhrZnOfXmrODK06l+ZhQ/LSUTRAZwzkv
UV6PSc+Pbq7ijYrurH4attPI8IkelzExhIPHziGQCTMnIN3pXSr6RT2zEtDG7LNR
98MHtKkuNC8bsuYFhyqGRbMLr9GANiByzu3rh1ccHuMbB42FR8/Z7X6UCDVZtP80
Bp4g6WwVG6A=
=xNYt
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Jamie Lokier]

Tonnerre wrote:
> > > I'll write you a small daemon based on libmagic which stores the
> > > file attributes in xattrs, or if they're not supported, in some
> > > MacOS/Xish per-directory files. Even a file manager ("finder") can
> > > do that, there's not even the need for a daemon.

Jamie Lokier wrote:
> > (For example, if I edit an HTML file which is encoded in iso-8859-1,
> > change it to utf-8 and indicate that in a META element, and save it
> > under the same name, the full content-type should change from
> > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
> > 
> > I don't see how you can do that without kernel support.

Dave Kleikamp wrote:
> Your html editor should do that.

My html editor is "vi".  If I am supposed to manually set the
content-type attribute after exiting vi, doesn't that rather
invalidate the idea of a "small daemon based on libmagic" which sets
it for me?

That was the sole point of my statement.

-- Jamie

Re: silent semantic changes with reiser4

[Alan Cox]

On Mer, 2004-09-01 at 21:16, Jamie Lokier wrote:
> (For example, if I edit an HTML file which is encoded in iso-8859-1,
> change it to utf-8 and indicate that in a META element, and save it
> under the same name, the full content-type should change from
> "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
> 
> I don't see how you can do that without kernel support.
> 
> Don't say dnotify or inotify, because neither would work.

inotify done right is useful here as well as in a lot of other desktop
cases where dnotify doesn't really scale. Its enough to let me

	- Find the new file
	- Virus scan it
	- Classify its possible type heirarchies
	- Index it

Alan

Re: silent semantic changes with reiser4

[Jamie Lokier]

Alan Cox wrote:
> On Mer, 2004-09-01 at 21:16, Jamie Lokier wrote:
> > (For example, if I edit an HTML file which is encoded in iso-8859-1,
> > change it to utf-8 and indicate that in a META element, and save it
> > under the same name, the full content-type should change from
> > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
> > 
> > I don't see how you can do that without kernel support.
> > 
> > Don't say dnotify or inotify, because neither would work.
> 
> inotify done right is useful here as well as in a lot of other desktop
> cases where dnotify doesn't really scale. Its enough to let me
> 
> 	- Find the new file
> 	- Virus scan it
> 	- Classify its possible type heirarchies
> 	- Index it

Indeed it does, but it fails for the example I was commenting on to
which you replied..

      1. The file /var/www/site/index.html's written in vi.
      2. "The daemon" (that's what I objected to) receives inotify.
         blocks waiting for scheduler, however...
      3. Seeing that vi is now finished, I phone person on other side
         of world and say the updated file is now available.
      4. Person fetches http://site/index.html
         ...oh!  They receive the wrong content-type (bollocks!)
      5. Eventually after paging & scheduling deems it appropriate,
         "The daemon" gets to run, looks at file, updates content-type.

In other words I was criticising Tonnerre's idea of a daemon which
updates the xattrs by looking at file contents...  by definition, a
daemon runs in the background, and the whole point of it updating a
content-type xattr was obviously so that programs like httpd could
just _use_ that xattr.

That doesn't work, even if that daemon uses inotify.

Basically, anytime where you want an ordering guarantee that something
will be recalculated in the interval after a file is modified and
before the calculated result is used, you have to be very careful
about exactly which code is using dnotify or inotify to achieve that.
Single threaded programs can use it easily, but a multi-threaded file
server has to be very careful about ordering if it's to avoid glitches.

It is exactly a real problem I have faced in a multi-threaded HTTP
server which uses dnotify to detect changes to prerequisite files and
also changes to the path walked and permissions on it, and thus
invalidate cached generated pages, thus giving strong cache
guarantees.  Performance is great, if you ignore the race conditions...

(One of the more popular small web servers, lighttpd (PHP people seem
to like it), also uses dnotify.  Apparently it provides a big
performance boost, just by avoiding stat() calls.  So I'm not alone in
using dnotify for such things.)

-- Jamie

Re: silent semantic changes with reiser4

[Christer Weinigel]

Jamie Lokier <jamie@shareable.org> writes:

> (For example, if I edit an HTML file which is encoded in iso-8859-1,
> change it to utf-8 and indicate that in a META element, and save it
> under the same name, the full content-type should change from
> "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
>
> I don't see how you can do that without kernel support.

charset_cache = dbm(os.getenv('HOME') + '/.charset_cache')

def get_charset(path):
    file_mtime = get_mtime(path)
    cache_tuple = dbm.get(path, (None, None))
    if file_mtime != cache_tuple[0]:
        cache_tuple = (file_mtime, figure_out_charset(path))
        dbm.put(path, cache_tuple)
    return cache_tuple[1]

This code is guaranteed to always give you an up to date charset for a
file, provided that the mtime is guaranteed to change every time the
file changes.

If the file can change during the call to get_charset you'll have to
lock the file while you are working with it or handle it some other
way (comparing mtime before and after and retrying if they differ),
but that is true even if you do it all in the kernel.  This has the
added advantage that the cache is updated lazily, so the charset is
never calculated unless someone really needs it.

If the disk gets full, it is time to shrink the cache and preferably
stale entries are evicted first.  If that's not good enough, a
notifier such as dnotify or inotify can be used to invalidate the
cache immediately after the file has changed.

If this gets popular, the cache management can be moved to a daemon
(naturally with all the security aspects in mind). 

> Don't say dnotify or inotify, because neither would work.

Why not?  The approach above works on any filesystem, even without
dnotify or inotify but will be more efficient with them.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christer Weinigel wrote:
> Jamie Lokier <jamie@shareable.org> writes:
> 
> > (For example, if I edit an HTML file which is encoded in iso-8859-1,
> > change it to utf-8 and indicate that in a META element, and save it
> > under the same name, the full content-type should change from
> > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
> >
> > I don't see how you can do that without kernel support.
> 
> [code which runs in the program which needs to read the content-type]
> 
> This code is guaranteed to always give you an up to date charset for a
> file, provided that the mtime is guaranteed to change every time the
> file changes.

...which it isn't, but ignoring that.

Yes we all know that code will work, turning a blind eye to occasional
mtime failures.

What I objected to isn't that, but Tonerre's idea of a daemon (i.e. in
the background, that's what daemons do) updating xattrs.  Your code
will work.  Tonerre's has a serious race condition.

> > Don't say dnotify or inotify, because neither would work.
> 
> Why not?  The approach above works on any filesystem, even without
> dnotify or inotify but will be more efficient with them.

The above approach is in fact better with dnotify/inotify, when they
area available, because then it doesn't have mtime problems.

My objection is that dnotify & inotify don't fix the problems of a
"daemon" (i.e. in the background) updating an xattr - because the
daemon might not run scheduled between the write and when the file
needs to be used.

That said, yours has practical problems too.  The intended purpose (*)
of storing content-type in an xattr was so that programs which need to
read content without understanding it may just do so (e.g. mail agents,
httpd), and are distinct from the programs which compute and set it
(e.g. programs which understand specific content formats).

(*) I don't necessarily agree with that purpose.

-- Jamie

Re: silent semantic changes with reiser4

[Jamie Lokier]

Andrew Morton wrote:
> No.  All of the applications which you initially identified can be
> implemented by putting the various bits of data into a single file and
> getting applications to agree on the format of that file.
> 
> For example, some image file formats already support embedded metadata, do
> they not?

Same for MP3s and their metadata tags.  _That_ kind of data should be
part of the contents of the file: it is intended to be transported.

It is useful to auto-extract it from the file's data stream, using
format-specific programs, and offer that metadata in separate files
where it is easy to use e.g. by shell scripts and indexing programs,
which can then work independently of file formats.

That kind of metadata should be _derived_ from the file's data stream,
and ideally the extraction should be automatic, with userspace
assistance.  It may be useful for the filesystem to record what's
extracted, for fast access and indexing.

The same applies to thumbnails.  The only difference is that these may
take longer to compute.  But they're the same thing: something
calculated from the file's data stream, which may (or may not) be
stored in the filesystem so it doesn't have to be recomputed each
time.  It might even be worth expiring the metadata from disk if it
hasn't been used for a while and can be recomputed.

These are the same as looking into archives or structured files.  The
only difference is that the unpacked views might not be stored on the
filesystem (but they might be, if the filesystem knows how to cache
views for faster future access).[*]

Those are all examples of data derived from the data stream of a flat
file.  They are useful applications, and all work fine with FTP, cp,
cat <a >b, and email attachments, _no application changes required_.

There's another kind of metadata which isn't intended for transport.
That would be things like the classic attributes: permissions,
timestamps.  It would also include things like author information when
the file format cannot carry it.

-- Jamie


[*] - A fancier variant, which is hard without good fs support, it
writing to structured files through a view.  I.e. modifying files
inside a .tar.gz, or (this is useful) inside an
{MS,Open,Gnome,K}Office container file or *SQL database table.  To do
this efficiently, the file's main data stream is regenerated on demand
from the view's data.  As I said, it requires good, special fs support
to do it efficiently (fancy caching).  With appropriate userspace
libraries you can make apps work with container files on all other
OSes (just like .tar.gz and *Office containers at the moment) but are
significantly faster to work with on Linux (no need to serialise and
(de-)compress except on demand) and more visible to the standard unix
scripting tools.

Re: silent semantic changes with reiser4

[James Bruce]

Andrew Morton wrote:

>No.  All of the applications which you initially identified can be
>implemented by putting the various bits of data into a single file and
>getting applications to agree on the format of that file.
>  
>

So in order to avoid breaking backup and file utilities, we'd instead 
break every application that reads files?  That way surely lies madness 
:)  This /could/ possibly work with lots of cooperation from glibc, but 
we'd also need a way to insert and delete bytes in a file... moving 1MB 
of attributes/streams just to change a 20 character author attribute 
seems a bit silly.  In addition, there are little things, such as how 
adding a thumbnail image shouldn't change the modification time of the 
original picture IMO, but it does if you embed it in the file.

If it should be in userspace, we could take the OS-X approach of using 
directories for everything, and the "data" that would be in a hybrid 
directory-file is a specially named file under that directory.  Programs 
for instance are actually directories, with the elf file underneath it.  
When you click on the "directory-program" in the gui it runs the 
associated elf file rather than opening the directory.  I think that 
approach is promising, but so few unix programs have any indirection for 
file access it'd be hell to teach them all how it works.  Of course with 
emminent breakage of some sort, maybe now is the time...

 - Jim Bruce

Re: silent semantic changes with reiser4

[Jamie Lokier]

James Bruce wrote:
> moving 1MB of attributes/streams just to change a 20 character
> author attribute seems a bit silly.

That's what we do now.  Try editing an MP3's id tags.

> If it should be in userspace, we could take the OS-X approach of using 
> directories for everything, and the "data" that would be in a hybrid 
> directory-file is a specially named file under that directory.  Programs 
> for instance are actually directories, with the elf file underneath it.  
> When you click on the "directory-program" in the gui it runs the 
> associated elf file rather than opening the directory.  I think that 
> approach is promising, but so few unix programs have any indirection for 
> file access it'd be hell to teach them all how it works.

The interfaces we've been talking about are very close to that.  The
indirection is automatic: instead of a document (or program) looking
like a directory, it'll look like a file, and reading (or executing)
the file will do the indirection, with kernel control (and so
consistency and performance) via an appropriate flat<->structured
relation helper.

-- Jamie

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 18:58, Jamie Lokier wrote:
> James Bruce wrote:
> > moving 1MB of attributes/streams just to change a 20 character
> > author attribute seems a bit silly.
>
> That's what we do now.  Try editing an MP3's id tags.

What's the problem? Let's say a file has three streams/forks/whatever: the 
unnamed one, and two additional named. The filesystem can treat these three 
streams like three independent (but related) files on disk so, if the author 
attribute belongs to the first named stream, only the first named stream (and 
its corresponding file on the backend) would need to be modified. No need to 
move MBs of data around.

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrew Morton wrote:

>
>No.  All of the applications which you initially identified can be
>implemented by putting the various bits of data into a single file and
>getting applications to agree on the format of that file.
>
>For example, some image file formats already support embedded metadata, do
>they not?
>
>-
>  
>
and that leads to vasts numbers of file formats, all of which require 
special code to parse them, and apps that don't duplicate that code 
can't access the data, and it is a software engineering mistake.  
Uniform interfaces greatly reduce the cost of an OS and increase its 
expressive power.  Closure is the most important and least understood 
principle of OS design.

By contrast, suppose everything was stored in files and directories.  
Every app can afford the coding cost to learn about files and 
directories.  Data can freely bounce from one object to another in the 
OS because the APIs for the objects are all the same.  That's increased 
expressive power.  The number of connections between objects determines 
the expressive power of the OS, not the number of objects in it.  
Unified namespaces do a lot for an OS.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrew Morton wrote:

>
>No.  All of the applications which you initially identified can be
>implemented by putting the various bits of data into a single file and
>getting applications to agree on the format of that file.
>
>For example, some image file formats already support embedded metadata, do
>they not?
>
>-
>  
>
and that leads to vasts numbers of file formats, all of which require 
special code to parse them, and apps that don't duplicate that code 
can't access the data, and it is a software engineering mistake.  
Uniform interfaces greatly reduce the cost of an OS and increase its 
expressive power.  Closure is the most important and least understood 
principle of OS design.

By contrast, suppose everything was stored in files and directories.  
Every app can afford the coding cost to learn about files and 
directories.  Data can freely bounce from one object to another in the 
OS because the APIs for the objects are all the same.  That's increased 
expressive power.  The number of connections between objects determines 
the expressive power of the OS, not the number of objects in it.  
Unified namespaces do a lot for an OS.

Hans

Re: silent semantic changes with reiser4

[Bernd Petrovitsch]

[ shortened Cc: of reiser- and other folks since it is no longer
reiser-specific - sorry, if it was too much ]
[ fixed missing attribution ]

On Thu, 2004-08-26 at 12:51 +0200, Spam wrote:
> On Thu, 2004-08-26 at 03:24 -0700, Andrew Morton wrote:
> > Spam <spam@tnonline.net> wrote:
[...]
> >>   Applications  that support the new features will benefit, all others
> >>   will continue to work without destroying data.
> 
> > Sorry, but that all sounds a bit fluffy.   Please provide some examples.

It is too fluffy.

>   We  already had the examples with cp and mv. Both should continue to
>   work and the files will still be copied. The same with Konqueror and

... after they are patched to support streams. As stated each stream has
an own fd, so - from the user-space perspective - one has to open *all*
streams of this file, read them and write them into the destination.

>   Nautilus.  Files  and  their  meta-files/streams/attributes  will be
>   retained as long as applications are using the OS API.

The OS-API is *one* file descriptor where you can read, write, mmap, ...
on. Therefore stream selection must happen with open() since open()
returns a fd (which uniquely identify a stream). Voila.
And actually this makes transition actually possible: Old apps simply
ignore streams (yes, there will be a default stream, which is used id
nothing else is specified) and throw them away (think of e.g. patch
which opens a new file, writes into it, unlink's the old one and mv's
the new one on the old place) until they are modified to support them.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: silent semantic changes with reiser4

[Hans Reiser]

What is important is that it all be one packaged solution from the same 
maintainer.  That is necessary for coherency of solution.

There is also benefit to it being one coherent body of code, but this is 
harder to articulate.  It is always hard to articulate why functionality 
belongs in the same program rather than split into multiple programs.  
Filesystems should provide primitives for ordering, structuring, and 
naming objects.  It is hard to say why we should do all of that, but on 
some level I just know it is our task.

Hans

Spam wrote:

>
>
>  You  said  it  would be socially hard. I think it would be very much
>  close to impossible to get it right. Imagine that Gnome and Nautilus
>  would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
>  app-xyz  would  implement  this anytime soon and in the meantime the
>  data is at risk.
>
>
>  
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

What is important is that it all be one packaged solution from the same 
maintainer.  That is necessary for coherency of solution.

There is also benefit to it being one coherent body of code, but this is 
harder to articulate.  It is always hard to articulate why functionality 
belongs in the same program rather than split into multiple programs.  
Filesystems should provide primitives for ordering, structuring, and 
naming objects.  It is hard to say why we should do all of that, but on 
some level I just know it is our task.

Hans

Spam wrote:

>
>
>  You  said  it  would be socially hard. I think it would be very much
>  close to impossible to get it right. Imagine that Gnome and Nautilus
>  would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
>  app-xyz  would  implement  this anytime soon and in the meantime the
>  data is at risk.
>
>
>  
>
>  
>

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Spam wrote:
>   Because  having user space tools and code will make it not work with
>   everything. Keeping stuff in the kernel should make the new features
>   transparent to the applications.

But having it in the kernel has the same problem. If you read the
Solaris documentation you will see that a bunch of utilties had to
get a new commandline option to be able to access the metadata and
a special utility was added for other applications. If you look
at windows you will see that you need to use a filename like
<realfilename>:<streamname>:$DATA (which obviously does not work for
single-character filenames).

Ignoring samba for a bit which just needs streams to stay compatible
with windows I see few reasons for using streams:

* files are more complex these days and tend to include multiple
  different things: images with thumbnails and exif data, 'office'
  documents containing both text and images

* standard way to add common metadata to a file which can be used for
  searching tools (author, copyright, keywords, etc.)

But both can already be done in userland (modern image formats can store
thumbnails and exif data internally, applications use tar or zip-like
files for documents, etc.). The metadata part is a lot more complicated
as well since the behaviour of attributes might need to be complex:
if I change an image using an application that is not stream-aware, what
should happen to its thumbnail? 

The only common benefits I can see are standardisation and optimization:
instead of every file format or application defining a way to specify 
metadata for a file you get a common API defined by the OS (but you'll
still need to standardize on attribute names and formats, so plenty of
room that will still not help), and instead of parsing different files
or XML streams you can directly access a bit of metadata.

So far I'm not convinced that streams are worth the effort. Not that my
opinion is all that relevant here, but still :)

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Hans Reiser]

Wichert Akkerman wrote:

>Previously Spam wrote:
>  
>
>>  Because  having user space tools and code will make it not work with
>>  everything. Keeping stuff in the kernel should make the new features
>>  transparent to the applications.
>>    
>>
>
>But having it in the kernel has the same problem. If you read the
>Solaris documentation you will see that a bunch of utilties had to
>get a new commandline option to be able to access the metadata and
>a special utility was added for other applications.
>
They did it wrong.

> If you look
>at windows you will see that you need to use a filename like
><realfilename>:<streamname>:$DATA (which obviously does not work for
>single-character filenames).
>  
>
again, they did it wrong.  See Rob Pike's "The Hideous Name" for why 
that is wrong.

>Ignoring samba for a bit which just needs streams to stay compatible
>with windows I see few reasons for using streams:
>
>* files are more complex these days and tend to include multiple
>  different things: images with thumbnails and exif data, 'office'
>  documents containing both text and images
>
>* standard way to add common metadata to a file which can be used for
>  searching tools (author, copyright, keywords, etc.)
>
>But both can already be done in userland (modern image formats can store
>thumbnails and exif data internally, applications use tar or zip-like
>files for documents, etc.). The metadata part is a lot more complicated
>as well since the behaviour of attributes might need to be complex:
>if I change an image using an application that is not stream-aware, what
>should happen to its thumbnail? 
>
>The only common benefits I can see are standardisation and optimization:
>instead of every file format or application defining a way to specify 
>metadata for a file you get a common API defined by the OS (but you'll
>still need to standardize on attribute names and formats, so plenty of
>room that will still not help), and instead of parsing different files
>or XML streams you can directly access a bit of metadata.
>
>So far I'm not convinced that streams are worth the effort. Not that my
>opinion is all that relevant here, but still :)
>
>Wichert.
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Wichert Akkerman wrote:

>Previously Spam wrote:
>  
>
>>  Because  having user space tools and code will make it not work with
>>  everything. Keeping stuff in the kernel should make the new features
>>  transparent to the applications.
>>    
>>
>
>But having it in the kernel has the same problem. If you read the
>Solaris documentation you will see that a bunch of utilties had to
>get a new commandline option to be able to access the metadata and
>a special utility was added for other applications.
>
They did it wrong.

> If you look
>at windows you will see that you need to use a filename like
><realfilename>:<streamname>:$DATA (which obviously does not work for
>single-character filenames).
>  
>
again, they did it wrong.  See Rob Pike's "The Hideous Name" for why 
that is wrong.

>Ignoring samba for a bit which just needs streams to stay compatible
>with windows I see few reasons for using streams:
>
>* files are more complex these days and tend to include multiple
>  different things: images with thumbnails and exif data, 'office'
>  documents containing both text and images
>
>* standard way to add common metadata to a file which can be used for
>  searching tools (author, copyright, keywords, etc.)
>
>But both can already be done in userland (modern image formats can store
>thumbnails and exif data internally, applications use tar or zip-like
>files for documents, etc.). The metadata part is a lot more complicated
>as well since the behaviour of attributes might need to be complex:
>if I change an image using an application that is not stream-aware, what
>should happen to its thumbnail? 
>
>The only common benefits I can see are standardisation and optimization:
>instead of every file format or application defining a way to specify 
>metadata for a file you get a common API defined by the OS (but you'll
>still need to standardize on attribute names and formats, so plenty of
>room that will still not help), and instead of parsing different files
>or XML streams you can directly access a bit of metadata.
>
>So far I'm not convinced that streams are worth the effort. Not that my
>opinion is all that relevant here, but still :)
>
>Wichert.
>
>  
>

Re: silent semantic changes with reiser4

[Christer Weinigel]

Spam <spam@tnonline.net> writes:

>   Keeping stuff in the kernel should make the new features
>   transparent to the applications.

No, it will make just one special case, rename within the same
filesystem, work.  (Well, two special cases, rm will also delete
the other forks). 

Unless we add a new copy(2) syscall (which would be nice for other
reasons) all applicatons that copy files will fail to copy the
streams.  So no working cp command, no nautilus nor konqueror without
changes to the application.  And if you have to change the
applications anyway, isn't it much better to agree on a shared library
in userspace that everyone uses?  Which has the added bonus that it
can be made to work on top of Linux, Windows, Ultrix and VMS?

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Christer Weinigel]

Spam <spam@tnonline.net> writes:

>   Keeping stuff in the kernel should make the new features
>   transparent to the applications.

No, it will make just one special case, rename within the same
filesystem, work.  (Well, two special cases, rm will also delete
the other forks). 

Unless we add a new copy(2) syscall (which would be nice for other
reasons) all applicatons that copy files will fail to copy the
streams.  So no working cp command, no nautilus nor konqueror without
changes to the application.  And if you have to change the
applications anyway, isn't it much better to agree on a shared library
in userspace that everyone uses?  Which has the added bonus that it
can be made to work on top of Linux, Windows, Ultrix and VMS?

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> writes:

>>   Keeping stuff in the kernel should make the new features
>>   transparent to the applications.

> No, it will make just one special case, rename within the same
> filesystem, work.  (Well, two special cases, rm will also delete
> the other forks). 

> Unless we add a new copy(2) syscall (which would be nice for other
> reasons) all applicatons that copy files will fail to copy the
> streams.

  How  so?  The whole idea is that the underlaying OS that handles the
  copying  should  also  know  to  copy  everything, otherwise you can
  implement  everything  into  applications  and  just  skip the whole
  filesystem part.

  Of  course,  it  won't  work if applications themselves are actually
  transferring  the  data  and  create the new files directly onto the
  filesystem. This just seem so wrong.

  ~S

>   So no working cp command, no nautilus nor konqueror without
> changes to the application.  And if you have to change the
> applications anyway, isn't it much better to agree on a shared library
> in userspace that everyone uses?  Which has the added bonus that it
> can be made to work on top of Linux, Windows, Ultrix and VMS?

>   /Christer

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> writes:

>>   Keeping stuff in the kernel should make the new features
>>   transparent to the applications.

> No, it will make just one special case, rename within the same
> filesystem, work.  (Well, two special cases, rm will also delete
> the other forks). 

> Unless we add a new copy(2) syscall (which would be nice for other
> reasons) all applicatons that copy files will fail to copy the
> streams.

  How  so?  The whole idea is that the underlaying OS that handles the
  copying  should  also  know  to  copy  everything, otherwise you can
  implement  everything  into  applications  and  just  skip the whole
  filesystem part.

  Of  course,  it  won't  work if applications themselves are actually
  transferring  the  data  and  create the new files directly onto the
  filesystem. This just seem so wrong.

  ~S

>   So no working cp command, no nautilus nor konqueror without
> changes to the application.  And if you have to change the
> applications anyway, isn't it much better to agree on a shared library
> in userspace that everyone uses?  Which has the added bonus that it
> can be made to work on top of Linux, Windows, Ultrix and VMS?

>   /Christer

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Spam wrote:
>   How  so?  The whole idea is that the underlaying OS that handles the
>   copying  should  also  know  to  copy  everything, otherwise you can
>   implement  everything  into  applications  and  just  skip the whole
>   filesystem part.

UNIX doesn't have a copy systemcall, applications copy the data
manually.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Spam]

> Previously Spam wrote:
>>   How  so?  The whole idea is that the underlaying OS that handles the
>>   copying  should  also  know  to  copy  everything, otherwise you can
>>   implement  everything  into  applications  and  just  skip the whole
>>   filesystem part.

> UNIX doesn't have a copy systemcall, applications copy the data
> manually.

  Oh, this is very unfortunate and should be a bigger issue to fix.

> Wichert.

Re: silent semantic changes with reiser4

[Spam]

> Previously Spam wrote:
>>   How  so?  The whole idea is that the underlaying OS that handles the
>>   copying  should  also  know  to  copy  everything, otherwise you can
>>   implement  everything  into  applications  and  just  skip the whole
>>   filesystem part.

> UNIX doesn't have a copy systemcall, applications copy the data
> manually.

  Oh, this is very unfortunate and should be a bigger issue to fix.

> Wichert.

Re: silent semantic changes with reiser4

[Bernd Petrovitsch]

On Thu, 2004-08-26 at 13:43 +0200, Spam wrote:
> > Previously Spam wrote:
> >>   How  so?  The whole idea is that the underlaying OS that handles the
> >>   copying  should  also  know  to  copy  everything, otherwise you can
> >>   implement  everything  into  applications  and  just  skip the whole
> >>   filesystem part.
> 
> > UNIX doesn't have a copy systemcall, applications copy the data
> > manually.
> 
>   Oh, this is very unfortunate and should be a bigger issue to fix.

Then you have to rewrite POSIX und SuSv3.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: silent semantic changes with reiser4

[Hans Reiser]

Bernd Petrovitsch wrote:

>On Thu, 2004-08-26 at 13:43 +0200, Spam wrote:
>  
>
>>>Previously Spam wrote:
>>>      
>>>
>>>>  How  so?  The whole idea is that the underlaying OS that handles the
>>>>  copying  should  also  know  to  copy  everything, otherwise you can
>>>>  implement  everything  into  applications  and  just  skip the whole
>>>>  filesystem part.
>>>>        
>>>>
>>>UNIX doesn't have a copy systemcall, applications copy the data
>>>manually.
>>>      
>>>
>>  Oh, this is very unfortunate and should be a bigger issue to fix.
>>    
>>
>
>Then you have to rewrite POSIX und SuSv3.
>
>	Bernd
>  
>
Spam is right.  Posix is a standard, not a vision, and the future is 
always a vision not a standard.

Re: silent semantic changes with reiser4

[Hans Reiser]

Bernd Petrovitsch wrote:

>On Thu, 2004-08-26 at 13:43 +0200, Spam wrote:
>  
>
>>>Previously Spam wrote:
>>>      
>>>
>>>>  How  so?  The whole idea is that the underlaying OS that handles the
>>>>  copying  should  also  know  to  copy  everything, otherwise you can
>>>>  implement  everything  into  applications  and  just  skip the whole
>>>>  filesystem part.
>>>>        
>>>>
>>>UNIX doesn't have a copy systemcall, applications copy the data
>>>manually.
>>>      
>>>
>>  Oh, this is very unfortunate and should be a bigger issue to fix.
>>    
>>
>
>Then you have to rewrite POSIX und SuSv3.
>
>	Bernd
>  
>
Spam is right.  Posix is a standard, not a vision, and the future is 
always a vision not a standard.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 01:05:02AM -0700, Hans Reiser wrote:
> Spam is right.  Posix is a standard, not a vision, and the future is 
> always a vision not a standard.

An old friend of mine always said "if someone has visions he needs to
stop somking mushrooms.."

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 09:41:07AM +0200, Bernd Petrovitsch wrote:
> > > UNIX doesn't have a copy systemcall, applications copy the data
> > > manually.
> > 
> >   Oh, this is very unfortunate and should be a bigger issue to fix.
> 
> Then you have to rewrite POSIX und SuSv3.

They don't say 'you must now have a copy syscall'.  Having one that's
actually used by system tools would be a great optimization for many
network or distributed filesystems.

Re: silent semantic changes with reiser4

[Bernd Petrovitsch]

On Fri, 2004-08-27 at 10:49 +0200, Christoph Hellwig wrote:

> On Fri, Aug 27, 2004 at 09:41:07AM +0200, Bernd Petrovitsch wrote:
> > > > UNIX doesn't have a copy systemcall, applications copy the data
> > > > manually.
> > > 
> > >   Oh, this is very unfortunate and should be a bigger issue to fix.
> > 
> > Then you have to rewrite POSIX und SuSv3.
> 
> They don't say 'you must now have a copy syscall'.  Having one that's
> actually used by system tools would be a great optimization for many
> network or distributed filesystems.

ACK. But actually spam apparently assumes that the kernel - userspace
interface is at the wrong abstraction level. And this is conceptually
bound IMHO to POSIX.
And it does not solve the problem, that the read() and write() sys-calls
will not vanish. Even if a copy() sys-call exists.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: silent semantic changes with reiser4

[Hans Reiser]

Wichert Akkerman wrote:

>Previously Spam wrote:
>  
>
>>  How  so?  The whole idea is that the underlaying OS that handles the
>>  copying  should  also  know  to  copy  everything, otherwise you can
>>  implement  everything  into  applications  and  just  skip the whole
>>  filesystem part.
>>    
>>
>
>UNIX doesn't have a copy systemcall, applications copy the data
>manually.
>
>Wichert.
>
>  
>
See sys_reiser4()..... ;-)  you can go "A<-B".  I have hopes of getting 
drive vendors to support us in doing single disk copies without even 
leaving the disk drive.....

Re: silent semantic changes with reiser4

[Hans Reiser]

Wichert Akkerman wrote:

>Previously Spam wrote:
>  
>
>>  How  so?  The whole idea is that the underlaying OS that handles the
>>  copying  should  also  know  to  copy  everything, otherwise you can
>>  implement  everything  into  applications  and  just  skip the whole
>>  filesystem part.
>>    
>>
>
>UNIX doesn't have a copy systemcall, applications copy the data
>manually.
>
>Wichert.
>
>  
>
See sys_reiser4()..... ;-)  you can go "A<-B".  I have hopes of getting 
drive vendors to support us in doing single disk copies without even 
leaving the disk drive.....

Re: silent semantic changes with reiser4

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

On Aug 26, 2004  16:56 -0700, Hans Reiser wrote:
> Wichert Akkerman wrote:
> >UNIX doesn't have a copy systemcall, applications copy the data
> >manually.
>
> See sys_reiser4()..... ;-)  you can go "A<-B".  I have hopes of getting 
> drive vendors to support us in doing single disk copies without even 
> leaving the disk drive.....

Yes, we've wanted to have a "copy" syscall for a while now also, so that
Lustre (or other network FS for that matter) can do remote copies of
large files without having to go through the VM of the client.  The work
on sendfile() in 2.6 is a step in that direction, I eagerly await the
day when "cp" will try to do a sys_sendfile() call first, before doing
the read+write looping it does now.

Consider for a network filesystem we are limited to 1/2 of the client
network bandwidth (say 50-75MB/s) for copying instead of the server-local
or multiple server-server bandwidth for copies (400-600MB/s or more).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/             http://members.shaw.ca/golinux/


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Stelian Ionescu]

On Thu, Aug 26, 2004 at 01:33:32PM +0200, Wichert Akkerman wrote:
>Previously Spam wrote:
>>   How  so?  The whole idea is that the underlaying OS that handles the
>>   copying  should  also  know  to  copy  everything, otherwise you can
>>   implement  everything  into  applications  and  just  skip the whole
>>   filesystem part.
>
>UNIX doesn't have a copy systemcall, applications copy the data
>manually.
couldn't sendfile(2) be used for that ?
perhaps a flag parameter could be added to sendfile(2) which specifies
whether to copy _all_ metadata if both fd's correspond to files, something
like either O_METADATA or O_NOMETADATA

-- 
Stelian Ionescu aka fe[nl]ix

Quidquid latine dictum sit, altum viditur.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christer Weinigel wrote:
> all applicatons that copy files will fail to copy the streams.  So
> no working cp command, no nautilus nor konqueror without changes to
> the application.

No true.  A lot of the fancy stuff depends on metadata which is merely
another view of the serialised contents in the flat file.  Nautilus &
Konqueror thumbnails are an example of that (I gave a few more in
other mails).  cp works fine with that, and so does ftp.  The metadata
is just recalculated when it's needed from the target file.

> And if you have to change the applications anyway, isn't it much
> better to agree on a shared library in userspace that everyone uses?
> Which has the added bonus that it can be made to work on top of
> Linux, Windows, Ultrix and VMS?

Ideally yes, a shared library _or_ at least an agreed representation.

However, with filesystem support you can improve performance by
avoiding unneed serialisation (write to your huge *Office
presentation, read it from another program, no need to wait for the
slow tar+compress stage yet it's _as if_ those were done), retain
computed metadata with coherency guarantees (e.g. real time search
indexes, crypto digests and such), and let the filesystem decide when
best to prune computed metadata or convert representations.

All of these things can work with a combination of userspace plugins
(not the same as reiser4 plugins), and filesystem support.

Without the filesystem support, you can still use the userspace
plugins -- so apps would still be portable -- but you don't get the
extras mentioned above.

-- Jamie

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 14:16, Christer Weinigel wrote:
> Spam <spam@tnonline.net> writes:
> >   Keeping stuff in the kernel should make the new features
> >   transparent to the applications.
>
> No, it will make just one special case, rename within the same
> filesystem, work.  (Well, two special cases, rm will also delete
> the other forks).

In traditional UNIX filesystems, copy loses some info too
(owner,group,mode bits). cp and tar was taught about this
and they now can preserve that info.

However, I do not think that hacking on cp, tar, etc everytime
we need to add something to our filesystems (xattrs, ACLs,
whatever) is a terribly attractive idea.

I think reiserfs is going to handle that by giving uniform access
to such ancillary data. It will be visible as file/meta/*
filelets ('small files') or something like that and can be saved/restored
by e.g. tar *without* needing to know what data is stored there, and
in what format.

I think Hans is not planning turning old "file is a stream of bytes"
into eight-stream octopus. One stream will remain as a 'main' one,
which contains actual data. Others will keep metadata, etc...

Note that even today's files aren't simple "streams of bytes"
because they also have names,mode bits etc.
--
vda

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 14:16, Christer Weinigel wrote:
> Spam <spam@tnonline.net> writes:
> >   Keeping stuff in the kernel should make the new features
> >   transparent to the applications.
>
> No, it will make just one special case, rename within the same
> filesystem, work.  (Well, two special cases, rm will also delete
> the other forks).

In traditional UNIX filesystems, copy loses some info too
(owner,group,mode bits). cp and tar was taught about this
and they now can preserve that info.

However, I do not think that hacking on cp, tar, etc everytime
we need to add something to our filesystems (xattrs, ACLs,
whatever) is a terribly attractive idea.

I think reiserfs is going to handle that by giving uniform access
to such ancillary data. It will be visible as file/meta/*
filelets ('small files') or something like that and can be saved/restored
by e.g. tar *without* needing to know what data is stored there, and
in what format.

I think Hans is not planning turning old "file is a stream of bytes"
into eight-stream octopus. One stream will remain as a 'main' one,
which contains actual data. Others will keep metadata, etc...

Note that even today's files aren't simple "streams of bytes"
because they also have names,mode bits etc.
--
vda

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Denis Vlasenko wrote:

> I think Hans is not planning turning old "file is a stream of bytes"
> into eight-stream octopus. One stream will remain as a 'main' one,
> which contains actual data. Others will keep metadata, etc...

This is exactly what the Samba people want, though. 

Office suites can store a document with embedded images
and spread sheets "easily" by putting the text, the
images and spread sheets all in different file streams.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Denis Vlasenko wrote:

> I think Hans is not planning turning old "file is a stream of bytes"
> into eight-stream octopus. One stream will remain as a 'main' one,
> which contains actual data. Others will keep metadata, etc...

This is exactly what the Samba people want, though. 

Office suites can store a document with embedded images
and spread sheets "easily" by putting the text, the
images and spread sheets all in different file streams.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

Am Donnerstag, den 26.08.2004, 10:12 -0400 schrieb Rik van Riel:

> > I think Hans is not planning turning old "file is a stream of bytes"
> > into eight-stream octopus. One stream will remain as a 'main' one,
> > which contains actual data. Others will keep metadata, etc...
> 
> This is exactly what the Samba people want, though. 
> 
> Office suites can store a document with embedded images
> and spread sheets "easily" by putting the text, the
> images and spread sheets all in different file streams.

Ouch.

I think Hans' idea is (I don't know if it is a good idea nor if it is
doable, but at least it sounds interesting) to have special compound
files where you can do something like this:

cp text.txt test.compound/test.txt
cp image.jpg test.compound/image.jpg

And if you read test.compound (the main stream) you get a special format
that contains all the components. You can copy that single stream of
bytes to another (reiser4) fs and then access test.compound/test.txt
again.

The only thing I'm worrying about with this approach is what happens if
someone tries to simultaneously open test.compound and
test.compound/test.txt.

Hans has an example somewhere where he does something like that
with /etc/passwd.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> I think Hans' idea is (I don't know if it is a good idea nor if it is
> doable, but at least it sounds interesting) to have special compound
> files where you can do something like this:
> 
> cp text.txt test.compound/test.txt
> cp image.jpg test.compound/image.jpg
> 
> And if you read test.compound (the main stream) you get a special format
> that contains all the components. You can copy that single stream of
> bytes to another (reiser4) fs and then access test.compound/test.txt
> again.

(To Rik especially), this is the design which more or less satisfies
lots of different goals at once.

I wrote:
> Properly implemented metadata can:
> 
>   (1) operate in both modes simultaneously;
>   (2) work with unaware applications;
>   (3) provide performance enhancements to aware applications;
>   (4) provide storage enhancements to both;
>   (5) provide useful features that work with standard unmodified unix tools,

(1) is by design.

You can copy these "files" safely using cp or cat or ftp.
You can email them.  They're flat files.  That's (2).

You can poke around inside them.

Applications which do poke around inside get performance gains,
because they don't have to serialise the flat format every time they
"load" or "save" a file.  For example, if you have a huge OpenOffice
document containing lots of sub-documents and pictures.  We can
imagine a version of OpenOffice which just opens the sub-documents,
and writes the ones which are changed after editing.  Also, another
program such as a viewer, post-processor or search indexing program
only needs to read the parts which have changed; it doesn't have to
decode the whole thing after a change.  That's (3).

We already know how useful this is with big mailboxes, hence people
switching to maildir.  Now imagine the attachments in each message are
also viewable, cat'able etc., and that the filesystem stores each
message compressed automatically, or in aggregate (your choice) and
that you can use both mbox and maildir tools on the same mailbox. That's (4).

You can ls, cp, grep, less, vi, gqview etc. the pieces of files, and
every supported file format needs just one tool to provide the
interface to its contents.  That's (5).

While I'm here I might as well add:

  (6) improve support for real time search indexing.

The "view coherency" mechanism which is needed for efficient
userspace VFS support has just the right properties to keep informing
an indexer of changes.  They could even enforce operation orderings if
we cared about _perfect_ real-timeness, although we probably don't.

  (7) yet another way of doing distros.

Why not just install .tgz or .zip files of entire binary packages, and
let the filesystem unpack the bits that are needed when they're
needed.  Some platforms (i.e. Java) do this already with .jar files
and it's quite popular.  This would extend it in a versatile way.

Rik, I hope I've explained how those 5 properties are satisfied by
Hans Reiser's brilliant design (ahem, great minds think alike, and
remember Linus likes files-as-directories too :).  But if I haven't
please do pick holes.

> The only thing I'm worrying about with this approach is what happens if
> someone tries to simultaneously open test.compound and
> test.compound/test.txt.

It is possible to implement it so that every change to one file
immediately affects the other in the appropriate way, _as if_ the
other file is regenerated every time before reading it.

This is what I mean by coherent views.  It can be done even when
userspace is doing the work, and the obvious mechanisms just happen to
have the property of being useful in other ways (like a better
dnotify, but I digress...).  They're just an ordinary cache coherency
protocols.

However, if you are writing a few bytes to test.compound, then the
coherent view in test.compound/test.txt is probably a corrupt or
non-existent file, because test.compound's intermediate state is a
truncated and/or corrupt archive file, until you've written some more.

That doesn't look nice, but it's exactly the same problem as you get
when you're writing to a file that another program is reading.  If you
do that on a live http server, for example editing a .html file, very
occasionally someone will see a truncated file.  Even if your editor
uses renames to update atomically, sometimes you need to update more
than one file atomically.  It can be done, but rarely is.

Is there a solution to this?  The obvious one is when you're writing
to test.compound, don't read from test.compound/test.txt.  That's _no
different_ from the advice which says when you're saving a file from
OpenOffice, don't load it into a document viewer during the time you're
writing it, and don't serve it.

The other advice is to save to a different file and call rename().
Again, you get pretty much the same results as now: it atomically
changes, and currently open references to the old contents will
continue to reference the old contents.

There's another solution: transactions.  reiser4 offers some kind of
transaction facility, doesn't it?  This is a fine example of when
might want to declare that a sequence of writes needs to be a single
transaction, visible only when the whole sequence is complete.  Perfect!

I don't know reiser4's interface to that, but it would make a lot of
sense to have a simple shell tool called "atomic", to wrap a sequence
of operations.  That would be useful for writing files _and_ useful
for those pesky "update several files on the web server at once"
moments.  (Yes I know you can build a hard link tree and rename the
root directory, although even that doesn't always work).

-- Jamie

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
> Christophe Saout wrote:

> > And if you read test.compound (the main stream) you get a special format
> > that contains all the components. You can copy that single stream of
> > bytes to another (reiser4) fs and then access test.compound/test.txt
> > again.
> 
> (To Rik especially), this is the design which more or less satisfies
> lots of different goals at once.

And if an unaware application reads the compound file
and then writes it out again, does the filesystem
interpret the contents and create the other streams ?

Unless I overlook something (please tell me what), the
scheme just proposed requires filesystems to look at
the content of files that is being written out, in
order to make the streams work.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
> Christophe Saout wrote:

> > And if you read test.compound (the main stream) you get a special format
> > that contains all the components. You can copy that single stream of
> > bytes to another (reiser4) fs and then access test.compound/test.txt
> > again.
> 
> (To Rik especially), this is the design which more or less satisfies
> lots of different goals at once.

And if an unaware application reads the compound file
and then writes it out again, does the filesystem
interpret the contents and create the other streams ?

Unless I overlook something (please tell me what), the
scheme just proposed requires filesystems to look at
the content of files that is being written out, in
order to make the streams work.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel:

> > > And if you read test.compound (the main stream) you get a special format
> > > that contains all the components. You can copy that single stream of
> > > bytes to another (reiser4) fs and then access test.compound/test.txt
> > > again.
> > 
> > (To Rik especially), this is the design which more or less satisfies
> > lots of different goals at once.
> 
> And if an unaware application reads the compound file
> and then writes it out again, does the filesystem
> interpret the contents and create the other streams ?
> 
> Unless I overlook something (please tell me what), the
> scheme just proposed requires filesystems to look at
> the content of files that is being written out, in
> order to make the streams work.

Yes, the main compound stream unaware applications would see would just
be a writable view of its contents. Probably not trivial to implement
but doable.

It should be a simple format. Something simliar to tar. The worst thing
that can happen is people start writing plugins for every existing
compound format out there. It should be the other way around, agree on a
simple compound format and encourage applications to use this one if the
want to use this advantage.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel:

> > > And if you read test.compound (the main stream) you get a special format
> > > that contains all the components. You can copy that single stream of
> > > bytes to another (reiser4) fs and then access test.compound/test.txt
> > > again.
> > 
> > (To Rik especially), this is the design which more or less satisfies
> > lots of different goals at once.
> 
> And if an unaware application reads the compound file
> and then writes it out again, does the filesystem
> interpret the contents and create the other streams ?
> 
> Unless I overlook something (please tell me what), the
> scheme just proposed requires filesystems to look at
> the content of files that is being written out, in
> order to make the streams work.

Yes, the main compound stream unaware applications would see would just
be a writable view of its contents. Probably not trivial to implement
but doable.

It should be a simple format. Something simliar to tar. The worst thing
that can happen is people start writing plugins for every existing
compound format out there. It should be the other way around, agree on a
simple compound format and encourage applications to use this one if the
want to use this advantage.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 06:04:42PM +0200, Christophe Saout wrote:
> It should be a simple format. Something simliar to tar. The worst thing
> that can happen is people start writing plugins for every existing
> compound format out there. It should be the other way around, agree on a
> simple compound format and encourage applications to use this one if the
> want to use this advantage.

or placing this into a userspace helper that the kernel can invoke
transparently.  If you're doing funny thing like that for your swap
device deadlocks are the last of your worries :)

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> or placing this into a userspace helper that the kernel can invoke
> transparently.

Yes, exactly this for most file formats.
Ideally, with _some_ user control over the handlers for their own
files a bit like Hurd.

Not just archive files: metadata pulled from audio files, images,
distro packages, etc.  In other words much the same as Gnome and KDE's
things, but with more emphasis on metadata extractors and efficient
on-disk cache interaction, and a sane page-cache coherency interface
(possibly FUSE).

Metadata extraction is useful for search and indexing purposes.
(That, and a way to hook change notifications up with indexers --
again comes with any proper coherency interface).

For new applications using a container, it's tempting to want one
format which is optimised for this.  But really many applications want
to write XML or compressed XML, or something else that isn't
especially efficient but is easy to work with on many OSes.  We should
support applications whatever format they need to work with, if
providing an interior view as a directory is useful for them.  (This
implies applications can introduce their own formats for files they
need to operate on.)  By the way, lazy updating of containers is an
even bigger advantage if you have to use XML as the flat format,
because the savings from not serialising when you don't need to are
more pronounced.

For some applications the "content" isn't structured in a way which
maps naturally to a plain archive file.  Hans' /etc/passwd is a simple
example: at least one of the "flat" representations, when there are
enough entries, probably ought to be a DB file for faster lookup.
(Though, it would be great if /etc/passwd could automagically get fast
lookup due to an automagic underlying representation, while still
appearing to be a flat file when you cat it).

Also, imagine "cd" into a MySQL table file.  It might just work.

By the way, we don't need containers to waste huge amounts of space
like the Microsoft Word "fast save" documents did.  Because our
containers are created on demand, there is no performance advantage in
them having unused holes, unlike the case were Word was saving changes
periodically.

-- Jamie

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Christophe Saout wrote:

| Yes, the main compound stream unaware applications would see would just
| be a writable view of its contents. Probably not trivial to implement
| but doable.

Not the main stream, I think, but we do need a way to make "eight-stream
octopus" files portable.  But there needs to be a copy(2) syscall, and
apps need to start supporting it.  After all, copy-on-write was planned
for Reiser4, now 4.1*.  And don't anyone dare suggest "userland library".

| It should be a simple format. Something simliar to tar. The worst thing

Simple, yes.  Similar to tar, no.

How about a serializer plugin?  foo.mp3/serialize (say) should return
some single stream which contains all of foo.mp3.  It could work for
directories as well, recursively, at least on the same filesystem --
you'd back up by doing

cp /serialize /backup/20040826

This has two potential problems.  One is security -- I want to be able
to serialize foo.mp3 without being root, but root doesn't want me to be
able to serialize /etc and thus get access to shadow.

The second problem is that in order to actually be used for backup, we
need snapshots, which are not done yet*.  Currently, it'd have to lock
the file/directory (recursively), which you don't want to do to entire
live filesystems...  So, no massive backups of live systems yet, but
definitely useful for sending around mp3s and such.

| that can happen is people start writing plugins for every existing
| compound format out there. It should be the other way around, agree on a
| simple compound format and encourage applications to use this one if the
| want to use this advantage.

For the format, I vote for whatever format we're using at the storage
layer.  It'd be a lot faster that way, both in execution time (I think)
and in development time (I'm sure).  Plus, imagine the restore:

cat /backup/20040826 > /dev/hda5
resizefs_reiser4 /dev/hda5


* Disclaimer -- all this is "I think, last I checked".


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQS4uhngHNmZLgCUhAQKTaw//WCHt0Umgaf0GaMUpEr0PLhiXhFTYQqKV
MKgpVt6SjP7lC9k3RbuCnPCLyOJGb9lIqo+Wd+B3b4TcQAQd4OYsSI6WiMPQjxXY
OdNnB7/xYkkd/ULw7/7IGcokyc4FRDjhTyOMe4KscQS/6uhdX1dCriCCzVq7iELx
u3HLulunHv1bwHbnZ6CyEZdQEr3j8XUu62+IXPTc6hWnfRRCvskV8Y4qyyVd8wFY
Kp44TxbJJSzDeWmuI/YYkxVJsPmSoxRLxmw9/B8D72KGNg1SYLnM4ueIc5IV8RH0
kG/o8tzwTRxE/eqfcHP+UBt6yXZR/i84DDogAmJFKP73g9X+U+J5o/kONOBQODhY
qY5LUwQu4UjlzXttMxC4NmZPESdUw85DgC/GhTQlsBMYBcmQHmpLAK67qt83+Kg0
1QqGPqJm4HL4weHLNknJ0DukLNKH7Bw5CVg335IwruaoiKUG7HDgthUhA319FCab
VvYa+bEfeBEw9RorSE/gWM9Hg73luM7eoSzKc9zHgx++cACTi8SBsTFWqI2FU6DX
UyWG5sNPybOiczpPsFu5S5FJBBNM3fLjkfeYgfqxUIjxGTNeUn/g/9I5ShUrCF1o
T+3nd1jFxfoehYGkfGB2YLpdeNVBv4EdKK16j39+iducfJrXnRlL1yjdo/Kte2uz
tTQZw3uTnhU=
=7bBF
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Christophe Saout wrote:

| Yes, the main compound stream unaware applications would see would just
| be a writable view of its contents. Probably not trivial to implement
| but doable.

Not the main stream, I think, but we do need a way to make "eight-stream
octopus" files portable.  But there needs to be a copy(2) syscall, and
apps need to start supporting it.  After all, copy-on-write was planned
for Reiser4, now 4.1*.  And don't anyone dare suggest "userland library".

| It should be a simple format. Something simliar to tar. The worst thing

Simple, yes.  Similar to tar, no.

How about a serializer plugin?  foo.mp3/serialize (say) should return
some single stream which contains all of foo.mp3.  It could work for
directories as well, recursively, at least on the same filesystem --
you'd back up by doing

cp /serialize /backup/20040826

This has two potential problems.  One is security -- I want to be able
to serialize foo.mp3 without being root, but root doesn't want me to be
able to serialize /etc and thus get access to shadow.

The second problem is that in order to actually be used for backup, we
need snapshots, which are not done yet*.  Currently, it'd have to lock
the file/directory (recursively), which you don't want to do to entire
live filesystems...  So, no massive backups of live systems yet, but
definitely useful for sending around mp3s and such.

| that can happen is people start writing plugins for every existing
| compound format out there. It should be the other way around, agree on a
| simple compound format and encourage applications to use this one if the
| want to use this advantage.

For the format, I vote for whatever format we're using at the storage
layer.  It'd be a lot faster that way, both in execution time (I think)
and in development time (I'm sure).  Plus, imagine the restore:

cat /backup/20040826 > /dev/hda5
resizefs_reiser4 /dev/hda5


* Disclaimer -- all this is "I think, last I checked".


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQS4uhngHNmZLgCUhAQKTaw//WCHt0Umgaf0GaMUpEr0PLhiXhFTYQqKV
MKgpVt6SjP7lC9k3RbuCnPCLyOJGb9lIqo+Wd+B3b4TcQAQd4OYsSI6WiMPQjxXY
OdNnB7/xYkkd/ULw7/7IGcokyc4FRDjhTyOMe4KscQS/6uhdX1dCriCCzVq7iELx
u3HLulunHv1bwHbnZ6CyEZdQEr3j8XUu62+IXPTc6hWnfRRCvskV8Y4qyyVd8wFY
Kp44TxbJJSzDeWmuI/YYkxVJsPmSoxRLxmw9/B8D72KGNg1SYLnM4ueIc5IV8RH0
kG/o8tzwTRxE/eqfcHP+UBt6yXZR/i84DDogAmJFKP73g9X+U+J5o/kONOBQODhY
qY5LUwQu4UjlzXttMxC4NmZPESdUw85DgC/GhTQlsBMYBcmQHmpLAK67qt83+Kg0
1QqGPqJm4HL4weHLNknJ0DukLNKH7Bw5CVg335IwruaoiKUG7HDgthUhA319FCab
VvYa+bEfeBEw9RorSE/gWM9Hg73luM7eoSzKc9zHgx++cACTi8SBsTFWqI2FU6DX
UyWG5sNPybOiczpPsFu5S5FJBBNM3fLjkfeYgfqxUIjxGTNeUn/g/9I5ShUrCF1o
T+3nd1jFxfoehYGkfGB2YLpdeNVBv4EdKK16j39+iducfJrXnRlL1yjdo/Kte2uz
tTQZw3uTnhU=
=7bBF
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[David Lang]

On Thu, 26 Aug 2004, David Masover wrote:

> Christophe Saout wrote:
>
> | Yes, the main compound stream unaware applications would see would just
> | be a writable view of its contents. Probably not trivial to implement
> | but doable.
>
> Not the main stream, I think, but we do need a way to make "eight-stream
> octopus" files portable.  But there needs to be a copy(2) syscall, and
> apps need to start supporting it.  After all, copy-on-write was planned
> for Reiser4, now 4.1*.  And don't anyone dare suggest "userland library".
>
> | It should be a simple format. Something simliar to tar. The worst thing
>
> Simple, yes.  Similar to tar, no.
>
> How about a serializer plugin?  foo.mp3/serialize (say) should return
> some single stream which contains all of foo.mp3.  It could work for
> directories as well, recursively, at least on the same filesystem --
> you'd back up by doing
>
> cp /serialize /backup/20040826
>
> This has two potential problems.  One is security -- I want to be able
> to serialize foo.mp3 without being root, but root doesn't want me to be
> able to serialize /etc and thus get access to shadow.
>
> The second problem is that in order to actually be used for backup, we
> need snapshots, which are not done yet*.  Currently, it'd have to lock
> the file/directory (recursively), which you don't want to do to entire
> live filesystems...  So, no massive backups of live systems yet, but
> definitely useful for sending around mp3s and such.
>
> | that can happen is people start writing plugins for every existing
> | compound format out there. It should be the other way around, agree on a
> | simple compound format and encourage applications to use this one if the
> | want to use this advantage.
>
> For the format, I vote for whatever format we're using at the storage
> layer.  It'd be a lot faster that way, both in execution time (I think)
> and in development time (I'm sure).  Plus, imagine the restore:
>
> cat /backup/20040826 > /dev/hda5
> resizefs_reiser4 /dev/hda5
>

one of the virtual metatags should be something about the method to 
serialize the data

if you are viewing a tarfile as a directory the VFS/Filesystem shoud know 
that the link between the whole thing and the contents is tar, and it 
should be possible for userspace programs to find this out by reading teh 
appropriate thing inside the virtual directory (and eventually change it 
as well, which would trigger regeneration of the base object, or at least 
noting that the current version is wrong and needs to be regenerated)

David Lang

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[David Lang]

On Thu, 26 Aug 2004, Christophe Saout wrote:

> Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel:
>
>>>> And if you read test.compound (the main stream) you get a special format
>>>> that contains all the components. You can copy that single stream of
>>>> bytes to another (reiser4) fs and then access test.compound/test.txt
>>>> again.
>>>
>>> (To Rik especially), this is the design which more or less satisfies
>>> lots of different goals at once.
>>
>> And if an unaware application reads the compound file
>> and then writes it out again, does the filesystem
>> interpret the contents and create the other streams ?
>>
>> Unless I overlook something (please tell me what), the
>> scheme just proposed requires filesystems to look at
>> the content of files that is being written out, in
>> order to make the streams work.
>
> Yes, the main compound stream unaware applications would see would just
> be a writable view of its contents. Probably not trivial to implement
> but doable.
>
> It should be a simple format. Something simliar to tar. The worst thing
> that can happen is people start writing plugins for every existing
> compound format out there. It should be the other way around, agree on a
> simple compound format and encourage applications to use this one if the
> want to use this advantage.
>

I also don't see why the VFS/Filesystem can't decide that (for example) 
this tar.gz is so active that instead of storing it as a tar.gz and 
providing a virtual directory of the contents that it instead stores the 
directory of the contents and makes the tar.gz virtual (regenerating it as 
needed or as extra system resources are available)

implementation wise I see headaches in doing this, but conceptually this 
is just an optimization that could take place in the future if we fine 
that it's needed.

David Lang

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Jamie Lokier]

David Lang wrote:
> I also don't see why the VFS/Filesystem can't decide that (for example) 
> this tar.gz is so active that instead of storing it as a tar.gz and 
> providing a virtual directory of the contents that it instead stores the 
> directory of the contents and makes the tar.gz virtual (regenerating it as 
> needed or as extra system resources are available)

Absolutely.  It could keep both views, if they're both actively being
used.  Or more than both, if there are more.  (You could think of
compression being an alternate view, and both compressed and
uncompressed may as well remain on disk if there's space and it's
being actively accessed.

> implementation wise I see headaches in doing this, but conceptually this 
> is just an optimization that could take place in the future if we fine 
> that it's needed.

untarring a big source tree takes ages.  I wouldn't like to do it
after every reboot, if there was a .tar.bz2 tree I looked at often.
That's why I have things like glibc untarred in my home directory.

With good cacheing, I could cd into the .tar.bz2 files and
_effectively_ have the performance of untarred source trees for the
ones I look at often on my disk -- automatically cleaned if the space
if needed for something else, too.  It would be quite nice.

-- Jamie

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
> 
> With good cacheing, I could cd into the .tar.bz2 files and
> _effectively_ have the performance of untarred source trees for the
> ones I look at often on my disk -- automatically cleaned if the space
> if needed for something else, too.  It would be quite nice.

What you're talking about is just a caching filesystem. Then you can have 
any deamon do whatever it wants to fill it - whether from tar-files or 
over the network or anything else.

I think you should ask David Howells about it..

		Linus

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 23:05, David Lang wrote:

> I also don't see why the VFS/Filesystem can't decide that (for example)
> this tar.gz is so active that instead of storing it as a tar.gz and
> providing a virtual directory of the contents that it instead stores the
> directory of the contents and makes the tar.gz virtual (regenerating it as
> needed or as extra system resources are available)

Because that would mean the kernel should "talk" the tar format, which is, 
IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to 
perform this task, instead.

Re: silent semantic changes with reiser4

[Chris Dukes]

On Fri, Aug 27, 2004 at 12:30:19AM +0200, Felipe Alfaro Solana wrote:
> On Thursday 26 August 2004 23:05, David Lang wrote:
> 
> > I also don't see why the VFS/Filesystem can't decide that (for example)
> > this tar.gz is so active that instead of storing it as a tar.gz and
> > providing a virtual directory of the contents that it instead stores the
> > directory of the contents and makes the tar.gz virtual (regenerating it as
> > needed or as extra system resources are available)
> 
> Because that would mean the kernel should "talk" the tar format, which is, 
> IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to 
> perform this task, instead.

Wasn't HURD supposed to do this for us?
-- 
Chris Dukes
Warning: Do not use the reflow toaster oven to prepare foods after
it has been used for solder paste reflow. 
http://www.stencilsunlimited.com/stencil_article_page5.htm

Re: silent semantic changes with reiser4

[David Lang]

On Fri, 27 Aug 2004, Felipe Alfaro Solana wrote:

> On Thursday 26 August 2004 23:05, David Lang wrote:
>
>> I also don't see why the VFS/Filesystem can't decide that (for example)
>> this tar.gz is so active that instead of storing it as a tar.gz and
>> providing a virtual directory of the contents that it instead stores the
>> directory of the contents and makes the tar.gz virtual (regenerating it as
>> needed or as extra system resources are available)
>
> Because that would mean the kernel should "talk" the tar format, which is,
> IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to
> perform this task, instead.
>

the kernel will definantly need the ability to use user-space code to do 
the transformations from one version to the other (if nothing else think 
of the thumbnail version of images, we don't want the image manipulation 
code in the kernel and we definantly want this sort of option available)

the interesting issue is going to be defining the kernel->user-space 
interface for doing the extractions.

David Lang

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> And if an unaware application reads the compound file
> and then writes it out again, does the filesystem
> interpret the contents and create the other streams ?

Yes, exactly that.  The streams are created on demand of course, and
by userspace helpers when that's appropriate which I suspect it almost
always is.

> Unless I overlook something (please tell me what), the
> scheme just proposed requires filesystems to look at
> the content of files that is being written out, in
> order to make the streams work.

Yes.  Hence the idea of coherent views between two files: writing to
one affects the content of the other, although the calcalation is only
done on demand (or when the fs wants to migrate the representation --
for example, creating the flat container prior to deleting the
regeneratable pieces in order to save space).

I haven't seen anything from Namesys that says they'll do that.  I
have the impression the streams are just generated in memory on the
fly, not stored on disk with a cacheing policy, but that's just an
impression.  (We've all seen the Namesys white papers, they're not
_that_ revealing). :)

I'm just pointing out how to do it _right_.  I think it will turn out
like this eventually, either next year or some time over the next
decade after some iterations.  Inevitable.  Mark my words, etc. :)

-- Jamie

Re: silent semantic changes with reiser4

[John Stoffel]

Jamie> Rik van Riel wrote:
>> And if an unaware application reads the compound file
>> and then writes it out again, does the filesystem
>> interpret the contents and create the other streams ?

Jamie> Yes, exactly that.  The streams are created on demand of
Jamie> course, and by userspace helpers when that's appropriate which
Jamie> I suspect it almost always is.

So how would a program that converts between a JPEG file (with exif
data) and a PNG work, such as ImageMagick?  Are we proposing to teach
the VFS (or worse yet each filesystem) how to do this?  

I've been following this discussion a bit and I'm not sure that I've
actually seen any concrete examples of where this is a *good* thing to
have.  People talk about only having to modify 20 bytes at a time
instead of reading and writing 1mb of data.  Isn't that what mmap()
does?

Now I can sorta understand the idea that having a directory look like
a file is neat, and certainly simplifies some aspects, but I think
that going all the way down to the logical conclusion here is a bit
silly.

To use the principle of blowing things up to make them very large or
very small, what happens if I decide that the best idea is to make all
files just be directories which contain single byte files?  Isn't that
the logical extreme here?  So my 1mb JPEG file is not just some image
data and header info in multiple files, but it's really just 1
million (ok 1024 * 1024) individual files that the VFS knows how to
put together.  Seems like the logical extreme.  Oh wait, maybe we
should be exposing a single file per bit instead! 

>From my point of view, there lies madness.  As Rik pointed out, how do
backup and restore tools work with this stuff?  Most people could care
less about how their data is organized, but they certainly care when
they can't restore it from backups.  

I'd really like to see a concrete example from Hans or other
proponents about why this makes things easier/faster/better to do.
Mostly, I've just seen "proof by vigorous handwaving" that it's a good
thing.

In alot of ways, I think people are going in the wrong direction, you
want to excapsulate and hide the details more, not expose them.
That's what a good API does, it hides the details while giving you a
rich set of semantics to manage your data.  

God knows I'm not smart enough or driven enough to actually come up
with my own ideas, but I certainly haven't seen anyone else (even
Linus) come up with an earth shattering arguement to say why this is
the right move to make.

As Linus says, most of the OS's job is to mediate access to
objects/data.  Why do we want to expose such low level data then?

John

   John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
	 stoffel@lucent.com - http://www.lucent.com - 978-952-7548

Re: silent semantic changes with reiser4

[John Stoffel]

Jamie> Rik van Riel wrote:
>> And if an unaware application reads the compound file
>> and then writes it out again, does the filesystem
>> interpret the contents and create the other streams ?

Jamie> Yes, exactly that.  The streams are created on demand of
Jamie> course, and by userspace helpers when that's appropriate which
Jamie> I suspect it almost always is.

So how would a program that converts between a JPEG file (with exif
data) and a PNG work, such as ImageMagick?  Are we proposing to teach
the VFS (or worse yet each filesystem) how to do this?  

I've been following this discussion a bit and I'm not sure that I've
actually seen any concrete examples of where this is a *good* thing to
have.  People talk about only having to modify 20 bytes at a time
instead of reading and writing 1mb of data.  Isn't that what mmap()
does?

Now I can sorta understand the idea that having a directory look like
a file is neat, and certainly simplifies some aspects, but I think
that going all the way down to the logical conclusion here is a bit
silly.

To use the principle of blowing things up to make them very large or
very small, what happens if I decide that the best idea is to make all
files just be directories which contain single byte files?  Isn't that
the logical extreme here?  So my 1mb JPEG file is not just some image
data and header info in multiple files, but it's really just 1
million (ok 1024 * 1024) individual files that the VFS knows how to
put together.  Seems like the logical extreme.  Oh wait, maybe we
should be exposing a single file per bit instead! 

From my point of view, there lies madness.  As Rik pointed out, how do
backup and restore tools work with this stuff?  Most people could care
less about how their data is organized, but they certainly care when
they can't restore it from backups.  

I'd really like to see a concrete example from Hans or other
proponents about why this makes things easier/faster/better to do.
Mostly, I've just seen "proof by vigorous handwaving" that it's a good
thing.

In alot of ways, I think people are going in the wrong direction, you
want to excapsulate and hide the details more, not expose them.
That's what a good API does, it hides the details while giving you a
rich set of semantics to manage your data.  

God knows I'm not smart enough or driven enough to actually come up
with my own ideas, but I certainly haven't seen anyone else (even
Linus) come up with an earth shattering arguement to say why this is
the right move to make.

As Linus says, most of the OS's job is to mediate access to
objects/data.  Why do we want to expose such low level data then?

John

   John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
	 stoffel@lucent.com - http://www.lucent.com - 978-952-7548

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 22:32, John Stoffel wrote:
> Jamie> Rik van Riel wrote:
> >> And if an unaware application reads the compound file
> >> and then writes it out again, does the filesystem
> >> interpret the contents and create the other streams ?
>
> Jamie> Yes, exactly that.  The streams are created on demand of
> Jamie> course, and by userspace helpers when that's appropriate which
> Jamie> I suspect it almost always is.
>
> So how would a program that converts between a JPEG file (with exif
> data) and a PNG work, such as ImageMagick?  Are we proposing to teach
> the VFS (or worse yet each filesystem) how to do this?
>
> I've been following this discussion a bit and I'm not sure that I've
> actually seen any concrete examples of where this is a *good* thing to
> have.  People talk about only having to modify 20 bytes at a time
> instead of reading and writing 1mb of data.  Isn't that what mmap()
> does?
>
> Now I can sorta understand the idea that having a directory look like
> a file is neat, and certainly simplifies some aspects, but I think
> that going all the way down to the logical conclusion here is a bit
> silly.
>
> To use the principle of blowing things up to make them very large or
> very small, what happens if I decide that the best idea is to make all
> files just be directories which contain single byte files?  Isn't that
> the logical extreme here?  So my 1mb JPEG file is not just some image
> data and header info in multiple files, but it's really just 1
> million (ok 1024 * 1024) individual files that the VFS knows how to
> put together.  Seems like the logical extreme.  Oh wait, maybe we
> should be exposing a single file per bit instead!

It is doable. It doesn't mean we shall do this.

I think some reiser4 supporters try to do too much too soon.
It is more sensible to do it in small steps.

I think we can start small and make all file metadata to be accessible
via file/meta/{uid,gid,mode,mtime,atime,...}.

Normal tools will be unable to see file/meta/ because file is not
a directory and file/ is not a directory (i.e. open(O_DIRECTORY)
will fail on them). Hardlinking to file/meta/* is not allowed.

However, file/. _is_ a directory. We can make a tar-like
tool which can do its work and save/restore metadata along with data
by just tarring/untarring file/meta/*. Notice how this 'new tar'
does not need to know about exact kind of metadata supported by fs.
You can add ACLs to the fs, and this 'new tar' will be able to save/restore
ACLs. Without any modufications.

Contrast this with existing tar which has hardcoded knowlendge about
uid,gid,mode,etc...
--
vda

Re: silent semantic changes with reiser4

[Jamie Lokier]

John Stoffel wrote:
> Jamie> Yes, exactly that.  The streams are created on demand of
> Jamie> course, and by userspace helpers when that's appropriate which
> Jamie> I suspect it almost always is.
> 
> So how would a program that converts between a JPEG file (with exif
> data) and a PNG work, such as ImageMagick?  Are we proposing to teach
> the VFS (or worse yet each filesystem) how to do this?  

No.  (1) That's what "userspace helpers" are for.  (2) What does image
format conversion have to do with viewing files in their component
parts?  (3) I suspect someone will write a plugin that does indeed
convert virtually every known image format to a common format
(probably PNG and something "raw") at some point.  Why not?  It's just
a small script (it would run ImageMagick!).

> I've been following this discussion a bit and I'm not sure that I've
> actually seen any concrete examples of where this is a *good* thing to
> have.  People talk about only having to modify 20 bytes at a time
> instead of reading and writing 1mb of data.  Isn't that what mmap()
> does?

Sure, if you think mmap() is an easy substitute for "parse the author
& license tags from this [unknown format] audio, video or font file".

> Now I can sorta understand the idea that having a directory look like
> a file is neat, and certainly simplifies some aspects, but I think
> that going all the way down to the logical conclusion here is a bit
> silly.
> 
> To use the principle of blowing things up to make them very large or
> very small, what happens if I decide that the best idea is to make all
> files just be directories which contain single byte files?  Isn't that
> the logical extreme here?  So my 1mb JPEG file is not just some image
> data and header info in multiple files, but it's really just 1
> million (ok 1024 * 1024) individual files that the VFS knows how to
> put together.  Seems like the logical extreme.  Oh wait, maybe we
> should be exposing a single file per bit instead! 

If you decide to do that, you are welcome to write the userspace
helper which creates that view in your directory.

Why not?  It's very silly, but it's no sillier then writing a program
to read your 1mb JPEG and write it out one file per bit, which you're
welcome to write right now.

> >From my point of view, there lies madness.  As Rik pointed out, how do
> backup and restore tools work with this stuff?  Most people could care
> less about how their data is organized, but they certainly care when
> they can't restore it from backups.  

Generated views are something which should _not_ be backed up and
restored.  That's not what they are for.

Auxiliary metadata, such as author info and signatures which cannot be
stored in the main file for some reason -- that should be backed up.

Permissions, that should be backed up.

But not views which are computed from the main file.  You don't need
to back them up, and they don't need to take any real space.  They're
virtual, just like an infinitely deep directory on a web server can be
virtual.  You can make those very silly too, if you want to.

(Some help from the filesystem with storing temporarily cached values
is fine, for performance, but we shouldn't pretend that generated
views are anything other than virtual).

> I'd really like to see a concrete example from Hans or other
> proponents about why this makes things easier/faster/better to do.
> Mostly, I've just seen "proof by vigorous handwaving" that it's a good
> thing.
> 
> In alot of ways, I think people are going in the wrong direction, you
> want to excapsulate and hide the details more, not expose them.
> That's what a good API does, it hides the details while giving you a
> rich set of semantics to manage your data.  

Would you like to propose an alternate API?  I was under the
impression that we were discussing one, right here in this thread.
It's called POSIX with an extension where you can cd into a file.  It
has a rich set of semantics to manage your data, and works with a lot
of familiar programs.

A better one would be welcome, if you have an idea.

> God knows I'm not smart enough or driven enough to actually come up
> with my own ideas, but I certainly haven't seen anyone else (even
> Linus) come up with an earth shattering arguement to say why this is
> the right move to make.

Come up with a better one otherwise it's right ;)

> As Linus says, most of the OS's job is to mediate access to
> objects/data.  Why do we want to expose such low level data then?

Eh?  In this case the OS is mediating by providing a uniform interface
between programs that access the data and programs that handle file
formats.

What is this "low level" data you speak of?  I would say something
like the "designer" of a font, or the "from address" and "subject" of
an email are very high level abstractions, and that's among the sorts
of things which are to be exposed by these interfaces.

-- Jamie

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote:
> Rik van Riel wrote:
> > And if an unaware application reads the compound file
> > and then writes it out again, does the filesystem
> > interpret the contents and create the other streams ?
> 
> Yes, exactly that.  The streams are created on demand of course, and
> by userspace helpers when that's appropriate which I suspect it almost
> always is.
> 
> > Unless I overlook something (please tell me what), the
> > scheme just proposed requires filesystems to look at
> > the content of files that is being written out, in
> > order to make the streams work.
> 
> Yes.  Hence the idea of coherent views between two files: writing to
> one affects the content of the other, although the calcalation is only
> done on demand (or when the fs wants to migrate the representation --
> for example, creating the flat container prior to deleting the
> regeneratable pieces in order to save space).
> 
> I haven't seen anything from Namesys that says they'll do that.  I
> have the impression the streams are just generated in memory on the
> fly, not stored on disk with a cacheing policy, but that's just an
> impression.  (We've all seen the Namesys white papers, they're not
> _that_ revealing). :)

would it be a tautology if I say that the streams implementation depends on
(reiser4 now, may be vfs later) plugins? All proposals including user-level
helpers, shadow caching/indexing may be implemented, why not?

> I'm just pointing out how to do it _right_.  I think it will turn out
> like this eventually, either next year or some time over the next
> decade after some iterations.  Inevitable.  Mark my words, etc. :)
> 
> -- Jamie

-- 
Alex.

Re: silent semantic changes with reiser4

[Hans Reiser]

Alex Zarochentsev wrote:

>On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote:
>  
>
>>Rik van Riel wrote:
>>    
>>
>>>And if an unaware application reads the compound file
>>>and then writes it out again, does the filesystem
>>>interpret the contents and create the other streams ?
>>>      
>>>
>>Yes, exactly that.  The streams are created on demand of course, and
>>by userspace helpers when that's appropriate which I suspect it almost
>>always is.
>>
>>    
>>
>>>Unless I overlook something (please tell me what), the
>>>scheme just proposed requires filesystems to look at
>>>the content of files that is being written out, in
>>>order to make the streams work.
>>>      
>>>
>>Yes.  Hence the idea of coherent views between two files: writing to
>>one affects the content of the other, although the calcalation is only
>>done on demand (or when the fs wants to migrate the representation --
>>for example, creating the flat container prior to deleting the
>>regeneratable pieces in order to save space).
>>
>>I haven't seen anything from Namesys that says they'll do that.
>>
We do that, err, it is our design to do that. Hmm, the plugins to do 
that seem to all be in the next to implement list and not on the 
implemented already list.....

However, with respect to streams, those are just files in a 
file-directory object in our scheme. If you want them unified, well, 
then you cat compound-document/pseudos/glued, or you link 
compound-document/pseudos/glued to compound-document/pseudos/body, where 
body is our term for the default stream, and then you just cat 
compound-document

If you have the impression I am being vaporwarish, well, you are 
right.... glued is not yet on the already implemented list.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Alex Zarochentsev wrote:

>On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote:
>  
>
>>Rik van Riel wrote:
>>    
>>
>>>And if an unaware application reads the compound file
>>>and then writes it out again, does the filesystem
>>>interpret the contents and create the other streams ?
>>>      
>>>
>>Yes, exactly that.  The streams are created on demand of course, and
>>by userspace helpers when that's appropriate which I suspect it almost
>>always is.
>>
>>    
>>
>>>Unless I overlook something (please tell me what), the
>>>scheme just proposed requires filesystems to look at
>>>the content of files that is being written out, in
>>>order to make the streams work.
>>>      
>>>
>>Yes.  Hence the idea of coherent views between two files: writing to
>>one affects the content of the other, although the calcalation is only
>>done on demand (or when the fs wants to migrate the representation --
>>for example, creating the flat container prior to deleting the
>>regeneratable pieces in order to save space).
>>
>>I haven't seen anything from Namesys that says they'll do that.
>>
We do that, err, it is our design to do that. Hmm, the plugins to do 
that seem to all be in the next to implement list and not on the 
implemented already list.....

However, with respect to streams, those are just files in a 
file-directory object in our scheme. If you want them unified, well, 
then you cat compound-document/pseudos/glued, or you link 
compound-document/pseudos/glued to compound-document/pseudos/body, where 
body is our term for the default stream, and then you just cat 
compound-document

If you have the impression I am being vaporwarish, well, you are 
right.... glued is not yet on the already implemented list.

Hans

Re: silent semantic changes with reiser4

[Diego Calleja]

El Thu, 26 Aug 2004 11:54:51 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió:

> And if an unaware application reads the compound file
> and then writes it out again, does the filesystem
> interpret the contents and create the other streams ?

If the old utils doesn't support streams there's no way to make them create
streams. There's not a real solution for that except fixing the old tools.

Why instead of being "compound" you just make test.compound to
behave as a real directory? ie:

# echo "foo" > streamfile
# echo "fake metadata" > streamfile/somemetadata
# cat streamfile
foo
# cat streamfile/somemetadata
fake metadata

Now the old backup utils could see "streamfile" as a directory, ie:
old apps could see that "streamfile" is a directory and then make
their backups with the following contents:
streamfile/somemetadata
streamfile/defaultstream

The defaultstream would be what its name means. There could be more
default streams like "streamfile/compoundstream"

All this looks like reinventing the file/directory concept wheel. Instead of
adding support for streams and "use files as directories", why not orientate
it to "use directories as files? Streams could very well be provided
by directories containing files, with the addon of being able to cat
a directory (which would cat the output of a designed file inside
every directory like "defaultstream", then there could be other
"special files" like "compoundstream", although the idea of "fixed name-
special files" doesn't sounds very cool) All this could be very well
provided done userspace, but then couldn't userspace also supply the whole VFS 
functionality and still it lives in the kernel...?

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Diego Calleja wrote:

> All this looks like reinventing the file/directory concept wheel.
> Instead of adding support for streams and "use files as directories",
> why not orientate it to "use directories as files? Streams could very
> well be provided by directories containing files,

So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

This is getting surreal.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Diego Calleja wrote:

> All this looks like reinventing the file/directory concept wheel.
> Instead of adding support for streams and "use files as directories",
> why not orientate it to "use directories as files? Streams could very
> well be provided by directories containing files,

So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

This is getting surreal.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
>
> On Thu, 26 Aug 2004, Diego Calleja wrote:
> 
> > All this looks like reinventing the file/directory concept wheel.
> > Instead of adding support for streams and "use files as directories",
> > why not orientate it to "use directories as files? Streams could very
> > well be provided by directories containing files,
> 
> So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

No no. The stream you get from a directory is totally _independent_ of the 
contents of the directory. Anything else would be pointless.

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> No no. The stream you get from a directory is totally _independent_ of the 
> contents of the directory. Anything else would be pointless.

Surely it depends on the directory?

The stream from a "classic directory", i.e. something made with
mkdir(), mode drwxr-xr-x filled with files, might well be a reproducible
representation of its contents.

-- Jamie

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
> Linus Torvalds wrote:
> > 
> > No no. The stream you get from a directory is totally _independent_ of the 
> > contents of the directory. Anything else would be pointless.
> 
> Surely it depends on the directory?

Well, you _could_ make it do a "tar", but what's the point? That really 
doesn't add any value as far as user space is concerned.

Think of it this way: if the directory and the "default stream" associated 
with it aren't independent, then user space already has the information, 
and the stream is pointless. In fact, the stream is _worse_ than 
pointless, because it hides the fact that there is no independent 
information.

See my point? 

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> Well, you _could_ make it do a "tar", but what's the point? That really 
> doesn't add any value as far as user space is concerned.
> 
> Think of it this way: if the directory and the "default stream" associated 
> with it aren't independent, then user space already has the information, 
> and the stream is pointless. In fact, the stream is _worse_ than 
> pointless, because it hides the fact that there is no independent 
> information.
> 
> See my point? 

I see your point, but I agree 50%.

(1) One of the uses of "streams" is to have extra information which
    accompanies a file, and is separate from the flat file's information.

    For example, metadata like "this C source file is in UTF-8",
    "the author is Jamie (nowhere to put it in the data file)" and
    "Jamie's signature".

(2) Filesystem settings like "compress this", and perhaps "permissions".

(3) Filesystem information like "this file has been compressed by 59%,
    uses 5 1k blocks".

(4) Another use, which is more akin to containers, is to _present_ the
    information of the flat file in a different way.

    For example, the id tags and comments of any media file, the component
    parts of any media file, license (of course :), and expanded or
    decoded views such as inside an archive file, filesystem image, or
    other structured storage file (.doc :).

    Those alternate views are useful just because they can be accessed
    using ordinary unix tools, no matter what the obscure codec format
    provided it's supported.  With a bit of effort a standard
    presentation for certain kinds of data may arise.

(5) There is also data which is _calculated_ from the flat file or its
    extra metadata, and which the kernel guarantees is either
    consistent with the file, not set at all, or at least is
    invalidated if the file is changed.  It may be calculated on demand
    through the kernel interface, or applications may simply have to
    calculate it themselves when they don't have an up to date value.

    For example MD5 or SHA1 digests of the flat file, or of its parts.
    The key thing is that these are always up to date, not dependent
    on weak tests like mtime.

    Partial checksums for speeding up rsync and other delta algorithms.
    Computed thumbnails, MP3 average power levels, character encoding
    extracted from parsing old HTML META tags.  Anything really that
    can be calculated and usefully cached.

    This is similar to (4), but I'm emphasising things that
    may take a while to compute, may be cached with the filesystem
    itself helping to decide when to evict, and writing them won't
    affect the data they're calculated from, if writing is allowed at all.

    The only really important of these are MD5 or SHA1 digests, or
    some other way of tracking modifications like a strong serial number.

    All other computed cached data can be maintained by applications
    elsewhere, if they know when the data has been modified since the
    last time.  (Though there are advantages to making the data
    disappear if the file is deleted, or move if it's renamed).


The thing with these different categories is that we want different
behaviour when serialising the data for a backup or to copy it.

We probably always (1) to be copied.

(2) is questionable: it's like unix file permissions, you might want
to copy it; you might not.

(3) Shouldn't be copied or stored except for special reasons.  It's a
bit like /proc: it's not meant to be stored on your backup CD and it's
definitely not meant to be written during a restore. :)

(4) Should never be copied if you're copying the flat file, because
it's exactly the same information, just accessed differently.  You
might want to copy it by itself for some reason though, and you
certainly would like to work with it from scripts and applications.
These streams are special in another way: they might themselves
decompose into further streams.  Just think of a AVI file containing
an OGG file containing a FLAC containing metadata and data streams, or
something like that.  They happen.

(5) shouldn't be copied if you're transmitting a file, as a matter of
data integrity among other things.  You might want it on the backup,
then again maybe only some of the values.  Probably never the crypto digests.


To summarise: most data inside a directory-as-file should _not_
usually be copied and transported.  A little of it, (1), should be
copied if the serialised form will have it, and some of the other
metadata should be copied under special circumstances.

I've suggested that we could simply not have (1), and recommend that
only reproducible, non-essential data appears inside the directory.
Everything essential would be part of the serialised, flat file.

However, you seem to like (1), metadata which should be serialised and
copied around, and which is not part of the flat file.

Note that (1) will only be copied around on systems which handle
"extra forks" anyway -- things like Samba and MacOS compatibility.
FTP and HTTP obviously won't copy the metadata of (1), although HTTP
WebDAV might allow it to be visible as properties.

All that suggests a common structure for the directory inside a
"container" -- one where the metadata to be copied by default is
clearly distinguished from the metadata which should never by copied,
or which is cached, or which is a view.  Possibly it should be
distinguished by name, although permission bits or (not sure I like
this) meta-metadata could serve the same role without being name
dependent.

-- Jamie

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> No no. The stream you get from a directory is totally _independent_ of the 
> contents of the directory. Anything else would be pointless.

It's a relief to know that nobody's taking my humorous
suggestion seriously, but now we still have the "standard
Unix tools can't manipulate files" problem...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> No no. The stream you get from a directory is totally _independent_ of the 
> contents of the directory. Anything else would be pointless.

It's a relief to know that nobody's taking my humorous
suggestion seriously, but now we still have the "standard
Unix tools can't manipulate files" problem...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
> 
> It's a relief to know that nobody's taking my humorous
> suggestion seriously, but now we still have the "standard
> Unix tools can't manipulate files" problem...

I disagree. They can manipulate the files a whole lot better than they can 
manipulate xattr's.

For example, you _could_ probably (but hey, maybe "tar" tries to strip
slashes off the end of filenames, so this might not work due to silly
reasons like that) back up a compound file with

	tar cvf file.tar file file/

although unpacking it would require that tar be taught about the thing. 
And you definitely could write a script to do the thing, ie even with an 
unmodified tar you could do

	tar cvf file-archive.tar file
	cd file
	tar cvf ../attribute-archive.tar .

which is a hell of a lot better than what you can do with the fsattr 
interfaces and unmodified legacy applications.

So one of the advantages of "dir-as-file/file-as-dir" is exactly that you
_can_ manipulate the data with legacy tools. Sure, things that traverse a
directory tree might need some (likely fairly trivial) modifications if
they really want to take advantage of the subfiles, but that's still
likely to be _much_ less of an issue than with fsattr's that have a 
totally different model entirely.

		Linus

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> For example, you _could_ probably (but hey, maybe "tar" tries to strip
> slashes off the end of filenames, so this might not work due to silly
> reasons like that) back up a compound file with
> 
> 	tar cvf file.tar file file/

So you'd have both a file and a directory that just happen
to have the same name ?  How would this work in the dcache?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> For example, you _could_ probably (but hey, maybe "tar" tries to strip
> slashes off the end of filenames, so this might not work due to silly
> reasons like that) back up a compound file with
> 
> 	tar cvf file.tar file file/

So you'd have both a file and a directory that just happen
to have the same name ?  How would this work in the dcache?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
> 
> So you'd have both a file and a directory that just happen
> to have the same name ?  How would this work in the dcache?

There would be only one entry in the dcache. The lookup will select 
whether it opens the file or the directory based on O_DIRECTORY (and 
usage, of course - if it's in the middle of a path, it obviously needs to 
be opened as a directory regardless).

That's not the problem. The problem from a dcache standpoint ends up being 
when the file has a link, and you have two paths to the same sub-file 
through two different ways:

	.. create file 'x' with named stream 'y' ...
	ln x z
	ls -l x/y z/y	/* it's the same attribute!! */

but this is actually exactly the same thing that we already have with 
mounts, ie it is equivalent (from a dentry standpoint) to

	.. create directory 'x' with file 'y' ..
	mkdir z
	mount --bind x z
	ls -l x/y z/y	/* It's the same file!! */

so none of this is really anything "new" from a dcache standpoint.

Except for all the details, of course ;)

		Linus

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:
> On Thu, 26 Aug 2004, Rik van Riel wrote:
> > 
> > So you'd have both a file and a directory that just happen
> > to have the same name ?  How would this work in the dcache?
> 
> There would be only one entry in the dcache. The lookup will select 
> whether it opens the file or the directory based on O_DIRECTORY (and 
> usage, of course - if it's in the middle of a path, it obviously needs to 
> be opened as a directory regardless).

Hmmm, I just straced  "cp /bin/bash /tmp".
One line stood out as a potential problem:

open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4

What do we do with O_CREAT ?

Do we always allow both a directory and a file to be created with
the same name ?

Does this create a new class of "symlink attack" style security
holes ?


-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:
> On Thu, 26 Aug 2004, Rik van Riel wrote:
> > 
> > So you'd have both a file and a directory that just happen
> > to have the same name ?  How would this work in the dcache?
> 
> There would be only one entry in the dcache. The lookup will select 
> whether it opens the file or the directory based on O_DIRECTORY (and 
> usage, of course - if it's in the middle of a path, it obviously needs to 
> be opened as a directory regardless).

Hmmm, I just straced  "cp /bin/bash /tmp".
One line stood out as a potential problem:

open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4

What do we do with O_CREAT ?

Do we always allow both a directory and a file to be created with
the same name ?

Does this create a new class of "symlink attack" style security
holes ?


-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 673 bytes --]

Am Donnerstag, den 26.08.2004, 14:59 -0400 schrieb Rik van Riel:

> open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4
> 
> What do we do with O_CREAT ?
> 
> Do we always allow both a directory and a file to be created with
> the same name ?

I would say that the directory under a file is implicit. There are still
the pseudo-files a filesystem might want to provide or the VFS if there
should be an agreement to generally provide file/pseudo/uid or possibly
some security hooks. Some filesystem might want to provide those pseudo
files but doesn't have the technical possibility to store non-pseudeo
files from the user under such a directory.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 673 bytes --]

Am Donnerstag, den 26.08.2004, 14:59 -0400 schrieb Rik van Riel:

> open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4
> 
> What do we do with O_CREAT ?
> 
> Do we always allow both a directory and a file to be created with
> the same name ?

I would say that the directory under a file is implicit. There are still
the pseudo-files a filesystem might want to provide or the VFS if there
should be an agreement to generally provide file/pseudo/uid or possibly
some security hooks. Some filesystem might want to provide those pseudo
files but doesn't have the technical possibility to store non-pseudeo
files from the user under such a directory.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
> 
> Hmmm, I just straced  "cp /bin/bash /tmp".
> One line stood out as a potential problem:
> 
> open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4
> 
> What do we do with O_CREAT ?
> 
> Do we always allow both a directory and a file to be created with
> the same name ?

Either I am confused, or you are.

To me, a filesystem that allows this thing doesn't really _have_ the 
concept of "directory vs file". It's just a "filesystem object", and it 
can act as _both_ a directory and a file.

So when you create "/tmp/bash" - assuming /tmp supports the file-as-dir 
semantics at all, you're just creating a new inode - the same you always 
have. When you write to that inode, it writes to the default stream. 
There's no special cases here.

Now, after you have your regular /tmp/bash, you can then start adding 
named streams to it, ie you can do

	open("/tmp/bash/icon", O_WRONLY|O_CREAT|O_LARGEFILE, 0755);

and that will create the "icon" named stream. See? 

So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
contains both a standard data stream (the "file" part) _and_ pointers to
other named streams (the "directory" part).

Hey, think of it as a wave-particle duality. Both "modes" exist at the
same time, and cannot be separated from each other. Which one you see
depends entirely on your "experiment", ie how you open the file.

> Does this create a new class of "symlink attack" style security
> holes ?

I don't believe so. 

		Linus

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> contains both a standard data stream (the "file" part) _and_ pointers to
> other named streams (the "directory" part).

OK, that makes sense.  

> Hey, think of it as a wave-particle duality. Both "modes" exist at the
> same time, and cannot be separated from each other. Which one you see
> depends entirely on your "experiment", ie how you open the file.

Guess I'm scared again now.  We need to make sure that
backup programs don't fall victim to the uncertainty
principle ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> contains both a standard data stream (the "file" part) _and_ pointers to
> other named streams (the "directory" part).

OK, that makes sense.  

> Hey, think of it as a wave-particle duality. Both "modes" exist at the
> same time, and cannot be separated from each other. Which one you see
> depends entirely on your "experiment", ie how you open the file.

Guess I'm scared again now.  We need to make sure that
backup programs don't fall victim to the uncertainty
principle ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

Am Donnerstag, den 26.08.2004, 15:44 -0400 schrieb Rik van Riel:

> > Hey, think of it as a wave-particle duality. Both "modes" exist at the
> > same time, and cannot be separated from each other. Which one you see
> > depends entirely on your "experiment", ie how you open the file.
> 
> Guess I'm scared again now.  We need to make sure that
> backup programs don't fall victim to the uncertainty
> principle ;)

At least you can be sure that the backup will go into a defined state as
soon as you observe it.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> contains both a standard data stream (the "file" part) _and_ pointers to
> other named streams (the "directory" part).

Thinking about it some more, how would file managers and
file chosers handle this situation ?

Currently the user browses the directory tree and when
the user clicks on something, one of the following 
happens:

1) if it is a directory, the file manager/choser changes
   into that directory

2) if it is a file, the file is opened

Now how do we present things to users ?

How will users know when an object can only be chdired
into, or only be opened ?

For objects that do both, how does the user choose ?

Do we really want to have a file paradigm that's different
from the other OSes out there ?

What happens when users want to transfer data from Linux
to another system ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> contains both a standard data stream (the "file" part) _and_ pointers to
> other named streams (the "directory" part).

Thinking about it some more, how would file managers and
file chosers handle this situation ?

Currently the user browses the directory tree and when
the user clicks on something, one of the following 
happens:

1) if it is a directory, the file manager/choser changes
   into that directory

2) if it is a file, the file is opened

Now how do we present things to users ?

How will users know when an object can only be chdired
into, or only be opened ?

For objects that do both, how does the user choose ?

Do we really want to have a file paradigm that's different
from the other OSes out there ?

What happens when users want to transfer data from Linux
to another system ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

Am Donnerstag, den 26.08.2004, 16:10 -0400 schrieb Rik van Riel:

> > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> > contains both a standard data stream (the "file" part) _and_ pointers to
> > other named streams (the "directory" part).
> 
> Thinking about it some more, how would file managers and
> file chosers handle this situation ?

I would say they don't. That's where metadata and pseudo-files sit. It's
up to the applications to do something useful with the data.

> Do we really want to have a file paradigm that's different
> from the other OSes out there ?

Some other OSes have similar things.

> What happens when users want to transfer data from Linux
> to another system ?

No one should store really important data inside file/ that makes the
file completely useless. I just love these postscript fonts on the Mac.
The actual main file stream is 0 bytes in size. The actual font is
inside the resource fork. When I want to convert it the Mac user needs
to pack it into a compound file using a special tool and then I can
decode it using another tool to access the PFB file. Crap.

It's useful for metadata like keywords, ACLs and these things. If you
can pack it using an aware backup program, fine. If you don't, you still
have the data.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> I just love these postscript fonts on the Mac.  The actual main file
> stream is 0 bytes in size. The actual font is inside the resource
> fork. When I want to convert it the Mac user needs to pack it into a
> compound file using a special tool and then I can decode it using
> another tool to access the PFB file. Crap.

I thought it was a cunning way to stop me from copying those nice
fonts onto my Linux box.  The original DRM.  Thanks for the tip :)

-- Jamie

Re: silent semantic changes with reiser4

[Stephen Wille Padnos]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Linus Torvalds wrote:
>  
>
>>So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
>>contains both a standard data stream (the "file" part) _and_ pointers to
>>other named streams (the "directory" part).
>>    
>>
>Thinking about it some more, how would file managers and
>file chosers handle this situation ?
>
>Currently the user browses the directory tree and when
>the user clicks on something, one of the following 
>happens:
>
>1) if it is a directory, the file manager/choser changes
>   into that directory
>  
>
How does the file manager / chooser decide whether you're trying to move 
into a directory, or the meta-data-directory for a directory?
It's not just files that should have metadata - directories need* them 
too.  Making it possible to see attributes as a directory under a file 
is great, but you'd still need an O_META flag for accessing directory 
metadata (since there are already files under a directory).

>2) if it is a file, the file is opened
>
>Now how do we present things to users ?
>
>How will users know when an object can only be chdired
>into, or only be opened ?
>
>For objects that do both, how does the user choose ?
>
>Do we really want to have a file paradigm that's different
>from the other OSes out there ?
>  
>
MacOS does, Be did (sort of).  I'm not sure it would be the end of the 
world, as long as the data can be extracted.

>What happens when users want to transfer data from Linux
>to another system ?
>  
>
That would depend on the other system.  Data is easy, metadata is hard.
It would be possible to create an XML schema for metadata, and if 
requested (O_EVERYTHING?), the file data is returned with all metadata 
in XML tags.  (not advocating this, just an idea :)
- Steve

* I say need in the same way as one *needs* to upgrade their 2GHz 
computer - it would be nice :)

Re: silent semantic changes with reiser4

[Claus Färber]

Stephen Wille Padnos <spadnos@sover.net> schrieb/wrote:
> How does the file manager / chooser decide whether you're trying to
> move into a directory, or the meta-data-directory for a directory?
> It's not just files that should have metadata - directories need* them
> too.  Making it possible to see attributes as a directory under a file
> is great, but you'd still need an O_META flag for accessing directory
> metadata (since there are already files under a directory).

A simple convention that meta data files start with, say ".$", would be  
enough.

Claus
-- 
http://www.faerber.muc.de

Re: silent semantic changes with reiser4

[Claus Färber]

Stephen Wille Padnos <spadnos@sover.net> schrieb/wrote:
> How does the file manager / chooser decide whether you're trying to
> move into a directory, or the meta-data-directory for a directory?
> It's not just files that should have metadata - directories need* them
> too.  Making it possible to see attributes as a directory under a file
> is great, but you'd still need an O_META flag for accessing directory
> metadata (since there are already files under a directory).

A simple convention that meta data files start with, say ".$", would be  
enough.

Claus
-- 
http://www.faerber.muc.de

Re: silent semantic changes with reiser4

[Claus Färber]

Stephen Wille Padnos <spadnos@sover.net> schrieb/wrote:
> How does the file manager / chooser decide whether you're trying to
> move into a directory, or the meta-data-directory for a directory?
> It's not just files that should have metadata - directories need* them
> too.  Making it possible to see attributes as a directory under a file
> is great, but you'd still need an O_META flag for accessing directory
> metadata (since there are already files under a directory).

A simple convention that meta data files start with, say ".$", would be  
enough.

Claus
-- 
http://www.faerber.muc.de

Re: silent semantic changes with reiser4

[Alan Cox]

On Maw, 2004-08-31 at 12:01, Claus Färber wrote:
> A simple convention that meta data files start with, say ".$", would be  
> enough.

POSIX/SuS don't permit this. The only "free" namespace is that starting
"//" (and not as some desktops seem to think foo://). Remember always
send GUI desktop users files called http://read.me  .. its fun 8)

The // doesn't really help because you don't want extensions at the path
top. 

Another interesting question btw with substreams of a file is what the
semantics of fchown, fstat, fchmod, fchdir etc are, or of mounting a
file over a substream.

Alan

Re: silent semantic changes with reiser4

[Alan Cox]

On Maw, 2004-08-31 at 12:01, Claus Färber wrote:
> A simple convention that meta data files start with, say ".$", would be  
> enough.

POSIX/SuS don't permit this. The only "free" namespace is that starting
"//" (and not as some desktops seem to think foo://). Remember always
send GUI desktop users files called http://read.me  .. its fun 8)

The // doesn't really help because you don't want extensions at the path
top. 

Another interesting question btw with substreams of a file is what the
semantics of fchown, fstat, fchmod, fchdir etc are, or of mounting a
file over a substream.

Alan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
> 
> Thinking about it some more, how would file managers and file chosers
> handle this situation ?

The same way they already handle it on other platforms that support it? By 
taking advantage of it..

People literally use this for icon and preview information, so some of the
stuff shows up very much in file managers. And I'd assume that a normal
double-click would just open the main file, and you can right-click for 
management information, including opening the file.

If you want an entity that acts as a directory, you create a directory. 

Directories don't go away - you still have S_ISDIR() and S_ISREG(), and 
they still return information. A file that has associated information is 
still a _file_, and people should treat it that way, it's just that it 
also has a list of named sub-streams. You can open it as a directory, but 
the stat information clearly says it is a file, and the "directory" view 
is very much associated with _that_ file.

I definitely don't think you want the file manager to act as if a named 
stream is exactly the same as a stand-alone file. They have all the same 
operations, but there's no question that there are differences too. 
Especially on a conceptual level - but for most filesystems there are 
likely real technical differences too.

For example, it's likely that most filesystems would _not_ allow linking 
of a named stream anywhere else. And you might not be able to change the 
permissions or date on the named stream either, since it may or may not 
have a separate date/permission thing from the container.

So don't believe that just because the named streams are _named_ like real 
files, that they suddenly have any existence beyond the container that 
they are part of.

There may be other limitations too - again depending on how the filesystem
actually implements named streams. It might not support more than one
level of naming, for example - so you might not be able to create a
directory structure within the named streams, for example.

In short: the fact that the VFS layer exposes the _capability_ to see the 
named streams as a full POSIX filesystem of its own does _not_ mean that 
the low-level filesystem necessarily allows the full possible semantics. 
So you shouldn't design your apps that know about named streams to think 
they are normal directories.

The directory thing is just a very powerful naming scheme, and one that 
fits into the existing UNIX model.

> Do we really want to have a file paradigm that's different
> from the other OSes out there ?

Different from what other OSes?

Last I looked, Windows, Solaris, and OSX all supported named streams. What 
other OS's are out there that you care about? 

In other words, this is _not_ a different paradigm from what others do.  
The discussion is whether we want to implement it at all, and more
importantly about syntactic issues (ie we clearly already implement
extended attributes, just with a much more limited syntactic power).

We don't have to go all the way. Solaris has "openat()", which is kind of 
a half-way there - not really directly available in the same namespace, 
but at least the result is available as a real file interface (as opposed 
to the Linux "xattr" interfaces that are _totally_ special-cased system 
calls).

In other words - the paradigm is already there. It's just that currently
it's pretty much unusable under Linux, because it requires so much
specialized knowledge that it's not worth it to modify existing apps. And
the interfaces are really very limited, so even if you _do_ end up using 
the specialized Linux interfaces, you can't actually do a lot of what you 
might want to do.

			Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>
>For example, it's likely that most filesystems would _not_ allow linking 
>of a named stream anywhere else.
>
Why?  It is just a file that inherits its stat data and is referenced by 
a concatenation method of its parent directory.  The link will also 
inherit its stat data from the original parent directory.

> And you might not be able to change the 
>permissions or date on the named stream either, since it may or may not 
>have a separate date/permission thing from the container.
>  
>
You should be able to change the permission and data, but when you 
change it, you change it for the whole container.

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>
>For example, it's likely that most filesystems would _not_ allow linking 
>of a named stream anywhere else.
>
Why?  It is just a file that inherits its stat data and is referenced by 
a concatenation method of its parent directory.  The link will also 
inherit its stat data from the original parent directory.

> And you might not be able to change the 
>permissions or date on the named stream either, since it may or may not 
>have a separate date/permission thing from the container.
>  
>
You should be able to change the permission and data, but when you 
change it, you change it for the whole container.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Hans Reiser wrote:
>
> Linus Torvalds wrote:
> >
> >For example, it's likely that most filesystems would _not_ allow linking 
> >of a named stream anywhere else.
>
> Why?  It is just a file that inherits its stat data and is referenced by 
> a concatenation method of its parent directory.  The link will also 
> inherit its stat data from the original parent directory.

In reiser4, maybe. Not in general. Look at what other filesystems do for 
attributes..

> > And you might not be able to change the 
> >permissions or date on the named stream either, since it may or may not 
> >have a separate date/permission thing from the container.
>
> You should be able to change the permission and data, but when you 
> change it, you change it for the whole container.

Why? I'd much rather disallow it, than allow it and then have "nonlocal" 
effects.

I can also see a filesystem validly having _extra_ permissions (ie apart
from the container) on it too.

Not all the world is reiser4. 

		Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>  
>
>>>And you might not be able to change the 
>>>permissions or date on the named stream either, since it may or may not 
>>>have a separate date/permission thing from the container.
>>>      
>>>
>>You should be able to change the permission and data,
>>
^data^date

>> but when you 
>>change it, you change it for the whole container.
>>    
>>
>
>Why? I'd much rather disallow it, than allow it and then have "nonlocal" 
>effects.
>  
>
Sometimes you want the nonlocal effects and sometimes you don't, and by 
decomposing streams into smaller primitives we can let users choose as 
they want.

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>  
>
>>>And you might not be able to change the 
>>>permissions or date on the named stream either, since it may or may not 
>>>have a separate date/permission thing from the container.
>>>      
>>>
>>You should be able to change the permission and data,
>>
^data^date

>> but when you 
>>change it, you change it for the whole container.
>>    
>>
>
>Why? I'd much rather disallow it, than allow it and then have "nonlocal" 
>effects.
>  
>
Sometimes you want the nonlocal effects and sometimes you don't, and by 
decomposing streams into smaller primitives we can let users choose as 
they want.

Re: silent semantic changes with reiser4

[viro]

On Thu, Aug 26, 2004 at 04:47:39PM -0700, Hans Reiser wrote:

> Sometimes you want the nonlocal effects and sometimes you don't, and by 
> decomposing streams into smaller primitives we can let users choose as 
> they want.

Right.  Now, would you kindly post the detailed technical analysis of your
wonderful design that handles that stuff safely?  With proof of correctness,
please.  Since we are expected to take your code and use it as "Uber"
replacement for the existing one, it surely would not be too much to expect,
would it?

And no, "we have a nice presentation somewhere on namesys" does not
qualify.

I apologize for the obvious posting in a thread already full of noise,
but I would like to make *very* sure that lack of ripping you another
one does *not* mutate into "no objections from viro" in a lovely thread
on SlashDot in a week or so.

Re: silent semantic changes with reiser4

[Hans Reiser]

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Thu, Aug 26, 2004 at 04:47:39PM -0700, Hans Reiser wrote:
>
>  
>
>>Sometimes you want the nonlocal effects and sometimes you don't, and by 
>>decomposing streams into smaller primitives we can let users choose as 
>>they want.
>>    
>>
>
>Right.  Now, would you kindly post the detailed technical analysis of your
>wonderful design that handles that stuff safely?  With proof of correctness,
>please.  Since we are expected to take your code and use it as "Uber"
>replacement for the existing one, it surely would not be too much to expect,
>would it?
>
>And no, "we have a nice presentation somewhere on namesys" does not
>qualify.
>
>I apologize for the obvious posting in a thread already full of noise,
>but I would like to make *very* sure that lack of ripping you another
>one does *not* mutate into "no objections from viro" in a lovely thread
>on SlashDot in a week or so.
>
>
>  
>
Over the next 6-18 months, one piece at a time, it will all fall into 
place. 

Reiser4 as it is shipping today, consists of close to the minimal 
functionality necessary to replace reiser3, albeit done according to our 
plugin model, and with vastly higher performance.  The full solution is 
not yet written, wait for it.  I understand why you think deadlocks are 
everything, but avoiding them is only a small, albeit necessary, piece 
of the task.

Thanks though for pointing out issues with unlinking files that have 
been cd'd to.

Re: silent semantic changes with reiser4

[Hans Reiser]

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Thu, Aug 26, 2004 at 04:47:39PM -0700, Hans Reiser wrote:
>
>  
>
>>Sometimes you want the nonlocal effects and sometimes you don't, and by 
>>decomposing streams into smaller primitives we can let users choose as 
>>they want.
>>    
>>
>
>Right.  Now, would you kindly post the detailed technical analysis of your
>wonderful design that handles that stuff safely?  With proof of correctness,
>please.  Since we are expected to take your code and use it as "Uber"
>replacement for the existing one, it surely would not be too much to expect,
>would it?
>
>And no, "we have a nice presentation somewhere on namesys" does not
>qualify.
>
>I apologize for the obvious posting in a thread already full of noise,
>but I would like to make *very* sure that lack of ripping you another
>one does *not* mutate into "no objections from viro" in a lovely thread
>on SlashDot in a week or so.
>
>
>  
>
Over the next 6-18 months, one piece at a time, it will all fall into 
place. 

Reiser4 as it is shipping today, consists of close to the minimal 
functionality necessary to replace reiser3, albeit done according to our 
plugin model, and with vastly higher performance.  The full solution is 
not yet written, wait for it.  I understand why you think deadlocks are 
everything, but avoiding them is only a small, albeit necessary, piece 
of the task.

Thanks though for pointing out issues with unlinking files that have 
been cd'd to.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> For objects that do both, how does the user choose ?
> 
> Do we really want to have a file paradigm that's different
> from the other OSes out there ?

What does MacOS X do?  Someone said that documents are directories in
it; it must know how to handle that.

What does Windows do when you click on a .zip file and WinZIP is
installed?  It opens the .zip file and lets you explore inside.  When
you click on a .doc file, though, it opens a viewer or editor -- you
don't get the option to look inside.

> What happens when users want to transfer data from Linux
> to another system ?

This is why I favour storing all essential metadata (about the file's
content) inside the file's contents, the primary stream.

We have another problem: what happens when users want to transfer data
from Windows (with secondary streams) and MacOS (with resource forks)?

-- Jamie

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Thu, Aug 26, 2004 at 09:48:41PM +0100, Jamie Lokier wrote:
> 
> This is why I favour storing all essential metadata (about the file's
> content) inside the file's contents, the primary stream.
> 
> We have another problem: what happens when users want to transfer data
> from Windows (with secondary streams) and MacOS (with resource forks)?

I can tell you that - right now we (Samba) has to drop all but the primary stream,
because the Linux filesystems don't have data stream support. Oh look,
we're losing user data (that's a *bad* thing :-).

Jeremy.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:52:18PM -0700, Jeremy Allison wrote:
> On Thu, Aug 26, 2004 at 09:48:41PM +0100, Jamie Lokier wrote:
> > 
> > This is why I favour storing all essential metadata (about the file's
> > content) inside the file's contents, the primary stream.
> > 
> > We have another problem: what happens when users want to transfer data
> > from Windows (with secondary streams) and MacOS (with resource forks)?
> 
> I can tell you that - right now we (Samba) has to drop all but the primary stream,
> because the Linux filesystems don't have data stream support. Oh look,
> we're losing user data (that's a *bad* thing :-).

Maybe you should learn from netatalk.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Fri, Aug 27, 2004 at 10:19:56AM +0100, Christoph Hellwig wrote:
> 
> Maybe you should learn from netatalk.

I am well aware of what netatalk does. However netatalk
isn't as widely used as Samba. Things they can get away
with would cause our user community to flay us alive.

We need a proper solution, not a nasty hack.

That's like me telling you to "learn from *BSD". You have
different user constituencies, you have to serve yours,
I have to serve mine.

Jeremy.

Re: silent semantic changes with reiser4

[Brad Boyer]

On Fri, Aug 27, 2004 at 08:56:29PM -0700, Jeremy Allison wrote:
> I am well aware of what netatalk does. However netatalk
> isn't as widely used as Samba. Things they can get away
> with would cause our user community to flay us alive.
> 
> We need a proper solution, not a nasty hack.
> 
> That's like me telling you to "learn from *BSD". You have
> different user constituencies, you have to serve yours,
> I have to serve mine.

While I do think that there are some hacks in netatalk, the
hacks there are not nearly as bad as the hacks that made it
into the kernel to support exporting a native Mac filesystem.
Take a look at the fork code in the hfs filesystem from 2.4
or before for exporting to netatalk.  It's disgusting, and
mostly because there is no easy way to export the resource
fork and extra metadata that is essential to the client. The
2.6 code doesn't have that anymore, but I don't think it
actually works with netatalk either. Perhaps someone can
correct me on that, but I don't see how it could work.

I would hope that something comes out of this whole discussion
that will allow a much better interface between the hfs/hfsplus
code in the kernel and netatalk.

	Brad Boyer
	flar@allandria.com

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 22:10, Rik van Riel wrote:

> Do we really want to have a file paradigm that's different
> from the other OSes out there ?
>
> What happens when users want to transfer data from Linux
> to another system ?

I think it depends on the transport being used: some transports could allow 
for metadata (i.e. a MIME-compatible transport) while some others don't. Of 
course, the remote side needs to also support metadata (i,e. using a 
MIME-compatible transport against a remote host that doesn't understand 
metadata is certainly impossible).

For example, with FTP transports, I guess the only possible option is to 
transfer the unnamed/default stream. NFS should allow metadata, but this 
probably needs some kind of extensions to the NFS protocol.

Re: silent semantic changes with reiser4

[Horst von Brand]

Rik van Riel <riel@redhat.com> said:
> On Thu, 26 Aug 2004, Linus Torvalds wrote:
> > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> > contains both a standard data stream (the "file" part) _and_ pointers to
> > other named streams (the "directory" part).

> Thinking about it some more, how would file managers and
> file chosers handle this situation ?
> 
> Currently the user browses the directory tree and when
> the user clicks on something, one of the following 
> happens:
> 
> 1) if it is a directory, the file manager/choser changes
>    into that directory
> 
> 2) if it is a file, the file is opened

And now you have a mess. Is it "really" a file, or a directory? Why not
just keep both well apart, and stay happy? I just fail to see what could be
gained by having directories that really aren't, and files that aren't
either. Use a directory if you want one, use a file elsewhere.

> Now how do we present things to users ?
> 
> How will users know when an object can only be chdired
> into, or only be opened ?

Easy: It is a directory, or it isn't.

> For objects that do both, how does the user choose ?

Don't give silly choices.

> Do we really want to have a file paradigm that's different
> from the other OSes out there ?

I vote "no". There are/have been OSes with weird "files", none of them
survived to get anywhere as popular as Unix and "file == stream of
bytes". Even with much simpler variants like files as sequences of
records. For a good reason: The Unix way is simple, and extremely flexible,
as my proggie can define at its own whim how to handle what's inside. If a
single stream isn't enough, we have directories. No need to innovate there.

> What happens when users want to transfer data from Linux
> to another system ?

Or between Linux systems with different kernels that happen to implement
different views/metadata on files.

Please do remember devfs: It sounded like a cool idea, got into the kernel
just to be thrown out later because nobody used it. Much heat was
generated, nothing of permanent value. This looks the same: A very vocal
tiny minority is clamoring for something completely non-Unix for totally
bogus reasons. What happened to "code talks, bullshit walks"? There is _no_
code (== real-world, user programs that can't be done efficiently enough
without this), so this nonsense should just be thrown out, and everybody go
back to real hacking.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Jamie Lokier]

Horst von Brand wrote:
> Please do remember devfs: It sounded like a cool idea, got into the kernel
> just to be thrown out later because nobody used it.

Are you kidding?  Lots of distros and many embedded Linuxes use devfs.

And udev _still_ doesn't create device nodes properly.  (Hint: I have
to run two modprobe commands before pppd works.  I have to run
modprobe before openvpn or bochs work.)

> What happened to "code talks, bullshit walks"?

devfs is a fine example of why code isn't enough.  With devfs the code
came first, the >1 year of strategic bullshit politics from the "it's
not traditional unix" crowd came later, then it went in, then lots of
people used it, then it was replaced by something which still doesn't
work as well as 2.4 does with or without devfs, and people are still
using it despite it's faults.

-- Jamie

Re: silent semantic changes with reiser4

[Olaf Hering]

 On Wed, Sep 01, Jamie Lokier wrote:

> And udev _still_ doesn't create device nodes properly.  (Hint: I have
> to run two modprobe commands before pppd works.  I have to run
> modprobe before openvpn or bochs work.)

Just because it is supposed to work that way.

-- 
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG

Re: silent semantic changes with reiser4

[Olaf Hering]

 On Wed, Sep 01, Jamie Lokier wrote:

> And udev _still_ doesn't create device nodes properly.  (Hint: I have
> to run two modprobe commands before pppd works.  I have to run
> modprobe before openvpn or bochs work.)

Just because it is supposed to work that way.

-- 
USB is for mice, FireWire is for men!

sUse lINUX ag, n√úRNBERG
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[Olaf Hering]

 On Wed, Sep 01, Jamie Lokier wrote:

> And udev _still_ doesn't create device nodes properly.  (Hint: I have
> to run two modprobe commands before pppd works.  I have to run
> modprobe before openvpn or bochs work.)

Just because it is supposed to work that way.

-- 
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>
>To me, a filesystem that allows this thing doesn't really _have_ the 
>concept of "directory vs file". It's just a "filesystem object", and it 
>can act as _both_ a directory and a file.
>
>  
>
I agree.

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>
>To me, a filesystem that allows this thing doesn't really _have_ the 
>concept of "directory vs file". It's just a "filesystem object", and it 
>can act as _both_ a directory and a file.
>
>  
>
I agree.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> What do we do with O_CREAT ?
> 
> Do we always allow both a directory and a file to be created with
> the same name ?

I haven't analysed it thoroughly.  I suspect it's fine as long as:

(1) O_CREAT creates something with "file-like
    attributes", meaning stat() says it's a regular file.

(2) File-like means it can be unlinked, linked and renamed, even if
    someone has something inside it open.  Nothing that can be created
    inside it will prevent it from being unlinked (unlike a
    directory-like object, where rmdir() will refuse if it's not empty).

    File-like also means that programs like "cp from to_my_new_file"
    won't do anything so silly as to write _inside_, the way that
    "cp from to_my_new_file/from" would.

    Basic utilities will need to be checked to make sure they don't
    try appending "/" as their method of deciding if the target object
    is a directory and should be entered.

> Does this create a new class of "symlink attack" style security
> holes ?

Yes, but they don't need O_CREAT.

An adversary creates a symlink to metadata inside your file.  You
write to it: it has interesting effects that weren't anticipated, such
as either modifying another (virtual) file, or altering permissions or
other parameters which writing doesn't normally do.

This is very difficult to prevent.

We are creating a way for scripts and classic unix tools to have easy
access to fancy attributes which may affect the security: things like
being able to change a file's permissions just by writing to the
appropriate path, or reading characteristics of files that shouldn't
be visible.  In Hans Reiser's example of expanded /etc/passwd, atimes
and mtimes of individual passwd entries is security information that
perhaps shouldn't be exposed.

In a way, these holes are similar to /proc, in that just writing has
side effects which might not be acceptable.

The solution is the same as for /proc (I hope): make sure the read
permissions on all metadata inside a directory branch are restricted
to the permissions of the file branch, and write permissions even more
restricted at least by default.

For the paranoid, one way is to make new files "not readable or
executable as directories".  It's possible for the file and directory
branches to have different permission bits, though I could see that
getting a bit annoying.  Other possibilities abound.

-- Jamie

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2037 bytes --]

Am Donnerstag, den 26.08.2004, 20:29 +0100 schrieb Jamie Lokier:

> (1) O_CREAT creates something with "file-like
>     attributes", meaning stat() says it's a regular file.
> 
> (2) File-like means it can be unlinked, linked and renamed, even if
>     someone has something inside it open.  Nothing that can be created
>     inside it will prevent it from being unlinked (unlike a
>     directory-like object, where rmdir() will refuse if it's not empty).

I would say hat the file and all of its directory gets invalidated and
deleted as soon as the last opener of the file's main stream or some of
its directory content disappears. How would that be?

> > Does this create a new class of "symlink attack" style security
> > holes ?
> 
> Yes, but they don't need O_CREAT.
> 
> An adversary creates a symlink to metadata inside your file.  You
> write to it: it has interesting effects that weren't anticipated, such
> as either modifying another (virtual) file, or altering permissions or
> other parameters which writing doesn't normally do.

Hmm.

> This is very difficult to prevent.

Right. But to change some properties you usually need to be the owner of
that file anyway, or root.

> In Hans Reiser's example of expanded /etc/passwd, atimes
> and mtimes of individual passwd entries is security information that
> perhaps shouldn't be exposed.

Only assuming mtime/ctime/atime/rwx/uid/gid of file contents can
actually be independant of those from the main stream. I would say that
the rights (read and write) for non-pseudo sub-files should the same as
the file, uid and gid 644 (only owner can change these), something like
that. mtime, atime and ctime always show the same as the file itself.
Hmm.

> The solution is the same as for /proc (I hope): make sure the read
> permissions on all metadata inside a directory branch are restricted
> to the permissions of the file branch, and write permissions even more
> restricted at least by default.

That's what I was thinking.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2037 bytes --]

Am Donnerstag, den 26.08.2004, 20:29 +0100 schrieb Jamie Lokier:

> (1) O_CREAT creates something with "file-like
>     attributes", meaning stat() says it's a regular file.
> 
> (2) File-like means it can be unlinked, linked and renamed, even if
>     someone has something inside it open.  Nothing that can be created
>     inside it will prevent it from being unlinked (unlike a
>     directory-like object, where rmdir() will refuse if it's not empty).

I would say hat the file and all of its directory gets invalidated and
deleted as soon as the last opener of the file's main stream or some of
its directory content disappears. How would that be?

> > Does this create a new class of "symlink attack" style security
> > holes ?
> 
> Yes, but they don't need O_CREAT.
> 
> An adversary creates a symlink to metadata inside your file.  You
> write to it: it has interesting effects that weren't anticipated, such
> as either modifying another (virtual) file, or altering permissions or
> other parameters which writing doesn't normally do.

Hmm.

> This is very difficult to prevent.

Right. But to change some properties you usually need to be the owner of
that file anyway, or root.

> In Hans Reiser's example of expanded /etc/passwd, atimes
> and mtimes of individual passwd entries is security information that
> perhaps shouldn't be exposed.

Only assuming mtime/ctime/atime/rwx/uid/gid of file contents can
actually be independant of those from the main stream. I would say that
the rights (read and write) for non-pseudo sub-files should the same as
the file, uid and gid 644 (only owner can change these), something like
that. mtime, atime and ctime always show the same as the file itself.
Hmm.

> The solution is the same as for /proc (I hope): make sure the read
> permissions on all metadata inside a directory branch are restricted
> to the permissions of the file branch, and write permissions even more
> restricted at least by default.

That's what I was thinking.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Herbert Poetzl]

On Thu, Aug 26, 2004 at 11:55:07AM -0700, Linus Torvalds wrote:
> 
> 
> On Thu, 26 Aug 2004, Rik van Riel wrote:
> > 
> > So you'd have both a file and a directory that just happen
> > to have the same name ?  How would this work in the dcache?
> 
> There would be only one entry in the dcache. The lookup will select 
> whether it opens the file or the directory based on O_DIRECTORY (and 
> usage, of course - if it's in the middle of a path, it obviously needs to 
> be opened as a directory regardless).

hmm, that might be interesting for applications
like apache which allow to append arguments by
'building' a 'virtual' path ...

http://www.kernel.org/list.cgi/some/more/args

where the web server actually decides where the
script ends and the arguments start by checking
the filesystem ...

> That's not the problem. The problem from a dcache standpoint ends up being 
> when the file has a link, and you have two paths to the same sub-file 
> through two different ways:
> 
> 	.. create file 'x' with named stream 'y' ...
> 	ln x z
> 	ls -l x/y z/y	/* it's the same attribute!! */
> 
> but this is actually exactly the same thing that we already have with 
> mounts, ie it is equivalent (from a dentry standpoint) to
> 
> 	.. create directory 'x' with file 'y' ..
> 	mkdir z
> 	mount --bind x z
> 	ls -l x/y z/y	/* It's the same file!! */
> 
> so none of this is really anything "new" from a dcache standpoint.
> 
> Except for all the details, of course ;)

if the file is removed, but some attributes are
'locked' by access, will the 'other' attributes
remain visible or disappear ... leaving a partial
'view' of the contents? or did I get the idea
completely wrong?

TIA,
Herbert

> 		Linus
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Linus Torvalds wrote:
[...]
| 	tar cvf file.tar file file/

I don't like it.  Imagine this:
$ cat file/perm
040700 drwx------
$ cat file/perm/user/writable
1
$ cat file/perm/uid
0

Do that tar, and you lose the uid.  This is an example, and has plenty
of flaws.  The point is that often there may be files/directories within
files/directories.

In order to make this work, you either have to patch tar or split every
file/directory into a file/directory pair, recursively, using some sort
of mangling.  "file" becomes file.f and file.d.  Then tar works.

To be honest, I like my "serializer" better.  It just feels faster and
cleaner.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQS4z1XgHNmZLgCUhAQKXAQ//eJ9tPFFEf1kpFY7gYUhMSUAAs1wIvVrN
pOVyLPJwGTv4oIaq8/8Op+XmQt1BWfnFdgwSal5R7vuLlrj0/Lp714I4sTOb35+6
q4hB1Q6X1oKbS1QIVWM9zw8PaebTY1GZDYITqWEN8xFN2cx41m4HZZ9P2TXNGPWx
1TqeBAaR07N1LaDzDHWvXu2v8pP6Wjy8mmWbtHbjN7U8oLEAUVXmT/ZJ5MpVK6z6
FTTA6NnARpEDtWebTJit14IQYG88syl31+lNL/N87iTveYI3+6gAQniajkMda565
PnPbVCd/aLHpx4hNLYDiNZwGkhmCgA/V3/vHJGT1vbTWN56zFLF7XQqE0BMfR5k6
ozwzBZ368+suEs0pDplCJq7ru0BF3j49r2Ad7aILCJOENvN0uVzLNxmu+LfvARTm
gG7UNkJYcAjwmUaE1vXEACnwKhtyyxIpRbsb9ALqbDS9KXB7eAaI2FwLPXCU/L61
qHWHN47KllWdfT4fttwPd/Lag7UpKu9McBPnp9ZOTuYGY6QTgflBG1C7kallcJUw
7cZh831PRqmnj2hWcAlSvinD3ClxaF9swewMPfQJ9I10n7p6w9Q9EJEXUX6+gUhv
5MD6q2WxqNVM/i5Y/2s3GQ+3lJ2VvnWHkJgc7JkmP0H1N0vocYWhKaofMOQXAy57
/ocTFfcwzUc=
=YrfX
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Hans Reiser]

Why are you guys even considering going to any pain at all to distort 
semantics for the sake of backup?  tar is easy, we'll fix it and send in 
a patch. 

Re: silent semantic changes with reiser4

[Hans Reiser]

Why are you guys even considering going to any pain at all to distort 
semantics for the sake of backup?  tar is easy, we'll fix it and send in 
a patch. 

Re: silent semantic changes with reiser4

[Rik van Riel]

On Fri, 27 Aug 2004, Hans Reiser wrote:

> Why are you guys even considering going to any pain at all to distort 
> semantics for the sake of backup?  tar is easy, we'll fix it and send in 
> a patch. 

Because not everybody uses tar.  Quite a few people use a
network backup system, while others use duplicity, RPM uses
cpio internally and big companies tend to use proprietary
network backup suites.

Breaking people's setup is something to worry about.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Fri, 27 Aug 2004, Hans Reiser wrote:

> Why are you guys even considering going to any pain at all to distort 
> semantics for the sake of backup?  tar is easy, we'll fix it and send in 
> a patch. 

Because not everybody uses tar.  Quite a few people use a
network backup system, while others use duplicity, RPM uses
cpio internally and big companies tend to use proprietary
network backup suites.

Breaking people's setup is something to worry about.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Fri, 27 Aug 2004, Hans Reiser wrote:
>
>  
>
>>Why are you guys even considering going to any pain at all to distort 
>>semantics for the sake of backup?  tar is easy, we'll fix it and send in 
>>a patch. 
>>    
>>
>
>Because not everybody uses tar.  Quite a few people use a
>network backup system, while others use duplicity, RPM uses
>cpio internally and big companies tend to use proprietary
>network backup suites.
>
>Breaking people's setup is something to worry about.
>
>  
>
It is the tail that should not wag the dog.  New semantics are going to 
break backups other than dd.   We need a LOT of new semantics if we 
don't want to be inferior to Apple and MS.

We should implement backup plugin methods though, so that we only have 
to break things once....

Re: silent semantic changes with reiser4

[Lee Revell]

On Fri, 2004-08-27 at 14:27, Hans Reiser wrote:
> Rik van Riel wrote:
> >On Fri, 27 Aug 2004, Hans Reiser wrote:
> >
> >>Why are you guys even considering going to any pain at all to distort 
> >>semantics for the sake of backup?  tar is easy, we'll fix it and send in 
> >>a patch. 
> >
> >Because not everybody uses tar.  Quite a few people use a
> >network backup system, while others use duplicity, RPM uses
> >cpio internally and big companies tend to use proprietary
> >network backup suites.
> >
> >Breaking people's setup is something to worry about.
> >
> It is the tail that should not wag the dog.  New semantics are going to 
> break backups other than dd.   We need a LOT of new semantics if we 
> don't want to be inferior to Apple and MS.
> 
> We should implement backup plugin methods though, so that we only have 
> to break things once....

Every Linux shop I know of uses amanda anyway, which will not be
affected by these changes, as it uses dump/restore.  restore will need
minor changes, I think dump might not have to be changed at all.

If your backup system can't easily handle these changes, that's a pretty
good indication that it's poorly designed.

Lee

Re: silent semantic changes with reiser4

[Rik van Riel]

On Fri, 27 Aug 2004, Hans Reiser wrote:

> It is the tail that should not wag the dog.

That's no reason for dismissing technical questions with
vigorous handwaving.

> New semantics are going to break backups other than dd.  We need a LOT
> of new semantics if we don't want to be inferior to Apple and MS.

Technical objections need to be addressed, otherwise Linux
will end up inferior to Apple and MS.  Whatever choice we
end up making, it should be something that's both workable
and elegant.

It would be really nice if all the technical issues were
worked out on linux-kernel before the files-as-directories
stuff gets merged.  There are quite a few technical questions
in the thread that haven't been answered yet...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Fri, 27 Aug 2004, Hans Reiser wrote:

> It is the tail that should not wag the dog.

That's no reason for dismissing technical questions with
vigorous handwaving.

> New semantics are going to break backups other than dd.  We need a LOT
> of new semantics if we don't want to be inferior to Apple and MS.

Technical objections need to be addressed, otherwise Linux
will end up inferior to Apple and MS.  Whatever choice we
end up making, it should be something that's both workable
and elegant.

It would be really nice if all the technical issues were
worked out on linux-kernel before the files-as-directories
stuff gets merged.  There are quite a few technical questions
in the thread that haven't been answered yet...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>It would be really nice if all the technical issues were
>worked out on linux-kernel before the files-as-directories
>stuff gets merged.  There are quite a few technical questions
>in the thread that haven't been answered yet...
>
>  
>
Well, did I answer the technical question with the 
filename/pseudos/backup method answer in your opinion?

Re: silent semantic changes with reiser4

[Christian Mayrhuber]

On Friday 27 August 2004 16:12, Rik van Riel wrote:

> Because not everybody uses tar.  Quite a few people use a
> network backup system, while others use duplicity, RPM uses
> cpio internally and big companies tend to use proprietary
> network backup suites.
Big backup systems support scripts that can be run prior to
backup and post restore. You aready have to use star to
do a metadata backup the acl's/ea into some tgz, which
can be unpacked after restore.

> Breaking people's setup is something to worry about.
Yeah, maybe. But as usual on UN*X  like systems most admins
are able to come up with a quick shell/perl script to solve their
particular problem.

-- 
lg, Chris

Re: silent semantic changes with reiser4

[Rik van Riel]

On Fri, 27 Aug 2004, Hans Reiser wrote:

> Why are you guys even considering going to any pain at all to distort 
> semantics for the sake of backup?  tar is easy, we'll fix it and send in 
> a patch. 

It's not as easy as you make it out, and not just because
there are a few dozen backup programs that need fixing.

The problem is more fundamental than that.  Some of the
file streams proposed need to be backed up, while others
are alternative presentations of the file, which should
not be backed up.

Currently I see no way to distinguish between the stuff
that should be backed up and the stuff that shouldn't.

That problem needs to be resolved before we can even start
thinking about fixing archivers...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Fri, 27 Aug 2004, Hans Reiser wrote:

> Why are you guys even considering going to any pain at all to distort 
> semantics for the sake of backup?  tar is easy, we'll fix it and send in 
> a patch. 

It's not as easy as you make it out, and not just because
there are a few dozen backup programs that need fixing.

The problem is more fundamental than that.  Some of the
file streams proposed need to be backed up, while others
are alternative presentations of the file, which should
not be backed up.

Currently I see no way to distinguish between the stuff
that should be backed up and the stuff that shouldn't.

That problem needs to be resolved before we can even start
thinking about fixing archivers...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Spam]

> On Fri, 27 Aug 2004, Hans Reiser wrote:

>> Why are you guys even considering going to any pain at all to distort
>> semantics for the sake of backup?  tar is easy, we'll fix it and send in
>> a patch. 

> It's not as easy as you make it out, and not just because
> there are a few dozen backup programs that need fixing.

> The problem is more fundamental than that.  Some of the
> file streams proposed need to be backed up, while others
> are alternative presentations of the file, which should
> not be backed up.

  No, not really. This is a user decision and should be options in the
  backup  software.  I don't think it is up to the kernel, filesystem,
  or  the  OS  in  general to decide what information the user want to
  retain or not.

> Currently I see no way to distinguish between the stuff
> that should be backed up and the stuff that shouldn't.

> That problem needs to be resolved before we can even start
> thinking about fixing archivers...

  The  archivers  should,  as  I  said,  allow  the user to choose. It
  shouldn't be automatic. Default, should IMO be to store everything.

  ~S

Re: silent semantic changes with reiser4

[Spam]

> On Fri, 27 Aug 2004, Hans Reiser wrote:

>> Why are you guys even considering going to any pain at all to distort
>> semantics for the sake of backup?  tar is easy, we'll fix it and send in
>> a patch. 

> It's not as easy as you make it out, and not just because
> there are a few dozen backup programs that need fixing.

> The problem is more fundamental than that.  Some of the
> file streams proposed need to be backed up, while others
> are alternative presentations of the file, which should
> not be backed up.

  No, not really. This is a user decision and should be options in the
  backup  software.  I don't think it is up to the kernel, filesystem,
  or  the  OS  in  general to decide what information the user want to
  retain or not.

> Currently I see no way to distinguish between the stuff
> that should be backed up and the stuff that shouldn't.

> That problem needs to be resolved before we can even start
> thinking about fixing archivers...

  The  archivers  should,  as  I  said,  allow  the user to choose. It
  shouldn't be automatic. Default, should IMO be to store everything.

  ~S

Re: silent semantic changes with reiser4

[Spam]

> On Fri, 27 Aug 2004, Hans Reiser wrote:

>> Why are you guys even considering going to any pain at all to distort
>> semantics for the sake of backup?  tar is easy, we'll fix it and send in
>> a patch. 

> It's not as easy as you make it out, and not just because
> there are a few dozen backup programs that need fixing.

> The problem is more fundamental than that.  Some of the
> file streams proposed need to be backed up, while others
> are alternative presentations of the file, which should
> not be backed up.

  No, not really. This is a user decision and should be options in the
  backup  software.  I don't think it is up to the kernel, filesystem,
  or  the  OS  in  general to decide what information the user want to
  retain or not.

> Currently I see no way to distinguish between the stuff
> that should be backed up and the stuff that shouldn't.

> That problem needs to be resolved before we can even start
> thinking about fixing archivers...

  The  archivers  should,  as  I  said,  allow  the user to choose. It
  shouldn't be automatic. Default, should IMO be to store everything.

  ~S

Re: [SPAM] Re: silent semantic changes with reiser4

[Jamie Lokier]

Spam wrote:
> > The problem is more fundamental than that.  Some of the
> > file streams proposed need to be backed up, while others
> > are alternative presentations of the file, which should
> > not be backed up.
> 
>   No, not really. This is a user decision and should be options in the
>   backup  software.  I don't think it is up to the kernel, filesystem,
>   or  the  OS  in  general to decide what information the user want to
>   retain or not.

It is helpful for the OS, or a naming convention, to indicate what
_is information_ though.

It makes no sense to backup two or more copies of the _same
information_, and it makes even less sense to try to restore them as
it'll either be slow, fail (you can't always write to alternative
presentations), or cause unwanted side effects.

Just like when you backup a dynamic web site.  You store the files
which the server is using.  You don't use "wget" to store the
generated pages, that's not a useful backup and you can't restore from it.

> > Currently I see no way to distinguish between the stuff
> > that should be backed up and the stuff that shouldn't.
> 
> > That problem needs to be resolved before we can even start
> > thinking about fixing archivers...
> 
>   The  archivers  should,  as  I  said,  allow  the user to choose. It
>   shouldn't be automatic. Default, should IMO be to store everything.

Don't try to store different views of the same thing.
When you try to restore, you _won't_ necessarily get back what you stored.

Whereas if you follow the OS's advice, and skip virtual files, then
backup and restore will recreate the filesystem, which is what you want.

_That's_ storing everything.  It's what you want from a backup.

-- Jamie

Re: [SPAM] Re: silent semantic changes with reiser4

[Hendrik Visage]

On Fri, Aug 27, 2004 at 04:41:56PM +0100, Jamie Lokier wrote:
> 
> It is helpful for the OS, or a naming convention, to indicate what
> _is information_ though.
> 
> It makes no sense to backup two or more copies of the _same
> information_, and it makes even less sense to try to restore them as
> it'll either be slow, fail (you can't always write to alternative
> presentations), or cause unwanted side effects.

You sure you don't mean what is data (on disk) and what is information, ie.
interpretted data?

Following this whole threat is quite interesting, but I think there are
two things that need to be pointed out (which I might've missed) is that
hardlinks would be backed up multiple times if the archiver doesn't know
the semantics of the filesystem. This is the reason why the <fs-of-choice>dump
usually are faster than tar/cpio/etc. as they understand the low level and
can optimize at that level.

Also, a user might only have access to a certain subset of information, 
which he might to backup but not the rest of the stream/data, and such the
choice should be the user's.

> Just like when you backup a dynamic web site.  You store the files
> which the server is using.  You don't use "wget" to store the
> generated pages, that's not a useful backup and you can't restore from it.

That depends ;^)
But in the general case it's true.

Re: silent semantic changes with reiser4

[Spam]

  

> Spam wrote:
>> > The problem is more fundamental than that.  Some of the
>> > file streams proposed need to be backed up, while others
>> > are alternative presentations of the file, which should
>> > not be backed up.
>> 
>>   No, not really. This is a user decision and should be options in the
>>   backup  software.  I don't think it is up to the kernel, filesystem,
>>   or  the  OS  in  general to decide what information the user want to
>>   retain or not.

> It is helpful for the OS, or a naming convention, to indicate what
> _is information_ though.

> It makes no sense to backup two or more copies of the _same
> information_, and it makes even less sense to try to restore them as
> it'll either be slow, fail (you can't always write to alternative
> presentations), or cause unwanted side effects.

> Just like when you backup a dynamic web site.  You store the files
> which the server is using.  You don't use "wget" to store the
> generated pages, that's not a useful backup and you can't restore from it.

  I do not agree. Everything can be considered information, even if it
  is derived from an already existing file.

  If  the  user  wants  to  not  backup certain things then that is an
  option to tell the backup program.

>> > Currently I see no way to distinguish between the stuff
>> > that should be backed up and the stuff that shouldn't.
>> 
>> > That problem needs to be resolved before we can even start
>> > thinking about fixing archivers...
>> 
>>   The  archivers  should,  as  I  said,  allow  the user to choose. It
>>   shouldn't be automatic. Default, should IMO be to store everything.

> Don't try to store different views of the same thing.
> When you try to restore, you _won't_ necessarily get back what you stored.

  No, it would be restored as it were before with the meta information
  intact.

> Whereas if you follow the OS's advice, and skip virtual files, then
> backup and restore will recreate the filesystem, which is what you want.

> _That's_ storing everything.  It's what you want from a backup.

  It would be storing everything but the virtual files.


> -- Jamie

Re: silent semantic changes with reiser4

[Spam]

  

> Spam wrote:
>> > The problem is more fundamental than that.  Some of the
>> > file streams proposed need to be backed up, while others
>> > are alternative presentations of the file, which should
>> > not be backed up.
>> 
>>   No, not really. This is a user decision and should be options in the
>>   backup  software.  I don't think it is up to the kernel, filesystem,
>>   or  the  OS  in  general to decide what information the user want to
>>   retain or not.

> It is helpful for the OS, or a naming convention, to indicate what
> _is information_ though.

> It makes no sense to backup two or more copies of the _same
> information_, and it makes even less sense to try to restore them as
> it'll either be slow, fail (you can't always write to alternative
> presentations), or cause unwanted side effects.

> Just like when you backup a dynamic web site.  You store the files
> which the server is using.  You don't use "wget" to store the
> generated pages, that's not a useful backup and you can't restore from it.

  I do not agree. Everything can be considered information, even if it
  is derived from an already existing file.

  If  the  user  wants  to  not  backup certain things then that is an
  option to tell the backup program.

>> > Currently I see no way to distinguish between the stuff
>> > that should be backed up and the stuff that shouldn't.
>> 
>> > That problem needs to be resolved before we can even start
>> > thinking about fixing archivers...
>> 
>>   The  archivers  should,  as  I  said,  allow  the user to choose. It
>>   shouldn't be automatic. Default, should IMO be to store everything.

> Don't try to store different views of the same thing.
> When you try to restore, you _won't_ necessarily get back what you stored.

  No, it would be restored as it were before with the meta information
  intact.

> Whereas if you follow the OS's advice, and skip virtual files, then
> backup and restore will recreate the filesystem, which is what you want.

> _That's_ storing everything.  It's what you want from a backup.

  It would be storing everything but the virtual files.


> -- Jamie

Re: silent semantic changes with reiser4

[Spam]

  

> Spam wrote:
>> > The problem is more fundamental than that.  Some of the
>> > file streams proposed need to be backed up, while others
>> > are alternative presentations of the file, which should
>> > not be backed up.
>> 
>>   No, not really. This is a user decision and should be options in the
>>   backup  software.  I don't think it is up to the kernel, filesystem,
>>   or  the  OS  in  general to decide what information the user want to
>>   retain or not.

> It is helpful for the OS, or a naming convention, to indicate what
> _is information_ though.

> It makes no sense to backup two or more copies of the _same
> information_, and it makes even less sense to try to restore them as
> it'll either be slow, fail (you can't always write to alternative
> presentations), or cause unwanted side effects.

> Just like when you backup a dynamic web site.  You store the files
> which the server is using.  You don't use "wget" to store the
> generated pages, that's not a useful backup and you can't restore from it.

  I do not agree. Everything can be considered information, even if it
  is derived from an already existing file.

  If  the  user  wants  to  not  backup certain things then that is an
  option to tell the backup program.

>> > Currently I see no way to distinguish between the stuff
>> > that should be backed up and the stuff that shouldn't.
>> 
>> > That problem needs to be resolved before we can even start
>> > thinking about fixing archivers...
>> 
>>   The  archivers  should,  as  I  said,  allow  the user to choose. It
>>   shouldn't be automatic. Default, should IMO be to store everything.

> Don't try to store different views of the same thing.
> When you try to restore, you _won't_ necessarily get back what you stored.

  No, it would be restored as it were before with the meta information
  intact.

> Whereas if you follow the OS's advice, and skip virtual files, then
> backup and restore will recreate the filesystem, which is what you want.

> _That's_ storing everything.  It's what you want from a backup.

  It would be storing everything but the virtual files.


> -- Jamie

Re: silent semantic changes with reiser4

[Rik van Riel]

On Sat, 28 Aug 2004, Spam wrote:

> > Whereas if you follow the OS's advice, and skip virtual files, then
> > backup and restore will recreate the filesystem, which is what you want.
> 
> > _That's_ storing everything.  It's what you want from a backup.
> 
>   It would be storing everything but the virtual files.

Thing is, there is no way to distinguish between what are
virtual files and what are actual streams hidden inside a
file.  You don't know what should and shouldn't be backed
up...

Hans says there must be a way, somehow, but I haven't seen
him tell this mailing list what exactly it is ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Sat, 28 Aug 2004, Spam wrote:

> > Whereas if you follow the OS's advice, and skip virtual files, then
> > backup and restore will recreate the filesystem, which is what you want.
> 
> > _That's_ storing everything.  It's what you want from a backup.
> 
>   It would be storing everything but the virtual files.

Thing is, there is no way to distinguish between what are
virtual files and what are actual streams hidden inside a
file.  You don't know what should and shouldn't be backed
up...

Hans says there must be a way, somehow, but I haven't seen
him tell this mailing list what exactly it is ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Fri, 27 Aug 2004, Rik van Riel wrote:
> 
> Thing is, there is no way to distinguish between what are
> virtual files and what are actual streams hidden inside a
> file.  You don't know what should and shouldn't be backed
> up...

I think that lack of distinguishing poiwer is more serious for 
directories. The more I think I think about it, the more I wonder whether 
Solaris did things right - having a special operation to "cross the 
boundary".

I suspect Solaris did it that way because it's a hell of a lot easier to 
do it like that, but regardless, it would solve the issue of real 
directories having both real children _and_ the "extra streams".

		Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Just have a special name instead of a special boundary, or, better, have 
a filename/pseudos/backup method that outputs everything needed to 
backup the object "filename".

Hans

Linus Torvalds wrote:

>On Fri, 27 Aug 2004, Rik van Riel wrote:
>  
>
>>Thing is, there is no way to distinguish between what are
>>virtual files and what are actual streams hidden inside a
>>file.  You don't know what should and shouldn't be backed
>>up...
>>    
>>
>
>I think that lack of distinguishing poiwer is more serious for 
>directories. The more I think I think about it, the more I wonder whether 
>Solaris did things right - having a special operation to "cross the 
>boundary".
>
>I suspect Solaris did it that way because it's a hell of a lot easier to 
>do it like that, but regardless, it would solve the issue of real 
>directories having both real children _and_ the "extra streams".
>
>		Linus
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Just have a special name instead of a special boundary, or, better, have 
a filename/pseudos/backup method that outputs everything needed to 
backup the object "filename".

Hans

Linus Torvalds wrote:

>On Fri, 27 Aug 2004, Rik van Riel wrote:
>  
>
>>Thing is, there is no way to distinguish between what are
>>virtual files and what are actual streams hidden inside a
>>file.  You don't know what should and shouldn't be backed
>>up...
>>    
>>
>
>I think that lack of distinguishing poiwer is more serious for 
>directories. The more I think I think about it, the more I wonder whether 
>Solaris did things right - having a special operation to "cross the 
>boundary".
>
>I suspect Solaris did it that way because it's a hell of a lot easier to 
>do it like that, but regardless, it would solve the issue of real 
>directories having both real children _and_ the "extra streams".
>
>		Linus
>
>
>  
>

Re: silent semantic changes with reiser4

[Spam]

  I  am  thinking  that  is  there  was a proper API for accessing the
  filesystem  then this problem wouldn't arise because things could be
  done  behind  the curtains inside the API, instead of having all the
  tools to be rewritten to know.

  Think of FAT32, for example, the new filenames will be retained even
  if  you use old tools on them. It is only if you mount it as a FAT16
  volume they will be lost.

  Even  if  this  is fixed for Reiser4 now. Next time someone wants to
  make  changes like this we have the exact same problem yet again.
  

> Just have a special name instead of a special boundary, or, better, have
> a filename/pseudos/backup method that outputs everything needed to 
> backup the object "filename".

> Hans

> Linus Torvalds wrote:

>>On Fri, 27 Aug 2004, Rik van Riel wrote:
>>  
>>
>>>Thing is, there is no way to distinguish between what are
>>>virtual files and what are actual streams hidden inside a
>>>file.  You don't know what should and shouldn't be backed
>>>up...
>>>    
>>>
>>
>>I think that lack of distinguishing poiwer is more serious for 
>>directories. The more I think I think about it, the more I wonder whether
>>Solaris did things right - having a special operation to "cross the 
>>boundary".
>>
>>I suspect Solaris did it that way because it's a hell of a lot easier to
>>do it like that, but regardless, it would solve the issue of real 
>>directories having both real children _and_ the "extra streams".
>>
>>		Linus
>>
>>
>>  
>>

Re: silent semantic changes with reiser4

[Spam]

  I  am  thinking  that  is  there  was a proper API for accessing the
  filesystem  then this problem wouldn't arise because things could be
  done  behind  the curtains inside the API, instead of having all the
  tools to be rewritten to know.

  Think of FAT32, for example, the new filenames will be retained even
  if  you use old tools on them. It is only if you mount it as a FAT16
  volume they will be lost.

  Even  if  this  is fixed for Reiser4 now. Next time someone wants to
  make  changes like this we have the exact same problem yet again.
  

> Just have a special name instead of a special boundary, or, better, have
> a filename/pseudos/backup method that outputs everything needed to 
> backup the object "filename".

> Hans

> Linus Torvalds wrote:

>>On Fri, 27 Aug 2004, Rik van Riel wrote:
>>  
>>
>>>Thing is, there is no way to distinguish between what are
>>>virtual files and what are actual streams hidden inside a
>>>file.  You don't know what should and shouldn't be backed
>>>up...
>>>    
>>>
>>
>>I think that lack of distinguishing poiwer is more serious for 
>>directories. The more I think I think about it, the more I wonder whether
>>Solaris did things right - having a special operation to "cross the 
>>boundary".
>>
>>I suspect Solaris did it that way because it's a hell of a lot easier to
>>do it like that, but regardless, it would solve the issue of real 
>>directories having both real children _and_ the "extra streams".
>>
>>		Linus
>>
>>
>>  
>>

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Hans" == Hans Reiser <reiser@namesys.com> writes:

Hans> Just have a special name instead of a special boundary, or,
Hans> better, have a filename/pseudos/backup method that outputs
Hans> everything needed to backup the object "filename".

For the Reiser4 case, this could also be solved using views, right?
Just create a view that exposes everything that should be backed up,
and nothing else?  What "instructions" would a backup program need,
other than knowing what files to backup, and what files to not backup?

Where are views going to be handled?  It seems like something that would
be useful for every other filesystem, and so shouldn't be handled too
close to Reiser4.  (Or is that a debate/discussion/flamewar we don't
want to get into right now? ;-) )

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Hans" == Hans Reiser <reiser@namesys.com> writes:

Hans> Just have a special name instead of a special boundary, or,
Hans> better, have a filename/pseudos/backup method that outputs
Hans> everything needed to backup the object "filename".

For the Reiser4 case, this could also be solved using views, right?
Just create a view that exposes everything that should be backed up,
and nothing else?  What "instructions" would a backup program need,
other than knowing what files to backup, and what files to not backup?

Where are views going to be handled?  It seems like something that would
be useful for every other filesystem, and so shouldn't be handled too
close to Reiser4.  (Or is that a debate/discussion/flamewar we don't
want to get into right now? ;-) )

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Helge Hafting]

On Fri, Aug 27, 2004 at 07:05:35PM -0700, Linus Torvalds wrote:
> 
> 
> On Fri, 27 Aug 2004, Rik van Riel wrote:
> > 
> > Thing is, there is no way to distinguish between what are
> > virtual files and what are actual streams hidden inside a
> > file.  You don't know what should and shouldn't be backed
> > up...
> 
> I think that lack of distinguishing poiwer is more serious for 
> directories. The more I think I think about it, the more I wonder whether 
> Solaris did things right - having a special operation to "cross the 
> boundary".
> 
> I suspect Solaris did it that way because it's a hell of a lot easier to 
> do it like that, but regardless, it would solve the issue of real 
> directories having both real children _and_ the "extra streams".

There are many ways of doing this. Several extra streams to a directory
that aren't ordinary files in the directory?

It seems to me that we can get a lot of nice functionality in a simpler way:
Instead of thinking about a number of streams attached to something
that is either an ordinary file or directory, just say that the only
change will be that a directory may have a _single_ file stream in
addition to being a plain directory.

The conceptual change is smaller.  Still you achieve multiple
streams, the "main" stream in such a dir+file is the single stream
attached to the directory.  All the "extra streams" are files inside the 
(normal) directory.

* This way, the extra streams aren't "special" in any way and
  should work fine with existing tools. 
* Serving such a filesystem to clients that use multiple streams
  will allow any number of streams per file for them
* Moving the directory+file thing will move both the directory and
  the special stream, and all the extra streams follow because they're
  �part of the directory as usual�.

If the VFS is to be extended in order to support file-as-directory (or
vice versa) then hopefully it can be done in a simple way.

Helge Hafting
 

Re: silent semantic changes with reiser4

[Helge Hafting]

On Fri, Aug 27, 2004 at 07:05:35PM -0700, Linus Torvalds wrote:
> 
> 
> On Fri, 27 Aug 2004, Rik van Riel wrote:
> > 
> > Thing is, there is no way to distinguish between what are
> > virtual files and what are actual streams hidden inside a
> > file.  You don't know what should and shouldn't be backed
> > up...
> 
> I think that lack of distinguishing poiwer is more serious for 
> directories. The more I think I think about it, the more I wonder whether 
> Solaris did things right - having a special operation to "cross the 
> boundary".
> 
> I suspect Solaris did it that way because it's a hell of a lot easier to 
> do it like that, but regardless, it would solve the issue of real 
> directories having both real children _and_ the "extra streams".

There are many ways of doing this. Several extra streams to a directory
that aren't ordinary files in the directory?

It seems to me that we can get a lot of nice functionality in a simpler way:
Instead of thinking about a number of streams attached to something
that is either an ordinary file or directory, just say that the only
change will be that a directory may have a _single_ file stream in
addition to being a plain directory.

The conceptual change is smaller.  Still you achieve multiple
streams, the "main" stream in such a dir+file is the single stream
attached to the directory.  All the "extra streams" are files inside the 
(normal) directory.

* This way, the extra streams aren't "special" in any way and
  should work fine with existing tools. 
* Serving such a filesystem to clients that use multiple streams
  will allow any number of streams per file for them
* Moving the directory+file thing will move both the directory and
  the special stream, and all the extra streams follow because they're
  �part of the directory as usual�.

If the VFS is to be extended in order to support file-as-directory (or
vice versa) then hopefully it can be done in a simple way.

Helge Hafting
 
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004, Helge Hafting wrote:
>
> > I think that lack of distinguishing power is more serious for 
> > directories. The more I think I think about it, the more I wonder whether 
> > Solaris did things right - having a special operation to "cross the 
> > boundary".
> > 
> > I suspect Solaris did it that way because it's a hell of a lot easier to 
> > do it like that, but regardless, it would solve the issue of real 
> > directories having both real children _and_ the "extra streams".
> 
> There are many ways of doing this. Several extra streams to a directory
> that aren't ordinary files in the directory?

Well.. Yes. We already have "." and "..", which are "special extra
streams" in a sense. However, people expect them, and know to ignore them. 
The same wouldn't be true of new naming.

> It seems to me that we can get a lot of nice functionality in a simpler way:
> Instead of thinking about a number of streams attached to something
> that is either an ordinary file or directory, just say that the only
> change will be that a directory may have a _single_ file stream in
> addition to being a plain directory.

That doesn't really help us. What would the name be, and how could you 
avoid clashes? 

> If the VFS is to be extended in order to support file-as-directory (or
> vice versa) then hopefully it can be done in a simple way.

I'm pretty confident that we can extend the VFS layer to support named
streams (see the technical discussion with Al, rather than the flames in
this thread). I also clearly believe that it is worth it, but I'm starting
to wonder if we should have a special open flag to make people select the
stream.

If you look at the Solaris interface, the _nice_ part about "openat()" is 
that you can do something like

	file = open(filename, O_RDONLY);
	if (file < 0)
		return -ENOENT;
	icon = openat(file, "icon", O_RDONLY | O_XATTR);
	if (icon < 0)
		icon = default_icon_file;
	..

and it will work regardless of whether "filename" is a directory or a 
regular file, if I've understood correctly.

Now, I think that makes sense for several reasons:
 - single case
 - race-free (think "stat()" vs "fstat()" races).
 - I think we want to do "openat()" regardless of whether we ever 
   support extended attributes or not ("openat()" is nice for doing 
   "namei()" in user space even in the absense of any attributes or 
   named streams).

So what we can do is
 - implement openat() regardless, and expect to do the Solaris thing for 
   it if we ever do streams.
 - _also_ support the "implied named attributes" for regular files, so 
   that you don't have to use "openat()" to access them.

Comments? Does anybody hate "openat()" for any reason (regardless of 
attributes)? We can easily support it, we'd just need to pass in the file 
to use as part of the "nameidata" thing or add an argument (it would also 
possibly be cleaner if we made "fs->pwd" be a "struct file").

		Linus

Re: silent semantic changes with reiser4

[Rik van Riel]

On Sat, 28 Aug 2004, Linus Torvalds wrote:
> Well.. Yes. We already have "." and "..", which are "special extra
> streams" in a sense. However, people expect them, and know to ignore them. 
> The same wouldn't be true of new naming.

If the streams that don't contain data weren't real
files, but instead fifos, backup programs would ignore
them fine.

Tar seems to ignore special nodes ...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Sat, 28 Aug 2004, Linus Torvalds wrote:
> Well.. Yes. We already have "." and "..", which are "special extra
> streams" in a sense. However, people expect them, and know to ignore them. 
> The same wouldn't be true of new naming.

If the streams that don't contain data weren't real
files, but instead fifos, backup programs would ignore
them fine.

Tar seems to ignore special nodes ...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[viro]

On Sat, Aug 28, 2004 at 11:09:38AM -0700, Linus Torvalds wrote:
> Comments? Does anybody hate "openat()" for any reason (regardless of 
> attributes)? We can easily support it, we'd just need to pass in the file 
> to use as part of the "nameidata" thing or add an argument (it would also 
> possibly be cleaner if we made "fs->pwd" be a "struct file").

What would your openat() produce?  Normal struct file?  Then what's going
to be its vfsmount/dentry and what will they be attached to?

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> What would your openat() produce?  Normal struct file?  Then what's going
> to be its vfsmount/dentry and what will they be attached to?

Normal file descriptor, exactly like "open()".

And it's going to have all the same vfsmount/dentry thing it would have if 
you looked it up the whole way. I don't understand your question..

Ignore the O_XATTR thing for a while, and assume it's just a convenient
combination of "fchdir + open" (plus "fchdir back" of course, but that's 
beside the point - openat() doesn't ever really change cwd).

Going back to O_XATTR: that would end up doing the "special vfsmount"  
magic at the beginning of the lookup. If the dentry you started with
wasn't marked D_HYBRID, it would just return -ENOTDIR.

So we could do openat() _without_ any of the lookup_mnt() etc special 
cases. This interface is independent of whether we want to expose the 
attributes through a normal lookup - we can do either, both, or neither as 
we choose.

The nice thing about "openat()" is
 - people can definitely find uses for it even without attributes. 
 - it's "portable". Well, at least somebody else does the same thing, 
   which is nice for user-space developers. You don't use a Linux-only 
   interface, you use a Linux/Solaris one, which makes a lot of people a 
   lot more happy.

Remember, portability has always been very important to Linux, and
Linux-only features while nice are certainly not as nice as features you
can also find in other places.

NIH is a disease.

		Linus

Re: silent semantic changes with reiser4

[viro]

On Sat, Aug 28, 2004 at 11:44:52AM -0700, Linus Torvalds wrote:
> Going back to O_XATTR: that would end up doing the "special vfsmount"  
> magic at the beginning of the lookup. If the dentry you started with
> wasn't marked D_HYBRID, it would just return -ENOTDIR.

OK, let me restate the question - what do we get from pwd if we do
fchdir() to such beast?

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
>
> On Sat, Aug 28, 2004 at 11:44:52AM -0700, Linus Torvalds wrote:
> > Going back to O_XATTR: that would end up doing the "special vfsmount"  
> > magic at the beginning of the lookup. If the dentry you started with
> > wasn't marked D_HYBRID, it would just return -ENOTDIR.
> 
> OK, let me restate the question - what do we get from pwd if we do
> fchdir() to such beast?

We'll see the "attribute path". We could (if we want to) mark the point 
where we walked into attribute space somehow (since we can see it by just 
looking at the vfsmount: d_root/d_mountpoint being the same), but even if 
we don't, we'll get a sane-looking path.

(It's not just "getcwd()", it's /proc interfaces too, for opening files,
and we already have the notion of magic markers like "(deleted)" to show
human-readable information).

The question about what to do at the "attribute point" (and there may
actually be several, if the filesystem supports attributes on attribute
files) likely depends on whether we support the previously discussed
"lookup()" magic for attributes.

So if we do support it, I think we should just make the attribute point
look exactly like a normal directory, since that path would work as-is. If
we don't support it (and at directory points), we might want to just
consider it a special root.

NOTE! Anybody who tries to use the "getcwd()" string as a real path is 
already broken - you have pathname permissions that may not make it 
possible to look up the path you see.

So we have multiple options:

 - support "file/attribute" lookup: show the path as-is, so you'd show it 
   as
	"/path/to/file/attribute"

 - alternatively, even if you _do_ support the normal lookup, show it with 
   a double slash (which will still be a valid path), just as a visual 
   clue:
	"/path/to/file//attribute"

 - for directories, or if we do _not_ support the extended lookup format,
   we could show it the same way we show deleted files, something like 
   this:
	"/path/to/file/attribute (attr)"

 - using "http notation" for non-standard-namespaces (we already do this
   for sockets and pipes, for example)

	"attr:attribute@/path/to/file"

pick your poison.

		Linus

Re: silent semantic changes with reiser4

[Andries Brouwer]

On Sat, Aug 28, 2004 at 12:16:47PM -0700, Linus Torvalds wrote:

If I see it correctly, you want to group a file and some
ancillary files together.

The Unix way would be to make a directory and put them all there:
	xterm/xterm
	xterm/xterm.icon

But you are unsatisfied and want
	xterm
	xterm/xterm.icon

As long as we agree that the latter really means the former,
there are no problems in finding out what should happen.

The conclusion is, that a directory carries an additional bit
that says "if I am opened as a regular file then use the file
of the same name inside".

Now there is no attribute space, just a shorthand.

Andries

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004, Andries Brouwer wrote:
> 
> If I see it correctly, you want to group a file and some
> ancillary files together.
> 
> The Unix way would be to make a directory and put them all there:
> 	xterm/xterm
> 	xterm/xterm.icon
> 
> But you are unsatisfied and want
> 	xterm
> 	xterm/xterm.icon
> 
> As long as we agree that the latter really means the former,
> there are no problems in finding out what should happen.
> 
> The conclusion is, that a directory carries an additional bit
> that says "if I am opened as a regular file then use the file
> of the same name inside".
> 
> Now there is no attribute space, just a shorthand.

It's more than a shorthand, though. _Much_ more.

There's the small issue of atomicity and locking.

There's the small issue of hardlinks.

Both of those are why it would need special support.

		Linus

Re: silent semantic changes with reiser4

[Andries Brouwer]

On Sat, Aug 28, 2004 at 12:46:10PM -0700, Linus Torvalds wrote:

> > Now there is no attribute space, just a shorthand.
> 
> It's more than a shorthand, though. _Much_ more.
> ...
> Both of those are why it would need special support.

Yes - support, I do not argue against that.
But I argue against an attribute space.

This xterm.icon was xterm.icons - with both large and small
icons. And these icons have copyrights. Etc. There is just
an arbitrary tree below this xterm filesystem object.
The files below are in no way different from the files above.

Unix says: everything is a file. Now we get: everything is
a directory. And some directories have a file attached.

Andries

Re: silent semantic changes with reiser4

[viro]

On Sat, Aug 28, 2004 at 12:16:47PM -0700, Linus Torvalds wrote:
> So if we do support it, I think we should just make the attribute point
> look exactly like a normal directory, since that path would work as-is. If
> we don't support it (and at directory points), we might want to just
> consider it a special root.
> 
> NOTE! Anybody who tries to use the "getcwd()" string as a real path is 
> already broken - you have pathname permissions that may not make it 
> possible to look up the path you see.

OK, forget getcwd().  What does lookup of .. do from that point?  *Especially*
for stuff you've got from regular files.  That's the decision that needs to
be made.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> OK, forget getcwd().  What does lookup of .. do from that point?  *Especially*
> for stuff you've got from regular files.  That's the decision that needs to
> be made.

I think that will decide on whether we expose attributes through the 
normal namespace or not.

If we do expose them in the normal namespace, then ".." should work the
way the namespace looks: if you do ".." on the "attribute directory" of a
file, you get the directory that the file was in. Ie an old-style 
user-space "getcwd()" would give the right path (well, an old-style 
user-space getcwd() would probably refuse the file on the base that it is 
S_IFREG, but ignoring that..)

If we _don't_ expose it in the normal namespace, we should should either
just error out (logically you'd get the file itself, but I really don't
wan tto have ".." return a non-directory, because _that_ really might
confuse things), or you'd just return the same directory (ie it would be a
"local root" in the namespace you got moved to).

So let's try to be self-consistent with how we expose it in the normal 
namespace.

		Linus

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Sat, Aug 28, 2004 at 10:12:39PM -0700, Linus Torvalds wrote:
> 
> If we do expose them in the normal namespace, then ".." should work the
> way the namespace looks: if you do ".." on the "attribute directory" of a
> file, you get the directory that the file was in. Ie an old-style 
> user-space "getcwd()" would give the right path (well, an old-style 
> user-space getcwd() would probably refuse the file on the base that it is 
> S_IFREG, but ignoring that..)

Errr, why don't we check what Solaris does in these cases and
see if that makes sense. They do seem to have made some progress
along this path and it wouldn't make sense to create new semantics
just for the sake of it.

Jeremy.

Re: silent semantic changes with reiser4

[Paul Jakma]

On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:

> OK, let me restate the question - what do we get from pwd if we do
> fchdir() to such beast?

On solaris: ENOTDIR from getcwd(), with 'pwd' executed inside a shell 
started with runat.

.. entry exists, but doesnt lead anywhere, cd .. returning ENOTDIR 
too.

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
If you didn't get caught, did you really do it?

Re: silent semantic changes with reiser4

[Hans Reiser]

I object to openat().....

My reason is that the things that distinguish between objects should be 
the names, not the choice of system call.  The reason for this is that 
it improves closure and namespace unification to do so, because it 
allows all the objects to be accessed within the same namespace.

Yes, it can be useful to allow a namespace to exclude some objects, but 
that exclusion should not be mandated.  If you want to exclude, you 
should cd or chroot to /proc/nopseudos and find a view of the filesystem 
that excludes metas, or mount with -nopseuodos.

Do you see why I say this?  I can say a lot more about the damage of 
fragmenting namespaces into multiple apis.... Why look at xattrs....;-)

Hans


Linus Torvalds wrote:

>
>
>I'm pretty confident that we can extend the VFS layer to support named
>streams (see the technical discussion with Al, rather than the flames in
>this thread). I also clearly believe that it is worth it, but I'm starting
>to wonder if we should have a special open flag to make people select the
>stream.
>
>If you look at the Solaris interface, the _nice_ part about "openat()" is 
>that you can do something like
>
>	file = open(filename, O_RDONLY);
>	if (file < 0)
>		return -ENOENT;
>	icon = openat(file, "icon", O_RDONLY | O_XATTR);
>	if (icon < 0)
>		icon = default_icon_file;
>	..
>
>and it will work regardless of whether "filename" is a directory or a 
>regular file, if I've understood correctly.
>
>Now, I think that makes sense for several reasons:
> - single case
> - race-free (think "stat()" vs "fstat()" races).
> - I think we want to do "openat()" regardless of whether we ever 
>   support extended attributes or not ("openat()" is nice for doing 
>   "namei()" in user space even in the absense of any attributes or 
>   named streams).
>
>So what we can do is
> - implement openat() regardless, and expect to do the Solaris thing for 
>   it if we ever do streams.
> - _also_ support the "implied named attributes" for regular files, so 
>   that you don't have to use "openat()" to access them.
>
>Comments? Does anybody hate "openat()" for any reason (regardless of 
>attributes)? We can easily support it, we'd just need to pass in the file 
>to use as part of the "nameidata" thing or add an argument (it would also 
>possibly be cleaner if we made "fs->pwd" be a "struct file").
>
>		Linus
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

I object to openat().....

My reason is that the things that distinguish between objects should be 
the names, not the choice of system call.  The reason for this is that 
it improves closure and namespace unification to do so, because it 
allows all the objects to be accessed within the same namespace.

Yes, it can be useful to allow a namespace to exclude some objects, but 
that exclusion should not be mandated.  If you want to exclude, you 
should cd or chroot to /proc/nopseudos and find a view of the filesystem 
that excludes metas, or mount with -nopseuodos.

Do you see why I say this?  I can say a lot more about the damage of 
fragmenting namespaces into multiple apis.... Why look at xattrs....;-)

Hans


Linus Torvalds wrote:

>
>
>I'm pretty confident that we can extend the VFS layer to support named
>streams (see the technical discussion with Al, rather than the flames in
>this thread). I also clearly believe that it is worth it, but I'm starting
>to wonder if we should have a special open flag to make people select the
>stream.
>
>If you look at the Solaris interface, the _nice_ part about "openat()" is 
>that you can do something like
>
>	file = open(filename, O_RDONLY);
>	if (file < 0)
>		return -ENOENT;
>	icon = openat(file, "icon", O_RDONLY | O_XATTR);
>	if (icon < 0)
>		icon = default_icon_file;
>	..
>
>and it will work regardless of whether "filename" is a directory or a 
>regular file, if I've understood correctly.
>
>Now, I think that makes sense for several reasons:
> - single case
> - race-free (think "stat()" vs "fstat()" races).
> - I think we want to do "openat()" regardless of whether we ever 
>   support extended attributes or not ("openat()" is nice for doing 
>   "namei()" in user space even in the absense of any attributes or 
>   named streams).
>
>So what we can do is
> - implement openat() regardless, and expect to do the Solaris thing for 
>   it if we ever do streams.
> - _also_ support the "implied named attributes" for regular files, so 
>   that you don't have to use "openat()" to access them.
>
>Comments? Does anybody hate "openat()" for any reason (regardless of 
>attributes)? We can easily support it, we'd just need to pass in the file 
>to use as part of the "nameidata" thing or add an argument (it would also 
>possibly be cleaner if we made "fs->pwd" be a "struct file").
>
>		Linus
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>

The Necessity of File Types (was silent semantic changes with reiser4)

[Alexander G. M. Smith]

Linus Torvalds wrote:
> I'm pretty confident that we can extend the VFS layer to support
> named streams [...]

Hans Reiser wrote on Sat, 28 Aug 2004 15:29:33 -0700:
> I object to openat().....
> 
> My reason is that the things that distinguish between objects should be 
> the names, not the choice of system call.  The reason for this is that 
> it improves closure and namespace unification to do so, because it 
> allows all the objects to be accessed within the same namespace.

I have to agree with Hans there.  Just have one main kind of file system object.  No separate stream system, other than as an optional alternative way of viewing things.  Treat it as a file or a directory or as an attribute as the context implies.  Have just one open call to open it, another to enumerate its children, and another to read and write data to it.  Paths to file system objects, no matter what their purpose, are all just plain paths (you can get fancy later with sub-name-spaces and name space boundaries).

But how do you tell what kind of thing it is?  How do you tell if it is important enough to back up?  How do you tell that a thing is intended to be treated as a directory containing other things rather than as a stand-alone object?  File types are needed for that!

In BeOS we had a MIME type database and all important files were given an attribute of BEOS:TYPE which contained their MIME string.  A system-wide database associated applications that handled each type, and also listed the standard attributes for each MIME type (such as Subject/From/To/... for "text/e-mail" files).  It even included some extra info about which attributes should be shown to the user and which were editable by the user in the desktop directory window GUI.  By the way, the database was just a tree of files (one for each MIME type using the same structure as MIME names) with attributes attached to them containing the relevant meta-meta-data.

Unclassified files (ones with no type, such as those generated by "cp" and other non-attribute aware programs) were treated as application/octet-stream.  There was also a background task that would try to identify them when the computer was idle, or if you tried to open them in a file requestor.  It used the file extension and also looked for characteristic signature bytes to pick the type.

Unfortunately BeOS had fragmented the type system by having a separate system of 4 character codes to identify the type of attributes (since attributes had their own API separate from files).  That would tell you the attribute was an INT32 or  C-String or Date.

I'd like to see one unified type system for the unified file/directory/attribute thing (has anyone named those multipurpose file system objects yet or can I call them Fildirutes?).  That means a global list of types and an attribute attached to all files needing a type.  That standard child file (used as an attribute) would be named something like "..meta.mimetype".  The global list would contain information about whether that type of thing needed to be backed up, what its standard computed attributes would be, and all sorts of common properties that all files of that type need.  Plus I'd add the primitive types, so "primitive/int32" and "primitive/date64" would be possible types for things which are simple attributes.

If you're worried about extra files, those file types could be stored as a unique ID (I recommend 8 character codes, using semi-readable text, for portability reasons) in each file inode.  The type would only appear as a file to users of the file system.

Another need for the type system is the need to index attributes.  Someone has to tell the OS that "MAIL:date-received" is a 64 bit number of microseconds since the start of time, so that the OS knows how to sort the index for that attribute.

Jan Harkes wrote on Sat, 28 Aug 2004 12:11:14 -0400:
> From what I saw in one part of the discussion, it allows for infinite
> depth recursion (file/metas/metas/metas/...). Some applications are
> going to have a lot of fun with that.

Some of the type information can be faked out as appearing as files.  So looking for the MIME type of a MIME type file (thing/..meta.mimetype/..meta.mimetype) would pretend to be a file that contained "text/mime-string", but not actually represent anything on disk.  Instead stock answers for the really primitive data types would be generated by the file system, recursively if needed.

The recursion should be there.  Just like it is for "." and "..", which "ls -R" for some reason doesn't traverse :-)

By the way, I disagree on Hans with using a directory for the meta data, I'd prefer to just have "..meta." as the name prefix of the metadata things.  Otherwise there's an extra directory level in there, making ..meta/ different from other directories (making browsing it a bit awkward since it doesn't have metadata of its own - like window coordinates for displaying the directory).

- Alex

Re: The Necessity of File Types (was silent semantic changes with reiser4)

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander G. M. Smith wrote:
[...]
| By the way, I disagree on Hans with using a directory for the meta
data, I'd prefer to just have "..meta." as the name prefix of the
metadata things.  Otherwise there's an extra directory level in there,
making ..meta/ different from other directories (making browsing it a
bit awkward since it doesn't have metadata of its own - like window
coordinates for displaying the directory).

You'd rather do "cat ..meta.pseudo" than "ls ..metas/"?
And we already have weird directories.  /proc doesn't do things that
/tmp does.  I also doubt proc will ever store that kind of metadata.

I don't like either "metas" or "..metas", but you'll have to look in the
archives for why.  I am NOT starting that again.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTPJW3gHNmZLgCUhAQKuEw/+PjrDfSlZvuTUwiOSa/kU05qjTKkIp05F
nBkrSIUKSUfoYN3gILucIGOjjJ5qBRW3wLIwWyO+raOAzHFTdylvR1rEWP9h/JUG
Psz3X8RE8cubiuNwjyJawI87cPX8HW1xQxgkiiquZN2L5XbvHsFsbwAJV1tJLnrM
2lL6IBUXcxDIh7pdB0/z+4t945ElT3Rl09CLP/MfP4GNvt/UWjhbeHNWdeLq93C+
w0irn6VIsm+1+GKqEhpYLLMj9V4cbBY6m7XMuBzH+gEyu5exAMqGzPLGDiG/I6y/
rXI6IF94i2p4sLMv0LmtEvQsV37LFIKRBrWVScV5oiQ93LPVLG6pYmAi6RLsHN03
QazMmcRwjVvrTAf0KS21KXaE0qYbvgoQpteUEkPZLRkczSmiry2d3e6gBQlCV7xj
9Hzkm9HxR0BjxmRaMVLEWZ86t08OHkl0Ab186DiHUC5RlL2FPIvdyYthXo9wwtI/
XlA03Rvc3kl6DVRO8UM/+fa8v5MDn58ZMI2QgdPrU0V2Q9bGv/woByszXaDA+IVJ
ZiHQ8IgShYlkA279DffOrZnDPnZMOWa7iRYetX5PYPoAyfkrmFKofT8va+dDL7pf
HA5crUR5ws1EZZ8lUfrtXXnsBeEn9PS0lUHvJY7zZ4tJfZ65Fa+IiDpK5p+1tJZi
ylCBNEidK/8=
=CiZ9
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004, Hans Reiser wrote:
>
> I object to openat().....

Sound slike you object to O_XATTRS, not openat() itself.

Realize that openat() works independently of any special streams, it's
fundamentally a "look up name starting from this file" (rather than
"starting from root" or "starting from cwd").

		Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Sat, 28 Aug 2004, Hans Reiser wrote:
>  
>
>>I object to openat().....
>>    
>>
>
>Sound slike you object to O_XATTRS, not openat() itself.
>
>Realize that openat() works independently of any special streams, it's
>fundamentally a "look up name starting from this file" (rather than
>"starting from root" or "starting from cwd").
>
>		Linus
>
>
>  
>
well, isn't that namespace fragmentation by definition?  If you can't go

cat filenameA/metas/permissions > filenameB/metas/permissions

find / -exec cat {}/permissions \; | grep 4777 | wc -l

then aren't you missing the whole point of this exercise which is to 
allow the whole OS to be better integrated into a more unified namespace 
so that data can leap from one tool to another, and one container to 
another, with the greatest of ease?  If cat cannot access the metadata 
without changing the code of cat, then all the elegance goes poof.

It completely baffles me what disabling filenameA/.. does for us.  Why 
add asymmetries?  Ease of implementation is no excuse for such asymmetry.

Tomorrow I am going to send a little essay I wrote this evening on these 
metafiles.

Re: silent semantic changes with reiser4

[CaT]

On Sun, Aug 29, 2004 at 02:36:50AM -0700, Hans Reiser wrote:
> Linus Torvalds wrote:
> >On Sat, 28 Aug 2004, Hans Reiser wrote:
> >>I object to openat().....
> >
> >Sound slike you object to O_XATTRS, not openat() itself.
> >
> >Realize that openat() works independently of any special streams, it's
> >fundamentally a "look up name starting from this file" (rather than
> >"starting from root" or "starting from cwd").
>
> well, isn't that namespace fragmentation by definition?  If you can't go
> 
> cat filenameA/metas/permissions > filenameB/metas/permissions
> 
> find / -exec cat {}/permissions \; | grep 4777 | wc -l

My apologies if this question has been raised (and answered) elsewhere.
I tried to keep up with the flamefest/discussion but it wasn't always
possible.

How would you do it for directories without making certain names
illegal? ie taking the above you'd do:

cat dirA/metas/permissions > dirB/metas/permissions

But how does that distinguish an access to the metas for dirA and dirB
from an access to a file called permissions within the metas directories
of dirA and dirB?

I think what Linus is trying to do, as far as naming, is create a
consistant access scheme that works for all types of objects in the
fs without making certain names illegal (and with plugins, those
names are not even a fixed thing I believe).

Feel free to correct me if I'm wrong, ofcourse. :)

-- 
    Red herrings strewn hither and yon.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "CaT" == CaT  <cat@zip.com.au> writes:

CaT> My apologies if this question has been raised (and answered)
CaT> elsewhere.  I tried to keep up with the flamefest/discussion but it
CaT> wasn't always possible.

CaT> How would you do it for directories without making certain names
CaT> illegal? ie taking the above you'd do:

CaT> cat dirA/metas/permissions > dirB/metas/permissions

Yes.  Directories are not allowed to contain objects with a filename
"metas".  The naming of "metas" has been discussed quite a bit on the
Reiserfs list, and I don't think there was much consensus that came out
of it.  Currently for Reiser4, AFAIK, the "metas" name is a compile-time
option that you can change by changing a #define, if you're worried
about name clashes.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "CaT" == CaT  <cat@zip.com.au> writes:

CaT> My apologies if this question has been raised (and answered)
CaT> elsewhere.  I tried to keep up with the flamefest/discussion but it
CaT> wasn't always possible.

CaT> How would you do it for directories without making certain names
CaT> illegal? ie taking the above you'd do:

CaT> cat dirA/metas/permissions > dirB/metas/permissions

Yes.  Directories are not allowed to contain objects with a filename
"metas".  The naming of "metas" has been discussed quite a bit on the
Reiserfs list, and I don't think there was much consensus that came out
of it.  Currently for Reiser4, AFAIK, the "metas" name is a compile-time
option that you can change by changing a #define, if you're worried
about name clashes.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Sun, Aug 29, 2004 at 05:05:17PM -0400, Hubert Chan wrote:
>Reiserfs list, and I don't think there was much consensus that came out

It's like Gentoo users and patch sets ;)

>of it.  Currently for Reiser4, AFAIK, the "metas" name is a compile-time
>option that you can change by changing a #define, if you're worried
>about name clashes.

http://mjt.nysv.org/reiser/reiser4.metas.patch

-- 
mjt

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004, Hans Reiser wrote:
> >
> >Realize that openat() works independently of any special streams, it's
> >fundamentally a "look up name starting from this file" (rather than
> >"starting from root" or "starting from cwd").
>
> well, isn't that namespace fragmentation by definition?

No.

There's no difference between

	fd = open("/usr/bin/yes", O_RDWR);

and

	dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY);
	fd = openat(dirfd, "yes", O_RDWR);

apart from error checking and permissions..

No new namespace.

Only O_XATTR implies a "namespace change".

		Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Sun, 29 Aug 2004, Hans Reiser wrote:
>  
>
>>>Realize that openat() works independently of any special streams, it's
>>>fundamentally a "look up name starting from this file" (rather than
>>>"starting from root" or "starting from cwd").
>>>      
>>>
>>well, isn't that namespace fragmentation by definition?
>>    
>>
>
>No.
>
>There's no difference between
>
>	fd = open("/usr/bin/yes", O_RDWR);
>
>and
>
>	dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY);
>	fd = openat(dirfd, "yes", O_RDWR);
>  
>
The difference is that cat uses open() not openat().

With your model, can I do:

cat filenameA/metas/permissions > filenameB/metas/permissions

find / -exec cat {}/permissions \; | grep 4777 | wc -l

If yes, then we are talking past each other somehow rather than 
disagreeing. If metafiles can be opened with both open and openat() in 
your model, then we are discussing some small detail.

I think the answer is no though, in which case you are missing the point 
of the new design. Is the answer no?

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Sun, 2004-08-29 at 12:53, Hans Reiser wrote:
> With your model, can I do:
> 
> cat filenameA/metas/permissions > filenameB/metas/permissions
> 

runat filenameA "cat permissions" | runat filenameB "cat > permissions"

> find / -exec cat {}/permissions \; | grep 4777 | wc -l

find / -exec runat {} "cat permissions" \; | grep 4777 | wc -l

Although, whether or not the kernel exposes file permissions as an
attribute named permissions is up to the filesystem. (And the wrong
thing to do, in my opinion, but that's irrelevant.)

> If yes, then we are talking past each other somehow rather than 
> disagreeing. If metafiles can be opened with both open and openat() in 
> your model, then we are discussing some small detail.

It can be opened by both, but in order to do it via open(2), you need to
fchdir(2) to the attribute directory as returned by openat(2) and then
use a relative pathname.

> I think the answer is no though, in which case you are missing the point 
> of the new design. Is the answer no?
-- 
Nicholas Miell <nmiell@comcast.net>

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Sun, 2004-08-29 at 12:53, Hans Reiser wrote:
> With your model, can I do:
> 
> cat filenameA/metas/permissions > filenameB/metas/permissions
> 

runat filenameA "cat permissions" | runat filenameB "cat > permissions"

> find / -exec cat {}/permissions \; | grep 4777 | wc -l

find / -exec runat {} "cat permissions" \; | grep 4777 | wc -l

Although, whether or not the kernel exposes file permissions as an
attribute named permissions is up to the filesystem. (And the wrong
thing to do, in my opinion, but that's irrelevant.)

> If yes, then we are talking past each other somehow rather than 
> disagreeing. If metafiles can be opened with both open and openat() in 
> your model, then we are discussing some small detail.

It can be opened by both, but in order to do it via open(2), you need to
fchdir(2) to the attribute directory as returned by openat(2) and then
use a relative pathname.

> I think the answer is no though, in which case you are missing the point 
> of the new design. Is the answer no?
-- 
Nicholas Miell <nmiell@comcast.net>

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Nicholas" == Nicholas Miell <nmiell@comcast.net> writes:

Nicholas> On Sun, 2004-08-29 at 12:53, Hans Reiser wrote:
>> With your model, can I do:
>> 
>> cat filenameA/metas/permissions > filenameB/metas/permissions
>> 

Nicholas> runat filenameA "cat permissions" | runat filenameB "cat > permissions"

The world is more than the commandline.  Say I have an application icon
stored in xterm/icon.png.  Can I just open xterm/icon.png in Gimp
without having to open up a new instance of Gimp, and just use the old
instance that I was using for other images?  Or Can I use emacs to edit
xterm/description.txt without having to open up a new instance of emacs?
Will scripts be able to use metafiles and normal files at the same time?

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Nicholas" == Nicholas Miell <nmiell@comcast.net> writes:

Nicholas> On Sun, 2004-08-29 at 12:53, Hans Reiser wrote:
>> With your model, can I do:
>> 
>> cat filenameA/metas/permissions > filenameB/metas/permissions
>> 

Nicholas> runat filenameA "cat permissions" | runat filenameB "cat > permissions"

The world is more than the commandline.  Say I have an application icon
stored in xterm/icon.png.  Can I just open xterm/icon.png in Gimp
without having to open up a new instance of Gimp, and just use the old
instance that I was using for other images?  Or Can I use emacs to edit
xterm/description.txt without having to open up a new instance of emacs?
Will scripts be able to use metafiles and normal files at the same time?

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004, Hans Reiser wrote:
>
> The difference is that cat uses open() not openat().

Hans. Step back a bit. Stop this idiocy about thinking that this is all 
about you, and all about your reiser4 "metas" etc.

This is about VFS interfaces, and about being generic and _useful_.

Quite frankly, there is absolutely _zero_ point to make attributes just 
because reiser4 can implement them. Zero. Nada. Zilch. It just won't 
matter, because people are NOT GOING TO SWITCH TO REISER4 IN MASS.

So in order to make this useful, it has to be:
 - stable. This means that if Al Viro asks about locking and aliasing
   issues, you don't ignore it, you ask "how high?"
 - portable. Preferably across operating systems, but at the very _least_ 
   across filesystems. See the NFSv4 thing, for example.
 - useful to the people who would actually use it. Right now that's samba, 
   and not a whole lot else.  For everybody else, the pain of not having 
   _all_ major filesystems support it is probably so insurmountable that
   it's just not going to happen.

Do you get it? If you can't get your mind out of your little current 
reiser4 issues, don't even bother talking about it.

Look through the thread on "possible design issues for hybrids" for me
actually talking with Al on how to handle problem #1. That's one important
thing, and in fact, that one is important whether we expose it to a normal
"open()" call or not, since all the same aliasing issues exist
_regardless_ of interfaces.

"openat()" happens to be _one_ solution to #2 and #3. You haven't given 
any constructive input to _any_ of the problems, and you keep on raising 
totally irrelevant issues. Face the music, Hans. If you can't solve those 
three problems, whatever you do simply DOES NOT MATTER.

In particular, in the two last emails to you, I tried to show that
"openat()" is _independent_ of streams/attributes. That it has semantics
that have absolutely _nothing_ to do with metas. And in BOTH CASES you
could not wrap your brain around that fact, and kept on harping about
meta's.

There are bigger issues here. I'm perfectly happy to _also_ expose the
streams as regular filenames, but I've tried to explain that that doesn't
work for directories, for example, so YOU CANNOT -JUST- talk about the
filename-only thing. You do need to have a different namespace.  Full
stop. What's so hard to understand about that?

So stop ignoring the issues. Stop ignoring the fact that not everything 
revolves around how you happened to implement something. Get with the 
program. 

		Linus

Re: silent semantic changes with reiser4

[Alexander Lyamin]

Sat, Aug 28, 2004 at 10:01:23PM -0700, Linus Torvalds wrote:
> 
> 
> On Sat, 28 Aug 2004, Hans Reiser wrote:
> >
> > I object to openat().....
> 
> Sound slike you object to O_XATTRS, not openat() itself.

xattr deserve to be objected. 
quickiehackie design practice.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: silent semantic changes with reiser4

[viro]

On Sat, Aug 28, 2004 at 10:01:23PM -0700, Linus Torvalds wrote:
> 
> 
> On Sat, 28 Aug 2004, Hans Reiser wrote:
> >
> > I object to openat().....
> 
> Sound slike you object to O_XATTRS, not openat() itself.
> 
> Realize that openat() works independently of any special streams, it's
> fundamentally a "look up name starting from this file" (rather than
> "starting from root" or "starting from cwd").

Linus, openat() is trivial to implement in userland *IF* we are talking
about stuff already in namespace.  Proof:

int openat(int fd, char *path, int flags, mode_t mode)
{
	int len = snprintf(NULL, 0, "/proc/self/%d/%s", fd, path);
	char name[len + 1];
	sprintf(name, "/proc/self/%d/%s", fd, path);
	return open(name, flags, mode);
}

7 lines.  Replace VLA with alloca() or malloc() in pre-C99 environment.

In other words, there's nothing magical about syscall itself - compatibility
with Solaris is not an issue here.  And basic functionality (i.e. what you've
described above) can be trivially achieved with no extra primitives at all.

Now, if you want to use that puppy as explicit indicator of magic steps taken,
fine, but that again says nothing about the primitives we want to express the
magic steps in question - reproducing openat() as one of the libc functions
won't be hard anyway.

So I have to agree with Hans on that one - I believe that as a fundamental
primitive it's wrong choice.  Convenient to have in libc - sure, why not?

Re: silent semantic changes with reiser4

[John Stoffel]

Hans> My reason is that the things that distinguish between objects
Hans> should be the names, not the choice of system call.  The reason
Hans> for this is that it improves closure and namespace unification
Hans> to do so, because it allows all the objects to be accessed
Hans> within the same namespace.

So who decides what the names mean in a specific context?  How do we
get consistency across applications?  In the Apple and MicroSoft
cases, it's a single company which sets the standards for the names of
the streams and what they mean.  I have no knowledge of how Sun is
doing this.  I don't see us setting up a single naming authority and
enforcement mechanism.  

I went and read the Pike paper on namespaces which Hans quotes much
earlier in this thread, and I think it isn't applicable to this
thread.  In the paper Pike compares the MVS, VMS, DOS and Unix name
spaces for disk layouts.  It's an interesting paper, though a bit
dated since it's 20 years old.  The concepts are good, but the
examples aren't any more.  How many Unix systems today have
/n/faces/... to link email addresses to digitized images of people
faces? 

The big point of this paper though was to make the syntax simple, and
the semantics clear and unambiguous.  For example, they point out how
the "Cedar" file system added in VMS style version numbers to
filenames with the syntax "<name>!<version>" and they pointedly ask
what does "/usr/rob!3/bin/cat-v mean?  Which I think just points out
that the syntax of streams (metas) needs be carefully thought out, and
how that syntax implies various semantics.  

Lots of people keep asking for ACLs, but time and time again people
point out that while the Unix permission model isn't as flexible, it
is _simpler_ and easier to keep track of.  

As an example of complexity, just keeping track of our /etc/sudoers
file with all the users, hosts and commands that can be defined is a
huge hassle.  You pretty much need to have a 'lint' style program to
do the consistency checking.  

Hans> Yes, it can be useful to allow a namespace to exclude some
Hans> objects, but that exclusion should not be mandated.  

Conversely, should a namespace *require* that objects always be there?
What happens when I mount an NFS filesystem onto a Resierfs4 tree?
How does the user/apps determine when the semantics of the underlying
namespace have changed?  

Hans> If you want to exclude, you should cd or chroot to
Hans> /proc/nopseudos and find a view of the filesystem that excludes
Hans> metas, or mount with -nopseuodos.

Hans> Do you see why I say this?  I can say a lot more about the
Hans> damage of fragmenting namespaces into multiple apis.... Why look
Hans> at xattrs....;-)

I think one of the advantages of the xattrs is that the core namespace
doesn't change, nor do the semantics.  They are layered on top and can
be explicitly looked at.  

A real life problem of namespaces is Network Appliance Filers.  If you
have a filer with both NFS and CIFS exports, it will nicely manage the
CIFS ACLs in harmony (for some level you determine) with the NFS and
Unix permissions.  But to backup that data, and to store that
information, you need to use either their 'dump' tool or NDMP.  If you
NFS mount the filesystem to your backup host and then use a simple tar
to back it up, you lose the information since that window you use to
look into the namespace is restricted.  

It's a gotcha.  Until we can (or maybe computing in general) come up
with a new namespace paradym which we can all agree to, this is going
to be a big issue.  

John

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "John" == John Stoffel <stoffel@lucent.com> writes:

[...]

John> I went and read the Pike paper on namespaces which Hans quotes
John> much earlier in this thread, and I think it isn't applicable to
John> this thread.  In the paper Pike compares the MVS, VMS, DOS and
John> Unix name spaces for disk layouts.  It's an interesting paper,
John> though a bit dated since it's 20 years old.  The concepts are
John> good, but the examples aren't any more.  How many Unix systems
John> today have /n/faces/... to link email addresses to digitized
John> images of people faces?

IIRC, gdm uses some directory in /usr/share to match usernames to
pictures (along with ~<username>/<somefile>).  But I can't find the
directory nor remember exactly what the name is.

Then there's the gconf database, which uses the filesystem to store its
hierarchical structure.  Or the .cddb/ database that uses filenames to
match the discid to the data.  Or on Debian systems, there's
/var/lib/dpkg/info database that attaches (post|pre)(inst|rm) scripts,
file lists, etc. to packages.  (Although I would have organized that
directory differently.)

And then there's the fact that there would be more examples if
filesystems were more expressive.  Like my LDAP database that sometimes
gets corrupted when the power gets pulled.  It's stored in a number of
opaque files, so I don't know how it gets corrupted, and I can't use
the normal fsck to check it.  Wouldn't it be nice if the database could
be stored as normal files, using the filesystem hierarchy to mimic the
LDAP hierarchy?  Then if something gets corrupted, fsck will fix it, or
I can easily see which record contains the garbage.  The problem is
that the filesystem can't do queries, so I'll have to wait until
Reiser6.

John> The big point of this paper though was to make the syntax simple,
John> and the semantics clear and unambiguous.  For example, they point
John> out how the "Cedar" file system added in VMS style version numbers
John> to filenames with the syntax "<name>!<version>" and they pointedly
John> ask what does "/usr/rob!3/bin/cat-v mean?  Which I think just
John> points out that the syntax of streams (metas) needs be carefully
John> thought out, and how that syntax implies various semantics.

The problem isn't just with things being confusing.  It's also about
making things accessible through a unified interface.  If we take the
attributes example, if we need to use a special xattr syscall, then I
can't use GIMP to edit icon attribute of xterm, or use emacs/vim/nano/ed
to edit the comment attribute of sound_of_silence.ogg (or the lyrics
attribute of sound_of_silence.ogg).  I need to fire up some stupid
program to do batch edits of the attributes of my ogg collection because
the FreeDB database had the wrong CD title info for the CD that I just
ripped and I didn't catch the error before I ripped it.  If attributes
were exposed through the regular filesystem interface (possibly through
some userspace helper to do the actual OGG parsing), I could just type
a couple of lines on the command line, and be done with it.

John> Lots of people keep asking for ACLs, but time and time again
John> people point out that while the Unix permission model isn't as
John> flexible, it is _simpler_ and easier to keep track of.

Not if you need to add a whole bunch of groups just to do the things
you want to do.  I personally haven't done that.  Why?  Because I would
rather deal with suboptimal permissions than having to add all those
groups.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Hubert" == Hubert Chan <hubert@uhoreg.ca> writes:
>>>>> "John" == John Stoffel <stoffel@lucent.com> writes:

[...]

John> The big point of this paper though was to make the syntax simple,
John> and the semantics clear and unambiguous.  For example, they point
John> out how the "Cedar" file system added in VMS style version numbers
John> to filenames with the syntax "<name>!<version>" and they pointedly
John> ask what does "/usr/rob!3/bin/cat-v mean?  Which I think just
John> points out that the syntax of streams (metas) needs be carefully
John> thought out, and how that syntax implies various semantics.

Hubert> The problem isn't just with things being confusing.  It's also
Hubert> about making things accessible through a unified interface. ...

By the way, "The Hideous Name" mentions this just briefly (in the
"Properties of Name Spaces" section).  A better paper, IMHO, about this
is "The Use of Name Spaces in Plan 9" (Rob Pike, Dave Presotto, Ken
Thompson, Howard Trickey, Phil Winterbottom; Operating Systems Review,
Num. 2, Vol 27, April 1993 pp. 72-76)

http://www.cs.bell-labs.com/sys/doc/names.pdf
or
http://www.cs.bell-labs.com/sys/doc/names.html

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Sat, Aug 28, 2004 at 11:09:38AM -0700, Linus Torvalds wrote:
> 
> If you look at the Solaris interface, the _nice_ part about "openat()" is 
> that you can do something like
> 
> 	file = open(filename, O_RDONLY);
> 	if (file < 0)
> 		return -ENOENT;
> 	icon = openat(file, "icon", O_RDONLY | O_XATTR);
> 	if (icon < 0)
> 		icon = default_icon_file;
> 	..
> 
> and it will work regardless of whether "filename" is a directory or a 
> regular file, if I've understood correctly.
> 
> Now, I think that makes sense for several reasons:
>  - single case
>  - race-free (think "stat()" vs "fstat()" races).
>  - I think we want to do "openat()" regardless of whether we ever 
>    support extended attributes or not ("openat()" is nice for doing 
>    "namei()" in user space even in the absense of any attributes or 
>    named streams).
> 
> So what we can do is
>  - implement openat() regardless, and expect to do the Solaris thing for 
>    it if we ever do streams.

Hurrah ! Definately the best way to go as far as Samba is concerned.
Makes our portability easier. Now the NT interface allows an open on
a named data stream without an open on the containing file (ie.
you can do CreateFile("filename::STREAMNAME",....) without doing
CreateFile("filename",..." first, but I personally think those
semantics are a nasty hack. It makes much more sense (IMHO) to
ensure that you already have an open fd on a file before allowing
access to the named streams inside, Windows be damned. If we have
some semantic issues mapping from the Windows semantics to Solaris/Linux
semantics then so be it, it's a Samba problem so to speak :-).

Thanks,

	Jeremy.

Re: silent semantic changes with reiser4

[Helge Hafting]

Linus Torvalds wrote:

>On Sat, 28 Aug 2004, Helge Hafting wrote:
>  
>
>>>I think that lack of distinguishing power is more serious for 
>>>directories. The more I think I think about it, the more I wonder whether 
>>>Solaris did things right - having a special operation to "cross the 
>>>boundary".
>>>
>>>I suspect Solaris did it that way because it's a hell of a lot easier to 
>>>do it like that, but regardless, it would solve the issue of real 
>>>directories having both real children _and_ the "extra streams".
>>>      
>>>
>>There are many ways of doing this. Several extra streams to a directory
>>that aren't ordinary files in the directory?
>>    
>>
>
>Well.. Yes. We already have "." and "..", which are "special extra
>streams" in a sense. However, people expect them, and know to ignore them. 
>The same wouldn't be true of new naming.
>
>  
>
>>It seems to me that we can get a lot of nice functionality in a simpler way:
>>Instead of thinking about a number of streams attached to something
>>that is either an ordinary file or directory, just say that the only
>>change will be that a directory may have a _single_ file stream in
>>addition to being a plain directory.
>>    
>>
>
>That doesn't really help us. What would the name be, and how could you 
>avoid clashes? 
>  
>
The name for the file stream and the directory would be the same,
distinguished by how they're used.  I.e. fopen("filedirname", "rw")
gets the stream, while chdir("filedirname") changes into the directory.
fopen("filedirname/substream", "rw"); opens some stream inside
the directory - as usual.

My idea is that the "extra streams" aren't special at all, they
are simply files in a directory and will therefore work with
any normal tool.  cd'ing into such a directory won't be special
either - because it is a plain directory. A collection of named streams
is usually a bunch of files in a directory - nothing new there.

Someone serving files for clients using
other os'es may attach special meaning to them, but I don't see the need
for a special meaning under linux.

So the only new thing is that the directory have a stream of
its own.  Then there must be a way to turn a ordinary
file into such a directory - people may then implement
thumbnails, summaries and so on as substreams.  Where
the substreams really are plain files in this slightly
special directory.

Perhaps mkdir with an existing filename could succeed,
creating a mixed file/directory, if the underlying fs support it.

>  
>
>>If the VFS is to be extended in order to support file-as-directory (or
>>vice versa) then hopefully it can be done in a simple way.
>>    
>>
>
>I'm pretty confident that we can extend the VFS layer to support named
>streams (see the technical discussion with Al, rather than the flames in
>this thread). I also clearly believe that it is worth it, but I'm starting
>to wonder if we should have a special open flag to make people select the
>stream.
>
>If you look at the Solaris interface, the _nice_ part about "openat()" is 
>that you can do something like
>
>	file = open(filename, O_RDONLY);
>	if (file < 0)
>		return -ENOENT;
>	icon = openat(file, "icon", O_RDONLY | O_XATTR);
>	if (icon < 0)
>		icon = default_icon_file;
>	..
>
>and it will work regardless of whether "filename" is a directory or a 
>regular file, if I've understood correctly.
>  
>
An alternative: support open("filename/icon", O_CREAT);
and have this turn "filename" into file-as-directory if it wasn't so
already.  Then it works both with files and directories.
A new flag might be needed if old programs somehow depend on the stuff
above failing for plain files.

>Now, I think that makes sense for several reasons:
> - single case
> - race-free (think "stat()" vs "fstat()" races).
> - I think we want to do "openat()" regardless of whether we ever 
>   support extended attributes or not ("openat()" is nice for doing 
>   "namei()" in user space even in the absense of any attributes or 
>   named streams).
>
>So what we can do is
> - implement openat() regardless, and expect to do the Solaris thing for 
>   it if we ever do streams.
> - _also_ support the "implied named attributes" for regular files, so 
>   that you don't have to use "openat()" to access them.
>
>Comments? Does anybody hate "openat()" for any reason (regardless of 
>attributes)? We can easily support it, we'd just need to pass in the file 
>to use as part of the "nameidata" thing or add an argument (it would also 
>possibly be cleaner if we made "fs->pwd" be a "struct file").
>
>  
>
No problem with openat()

Helge Hafting

Re: silent semantic changes with reiser4

[Helge Hafting]

Linus Torvalds wrote:

>On Sat, 28 Aug 2004, Helge Hafting wrote:
>  
>
>>>I think that lack of distinguishing power is more serious for 
>>>directories. The more I think I think about it, the more I wonder whether 
>>>Solaris did things right - having a special operation to "cross the 
>>>boundary".
>>>
>>>I suspect Solaris did it that way because it's a hell of a lot easier to 
>>>do it like that, but regardless, it would solve the issue of real 
>>>directories having both real children _and_ the "extra streams".
>>>      
>>>
>>There are many ways of doing this. Several extra streams to a directory
>>that aren't ordinary files in the directory?
>>    
>>
>
>Well.. Yes. We already have "." and "..", which are "special extra
>streams" in a sense. However, people expect them, and know to ignore them. 
>The same wouldn't be true of new naming.
>
>  
>
>>It seems to me that we can get a lot of nice functionality in a simpler way:
>>Instead of thinking about a number of streams attached to something
>>that is either an ordinary file or directory, just say that the only
>>change will be that a directory may have a _single_ file stream in
>>addition to being a plain directory.
>>    
>>
>
>That doesn't really help us. What would the name be, and how could you 
>avoid clashes? 
>  
>
The name for the file stream and the directory would be the same,
distinguished by how they're used.  I.e. fopen("filedirname", "rw")
gets the stream, while chdir("filedirname") changes into the directory.
fopen("filedirname/substream", "rw"); opens some stream inside
the directory - as usual.

My idea is that the "extra streams" aren't special at all, they
are simply files in a directory and will therefore work with
any normal tool.  cd'ing into such a directory won't be special
either - because it is a plain directory. A collection of named streams
is usually a bunch of files in a directory - nothing new there.

Someone serving files for clients using
other os'es may attach special meaning to them, but I don't see the need
for a special meaning under linux.

So the only new thing is that the directory have a stream of
its own.  Then there must be a way to turn a ordinary
file into such a directory - people may then implement
thumbnails, summaries and so on as substreams.  Where
the substreams really are plain files in this slightly
special directory.

Perhaps mkdir with an existing filename could succeed,
creating a mixed file/directory, if the underlying fs support it.

>  
>
>>If the VFS is to be extended in order to support file-as-directory (or
>>vice versa) then hopefully it can be done in a simple way.
>>    
>>
>
>I'm pretty confident that we can extend the VFS layer to support named
>streams (see the technical discussion with Al, rather than the flames in
>this thread). I also clearly believe that it is worth it, but I'm starting
>to wonder if we should have a special open flag to make people select the
>stream.
>
>If you look at the Solaris interface, the _nice_ part about "openat()" is 
>that you can do something like
>
>	file = open(filename, O_RDONLY);
>	if (file < 0)
>		return -ENOENT;
>	icon = openat(file, "icon", O_RDONLY | O_XATTR);
>	if (icon < 0)
>		icon = default_icon_file;
>	..
>
>and it will work regardless of whether "filename" is a directory or a 
>regular file, if I've understood correctly.
>  
>
An alternative: support open("filename/icon", O_CREAT);
and have this turn "filename" into file-as-directory if it wasn't so
already.  Then it works both with files and directories.
A new flag might be needed if old programs somehow depend on the stuff
above failing for plain files.

>Now, I think that makes sense for several reasons:
> - single case
> - race-free (think "stat()" vs "fstat()" races).
> - I think we want to do "openat()" regardless of whether we ever 
>   support extended attributes or not ("openat()" is nice for doing 
>   "namei()" in user space even in the absense of any attributes or 
>   named streams).
>
>So what we can do is
> - implement openat() regardless, and expect to do the Solaris thing for 
>   it if we ever do streams.
> - _also_ support the "implied named attributes" for regular files, so 
>   that you don't have to use "openat()" to access them.
>
>Comments? Does anybody hate "openat()" for any reason (regardless of 
>attributes)? We can easily support it, we'd just need to pass in the file 
>to use as part of the "nameidata" thing or add an argument (it would also 
>possibly be cleaner if we made "fs->pwd" be a "struct file").
>
>  
>
No problem with openat()

Helge Hafting

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Helge" == Helge Hafting <helge.hafting@hist.no> writes:

Helge> Linus Torvalds wrote:

[...]

>> That doesn't really help us. What would the name be, and how could you
>> avoid clashes?

Helge> The name for the file stream and the directory would be the same,
Helge> distinguished by how they're used.  I.e. fopen("filedirname",
Helge> "rw") gets the stream, while chdir("filedirname") changes into
Helge> the directory.  fopen("filedirname/substream", "rw"); opens some
Helge> stream inside the directory - as usual.

The problem is that filedirname now cannot contain a file called
"substream", or else it will clash with the name of the substream.

One solution is to put all streams into another subdirectory, much like
"metas" in Reiser4.  So to access a stream named "substream", you open
filedirname/streams/substream (replace "streams" with some carefully
chosen name that is user-friendly and highly unlikely to produce
clashes, such as "..streams", following the original "metas" proposal
which has been discussed to death on the ReiserFS list.  Or
"pseudos/streams", or one of countless variations).

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Helge" == Helge Hafting <helge.hafting@hist.no> writes:

Helge> Linus Torvalds wrote:

[...]

>> That doesn't really help us. What would the name be, and how could you
>> avoid clashes?

Helge> The name for the file stream and the directory would be the same,
Helge> distinguished by how they're used.  I.e. fopen("filedirname",
Helge> "rw") gets the stream, while chdir("filedirname") changes into
Helge> the directory.  fopen("filedirname/substream", "rw"); opens some
Helge> stream inside the directory - as usual.

The problem is that filedirname now cannot contain a file called
"substream", or else it will clash with the name of the substream.

One solution is to put all streams into another subdirectory, much like
"metas" in Reiser4.  So to access a stream named "substream", you open
filedirname/streams/substream (replace "streams" with some carefully
chosen name that is user-friendly and highly unlikely to produce
clashes, such as "..streams", following the original "metas" proposal
which has been discussed to death on the ReiserFS list.  Or
"pseudos/streams", or one of countless variations).

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Spam" == Spam  <spam@tnonline.net> writes:

> Rik van Riel wrote:
>> On Fri, 27 Aug 2004, Hans Reiser wrote:
>>> Why are you guys even considering going to any pain at all to
>>> distort semantics for the sake of backup?  tar is easy, we'll fix it
>>> and send in a patch.

>> It's not as easy as you make it out, and not just because there are a
>> few dozen backup programs that need fixing.

>> The problem is more fundamental than that.  Some of the file streams
>> proposed need to be backed up, while others are alternative
>> presentations of the file, which should not be backed up.

Spam>   No, not really. This is a user decision and should be options in
Spam> the backup software.  I don't think it is up to the kernel,
Spam> filesystem, or the OS in general to decide what information the
Spam> user want to retain or not.

Why not just define an attribute named something like "do-not-backup"?
Then whatever program that generates the thumbnail can automatically add
the do-not-backup bit, and the backup software knows to ignore it.
(Obviously, that bit should apply recursively down the subtree.)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Spam" == Spam  <spam@tnonline.net> writes:

> Rik van Riel wrote:
>> On Fri, 27 Aug 2004, Hans Reiser wrote:
>>> Why are you guys even considering going to any pain at all to
>>> distort semantics for the sake of backup?  tar is easy, we'll fix it
>>> and send in a patch.

>> It's not as easy as you make it out, and not just because there are a
>> few dozen backup programs that need fixing.

>> The problem is more fundamental than that.  Some of the file streams
>> proposed need to be backed up, while others are alternative
>> presentations of the file, which should not be backed up.

Spam>   No, not really. This is a user decision and should be options in
Spam> the backup software.  I don't think it is up to the kernel,
Spam> filesystem, or the OS in general to decide what information the
Spam> user want to retain or not.

Why not just define an attribute named something like "do-not-backup"?
Then whatever program that generates the thumbnail can automatically add
the do-not-backup bit, and the backup software knows to ignore it.
(Obviously, that bit should apply recursively down the subtree.)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Spam]

  

>>>>>> "Spam" == Spam  <spam@tnonline.net> writes:

>> Rik van Riel wrote:
>>> On Fri, 27 Aug 2004, Hans Reiser wrote:
>>>> Why are you guys even considering going to any pain at all to
>>>> distort semantics for the sake of backup?  tar is easy, we'll fix it
>>>> and send in a patch.

>>> It's not as easy as you make it out, and not just because there are a
>>> few dozen backup programs that need fixing.

>>> The problem is more fundamental than that.  Some of the file streams
>>> proposed need to be backed up, while others are alternative
>>> presentations of the file, which should not be backed up.

Spam>>   No, not really. This is a user decision and should be options in
Spam>> the backup software.  I don't think it is up to the kernel,
Spam>> filesystem, or the OS in general to decide what information the
Spam>> user want to retain or not.

> Why not just define an attribute named something like "do-not-backup"?
> Then whatever program that generates the thumbnail can automatically add
> the do-not-backup bit, and the backup software knows to ignore it.
> (Obviously, that bit should apply recursively down the subtree.)

  This  is  about  the  same  idea  as the archive flag in DOS/Windows
  environments.  It  didn't really work to well IMO. If meta files are
  definable  by  users/application then the backup system could create
  its own meta files with the specific information it needs.

  ~S

  

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Spam" == Spam  <spam@tnonline.net> writes:

> Hubert Chan wrote:
>> Why not just define an attribute named something like
>> "do-not-backup"?  Then whatever program that generates the thumbnail
>> can automatically add the do-not-backup bit, and the backup software
>> knows to ignore it.  (Obviously, that bit should apply recursively
>> down the subtree.)

Spam>   This is about the same idea as the archive flag in DOS/Windows
Spam> environments.  It didn't really work to well IMO.

Similar, but not the same.  IIRC, the purpose of the archive flag was to
mark which files had been modified since the last backup, wherein you
get into conflicts if you have several backup-like utilities.  (I'm
trying to forget about the DOS days. ;-) ) A do-not-backup attribute
would flag that a file should never be backed up.

Spam> If meta files are definable by users/application then the backup
Spam> system could create its own meta files with the specific
Spam> information it needs.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Spam" == Spam  <spam@tnonline.net> writes:

> Hubert Chan wrote:
>> Why not just define an attribute named something like
>> "do-not-backup"?  Then whatever program that generates the thumbnail
>> can automatically add the do-not-backup bit, and the backup software
>> knows to ignore it.  (Obviously, that bit should apply recursively
>> down the subtree.)

Spam>   This is about the same idea as the archive flag in DOS/Windows
Spam> environments.  It didn't really work to well IMO.

Similar, but not the same.  IIRC, the purpose of the archive flag was to
mark which files had been modified since the last backup, wherein you
get into conflicts if you have several backup-like utilities.  (I'm
trying to forget about the DOS days. ;-) ) A do-not-backup attribute
would flag that a file should never be backed up.

Spam> If meta files are definable by users/application then the backup
Spam> system could create its own meta files with the specific
Spam> information it needs.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Spam" == Spam  <spam@tnonline.net> writes:

Spam>   This is about the same idea as the archive flag in DOS/Windows
Spam> environments.  It didn't really work to well IMO. If meta files
Spam> are definable by users/application then the backup system could
Spam> create its own meta files with the specific information it needs.

True enough.  Sometimes you'll want to backup the thumbnail, sometimes
you won't.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Spam" == Spam  <spam@tnonline.net> writes:

Spam>   This is about the same idea as the archive flag in DOS/Windows
Spam> environments.  It didn't really work to well IMO. If meta files
Spam> are definable by users/application then the backup system could
Spam> create its own meta files with the specific information it needs.

True enough.  Sometimes you'll want to backup the thumbnail, sometimes
you won't.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>
>Currently I see no way to distinguish between the stuff
>that should be backed up and the stuff that shouldn't.
>
>  
>
We create filename/pseudos/backup, and that tells the archiver what to 
do.....

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>
>Currently I see no way to distinguish between the stuff
>that should be backed up and the stuff that shouldn't.
>
>  
>
We create filename/pseudos/backup, and that tells the archiver what to 
do.....

Re: silent semantic changes with reiser4

[Paul Jackson]

Hans wrote:
> We create filename/pseudos/backup, and that tells the archiver what to 
> do.....

Instead of exposing the old semantics under a new interface, why not
expose the new semantics under a new interface.

There exist plenty of programs that know the old Unix semantics.  There
don't exist many working programs that use the new semantics that you're
adding.

I raise again the example of how Windows adapted to long filenames.  Old
DOS and FAT programs, including my Unix backups of today, see a 8.3 name
space.  Only code that knows the new magic sees the long names.

If given the choice of breaking much old, existing stuff, or some new,
mostly not yet existing stuff, does not it make more sense to break what
mostly doesn't exist yet?

One possible way to do this, of no doubt many:

 * Stealing a corner of the existing filename space for
   some magic names with the new semantics.

 * A new option on open(2), hence opendir(3), that lights up
   these magic names.

 * Doing any of the classic pathname calls with such a
   new magic name exposes the new semantics - such calls
   as:
	access execve mkdir mknod mount readlink
	rename rmdir stat truncate unlink

This means essentially constructing a map between old and new,
such that changes made in either view are sane and visible
from the other view.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Alexander Lyamin]

Fri, Aug 27, 2004 at 11:08:57PM -0700, Paul Jackson wrote:
> Hans wrote:
> > We create filename/pseudos/backup, and that tells the archiver what to 
> > do.....
> 
> Instead of exposing the old semantics under a new interface, why not
> expose the new semantics under a new interface.

yeah. that could work.

> There exist plenty of programs that know the old Unix semantics.  There
> don't exist many working programs that use the new semantics that you're
> adding.
> 
> I raise again the example of how Windows adapted to long filenames.  Old
> DOS and FAT programs, including my Unix backups of today, see a 8.3 name
> space.  Only code that knows the new magic sees the long names.
> 
> If given the choice of breaking much old, existing stuff, or some new,
> mostly not yet existing stuff, does not it make more sense to break what
> mostly doesn't exist yet?
> 
> One possible way to do this, of no doubt many:
> 
>  * Stealing a corner of the existing filename space for
>    some magic names with the new semantics.
> 
>  * A new option on open(2), hence opendir(3), that lights up
>    these magic names.
> 
>  * Doing any of the classic pathname calls with such a
>    new magic name exposes the new semantics - such calls
>    as:
> 	access execve mkdir mknod mount readlink
> 	rename rmdir stat truncate unlink
> 
> This means essentially constructing a map between old and new,
> such that changes made in either view are sane and visible
> from the other view.

It would be intresting to hear comments from Hans Reiser on proposals stated  above...


-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: silent semantic changes with reiser4

[Hans Reiser]

Alexander Lyamin wrote:

>Fri, Aug 27, 2004 at 11:08:57PM -0700, Paul Jackson wrote:
>  
>
>>
>>One possible way to do this, of no doubt many:
>>
>> * Stealing a corner of the existing filename space for
>>   some magic names with the new semantics.
>>
>> * A new option on open(2), hence opendir(3), that lights up
>>   these magic names.
>>
>> * Doing any of the classic pathname calls with such a
>>   new magic name exposes the new semantics - such calls
>>   as:
>>	access execve mkdir mknod mount readlink
>>	rename rmdir stat truncate unlink
>>
>>This means essentially constructing a map between old and new,
>>such that changes made in either view are sane and visible
>>from the other view.
>>    
>>
>
>It would be intresting to hear comments from Hans Reiser on proposals stated  above...
>
>
>  
>
just use a view, and skip the options on the system calls.  if you cd to 
/nometas/your_home_directory_path you don't see the metafiles.  Why is a 
view better than a syscall flag?  Because it lets the user choose what 
he wants without recompiling to do it.  This kind of a view requires no 
coding because you can just mount the root filesystem two ways, one with 
the -nopseudos mount option, and one without it.

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 11:28:42AM -0700, Hans Reiser wrote:
> just use a view, and skip the options on the system calls.  if you cd to 
> /nometas/your_home_directory_path you don't see the metafiles.  Why is a 
> view better than a syscall flag?  Because it lets the user choose what 
> he wants without recompiling to do it.  This kind of a view requires no 
> coding because you can just mount the root filesystem two ways, one with 
> the -nopseudos mount option, and one without it.

*What*?

OK, now I want detailed explanation of the reasons why that doesn't create
cache coherency problems.

Do you have an analysis of locking in the entire thing?

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 07:36:29PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> > he wants without recompiling to do it.  This kind of a view requires no 
> > coding because you can just mount the root filesystem two ways, one with 
> > the -nopseudos mount option, and one without it.
> 
> *What*?
> 
> OK, now I want detailed explanation of the reasons why that doesn't create
> cache coherency problems.
> 
> Do you have an analysis of locking in the entire thing?

And I am very, very serious about that - we are talking about very nasty
minefield and design choices in that area have fundamental impact on the
entire layer, wherever it is located.

It's *NOT* something that you can leave until later and hope it somehow
falls into place - it can be merged in steps, but you MUST know the goal
on that level.  To rearchitect later might be possible (even though you
will a hell of a time avoiding plugins breakage), but it will be *hard*.

Re: silent semantic changes with reiser4

[Hans Reiser]

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Sun, Aug 29, 2004 at 07:36:29PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
>  
>
>>>he wants without recompiling to do it.  This kind of a view requires no 
>>>coding because you can just mount the root filesystem two ways, one with 
>>>the -nopseudos mount option, and one without it.
>>>      
>>>
>>*What*?
>>
>>OK, now I want detailed explanation of the reasons why that doesn't create
>>cache coherency problems.
>>
>>Do you have an analysis of locking in the entire thing?
>>    
>>
>
>And I am very, very serious about that - we are talking about very nasty
>minefield and design choices in that area have fundamental impact on the
>entire layer, wherever it is located.
>
>It's *NOT* something that you can leave until later and hope it somehow
>falls into place - it can be merged in steps, but you MUST know the goal
>on that level.  To rearchitect later might be possible (even though you
>will a hell of a time avoiding plugins breakage), but it will be *hard*.
>
>
>  
>
How about if you educate me on the problems you see for a bit before I 
respond? I think it might help us move into a constructive discussion.

You are more experienced with your code than I am.... but I firmly 
believe that having different views of the same filesystem in which some 
files are visible in one view but not in another is technically feasible 
for some VFS, though I'd like to learn about the problems you see for 
doing it in the current VFS.

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 01:06:41PM -0700, Hans Reiser wrote:

> How about if you educate me on the problems you see for a bit before I 
> respond? I think it might help us move into a constructive discussion.

A slew of cache coherency issues (and memory consumption on top of that).
Rigth now we have a signle dentry tree for all fs instances, no matter
how many times it and its subtrees are visible in the tree.  Which,
obviously, avoids all that crap.  As soon as we start trying to have
multiple trees over the same fs, we are in for a *lot* of fun.

The basic question is how do you propagate the changes from one tree
to another and what do you do when change to "invisible" entry happens
via a tree where it is visible.  Unfortunately, all obvious solutions
either hit the pathname resolution and hit it *hard* (both in scalability
and in price of uncontended case) or create a mess for stuff like
NFS silly-rename semantics, VFAT aliases, etc.

Saying that some unspecified data structures might work doesn't help.
obviously - any variant will have tradeoffs of its own and has to be
discussed individually.  fsdevel is there for exactly that sort of
stuff, so if you have any specific proposals you want to discuss, you
are more than welcome.

Again, right now all cache coherency issues are sidestepped by having a single
instance of dentry tree per superblock.  Each mount/binding is represented by
vfsmount and _refers_ to a (sub)tree of dentry tree of that fs - dentries
and inodes are the same (ditto for private data structures owned by filesystem,
obviously).

So we have a forest of dentry trees (one per filesystem) and forest of
vfsmount trees (one tree per namespace).  Each vfsmount corresponds to
a chunk of namespace - think of the full user-visible tree cut into
pieces by mountpoints.  For every piece we know
	a) what fs it's from (->mnt_sb)
	b) what subtree of that fs it corresponds (->mnt_root)
	c) what piece it's attached to (->mnt_parent)
	d) where in that piece it's attached (->mnt_mountpoint)
	e) where to find its siblings and children
	f) which namespace (== tree in vfsmount forest) it's from

Point in a namespace is determined by pair (vfsmount, dentry) - which
piece we are looking at and where in that piece we are.  Pathname lookups
operate on such pairs - as long as we do not cross into another piece
we just step from dentry to dentry, when we cross mountpoint towards root,
we flip to (mnt_parent, mnt_mountpoint) first, when we cross mountpoint
into mounted filesystem, we do hash lookup (lookup_mnt()) by our pair,
find vfsmount with such mnt_parent and mnt_mountpoint and step into
(vfsmount, vfsmount->mnt_root).

Lifetime of vfsmounts is controlled by a simple refcount.  Having vfsmount
attached to tree contributes +1, so does having it pinned down by lookup/
chdir/chroot/opened file.  When the last vfsmount goes away, filesystem is
shut down (basically, that's what happens on umount).  Binding simply creates
a vfsmount pointing to target and attaches it at source.

Cloning a namespace copies the vfsmount tree of parent process' namespace
and flips vfsmounts of our root and cwd into the corresponding nodes in that
copy.  Normally, fork()/clone() just increments namespace refcount instead
of cloning it, so all children share namespace with parent.

When all processes in a namespace are gone its refcount goes to zero.  At
that point we dissolve all attachments in its vfsmount tree (which drops
a reference to each vfsmount and can in turn lead to fs shutdowns).

Lazy umount does similar operation with specified subtree - it detaches
all pieces mounted anywhere in that subtree and drops reference to each.
Ones that are not busy will be shut down immediately, ones that are will
go away as soon as they stop being busy.

vfsmounts can have flags of their own - one of the pending projects is to
allow individual vfsmounts to be read-only.  Noexec, nosuid and nodev are
already per-mountpoint.

Tree of vfsmounts is protected by
	a) vfsmount spinlock (protects hash lookups, mostly)
	b) per-namespace semaphore (that one is obviously blocking)
	c) in cases when we can go from 0 to 1 vfsmounts with given dentry
as a mountpoint, we also hold ->d_inode->i_sem on the mountpoint-to-be
(that closes races between rmdir/mount, rename/mount, etc.)

Dentry tree is messier - these days we have dcache_lock and RCU stuff.
If you can help getting the documentation out of these guys, you've got
a *lot* of thanks from a lot of people.  As it is, it's RTFS country.
I can answer specific questions there, but I won't even try to produce
readable manual covering the entire thing.

Directory operations have exclusion based on ->i_sem - see
Documentation/filesystems/directory-locking for description and proof
of correctness.

For operations that can destroy an object (unlink/rmdir/overwriting rename)
we try to unhash the victim dentry first, so the filesystems that can't
handle unlinked-but-busy stuff can detect such attempt by seeing the
victim still hashed _and_ can be sure that if it's not hashed, nobody will
come and see it (lookup would have to go to filesystem code, since we
have the sucker unhashed and we have exclusion there).  If operation is
unsuccessful, we rehash the victim.

Files involved:
	fs/dcache.c
	fs/super.c
	fs/inode.c
	fs/namespace.c
	include/linux/dcache.h
	include/linux/fs.h
	include/linux/mount.h

If you need more details on any specific area (or gaps in the above) - just
ask.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> A slew of cache coherency issues (and memory consumption on top of that).
> Rigth now we have a signle dentry tree for all fs instances, no matter
> how many times it and its subtrees are visible in the tree.  Which,
> obviously, avoids all that crap.  As soon as we start trying to have
> multiple trees over the same fs, we are in for a *lot* of fun.

Al, I think you should make the argument a bit more specific, because I 
doubt a lot of people understand just what the problems are with aliased 
names. Just a few examples of the problems involved will illuminate things 
very well, I think. People who haven't been intimate with the name caches 
probably simply don't understand why you worry so much.

I'll start out with some trivial examples, just to let Hans and others get
an idea about what the issues are. I think examples of problems are often 
better ways to explain them than the abstract issues themselves.

Aliases: let's say that you have filename "a" hard-linked to filename "b", 
and you have a directory structure of streams under there. So you have

	a/file1
	a/dir1/file2
	a/dir2/file3

and (through the hard-link with "b") you have aliases of all these same 
names available as "b/file1", "b/dir1/file2" etc).

Now, imagine that you have two processes doing

	mv a/dir1 a/dir2/newdir

and

	mv b/dir2 b/dir1/newdir

at the same time. Both of them MUST NOT SUCCEED, for pretty obvious 
reasons (you'd have moved two directories within each other, and now 
neither would be accessible any more).

How do you handle locking for this situation?

Another interesting case is what happens when you have looked up and cache
the filename "a/file1" and then another process does "rm b/file1". How do
you update the _other_ cached copy, since they had two different names,
but _both_ names went away at the same time. Also again, how do you handle
locking?

The general VFS layer has a lot of rules, and avoids these problems by
simply never having aliases between two directories. If the same directory
shows up multiple times (which can happen with bind mounts), they have the
exact same dentry for the directory, it's just found through two different
vfsmount instances. That's why vfsmounts exist - they allow the same name
cache entry to show up in different places at the same time.

So when we do a bind mount, and the same directory shows up under two
different names "a" and "b", and we do a "rm b/file1", it _automatically_
disappears from "a/file1" too, simply by virtue of "a" and "b" literally
being the same dentry. No aliasing ever happens, and this makes coherency
and locking much easier (which is not to say that they are trivial, but
they are pretty damn clear in comparison to the alternatives).

What Al (and others) worries about is that the reiser4 name handling has
_none_ of these issues figured out and protected against. You can protect 
against them by taking very heavy locks (you can trivially protect against 
all races by taking one large lock around any operation), but the fact is, 
that is just not an option for high-performance name lookup. 

These aliasing/locking rules need to be global and wll-though-out. Not 
just fix the two examples above, but be shown to be safe _in_general_. 

		Linus

Re: silent semantic changes with reiser4

[Linus Torvalds]

[ Linux-kernel cc'd, because I don't think the question is stupid, and I 
  can't even fully answer the kNFSd thing other than point to it as a
  problem. ]

On Mon, 30 Aug 2004, Grzegorz Kulewski wrote:
> 
> Sorry if my qestion is stupid, but why can't we deal with (hard)links to 
> directories in (nearly) same way we deal with bind mounts (= making 
> exactly one object representing target and only referencing to it)?

On a VFS level we could, these days, I think. But realize that bind mounts
and the vfsmounts are pretty recent things.

We don't have any filesystems that support the notion, though, and we 
don't have any interfaces for the filesystem to tell us about it right 
now. The VFS layer could try to figure it out on its own from aliasing 
information, so the latter may be a non-issue, but the former is why 
nobody does it.

And even if Linux _these days_ could handle hardlinked directories, the
fact is that they would cause slightly more memory usage (due to the
vfsmounts), and that nobody else can handle such filesystems - including
older versions of Linux. So nobody would likely use the feature (not to
mention that nobody is even really asking for it ;).

And the lack of filesystem support is not theoretical. It's not easy to 
just retrofit directory hardlinks on a UNIX filesystem. The ".." entry 
actually exists on _disk_ on traditional unix filesystems, and with 
hardlinks on directories, that's a real problem. A hardlinked directory 
has multiple parents.

Also, while the VFS layer no longer cares (to it, ".." is purely virtual,
and it never uses it), the NFS export routines still do actually want to
get the on-disk parent. A filesystem that can't do that may be unable to
be exported with full semantics (ie you might get ESTALE errors after
server reboots, although you'd have to ask somebody with more kNFSd
knowledge than me on exactly why that is the case ;)

			Linus

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 03:37:16PM -0700, Linus Torvalds wrote:
> > Sorry if my qestion is stupid, but why can't we deal with (hard)links to 
> > directories in (nearly) same way we deal with bind mounts (= making 
> > exactly one object representing target and only referencing to it)?
> 
> On a VFS level we could, these days, I think. But realize that bind mounts
> and the vfsmounts are pretty recent things.
 
Bindings won't replace hardlinks.
	a) lifetime rules and keeping stuff busy
	b) who's bound on top?  Note that for real hardlinks to directories
(not just "directory on top of file" hybrids) it's a serious question
	c) for real hardlinks we would want at least rename() working

Re: silent semantic changes with reiser4

[Neil Brown]

On Sunday August 29, torvalds@osdl.org wrote:
> 
> Also, while the VFS layer no longer cares (to it, ".." is purely virtual,
> and it never uses it), the NFS export routines still do actually want to
> get the on-disk parent. A filesystem that can't do that may be unable to
> be exported with full semantics (ie you might get ESTALE errors after
> server reboots, although you'd have to ask somebody with more kNFSd
> knowledge than me on exactly why that is the case ;)
> 

The VFS requires all directories to have full paths from the
filesystem root in the dcache.  This is needed for loop detection when
renaming directories.

suppose and NFS client: (UPPERCASE words refer to file handles)
 Says "lookup ROOT 'a'" and gets A  (file handle for /a)
 Says "lookup A 'b'" and gets B     (file handle for /a/b)
 waits for /a and /a/b to be flushed from the dcache
 Says "RENAME ROOT 'a' to B 'z'"    (i.e. asks for /a to be moved to /a/b/c)

how can the VFS detect that that is not allowed?  It needs to know
that B is a subdirectory of /a.  The complete path cannot fit in
filehandle, so to needs to be computed by repeated lookup of "..",
which must be on disk.


Even without that, storing ".." on disk makes lots of sense for
fsck-like tools.

NeilBrown

Re: silent semantic changes with reiser4

[Stephan von Krawczynski]

On Sun, 29 Aug 2004 15:37:16 -0700 (PDT)
Linus Torvalds <torvalds@osdl.org> wrote:

> [...]
> And even if Linux _these days_ could handle hardlinked directories, the
> fact is that they would cause slightly more memory usage (due to the
> vfsmounts), and that nobody else can handle such filesystems - including
> older versions of Linux. So nobody would likely use the feature (not to
> mention that nobody is even really asking for it ;).

Huh? Me about a year ago ;-)
Been in fact pretty much boo'd for it :-)

I therefore declare as this years hot issue:
How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being
available for years I guess, but with 2.6 supporting lots of GIDs becoming very
actual...
:-)))
Sorry for being a bit off-topic but shouldn't we first solve the obvious
(broken) and simple issues around fs?

Regards,
Stephan

Re: silent semantic changes with reiser4

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 12:51:56PM +0200, Stephan von Krawczynski wrote:
> 
> I therefore declare as this years hot issue:
> How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being
                       ^^
The limit for NFS is 16.


> available for years I guess, but with 2.6 supporting lots of GIDs becoming very
> actual...

thank you for reminding me that I still need to port it to 2.6. The patch came
into existence around 2.2.17 IIRC.

-- 
Frank

Re: silent semantic changes with reiser4

[Stephan von Krawczynski]

On Thu, 2 Sep 2004 13:26:23 +0200
Frank van Maarseveen <frankvm@xs4all.nl> wrote:

> On Thu, Sep 02, 2004 at 12:51:56PM +0200, Stephan von Krawczynski wrote:
> > 
> > I therefore declare as this years hot issue:
> > How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being
>                        ^^
> The limit for NFS is 16.

Yes, of course. Sorry for that exaggeration ;-)

> > available for years I guess, but with 2.6 supporting lots of GIDs becoming
> > very actual...
> 
> thank you for reminding me that I still need to port it to 2.6. The patch
> came into existence around 2.2.17 IIRC.

Oh it will be very welcome once your done, the only way to circumvent the
situation currently is to give world-rights, which is of course _bad_.

Thanks in advance for this patch,
Stephan

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds:
> The general VFS layer has a lot of rules, and avoids these problems by
> simply never having aliases between two directories. If the same directory
> shows up multiple times (which can happen with bind mounts), they have the
> exact same dentry for the directory, it's just found through two different
> vfsmount instances. That's why vfsmounts exist - they allow the same name
> cache entry to show up in different places at the same time.

So could you explain what is stopping us from reducing the whole problem
to the bind mount problem? IOW have "a/" be a directory that acts as if
it is dynamically bind mounted on top of the file "a".


Is it just the fantasy of supporting hard-links across "stream
boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
nobody wants to have to add cyclic graph detection to their filesystems
anyway. 8-)

What other issues would need to be addressed?

Cheers,
  Trond

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds:
> The general VFS layer has a lot of rules, and avoids these problems by
> simply never having aliases between two directories. If the same directory
> shows up multiple times (which can happen with bind mounts), they have the
> exact same dentry for the directory, it's just found through two different
> vfsmount instances. That's why vfsmounts exist - they allow the same name
> cache entry to show up in different places at the same time.

So could you explain what is stopping us from reducing the whole problem
to the bind mount problem? IOW have "a/" be a directory that acts as if
it is dynamically bind mounted on top of the file "a".


Is it just the fantasy of supporting hard-links across "stream
boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
nobody wants to have to add cyclic graph detection to their filesystems
anyway. 8-)

What other issues would need to be addressed?

Cheers,
  Trond
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 07:17:10PM -0400, Trond Myklebust wrote:
> På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds:
> > The general VFS layer has a lot of rules, and avoids these problems by
> > simply never having aliases between two directories. If the same directory
> > shows up multiple times (which can happen with bind mounts), they have the
> > exact same dentry for the directory, it's just found through two different
> > vfsmount instances. That's why vfsmounts exist - they allow the same name
> > cache entry to show up in different places at the same time.
> 
> So could you explain what is stopping us from reducing the whole problem
> to the bind mount problem? IOW have "a/" be a directory that acts as if
> it is dynamically bind mounted on top of the file "a".
> 
> 
> Is it just the fantasy of supporting hard-links across "stream
> boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
> nobody wants to have to add cyclic graph detection to their filesystems
> anyway. 8-)
> 
> What other issues would need to be addressed?

Go upthread and read.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004, Trond Myklebust wrote:
> 
> So could you explain what is stopping us from reducing the whole problem
> to the bind mount problem? IOW have "a/" be a directory that acts as if
> it is dynamically bind mounted on top of the file "a".

Hey, I suggested people do exactly that. See the other thread between me
and Al Viro on exact implementation issues - even some code snippets ;)

The problem really ends up being directories with attributes (where we
can't just overmount the existing directory). That's where "openat()" 
helps us. 

(The other problem is the purely _practical_ problem of reiser4 going
behind the VFS layer, and thus _not_ getting the aliasing and locking
right, but that's an implementation issue in my book, and nothing really
fundamental).

> Is it just the fantasy of supporting hard-links across "stream
> boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
> nobody wants to have to add cyclic graph detection to their filesystems
> anyway. 8-)

It's easy enough to do the graph detection at the VFS layer, exactly 
because of the density of the dentry graph. 

(Of course, nfs-exporting a filesystem breaks that density, thanks to the
"lookup by fh" stuff, so we might not allow it for an NFS client). 

I suspect most filesystems wouldn't allow hard-links across "stream 
boundaries" in the first place. I _suspect_ that most stream 
implementations would bind the attributes more tightly than that to the 
file that owns them. reiser4 migth be the only one that might ever support 
something like that.

> What other issues would need to be addressed?

 - Whether we want it in the first place
 - whether we need the "separate namespace" thing that O_XATTR and
   openat() brings us (they certainly solve the problem, but maybe it is
   solvable another way too)
 - how to actually test this out in practice (ie getting reiser4 to do the
   proper thing wrt the VFS layer, but preferably _also_ having another
   filesystem like NFSv4 or cifs that actually uses this and shows what
   the problems are).
 - whether it makes any sense at all unless we also make at least a few 
   other filesystems support it, so that people start using it as an 
   "expected feature" rather than a "works only on a couple of machines".

And probably others.

			Linus

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 19:54, skreiv Linus Torvalds:

> The problem really ends up being directories with attributes (where we
> can't just overmount the existing directory). That's where "openat()" 
> helps us. 

Well, yes there has to be a distinction between a true bind mount which
actually covers the file or directory, and something like the stream
"bind mount" which doesn't.

The stream "bind mount" is just there to allow you to root the
attributes in a single tree. It can be made functionally entirely
equivalent to the openat(), but uses pathname semantics (e.g., "//") to
denote the attribute fork instead of an extra function call.

> > Is it just the fantasy of supporting hard-links across "stream
> > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
> > nobody wants to have to add cyclic graph detection to their filesystems
> > anyway. 8-)
> 
> It's easy enough to do the graph detection at the VFS layer, exactly 
> because of the density of the dentry graph. 

Don't you end up having to lock the entire paths b/c/d and a/e/f in
order to prevent "ln a b/c/d/a; ln b a/e/f/b"?

> > What other issues would need to be addressed?

>  - how to actually test this out in practice (ie getting reiser4 to do the
>    proper thing wrt the VFS layer, but preferably _also_ having another
>    filesystem like NFSv4 or cifs that actually uses this and shows what
>    the problems are).

As I said, NFSv4 can be made ready pretty quickly: Bruce is already
finishing up the xattr implementation.

>  - whether it makes any sense at all unless we also make at least a few 
>    other filesystems support it, so that people start using it as an 
>    "expected feature" rather than a "works only on a couple of machines".

NTFS? ;-)

Cheers,
  Trond

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 19:54, skreiv Linus Torvalds:

> The problem really ends up being directories with attributes (where we
> can't just overmount the existing directory). That's where "openat()" 
> helps us. 

Well, yes there has to be a distinction between a true bind mount which
actually covers the file or directory, and something like the stream
"bind mount" which doesn't.

The stream "bind mount" is just there to allow you to root the
attributes in a single tree. It can be made functionally entirely
equivalent to the openat(), but uses pathname semantics (e.g., "//") to
denote the attribute fork instead of an extra function call.

> > Is it just the fantasy of supporting hard-links across "stream
> > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
> > nobody wants to have to add cyclic graph detection to their filesystems
> > anyway. 8-)
> 
> It's easy enough to do the graph detection at the VFS layer, exactly 
> because of the density of the dentry graph. 

Don't you end up having to lock the entire paths b/c/d and a/e/f in
order to prevent "ln a b/c/d/a; ln b a/e/f/b"?

> > What other issues would need to be addressed?

>  - how to actually test this out in practice (ie getting reiser4 to do the
>    proper thing wrt the VFS layer, but preferably _also_ having another
>    filesystem like NFSv4 or cifs that actually uses this and shows what
>    the problems are).

As I said, NFSv4 can be made ready pretty quickly: Bruce is already
finishing up the xattr implementation.

>  - whether it makes any sense at all unless we also make at least a few 
>    other filesystems support it, so that people start using it as an 
>    "expected feature" rather than a "works only on a couple of machines".

NTFS? ;-)

Cheers,
  Trond

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004, Trond Myklebust wrote:
> 
> Well, yes there has to be a distinction between a true bind mount which
> actually covers the file or directory, and something like the stream
> "bind mount" which doesn't.
> 
> The stream "bind mount" is just there to allow you to root the
> attributes in a single tree. It can be made functionally entirely
> equivalent to the openat(), but uses pathname semantics (e.g., "//") to
> denote the attribute fork instead of an extra function call.

Using '//' would be nice, but would break real apps. If I remember
correctly, POSIX specifies that '//' can be special at the _beginning_ of
a path, but in the middle, it has to act like a single '/'.

And that's not just theory - it's quite common for programs to just 
concatenate a directory name (which may or may not end with a slash) with 
another path-name that starts with a slash. So you _will_ see existing 
scripts and programs using things like "/usr/include//sys/type.h", and 
they'd break if "//" would switch from "regular namespace" to "attribute 
namespace".

So I don't see any way to extend pathname semantics to distinguish between 
"directory contents" and "directory attribute stream". 

> > It's easy enough to do the graph detection at the VFS layer, exactly 
> > because of the density of the dentry graph. 
> 
> Don't you end up having to lock the entire paths b/c/d and a/e/f in
> order to prevent "ln a b/c/d/a; ln b a/e/f/b"?

That's not the problem - since it's in memory, we can just get the dcache 
lock, and do it locked for t least local filesystems.

However, being prodded by Andries, I think I'm wrong _anyway_. Since the
dcache is only "dense" down one path to the root, and doesn't contain all
the alternate ways of getting to a particular directory, I came to the
conclusion that the VFS layer can't actually do cyclic detection after
all...

So together with the fact that nobody really _wants_ hardlinks to 
directories, I think the right answer is "no". It's not a problem as long 
as the attributes streams are always tied to the file/directory they are 
attributes of - then the "directory link" is really just a file link, and 
can't cause any cycles.

> >  - how to actually test this out in practice (ie getting reiser4 to do the
> >    proper thing wrt the VFS layer, but preferably _also_ having another
> >    filesystem like NFSv4 or cifs that actually uses this and shows what
> >    the problems are).
> 
> As I said, NFSv4 can be made ready pretty quickly: Bruce is already
> finishing up the xattr implementation.

Do we have any servers that implement it? I think NFSv4 might be a good 
test-case if so.

> >  - whether it makes any sense at all unless we also make at least a few 
> >    other filesystems support it, so that people start using it as an 
> >    "expected feature" rather than a "works only on a couple of machines".
> 
> NTFS? ;-)

Hey, I see the smiley, but I'd still like to point out that not many 
people use it under Linux, and while I think writing to it might be stable 
these days, I don't believe named streams are necessarily going to 
materialize all that quickly..

		Linus

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote:
> And that's not just theory - it's quite common for programs to just 
> concatenate a directory name (which may or may not end with a slash) with 
> another path-name that starts with a slash. So you _will_ see existing 
> scripts and programs using things like "/usr/include//sys/type.h", and 
> they'd break if "//" would switch from "regular namespace" to "attribute 
> namespace".
> 
> So I don't see any way to extend pathname semantics to distinguish between 
> "directory contents" and "directory attribute stream". 

I do, actually.  There might be a way and it is kinda-sorta similar to
your openat() variant, but lives in normal namespace.  No, I'm not too
fond of that, but since we are discussing weird variants anyway...

	a) associated directory tree of object is not automounted on top
of it.  Instead of that, we always do detached vfsmount (and do it on demand -
see below)
	b) we have a bunch of pseudo-symlinks in /proc/<pid>/fd/ - same
kind as what we already have there, but instead of (file->f_vfsmount,
file->f_dentry) they lead to associated vfsmount (allocated if needed).
Once we get a reference to such guy, we can
	* do further lookups
	* chdir there and poke around
	* hell, we can even bind it someplace (that will require slight change
in attach_mnt() logics, but it's not hard) and get it permanently mounted

Since it's not attached anywhere, normal GC logics works just fine.  And
yes, they are usable from scripts, etc. -
	exec 42<foo/bar/baz
	cat /proc/self/fd/#42/whatever/crap/you/want
and enjoy.

> However, being prodded by Andries, I think I'm wrong _anyway_. Since the
> dcache is only "dense" down one path to the root, and doesn't contain all
> the alternate ways of getting to a particular directory, I came to the
> conclusion that the VFS layer can't actually do cyclic detection after
> all...

<blinks>
<rereads a bunch of earlier postings>

Oh, _that_'s what you meant...  No, we definitely have no chance in hell
to catch loops, dcache or not.  It costs too much - we need to examine
a *lot* of nodes to do that and all of them would have to be read at some
point.  We either need entire fs tree locked in core or we might have to
reread it on every rename().  The former will kill us on memory use, the
latter - on amount of IO.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Mon, 30 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > 
> > So I don't see any way to extend pathname semantics to distinguish between 
> > "directory contents" and "directory attribute stream". 
> 
> I do, actually.  There might be a way and it is kinda-sorta similar to
> your openat() variant, but lives in normal namespace.  No, I'm not too
> fond of that, but since we are discussing weird variants anyway...
> 
> 	a) associated directory tree of object is not automounted on top
> of it.  Instead of that, we always do detached vfsmount (and do it on demand -
> see below)
> 	b) we have a bunch of pseudo-symlinks in /proc/<pid>/fd/ - same
> kind as what we already have there, but instead of (file->f_vfsmount,
> file->f_dentry) they lead to associated vfsmount (allocated if needed).
> Once we get a reference to such guy, we can
> 	* do further lookups
> 	* chdir there and poke around
> 	* hell, we can even bind it someplace (that will require slight change
> in attach_mnt() logics, but it's not hard) and get it permanently mounted

Well, the above _is_ the same as "openat()", really. It's just using a
filesystem starting point to emulate a new system call. Same thing
conceptually. You could do pretty much any system call as a filesystem
action if you wanted to ;)

I don't disagree with doing so - as a way to expose the new system call to
scripts. But I don't think you're being entirely intellectually honest if
you think this suddendly makes it be "one namespace". It's still a
secondary namespace rooted in an entry in the normal ones - exactly like
"openat()". 

For a non-script, a native "openat()" interface would be more efficient
and less confusing, and conceptually no different from yours.  No reason
we couldn't have both, since they are 100% equivalent and would share the
same code anyway...

		Linus

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 08:55:02PM -0700, Linus Torvalds wrote:
> Well, the above _is_ the same as "openat()", really. It's just using a
> filesystem starting point to emulate a new system call. Same thing
> conceptually. You could do pretty much any system call as a filesystem
> action if you wanted to ;)

Umm...  Yes and no - open() is not the only thing you can do there.  You
can emulate some of the other stuff with open() + fchdir() + syscall, but...
Note that it also gives you access to other tasks' files (provided that
tasks are yours, so you can get to their /proc/.../fd).

> I don't disagree with doing so - as a way to expose the new system call to
> scripts. But I don't think you're being entirely intellectually honest if
> you think this suddendly makes it be "one namespace". It's still a
> secondary namespace rooted in an entry in the normal ones - exactly like
> "openat()". 

Well - it *does* expose these objects to all normal syscalls.  E.g. you
can unlink() a component in there without
	fd2 = open(".", ...);
	fd3 = openat(fd, ".", ....)
	fchdir(fd3);
	unlink("blah");
	fchdir(fd2);
and similar horrors (now have fun adding locking for multithreaded process,
etc.).

In that sense it is, indeed, the same namespace.

> For a non-script, a native "openat()" interface would be more efficient
> and less confusing, and conceptually no different from yours.  No reason
> we couldn't have both, since they are 100% equivalent and would share the
> same code anyway...

I'm not sure that openat() is the right interface for e.g. fileservers.
And no, I'm not saying that above is suitable for them - IMO neither variant
is what they need.  The thing is, fileserver almost certainly wants to
create and remove objects.  And you either end up with new syscalls for
doing *that* relative to opened fd or you do tons of fchdir(), which
makes benefits of openat() dubious in the best case.

Arguments about O_NOFOLLOW on the intermediate stages are bullshit, IMNSHO -
if they want to make some parts of tree inaccessible, they should simply
mkdir /tmp/FOAD; chmod 0 /tmp/FOAD; mount --bind /tmp/FOAD <blocked path>
in the namespace their daemon is running in.  And forget all that crap
about filtering pathnames and blocking symlinks on intermediate stages
(the latter is obviously worthless without the former since one can simply
substitute the symlink body in the pathname).

Re: silent semantic changes with reiser4

[viro]

On Mon, Aug 30, 2004 at 05:46:37AM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> Arguments about O_NOFOLLOW on the intermediate stages are bullshit, IMNSHO -
> if they want to make some parts of tree inaccessible, they should simply
> mkdir /tmp/FOAD; chmod 0 /tmp/FOAD; mount --bind /tmp/FOAD <blocked path>
> in the namespace their daemon is running in.  And forget all that crap
> about filtering pathnames and blocking symlinks on intermediate stages
> (the latter is obviously worthless without the former since one can simply
> substitute the symlink body in the pathname).

Ehh...  After looking at that for a while...  No, it's not that simple
and removing the stuff that way won't do what these guys want, at least
not without something else.  Frankly, what I've seen worries me a lot -
it looks like there is a missing primitive here that would be saner
than this sort of filtering.

It appears that most of this stuff would be covered by a fast way to tell
if the resulting object belongs to given subtree.  That could be arranged
(not without some changes, but doable), but I'm not sure that it's enough
to cover the stuff they are really trying to do.  It does look like an
interesting problem and current solutions certainly suck.  And I very
much doubt that "do a lookup if it doesn't run into anything that could
be too tricky for our pathname-based checks, otherwise let's do it step-by-step
from userland" is the right approach here.

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 22:31, skreiv Linus Torvalds:

> So together with the fact that nobody really _wants_ hardlinks to 
> directories, I think the right answer is "no". It's not a problem as
> long as the attributes streams are always tied to the file/directory
> they are attributes of - then the "directory link" is really just a
> file link, and can't cause any cycles.

Right.

> Do we have any servers that implement it? I think NFSv4 might be a good 
> test-case if so.

I believe Solaris Express already has an implementation, so I should
have at least one server implementation available in the basement.
So probably does Hummingbird, since their NFSv4 server runs on Windows.

CITI is working on the Linux code, but I'm not sure how far they've come
on the server side. I'll ask Bruce in the morning...

Cheers,
  Trond

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 22:31, skreiv Linus Torvalds:

> So together with the fact that nobody really _wants_ hardlinks to 
> directories, I think the right answer is "no". It's not a problem as
> long as the attributes streams are always tied to the file/directory
> they are attributes of - then the "directory link" is really just a
> file link, and can't cause any cycles.

Right.

> Do we have any servers that implement it? I think NFSv4 might be a good 
> test-case if so.

I believe Solaris Express already has an implementation, so I should
have at least one server implementation available in the basement.
So probably does Hummingbird, since their NFSv4 server runs on Windows.

CITI is working on the Linux code, but I'm not sure how far they've come
on the server side. I'll ask Bruce in the morning...

Cheers,
  Trond
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[Brad Boyer]

On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote:
> On Sun, 29 Aug 2004, Trond Myklebust wrote:
> > >  - how to actually test this out in practice (ie getting reiser4 to do the
> > >    proper thing wrt the VFS layer, but preferably _also_ having another
> > >    filesystem like NFSv4 or cifs that actually uses this and shows what
> > >    the problems are).
> > 
> > As I said, NFSv4 can be made ready pretty quickly: Bruce is already
> > finishing up the xattr implementation.
> 
> Do we have any servers that implement it? I think NFSv4 might be a good 
> test-case if so.

I know the mention of it in the past has brought up ridicule, but
netatalk requires some level of multi-stream support in the export
to the client, and currently just emulates it on the server side
by hiding the extra files in various directories. It should be
possible to modify it to use named streams/attributes for the
extra data. If that happens, it would be nice if hfs and hfsplus
could export the data so it could be used. The view exported by
netatalk does match the internal structure of hfs, after all.

(For those of you that aren't familiar with the Macintosh, a Mac
style filesystem has two peer streams, data and resource. They
are each a seekable bytestream, and are actually at the same
level inside the filesystem. In addition, there is a large set
of fixed length attributes stored, such as the file type info
and icon position. In addition, hfsplus also supports full
named streams and arbitrary metadata, tho that isn't used much.)

	Brad Boyer
	flar@allandria.com

Re: silent semantic changes with reiser4

[Alexander G. M. Smith]

Trond Myklebust wrote on Sun, 29 Aug 2004 19:17:10 -0400:
> Is it just the fantasy of supporting hard-links across "stream
> boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
> nobody wants to have to add cyclic graph detection to their filesystems
> anyway. 8-)

Been there, done that.  It works in a rough form in an experimental RAM file system for BeOS.  But deleting files requires a graph traversal to see if it is isolating just one item or a whole bunch of them.  It also implies keeping track of multiple parent directories for everything.  It does make it useful for the user - you can have files and directories in multiple places, so you don't have to decide on one classification for a file (or directory).  One other downside is a lot of locking complexity (or not if you don't need fine grained locking) to guarantee atomicity of the delete.

For user convenience, the low level delete function returns an error code if there is more than one item cut off from the root, but it could just as well delete all of them (sort of like rmdir giving an error if the directory isn't empty rather than just deleting everything).

- Alex

Re: silent semantic changes with reiser4

[Hans Reiser]

You are assuming that the hiding has to be embodied by the dentry cache. 
That sounds hard. Let's not do that; I am too lazy.

We can put it at a layer above that. Let us have a "mask" concept. For 
the purposes of this discussion, the mask can be made totally trivial 
and hardcoded, with none of the expressive power of my new DoD project, 
and can probably be coded in a few hours to exclude the metas. For my 
new DoD project we are creating a configurable, scalable, sophisticated 
"mask" abstraction, and you must successfully pass through the mask 
before you can ask if the filesystem contains X/Y/Z. (For the darpa 
project with the general purpose masks with specifications and all that 
we will take 6 months, not a few hours.)

The rest shall wait until after I get a bite to eat, ok?

Hans

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Sun, Aug 29, 2004 at 01:06:41PM -0700, Hans Reiser wrote:
>
>  
>
>>How about if you educate me on the problems you see for a bit before I 
>>respond? I think it might help us move into a constructive discussion.
>>    
>>
>
>A slew of cache coherency issues (and memory consumption on top of that).
>Rigth now we have a signle dentry tree for all fs instances, no matter
>how many times it and its subtrees are visible in the tree.  Which,
>obviously, avoids all that crap.  As soon as we start trying to have
>multiple trees over the same fs, we are in for a *lot* of fun.
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

I think we have to either:

1) Link only to the file and not the directory in the file-directory 
duality. This means we change the semantics of hard link so that it only 
links to the data and the standard metadata, and the optional 
attributes/streams/files-in-directory are not seen by the second link. 
You can think of this as meaning that hard links only connect to the 
file and not the directory in the file-directory wave-particle duality

or, 2) we should ask Alexander Smith to help with applying the graph 
traversal cycle detection code that he wrote.

I can go either way contentedly for now. 2) is the right long term 
solution. 1) is probably the right short term solution.

Ok, Linus and Viro, now I see why it was hard. Being able to effectively 
connect to compound documents only with symlinks is a bit distasteful, 
but it is quite livable, and I very much hope you decide it is better 
than fragmenting the namespace.

Hans

Re: silent semantic changes with reiser4

[Alexander G. M. Smith]

Hans Reiser wrote on Mon, 30 Aug 2004 00:42:13 -0700:
> or, 2) we should ask Alexander Smith to help with applying the graph 
> traversal cycle detection code that he wrote.

I put my foot into it, didn't I?  It's not too bad, one trick to simplify
things is to have the concept of a main parent and step parents.  The file
system name space graph is kept structured so that by always going upward
through the main parent, you're guaranteed to reach the root.  It doesn't
save anything on traversal when deleting files (you still have to visit
all the relevant nodes to check for cycles), but allows you to replace
hard links with a new kind of dynamic symbolic link.  The hard link
pretends to be a symbolic link presenting the "true path" to its target,
which changes as the target moves around (so it's not as reliable as real
hard link).  Old software doesn't see the cycles in the graph, just follows
the single true path to any particular object like it currently does, and
sees a bunch of symbolic links to everything else.  New software can see
the full parent list (not just a single "..") and the actual hard links.

> Ok, Linus and Viro, now I see why it was hard. Being able to effectively 
> connect to compound documents only with symlinks is a bit distasteful, 
> but it is quite livable, and I very much hope you decide it is better 
> than fragmenting the namespace.

Still, having hard links to attributes or directories of attributes is
useful.  For example, if several contact manager entries reference the
same person, it would be nice to have that represented as a hard link to
the attribute directory tree describing that person.  That's assuming you
implement contacts (name, address, etc) as a bunch of attributes in a
directory.

- Alex

Re: silent semantic changes with reiser4

[Paul Stewart]

On 2004.08.27 23:08, Paul Jackson wrote:
> Hans wrote:
> > We create filename/pseudos/backup, and that tells the archiver what  
> > to do.....
> 
> Instead of exposing the old semantics under a new interface, why not
> expose the new semantics under a new interface.

Here's another take on the same theme.  To see attrs on files, one can  
either use a newly developed application which can use special new  
syscalls/flags on syscalls a Paul Jackson recommends.  However from an  
old shell or application one can also open the attribute node on
/home/myself/foo.txt by checking out /attr/home/myself/foo.txt/, which  
points to the "as directory" node on the filesystem that foo.txt points  
to.

The strange part of this idea is that the /attr filesystem wouldn't be  
conventionally browsable.  An opendir() on any path would return the  
attributes, not the subdirectory contents, even if the target was a  
conventional directory.  File based operations (open(), unlink(), etc)  
would operate on the "as directory" node of the dirname() of the file  
argument.  The /attr filesystem would be completely virtual, honoring  
the permissions, crossing filesystems, and inheriting the root  
directory of the calling process.

Just a thought from a Linux fs layman -- certainly one of the Reiser  
unwashed masses.

--
Paul

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Mon, 30 Aug 2004, Paul Stewart wrote:
> 
> Here's another take on the same theme.  To see attrs on files, one can  
> either use a newly developed application which can use special new  
> syscalls/flags on syscalls a Paul Jackson recommends.  However from an  
> old shell or application one can also open the attribute node on
> /home/myself/foo.txt by checking out /attr/home/myself/foo.txt/, which  
> points to the "as directory" node on the filesystem that foo.txt points  
> to.

This is the same idea as Al Viro's /proc/self/fd/#42/attr issue, except 
yours has two fundamental problem: races and ambiguities.

If you open a filename in some "secondary" tree (be it /proc or //attr or 
whatever) based on the filename in the primary one, you have two issues 
that you need to work out:

 - how do you handle a name change in the primary tree at the same time as 
   lookup
 - how do you handle the ambiguity of
	//attr/usr/bin/emacs/icon
   (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the 
   "emacs/icon" attribute on "/usr/bin").

The ambiguity can be handled by saying that attributes only have one
component (ie only the _last_ component of a lookup is the attribute
name). But the race between primary tree and secondary tree cannot be
handled in a normal name-space.

What Al did was to avoid both by "fixing" the attribute lookup point with 
another open - _exactly_ the same way "openat()" handles it. So Al's 
naming convention avoids both the ambiguity and the primary tree name 
change races by first opening the primary tree file, and then explicitly 
using that file as the "anchor" in the secondary tree. He did it in /proc, 
where we obviously already do export an open fd as an anchor-point.

> The strange part of this idea is that the /attr filesystem wouldn't be  
> conventionally browsable.

That may make it non-intuitive to use, but that's not the real problem.  
See above.

		Linus

Re: silent semantic changes with reiser4

[Alexander Lyamin]

Fri, Aug 27, 2004 at 11:28:51AM -0700, Hans Reiser wrote:
> Rik van Riel wrote:
> 
> >
> >
> >Currently I see no way to distinguish between the stuff
> >that should be backed up and the stuff that shouldn't.
> >
> > 
> >
> We create filename/pseudos/backup, and that tells the archiver what to 
> do.....

its ugly idea.

representations should be distinguished from sources. and its reasonably
to put this distinguisher somewhere in namespace. but filename/pseudos/backup
is plain ugly.
-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander Lyamin wrote:
| Fri, Aug 27, 2004 at 11:28:51AM -0700, Hans Reiser wrote:
|
|>Rik van Riel wrote:
|>
|>
|>>
|>>Currently I see no way to distinguish between the stuff
|>>that should be backed up and the stuff that shouldn't.
|>>
|>>
|>>
|>
|>We create filename/pseudos/backup, and that tells the archiver what to
|>do.....
|
|
| its ugly idea.
|
| representations should be distinguished from sources. and its reasonably
| to put this distinguisher somewhere in namespace. but
filename/pseudos/backup
| is plain ugly.

How about filename/metas/serialized, or filename/metas/fschunk?

And I did mention before about how it'd be nice for it to be an fs chunk...

Imagine 5000 tiny little files that, by the time reiser's done with
them, fit inside 5 blocks or so.  It's insanely cheaper to copy the five
blocks than to tar them up.  It also means that we don't have to worry
about supporting tar on other platforms -- if they don't have reiser4,
they need a separate converter.  If apps are going to use reiser4
metadata for anything significant, they probably would link against some
sort of userland implementation of reiser4 as a Windows library, say, to
access the files elsewhere.

That's extreme, but you get the idea.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTPKc3gHNmZLgCUhAQJ5xw//T+kLDCHlfQEaGM9XAeFDZU9f4m4PWjet
n2xODVIxaT7WmnZ4nQ3rRTiJjsXvRBcPdWUTFRZl6NbLqa/K6ESpUJAZsZynRhXd
NLJ9gEpeqcUrXBLrmTD5AbzENOLdqDIo2qQ+hde68H5vo+naa5xtTyrMiHNHQWbf
9P45WKXYt36madg4KjAhZcYB+T843fc/ItjLQBTwxBNOykBcZ82UUNwGeGbiqczm
uXftCD6BNtKsXnJxu2jVc8CK93lOw4SIf7BY09PFXgK2XTpAsKzOwwqpBtdxSzqD
lkz+nGEeYZk6GRA3N1OTbPJnboee7WHZzMyvXDMBsExfR6puV1SYA2NWhj9uGDmS
nYNZwdN+RimiouD1D9DD+oCQe/Gr66THicex5FM37cMBdaDf0jGblvpDdQRfkhcX
2KGLCt1ax620ia2945/+/+6iQqM7+oTQsVNziYjVGQ4gCNRzZh8SBtM0yUZuecNO
H84fWYLICGos+Qk7FyCuQOquTSVRWKadkiDaOSre2hFk+S9VmDWkghJSNGrnqoKi
pFYFrUzL+NXPHs6G8b8EKAaQQt+Z8iHE6xgdc+yS56Pdrq8apRzeQv2zhgZaRq9J
kCRgvYo40xFMX4B+EqexxIiwc6nPUD5dFbt0OnJMEk4pduFq64lMVdoQA8Qp4X8a
0feuVuaWHr0=
=ymeH
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Hans Reiser]

David Masover wrote:

>
>
> And I did mention before about how it'd be nice for it to be an fs 
> chunk...

Please define fs chunk. My mind has an open door.;-)

>
> Imagine 5000 tiny little files that, by the time reiser's done with
> them, fit inside 5 blocks or so. It's insanely cheaper to copy the five
> blocks than to tar them up. It also means that we don't have to worry
> about supporting tar on other platforms -- if they don't have reiser4,
> they need a separate converter. If apps are going to use reiser4
> metadata for anything significant, they probably would link against some
> sort of userland implementation of reiser4 as a Windows library, say, to
> access the files elsewhere.
>
> That's extreme, but you get the idea.

Re: silent semantic changes with reiser4

[Hans Reiser]

David Masover wrote:

>
>
> And I did mention before about how it'd be nice for it to be an fs 
> chunk...

Please define fs chunk. My mind has an open door.;-)

>
> Imagine 5000 tiny little files that, by the time reiser's done with
> them, fit inside 5 blocks or so. It's insanely cheaper to copy the five
> blocks than to tar them up. It also means that we don't have to worry
> about supporting tar on other platforms -- if they don't have reiser4,
> they need a separate converter. If apps are going to use reiser4
> metadata for anything significant, they probably would link against some
> sort of userland implementation of reiser4 as a Windows library, say, to
> access the files elsewhere.
>
> That's extreme, but you get the idea.

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
| David Masover wrote:
|
|>
|>
|> And I did mention before about how it'd be nice for it to be an fs
|> chunk...
|
|
| Please define fs chunk. My mind has an open door.;-)

I think this is sort of like dump/restore, but for a subset of the tree.
~ So it's faster for all the reasons dump/restore are faster than tar for
a whole filesystem.  But I don't know all that much about either
dump/restore or reiser4 format plugins.

I'm defining an "fs chunk" as a complete, read-only reiser4 filesystem
of the data in question.  As a snapshot.  I'm imagining it being faster
to create and ultimately smaller and more intelligently packed than a
tarball.

What if you could get a blocklist for a directory?  Of course some
things would have to be changed to completely divorce it from the parent
filesystem and have it still be usable.

I'm imagining that literally grabbing a chunk of the filesystem -- using
its own storage layer -- would be faster than tarring.  Think like if
directories had blocklists as well as files.  It would also be smaller
- -- see below.  And it completely eliminates the question of "what may I
never back up" -- the fs already knows that.  Without having to be told
by plugins.  If the fs stores it to disk, the backup stores it, unless
it's part of the "cache" proposal.

This is a rough idea, and has issues.  For instance, how do we know
what's cache and what's not?  If you have foo.gz/ugz cached and someone
writes to foo.gz/ugz, I think that should be written to the cache and
the original foo.gz should be removed -- regenerated if necessary from
foo.gz/ugz.  But the reverse is true -- write to foo.gz and foo.gz/ugz
should go away, to be regenerated if necessary.  Only one should go into
the backup, though.  I think foo.gz goes into the backup and is
regenerated from foo.gz/ugz if necessary.  More generally, when you have
two redundant streams, backup the smaller one.

|>
|> Imagine 5000 tiny little files that, by the time reiser's done with
|> them, fit inside 5 blocks or so. It's insanely cheaper to copy the five
|> blocks than to tar them up. It also means that we don't have to worry
|> about supporting tar on other platforms -- if they don't have reiser4,
|> they need a separate converter. If apps are going to use reiser4
|> metadata for anything significant, they probably would link against some
|> sort of userland implementation of reiser4 as a Windows library, say, to
|> access the files elsewhere.
|>
|> That's extreme, but you get the idea.
|
|

P.S.  Sorry for the dupe, Hans, accidently hit "reply" instead of
"reply-to-all"...

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUuS3gHNmZLgCUhAQKWCQ/+LTNqHvndRAMqzdAJ0rugZl30bqqPmrFs
vNqSwEx09RUxN6caOd+C4F7oVj/as0AJrNqyN5YR6/t6BKGxh+xEDnKdKiIRR5qp
/TJmsMSkh54MSqaZZPL5EB6/DjWscBN7zKCaaCN6f8xB6KU2BCOK5pNSHDvYhISA
uyD8AXrsOLRGC82dNtmh68KdrG+7VcTZMohulXNnosLLRJSuKeYEphB0i5HrvXQn
nZ010yb6LnmKtNKD+w7bFxt3ul+lQo8gti7zBJqb7mV4pa1MnQNhGZRgvey/Pq/l
ahXrrABauUFNwM4PfFWUoY9zjOdCufQpWev/mNGuoo9BS2ZiRg8LXoUpf2bOvi6I
yOTMRb37w6+w37Jv8B0vYPehW3bmuJpb+jDyZ5AYS0GygJmm3bEIFZYupYt2pU/v
7EBIzLszCEnqPbm0glGdWT5jyYiZGucvDK+W4qQ3ymEqaBIdGIti4G56mQ7bTQer
DPVT4dv38OE/PEtKb6xLQUaos7hBg+sWM5+mT9B+7kX/cYDhtdBkZpYqf10jAg34
4ufl2NS48CxqCcml7RayPVAuKT1ZN9sDcfXVYnPw33R94/lziZnfJrIInbnf9MZK
+P9v6WlFi6+9GZjwqFK9ZCJBoTCKRq1lGiy5flTDf1APAgpmU6CS/0Da4s1QiLLI
YRoeX8Nv8Us=
=WFSh
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
| David Masover wrote:
|
|>
|>
|> And I did mention before about how it'd be nice for it to be an fs
|> chunk...
|
|
| Please define fs chunk. My mind has an open door.;-)

I think this is sort of like dump/restore, but for a subset of the tree.
~ So it's faster for all the reasons dump/restore are faster than tar for
a whole filesystem.  But I don't know all that much about either
dump/restore or reiser4 format plugins.

I'm defining an "fs chunk" as a complete, read-only reiser4 filesystem
of the data in question.  As a snapshot.  I'm imagining it being faster
to create and ultimately smaller and more intelligently packed than a
tarball.

What if you could get a blocklist for a directory?  Of course some
things would have to be changed to completely divorce it from the parent
filesystem and have it still be usable.

I'm imagining that literally grabbing a chunk of the filesystem -- using
its own storage layer -- would be faster than tarring.  Think like if
directories had blocklists as well as files.  It would also be smaller
- -- see below.  And it completely eliminates the question of "what may I
never back up" -- the fs already knows that.  Without having to be told
by plugins.  If the fs stores it to disk, the backup stores it, unless
it's part of the "cache" proposal.

This is a rough idea, and has issues.  For instance, how do we know
what's cache and what's not?  If you have foo.gz/ugz cached and someone
writes to foo.gz/ugz, I think that should be written to the cache and
the original foo.gz should be removed -- regenerated if necessary from
foo.gz/ugz.  But the reverse is true -- write to foo.gz and foo.gz/ugz
should go away, to be regenerated if necessary.  Only one should go into
the backup, though.  I think foo.gz goes into the backup and is
regenerated from foo.gz/ugz if necessary.  More generally, when you have
two redundant streams, backup the smaller one.

|>
|> Imagine 5000 tiny little files that, by the time reiser's done with
|> them, fit inside 5 blocks or so. It's insanely cheaper to copy the five
|> blocks than to tar them up. It also means that we don't have to worry
|> about supporting tar on other platforms -- if they don't have reiser4,
|> they need a separate converter. If apps are going to use reiser4
|> metadata for anything significant, they probably would link against some
|> sort of userland implementation of reiser4 as a Windows library, say, to
|> access the files elsewhere.
|>
|> That's extreme, but you get the idea.
|
|

P.S.  Sorry for the dupe, Hans, accidently hit "reply" instead of
"reply-to-all"...

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUuS3gHNmZLgCUhAQKWCQ/+LTNqHvndRAMqzdAJ0rugZl30bqqPmrFs
vNqSwEx09RUxN6caOd+C4F7oVj/as0AJrNqyN5YR6/t6BKGxh+xEDnKdKiIRR5qp
/TJmsMSkh54MSqaZZPL5EB6/DjWscBN7zKCaaCN6f8xB6KU2BCOK5pNSHDvYhISA
uyD8AXrsOLRGC82dNtmh68KdrG+7VcTZMohulXNnosLLRJSuKeYEphB0i5HrvXQn
nZ010yb6LnmKtNKD+w7bFxt3ul+lQo8gti7zBJqb7mV4pa1MnQNhGZRgvey/Pq/l
ahXrrABauUFNwM4PfFWUoY9zjOdCufQpWev/mNGuoo9BS2ZiRg8LXoUpf2bOvi6I
yOTMRb37w6+w37Jv8B0vYPehW3bmuJpb+jDyZ5AYS0GygJmm3bEIFZYupYt2pU/v
7EBIzLszCEnqPbm0glGdWT5jyYiZGucvDK+W4qQ3ymEqaBIdGIti4G56mQ7bTQer
DPVT4dv38OE/PEtKb6xLQUaos7hBg+sWM5+mT9B+7kX/cYDhtdBkZpYqf10jAg34
4ufl2NS48CxqCcml7RayPVAuKT1ZN9sDcfXVYnPw33R94/lziZnfJrIInbnf9MZK
+P9v6WlFi6+9GZjwqFK9ZCJBoTCKRq1lGiy5flTDf1APAgpmU6CS/0Da4s1QiLLI
YRoeX8Nv8Us=
=WFSh
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Martin J. Bligh]

--On Thursday, August 26, 2004 11:15:19 -0700 Linus Torvalds <torvalds@osdl.org> wrote:
> On Thu, 26 Aug 2004, Rik van Riel wrote:
>> 
>> It's a relief to know that nobody's taking my humorous
>> suggestion seriously, but now we still have the "standard
>> Unix tools can't manipulate files" problem...
> 
> I disagree. They can manipulate the files a whole lot better than they can 
> manipulate xattr's.
> 
> For example, you _could_ probably (but hey, maybe "tar" tries to strip
> slashes off the end of filenames, so this might not work due to silly
> reasons like that) back up a compound file with
> 
> 	tar cvf file.tar file file/
> 
> although unpacking it would require that tar be taught about the thing. 
> And you definitely could write a script to do the thing, ie even with an 
> unmodified tar you could do
> 
> 	tar cvf file-archive.tar file
> 	cd file
> 	tar cvf ../attribute-archive.tar .
> 
> which is a hell of a lot better than what you can do with the fsattr 
> interfaces and unmodified legacy applications.
> 
> So one of the advantages of "dir-as-file/file-as-dir" is exactly that you
> _can_ manipulate the data with legacy tools. Sure, things that traverse a
> directory tree might need some (likely fairly trivial) modifications if
> they really want to take advantage of the subfiles, but that's still
> likely to be _much_ less of an issue than with fsattr's that have a 
> totally different model entirely.

What would "test -d" and "test -f" return on these magic beasties? I can't
think of any combinations that wouldn't confuse the crap out of userspace.

M.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Martin J. Bligh wrote:
> 
> What would "test -d" and "test -f" return on these magic beasties? I can't
> think of any combinations that wouldn't confuse the crap out of userspace.

"It's a feature".

The S_ISDIR/S_ISREG tests show real information: it shows not only user
intent ("you should consider this a file, even if it has attributes"), but
also whether it is a directory or a container.

And there's a real technical difference there: the streams contained
within a file are bound to that file. The files contained within a
directory are _independent_ of that directory. Big difference. HUGE
difference.

So it's not confusing. If it tests as a file, you think of it as a file.  
It may have attributes aka named streams associated with it, and you may
be able to open those attributes by treating the file as a directory, but
that doesn't really change the fact that it's a file.

The _big_ difference is that when you can make the compound object _look_ 
like a directory, that means that you can now manage the attributes with 
standard tools. They are still attributes, though.

		Linus

Re: silent semantic changes with reiser4

[Martin J. Bligh]

> On Thu, 26 Aug 2004, Martin J. Bligh wrote:
>> 
>> What would "test -d" and "test -f" return on these magic beasties? I can't
>> think of any combinations that wouldn't confuse the crap out of userspace.
> 
> "It's a feature".
> 
> The S_ISDIR/S_ISREG tests show real information: it shows not only user
> intent ("you should consider this a file, even if it has attributes"), but
> also whether it is a directory or a container.
> 
> And there's a real technical difference there: the streams contained
> within a file are bound to that file. The files contained within a
> directory are _independent_ of that directory. Big difference. HUGE
> difference.
> 
> So it's not confusing. If it tests as a file, you think of it as a file.  
> It may have attributes aka named streams associated with it, and you may
> be able to open those attributes by treating the file as a directory, but
> that doesn't really change the fact that it's a file.
> 
> The _big_ difference is that when you can make the compound object _look_ 
> like a directory, that means that you can now manage the attributes with 
> standard tools. They are still attributes, though.

I think what you're saying is that they'd both return positive, right?

Unfortunately I think lots of code in userspace assumes files / directories
are mutually exclusive conditions, and acts accordingly ... eg:

if [ -f file ]
	do file stuff
else
	do directory stuff
fi.

Now I'm not saying they're not broken ... but seems like a dangerous thing 
to change in the midst of a stable kernel series.

M.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

Am Donnerstag, den 26.08.2004, 14:00 -0700 schrieb Martin J. Bligh:

> > "It's a feature".
> > 
> > The S_ISDIR/S_ISREG tests show real information: it shows not only user
> > intent ("you should consider this a file, even if it has attributes"), but
> > also whether it is a directory or a container.
> > 
> > And there's a real technical difference there: the streams contained
> > within a file are bound to that file. The files contained within a
> > directory are _independent_ of that directory. Big difference. HUGE
> > difference.
> > 
> > So it's not confusing. If it tests as a file, you think of it as a file.  
> > It may have attributes aka named streams associated with it, and you may
> > be able to open those attributes by treating the file as a directory, but
> > that doesn't really change the fact that it's a file.
> > 
> > The _big_ difference is that when you can make the compound object _look_ 
> > like a directory, that means that you can now manage the attributes with 
> > standard tools. They are still attributes, though.
> 
> I think what you're saying is that they'd both return positive, right?

No. A file is still a file and S_ISREG will thus return false.
The application should just try to open it as a directory if it wants to
access the assiociated embedded attribute files. If the open fails,
well, it knows that the filesystem (or Linux version, whatever) doesn't
support this feature.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

Am Donnerstag, den 26.08.2004, 14:00 -0700 schrieb Martin J. Bligh:

> > "It's a feature".
> > 
> > The S_ISDIR/S_ISREG tests show real information: it shows not only user
> > intent ("you should consider this a file, even if it has attributes"), but
> > also whether it is a directory or a container.
> > 
> > And there's a real technical difference there: the streams contained
> > within a file are bound to that file. The files contained within a
> > directory are _independent_ of that directory. Big difference. HUGE
> > difference.
> > 
> > So it's not confusing. If it tests as a file, you think of it as a file.  
> > It may have attributes aka named streams associated with it, and you may
> > be able to open those attributes by treating the file as a directory, but
> > that doesn't really change the fact that it's a file.
> > 
> > The _big_ difference is that when you can make the compound object _look_ 
> > like a directory, that means that you can now manage the attributes with 
> > standard tools. They are still attributes, though.
> 
> I think what you're saying is that they'd both return positive, right?

No. A file is still a file and S_ISREG will thus return false.
The application should just try to open it as a directory if it wants to
access the assiociated embedded attribute files. If the open fails,
well, it knows that the filesystem (or Linux version, whatever) doesn't
support this feature.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Martin J. Bligh wrote:
> 
> I think what you're saying is that they'd both return positive, right?

No. I'd say that a file would look like a file, even if it has attributes.

It wouldn't show as a directory at all - unless you start looking at 
attributes. Because it really _is_ a file, and it's "directory aspect" is 
really nothing but a way to make its named streams visible.

So you really should consider it a perfectly regular file, and so only 
S_ISREG() will return true, and S_ISDIR() will return false.

Think of it this way: when you add a named attribute to current files 
using the xattr interfaces, do you start thinking of the file as a 
directory? No. Even though it actually now is a starting point for finding 
more information.

It's just that thanks to it's directory aspect, people and apps that 
_care_ about the attributes suddenly can trivially access them. You can 
access them in shell scripts, you can access them in programs, you can 
access them in perl. With no special knowledge necessary.

Remember: in the file-as-directory model, we're always talking about just 
_one_ object. That one object just happens to have namd streams or 
attributes or whatever you want to call them associated with it, and you 
can _see_ them through the normal directory interfaces. But that doesn't 
really make the object a "directory". It's still a file.

In other words, the "directory" part is just a _view_ into the file. A
view that potentially exposes a lot _more_ of the file, but we're still
talking about the same file.

In contrast, a S_IFDIR-like _directory_ is something else entirely. When 
you view the things in that, you aren't looking at data "inside" the 
directory. You're looking at somethign totally independent.

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> In other words, the "directory" part is just a _view_ into the file. A
> view that potentially exposes a lot _more_ of the file, but we're still
> talking about the same file.
> 
> In contrast, a S_IFDIR-like _directory_ is something else entirely. When 
> you view the things in that, you aren't looking at data "inside" the 
> directory. You're looking at somethign totally independent.

One of the constraints is that you can create any name in an
S_IFDIR-like directory, but you probably can't create any name "in" an
S_IFREG-like file, because the names have meaning -- except in
designated subtrees of the view intended for holding arbitrary names.

-- Jamie

Re: silent semantic changes with reiser4

[David Greaves]

Linus Torvalds wrote:

>On Thu, 26 Aug 2004, Martin J. Bligh wrote:
>  
>
>>I think what you're saying is that they'd both return positive, right?
>>    
>>
>
>No. I'd say that a file would look like a file, even if it has attributes.
>
>It wouldn't show as a directory at all - unless you start looking at 
>attributes. Because it really _is_ a file, and it's "directory aspect" is 
>really nothing but a way to make its named streams visible.
>
>So you really should consider it a perfectly regular file, and so only 
>S_ISREG() will return true, and S_ISDIR() will return false.
>
>  
>
ie, logically:

S_ISREG() no longer precludes O_DIRECTORY from succeeding
conversley
O_DIRECTORY succeeding no longer implies S_ISDIR()

However, apps making those assumptions will not notice new behaviour.

?

David

Re: silent semantic changes with reiser4

[Miquel van Smoorenburg]

In article <57730000.1093554054@flay>,
Martin J. Bligh <mbligh@aracnet.com> wrote:
>> On Thu, 26 Aug 2004, Martin J. Bligh wrote:
>>> 
>>> What would "test -d" and "test -f" return on these magic beasties? I can't
>>> think of any combinations that wouldn't confuse the crap out of userspace.
>> 
>I think what you're saying is that they'd both return positive, right?

test -f file  -> true
test -f file/ -> false
test -d file  -> false
test -d file/ -> true

Mike.
-- 
"In times of universal deceit, telling the truth becomes
 a revolutionary act." -- George Orwell.

Re: silent semantic changes with reiser4

[Chris Wright]

* Linus Torvalds (torvalds@osdl.org) wrote:
> The _big_ difference is that when you can make the compound object _look_ 
> like a directory, that means that you can now manage the attributes with 
> standard tools. They are still attributes, though.

For things that actually use such attributes to determine file access, do the
attributes get attributes?

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: silent semantic changes with reiser4

[Anton Altaparmakov]

On Thu, 2004-08-26 at 21:54, Linus Torvalds wrote:
> The S_ISDIR/S_ISREG tests show real information: it shows not only user
> intent ("you should consider this a file, even if it has attributes"), but
> also whether it is a directory or a container.
> 
> And there's a real technical difference there: the streams contained
> within a file are bound to that file. The files contained within a
> directory are _independent_ of that directory. Big difference. HUGE
> difference.
> 
> So it's not confusing. If it tests as a file, you think of it as a file.  
> It may have attributes aka named streams associated with it, and you may
> be able to open those attributes by treating the file as a directory, but
> that doesn't really change the fact that it's a file.

What about the attributes/named streams of a directory though?  If you
open it as a directory, you would get the files inside the directory. 
So how do you get at the attributes and named streams of the directory
itself using this interface?

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/

Re: silent semantic changes with reiser4

[Anton Altaparmakov]

On Thu, 2004-08-26 at 21:54, Linus Torvalds wrote:
> The S_ISDIR/S_ISREG tests show real information: it shows not only user
> intent ("you should consider this a file, even if it has attributes"), but
> also whether it is a directory or a container.
> 
> And there's a real technical difference there: the streams contained
> within a file are bound to that file. The files contained within a
> directory are _independent_ of that directory. Big difference. HUGE
> difference.
> 
> So it's not confusing. If it tests as a file, you think of it as a file.  
> It may have attributes aka named streams associated with it, and you may
> be able to open those attributes by treating the file as a directory, but
> that doesn't really change the fact that it's a file.

What about the attributes/named streams of a directory though?  If you
open it as a directory, you would get the files inside the directory. 
So how do you get at the attributes and named streams of the directory
itself using this interface?

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 20:57, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Linus Torvalds wrote:
> > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> >
> > No no. The stream you get from a directory is totally _independent_ of
> > the contents of the directory. Anything else would be pointless.
>
> It's a relief to know that nobody's taking my humorous
> suggestion seriously, but now we still have the "standard
> Unix tools can't manipulate files" problem...

Is it possible to sufficiently hide "dirs inside files"
so that old tools will be unable to see them?

I just checked:

ls -d /foo  does lstat64("/foo", ...)
ls -d /foo/ does lstat64("/foo", ...)
	but
ls -d /foo/. does lstat64("/foo/.", ...)

Will it work out if "dir inside file" will only be visible when referred as "file/."?
--
vda

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 20:57, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Linus Torvalds wrote:
> > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> >
> > No no. The stream you get from a directory is totally _independent_ of
> > the contents of the directory. Anything else would be pointless.
>
> It's a relief to know that nobody's taking my humorous
> suggestion seriously, but now we still have the "standard
> Unix tools can't manipulate files" problem...

Is it possible to sufficiently hide "dirs inside files"
so that old tools will be unable to see them?

I just checked:

ls -d /foo  does lstat64("/foo", ...)
ls -d /foo/ does lstat64("/foo", ...)
	but
ls -d /foo/. does lstat64("/foo/.", ...)

Will it work out if "dir inside file" will only be visible when referred as "file/."?
--
vda

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Denis Vlasenko wrote:
> 
> Is it possible to sufficiently hide "dirs inside files"
> so that old tools will be unable to see them?

Certainly possible.

> I just checked:
> 
> ls -d /foo  does lstat64("/foo", ...)
> ls -d /foo/ does lstat64("/foo", ...)
> 	but
> ls -d /foo/. does lstat64("/foo/.", ...)
> 
> Will it work out if "dir inside file" will only be visible when referred as "file/."?

That would likely be the _easiest_ approach for the kernel, and does solve 
the problem with some apps knowingly removing the trailing '/'.

Note that we could try this out with existing filesystems with very 
minimal changes:

 - make directory bind mounts work on top of files ("graft_tree()")
 - make open_namei() and friend _not_ do the mount-point following for the 
   last component if it's a non-directory.
 - probably some trivial fixups I haven't thought about. There might be 
   some places that use "S_ISDIR()" to check for whether something can be 
   looked up, but the main path walking already just checks whether there
   is a ".lookup" operation or not.

This would already allow people to "try out" how different applications 
would react to a file that can show up both as a directory and a file. The 
patch might end up being less than 25 lines or so, the difficulty is in 
finding all the right places.

Al, anything I missed?

(And yes, it's a quick hack, but it's a quick hack that would probably
mimic a good part of what we would have to do internally in the VFS layer
to support this notion anyway).

		Linus

Re: silent semantic changes with reiser4

[viro]

[reposted in thread]

On Thu, Aug 26, 2004 at 11:46:33AM -0700, Linus Torvalds wrote:
> Note that we could try this out with existing filesystems with very 
> minimal changes:
> 
>  - make directory bind mounts work on top of files ("graft_tree()")
>  - make open_namei() and friend _not_ do the mount-point following for the 
>    last component if it's a non-directory.
>  - probably some trivial fixups I haven't thought about. There might be 
>    some places that use "S_ISDIR()" to check for whether something can be 
>    looked up, but the main path walking already just checks whether there
>    is a ".lookup" operation or not.
> 
> This would already allow people to "try out" how different applications 
> would react to a file that can show up both as a directory and a file. The 
> patch might end up being less than 25 lines or so, the difficulty is in 
> finding all the right places.

The real issue is what to do with unlink() et.al. on these guys.  Note
that "unlink is OK if all we have there is a bunch of directory mounts"
won't work well - we have no good way to check that condition.

Even funnier one is what we do if we have directory mounted there *and*
have something mounted on stuff in that directory.

Yes, that's one of the probable directions for such stuff, but there's a
lot of fun semantics questions and answers to them will matter a lot.

Hey, if we lose the "can't unlink/rmdir/rename over something that is
a mountpoint in other life" - I'm happy and we can get a lot of much
more interesting stuff to work.  It will take some work (e.g. making
sure we can find all vfsmounts over given mountpoint and sorting out
the locking issues, which won't be trivial), but the main obstacle in
that direction is not in architecture - it's in SuS and tradition; as
the matter of fact, our life would be much easier if we stopped trying
to give -EBUSY here and just dissolved all subtrees mounted on anything
that has that dentry.

Re: silent semantic changes with reiser4

[viro]

On Thu, Aug 26, 2004 at 08:13:23PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
 
> Hey, if we lose the "can't unlink/rmdir/rename over something that is
> a mountpoint in other life" - I'm happy and we can get a lot of much
> more interesting stuff to work.  It will take some work (e.g. making
> sure we can find all vfsmounts over given mountpoint and sorting out
> the locking issues, which won't be trivial), but the main obstacle in
> that direction is not in architecture - it's in SuS and tradition; as
> the matter of fact, our life would be much easier if we stopped trying
> to give -EBUSY here and just dissolved all subtrees mounted on anything
> that has that dentry.

Argh...  OK, now I remember why I went for -EBUSY for unlink() (we obviously
are not bound by SuS on that one).  Consider the following scenario:
	* local file foo got something else bound on it for a while
	* we are tight on space - time to clean up
	* oh, look - contents of foo is junk
	* rm foo
	* ... oh, fuck, there goes the underlying file.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

Am Donnerstag, den 26.08.2004, 21:32 +0100 schrieb :

> Argh...  OK, now I remember why I went for -EBUSY for unlink() (we obviously
> are not bound by SuS on that one).  Consider the following scenario:
> 	* local file foo got something else bound on it for a while
> 	* we are tight on space - time to clean up
> 	* oh, look - contents of foo is junk
> 	* rm foo
> 	* ... oh, fuck, there goes the underlying file.

Right. Good thinking. You can't delete a directory while there regular
files in it either.

Another question: /mnt/test is a mountpoint. Why can I rename `mnt' but
can't rename `test'?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

Am Donnerstag, den 26.08.2004, 21:32 +0100 schrieb :

> Argh...  OK, now I remember why I went for -EBUSY for unlink() (we obviously
> are not bound by SuS on that one).  Consider the following scenario:
> 	* local file foo got something else bound on it for a while
> 	* we are tight on space - time to clean up
> 	* oh, look - contents of foo is junk
> 	* rm foo
> 	* ... oh, fuck, there goes the underlying file.

Right. Good thinking. You can't delete a directory while there regular
files in it either.

Another question: /mnt/test is a mountpoint. Why can I rename `mnt' but
can't rename `test'?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> Argh...  OK, now I remember why I went for -EBUSY for unlink() (we obviously
> are not bound by SuS on that one).  Consider the following scenario:
> 	* local file foo got something else bound on it for a while
> 	* we are tight on space - time to clean up
> 	* oh, look - contents of foo is junk
> 	* rm foo
> 	* ... oh, fuck, there goes the underlying file.

Hey, that's a valid reason for doing -EBUSY for normal bind-mounts, but it
actually _is_ what we want for an "implied-by-way-of-container-mount". 
After all, when you do a "rm foo", you do mean "remove the container foo".

I replied to your earlier off-list mail in private, so let's re-iterate
for the list: the easiest way to handle this is to just have a "mount
option", and have "MNT_ALLOWUNLINK" that gets set for containers, and that
users could possibly choose to set for regular mounts too (as a mount
option) if they really want to (and if we want them to).

So there's no reason we'd have to drop existing mount behaviour only 
because we also have special files that look like mountpoints. 

		Linus

[some sanity for a change] possible design issues for hybrids

[viro]

[subject changed and *please* let's keep the wankfest out of that branch;
we are talking about possible ways to handle hybrids, *NOT* their desirability,
effect on DARPA funding, size of KDE developers' genitals experienced by
Aunt Tillie, etc.]

On Thu, Aug 26, 2004 at 01:47:37PM -0700, Linus Torvalds wrote:
> Hey, that's a valid reason for doing -EBUSY for normal bind-mounts, but it
> actually _is_ what we want for an "implied-by-way-of-container-mount". 
> After all, when you do a "rm foo", you do mean "remove the container foo".
> 
> I replied to your earlier off-list mail in private, so let's re-iterate
> for the list: the easiest way to handle this is to just have a "mount
> option", and have "MNT_ALLOWUNLINK" that gets set for containers, and that
> users could possibly choose to set for regular mounts too (as a mount
> option) if they really want to (and if we want them to).
> 
> So there's no reason we'd have to drop existing mount behaviour only 
> because we also have special files that look like mountpoints. 

All right, let's see where that would take us.

1) we would need to find all vfsmounts over given dentry.  Probably a cyclic
list (we want to check if there are normal mounts/bindings among those and
we want to dissolve them if there's none).

2) we would need to do something about locking, since mount trees in other
guys' namespaces are protected by semaphores of their own.

3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
it.  MNT_DETACH will have no problems, but normal umount() is a different
story.  Note that it's not just hybrid-related problem - implementing the
mount traps will cause the same kind of trouble,

4) OK, we have those hybrids and want to create vfsmounts when crossing a
mountpoint.  When do they go away, anyway?  When we don't reference them
anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
happens explicitly - outside of the "dropping the final reference" path.
Might become a locking issue.

5) Creation of these vfsmounts: fs should somehow tell us whether it wants
one or not (at the very least, we should stop *somewhere*).  Can we use
the same dentry/inode?  I'm not sure and I really doubt that we'd like that.

6) if it's a method, where should it live, *especially* if we want them on
device nodes.  Note that inode_operations belongs to underlying fs, so it's
not particulary good place for device case.

7) automount folks want partially shared mount trees (well, mirrored,
actually).  The basic idea is that while namespace boundary is a trust
boundary, we might want to be able to say "I trust this guy to handle
that subtree under /home/stuff/foobar/mounts".  It's the same situation
as with shared mappings (I want separate address space, but I'm willing
to share that chunk of memory with other process), except that we want
it to be allowed to become asymmetric (shared r/o mapping with somebody
else having it r/w).   That stuff (and mount traps) is the next pending
major work on the mount trees.  We probably want hybrids work to go with it,
since they affect the same data structures and need more or less the same
changes.
[And yes, this is the open season on design discussions for shared subtrees -
automount folks are welcome to join]

8) what should happen when something is mounted on top of directory-over-file?
How do we treat such beasts?  What are the implications?

9) how do we recognize such mountpoints in the path lookups?  It *is* a
hot path, so we should be careful in that area; the impact will be felt
by everything in the system.

10) how do we deal with directories, anyway?  Mixing "attributes" with
normal directory contents is going to be fun, what with lseek() insanity.
That's not an issue for hybrids, but it is one for anybody who wants
any sort of common metadata exported that way.  Note that it's really
important for fs writers - having two different pieces of code to export
the same information (for directories and non-directories resp.) is going
to become a prime breeding ground for bugs.

11) if we go for your "here's stuff that belongs in device node viewed
as directory", how would that play with fs metadata exporters?  Again,
due to the insanity of lseek() on directories it's *very* hard to deal
with unions, when parts of directory come from different chunks of code.

That's it for starters.  Technical answers/questions/comments are welcome.
Generic masturbation => over there in the parent thread, please.

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

[ This is quite possibly just impossible and buggy, but here's my
  implementation notes. You asked for them. ]

On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> All right, let's see where that would take us.
> 
> 1) we would need to find all vfsmounts over given dentry.  Probably a cyclic
> list (we want to check if there are normal mounts/bindings among those and
> we want to dissolve them if there's none).

Not per-inode? dentries are a bitmore memory-constrained than inodes, and 
we only need this for filesystems that want to support it, so we wouldn't 
need to put this information in each dentry.

Since the vfsmounts have back-pointers to the dentry they are mounted on,
you can still do a "per-dentry" traversal, by just doign the inode list
and checking the dentry pointer. No?

Alternatively, we could just put the list on an external hash-chain 
entirely, and hash off the dentry. It depends on how often we end up 
needing it.

Or we could just put it in the dentry itself. I'd hate to make it any 
bigger than it already is, but maybe it doesn't matter that much.

> 2) we would need to do something about locking, since mount trees in other
> guys' namespaces are protected by semaphores of their own.

Ok, I'll admit that I don't know how to handle namespaces. These things 
should just go into a global namespace, and I was kind of assuming it 
would happen automatically in "lookup_mnt()" or something like that. A 
special case in lookup_mnt which says something like "if you didn't find a 
vfsmount, we create a new one for you".

It should be reasonably easy to create new ones on-the-fly, since we'd
have all the information (the parent vfsmount comes stated, and the
vfsmount we create would point to the same things that the "base" one
would).

> 3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
> it.  MNT_DETACH will have no problems, but normal umount() is a different
> story.  Note that it's not just hybrid-related problem - implementing the
> mount traps will cause the same kind of trouble,

Don't allow umount. It's not something the user can unmount - the mount is 
"implied" in the file. 

> 4) OK, we have those hybrids and want to create vfsmounts when crossing a
> mountpoint.  When do they go away, anyway?  When we don't reference them
> anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
> happens explicitly - outside of the "dropping the final reference" path.
> Might become a locking issue.

Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously 
clossed something over when I blathered about the "create the vfsmount on 
the fly" thing above ;)

> 5) Creation of these vfsmounts: fs should somehow tell us whether it wants
> one or not (at the very least, we should stop *somewhere*).  Can we use
> the same dentry/inode?  I'm not sure and I really doubt that we'd like that.

Why not? When doing the ->lookup() operation, the filesystem would create
the vfsmount and bind it to the current vfsmount. That guarantees that it
has a vfsmount, and will mean that it will show up positive with the
"d_mountpoint()" query, which in turn will cause us to do the
"lookup_mnt()".

Which in turn will create the other vfsmounts as needed, if you have 
multiple namespaces.

So I _think_ creation is easy. Getting rid of the dang things migth be 
harder.

> 6) if it's a method, where should it live, *especially* if we want them on
> device nodes.  Note that inode_operations belongs to underlying fs, so it's
> not particulary good place for device case.

Why not just let the existing .lookup method initialize the mount-point 
thing? After that, it's all in the VFS layer (I'd hate to have filesystems 
mess around with vfsmounts - they'll just get it wrong).

> 7) automount folks want partially shared mount trees (well, mirrored,
> actually).

I don't think you can get partial sharing on one of these puppies. You'd 
always have one vfsmount per namespace (well, lazily created, so maybe in 
practice you'd see a lot fewer).

> 8) what should happen when something is mounted on top of directory-over-file?
> How do we treat such beasts?  What are the implications?

Allow file-on-file mounts - it will just totally hide the thing (in that
namespace, at least). But don't allow the dir-on-file thing (that we
already don't allow).

> 9) how do we recognize such mountpoints in the path lookups?  It *is* a
> hot path, so we should be careful in that area; the impact will be felt
> by everything in the system.

I don't think you'll have any special cases. Same d_mountpount(), same 
lookup_mnt().

> 10) how do we deal with directories, anyway?  Mixing "attributes" with
> normal directory contents is going to be fun, what with lseek() insanity.

You couldn't get at the attributes that way anyway, so I think the point 
is moot. The "real" directory always takes over.

Crazy people could try to just use the regular "xattrs" interfaces if they 
really want attributes on directories. You wouldn't ever be able to use 
the "easy" one.

> 11) if we go for your "here's stuff that belongs in device node viewed
> as directory", how would that play with fs metadata exporters?  Again,
> due to the insanity of lseek() on directories it's *very* hard to deal
> with unions, when parts of directory come from different chunks of code.

Don't go there. See above. Directories would be just plain directories, 
you could never see their metadata. Same goes for at least symlinks, and 
possibly other filetypes too (ie at least initially, a block or character 
special device will just take over the whole "file_operations", which 
includes "readdir", so it's actually hard to have the filesystem do 
anything about those).

		Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 03:04:21PM -0700, Linus Torvalds wrote:
 
> > 2) we would need to do something about locking, since mount trees in other
> > guys' namespaces are protected by semaphores of their own.
> 
> Ok, I'll admit that I don't know how to handle namespaces. These things 
> should just go into a global namespace, and I was kind of assuming it 
> would happen automatically in "lookup_mnt()" or something like that. A 
> special case in lookup_mnt which says something like "if you didn't find a 
> vfsmount, we create a new one for you".
> 
> It should be reasonably easy to create new ones on-the-fly, since we'd
> have all the information (the parent vfsmount comes stated, and the
> vfsmount we create would point to the same things that the "base" one
> would).

Erm...  What do we do upon unlink()?  I'm killing a file, fs it's in is
mounted in a dozen of places (no namespaces, just chroot jails, whatever).
We need to find all vfsmounts to be killed by that.

And BTW that's an argument against anchoring that list in inode - unlink()
on foo should not screw bar/... even if bar and foo are links to the same
file.  So we'll need to check for dentry match anyway.
 
> > 3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
> > it.  MNT_DETACH will have no problems, but normal umount() is a different
> > story.  Note that it's not just hybrid-related problem - implementing the
> > mount traps will cause the same kind of trouble,
> 
> Don't allow umount. It's not something the user can unmount - the mount is 
> "implied" in the file. 

See below.

> > 4) OK, we have those hybrids and want to create vfsmounts when crossing a
> > mountpoint.  When do they go away, anyway?  When we don't reference them
> > anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
> > happens explicitly - outside of the "dropping the final reference" path.
> > Might become a locking issue.
> 
> Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously 
> clossed something over when I blathered about the "create the vfsmount on 
> the fly" thing above ;)

> > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants
> > one or not (at the very least, we should stop *somewhere*).  Can we use
> > the same dentry/inode?  I'm not sure and I really doubt that we'd like that.
> 
> Why not? When doing the ->lookup() operation, the filesystem would create
> the vfsmount and bind it to the current vfsmount. That guarantees that it
> has a vfsmount, and will mean that it will show up positive with the
> "d_mountpoint()" query, which in turn will cause us to do the
> "lookup_mnt()".

Several paragraphs below you are saying that you don't like fs messing with
vfsmounts.  Use of ->lookup() would mean that we should not only create
and attach vfsmounts from within fs code, but would actually have to make
->lookup() return vfsmount+dentry, AFAICS.
 
> > 6) if it's a method, where should it live, *especially* if we want them on
> > device nodes.  Note that inode_operations belongs to underlying fs, so it's
> > not particulary good place for device case.
> 
> Why not just let the existing .lookup method initialize the mount-point 
> thing? After that, it's all in the VFS layer (I'd hate to have filesystems 
> mess around with vfsmounts - they'll just get it wrong).

> Allow file-on-file mounts - it will just totally hide the thing (in that
> namespace, at least). But don't allow the dir-on-file thing (that we
> already don't allow).

Err...  What about dir-on-dir-that-is-on-file?  I.e. mount on foo/. when foo
is a file?
 
> > 9) how do we recognize such mountpoints in the path lookups?  It *is* a
> > hot path, so we should be careful in that area; the impact will be felt
> > by everything in the system.

> I don't think you'll have any special cases. Same d_mountpount(), same 
> lookup_mnt().

See above on use ->lookup()

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > 
> > It should be reasonably easy to create new ones on-the-fly, since we'd
> > have all the information (the parent vfsmount comes stated, and the
> > vfsmount we create would point to the same things that the "base" one
> > would).
> 
> Erm...  What do we do upon unlink()?  I'm killing a file, fs it's in is
> mounted in a dozen of places (no namespaces, just chroot jails, whatever).
> We need to find all vfsmounts to be killed by that.

But that should be trivial: that's what the per-inode vfsmount list was 
(your first question in the last email).

> And BTW that's an argument against anchoring that list in inode - unlink()
> on foo should not screw bar/... even if bar and foo are links to the same
> file.  So we'll need to check for dentry match anyway.

And again - I talked about this in the previous email. Even if you anchor 
the list in "struct inode", or you do it with a totally external 
hash-list, you'll always have the "vfsmount->mnt_mountpoint" pointer to 
point to the dentry. So you can just iterate over the list, and 
cherry-pick the ones that point to the dentry you are removing.

>  
> > > 3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
> > > it.  MNT_DETACH will have no problems, but normal umount() is a different
> > > story.  Note that it's not just hybrid-related problem - implementing the
> > > mount traps will cause the same kind of trouble,
> > 
> > Don't allow umount. It's not something the user can unmount - the mount is 
> > "implied" in the file. 
> 
> See below.
> 
> > > 4) OK, we have those hybrids and want to create vfsmounts when crossing a
> > > mountpoint.  When do they go away, anyway?  When we don't reference them
> > > anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
> > > happens explicitly - outside of the "dropping the final reference" path.
> > > Might become a locking issue.
> > 
> > Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously 
> > clossed something over when I blathered about the "create the vfsmount on 
> > the fly" thing above ;)
> 
> > > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants
> > > one or not (at the very least, we should stop *somewhere*).  Can we use
> > > the same dentry/inode?  I'm not sure and I really doubt that we'd like that.
> > 
> > Why not? When doing the ->lookup() operation, the filesystem would create
> > the vfsmount and bind it to the current vfsmount. That guarantees that it
> > has a vfsmount, and will mean that it will show up positive with the
> > "d_mountpoint()" query, which in turn will cause us to do the
> > "lookup_mnt()".
> 
> Several paragraphs below you are saying that you don't like fs messing with
> vfsmounts.  Use of ->lookup() would mean that we should not only create
> and attach vfsmounts from within fs code, but would actually have to make
> ->lookup() return vfsmount+dentry, AFAICS.

No, lookup would just return the dentry, but the dentry would already be 
filled in with the mount-point information.

And you can do that with a simple vfs helper function, ie the filesystem 
itself would just need to do

	pseudo_mount(dentry, inode);

thing - which just fills in dentry->d_mountpoint with a new vfsmount
thing. It would allocate a new root dentry (for the pseudo-mount) and a
new vfsmount, and make dentry->d_mountpoint point to it.

IOW, the filesystem itself would never mess around with d_mountpoint 
itself.

> Err...  What about dir-on-dir-that-is-on-file?  I.e. mount on foo/. when foo
> is a file?

Hmm.. We might as well allow it, I suspect. It's not like it should hurt.  
We'd end up following the mount-chain twice, but we already have that
issue with multi-mount cases..

		Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 03:45:09PM -0700, Linus Torvalds wrote:
> No, lookup would just return the dentry, but the dentry would already be 
> filled in with the mount-point information.
> 
> And you can do that with a simple vfs helper function, ie the filesystem 
> itself would just need to do
> 
> 	pseudo_mount(dentry, inode);
> 
> thing - which just fills in dentry->d_mountpoint with a new vfsmount
> thing. It would allocate a new root dentry (for the pseudo-mount) and a
> new vfsmount, and make dentry->d_mountpoint point to it.

What dentry->d_mountpoint?  No such thing...

Note that we can't get vfsmount by dentry - that's the point of having these
guys in the first place.  So I'm not sure what you are trying to do here -
dentry + inode is definitely not enough to attach any vfsmounts anywhere.

That's not about namespaces - same fs mounted in several places will give
the same problem - one dentry, many vfsmounts.  And we obviously *can't*
have one vfsmount for all of them - if the same fs is mounted on /foo and
/bar, we will have the same dentry for /foo/splat and /bar/splat.  So
what should we get for /foo/splat/. and /bar/splat/.?  Same dentry *and*
same vfsmount?  I'd expect .. from the former to give /foo and from the
latter - /bar...

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> What dentry->d_mountpoint?  No such thing...

Sorry - set "dentry->d_mounted++" + "add vfsmount/dentry to hashes".

Yes, it's not a direct list off the dentry, but it effectively is the same
thing.

So basically: the "d_mounted++" just makes sure we get into
"lookup_mnt()". That's where we will usually find the actual mount thing.

And that's also where the special case comes in: if we _don't_ find the 
mount thing there, that's where we need to create it. That will only 
happen if somebody looks it up using another namespace, though, so it 
should be rare.

And when it does happen, we can just create a new vfsmount - we have all
the information there (we'll have to walk the per-inode-or-whatever
vfsmount list to find all the information to populate the thing with, of
course. But we need that list _anyway_, so it should be a fairly 
straightforward special case).

			Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 04:24:51PM -0700, Linus Torvalds wrote:
> So basically: the "d_mounted++" just makes sure we get into
> "lookup_mnt()". That's where we will usually find the actual mount thing.
> 
> And that's also where the special case comes in: if we _don't_ find the 
> mount thing there, that's where we need to create it. That will only 
> happen if somebody looks it up using another namespace, though, so it 
> should be rare.

No.  Trivial example:

mount --bind /foo /bar
mount /dev/sda1 /bar/baz

do lookup for /foo/baz.  No namespaces involved, no vfsmounts found, d_mounted
positive and we certainly do *not* want anything to be created at that point.

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:

> On Thu, Aug 26, 2004 at 04:24:51PM -0700, Linus Torvalds wrote:
> > So basically: the "d_mounted++" just makes sure we get into
> > "lookup_mnt()". That's where we will usually find the actual mount thing.
> > 
> > And that's also where the special case comes in: if we _don't_ find the 
> > mount thing there, that's where we need to create it. That will only 
> > happen if somebody looks it up using another namespace, though, so it 
> > should be rare.
> 
> No.  Trivial example:
> 
> mount --bind /foo /bar
> mount /dev/sda1 /bar/baz
> 
> do lookup for /foo/baz.  No namespaces involved, no vfsmounts found, d_mounted
> positive and we certainly do *not* want anything to be created at that point.

Right. We obviously need to mark the dentry somehow, and only do the 
"create vfsmount" special case in this special case. If we didn't do that, 
then it wouldn't be a special case, now would it?

So clearly lookup_mnt() needs to check the dentry in the failure case. The 
marking could be in any of three places
 - mark the dentry itself by just using a dentry flag ("DCACHE_AUTOVFSMNT")
   or by having a dentry operation for this.
 - mark the inode itself (same logic as dentry)
 - look up the first vfsmount (on the inode list), and look if that one is 
   of the automatic type.

Clearly we should not _always_ create a vfsmount, that would just break 
the existign logic.

		Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 04:57:01PM -0700, Linus Torvalds wrote:
> Right. We obviously need to mark the dentry somehow, and only do the 
> "create vfsmount" special case in this special case. If we didn't do that, 
> then it wouldn't be a special case, now would it?
> 
> So clearly lookup_mnt() needs to check the dentry in the failure case. The 
> marking could be in any of three places
>  - mark the dentry itself by just using a dentry flag ("DCACHE_AUTOVFSMNT")
>    or by having a dentry operation for this.
>  - mark the inode itself (same logic as dentry)
>  - look up the first vfsmount (on the inode list), and look if that one is 
>    of the automatic type.
> 
> Clearly we should not _always_ create a vfsmount, that would just break 
> the existign logic.

Hmm...  IOW, you are suggesting to use normal trigger for lookup_mnt() and
then have extra check in lookup_mnt() failure exit.  That will probably
work (and I'd prefer to mark dentry here).

Freeing these guys will not be fun.  The final reference could be given
up under very different locking conditions - it's certainly a blocking
operation (it can trigger final umount, among other things), but having
it trigger removal from mount tree(s) can get ugly.  We might be able to
tweak that, but it will probably need require major surgery in several
places.

Right now we assume that vfsmount lock *and* per-namespace semaphore are
held when detaching from the tree (attaching what could be the first at
given dentry --- same + i_sem on its ->i_sem).  We also assume that vfsmount is
detached before refcount reaches zero.  The thing is, that final refcount
can be dropped while holding namespace semaphore.  It's not fatal (we'll
need to hold onto these guys until dropping that semaphore or add an analog
of mntput() for places under namespace sem), but it can get messy.

Situation with unlink(): we need to (a) check that all vfsmounts over
given point are "hybrid" ones (locking is not a problem at that point)
and (b) after successful operation (unlink() and rename() alike, AFAICS)
we want to go and kill everything under the victim dentry.  *That*
can get very ugly, since fs could be mounted in a lot of places and/or
in a lot of namespaces.

Note that we also need to do in VFS what NFS tries to do in revalidation -
if dentry is invalidated, everything under it should be detached and
dropped.  So it's not something new - we do need something that would
be called outside of any superblock/namespace/vfsmount locks (both
unlink() and revalidation qualify) and would kill the vfsmounts that
become unreachable.

Locking rules for attaching might need adjustment (the logics with ->i_sem
is that we want !d_mountpoint() stay true around call of ->unlink(),
->rmdir() and ->rename(); so cloning or replacing doesn't need that
protection.  Now it might become interesting - we'll need to be careful
in pivot_root(2) and possibly some other places).

I'll see what can be done (it'll definitely take careful looking at the
code), but there is a chance of things getting very ugly.

BTW, shared subtrees *are* compatible with that puppy - there's a lot
of corner cases to look at, but AFAICS they are all doable.  Lifetime
rules can get ugly in some of them, but that's on par with the situation
without sharing.

One thing that looks like a bad interface: we get forcible "use same
dentry for file and directory" with that design.  That can lead to
a big can of worms - without a large audit I wouldn't bet a dime on
ability to get that right (and same applies to reiser4 code as-is,
for the same reasons).  No idea how bad it will turn out to be - right
now all I can say is that we might run into trouble and that we almost
certainly will get extra "be careful not to do <list of things>" rules
out of that.

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> Hmm...  IOW, you are suggesting to use normal trigger for lookup_mnt() and
> then have extra check in lookup_mnt() failure exit.

Yes. That should be an _extremely_ rare failure case anyway, unless people 
start using namespaces more.

> Freeing these guys will not be fun.

I agree. I frankly have no idea how to do it. I think we need to keep them
around for as long as the dentry lifetime, I suspect. That would likely
imply that the vfsmount can't increase the dentry refcount - and we'd have
some special code in dentry_iput() or something to get rid of them.

> Situation with unlink(): we need to (a) check that all vfsmounts over
> given point are "hybrid" ones (locking is not a problem at that point)
> and (b) after successful operation (unlink() and rename() alike, AFAICS)
> we want to go and kill everything under the victim dentry.  *That*
> can get very ugly, since fs could be mounted in a lot of places and/or
> in a lot of namespaces.

It _might_ be acceptable to just do the normal shrink_dentry thing, and 
then return EBUSY if somebody is still there. 

> Locking rules for attaching might need adjustment (the logics with ->i_sem
> is that we want !d_mountpoint() stay true around call of ->unlink(),
> ->rmdir() and ->rename()

That should be ok. Since we'd only do the d_mount stuff on a "->lookup()", 
and since the lookup will take ->i_sem anyway, we should be fine, no?

> One thing that looks like a bad interface: we get forcible "use same
> dentry for file and directory" with that design.

Hmm.. Wouldn't the "directory" dentry always be the fake-mountpoint root 
one? So we'd always have two dentries, although they are tied together 
pretty tightly..

		Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 07:46:41PM -0700, Linus Torvalds wrote:
> 
> 
> On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > 
> > Hmm...  IOW, you are suggesting to use normal trigger for lookup_mnt() and
> > then have extra check in lookup_mnt() failure exit.
> 
> Yes. That should be an _extremely_ rare failure case anyway, unless people 
> start using namespaces more.

Namespaces actually are irrelevant - a box with several dozens of chroot
jail is not particulary exotic thing and it's equivalent to our case.

However, extra check of a flag at that point won't be too much overhead
anyway.
 
> > Freeing these guys will not be fun.
> 
> I agree. I frankly have no idea how to do it. I think we need to keep them
> around for as long as the dentry lifetime, I suspect. That would likely
> imply that the vfsmount can't increase the dentry refcount - and we'd have
> some special code in dentry_iput() or something to get rid of them.

My plan is to drop them as soon as their refcount goes to zero.  In other
words, make the final mntput() do removal from tree for such vfsmounts.
Will take some surgery, but it's definitely safer than playing with ->d_iput()
et.al.

> > can get very ugly, since fs could be mounted in a lot of places and/or
> > in a lot of namespaces.
> 
> It _might_ be acceptable to just do the normal shrink_dentry thing, and 
> then return EBUSY if somebody is still there. 

We need it for graceful invalidation in case of NFS and friends, though.
I think it might be doable; if it isn't, we'll see what else can be done,
but it's worth investigating.
 
> > Locking rules for attaching might need adjustment (the logics with ->i_sem
> > is that we want !d_mountpoint() stay true around call of ->unlink(),
> > ->rmdir() and ->rename()
> 
> That should be ok. Since we'd only do the d_mount stuff on a "->lookup()", 
> and since the lookup will take ->i_sem anyway, we should be fine, no?

Oh, I'm not worried about that kind of incrementing ->d_mount.  It's OK.
It's existing places that worry me - e.g. if we get pivot_root() moving
that hybrid-created vfsmount around and suddenly find that we had replaced
"killable" vfsmount with normal one, while holding only vfsmount lock.
That would screw the logics in unlink() if we get a race.

It's not that hard to do, just that we need to take a good look at that
kind of corner cases.

> > One thing that looks like a bad interface: we get forcible "use same
> > dentry for file and directory" with that design.
> 
> Hmm.. Wouldn't the "directory" dentry always be the fake-mountpoint root 
> one? So we'd always have two dentries, although they are tied together 
> pretty tightly..

No.  We don't want to allocate dentries outside of fs code - if nothing
else, they may need to have proper ->d_op set, etc.  So the only way I
see here is to have lookup_mnt() set the dentry it got to ->mnt_root of
new vfsmount.  Anything else opens such a can of worms that I don't want
to think of audit involved.

We *might* have a secondary dentry allocated by ->lookup() and stored somewhere
in the main one.  I'm not sure it's a good way to handle the things, though.

I'll try to put together what we got so far + notes on mount-traps and
post more coherent and detailed variant of original post.  Hopefully,
writing it down would help to sort the things out...

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > 
> > I agree. I frankly have no idea how to do it. I think we need to keep them
> > around for as long as the dentry lifetime, I suspect. That would likely
> > imply that the vfsmount can't increase the dentry refcount - and we'd have
> > some special code in dentry_iput() or something to get rid of them.
> 
> My plan is to drop them as soon as their refcount goes to zero.  In other
> words, make the final mntput() do removal from tree for such vfsmounts.

Can't do that. At least not if you do my plan. 

Even with a zero refcount, the dentry is still valid, and won't see a new 
"->lookup()". And since in my scheme the vfsmount is populated only at 
lookup (and copied later), you can't drop _all_ the vfsmount's. You'd have 
to at least leave one of them, in order to know how to re-create the 
others.

Or do you have a cunning plan?

> > > can get very ugly, since fs could be mounted in a lot of places and/or
> > > in a lot of namespaces.
> > 
> > It _might_ be acceptable to just do the normal shrink_dentry thing, and 
> > then return EBUSY if somebody is still there. 
> 
> We need it for graceful invalidation in case of NFS and friends, though.
> I think it might be doable; if it isn't, we'll see what else can be done,
> but it's worth investigating.

Hey, succeeding and leaving it pending is obviously the user-friendly
thing to do, so I won't argue against it. It's more of a "if it's too hard
to do right, EBUSY is probably an acceptable fallback".

> > Hmm.. Wouldn't the "directory" dentry always be the fake-mountpoint root 
> > one? So we'd always have two dentries, although they are tied together 
> > pretty tightly..
> 
> No.  We don't want to allocate dentries outside of fs code - if nothing
> else, they may need to have proper ->d_op set, etc.  So the only way I
> see here is to have lookup_mnt() set the dentry it got to ->mnt_root of
> new vfsmount.  Anything else opens such a can of worms that I don't want
> to think of audit involved.

Hmm. If you use the same dentry for mnt_root as for mnt_mountpoint, how do 
you avoid the infinite loop in follow_mount? Another magic special case?

			Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 08:57:40PM -0700, Linus Torvalds wrote:
> Can't do that. At least not if you do my plan. 
> 
> Even with a zero refcount, the dentry is still valid, and won't see a new 
> "->lookup()". And since in my scheme the vfsmount is populated only at 
> lookup (and copied later), you can't drop _all_ the vfsmount's. You'd have 
> to at least leave one of them, in order to know how to re-create the 
> others.
> 
> Or do you have a cunning plan?

a) set ->d_mount to 1 and set DCACHE_HYBRID at ->lookup() time
b) in lookup_mnt():
	if (unlikely(dentry has DCACHE_HYBRID)) {
		if (linkely(dentry->d_mount == 1))
			goto New;
	}
	do the normal series of hash lookups
	if (likely(dentry doesn't have DCACHE_HYBRID))
		return;
	if (vfsmount is marked as hybrid && dentry == vfsmount->mnt_dentry)
		return;
New:
	allocate new vfsmount and mark it as hybrid
	attach it on top of our one and go there

Does that qualify as cunning?  With that approach we have nothing non-trivial
to do at lookup time and lookup_mnt() will DTRT with minimal fuss.
 
> Hmm. If you use the same dentry for mnt_root as for mnt_mountpoint, how do 
> you avoid the infinite loop in follow_mount? Another magic special case?

See above.  And yes, I'd love to get a separate dentry (and inode, while
we are at it).  But I don't see how to do that.

One note: we might split lookup_mnt() in two functions - one for the
middle of lookup, another for final (non-triggering) step.

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
>
> > Or do you have a cunning plan?
> 
> a) set ->d_mount to 1 and set DCACHE_HYBRID at ->lookup() time
> b) in lookup_mnt():
> 	if (unlikely(dentry has DCACHE_HYBRID)) {
> 		if (linkely(dentry->d_mount == 1))
> 			goto New;
> 	}
> 	do the normal series of hash lookups
> 	if (likely(dentry doesn't have DCACHE_HYBRID))
> 		return;
> 	if (vfsmount is marked as hybrid && dentry == vfsmount->mnt_dentry)
> 		return;

Hmm.. Here we shouldn't have a vfsmount. If we have one, that means that 
we found it in the hashes, and we should just have returned it 
immediately (whether hybrid or not doesn't matter at that point).

Also, I don't see the magic with "dentry->d_mount == 1", since we may have 
allowed the dentry to be mounted on for real at one of the places, without 
losing it's "hydridity" (it's not a word, but dammit, it should be one) 
for that.

Is that part just an optimization to avoid the hash lookups?

> New:
> 	allocate new vfsmount and mark it as hybrid
> 	attach it on top of our one and go there
> 
> Does that qualify as cunning?  With that approach we have nothing non-trivial
> to do at lookup time and lookup_mnt() will DTRT with minimal fuss.

Ok, I think it qualifies as cunning enough. It has the nice property of 
not apparently needing any vfsmount-related code at all at lookup() time. 
Which is a bonus. 

> > Hmm. If you use the same dentry for mnt_root as for mnt_mountpoint, how do 
> > you avoid the infinite loop in follow_mount? Another magic special case?
> 
> See above.  And yes, I'd love to get a separate dentry (and inode, while
> we are at it).  But I don't see how to do that.
> 
> One note: we might split lookup_mnt() in two functions - one for the
> middle of lookup, another for final (non-triggering) step.

Yes. We don't necessarily even want to trigger the special mountpoint case
at all if we don't need it - which normal usage probably wouldn't ever
trigger. So I agree 100% - that way I think you actually allocate the
vfsmount until somebody really starts doing the "new things". Was that
your plan?

		Linus

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Thu, 26 Aug 2004, Linus Torvalds wrote:
>
> > 	if (vfsmount is marked as hybrid && dentry == vfsmount->mnt_dentry)
> > 		return;
> 
> Hmm.. Here we shouldn't have a vfsmount. If we have one, that means that 
> we found it in the hashes, and we should just have returned it 
> immediately (whether hybrid or not doesn't matter at that point).

Ahh, I misunderstood you. 

You mean that if the _old_ vfsmount was already a hybrid one, we won't try 
to recurse if we didn't find any other, so we return NULL at that point.

Got it. Yes, I bow before your cunningness.

		Linus

Re: [some sanity for a change] possible design issues for hybrids

[Anton Altaparmakov]

On Thu, 2004-08-26 at 23:04, Linus Torvalds wrote:
> [ This is quite possibly just impossible and buggy, but here's my
>   implementation notes. You asked for them. ]
> On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > All right, let's see where that would take us.
> > 3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
> > it.  MNT_DETACH will have no problems, but normal umount() is a different
> > story.  Note that it's not just hybrid-related problem - implementing the
> > mount traps will cause the same kind of trouble,
> 
> Don't allow umount. It's not something the user can unmount - the mount is 
> "implied" in the file. 
> 
> > 4) OK, we have those hybrids and want to create vfsmounts when crossing a
> > mountpoint.  When do they go away, anyway?  When we don't reference them
> > anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
> > happens explicitly - outside of the "dropping the final reference" path.
> > Might become a locking issue.
> 
> Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously 
> clossed something over when I blathered about the "create the vfsmount on 
> the fly" thing above ;)

If it is created on the fly, it should be "easy" to destroy on the fly
using time-based expiry, i.e. a kernel daemon going over all of those
beasts every X seconds (X = 5 perhaps?) and doing something like:

for (each vfsmount) {
	lock_vfsmount(vfsmount);
	if (MOUNT_IS_BUSY(vfsmount)) {
		unlock_vfsmount(vfsmount);
		continue;
	}
	if (current_time() < (vfsmount->last_used_time +
			vfsmount->expire_after)) {
		unlock_vfsmount(vfsmount);
		continue;
	}
	destroy_locked_vfsmount(vfsmount);
}

Wouldn't that work?

> > 10) how do we deal with directories, anyway?  Mixing "attributes" with
> > normal directory contents is going to be fun, what with lseek() insanity.
> 
> You couldn't get at the attributes that way anyway, so I think the point 
> is moot. The "real" directory always takes over.
> 
> Crazy people could try to just use the regular "xattrs" interfaces if they 
> really want attributes on directories. You wouldn't ever be able to use 
> the "easy" one.

But that defeats the whole point of the hybrid objects!  We might as
well just keep the xattrs interface and throw away the new one if we
will have to keep the old one anyway so that we can do named
streams/attributes on directories.  Windows (and other OS?) certainly do
allow them and do use them on directories as well as files...

> > 11) if we go for your "here's stuff that belongs in device node viewed
> > as directory", how would that play with fs metadata exporters?  Again,
> > due to the insanity of lseek() on directories it's *very* hard to deal
> > with unions, when parts of directory come from different chunks of code.
> 
> Don't go there. See above. Directories would be just plain directories, 
> you could never see their metadata. Same goes for at least symlinks, and 
> possibly other filetypes too (ie at least initially, a block or character 
> special device will just take over the whole "file_operations", which 
> includes "readdir", so it's actually hard to have the filesystem do 
> anything about those).

Ah, but at least in other OS (Windows is the one I know about)
directories _also_ have the same semantics as files with respect to
named streams/attributes.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/

Re: [some sanity for a change] possible design issues for hybrids

[Anton Altaparmakov]

On Thu, 2004-08-26 at 23:04, Linus Torvalds wrote:
> [ This is quite possibly just impossible and buggy, but here's my
>   implementation notes. You asked for them. ]
> On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > All right, let's see where that would take us.
> > 3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
> > it.  MNT_DETACH will have no problems, but normal umount() is a different
> > story.  Note that it's not just hybrid-related problem - implementing the
> > mount traps will cause the same kind of trouble,
> 
> Don't allow umount. It's not something the user can unmount - the mount is 
> "implied" in the file. 
> 
> > 4) OK, we have those hybrids and want to create vfsmounts when crossing a
> > mountpoint.  When do they go away, anyway?  When we don't reference them
> > anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
> > happens explicitly - outside of the "dropping the final reference" path.
> > Might become a locking issue.
> 
> Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously 
> clossed something over when I blathered about the "create the vfsmount on 
> the fly" thing above ;)

If it is created on the fly, it should be "easy" to destroy on the fly
using time-based expiry, i.e. a kernel daemon going over all of those
beasts every X seconds (X = 5 perhaps?) and doing something like:

for (each vfsmount) {
	lock_vfsmount(vfsmount);
	if (MOUNT_IS_BUSY(vfsmount)) {
		unlock_vfsmount(vfsmount);
		continue;
	}
	if (current_time() < (vfsmount->last_used_time +
			vfsmount->expire_after)) {
		unlock_vfsmount(vfsmount);
		continue;
	}
	destroy_locked_vfsmount(vfsmount);
}

Wouldn't that work?

> > 10) how do we deal with directories, anyway?  Mixing "attributes" with
> > normal directory contents is going to be fun, what with lseek() insanity.
> 
> You couldn't get at the attributes that way anyway, so I think the point 
> is moot. The "real" directory always takes over.
> 
> Crazy people could try to just use the regular "xattrs" interfaces if they 
> really want attributes on directories. You wouldn't ever be able to use 
> the "easy" one.

But that defeats the whole point of the hybrid objects!  We might as
well just keep the xattrs interface and throw away the new one if we
will have to keep the old one anyway so that we can do named
streams/attributes on directories.  Windows (and other OS?) certainly do
allow them and do use them on directories as well as files...

> > 11) if we go for your "here's stuff that belongs in device node viewed
> > as directory", how would that play with fs metadata exporters?  Again,
> > due to the insanity of lseek() on directories it's *very* hard to deal
> > with unions, when parts of directory come from different chunks of code.
> 
> Don't go there. See above. Directories would be just plain directories, 
> you could never see their metadata. Same goes for at least symlinks, and 
> possibly other filetypes too (ie at least initially, a block or character 
> special device will just take over the whole "file_operations", which 
> includes "readdir", so it's actually hard to have the filesystem do 
> anything about those).

Ah, but at least in other OS (Windows is the one I know about)
directories _also_ have the same semantics as files with respect to
named streams/attributes.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/

Re: [some sanity for a change] possible design issues for hybrids

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 09:56:39AM +0100, Anton Altaparmakov wrote:
> If it is created on the fly, it should be "easy" to destroy on the fly
> using time-based expiry, i.e. a kernel daemon going over all of those
> beasts every X seconds (X = 5 perhaps?) and doing something like:
> 
> for (each vfsmount) {
> 	lock_vfsmount(vfsmount);
> 	if (MOUNT_IS_BUSY(vfsmount)) {
> 		unlock_vfsmount(vfsmount);
> 		continue;
> 	}
> 	if (current_time() < (vfsmount->last_used_time +
> 			vfsmount->expire_after)) {
> 		unlock_vfsmount(vfsmount);
> 		continue;
> 	}
> 	destroy_locked_vfsmount(vfsmount);
> }
> 
> Wouldn't that work?

That would work for a low number of them.  But with Hans' "visions" we'd
have a damn lot of them at which point this isn't really scalable.

Re: silent semantic changes with reiser4

[Jonathan Abbey]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

On Thu, Aug 26, 2004 at 09:28:41PM +0300, Denis Vlasenko wrote:
|
| Is it possible to sufficiently hide "dirs inside files"
| so that old tools will be unable to see them?
| 
| I just checked:
| 
| ls -d /foo  does lstat64("/foo", ...)
| ls -d /foo/ does lstat64("/foo", ...)
| 	but
| ls -d /foo/. does lstat64("/foo/.", ...)
| 
| Will it work out if "dir inside file" will only be visible when referred as "file/."?

I'm used to using ls symlink/. to get ls to show me the directory on
the far side of a symbolic link.  That's a pretty analagous case to
the one we're discussing here, I think?

 Jon

-- 
-------------------------------------------------------------------------------
Jonathan Abbey 				              jonabbey@arlut.utexas.edu
Applied Research Laboratories                 The University of Texas at Austin
GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg

[-- Attachment #2: Type: application/pgp-signature, Size: 185 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Jonathan Abbey wrote:
> | Will it work out if "dir inside file" will only be visible when
> referred as "file/."?
> 
> I'm used to using ls symlink/. to get ls to show me the directory on
> the far side of a symbolic link.  That's a pretty analagous case to
> the one we're discussing here, I think?

By the way, do symlinks have metadata?  Where do you find it? :)

-- Jamie

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier:

> > | Will it work out if "dir inside file" will only be visible when
> > referred as "file/."?
> > 
> > I'm used to using ls symlink/. to get ls to show me the directory on
> > the far side of a symbolic link.  That's a pretty analagous case to
> > the one we're discussing here, I think?
> 
> By the way, do symlinks have metadata?  Where do you find it? :)

Oops. :)

At least in reiser4 they don't have, or at least you can't access them.
ln -s foo bar; cd bar/metas shows me the content of foo/metas.

symlinks are special anyway. Their rights are 777. The only thing they
can have is an owner. No chance to do a symlink/metas/uid then. Hmm.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier:

> > | Will it work out if "dir inside file" will only be visible when
> > referred as "file/."?
> > 
> > I'm used to using ls symlink/. to get ls to show me the directory on
> > the far side of a symbolic link.  That's a pretty analagous case to
> > the one we're discussing here, I think?
> 
> By the way, do symlinks have metadata?  Where do you find it? :)

Oops. :)

At least in reiser4 they don't have, or at least you can't access them.
ln -s foo bar; cd bar/metas shows me the content of foo/metas.

symlinks are special anyway. Their rights are 777. The only thing they
can have is an owner. No chance to do a symlink/metas/uid then. Hmm.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Nikita Danilov]

Christophe Saout writes:
 > Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier:
 > 
 > > > | Will it work out if "dir inside file" will only be visible when
 > > > referred as "file/."?
 > > > 
 > > > I'm used to using ls symlink/. to get ls to show me the directory on
 > > > the far side of a symbolic link.  That's a pretty analagous case to
 > > > the one we're discussing here, I think?
 > > 
 > > By the way, do symlinks have metadata?  Where do you find it? :)
 > 
 > Oops. :)
 > 
 > At least in reiser4 they don't have, or at least you can't access them.

They do.

 > ln -s foo bar; cd bar/metas shows me the content of foo/metas.

That's because lookup for "bar" performs symlink resolution.
 
 > 
 > symlinks are special anyway. Their rights are 777. The only thing they

Symlink is represented by normal inode in UNIX. It has full set of
attributes. It can even be hard-linked.

 > can have is an owner. No chance to do a symlink/metas/uid then. Hmm.
 > 

Nikita.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:

>  > At least in reiser4 they don't have, or at least you can't access them.
> 
> They do.
> 
>  > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
> 
> That's because lookup for "bar" performs symlink resolution.

So I can't access them and it is pointless. ;-)

BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
this makes sense. :)


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:

>  > At least in reiser4 they don't have, or at least you can't access them.
> 
> They do.
> 
>  > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
> 
> That's because lookup for "bar" performs symlink resolution.

So I can't access them and it is pointless. ;-)

BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
this makes sense. :)


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Emil Larsson]

Christophe Saout wrote:

>Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:
>
>  
>
>> > At least in reiser4 they don't have, or at least you can't access them.
>>
>>They do.
>>
>> > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
>>
>>That's because lookup for "bar" performs symlink resolution.
>>    
>>
>
>So I can't access them and it is pointless. ;-)
>
>BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
>this makes sense. :)
>
>  
>
I prefer this one:

hepburn:/# mount
/dev/sda3 on / type reiser4 (rw)
hepburn:/# touch apa
hepburn:/# chmod +x apa
hepburn:/# cd apa
hepburn:/apa# ls
ls: reading directory .: Not a directory

So there is already file-as-dir with legacy "ls" - and it works just 
like people seem to want it to do with legacy apps. Not at all.

/Emil

Re: silent semantic changes with reiser4

[Nikita Danilov]

Christophe Saout writes:
 > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:
 > 
 > >  > At least in reiser4 they don't have, or at least you can't access them.
 > > 
 > > They do.
 > > 
 > >  > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
 > > 
 > > That's because lookup for "bar" performs symlink resolution.
 > 
 > So I can't access them and it is pointless. ;-)
 > 
 > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
 > this makes sense. :)

Why? foo/metas is a file system object just like foo. It has owner,
permission bits, so access to its meta-data should be provided, and
uniform way to provide access to the file system object meta-data is to
have these little magic files inside metas directory, which is a file
system object just like metas. It has owner^@^@^@^@*** - Lisp stack
overflow. RESET

Nikita.

 > 

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

Am Freitag, den 27.08.2004, 02:21 +0400 schrieb Nikita Danilov:

>  > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
>  > this makes sense. :)
> 
> Why? foo/metas is a file system object just like foo. It has owner,
> permission bits, so access to its meta-data should be provided, and
> uniform way to provide access to the file system object meta-data is to
> have these little magic files inside metas directory, which is a file
> system object just like metas. It has owner^@^@^@^@*** - Lisp stack
> overflow. RESET

Yes, of course. But who's going to need that? The meta files are there
to describe regular unix files. They're just exported using a file
system name space. I don't think we need metadata to describe metadata.
Assuming someone adds a directory inside a file where user can add some
attributes/streams/resource forks. As Linus said, they are just
attributes, even if they seem to behave like normal files, they're not,
they belong to the real file. Adding attributes to attributes to
attributes? That would be like adding ACLS to ACLS. Hello the
madness. ;-)

There really should be a point where you can't recurse further. Assume
there is a backup application that always tries to open every dentry it
finds as directory and then recurse. Oops. :-)


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

Am Freitag, den 27.08.2004, 02:21 +0400 schrieb Nikita Danilov:

>  > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
>  > this makes sense. :)
> 
> Why? foo/metas is a file system object just like foo. It has owner,
> permission bits, so access to its meta-data should be provided, and
> uniform way to provide access to the file system object meta-data is to
> have these little magic files inside metas directory, which is a file
> system object just like metas. It has owner^@^@^@^@*** - Lisp stack
> overflow. RESET

Yes, of course. But who's going to need that? The meta files are there
to describe regular unix files. They're just exported using a file
system name space. I don't think we need metadata to describe metadata.
Assuming someone adds a directory inside a file where user can add some
attributes/streams/resource forks. As Linus said, they are just
attributes, even if they seem to behave like normal files, they're not,
they belong to the real file. Adding attributes to attributes to
attributes? That would be like adding ACLS to ACLS. Hello the
madness. ;-)

There really should be a point where you can't recurse further. Assume
there is a backup application that always tries to open every dentry it
finds as directory and then recurse. Oops. :-)


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Hans Reiser]

Nikita Danilov wrote:

>Christophe Saout writes:
> > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:
> > 
> > >  > At least in reiser4 they don't have, or at least you can't access them.
> > > 
> > > They do.
> > > 
> > >  > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
> > > 
> > > That's because lookup for "bar" performs symlink resolution.
> > 
> > So I can't access them and it is pointless. ;-)
> > 
> > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
> > this makes sense. :)
>
>Why? foo/metas is a file system object just like foo. It has owner,
>permission bits, so access to its meta-data should be provided, and
>uniform way to provide access to the file system object meta-data is to
>have these little magic files inside metas directory, which is a file
>system object just like metas. It has owner^@^@^@^@*** - Lisp stack
>overflow. RESET
>
>Nikita.
>
> > 
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>
I think Christophe is a bit right here.  While in general having 
meta-meta objects makes sense, in this particular instance, I don't see 
the functional need for it.  Can you supply an example of where it would 
be useful?

Re: silent semantic changes with reiser4

[Hans Reiser]

Nikita Danilov wrote:

>Christophe Saout writes:
> > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:
> > 
> > >  > At least in reiser4 they don't have, or at least you can't access them.
> > > 
> > > They do.
> > > 
> > >  > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
> > > 
> > > That's because lookup for "bar" performs symlink resolution.
> > 
> > So I can't access them and it is pointless. ;-)
> > 
> > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
> > this makes sense. :)
>
>Why? foo/metas is a file system object just like foo. It has owner,
>permission bits, so access to its meta-data should be provided, and
>uniform way to provide access to the file system object meta-data is to
>have these little magic files inside metas directory, which is a file
>system object just like metas. It has owner^@^@^@^@*** - Lisp stack
>overflow. RESET
>
>Nikita.
>
> > 
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>
I think Christophe is a bit right here.  While in general having 
meta-meta objects makes sense, in this particular instance, I don't see 
the functional need for it.  Can you supply an example of where it would 
be useful?

Re: silent semantic changes with reiser4

[Stephen Wille Padnos]

Hans Reiser wrote:

> Nikita Danilov wrote:
>
>> Christophe Saout writes:
>> > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:
>> > > >  > At least in reiser4 they don't have, or at least you can't 
>> access them.
>> > > > > They do.
>> > > > >  > ln -s foo bar; cd bar/metas shows me the content of 
>> foo/metas.
>> > > > > That's because lookup for "bar" performs symlink resolution.
>> > > So I can't access them and it is pointless. ;-)
>> > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I 
>> don't think
>> > this makes sense. :)
>>
>> Why? foo/metas is a file system object just like foo. It has owner,
>> permission bits, so access to its meta-data should be provided, and
>> uniform way to provide access to the file system object meta-data is to
>> have these little magic files inside metas directory, which is a file
>> system object just like metas. It has owner^@^@^@^@*** - Lisp stack
>> overflow. RESET
>>
>> Nikita.
>>
>> > -
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>>
>>
>>  
>>
> I think Christophe is a bit right here.  While in general having 
> meta-meta objects makes sense, in this particular instance, I don't 
> see the functional need for it.  Can you supply an example of where it 
> would be useful? 

One example is to supply hints about what to do with the attribute if 
the file is copied / moved.
Several types of attributes have been mentioned:

1) Those that are essentially a cached object, which can be re-derived 
from the source file (e.g. thumbnail image)
2) Those that become invalid if the source file changes (e.g. hash value)
3) Those that should be maintained on file copy (e.g. author, copyright, 
etc)
4) Those that may not want to be copied with the file (e.g. security 
keys or the like)
etc.

This could be accomplished without meta-meta information (like 
permissions), but that wouldn't be very extensible, would it :)

- Steve

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>  
>
>No no. The stream you get from a directory is totally _independent_ of the 
>contents of the directory.
>
but the user can link filename/metas/body to filename/metas/tar, if we 
implement a tar plugin, if it happens that the user wants it to be 
totally dependent.

> Anything else would be pointless.
>
>		Linus
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>

Re: silent semantic changes with reiser4

[Diego Calleja]

El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió:

> So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

/bin could be separated (like linus said) but cat /bin/.compound could do
it. This is the /etc/passwd Hans' example, I think:

"If /new_syntax_access_path/big_directory_of_small_files/.glued is a plugin that
aggregates every file in big_directory_of_small_files with a delimiter
separating every file within the aggregation, then one can simply type emacs
/new_syntax_access_path/big_directory_of_small_files/.glued, and the filesystem
has done all the work emacs needs to be effective at this. Not a line of emacs
needs to be changed."

Re: silent semantic changes with reiser4

[Diego Calleja]

El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió:

> So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

/bin could be separated (like linus said) but cat /bin/.compound could do
it. This is the /etc/passwd Hans' example, I think:

"If /new_syntax_access_path/big_directory_of_small_files/.glued is a plugin that
aggregates every file in big_directory_of_small_files with a delimiter
separating every file within the aggregation, then one can simply type emacs
/new_syntax_access_path/big_directory_of_small_files/.glued, and the filesystem
has done all the work emacs needs to be effective at this. Not a line of emacs
needs to be changed."
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[Christophe Saout]

On Thu, Aug 26, 2004 at 07:40:10PM +0200, Diego Calleja wrote:

> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> /bin could be separated (like linus said) but cat /bin/.compound could do
> it. This is the /etc/passwd Hans' example, I think:

Yes, but what about locking? If we have compound files with
individually accessible components a lock on the compound files
could lock its components at the same time.

But what with /usr/.compound? Is it read-only? What if you truncate it?
Does it atomically delete the whole /usr directory tree? What about reading?
It has to go through all these independent files, what if they change while
you are reading the compound file? Does it lock everything? Point-in-time
snapshot? If reiser4 can't do this. ;-)

I don't say that it's impossible to do but I think it's a lot harder than
doing it the other way round.

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Diego Calleja wrote:
> El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió:
> 
> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> /bin could be separated (like linus said) but cat /bin/.compound could
> do it. This is the /etc/passwd Hans' example, I think:

Arghhhh.  I wrote it down to ridicule the idea and now people
are taking it seriously ;(

It should be obvious enough that anything depending on the
kernel parsing file contents will lead to problems.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Diego Calleja wrote:
> El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió:
> 
> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> /bin could be separated (like linus said) but cat /bin/.compound could
> do it. This is the /etc/passwd Hans' example, I think:

Arghhhh.  I wrote it down to ridicule the idea and now people
are taking it seriously ;(

It should be obvious enough that anything depending on the
kernel parsing file contents will lead to problems.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> > /bin could be separated (like linus said) but cat /bin/.compound could
> > do it. This is the /etc/passwd Hans' example, I think:
> 
> Arghhhh.  I wrote it down to ridicule the idea and now people
> are taking it seriously ;(
> 
> It should be obvious enough that anything depending on the
> kernel parsing file contents will lead to problems.

This is one case where the kernel _isn't_ parsing file contents,
but I agree it's ridiculous :)

-- Jamie

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>Rik van Riel wrote:
>  
>
>>>/bin could be separated (like linus said) but cat /bin/.compound could
>>>do it. This is the /etc/passwd Hans' example, I think:
>>>      
>>>
>>Arghhhh.  I wrote it down to ridicule the idea and now people
>>are taking it seriously ;(
>>
>>It should be obvious enough that anything depending on the
>>kernel parsing file contents will lead to problems.
>>    
>>
>
>This is one case where the kernel _isn't_ parsing file contents,
>but I agree it's ridiculous :)
>
>-- Jamie
>  
>
Umm, no, when you write the glued /etc/passwd file it parses the 
delimiters in it to determine what subfile to put the parts of 
/etc/passwd into.

Inheritance with delimiters is the best way to make emacs effective for 
editing lots of small files, Stallman was right and deserves credit for it.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>Rik van Riel wrote:
>  
>
>>>/bin could be separated (like linus said) but cat /bin/.compound could
>>>do it. This is the /etc/passwd Hans' example, I think:
>>>      
>>>
>>Arghhhh.  I wrote it down to ridicule the idea and now people
>>are taking it seriously ;(
>>
>>It should be obvious enough that anything depending on the
>>kernel parsing file contents will lead to problems.
>>    
>>
>
>This is one case where the kernel _isn't_ parsing file contents,
>but I agree it's ridiculous :)
>
>-- Jamie
>  
>
Umm, no, when you write the glued /etc/passwd file it parses the 
delimiters in it to determine what subfile to put the parts of 
/etc/passwd into.

Inheritance with delimiters is the best way to make emacs effective for 
editing lots of small files, Stallman was right and deserves credit for it.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>
>Why not just install .tgz or .zip files of entire binary packages, and
>let the filesystem unpack the bits that are needed when they're
>needed.  Some platforms (i.e. Java) do this already with .jar files
>and it's quite popular.  This would extend it in a versatile way.
>  
>
I like this notion of a plugin that would install a package if one of 
the package's executables was invoked by its name.

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 17:12, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Denis Vlasenko wrote:
> > I think Hans is not planning turning old "file is a stream of bytes"
> > into eight-stream octopus. One stream will remain as a 'main' one,
> > which contains actual data. Others will keep metadata, etc...
>
> This is exactly what the Samba people want, though.
>
> Office suites can store a document with embedded images
> and spread sheets "easily" by putting the text, the
> images and spread sheets all in different file streams.

Kinda far reaching. It's hard to assess whether that is good or bad idea.
Can we start small, like putting metadata (e.g. ACLs) into these streams?
--
vda

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 17:12, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Denis Vlasenko wrote:
> > I think Hans is not planning turning old "file is a stream of bytes"
> > into eight-stream octopus. One stream will remain as a 'main' one,
> > which contains actual data. Others will keep metadata, etc...
>
> This is exactly what the Samba people want, though.
>
> Office suites can store a document with embedded images
> and spread sheets "easily" by putting the text, the
> images and spread sheets all in different file streams.

Kinda far reaching. It's hard to assess whether that is good or bad idea.
Can we start small, like putting metadata (e.g. ACLs) into these streams?
--
vda

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-08-26 at 16:19, Denis Vlasenko wrote:
> > Office suites can store a document with embedded images
> > and spread sheets "easily" by putting the text, the
> > images and spread sheets all in different file streams.
> 
> Kinda far reaching. It's hard to assess whether that is good or bad idea.
> Can we start small, like putting metadata (e.g. ACLs) into these streams?

Openoffice does this in user space and the user space vfs code desktops
use can handle zips so this "just works" already including over NFS,
unlike a kernel proposed method.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Sun, Aug 29, 2004 at 03:30:03PM +0100, Alan Cox wrote:
> 
> Openoffice does this in user space and the user space vfs code desktops
> use can handle zips so this "just works" already including over NFS,
> unlike a kernel proposed method.

Hurrah for OpenOffice. Now all you need to do is to
persuade Microsoft to store Word files in the same
format......

Jeremy.

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 10:30, skreiv Alan Cox:

> Openoffice does this in user space and the user space vfs code desktops
> use can handle zips so this "just works" already including over NFS,
> unlike a kernel proposed method.

NFS is not the roadblock here. Sure v2 and v3 support would be
impossible to do, but since this is all 2.6 work anyway, people have the
option of using NFSv4, which has full support for streams...

NFSv4 has an OPENATTR call which acts on files to return a filehandle
that works fine with both READDIR and LOOKUP, so if a VFS interface for
streams existed, we could code up full support tomorrow.
As it is, we're having to shoehorn this into the xattr interface. 8-(

Cheers,
  Trond

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 10:30, skreiv Alan Cox:

> Openoffice does this in user space and the user space vfs code desktops
> use can handle zips so this "just works" already including over NFS,
> unlike a kernel proposed method.

NFS is not the roadblock here. Sure v2 and v3 support would be
impossible to do, but since this is all 2.6 work anyway, people have the
option of using NFSv4, which has full support for streams...

NFSv4 has an OPENATTR call which acts on files to return a filehandle
that works fine with both READDIR and LOOKUP, so if a VFS interface for
streams existed, we could code up full support tomorrow.
As it is, we're having to shoehorn this into the xattr interface. 8-(

Cheers,
  Trond

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Sun, Aug 29, 2004 at 02:41:04PM -0400, Trond Myklebust wrote:
> 
> NFSv4 has an OPENATTR call which acts on files to return a filehandle
> that works fine with both READDIR and LOOKUP, so if a VFS interface for
> streams existed, we could code up full support tomorrow.
> As it is, we're having to shoehorn this into the xattr interface. 8-(

Yeah, that's what I'm really trying to avoid for Samba.
We're currently using the xattr interface for DOS attributes,
and soon for NT ACL support, but it doesn't really fit as
a streams interface (no lseek() support) although someone in
HP coded up a test interface to NT streams that map into xattrs
(most of the streams in a Word file for instance are quite
small) - but I really don't want to have to do this :-).

Jeremy.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Jeremy Allison wrote:
> (most of the streams in a Word file for instance are quite small)

Streams in a Word file?

Are you saying that when I copy a .doc file onto my Linux box and off,
I lose part of a Word document?

-- Jamie

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Sep 01, 2004 at 09:19:45PM +0100, Jamie Lokier wrote:
> Jeremy Allison wrote:
> > (most of the streams in a Word file for instance are quite small)
> 
> Streams in a Word file?

Yep.

> Are you saying that when I copy a .doc file onto my Linux box and off,
> I lose part of a Word document?

Right now no, because when Samba refuses the stream open, Word falls
back into a "tar"-like mode where it linearises the streams into the
data (it's a legacy mode for storing data on a FAT drive, not an NTFS
drive). However, the problem is that no currently supported Microsoft
OS doesn't have streams-capable NTFS support. 

This means that in a future MS-Office revision, this backwards support
may be broken by accident or by design (less likely, Microsoft really
don't do that kind of thing without very high level requests :-) and no
testers at Microsoft will notice (because they only test against MS servers).

Jeremy.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Jeremy Allison wrote:
> > Streams in a Word file?
> 
> Yep.
> 
> > Are you saying that when I copy a .doc file onto my Linux box and off,
> > I lose part of a Word document?
> 
> Right now no, because when Samba refuses the stream open, Word falls
> back into a "tar"-like mode where it linearises the streams into the
> data (it's a legacy mode for storing data on a FAT drive, not an NTFS
> drive). However, the problem is that no currently supported Microsoft
> OS doesn't have streams-capable NTFS support. 

I meant when I copy not using Samba.  For example, I copy the .doc
file in Windows NT to an FTP server.

Does the FTP operation magically linearise the .doc streams on demand?

Or does FTP lose part of the Word document?

-- Jamie

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Sep 01, 2004 at 09:31:01PM +0100, Jamie Lokier wrote:
> 
> I meant when I copy not using Samba.  For example, I copy the .doc
> file in Windows NT to an FTP server.
> 
> Does the FTP operation magically linearise the .doc streams on demand?
> 
> Or does FTP lose part of the Word document?

Good question. It depends if the Microsoft ftp client is streams-aware,
and understands the Microsoft OLE structured storage format and will do
the linearisation on demand or not. I must confess I haven't tested this,
as I don't ever run Windows other than on vmware sessions for Samba testing
these days :-).

Probably a non-Microsoft ftp client would lose part of the word doc.

Jeremy.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Jeremy Allison wrote:
> > I meant when I copy not using Samba.  For example, I copy the .doc
> > file in Windows NT to an FTP server.
> > 
> > Does the FTP operation magically linearise the .doc streams on demand?
> > Or does FTP lose part of the Word document?
> 
> Good question. It depends if the Microsoft ftp client is streams-aware,
> and understands the Microsoft OLE structured storage format and will do
> the linearisation on demand or not. I must confess I haven't tested this,
> as I don't ever run Windows other than on vmware sessions for Samba testing
> these days :-).
> 
> Probably a non-Microsoft ftp client would lose part of the word doc.

So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
will _all_ lose part of a Word document when they handle it on a
Window box?

Ouch!

The only sensible implementation I can imagine would be if the OS
linearised multi-stream Word documents into the non-stream format
automatically for all programs which don't know about streams.

Which is of course what I would like to implement for Linux...

- Jamie

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote:
> Jeremy Allison wrote:
> > > I meant when I copy not using Samba.  For example, I copy the .doc
> > > file in Windows NT to an FTP server.
> > > 
> > > Does the FTP operation magically linearise the .doc streams on demand?
> > > Or does FTP lose part of the Word document?
> > 
> > Good question. It depends if the Microsoft ftp client is streams-aware,
> > and understands the Microsoft OLE structured storage format and will do
> > the linearisation on demand or not. I must confess I haven't tested this,
> > as I don't ever run Windows other than on vmware sessions for Samba testing
> > these days :-).
> > 
> > Probably a non-Microsoft ftp client would lose part of the word doc.
> 
> So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> will _all_ lose part of a Word document when they handle it on a
> Window box?
> 
> Ouch!

Yep. It's the meta data that Word stores in streams that will get lost.

Jeremy.

Re: silent semantic changes with reiser4

[Lee Revell]

On Wed, 2004-09-01 at 16:51, Jeremy Allison wrote:
> On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote:
> > Jeremy Allison wrote:
> > > > I meant when I copy not using Samba.  For example, I copy the .doc
> > > > file in Windows NT to an FTP server.
> > > > 
> > > > Does the FTP operation magically linearise the .doc streams on demand?
> > > > Or does FTP lose part of the Word document?
> > > 
> > > Good question. It depends if the Microsoft ftp client is streams-aware,
> > > and understands the Microsoft OLE structured storage format and will do
> > > the linearisation on demand or not. I must confess I haven't tested this,
> > > as I don't ever run Windows other than on vmware sessions for Samba testing
> > > these days :-).
> > > 
> > > Probably a non-Microsoft ftp client would lose part of the word doc.
> > 
> > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > will _all_ lose part of a Word document when they handle it on a
> > Window box?
> > 
> > Ouch!
> 
> Yep. It's the meta data that Word stores in streams that will get lost.
> 

This is shocking.  When was this behavior introduced?

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Wed, 2004-09-01 at 16:51, Jeremy Allison wrote:
> On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote:
> > Jeremy Allison wrote:
> > > > I meant when I copy not using Samba.  For example, I copy the .doc
> > > > file in Windows NT to an FTP server.
> > > > 
> > > > Does the FTP operation magically linearise the .doc streams on demand?
> > > > Or does FTP lose part of the Word document?
> > > 
> > > Good question. It depends if the Microsoft ftp client is streams-aware,
> > > and understands the Microsoft OLE structured storage format and will do
> > > the linearisation on demand or not. I must confess I haven't tested this,
> > > as I don't ever run Windows other than on vmware sessions for Samba testing
> > > these days :-).
> > > 
> > > Probably a non-Microsoft ftp client would lose part of the word doc.
> > 
> > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > will _all_ lose part of a Word document when they handle it on a
> > Window box?
> > 
> > Ouch!
> 
> Yep. It's the meta data that Word stores in streams that will get lost.
> 

This is shocking.  When was this behavior introduced?

Lee

Re: silent semantic changes with reiser4

[Oliver Hunt]

The loss of forks in the file is exxactly the problem you used to have
when transferring native Mac files to a PC...

This meant in order to transfer files to different filesystem you
often needed to tar/zip/whatever them first.

Bare in mind this would let us do the whole MacOS thing of putting an
entire application(plus plugins, etc) inside one "file"...

--Oliver

On Wed, 1 Sep 2004 13:51:40 -0700, Jeremy Allison <jra@samba.org> wrote:
> On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote:
> > Jeremy Allison wrote:
> > > > I meant when I copy not using Samba.  For example, I copy the .doc
> > > > file in Windows NT to an FTP server.
> > > >
> > > > Does the FTP operation magically linearise the .doc streams on demand?
> > > > Or does FTP lose part of the Word document?
> > >
> > > Good question. It depends if the Microsoft ftp client is streams-aware,
> > > and understands the Microsoft OLE structured storage format and will do
> > > the linearisation on demand or not. I must confess I haven't tested this,
> > > as I don't ever run Windows other than on vmware sessions for Samba testing
> > > these days :-).
> > >
> > > Probably a non-Microsoft ftp client would lose part of the word doc.
> > 
> > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > will _all_ lose part of a Word document when they handle it on a
> > Window box?
> >
> > Ouch!
> 
> Yep. It's the meta data that Word stores in streams that will get lost.
> 
> 
> 
> Jeremy.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: silent semantic changes with reiser4

[Theodore Ts'o]

On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote:
> > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > will _all_ lose part of a Word document when they handle it on a
> > Window box?
> > 
> > Ouch!
> 
> Yep. It's the meta data that Word stores in streams that will get lost.

And this is why I believe that using streams in application is well,
ill-advised.  Indeed, one of my concerns with providing streams
support is that application authors may make the mistake of using it,
and we will be back to the bad old days (when MacOS made this mistake)
where you will need to binhex files before you ftp them (and unbinhex
them on the otherside) --- and if you forget, the resulting file will
be useless.

I understand why the Samba folks want this feature very badly;
however, hopefully other projects will know enough *not* to use
streams once they become available in Linux....

						- Ted

Re: silent semantic changes with reiser4

[Stuart Young]

On Thu, 2 Sep 2004 22:54, Theodore Ts'o wrote:
> On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote:
> > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > > will _all_ lose part of a Word document when they handle it on a
> > > Window box?
> > >
> > > Ouch!
> >
> > Yep. It's the meta data that Word stores in streams that will get lost.
>
> And this is why I believe that using streams in application is well,
> ill-advised.  Indeed, one of my concerns with providing streams
> support is that application authors may make the mistake of using it,
> and we will be back to the bad old days (when MacOS made this mistake)
> where you will need to binhex files before you ftp them (and unbinhex
> them on the otherside) --- and if you forget, the resulting file will
> be useless.

At least currently (to my knowledge anyway) all stream support in Windows is 
data that is not important, and that can be either regenerated from 
filesystem metadata or (more usually) the main file stream itself.

This sort of data is really where streams excel, by providing a way to access 
data that would otherwise take time/cpu to regenerate over and over, but that 
in itself is not indispensable. Good examples of this are indexes of data 
within a document, details of who owns/created/modified the document, common 
views or reformatting of the data, etc. With audio/video/graphics, you could 
store lower quality transforms of data (eg: stereo to mono, resolution 
reduction, thumbnails, etc) in the streams for a file. With a word document, 
it could be things like an index (assuming it's auto-generated from section 
headings). With a database, it could be the indexes, and a few views that are 
expensive time-wise to generate. All of these are easily regenerated from the 
original data stream, but takes a while. And if you've got the disk, why not 
use it?

If streams were always to be considered volatile, then you could do all sorts 
of interesting things with them. Any disk cleanup mechanism you have could 
also reap old streams specifically if the disk gets below a certain amount 
free. This means that old streams that are hanging about don't end up wasting 
all your disk space. Of course, you'd want a way to disable this (for servers 
mainly), and streams would have to be considered volatile on more than just 
Linux as a platform for this to be useful.

Note that I'm not particularly advocating streams here. I'm just pointing out 
'how' it could be useful. It could be very easy to misuse streams and cause 
huge problems (as per Ted's comments), but it's always good to know the other 
side of the argument.

-- 
 Stuart Young (aka Cef)
 cef-lkml@optusnet.com.au is for LKML and related email only

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Jeremy" == Jeremy Allison <jra@samba.org> writes:

Jeremy> Good question. It depends if the Microsoft ftp client is
Jeremy> streams-aware, and understands the Microsoft OLE structured
Jeremy> storage format and will do the linearisation on demand or not. I
Jeremy> must confess I haven't tested this, as I don't ever run Windows
Jeremy> other than on vmware sessions for Samba testing these days :-).

Jeremy> Probably a non-Microsoft ftp client would lose part of the word
Jeremy> doc.

What about email.  If someone uses a non-MS mail client, it'll lose
part of the Word doc?  Does that mean nobody can email me Word docs any
more?   <pause>  Wait a sec, that's a good thing! ;-)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Jeremy" == Jeremy Allison <jra@samba.org> writes:

Jeremy> Good question. It depends if the Microsoft ftp client is
Jeremy> streams-aware, and understands the Microsoft OLE structured
Jeremy> storage format and will do the linearisation on demand or not. I
Jeremy> must confess I haven't tested this, as I don't ever run Windows
Jeremy> other than on vmware sessions for Samba testing these days :-).

Jeremy> Probably a non-Microsoft ftp client would lose part of the word
Jeremy> doc.

What about email.  If someone uses a non-MS mail client, it'll lose
part of the Word doc?  Does that mean nobody can email me Word docs any
more?   <pause>  Wait a sec, that's a good thing! ;-)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[David Lang]

what happens when you ue one of the many software packages around that 
lets windows access a NFS server. when you copy the file to a NFS accessed 
drive does it loose part of the data?

David Lang

  On Wed, 1 Sep 2004, Jamie Lokier 
wrote:

> Date: Wed, 1 Sep 2004 21:31:01 +0100
> From: Jamie Lokier <jamie@shareable.org>
> To: Jeremy Allison <jra@samba.org>
> Cc: Trond Myklebust <trond.myklebust@fys.uio.no>,
>     Alan Cox <alan@lxorguk.ukuu.org.uk>,
>     Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua>,
>     Rik van Riel <riel@redhat.com>, Christer Weinigel <christer@weinigel.se>,
>     Spam <spam@tnonline.net>, Andrew Morton <akpm@osdl.org>,
>     wichert@wiggy.net, Linus Torvalds <torvalds@osdl.org>, reiser@namesys.com,
>     hch@lst.de, Linux Filesystem Development <linux-fsdevel@vger.kernel.org>,
>     Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, flx@namesys.com,
>     reiserfs-list@namesys.com
> Subject: Re: silent semantic changes with reiser4
> 
> Jeremy Allison wrote:
>>> Streams in a Word file?
>>
>> Yep.
>>
>>> Are you saying that when I copy a .doc file onto my Linux box and off,
>>> I lose part of a Word document?
>>
>> Right now no, because when Samba refuses the stream open, Word falls
>> back into a "tar"-like mode where it linearises the streams into the
>> data (it's a legacy mode for storing data on a FAT drive, not an NTFS
>> drive). However, the problem is that no currently supported Microsoft
>> OS doesn't have streams-capable NTFS support.
>
> I meant when I copy not using Samba.  For example, I copy the .doc
> file in Windows NT to an FTP server.
>
> Does the FTP operation magically linearise the .doc streams on demand?
>
> Or does FTP lose part of the Word document?
>
> -- Jamie
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Alan Cox]

On Mer, 2004-09-01 at 21:26, Jeremy Allison wrote:
> Right now no, because when Samba refuses the stream open, Word falls
> back into a "tar"-like mode where it linearises the streams into the
> data (it's a legacy mode for storing data on a FAT drive, not an NTFS
> drive). However, the problem is that no currently supported Microsoft
> OS doesn't have streams-capable NTFS support. 

That implies that samba can do the same transform set without kernel
help and you'd even get the advantages of being able to transfer the
stuff around sanely afterwards.

What I don't understand is the tie between Linux having such streams and
Windows doing it for Samba to work. Netatalk has always handle this for
Macintosh and portably. Presumably any Samba support would need to
handle OS's without wacky files for portability too ?

Re: silent semantic changes with reiser4

[Oliver Neukum]

Am Donnerstag, 2. September 2004 11:48 schrieb Alan Cox:
> What I don't understand is the tie between Linux having such streams and
> Windows doing it for Samba to work. Netatalk has always handle this for
> Macintosh and portably. Presumably any Samba support would need to
> handle OS's without wacky files for portability too ?

Can you do an atomic rename of all streams without kernel support?

	Regards
		Oliver

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-09-02 at 12:13, Oliver Neukum wrote:
> Am Donnerstag, 2. September 2004 11:48 schrieb Alan Cox:
> > What I don't understand is the tie between Linux having such streams and
> > Windows doing it for Samba to work. Netatalk has always handle this for
> > Macintosh and portably. Presumably any Samba support would need to
> > handle OS's without wacky files for portability too ?
> 
> Can you do an atomic rename of all streams without kernel support?

That depends how SAMBA chooses to handle the problem internally and how
it chooses to store the data. The netatalk people have atomicity from
the view of clients but not from the unix fs internal view.

Alan

Re: silent semantic changes with reiser4

[Jamie Lokier]

Alan Cox wrote:
> On Iau, 2004-08-26 at 16:19, Denis Vlasenko wrote:
> > > Office suites can store a document with embedded images
> > > and spread sheets "easily" by putting the text, the
> > > images and spread sheets all in different file streams.
> > 
> > Kinda far reaching. It's hard to assess whether that is good or bad idea.
> > Can we start small, like putting metadata (e.g. ACLs) into these streams?
> 
> Openoffice does this in user space and the user space vfs code desktops
> use can handle zips so this "just works" already including over NFS,
> unlike a kernel proposed method.

OpenOffice is very sensible, and I agree that a method of handling
document parts that _requires_ kernel support would be foolish.

However, please see my message "The argument for fs assistance in
handling archives" for how kernel support can improve performance of
programs like OpenOffice on large compound documents, while preserving
the sensible behaviour and remaining portable.

-- Jamie

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 16:00, Denis Vlasenko wrote:
>
> I think Hans is not planning turning old "file is a stream of bytes"
> into eight-stream octopus. One stream will remain as a 'main' one,
> which contains actual data. Others will keep metadata, etc...
>
> Note that even today's files aren't simple "streams of bytes"
> because they also have names,mode bits etc.

This is just exactly how NTFS does work: there's an unnamed stream (which is 
the one that the majority of userspace programs do manipulate) which contains 
the user data (the pixels from a PMG image, or text from a text file, etc.), 
and there are named streams which not all kind of applications do know how to 
handle.

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Andrew Morton wrote:

> All of which can be handled in userspace library code.
> 
> What compelling reason is there for doing this in the kernel?

There's a compelling reason to do it in userspace.  If an
unaware program copies or moves such a file with streams
inside, it doesn't break the streams and aware programs will
continue to see them.

OTOH, if we had the streams in the kernel, unaware applications
would continuously break the metadata and streams that the
streams aware programs expect !

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Andrew Morton wrote:

> All of which can be handled in userspace library code.
> 
> What compelling reason is there for doing this in the kernel?

There's a compelling reason to do it in userspace.  If an
unaware program copies or moves such a file with streams
inside, it doesn't break the streams and aware programs will
continue to see them.

OTOH, if we had the streams in the kernel, unaware applications
would continuously break the metadata and streams that the
streams aware programs expect !

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> > All of which can be handled in userspace library code.
> > 
> > What compelling reason is there for doing this in the kernel?
> 
> There's a compelling reason to do it in userspace.  If an
> unaware program copies or moves such a file with streams
> inside, it doesn't break the streams and aware programs will
> continue to see them.
> 
> OTOH, if we had the streams in the kernel, unaware applications
> would continuously break the metadata and streams that the
> streams aware programs expect !

You appear not to have read any of my mails on this topic.

Properly implemented metadata can:

  (1) operate in both modes simultaneously;
  (2) work with unaware applications;
  (3) provide performance enhancements to aware applications;
  (4) provide storage enhancements to both;
  (5) provide useful features that work with standard unmodified unix tools,

all at once.  That includes program copies.

-- Jamie

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Jamie Lokier wrote:

> Properly implemented metadata can:
[snip world peace]
> all at once.  That includes program copies.

Show me the design for such a properly implemented thing.

None of the designs I've seen proposed so far are able
to do what you claim.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Jamie Lokier wrote:

> Properly implemented metadata can:
[snip world peace]
> all at once.  That includes program copies.

Show me the design for such a properly implemented thing.

None of the designs I've seen proposed so far are able
to do what you claim.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Andrew Morton wrote:
>
>  
>
>>All of which can be handled in userspace library code.
>>
>>What compelling reason is there for doing this in the kernel?
>>    
>>
>
>There's a compelling reason to do it in userspace.  If an
>unaware program copies or moves such a file with streams
>inside, it doesn't break the streams and aware programs will
>continue to see them.
>
>OTOH, if we had the streams in the kernel, unaware applications
>would continuously break the metadata and streams that the
>streams aware programs expect !
>
>  
>
Well, first off, you don't want streams in the kernel, you want all the 
little pieces that can be composed together into a stream if you so 
choose.  Streams are ugly, the pieces are all cool.

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Andrew Morton wrote:
>
>  
>
>>All of which can be handled in userspace library code.
>>
>>What compelling reason is there for doing this in the kernel?
>>    
>>
>
>There's a compelling reason to do it in userspace.  If an
>unaware program copies or moves such a file with streams
>inside, it doesn't break the streams and aware programs will
>continue to see them.
>
>OTOH, if we had the streams in the kernel, unaware applications
>would continuously break the metadata and streams that the
>streams aware programs expect !
>
>  
>
Well, first off, you don't want streams in the kernel, you want all the 
little pieces that can be composed together into a stream if you so 
choose.  Streams are ugly, the pieces are all cool.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Thu, Aug 26, 2004 at 02:49:56AM -0700, Andrew Morton wrote:
> Spam <spam@tnonline.net> wrote:
> >
> >    Yes,  for  example  documents,  image  files  etc. The multiple data
> >    streams  can  contain thumbnails, info about who is editing the file
> >    (useful for networked files) etc. Could be used for version handling
> >    and much more.
> 
> All of which can be handled in userspace library code.
> 
> What compelling reason is there for doing this in the kernel?

Because without kernel support there is no way someone can
publish a new metadata type and have it automatically supported
by all application data files (ie. most apps ignore it, and only
apps that are aware of it can see it). Without kernel support
you have to have all apps agree on a data format. And
that's harder to do than getting linux kernel VFS engineers
to agree on things :-).

Jeremy.

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Jeremy Allison wrote:

> Because without kernel support there is no way someone can
> publish a new metadata type and have it automatically supported
> by all application data files (ie. most apps ignore it, and only
> apps that are aware of it can see it).

So your backup software ignores it, and after a restore you've
lost your new metadata ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Jeremy Allison wrote:

> Because without kernel support there is no way someone can
> publish a new metadata type and have it automatically supported
> by all application data files (ie. most apps ignore it, and only
> apps that are aware of it can see it).

So your backup software ignores it, and after a restore you've
lost your new metadata ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 20:17, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Jeremy Allison wrote:
> > Because without kernel support there is no way someone can
> > publish a new metadata type and have it automatically supported
> > by all application data files (ie. most apps ignore it, and only
> > apps that are aware of it can see it).
>
> So your backup software ignores it, and after a restore you've
> lost your new metadata ?

I think there's no easy way to fix this, except fixing the backup software to 
support xattrs's/streams/forks/whatever. Even on Windows, it has happened 
this way with older apps that don't support named streams.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Thu, Aug 26, 2004 at 02:17:04PM -0400, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Jeremy Allison wrote:
> 
> > Because without kernel support there is no way someone can
> > publish a new metadata type and have it automatically supported
> > by all application data files (ie. most apps ignore it, and only
> > apps that are aware of it can see it).
> 
> So your backup software ignores it, and after a restore you've
> lost your new metadata ?

Yes, but then you fix your backup software and everyone lives
happily ever after :-)......

It's a better solution than changing *every single application
in existance* (the "userspace library" approach).

IMHO of course.

Jeremy.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Jeremy" == Jeremy Allison <jra@samba.org> writes:

Jeremy> On Thu, Aug 26, 2004 at 02:17:04PM -0400, Rik van Riel wrote:
>> On Thu, 26 Aug 2004, Jeremy Allison wrote:
>> > Because without kernel support there is no way someone can publish
>> > a new metadata type and have it automatically supported by all
>> > application data files (ie. most apps ignore it, and only apps that
>> > are aware of it can see it).
>> 
>> So your backup software ignores it, and after a restore you've lost
>> your new metadata ?

Jeremy> Yes, but then you fix your backup software and everyone lives
Jeremy> happily ever after :-)......

Also, once you add support for file-as-dir, you can export any new
metadata that you want through that interface, so backup software will
be able to handle any new stuff you want to throw at it (ACLs, xattrs,
streams, etc.).

If anything, I'd say that needing to fix backup software is an argument
*for* adding file-as-dir semantics: no need to make any more changes
after this one (and it shouldn't be that big of a change, since backup
software already knows how to handle files, and how to handle dirs).

Jeremy> IMHO of course.

ditto.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Jeremy" == Jeremy Allison <jra@samba.org> writes:

Jeremy> On Thu, Aug 26, 2004 at 02:17:04PM -0400, Rik van Riel wrote:
>> On Thu, 26 Aug 2004, Jeremy Allison wrote:
>> > Because without kernel support there is no way someone can publish
>> > a new metadata type and have it automatically supported by all
>> > application data files (ie. most apps ignore it, and only apps that
>> > are aware of it can see it).
>> 
>> So your backup software ignores it, and after a restore you've lost
>> your new metadata ?

Jeremy> Yes, but then you fix your backup software and everyone lives
Jeremy> happily ever after :-)......

Also, once you add support for file-as-dir, you can export any new
metadata that you want through that interface, so backup software will
be able to handle any new stuff you want to throw at it (ACLs, xattrs,
streams, etc.).

If anything, I'd say that needing to fix backup software is an argument
*for* adding file-as-dir semantics: no need to make any more changes
after this one (and it shouldn't be that big of a change, since backup
software already knows how to handle files, and how to handle dirs).

Jeremy> IMHO of course.

ditto.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hans Reiser]

Well said.

Jeremy Allison wrote:

> Yes, but then you fix your backup software and everyone lives
>
>happily ever after :-)......
>
>It's a better solution than changing *every single application
>in existance* (the "userspace library" approach).
>
>IMHO of course.
>
>Jeremy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Well said.

Jeremy Allison wrote:

> Yes, but then you fix your backup software and everyone lives
>
>happily ever after :-)......
>
>It's a better solution than changing *every single application
>in existance* (the "userspace library" approach).
>
>IMHO of course.
>
>Jeremy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

I think Jeremy is accurate in saying the below.

Jeremy Allison wrote:

>>What compelling reason is there for doing this in the kernel?
>>    
>>
>
>
>Because without kernel support there is no way someone can
>publish a new metadata type and have it automatically supported
>by all application data files (ie. most apps ignore it, and only
>apps that are aware of it can see it). Without kernel support
>you have to have all apps agree on a data format. And
>that's harder to do than getting linux kernel VFS engineers
>to agree on things :-).
>
>Jeremy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

I think Jeremy is accurate in saying the below.

Jeremy Allison wrote:

>>What compelling reason is there for doing this in the kernel?
>>    
>>
>
>
>Because without kernel support there is no way someone can
>publish a new metadata type and have it automatically supported
>by all application data files (ie. most apps ignore it, and only
>apps that are aware of it can see it). Without kernel support
>you have to have all apps agree on a data format. And
>that's harder to do than getting linux kernel VFS engineers
>to agree on things :-).
>
>Jeremy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Anton Altaparmakov]

On Thu, 2004-08-26 at 00:32, Andrew Morton wrote:
> Spam <spam@tnonline.net> wrote:
> >
> > > In other words, if reiserfs does something special, we should make 
> > > standard interfaces for doing that special thing, so that everybody can
> > > do it without stepping on other peoples toes.
> > 
> >   Agreed  that would be the best. But how much time and effort will it
> >   be
> 
> Zero.
> 
> We can add these new features tomorrow, as reiser4-only features, with a
> plan in hand to generalise them later.
> 
> -->>__if__<<-- we think these are features which Linux should offer.

Please don't forget that if the reiser4 features are merged as they are
now, then we will likely be stuck with the API reiser4 chooses.  There
will be tools that will rely on it springing up no doubt.

Moving the reiser4 features to VFS later is fine and good, but what if
the VFS doesn't want the same API for those features?  Either we would
have to allow reiser4 to continue providing the old API even though the
VFS now provides a new, shiny API or we would have to break all existing
API users on reiser4.  Things like "I rebooted into the latest kernel
and my computer failed to boot because essential app FOO failed to
access the reiser4 API - Help!" spring to mind.

Yes, I know I am painting a rather black picture here and I know you
might well say "screw apps", its been done plenty of times in Linux
kernel development before...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/

Re: silent semantic changes with reiser4

[Anton Altaparmakov]

On Thu, 2004-08-26 at 00:32, Andrew Morton wrote:
> Spam <spam@tnonline.net> wrote:
> >
> > > In other words, if reiserfs does something special, we should make 
> > > standard interfaces for doing that special thing, so that everybody can
> > > do it without stepping on other peoples toes.
> > 
> >   Agreed  that would be the best. But how much time and effort will it
> >   be
> 
> Zero.
> 
> We can add these new features tomorrow, as reiser4-only features, with a
> plan in hand to generalise them later.
> 
> -->>__if__<<-- we think these are features which Linux should offer.

Please don't forget that if the reiser4 features are merged as they are
now, then we will likely be stuck with the API reiser4 chooses.  There
will be tools that will rely on it springing up no doubt.

Moving the reiser4 features to VFS later is fine and good, but what if
the VFS doesn't want the same API for those features?  Either we would
have to allow reiser4 to continue providing the old API even though the
VFS now provides a new, shiny API or we would have to break all existing
API users on reiser4.  Things like "I rebooted into the latest kernel
and my computer failed to boot because essential app FOO failed to
access the reiser4 API - Help!" spring to mind.

Yes, I know I am painting a rather black picture here and I know you
might well say "screw apps", its been done plenty of times in Linux
kernel development before...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/

Re: silent semantic changes with reiser4

[Christian Mayrhuber]

On Thursday 26 August 2004 11:03, Anton Altaparmakov wrote:

> Please don't forget that if the reiser4 features are merged as they are
> now, then we will likely be stuck with the API reiser4 chooses.  There
> will be tools that will rely on it springing up no doubt.
>
> Moving the reiser4 features to VFS later is fine and good, but what if
> the VFS doesn't want the same API for those features?  Either we would
> have to allow reiser4 to continue providing the old API even though the
> VFS now provides a new, shiny API or we would have to break all existing
> API users on reiser4.  Things like "I rebooted into the latest kernel
> and my computer failed to boot because essential app FOO failed to
> access the reiser4 API - Help!" spring to mind.

Andrew Morton wrote:
>b) accept the reiser4-only extensions with a view to turning them into
>   kernel-wide extensions at some time in the future, so all filesystems
>   will offer the extensions (as much as poss) or

If option b) is chosen Reiser4 can become a playground.

There is the reiser4() syscall which you surely don't want to implement for 
other filesystems.
Once there is some experience with this new fancy stuff the dust what
is useful/insecure, etc. and what is not will settle and can be condensed
into a vfs api.
Apps like samba and user scripts will have to be adapted once this is
the case, but this should not be to big a problem if this stuff is marked 
experimental.

People which want something stable can continue to use xattrs and a
magnitude of filesystems for now.

-- 
lg, Chris

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 12:45:47PM +0200, Christian Mayrhuber wrote:
> There is the reiser4() syscall which you surely don't want to implement for 
> other filesystems.

Actually a syscall that goes into filesystem code is the last thing we want.
But it's not in the submission, so let's keep that flame^H^H^H^Hdiscussion
for later.

> Once there is some experience with this new fancy stuff the dust what
> is useful/insecure, etc. and what is not will settle and can be condensed
> into a vfs api.
> Apps like samba and user scripts will have to be adapted once this is
> the case, but this should not be to big a problem if this stuff is marked 
> experimental.
> 
> People which want something stable can continue to use xattrs and a
> magnitude of filesystems for now.

Sure, no one stops you from playing around with new semantics.  But please
don't add them to the linux kernel stable series until we have semantics we
a) want to stick to for a while and b) actually work.

Re: silent semantic changes with reiser4

[Olivier Galibert]

On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote:
> Sure, no one stops you from playing around with new semantics.  But please
> don't add them to the linux kernel stable series until we have semantics we
> a) want to stick to for a while and b) actually work.

He's not proposing to add it to 2.4, is he?

  OG.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
> On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote:
> > Sure, no one stops you from playing around with new semantics.  But please
> > don't add them to the linux kernel stable series until we have semantics we
> > a) want to stick to for a while and b) actually work.
> 
> He's not proposing to add it to 2.4, is he?

We're talking about 2.6 here.  2.4 is the obsolete kernel series these days.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote:
>On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
>> He's not proposing to add it to 2.4, is he?
>We're talking about 2.6 here.  2.4 is the obsolete kernel series these days.

So run it in -mm or fork it to 2.7.

2.7 might not be a bad idea for trying such bleeding edge revolution
stuff as this.

-- 
mjt

Re: silent semantic changes with reiser4

[mjt]

On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote:
>On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
>> He's not proposing to add it to 2.4, is he?
>We're talking about 2.6 here.  2.4 is the obsolete kernel series these days.

So run it in -mm or fork it to 2.7.

2.7 might not be a bad idea for trying such bleeding edge revolution
stuff as this.

-- 
mjt

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote:
>On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
>> He's not proposing to add it to 2.4, is he?
>We're talking about 2.6 here.  2.4 is the obsolete kernel series these days.

So run it in -mm or fork it to 2.7.

2.7 might not be a bad idea for trying such bleeding edge revolution
stuff as this.

-- 
mjt

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:49:29PM +0300, Markus Törnqvist wrote:
> 2.7 might not be a bad idea for trying such bleeding edge revolution
> stuff as this.

I already suggested to Linus that 2.7 is probably the right place to
experiment with that.

Re: silent semantic changes with reiser4

[Christian Mayrhuber]

On Thursday 26 August 2004 14:49, Markus Törnqvist wrote:
> On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote:
> >On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
> >> He's not proposing to add it to 2.4, is he?
> >
> >We're talking about 2.6 here.  2.4 is the obsolete kernel series these
> > days.
>
> So run it in -mm or fork it to 2.7.
>
> 2.7 might not be a bad idea for trying such bleeding edge revolution
> stuff as this.

If I remember right, there are now some reasons for a 2.7:

1) udev / dynamic device numbers
2) enhanced filesystem semantics
maybe resulting in
3) visible automounter mountpoints

-- 
lg, Chris

Re: silent semantic changes with reiser4

[Christian Mayrhuber]

On Thursday 26 August 2004 14:49, Markus Törnqvist wrote:
> On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote:
> >On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
> >> He's not proposing to add it to 2.4, is he?
> >
> >We're talking about 2.6 here.  2.4 is the obsolete kernel series these
> > days.
>
> So run it in -mm or fork it to 2.7.
>
> 2.7 might not be a bad idea for trying such bleeding edge revolution
> stuff as this.

If I remember right, there are now some reasons for a 2.7:

1) udev / dynamic device numbers
2) enhanced filesystem semantics
maybe resulting in
3) visible automounter mountpoints

-- 
lg, Chris

Re: silent semantic changes with reiser4

[Adrian Bunk]

On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
> On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote:
> > Sure, no one stops you from playing around with new semantics.  But please
> > don't add them to the linux kernel stable series until we have semantics we
> > a) want to stick to for a while and b) actually work.
> 
> He's not proposing to add it to 2.4, is he?

2.6 is a stable series...

>   OG.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: silent semantic changes with reiser4

[Hans Reiser]

Spam wrote:

>
>>  The fact that one filesystem will
>>offer features which other filesystems do not and cannot offer makes me
>>queasy for some reason.
>>    
>>
>
>  This last sentence makes me wonder. Where is Linux heading? The idea
>  that   a  FS  cannot  contain  features that no other FS has is very
>  scary.
>
>  I  am  all  for  uniformity, but not at the expense of shutting down
>  advanced progress that Linux is so badly needing.
>
>  This talk about old UNIX seems like people want to still live in the
>  70'ies and not look forward. Please wake up!
>
>  ~S
>
>
>
>
>  
>
well said.

Re: silent semantic changes with reiser4

[Hans Reiser]

Spam wrote:

>
>>  The fact that one filesystem will
>>offer features which other filesystems do not and cannot offer makes me
>>queasy for some reason.
>>    
>>
>
>  This last sentence makes me wonder. Where is Linux heading? The idea
>  that   a  FS  cannot  contain  features that no other FS has is very
>  scary.
>
>  I  am  all  for  uniformity, but not at the expense of shutting down
>  advanced progress that Linux is so badly needing.
>
>  This talk about old UNIX seems like people want to still live in the
>  70'ies and not look forward. Please wake up!
>
>  ~S
>
>
>
>
>  
>
well said.

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>I had not intended to respond to this because I have nothing positive to 
>>say, but Andrew said I needed to respond and suggested I should copy 
>>Linus.
>>    
>>
>
>Yes, but I didn't say "flame Christoph and ignore the issues" ;)
>  
>
Oh....;-)

>There are lots of little things to do with implementation, coding style,
>module exports, deadlocks, what code goes where, etc.  These are all normal
>daily kernel business and we should set them aside for now and concentrate
>on the bigger issues.
>  
>
Yes, you are right, but I am not sure Viro will go along with that..... ;-)

>And as I see it, there are two big issues:
>
>a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
>   anticipate and 
>
>b) it does this within the context of just a single filesystem.
>
>I see three possible responses:
>
>a) accept the reiser4-only extensions as-is (possibly with post-review
>   modifications, of course) or
>
>b) accept the reiser4-only extensions with a view to turning them into
>   kernel-wide extensions at some time in the future, so all filesystems
>   will offer the extensions (as much as poss) or
>
>c) reject the extensions.
>
>
>My own order of preference is b) c) a). 
>
I don't object to b), though I think b) should wait for 2.7 and reiser4 
should not.

> The fact that one filesystem will
>offer features which other filesystems do not and cannot offer makes me
>queasy for some reason.
>  
>
Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
for Apple, and that means we need to put search engine and database 
functionality into the filesystem.  It takes 11 years of serious 
research to build a clean storage layer able to handle doing that.  
Reiser4 has done that, finally.  None of the other Linux filesystems 
have.  The next major release of ReiserFS is going to be bursting with 
semantic enhancements, because the prerequisites for them are in place 
now.  None of the other Linux filesystems have those prerequisites.  
They won't be able to keep up with the semantic enhancements.  This 
metafiles and file-directories stuff is actually fairly trivial stuff.

Look guys, in 1993 I anticipated the battle would be here, and I build 
the foundation for a defensive tower right at the spot MS and Apple are 
now maneuvering towards.  Help me get the next level on the tower before 
they get here.  It is one hell of a foundation, they won't be able to 
shake it, their trees are not as powerful.  Don't move reiser4 into vfs, 
use reiser4 as the vfs.  Don't write filesystems, write file plugins and 
disk format plugins and all the other kinds of plugins, and you won't be 
missing any expressive power that you really want....

Give Saveliev and I some credit.  10 years of hard work at an ivory 
tower nobody thought mattered.  Now the battle leaves the browser and 
swings our way.  Don't duplicate that infrastructure, use it. 

There is so much we could use help with if talented people like you 
chose to contribute.

>b) means that at some time in the future we need to hoist the reiser4
>extensions (at a conceptual level) (and probably with restrictions) up into
>the VFS.  This will involve much thought, argument and work.
>
>
>To get us started on this route it would really help me (and, probably,
>others) if you could describe what these API extensions are in a very
>simple way, without referring to incomprehehsible web pages,
>
what is not comprehensible....?

> and without
>using terms which non-reiser4 people don't understand.
>  
>
Well, I agree that there is value in defining things in more detail than 
we have.

>It would be best if each extension was addressed in a separate email
>thread.
>
>We also need to discuss what a reiser4 "module" is, what its capabilities
>are, and what licensing implications they have.
>
>Then, we can look at each one and say "yup, that makes sense - we want
>Linux to do that" and we can also think about how we would implement it at
>the VFS level.
>
>If we follow the above route I believe we can make progress in a technical
>direction and not get deadlocked on personal/political stuff.
>
>
>Now, an alternative to all the above is to just merge reiser4 as-is, after
>addressing all the lower-level coding issues.  And see what happens.  That
>may be a thing which Linus wishes to do.  I'm easy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
>  >To get us started on this route it would really help me (and, probably,
>  >others) if you could describe what these API extensions are in a very
>  >simple way, without referring to incomprehehsible web pages,
>  >
>  what is not comprehensible....?

Pretty much anything at www.namesys.com.  The amount of time which is
needed to extract the technical info which one is looking for vastly
exceeds a gnat-like attention span.

As a starting point, please prepare a bullet-point list of
userspace-visible changes which the filesystem introduces, or is planned to
introduce.

And describe the "plugin" system.  Why does the filesystem need such a
thing (other filesystems get their features via `patch -p1')?

And what are the licensing implications of plugins?  Are they derived
works?  Must they be GPL'ed?

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>> >To get us started on this route it would really help me (and, probably,
>> >others) if you could describe what these API extensions are in a very
>> >simple way, without referring to incomprehehsible web pages,
>> >
>> what is not comprehensible....?
>>    
>>
>
>Pretty much anything at www.namesys.com.  The amount of time which is
>needed to extract the technical info which one is looking for vastly
>exceeds a gnat-like attention span.
>
>As a starting point, please prepare a bullet-point list of
>userspace-visible changes which the filesystem introduces, or is planned to
>introduce.
>
>And describe the "plugin" system.  Why does the filesystem need such a
>thing (other filesystems get their features via `patch -p1')?
>  
>
It takes 6 months or more to become competent to change a usual 
filesystem.  Creating a new reiser4 plugin is a weekend programmer fun 
hack to do.  Weekend programmers matter, because they tend to have 
clever ideas based on understanding a need they have.   How many people 
can easily add new features to ext3 or reiserfs V3?  Very few. 

What happens if you need a disk format change?

Well, in V4, you can easily compose a plugin from plugin methods of 
other plugins, write a little piece of code with the one thing you want 
different, and add it in.  Disk format changes, no big deal, add a new 
disk format plugin, or a new item plugin, or a new node plugin, etc., 
and you got your new format.

There is a huge difference between code that is designed for reuse, and 
code that is not.  That is the difference between V3 and V4.  We were 
looking at our V3 balancing code, and it special cased each different 
kind of item (an item is a piece of something, which the balancing code 
chops objects into as it squeezes them into nodes).  It looked like the 
complexity of the balancing code was going to be N squared, where N was 
the number of different kinds of items.

So, we created item handlers, and wrote balancing code that could slice 
and dice and merge any item that implemented all of the operations 
required of an item handler.  From there it grew, and we made everything 
pluggable.

Adding features to the new code is far less than the time cost of adding 
features to the old code.  I've seen Nikita complain that something 
would take 6 weeks to do (changing key assignment algorithms), and then 
it takes him 3 afternoons, and it was because of the plugins, because 
when we did something similar in V3 it took 3 man months.

>And what are the licensing implications of plugins?  Are they derived
>works?  
>
Yes.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>> >To get us started on this route it would really help me (and, probably,
>> >others) if you could describe what these API extensions are in a very
>> >simple way, without referring to incomprehehsible web pages,
>> >
>> what is not comprehensible....?
>>    
>>
>
>Pretty much anything at www.namesys.com.  The amount of time which is
>needed to extract the technical info which one is looking for vastly
>exceeds a gnat-like attention span.
>
>As a starting point, please prepare a bullet-point list of
>userspace-visible changes which the filesystem introduces, or is planned to
>introduce.
>
>And describe the "plugin" system.  Why does the filesystem need such a
>thing (other filesystems get their features via `patch -p1')?
>  
>
It takes 6 months or more to become competent to change a usual 
filesystem.  Creating a new reiser4 plugin is a weekend programmer fun 
hack to do.  Weekend programmers matter, because they tend to have 
clever ideas based on understanding a need they have.   How many people 
can easily add new features to ext3 or reiserfs V3?  Very few. 

What happens if you need a disk format change?

Well, in V4, you can easily compose a plugin from plugin methods of 
other plugins, write a little piece of code with the one thing you want 
different, and add it in.  Disk format changes, no big deal, add a new 
disk format plugin, or a new item plugin, or a new node plugin, etc., 
and you got your new format.

There is a huge difference between code that is designed for reuse, and 
code that is not.  That is the difference between V3 and V4.  We were 
looking at our V3 balancing code, and it special cased each different 
kind of item (an item is a piece of something, which the balancing code 
chops objects into as it squeezes them into nodes).  It looked like the 
complexity of the balancing code was going to be N squared, where N was 
the number of different kinds of items.

So, we created item handlers, and wrote balancing code that could slice 
and dice and merge any item that implemented all of the operations 
required of an item handler.  From there it grew, and we made everything 
pluggable.

Adding features to the new code is far less than the time cost of adding 
features to the old code.  I've seen Nikita complain that something 
would take 6 weeks to do (changing key assignment algorithms), and then 
it takes him 3 afternoons, and it was because of the plugins, because 
when we did something similar in V3 it took 3 man months.

>And what are the licensing implications of plugins?  Are they derived
>works?  
>
Yes.

Hans

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 05:24, Hans Reiser wrote:
> Andrew Morton wrote:
> >
> >And describe the "plugin" system.  Why does the filesystem need such a
> >thing (other filesystems get their features via `patch -p1')?
> >  
> >
> It takes 6 months or more to become competent to change a usual 
> filesystem.  Creating a new reiser4 plugin is a weekend programmer fun 
> hack to do.  Weekend programmers matter, because they tend to have 
> clever ideas based on understanding a need they have.   How many people 
> can easily add new features to ext3 or reiserfs V3?  Very few. 
> 
> What happens if you need a disk format change?
> 
> Well, in V4, you can easily compose a plugin from plugin methods of 
> other plugins, write a little piece of code with the one thing you want 
> different, and add it in.  Disk format changes, no big deal, add a new 
> disk format plugin, or a new item plugin, or a new node plugin, etc., 
> and you got your new format.
> 

OK, real world example.  My roommate has an AKAI MPC-2000, a very
popular hardware sampler from the 90's.  The disk format is known,there
are a few utilities to edit the disks on a PC and extract the PCM
samples, but there are no tools to mount it on a modern PC.  Are you
saying that, since I know the MPC disk format, I could write a reiser4
plugin to mount an MPC drive?

If so, then Hans has an excellent point.  Users do want this kind of
thing, and it is worth having to fix tar et al.

Lee 

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 05:24, Hans Reiser wrote:
> Andrew Morton wrote:
> >
> >And describe the "plugin" system.  Why does the filesystem need such a
> >thing (other filesystems get their features via `patch -p1')?
> >  
> >
> It takes 6 months or more to become competent to change a usual 
> filesystem.  Creating a new reiser4 plugin is a weekend programmer fun 
> hack to do.  Weekend programmers matter, because they tend to have 
> clever ideas based on understanding a need they have.   How many people 
> can easily add new features to ext3 or reiserfs V3?  Very few. 
> 
> What happens if you need a disk format change?
> 
> Well, in V4, you can easily compose a plugin from plugin methods of 
> other plugins, write a little piece of code with the one thing you want 
> different, and add it in.  Disk format changes, no big deal, add a new 
> disk format plugin, or a new item plugin, or a new node plugin, etc., 
> and you got your new format.
> 

OK, real world example.  My roommate has an AKAI MPC-2000, a very
popular hardware sampler from the 90's.  The disk format is known,there
are a few utilities to edit the disks on a PC and extract the PCM
samples, but there are no tools to mount it on a modern PC.  Are you
saying that, since I know the MPC disk format, I could write a reiser4
plugin to mount an MPC drive?

If so, then Hans has an excellent point.  Users do want this kind of
thing, and it is worth having to fix tar et al.

Lee 

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 923 bytes --]

Am Donnerstag, den 26.08.2004, 15:26 -0400 schrieb Lee Revell:

> > Well, in V4, you can easily compose a plugin from plugin methods of 
> > other plugins, write a little piece of code with the one thing you want 
> > different, and add it in.  Disk format changes, no big deal, add a new 
> > disk format plugin, or a new item plugin, or a new node plugin, etc., 
> > and you got your new format.
> 
> OK, real world example.  My roommate has an AKAI MPC-2000, a very
> popular hardware sampler from the 90's.  The disk format is known,there
> are a few utilities to edit the disks on a PC and extract the PCM
> samples, but there are no tools to mount it on a modern PC.  Are you
> saying that, since I know the MPC disk format, I could write a reiser4
> plugin to mount an MPC drive?

No, the underlying storage must be a reiser4-like tree. If you want to
mount an MPC drive, write an MPC filesystem.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Hans Reiser]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 15:26 -0400 schrieb Lee Revell:
>
>  
>
>>>Well, in V4, you can easily compose a plugin from plugin methods of 
>>>other plugins, write a little piece of code with the one thing you want 
>>>different, and add it in.  Disk format changes, no big deal, add a new 
>>>disk format plugin, or a new item plugin, or a new node plugin, etc., 
>>>and you got your new format.
>>>      
>>>
>>OK, real world example.  My roommate has an AKAI MPC-2000, a very
>>popular hardware sampler from the 90's.  The disk format is known,there
>>are a few utilities to edit the disks on a PC and extract the PCM
>>samples, but there are no tools to mount it on a modern PC.  Are you
>>saying that, since I know the MPC disk format, I could write a reiser4
>>plugin to mount an MPC drive?
>>    
>>
>
>No, the underlying storage must be a reiser4-like tree. 
>
Not necessarily.  We just encourage it....  Reiser4 is a body of code 
that can be sliced and diced as you choose, and it is designed for easy 
slicing.

>If you want to
>mount an MPC drive, write an MPC filesystem.
>
>  
>
However, this last sentence is probably sensible advice.  A mount point 
is probably the right interface from reiser4 for what you want.

Re: silent semantic changes with reiser4

[Hans Reiser]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 15:26 -0400 schrieb Lee Revell:
>
>  
>
>>>Well, in V4, you can easily compose a plugin from plugin methods of 
>>>other plugins, write a little piece of code with the one thing you want 
>>>different, and add it in.  Disk format changes, no big deal, add a new 
>>>disk format plugin, or a new item plugin, or a new node plugin, etc., 
>>>and you got your new format.
>>>      
>>>
>>OK, real world example.  My roommate has an AKAI MPC-2000, a very
>>popular hardware sampler from the 90's.  The disk format is known,there
>>are a few utilities to edit the disks on a PC and extract the PCM
>>samples, but there are no tools to mount it on a modern PC.  Are you
>>saying that, since I know the MPC disk format, I could write a reiser4
>>plugin to mount an MPC drive?
>>    
>>
>
>No, the underlying storage must be a reiser4-like tree. 
>
Not necessarily.  We just encourage it....  Reiser4 is a body of code 
that can be sliced and diced as you choose, and it is designed for easy 
slicing.

>If you want to
>mount an MPC drive, write an MPC filesystem.
>
>  
>
However, this last sentence is probably sensible advice.  A mount point 
is probably the right interface from reiser4 for what you want.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 01:05:45AM -0700, Hans Reiser wrote:
> Not necessarily.  We just encourage it....  Reiser4 is a body of code 
> that can be sliced and diced as you choose, and it is designed for easy 
> slicing.

So what about slicing it into a simple posix fs as a start so we can review
that while still discussing semantic issues.  Now that would be a refreshing
thing after all that talking, no?

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote:
> OK, real world example.  My roommate has an AKAI MPC-2000, a very
> popular hardware sampler from the 90's.  The disk format is known,there
> are a few utilities to edit the disks on a PC and extract the PCM
> samples, but there are no tools to mount it on a modern PC.  Are you
> saying that, since I know the MPC disk format, I could write a reiser4
> plugin to mount an MPC drive?
> 
> If so, then Hans has an excellent point.  Users do want this kind of
> thing, and it is worth having to fix tar et al.

You don't need reiser4 for that, writing read-only linux filesystems is
trivial as soon as you have a specification of the ondisk format.

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 15:34, Christoph Hellwig wrote:
> On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote:
> > OK, real world example.  My roommate has an AKAI MPC-2000, a very
> > popular hardware sampler from the 90's.  The disk format is known,there
> > are a few utilities to edit the disks on a PC and extract the PCM
> > samples, but there are no tools to mount it on a modern PC.  Are you
> > saying that, since I know the MPC disk format, I could write a reiser4
> > plugin to mount an MPC drive?
> > 
> > If so, then Hans has an excellent point.  Users do want this kind of
> > thing, and it is worth having to fix tar et al.
> 
> You don't need reiser4 for that, writing read-only linux filesystems is
> trivial as soon as you have a specification of the ondisk format.
> 

Of course I could just write an MPC filesystem driver.  And, like Andrew
said, a filesystem normally gets new features with 'patch -p1'.  My
question, which was answered by a previous post, was whether the same
could be done in a reiser4 plugins.

Lee

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 15:34, Christoph Hellwig wrote:
> On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote:
> > OK, real world example.  My roommate has an AKAI MPC-2000, a very
> > popular hardware sampler from the 90's.  The disk format is known,there
> > are a few utilities to edit the disks on a PC and extract the PCM
> > samples, but there are no tools to mount it on a modern PC.  Are you
> > saying that, since I know the MPC disk format, I could write a reiser4
> > plugin to mount an MPC drive?
> > 
> > If so, then Hans has an excellent point.  Users do want this kind of
> > thing, and it is worth having to fix tar et al.
> 
> You don't need reiser4 for that, writing read-only linux filesystems is
> trivial as soon as you have a specification of the ondisk format.
> 

Of course I could just write an MPC filesystem driver.  And, like Andrew
said, a filesystem normally gets new features with 'patch -p1'.  My
question, which was answered by a previous post, was whether the same
could be done in a reiser4 plugins.

Lee

Re: silent semantic changes with reiser4

[Dmitry Baryshkov]

Hello,
On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote:
> On Thu, 2004-08-26 at 05:24, Hans Reiser wrote:
> > 
> > Well, in V4, you can easily compose a plugin from plugin methods of 
> > other plugins, write a little piece of code with the one thing you want 
> > different, and add it in.  Disk format changes, no big deal, add a new 
> > disk format plugin, or a new item plugin, or a new node plugin, etc., 
> > and you got your new format.
> > 
> 
> OK, real world example.  My roommate has an AKAI MPC-2000, a very
> popular hardware sampler from the 90's.  The disk format is known,there
> are a few utilities to edit the disks on a PC and extract the PCM
> samples, but there are no tools to mount it on a modern PC.  Are you
> saying that, since I know the MPC disk format, I could write a reiser4
> plugin to mount an MPC drive?
> 

Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4
plugins? From Hans' words it seems so. If this is correct, then maybe
reiser4 core should be updated to completely replace current VFS layer?
Then it's a good point to create a branch (in old development model it
would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and
rewrite all (or at least most used) FS as reiser4 plugins. Then
everybody will be happy.

But this looks too good to be true. Perhaps I misunderstood Hans' words
aboud 'new disk format', did I?

-- 
With best wishes
Dmitry Baryshkov

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov:

> Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4
> plugins? From Hans' words it seems so. If this is correct, then maybe
> reiser4 core should be updated to completely replace current VFS layer?
> Then it's a good point to create a branch (in old development model it
> would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and
> rewrite all (or at least most used) FS as reiser4 plugins. Then
> everybody will be happy.
> 
> But this looks too good to be true. Perhaps I misunderstood Hans' words
> aboud 'new disk format', did I?

No. You can change the format the reiser4 storage tree is stored in. As
long as other filesystems don't use the same underlying storage tree
this is not possible.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov:

> Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4
> plugins? From Hans' words it seems so. If this is correct, then maybe
> reiser4 core should be updated to completely replace current VFS layer?
> Then it's a good point to create a branch (in old development model it
> would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and
> rewrite all (or at least most used) FS as reiser4 plugins. Then
> everybody will be happy.
> 
> But this looks too good to be true. Perhaps I misunderstood Hans' words
> aboud 'new disk format', did I?

No. You can change the format the reiser4 storage tree is stored in. As
long as other filesystems don't use the same underlying storage tree
this is not possible.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Dmitry Baryshkov]

Hello,
On Thu, Aug 26, 2004 at 10:38:12PM +0200, Christophe Saout wrote:
> 
> No. You can change the format the reiser4 storage tree is stored in. As
> long as other filesystems don't use the same underlying storage tree
> this is not possible.
> 

Thank you for clarification. As I wrote, it's too good to be true.

-- 
With best wishes
Dmitry Baryshkov

Re: silent semantic changes with reiser4

[Nikita Danilov]

Christophe Saout writes:
 > Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov:
 > 
 > > Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4
 > > plugins? From Hans' words it seems so. If this is correct, then maybe
 > > reiser4 core should be updated to completely replace current VFS layer?
 > > Then it's a good point to create a branch (in old development model it
 > > would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and
 > > rewrite all (or at least most used) FS as reiser4 plugins. Then
 > > everybody will be happy.
 > > 
 > > But this looks too good to be true. Perhaps I misunderstood Hans' words
 > > aboud 'new disk format', did I?
 > 
 > No. You can change the format the reiser4 storage tree is stored in. As
 > long as other filesystems don't use the same underlying storage tree
 > this is not possible.

Wrong, plugin is called just below entry point from the VFS to the
file-system back-end. It can use reiser4 tree, or any storage layer it
wants. Or none at all: think about pseudo-files like
foo/metas/uid---they are also implemnted by plugins.

Programmer dedicated enough can (besides programming Fortran in any
language), write reiser4 plugin for any file-system. I will leave
description of this activity to people with better command over English
language.

Nikita.

 > 

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov:

> Wrong, plugin is called just below entry point from the VFS to the
> file-system back-end. It can use reiser4 tree, or any storage layer it
> wants. Or none at all: think about pseudo-files like
> foo/metas/uid---they are also implemnted by plugins.

Yes. But which plugins are we talking about?

What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?

Of course you can implement another filesystem inside the plugins but
they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov:

> Wrong, plugin is called just below entry point from the VFS to the
> file-system back-end. It can use reiser4 tree, or any storage layer it
> wants. Or none at all: think about pseudo-files like
> foo/metas/uid---they are also implemnted by plugins.

Yes. But which plugins are we talking about?

What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?

Of course you can implement another filesystem inside the plugins but
they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Nikita Danilov]

Christophe Saout writes:
 > Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov:
 > 
 > > Wrong, plugin is called just below entry point from the VFS to the
 > > file-system back-end. It can use reiser4 tree, or any storage layer it
 > > wants. Or none at all: think about pseudo-files like
 > > foo/metas/uid---they are also implemnted by plugins.
 > 
 > Yes. But which plugins are we talking about?
 > 
 > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?
 > 
 > Of course you can implement another filesystem inside the plugins but
 > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?
 > 

That was the message of my message.

Nikita.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote:
>  > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?
>  > 
>  > Of course you can implement another filesystem inside the plugins but
>  > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?
>  > 
> 
> That was the message of my message.

And I think Christophe and me already agreed in this thread that these
upper level plugin facility should go away before merge anyway.  We made
the mistake of not requesting removel of that one for XFS and now SGI
blocks it's removal.

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote:
>  
>
>> > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?
>> > 
>> > Of course you can implement another filesystem inside the plugins but
>> > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?
>> > 
>>
>>That was the message of my message.
>>    
>>
>
>And I think Christophe and me already agreed in this thread that these
>upper level plugin facility should go away before merge anyway.  We made
>the mistake of not requesting removel of that one for XFS and now SGI
>blocks it's removal.
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>
Your arrogance is misplaced for a man who has never done a significant 
piece of research.

Christoph, we are doing work, you are in the way, get out of the way.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote:
>  
>
>> > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?
>> > 
>> > Of course you can implement another filesystem inside the plugins but
>> > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?
>> > 
>>
>>That was the message of my message.
>>    
>>
>
>And I think Christophe and me already agreed in this thread that these
>upper level plugin facility should go away before merge anyway.  We made
>the mistake of not requesting removel of that one for XFS and now SGI
>blocks it's removal.
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>
Your arrogance is misplaced for a man who has never done a significant 
piece of research.

Christoph, we are doing work, you are in the way, get out of the way.

Hans

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 01:12:44AM -0700, Hans Reiser wrote:
> Your arrogance is misplaced for a man who has never done a significant 
> piece of research.
> 
> Christoph, we are doing work, you are in the way, get out of the way.

Could you please stop the namecalling.  And if you want to check who has
done significant work on Linux check the commit logs for yoour and for
my shortname.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2289 bytes --]

Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton:

> And describe the "plugin" system.  Why does the filesystem need such a
> thing (other filesystems get their features via `patch -p1')?

Ok, let me try. (Hans, please don't shoot me if I don't get every detail
right, I'm trying to simplify and translate)

The reiser4 core is just a big and fast storage layer using a single
tree. It is bound to a mount point and uses the page cache to manage its
memory. The only thing it can do on its own is to flush dirty data to
the disk when the VM wants to (memory pressure) or the VFS wants to
(unmount, sync) or some "plugin" wants to.
Additionally it's completely atomic (with respect to crashes, no
isolation like in databases), comparable to data=journal but without the
overhead of using a journal for everything.

Up to this point there are no users of this "database" and it does not
implement any of the VFS methods for a filesystem (except mount and
unmount perhaps).

All the functionality is provided to the plugins. You can insert,
remove, lookup or modify key:object pairs (with an index into the
object). The object is a sequence of units. For files a unit would be 1
byte and the index would be the byte offset. For directories the unit
would be 1 entry and the index would be the filename.

Now there are some plugins that define how the storage layout on the
disk is (some kind of "backend" plugins).

*And* there are plugins which are users of the "reiser4 client API" and
implement the actual VFS methods.

There's a UNIX directory plugin and a UNIX file plugin.

Directories, inodes and file content are just key:object pairs and the
plugins know how to operate on these.

There's a new plugin in work that also implements UNIX file semantics
but stores the data for that file encrypted and/or compressed.

These plugins live between the VFS and the storage layer.

Just like the filesystems live between the VFS and the block layer (Hans
would say that filesystems are VFS plugins ;-)).

> And what are the licensing implications of plugins?  Are they derived
> works?  Must they be GPL'ed?

I suppose yes, at least currently, since they can only be linked with
reiser4, there's no module infrastructure.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2289 bytes --]

Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton:

> And describe the "plugin" system.  Why does the filesystem need such a
> thing (other filesystems get their features via `patch -p1')?

Ok, let me try. (Hans, please don't shoot me if I don't get every detail
right, I'm trying to simplify and translate)

The reiser4 core is just a big and fast storage layer using a single
tree. It is bound to a mount point and uses the page cache to manage its
memory. The only thing it can do on its own is to flush dirty data to
the disk when the VM wants to (memory pressure) or the VFS wants to
(unmount, sync) or some "plugin" wants to.
Additionally it's completely atomic (with respect to crashes, no
isolation like in databases), comparable to data=journal but without the
overhead of using a journal for everything.

Up to this point there are no users of this "database" and it does not
implement any of the VFS methods for a filesystem (except mount and
unmount perhaps).

All the functionality is provided to the plugins. You can insert,
remove, lookup or modify key:object pairs (with an index into the
object). The object is a sequence of units. For files a unit would be 1
byte and the index would be the byte offset. For directories the unit
would be 1 entry and the index would be the filename.

Now there are some plugins that define how the storage layout on the
disk is (some kind of "backend" plugins).

*And* there are plugins which are users of the "reiser4 client API" and
implement the actual VFS methods.

There's a UNIX directory plugin and a UNIX file plugin.

Directories, inodes and file content are just key:object pairs and the
plugins know how to operate on these.

There's a new plugin in work that also implements UNIX file semantics
but stores the data for that file encrypted and/or compressed.

These plugins live between the VFS and the storage layer.

Just like the filesystems live between the VFS and the block layer (Hans
would say that filesystems are VFS plugins ;-)).

> And what are the licensing implications of plugins?  Are they derived
> works?  Must they be GPL'ed?

I suppose yes, at least currently, since they can only be linked with
reiser4, there's no module infrastructure.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 02:18:49PM +0200, Christophe Saout wrote:
> The reiser4 core is just a big and fast storage layer using a single
> tree. It is bound to a mount point and uses the page cache to manage its
> memory. The only thing it can do on its own is to flush dirty data to
> the disk when the VM wants to (memory pressure) or the VFS wants to
> (unmount, sync) or some "plugin" wants to.
> Additionally it's completely atomic (with respect to crashes, no
> isolation like in databases), comparable to data=journal but without the
> overhead of using a journal for everything.
> 
> Up to this point there are no users of this "database" and it does not
> implement any of the VFS methods for a filesystem (except mount and
> unmount perhaps).

Now that part is interesting from the how to sell it to non-Linux users
POV, but not for the linux kernel.

> All the functionality is provided to the plugins. You can insert,
> remove, lookup or modify key:object pairs (with an index into the
> object). The object is a sequence of units. For files a unit would be 1
> byte and the index would be the byte offset. For directories the unit
> would be 1 entry and the index would be the filename.
> 
> Now there are some plugins that define how the storage layout on the
> disk is (some kind of "backend" plugins).
> 
> *And* there are plugins which are users of the "reiser4 client API" and
> implement the actual VFS methods.

Here comes the problem.  All the access checking/race avoidance/loop
creation avoiced, in short the posix+extension semantics are implemented
in the Linux VFS layer.  If you want to allow a second access method
(e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
inside reiser4.  But this is absolutely not what we want, because we
moved to this common as all previous version in the different
filesystems were buggy, and in fact there are problems that are simply
not solveable at the level at all (rename on directories comes to mind).

So if you want to provide an additional API you'll have to go through
the VFS to get it right - ergo a plugin architecture for upper plugins
is worthless.

Plugins for the disk-format level are an interesting and workable idea
on the other hand.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig:

> Now that part is interesting from the how to sell it to non-Linux users
> POV, but not for the linux kernel.

Why? The question was what these plugins are exactly. This is the
answer.

> > *And* there are plugins which are users of the "reiser4 client API" and
> > implement the actual VFS methods.
> 
> Here comes the problem.  All the access checking/race avoidance/loop
> creation avoiced, in short the posix+extension semantics are implemented
> in the Linux VFS layer.  If you want to allow a second access method
> (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
> inside reiser4.

Are you actually listening? If you implement a filesystem there's a
place where you have to implement the Linux VFS methods. I'm talking
about inode_operations and these things. This has nothing to do with
doing anything outside the Linux VFS. And I'm not talking about these
metas either. These really don't belong inside the filesystem.

> So if you want to provide an additional API you'll have to go through
> the VFS to get it right - ergo a plugin architecture for upper plugins
> is worthless.

See, you didn't listen. ;-)

The reiser4 core doesn't know about inodes, directories or files. It's
the glue code between the VFS and the storage layer that does. It's
implemented as a plugin. This has nothing to do with semantic
enhancements yet. These should be removed for now and made a 2.7 topic.

Maybe Namesys could provide them as an additional patch from their
webpage for those who need it. The plugins that would benefit from this
(encrypted and compressed file storage) aren't ready yet anyway.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig:

> Now that part is interesting from the how to sell it to non-Linux users
> POV, but not for the linux kernel.

Why? The question was what these plugins are exactly. This is the
answer.

> > *And* there are plugins which are users of the "reiser4 client API" and
> > implement the actual VFS methods.
> 
> Here comes the problem.  All the access checking/race avoidance/loop
> creation avoiced, in short the posix+extension semantics are implemented
> in the Linux VFS layer.  If you want to allow a second access method
> (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
> inside reiser4.

Are you actually listening? If you implement a filesystem there's a
place where you have to implement the Linux VFS methods. I'm talking
about inode_operations and these things. This has nothing to do with
doing anything outside the Linux VFS. And I'm not talking about these
metas either. These really don't belong inside the filesystem.

> So if you want to provide an additional API you'll have to go through
> the VFS to get it right - ergo a plugin architecture for upper plugins
> is worthless.

See, you didn't listen. ;-)

The reiser4 core doesn't know about inodes, directories or files. It's
the glue code between the VFS and the storage layer that does. It's
implemented as a plugin. This has nothing to do with semantic
enhancements yet. These should be removed for now and made a 2.7 topic.

Maybe Namesys could provide them as an additional patch from their
webpage for those who need it. The plugins that would benefit from this
(encrypted and compressed file storage) aren't ready yet anyway.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:00:34PM +0200, Christophe Saout wrote:
> Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig:
> 
> > Now that part is interesting from the how to sell it to non-Linux users
> > POV, but not for the linux kernel.
> 
> Why? The question was what these plugins are exactly. This is the
> answer.

I explained it below..

> > > *And* there are plugins which are users of the "reiser4 client API" and
> > > implement the actual VFS methods.
> > 
> > Here comes the problem.  All the access checking/race avoidance/loop
> > creation avoiced, in short the posix+extension semantics are implemented
> > in the Linux VFS layer.  If you want to allow a second access method
> > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
> > inside reiser4.
> 
> Are you actually listening? If you implement a filesystem there's a
> place where you have to implement the Linux VFS methods. I'm talking
> about inode_operations and these things. This has nothing to do with
> doing anything outside the Linux VFS. And I'm not talking about these
> metas either. These really don't belong inside the filesystem.

as I wrote in this mail this absolutely _does_ belong in the filesystem,
it's a major part of it and isn't easily separatable.

> > So if you want to provide an additional API you'll have to go through
> > the VFS to get it right - ergo a plugin architecture for upper plugins
> > is worthless.
> 
> See, you didn't listen. ;-)
> 
> The reiser4 core doesn't know about inodes, directories or files. It's
> the glue code between the VFS and the storage layer that does. It's
> implemented as a plugin. This has nothing to do with semantic
> enhancements yet. These should be removed for now and made a 2.7 topic.

Oh yes, it is.  As soon as you use different access methods on an
overlapping set of objects you see exactly the problems I described.

If they don't overlap there's no point for the plugins to start with,
you'll better turn the core into a library that can be used by different
projects then.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig:

> > Why? The question was what these plugins are exactly. This is the
> > answer.
> 
> I explained it below..

It's hard to talk to you if you're not even trying to understand what
I'm trying to say. Andrew wanted to know what these plugins are. I tried
to give an answer and then you come along to tell me that it's not
relevant. Sure, Linux doesn't care about what the filesystem does
internally but Andrew wanted to know that.

> > Are you actually listening? If you implement a filesystem there's a
> > place where you have to implement the Linux VFS methods. I'm talking
> > about inode_operations and these things. This has nothing to do with
> > doing anything outside the Linux VFS. And I'm not talking about these
> > metas either. These really don't belong inside the filesystem.
> 
> as I wrote in this mail this absolutely _does_ belong in the filesystem,
> it's a major part of it and isn't easily separatable.

Huh?

Now you're completely confusing me.

First you say that that file-as-a-directory is crap then you say that it
does belong into the filesystem?

I'll try again:

inode_operations, etc... are implemented as a "reiser4 plugin". This
means that reiser4 is usable as filesystem, you can store files in a
directory hierarchy. There's absolute nothing special here. Absolutely
nothing. And yes, reiser4 plugins are invisible from the VFS. It looks
like a normal filesystem because it behaves like one.

The "metas" I was referring in the second part are this file-as-a-
directory thing. These don't belong in the filesystem and should
therefore be removed.

The latter doesn't necessarily have anything to do with plugins.
The plugins are plugins for the storage layer, not for some semantic
enhancement.

Note that I changed the topic to avoid further confusion.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig:

> > Why? The question was what these plugins are exactly. This is the
> > answer.
> 
> I explained it below..

It's hard to talk to you if you're not even trying to understand what
I'm trying to say. Andrew wanted to know what these plugins are. I tried
to give an answer and then you come along to tell me that it's not
relevant. Sure, Linux doesn't care about what the filesystem does
internally but Andrew wanted to know that.

> > Are you actually listening? If you implement a filesystem there's a
> > place where you have to implement the Linux VFS methods. I'm talking
> > about inode_operations and these things. This has nothing to do with
> > doing anything outside the Linux VFS. And I'm not talking about these
> > metas either. These really don't belong inside the filesystem.
> 
> as I wrote in this mail this absolutely _does_ belong in the filesystem,
> it's a major part of it and isn't easily separatable.

Huh?

Now you're completely confusing me.

First you say that that file-as-a-directory is crap then you say that it
does belong into the filesystem?

I'll try again:

inode_operations, etc... are implemented as a "reiser4 plugin". This
means that reiser4 is usable as filesystem, you can store files in a
directory hierarchy. There's absolute nothing special here. Absolutely
nothing. And yes, reiser4 plugins are invisible from the VFS. It looks
like a normal filesystem because it behaves like one.

The "metas" I was referring in the second part are this file-as-a-
directory thing. These don't belong in the filesystem and should
therefore be removed.

The latter doesn't necessarily have anything to do with plugins.
The plugins are plugins for the storage layer, not for some semantic
enhancement.

Note that I changed the topic to avoid further confusion.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 2520 bytes --]

On Thu, Aug 26, 2004 at 03:17:53PM +0200, Christophe Saout wrote:
> It's hard to talk to you if you're not even trying to understand what
> I'm trying to say. Andrew wanted to know what these plugins are. I tried
> to give an answer and then you come along to tell me that it's not
> relevant. Sure, Linux doesn't care about what the filesystem does
> internally but Andrew wanted to know that.

Huh?  I don understand what reiser4 plugins are, and I tell you that a
subset of these are _wrong_ for use in the kernel. 

> > > Are you actually listening? If you implement a filesystem there's a
> > > place where you have to implement the Linux VFS methods. I'm talking
> > > about inode_operations and these things. This has nothing to do with
> > > doing anything outside the Linux VFS. And I'm not talking about these
> > > metas either. These really don't belong inside the filesystem.
> > 
> > as I wrote in this mail this absolutely _does_ belong in the filesystem,
> > it's a major part of it and isn't easily separatable.
> 
> Huh?
> 
> Now you're completely confusing me.
> 
> First you say that that file-as-a-directory is crap then you say that it
> does belong into the filesystem?

I think you're talking about something different then me, I'm not
talking about the magic meta files but the VFS interface in general.

This VFS interface is an integral part of ævery filesystem, and it
doesn't make a whole lot to put it into a plugin.  If you want your
filesystem core portable it does to a certain extent make sense to
abstract them out, but as someone who's worked on a few such 'portable'
filesystems I can tell you that it doesn't work out as expected.

> I'll try again:
> 
> inode_operations, etc... are implemented as a "reiser4 plugin". This
> means that reiser4 is usable as filesystem, you can store files in a
> directory hierarchy. There's absolute nothing special here. Absolutely
> nothing. And yes, reiser4 plugins are invisible from the VFS. It looks
> like a normal filesystem because it behaves like one.

Now kill the whole plugin mess and let them talk directly to another.
My first mail explained to you why it doesn't make sense to have
multiple such plugins to work on a common subset of data.

> The latter doesn't necessarily have anything to do with plugins.
> The plugins are plugins for the storage layer, not for some semantic
> enhancement.

Again plugins below the pagecache (if that's what you call the "storage
layer") _do_ make sense.  plugins at the semantic layer don't.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 2774 bytes --]

On Thu, Aug 26, 2004 at 03:17:53PM +0200, Christophe Saout wrote:
> It's hard to talk to you if you're not even trying to understand what
> I'm trying to say. Andrew wanted to know what these plugins are. I tried
> to give an answer and then you come along to tell me that it's not
> relevant. Sure, Linux doesn't care about what the filesystem does
> internally but Andrew wanted to know that.

Huh?  I don understand what reiser4 plugins are, and I tell you that a
subset of these are _wrong_ for use in the kernel. 

> > > Are you actually listening? If you implement a filesystem there's a
> > > place where you have to implement the Linux VFS methods. I'm talking
> > > about inode_operations and these things. This has nothing to do with
> > > doing anything outside the Linux VFS. And I'm not talking about these
> > > metas either. These really don't belong inside the filesystem.
> > 
> > as I wrote in this mail this absolutely _does_ belong in the filesystem,
> > it's a major part of it and isn't easily separatable.
> 
> Huh?
> 
> Now you're completely confusing me.
> 
> First you say that that file-as-a-directory is crap then you say that it
> does belong into the filesystem?

I think you're talking about something different then me, I'm not
talking about the magic meta files but the VFS interface in general.

This VFS interface is an integral part of ævery filesystem, and it
doesn't make a whole lot to put it into a plugin.  If you want your
filesystem core portable it does to a certain extent make sense to
abstract them out, but as someone who's worked on a few such 'portable'
filesystems I can tell you that it doesn't work out as expected.

> I'll try again:
> 
> inode_operations, etc... are implemented as a "reiser4 plugin". This
> means that reiser4 is usable as filesystem, you can store files in a
> directory hierarchy. There's absolute nothing special here. Absolutely
> nothing. And yes, reiser4 plugins are invisible from the VFS. It looks
> like a normal filesystem because it behaves like one.

Now kill the whole plugin mess and let them talk directly to another.
My first mail explained to you why it doesn't make sense to have
multiple such plugins to work on a common subset of data.

> The latter doesn't necessarily have anything to do with plugins.
> The plugins are plugins for the storage layer, not for some semantic
> enhancement.

Again plugins below the pagecache (if that's what you call the "storage
layer") _do_ make sense.  plugins at the semantic layer don't.

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig:

> > First you say that that file-as-a-directory is crap then you say that it
> > does belong into the filesystem?
> 
> I think you're talking about something different then me, I'm not
> talking about the magic meta files but the VFS interface in general.
> 
> This VFS interface is an integral part of very filesystem, and it
> doesn't make a whole lot to put it into a plugin.

Right. That's why these plugins are linked in uncoditionally. It doesn't
work without them. Hence "plugins" is not a very good name.

> If you want your
> filesystem core portable it does to a certain extent make sense to
> abstract them out, but as someone who's worked on a few such 'portable'
> filesystems I can tell you that it doesn't work out as expected.

That's your opinion. reiser4 seems to work very well.

> Now kill the whole plugin mess and let them talk directly to another.

This is how reiserfs v3 did it. And Hans says that it actually
complicated things.

> My first mail explained to you why it doesn't make sense to have
> multiple such plugins to work on a common subset of data.

What I understood is that you can select exactly one plugin that e.g.
handles the file data. The default plugin is optimized for normal files,
another one could implement transparent compression or encryption. Some
of these plugins also give the storage layer hints how to group files
together to optimized performance. Neither of these things mess with the
VFS.

> > The latter doesn't necessarily have anything to do with plugins.
> > The plugins are plugins for the storage layer, not for some semantic
> > enhancement.
> 
> Again plugins below the pagecache (if that's what you call the "storage
> layer") _do_ make sense.  plugins at the semantic layer don't.

Yes, that's what I was trying to say.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig:

> > First you say that that file-as-a-directory is crap then you say that it
> > does belong into the filesystem?
> 
> I think you're talking about something different then me, I'm not
> talking about the magic meta files but the VFS interface in general.
> 
> This VFS interface is an integral part of very filesystem, and it
> doesn't make a whole lot to put it into a plugin.

Right. That's why these plugins are linked in uncoditionally. It doesn't
work without them. Hence "plugins" is not a very good name.

> If you want your
> filesystem core portable it does to a certain extent make sense to
> abstract them out, but as someone who's worked on a few such 'portable'
> filesystems I can tell you that it doesn't work out as expected.

That's your opinion. reiser4 seems to work very well.

> Now kill the whole plugin mess and let them talk directly to another.

This is how reiserfs v3 did it. And Hans says that it actually
complicated things.

> My first mail explained to you why it doesn't make sense to have
> multiple such plugins to work on a common subset of data.

What I understood is that you can select exactly one plugin that e.g.
handles the file data. The default plugin is optimized for normal files,
another one could implement transparent compression or encryption. Some
of these plugins also give the storage layer hints how to group files
together to optimized performance. Neither of these things mess with the
VFS.

> > The latter doesn't necessarily have anything to do with plugins.
> > The plugins are plugins for the storage layer, not for some semantic
> > enhancement.
> 
> Again plugins below the pagecache (if that's what you call the "storage
> layer") _do_ make sense.  plugins at the semantic layer don't.

Yes, that's what I was trying to say.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:35:07PM +0200, Christophe Saout wrote:
> > If you want your
> > filesystem core portable it does to a certain extent make sense to
> > abstract them out, but as someone who's worked on a few such 'portable'
> > filesystems I can tell you that it doesn't work out as expected.
> 
> That's your opinion. reiser4 seems to work very well.

So how many OSes is it ported to currently?  The problem is not that it
doesn't work but that it's really hard to maintain.  And remember that
this maintaince overhead is not just for Hans but for everyone that does
VFS-level work.  Unless of course we get a blanko permission to break it
as soon as fixing the mess becomes non-trivial.

> What I understood is that you can select exactly one plugin that e.g.
> handles the file data. The default plugin is optimized for normal files,
> another one could implement transparent compression or encryption. Some
> of these plugins also give the storage layer hints how to group files
> together to optimized performance. Neither of these things mess with the
> VFS.

compression or encryption must sit below the pagecache to work nicely,
and this hint things that usually sit at the pagecache level.  But let's
assume you have a valid use for different file_operations, why don't you
simply add in different file_operations instead of adding another
internal dispatch layer?  

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]

Am Donnerstag, den 26.08.2004, 15:40 +0200 schrieb Christoph Hellwig:

> > That's your opinion. reiser4 seems to work very well.
> 
> So how many OSes is it ported to currently?  The problem is not that it
> doesn't work but that it's really hard to maintain.  And remember that
> this maintaince overhead is not just for Hans but for everyone that does
> VFS-level work.

Where does it add overhead? It's just the interface between the reiser4
storage layer and the rest abstracted out.

Ok, there's the code that dispatches the VFS method calls to the plugin
call but I would say that's about it. I didn't develop reiser4 so I
don't know.

/* ->mkdir() VFS method in reiser4 inode_operations */
static int
reiser4_mkdir(struct inode *parent      /* inode of parent
                                         * directory */ ,
              struct dentry *dentry     /* dentry of new object to
                                         * create */ ,
              int mode /* new object's mode */ )
{
        reiser4_object_create_data data;

        reiser4_stat_inc_at(parent->i_sb, vfs_calls.mkdir);

        data.mode = S_IFDIR | mode;
        data.id = DIRECTORY_FILE_PLUGIN_ID;
        return invoke_create_method(parent, dentry, &data);
}

The invoke_create_method then just calls the create method of the plugin
after setting up a context and lookup up the plugin.

xfs_iops.c seems to do something similar.

>   Unless of course we get a blanko permission to break it
> as soon as fixing the mess becomes non-trivial.

Well, I think so. It's done relatively often.

> > What I understood is that you can select exactly one plugin that e.g.
> > handles the file data. The default plugin is optimized for normal files,
> > another one could implement transparent compression or encryption. Some
> > of these plugins also give the storage layer hints how to group files
> > together to optimized performance. Neither of these things mess with the
> > VFS.
> 
> compression or encryption must sit below the pagecache to work nicely,
> and this hint things that usually sit at the pagecache level.  But let's
> assume you have a valid use for different file_operations, why don't you
> simply add in different file_operations instead of adding another
> internal dispatch layer?  

I don't know, ask Hans. How could the VFS know it a filesystem wants to
do something specific with a file that is completely transparent to the
VFS?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

> > compression or encryption must sit below the pagecache to work nicely,
> > and this hint things that usually sit at the pagecache level.  But let's
> > assume you have a valid use for different file_operations, why don't you
> > simply add in different file_operations instead of adding another
> > internal dispatch layer?  
> 
> I don't know, ask Hans. How could the VFS know it a filesystem wants to
> do something specific with a file that is completely transparent to the
> VFS?

The VFS shouldn't, that the whole point.  That's why it allows the
filesystem to register different method tables for each object.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

Am Donnerstag, den 26.08.2004, 17:37 +0200 schrieb Christoph Hellwig:

> > > compression or encryption must sit below the pagecache to work nicely,
> > > and this hint things that usually sit at the pagecache level.  But let's
> > > assume you have a valid use for different file_operations, why don't you
> > > simply add in different file_operations instead of adding another
> > > internal dispatch layer?  
> > 
> > I don't know, ask Hans. How could the VFS know it a filesystem wants to
> > do something specific with a file that is completely transparent to the
> > VFS?
> 
> The VFS shouldn't, that the whole point.  That's why it allows the
> filesystem to register different method tables for each object.

Only the objects it can distinguish.

>         ops->file    = reiser4_file_operations;
>         ops->symlink = reiser4_symlink_inode_operations;
>         ops->special = reiser4_special_inode_operations;
>         ops->dentry  = reiser4_dentry_operations;
>         ops->as      = reiser4_as_operations;

How could reiser4 register other operations for files that should be
stored encrypted or compressed? It's all under reiser4_file_operations.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 05:48:54PM +0200, Christophe Saout wrote:
> > > I don't know, ask Hans. How could the VFS know it a filesystem wants to
> > > do something specific with a file that is completely transparent to the
> > > VFS?
> > 
> > The VFS shouldn't, that the whole point.  That's why it allows the
> > filesystem to register different method tables for each object.
> 
> Only the objects it can distinguish.

Yes, every inode can have different operation vectors.  Which is the
smallest possible object the VFS knows about.

> >         ops->file    = reiser4_file_operations;
> >         ops->symlink = reiser4_symlink_inode_operations;
> >         ops->special = reiser4_special_inode_operations;
> >         ops->dentry  = reiser4_dentry_operations;
> >         ops->as      = reiser4_as_operations;
> 
> How could reiser4 register other operations for files that should be
> stored encrypted or compressed? It's all under reiser4_file_operations.

Again, your confusing upper and lower plugins.  For things happening
below the pagecache you could register different address_space
operations which sometimes makes sense.  But you want e.g. different
inode_operations for directories vs symlinks vs files.

Please read through some linux filesystem code, okay :)

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 986 bytes --]

Am Donnerstag, den 26.08.2004, 18:06 +0200 schrieb Christoph Hellwig:

> Again, your confusing upper and lower plugins.  For things happening
> below the pagecache you could register different address_space
> operations which sometimes makes sense.  But you want e.g. different
> inode_operations for directories vs symlinks vs files.

Ok, I see your point. aops. Sorry.
Looking at the code this could be done. The wrappers that dispatch the
operations are really small and call the plugin that is registered with
the inode of the mapping. Instead it could have directly set the
corresponding operations. Right. The wrappers are doing a few things
before calling the plugin. That could be done the other way round too.
But that's more of an implementation issue and could still be changed.

> Please read through some linux filesystem code, okay :)

I'm partly familiar with that, I just either misunderstood you or wasn't
really thinking (probably the latter...).


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 06:14:19PM +0200, Christophe Saout wrote:
> Ok, I see your point. aops. Sorry.
> Looking at the code this could be done. The wrappers that dispatch the
> operations are really small and call the plugin that is registered with
> the inode of the mapping. Instead it could have directly set the
> corresponding operations. Right. The wrappers are doing a few things
> before calling the plugin. That could be done the other way round too.
> But that's more of an implementation issue and could still be changed.

I agree that it's an implementation issue.  But it's also a good proof
for how Hans tries to ignore all the existing infrastructure for various
reasons.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

Am Donnerstag, den 26.08.2004, 17:25 +0100 schrieb Christoph Hellwig:

> > Looking at the code this could be done. The wrappers that dispatch the
> > operations are really small and call the plugin that is registered with
> > the inode of the mapping. Instead it could have directly set the
> > corresponding operations. Right. The wrappers are doing a few things
> > before calling the plugin. That could be done the other way round too.
> > But that's more of an implementation issue and could still be changed.
> 
> I agree that it's an implementation issue.  But it's also a good proof
> for how Hans tries to ignore all the existing infrastructure for various
> reasons.

I definitely agree with you on that one.

It bugged me from the beginning because things like these are very
likely to suffer from unnecessary problems that have been solved in the
common code or might be solved when correctly done in the common code in
the future.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

Am Donnerstag, den 26.08.2004, 17:25 +0100 schrieb Christoph Hellwig:

> > Looking at the code this could be done. The wrappers that dispatch the
> > operations are really small and call the plugin that is registered with
> > the inode of the mapping. Instead it could have directly set the
> > corresponding operations. Right. The wrappers are doing a few things
> > before calling the plugin. That could be done the other way round too.
> > But that's more of an implementation issue and could still be changed.
> 
> I agree that it's an implementation issue.  But it's also a good proof
> for how Hans tries to ignore all the existing infrastructure for various
> reasons.

I definitely agree with you on that one.

It bugged me from the beginning because things like these are very
likely to suffer from unnecessary problems that have been solved in the
common code or might be solved when correctly done in the common code in
the future.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Hans Reiser]

Christophe Saout wrote:

>
>
>I don't know, ask Hans. How could the VFS know it a filesystem wants to
>do something specific with a file that is completely transparent to the
>VFS?
>
>  
>
To know what method to use, you must determine the pluginid, and then 
find the method within that plugin for that vfs operation.

As for overhead, well, who eats whose dust in the benchmarks....?

Re: reiser4 plugins

[Hans Reiser]

Christophe Saout wrote:

>
>
>I don't know, ask Hans. How could the VFS know it a filesystem wants to
>do something specific with a file that is completely transparent to the
>VFS?
>
>  
>
To know what method to use, you must determine the pluginid, and then 
find the method within that plugin for that vfs operation.

As for overhead, well, who eats whose dust in the benchmarks....?

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:
 > Christophe Saout wrote:
 > 
 > >
 > >
 > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
 > >do something specific with a file that is completely transparent to the
 > >VFS?
 > >
 > >  
 > >
 > To know what method to use, you must determine the pluginid, and then 
 > find the method within that plugin for that vfs operation.
 > 
 > As for overhead, well, who eats whose dust in the benchmarks....?

Whoever sponsors the benchmark usually wins. Had you forgotten that
mongo setup used by http://www.namesys.com/benchmarks.html was specially
`tuned' to reach peak reiser4 performance? Remember why you decided to
turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
_second_ stalls with reiser4 under high io loads (large atom is
obviously being flushed and everyone waits on it...). In my opinion, it
is such things that are of utmost importance for real reiser4
acceptance, not how to name `metas' sub-directory.

Nikita.

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

>Hans Reiser writes:
> > Christophe Saout wrote:
> > 
> > >
> > >
> > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
> > >do something specific with a file that is completely transparent to the
> > >VFS?
> > >
> > >  
> > >
> > To know what method to use, you must determine the pluginid, and then 
> > find the method within that plugin for that vfs operation.
> > 
> > As for overhead, well, who eats whose dust in the benchmarks....?
>
>Whoever sponsors the benchmark usually wins. Had you forgotten that
>mongo setup used by http://www.namesys.com/benchmarks.html was specially
>`tuned' to reach peak reiser4 performance? Remember why you decided to
>turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
>_second_ stalls with reiser4 under high io loads (large atom is
>obviously being flushed and everyone waits on it...). In my opinion, it
>is such things that are of utmost importance for real reiser4
>acceptance, not how to name `metas' sub-directory.
>
>Nikita.
>
>
>  
>
If you ask real users, they say that reiser4 is fast, and their 
experience matches our benchmark.  You can criticize the benchmark if 
you want, but then you should run your own and publish it.

There are still plenty of things needing fixing though, and you are 
right that they are more important than the pretext being attempted for 
keeping us out of the kernel.  Probably you could supply them with many 
superior pretexts.;-)

The 90 second stalls, those should be fixed by debugging copy on capture 
and turning it on, yes?  You can hardly claim I failed to push for the 
completion of copy on capture....  and of course the reason it is not 
done is simply that there is too much to do....  I (sincerely) hope vs 
enjoys his vacation, the whole team is working too hard.  I am doing 7 
day weeks now to try to make payroll for us and keep darpa happy at the 
same time.

fsync performance needs fixing badly, and Chris is right in his 
diagnosis of what we can do to fix it.

Hans

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

>Hans Reiser writes:
> > Christophe Saout wrote:
> > 
> > >
> > >
> > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
> > >do something specific with a file that is completely transparent to the
> > >VFS?
> > >
> > >  
> > >
> > To know what method to use, you must determine the pluginid, and then 
> > find the method within that plugin for that vfs operation.
> > 
> > As for overhead, well, who eats whose dust in the benchmarks....?
>
>Whoever sponsors the benchmark usually wins. Had you forgotten that
>mongo setup used by http://www.namesys.com/benchmarks.html was specially
>`tuned' to reach peak reiser4 performance? Remember why you decided to
>turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
>_second_ stalls with reiser4 under high io loads (large atom is
>obviously being flushed and everyone waits on it...). In my opinion, it
>is such things that are of utmost importance for real reiser4
>acceptance, not how to name `metas' sub-directory.
>
>Nikita.
>
>
>  
>
If you ask real users, they say that reiser4 is fast, and their 
experience matches our benchmark.  You can criticize the benchmark if 
you want, but then you should run your own and publish it.

There are still plenty of things needing fixing though, and you are 
right that they are more important than the pretext being attempted for 
keeping us out of the kernel.  Probably you could supply them with many 
superior pretexts.;-)

The 90 second stalls, those should be fixed by debugging copy on capture 
and turning it on, yes?  You can hardly claim I failed to push for the 
completion of copy on capture....  and of course the reason it is not 
done is simply that there is too much to do....  I (sincerely) hope vs 
enjoys his vacation, the whole team is working too hard.  I am doing 7 
day weeks now to try to make payroll for us and keep darpa happy at the 
same time.

fsync performance needs fixing badly, and Chris is right in his 
diagnosis of what we can do to fix it.

Hans

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:
 > Nikita Danilov wrote:
 > 
 > >Hans Reiser writes:
 > > > Christophe Saout wrote:
 > > > 
 > > > >
 > > > >
 > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
 > > > >do something specific with a file that is completely transparent to the
 > > > >VFS?
 > > > >
 > > > >  
 > > > >
 > > > To know what method to use, you must determine the pluginid, and then 
 > > > find the method within that plugin for that vfs operation.
 > > > 
 > > > As for overhead, well, who eats whose dust in the benchmarks....?
 > >
 > >Whoever sponsors the benchmark usually wins. Had you forgotten that
 > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
 > >`tuned' to reach peak reiser4 performance? Remember why you decided to
 > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
 > >_second_ stalls with reiser4 under high io loads (large atom is
 > >obviously being flushed and everyone waits on it...). In my opinion, it
 > >is such things that are of utmost importance for real reiser4
 > >acceptance, not how to name `metas' sub-directory.
 > >
 > >Nikita.
 > >
 > >
 > >  
 > >
 > If you ask real users, they say that reiser4 is fast, and their 
 > experience matches our benchmark.  You can criticize the benchmark if 

They experience 90 second stalls. And please, do not tell me how fast
reiser4 is, I spent a lot of time working with it, and I know very well
when it's fast, and when it's deadly slow.

 > you want, but then you should run your own and publish it.

I did, after which you told me to turn OVERWRITE and MODIFY phases off,
beause performance was horrible.

I not criticizing mongo benchmark per se. I think that it is
fundamentally wrong to use results that were deliberatly manipulated to
get best appearance to reiser4 (by omitting work-loads where it performs
poorly) as an argument. It's not clear who will, according to your
colorful expression, `eat dust' as a result of this. Or do you think
that users never overwrite of modify files in real life?

 > The 90 second stalls, those should be fixed by debugging copy on capture 
 > and turning it on, yes?  You can hardly claim I failed to push for the 

No. I described so many times to you, why COC is ineffectual here.

 > Hans

Nikita.

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

> > >Whoever sponsors the benchmark usually wins. Had you forgotten that
> > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
> > >`tuned' to reach peak reiser4 performance? Remember why you decided to
> > >turn OVERWRITE and MODIFY phases off?
>
What I should have done was what I did with fsync performance.  With 
fsync performance I told people that we had not yet tuned for it, please 
wait a bit and we will tune for it, for now it sucks.

Instead what I did was discuss with Zam at the time how it could be 
fixed, leave it off the website until Zam was given a chance to fix it, 
and then I managed to forget about it.   After release one remembers 
what all the things that should have been fixed before release were, sigh.

Zam and I both know what is needed to fix performance in these phases, 
and I just spoke with him on the phone.  I imagine it is 2 weeks of work 
for him, but it could be up to 6 weeks.  He will comment later.

The fix should consist of the following:

1) tweak the relocation policies (zam will say more about this, as he is 
the maintainer of them)

2) optimize overwrite set so that in the modify phase we behave like a 
write twice journaling filesystem, which means implement a reserved for 
the journal only area that exists as long as plenty of disk space is free.

3) finish the repacker (but we won't do that this month....)

The modify phase, which picks a random block to modify, is a worst case 
for a wandering log without a repacker.  We could actually do very very 
well at that kind of activity with a repacker, probably better than a 
fixed journal, but for now we should try acting like a write twice 
journal for atoms that small.

So, having realized my error thanks to the gracious kindness of how you 
pointed it out, we will modify the web site to say that we are not yet 
tuned for and currently suck at modifying random blocks in the middle of 
files, and appending small amounts to the end of them, and overwriting 
small files, but that we think we know what is needed to fix those things.

I think your characterization of my reasons was unkind and also unfair.  
I will pass on saying similarly unkind things about you as you are 
mostly a good person.  You never did seem to understand me.  You didn't 
understand the reasons for my technical designs (when they worked it 
made no sense to you that they did), and you didn't understand me as a 
person. 

I wish you well in your new job, and thank you for the hard work you did 
for me.

Hans

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

> > >Whoever sponsors the benchmark usually wins. Had you forgotten that
> > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
> > >`tuned' to reach peak reiser4 performance? Remember why you decided to
> > >turn OVERWRITE and MODIFY phases off?
>
What I should have done was what I did with fsync performance.  With 
fsync performance I told people that we had not yet tuned for it, please 
wait a bit and we will tune for it, for now it sucks.

Instead what I did was discuss with Zam at the time how it could be 
fixed, leave it off the website until Zam was given a chance to fix it, 
and then I managed to forget about it.   After release one remembers 
what all the things that should have been fixed before release were, sigh.

Zam and I both know what is needed to fix performance in these phases, 
and I just spoke with him on the phone.  I imagine it is 2 weeks of work 
for him, but it could be up to 6 weeks.  He will comment later.

The fix should consist of the following:

1) tweak the relocation policies (zam will say more about this, as he is 
the maintainer of them)

2) optimize overwrite set so that in the modify phase we behave like a 
write twice journaling filesystem, which means implement a reserved for 
the journal only area that exists as long as plenty of disk space is free.

3) finish the repacker (but we won't do that this month....)

The modify phase, which picks a random block to modify, is a worst case 
for a wandering log without a repacker.  We could actually do very very 
well at that kind of activity with a repacker, probably better than a 
fixed journal, but for now we should try acting like a write twice 
journal for atoms that small.

So, having realized my error thanks to the gracious kindness of how you 
pointed it out, we will modify the web site to say that we are not yet 
tuned for and currently suck at modifying random blocks in the middle of 
files, and appending small amounts to the end of them, and overwriting 
small files, but that we think we know what is needed to fix those things.

I think your characterization of my reasons was unkind and also unfair.  
I will pass on saying similarly unkind things about you as you are 
mostly a good person.  You never did seem to understand me.  You didn't 
understand the reasons for my technical designs (when they worked it 
made no sense to you that they did), and you didn't understand me as a 
person. 

I wish you well in your new job, and thank you for the hard work you did 
for me.

Hans

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:
 > Nikita Danilov wrote:
 > 
 > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that
 > > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
 > > > >`tuned' to reach peak reiser4 performance? Remember why you decided to
 > > > >turn OVERWRITE and MODIFY phases off?
 > >
 > What I should have done was what I did with fsync performance.  With 
 > fsync performance I told people that we had not yet tuned for it, please 
 > wait a bit and we will tune for it, for now it sucks.

Right, and instead of this you are now claiming that these `benchmarks'
show superior reiser4 performance. Moreover, you are doing this
aggressively and proposing other people to `eat the dust' over what
happened to only exist due to your failing to remember something. As
some other notable LKML poster put it `inform yourself before
posting'. :-)

 > 
 > Instead what I did was discuss with Zam at the time how it could be 
 > fixed, leave it off the website until Zam was given a chance to fix it, 

It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were
turned off after switching to large keys.

 > and then I managed to forget about it.   After release one remembers 
 > what all the things that should have been fixed before release were, sigh.
 > 

[...]

 > 
 > I think your characterization of my reasons was unkind and also unfair.  

What fairness and kindness one expects after styling others as `puppies'
in public, and commenting on their hair style in purportedly technical
argument?

[...]

 > 
 > Hans

Nikita.

Re: reiser4 plugins

[Konstantin Besch]

> I wish you well in your new job, and thank you for the hard work you did 
> for me.

>  Hans

Oh yeah , Nikita , had to say this : BUT thank you for that hard work you did
for us all. 
It seems it is 'bout time to move out of the company, producing public 
software, where leader of the project thanks participants for the work done 
with words "for me" at the end.

Always had my doubts that I should thanks Hans Reiser for the FS :)
And the facts are that my servers for now get ext3. 
Good Day in New York !

                --Sincerely.


On Saturday 28 August 2004 9:47 am, Nikita Danilov wrote:
> Hans Reiser writes:
>  > Nikita Danilov wrote:
>  > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that
>  > > > >mongo setup used by http://www.namesys.com/benchmarks.html was
>  > > > > specially `tuned' to reach peak reiser4 performance? Remember why
>  > > > > you decided to turn OVERWRITE and MODIFY phases off?
>  >
>  > What I should have done was what I did with fsync performance.  With
>  > fsync performance I told people that we had not yet tuned for it, please
>  > wait a bit and we will tune for it, for now it sucks.
>
> Right, and instead of this you are now claiming that these `benchmarks'
> show superior reiser4 performance. Moreover, you are doing this
> aggressively and proposing other people to `eat the dust' over what
> happened to only exist due to your failing to remember something. As
> some other notable LKML poster put it `inform yourself before
> posting'. :-)
>
>  > Instead what I did was discuss with Zam at the time how it could be
>  > fixed, leave it off the website until Zam was given a chance to fix it,
>
> It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were
> turned off after switching to large keys.
>
>  > and then I managed to forget about it.   After release one remembers
>  > what all the things that should have been fixed before release were,
>  > sigh.
>
> [...]
>
>  > I think your characterization of my reasons was unkind and also unfair.
>
> What fairness and kindness one expects after styling others as `puppies'
> in public, and commenting on their hair style in purportedly technical
> argument?
>
> [...]
>
>  > Hans
>
> Nikita.

Re: reiser4 plugins

[Hans Reiser]

Konstantin Besch wrote:

>>I wish you well in your new job, and thank you for the hard work you did 
>>for me.
>>    
>>
>
>  
>
>> Hans
>>    
>>
>
>Oh yeah , Nikita , had to say this : BUT thank you for that hard work you did
>for us all. 
>It seems it is 'bout time to move out of the company, producing public 
>software, where leader of the project thanks participants for the work done 
>with words "for me" at the end.
>
>Always had my doubts that I should thanks Hans Reiser for the FS :)
>And the facts are that my servers for now get ext3. 
>Good Day in New York !
>
>                --Sincerely.
>
>  
>
I paid his salary with my credit cards.  Under usual english rules and 
capitalist traditions, that means that saying that he was working for me 
is socially acceptable english, even if I understood that he worked for 
more than the money. 

Hans

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

> > 
> > Instead what I did was discuss with Zam at the time how it could be 
> > fixed, leave it off the website until Zam was given a chance to fix it, 
>
>It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were
>turned off after switching to large keys.
>
> 
>
The online repacker is the best technical fix for these problems.  Tight 
packing is generally more damaged in its layout by small changes than 
loose packing, and V4 is the ultimate tight packer.  It is not clear to 
me that making V4 pack more loosely is necessarily a good idea.  These 
phases disturb the original packing at flush time, and the other phases 
don't.  One solution to try is packing less tightly.  Maybe once every 
megabyte skip forward 1 megabyte, and once every 64 megabytes skip to a 
random disk location, when allocating large new atoms.  I think I prefer 
using a repacker, but we should try it and see.  The other solutions of 
the previous email should also help a lot.  The issues of layout in this 
are similar to the ones improved by the fibration plugin.

I remember talking with not just you but zam about how we could fix it, 
and there was too much in the queue of work, and everyone was 
complaining to me that we should be debugging not optimizing, and you 
were the only one who thought it was a big deal.  I guess you still 
think it is a big issue and I still think things are good enough to use 
without it.  I still think Zam understood the issues better than you did 
(allocation is his code not yours).

There are some layout optimizations that a repacker can do best.  Still, 
there are some rough spots in the current allocation policy, and we 
should look at it.

Probably you will have reason to howl if we setup a benchmark which 
disturbs things with these phases, runs the repacker (I hope to have one 
in 6 months), and then measures our read performance compared to other 
filesystems without a repacker....;-)

Hans

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:

[...]

 > 
 > I remember talking with not just you but zam about how we could fix it, 
 > and there was too much in the queue of work, and everyone was 
 > complaining to me that we should be debugging not optimizing, and you 
 > were the only one who thought it was a big deal.  I guess you still 

You are trying to evade the point. And the point is not how performance
in these phases (and their impact on other phases) can be improved, or
what excuses did you have for not `doing' it, but the fact that after it
was found that large keys behave badly, these phases were turned off,
others were switched to lexicographical operation, _and_ resulting
benchmark is used as a basis to call other people names.

 > think it is a big issue and I still think things are good enough to use 
 > without it.  I still think Zam understood the issues better than you did 
 > (allocation is his code not yours).
 > 
 > There are some layout optimizations that a repacker can do best.  Still, 
 > there are some rough spots in the current allocation policy, and we 
 > should look at it.
 > 
 > Probably you will have reason to howl if we setup a benchmark which 

Surely, as a `big dog' what other sounds can I produce? :)

 > disturbs things with these phases, runs the repacker (I hope to have one 
 > in 6 months), and then measures our read performance compared to other 
 > filesystems without a repacker....;-)

That's ok with me, but it would make sense to remove unfair benchmarks
from the site until then.

 > 
 > Hans

Nikita.

Re: reiser4 plugins

[Alex Zarochentsev]

On Fri, Aug 27, 2004 at 10:55:50PM +0400, Nikita Danilov wrote:
> Hans Reiser writes:
>  > Nikita Danilov wrote:
>  > 
>  > >Hans Reiser writes:
>  > > > Christophe Saout wrote:
>  > > > 
>  > > > >
>  > > > >
>  > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
>  > > > >do something specific with a file that is completely transparent to the
>  > > > >VFS?
>  > > > >
>  > > > >  
>  > > > >
>  > > > To know what method to use, you must determine the pluginid, and then 
>  > > > find the method within that plugin for that vfs operation.
>  > > > 
>  > > > As for overhead, well, who eats whose dust in the benchmarks....?
>  > >
>  > >Whoever sponsors the benchmark usually wins. Had you forgotten that
>  > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
>  > >`tuned' to reach peak reiser4 performance? Remember why you decided to
>  > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
>  > >_second_ stalls with reiser4 under high io loads (large atom is
>  > >obviously being flushed and everyone waits on it...). In my opinion, it
>  > >is such things that are of utmost importance for real reiser4
>  > >acceptance, not how to name `metas' sub-directory.
>  > >
>  > >Nikita.
>  > >
>  > >
>  > >  
>  > >
>  > If you ask real users, they say that reiser4 is fast, and their 
>  > experience matches our benchmark.  You can criticize the benchmark if 
> 
> They experience 90 second stalls. And please, do not tell me how fast
> reiser4 is, I spent a lot of time working with it, and I know very well
> when it's fast, and when it's deadly slow.
> 
>  > you want, but then you should run your own and publish it.
> 
> I did, after which you told me to turn OVERWRITE and MODIFY phases off,
> beause performance was horrible.

Hmm, I think not the modify/overwrite phases performance were the problem.  The
modify and overwrite mongo phases fragment the filesystem too much, it had a
bad effect visible in the read phase and even in delete phase.  

We measured/optimized read performance  in assumption that fs is not
fragmented, why we should measure bad effect of overwrite instead?  

other fs do not relocate already allocated blocks, negative effect of 
file set overwrite is zero for them.  We could do the same for reiser4
by tuning flush algorithm and journal blocks allocation.

If users want to see read and delete performance as they are, overwrite phase
should be excluded, at least until we teach flush algorithm to not break good
block allocation.

If one wants to measure overwite performance, it is easy to enable overwrite
phase in mongo, but read and delete speed will be affected (it was so when I
tried to optimize delete).

> I not criticizing mongo benchmark per se. I think that it is
> fundamentally wrong to use results that were deliberatly manipulated to
> get best appearance to reiser4 (by omitting work-loads where it performs
> poorly) as an argument. It's not clear who will, according to your
> colorful expression, `eat dust' as a result of this. 
> Or do you think
> that users never overwrite of modify files in real life?

not whole file set.  I mean it is unusual to overwrite all files in the fs.

>  > The 90 second stalls, those should be fixed by debugging copy on capture 
>  > and turning it on, yes?  You can hardly claim I failed to push for the 
> 
> No. I described so many times to you, why COC is ineffectual here.

There are other ideas :) but, it seems to me, if we will focus on benchmarks,
we never complete plugins, metafiles and other useful things.  What I want
from reiser4 is a possiblity to encrypt and compress ~/Mail/ :).

> 
>  > Hans
> 
> Nikita.

-- 
Alex.

Re: reiser4 plugins

[Steve Bergman]

On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
> >
> If you ask real users, they say that reiser4 is fast, and their 
> experience matches our benchmark.  You can criticize the benchmark if 
> you want, but then you should run your own and publish it.
> 


As a "real" desktop user who just converted all his partitions from ext3
to reiser4, I have not, as yet, noticed any startling performance
increase.  Being slightly slightly irked to hear that the benchmark
numbers that have been paraded around on Slashdot and the internet in
general, at ext3's expense, have had reiser4's "bad" results surgically
extracted, I am running my own benchmarks to get the real story on
reiser4/ext3 mongo performance on my, rather average, desktop hardware.

I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.

Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
argument at line 439.

I have done this to fix it:


--- mongo.pl    2004-08-27 17:07:01.681723313 -0500
+++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
@@ -429,8 +429,8 @@
        if ( -e ${ERR_FILE}) {
            &DIE ("\nEXITED WITH FAIL\n");
        }
-       my $real = (split ' ', $time_output[0])[1];
-       my $cpu  = (split ' ', $time_output[2])[1];
+       my $real = (split ' ', $time_output[1])[1];
+       my $cpu  = (split ' ', $time_output[3])[1];
  
        unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
          LOG "@time_output";


What it gets me is the "real" line of the "time" output for "STAT
REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
i.e. only system time is counted. I believe this was the intent of the
original code, but want to verify before continuing.

Thanks,
Steve Bergman

Re: reiser4 plugins

[Steve Bergman]

On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
> >
> If you ask real users, they say that reiser4 is fast, and their 
> experience matches our benchmark.  You can criticize the benchmark if 
> you want, but then you should run your own and publish it.
> 


As a "real" desktop user who just converted all his partitions from ext3
to reiser4, I have not, as yet, noticed any startling performance
increase.  Being slightly slightly irked to hear that the benchmark
numbers that have been paraded around on Slashdot and the internet in
general, at ext3's expense, have had reiser4's "bad" results surgically
extracted, I am running my own benchmarks to get the real story on
reiser4/ext3 mongo performance on my, rather average, desktop hardware.

I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.

Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
argument at line 439.

I have done this to fix it:


--- mongo.pl    2004-08-27 17:07:01.681723313 -0500
+++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
@@ -429,8 +429,8 @@
        if ( -e ${ERR_FILE}) {
            &DIE ("\nEXITED WITH FAIL\n");
        }
-       my $real = (split ' ', $time_output[0])[1];
-       my $cpu  = (split ' ', $time_output[2])[1];
+       my $real = (split ' ', $time_output[1])[1];
+       my $cpu  = (split ' ', $time_output[3])[1];
  
        unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
          LOG "@time_output";


What it gets me is the "real" line of the "time" output for "STAT
REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
i.e. only system time is counted. I believe this was the intent of the
original code, but want to verify before continuing.

Thanks,
Steve Bergman

Re: reiser4 plugins

[Hans Reiser]

Steve Bergman wrote:

>On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
>  
>
>>If you ask real users, they say that reiser4 is fast, and their 
>>experience matches our benchmark.  You can criticize the benchmark if 
>>you want, but then you should run your own and publish it.
>>
>>    
>>
>
>
>As a "real" desktop user who just converted all his partitions from ext3
>to reiser4, I have not, as yet, noticed any startling performance
>increase.  Being slightly slightly irked to hear that the benchmark
>numbers that have been paraded around on Slashdot and the internet in
>general, at ext3's expense, have had reiser4's "bad" results surgically
>extracted, I am running my own benchmarks to get the real story on
>reiser4/ext3 mongo performance on my, rather average, desktop hardware.
>
>I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.
>
>Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
>argument at line 439.
>
>I have done this to fix it:
>
>
>--- mongo.pl    2004-08-27 17:07:01.681723313 -0500
>+++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
>@@ -429,8 +429,8 @@
>        if ( -e ${ERR_FILE}) {
>            &DIE ("\nEXITED WITH FAIL\n");
>        }
>-       my $real = (split ' ', $time_output[0])[1];
>-       my $cpu  = (split ' ', $time_output[2])[1];
>+       my $real = (split ' ', $time_output[1])[1];
>+       my $cpu  = (split ' ', $time_output[3])[1];
>  
>        unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
>          LOG "@time_output";
>
>
>What it gets me is the "real" line of the "time" output for "STAT
>REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
>i.e. only system time is counted. I believe this was the intent of the
>original code, but want to verify before continuing.
>
>Thanks,
>Steve Bergman
>
>
>
>  
>
I didn't write this (more precisely, it only vaguely resembles what I 
wrote in 1996).  Are you saying that it reports system time as real 
time?  If yes, then it is an error, we need to go remove a bunch of 
numbers from our benchmarks, and thanks for finding it.

Zam, please comment.

Hans

Re: reiser4 plugins

[Steve Bergman]

On Fri, 2004-08-27 at 23:54 -0700, Hans Reiser wrote:
> >
> I didn't write this (more precisely, it only vaguely resembles what I 
> wrote in 1996).  Are you saying that it reports system time as real 
> time?  If yes, then it is an error, we need to go remove a bunch of 
> numbers from our benchmarks, and thanks for finding it.
> 
> Zam, please comment.
> 

No.  I'm saying that on my setup (kernel 2.6.8.1-mm4/perl 5.8.5/bash
3.00) the first line returned from running the test is the:

[1]+  Done ...

line which throws the array indexes off by 1 and I always get 0 for the
real time and, yes, the real time gets reported as the system time, I
think.  Plus I get a warning about the fact that "Done" is not numeric.
This is obviously a problem with my particular setup.

After bumping the indexes up by 1, I get the correct real time reported
as "STAT REAL_TIME".  And the system time is reported as "STAT
CPU_TIME".

"STAT CPU_UTIL", however is computed in a completely different way based
on numbers collected from /proc/stat.  If I'm, reading
linux/Documentation/filesystems/proc.txt correctly, it is trying to
return the (system time) / (total time).  The total time agrees with
"STAT REAL_TIME".  However, the (system time) that it comes up with here
is always considerably higher than "STAT CPU_TIME".

User error is quite possible, as I am:

1. Just getting to know mongo.
2. Not a perl guy.
3. There is obviously something mongo.pl doesn't like about my setup.

-Steve Bergman

P.S. In the 2 tests I've run, reiser4 is not doing all that badly
against ext3 in OVERWRITE and MODIFY, though ext3 does come in faster.
Reiser4 trails badly on APPEND however, ext3 (data=ordered, without
htree) being some 2.5 - 4 times faster. 

Re: reiser4 plugins

[Alex Zarochentsev]

On Sat, Aug 28, 2004 at 05:09:28AM -0500, Steve Bergman wrote:
> On Fri, 2004-08-27 at 23:54 -0700, Hans Reiser wrote:
> > >
> > I didn't write this (more precisely, it only vaguely resembles what I 
> > wrote in 1996).  Are you saying that it reports system time as real 
> > time?  If yes, then it is an error, we need to go remove a bunch of 
> > numbers from our benchmarks, and thanks for finding it.
> > 
> > Zam, please comment.
> > 
> 
> No.  I'm saying that on my setup (kernel 2.6.8.1-mm4/perl 5.8.5/bash
> 3.00) the first line returned from running the test is the:
> 
> [1]+  Done ...

> line which throws the array indexes off by 1 and I always get 0 for the
> real time and, yes, the real time gets reported as the system time, I
> think.  Plus I get a warning about the fact that "Done" is not numeric.
> This is obviously a problem with my particular setup.

Yes. but that real/sys time parsing isn't reliable.   

> After bumping the indexes up by 1, I get the correct real time reported
> as "STAT REAL_TIME".  And the system time is reported as "STAT
> CPU_TIME".
> 
> "STAT CPU_UTIL", however is computed in a completely different way based
> on numbers collected from /proc/stat.  If I'm, reading
> linux/Documentation/filesystems/proc.txt correctly, it is trying to
> return the (system time) / (total time).  The total time agrees with
> "STAT REAL_TIME".  However, the (system time) that it comes up with here
> is always considerably higher than "STAT CPU_TIME".

CPU_UTIL counts other background processes too.   The foreground process may just wait
when all work is done by pdflush.  I think CPU_UTIL is more useful than CPU_TIME.

> 
> User error is quite possible, as I am:
> 
> 1. Just getting to know mongo.
> 2. Not a perl guy.

i don't like perl too much, but seems Perl is perfect for the things you just did (the fixes).

> 3. There is obviously something mongo.pl doesn't like about my setup.
> 
> -Steve Bergman
> 
> P.S. In the 2 tests I've run, reiser4 is not doing all that badly
> against ext3 in OVERWRITE and MODIFY, though ext3 does come in faster.
> Reiser4 trails badly on APPEND however, ext3 (data=ordered, without
> htree) being some 2.5 - 4 times faster. 

-- 
Alex.

Re: reiser4 plugins

[Alex Zarochentsev]

On Sat, Aug 28, 2004 at 05:09:28AM -0500, Steve Bergman wrote:
> On Fri, 2004-08-27 at 23:54 -0700, Hans Reiser wrote:
> > >
> > I didn't write this (more precisely, it only vaguely resembles what I 
> > wrote in 1996).  Are you saying that it reports system time as real 
> > time?  If yes, then it is an error, we need to go remove a bunch of 
> > numbers from our benchmarks, and thanks for finding it.
> > 
> > Zam, please comment.
> > 
> 
> No.  I'm saying that on my setup (kernel 2.6.8.1-mm4/perl 5.8.5/bash
> 3.00) the first line returned from running the test is the:
> 
> [1]+  Done ...

> line which throws the array indexes off by 1 and I always get 0 for the
> real time and, yes, the real time gets reported as the system time, I
> think.  Plus I get a warning about the fact that "Done" is not numeric.
> This is obviously a problem with my particular setup.

Yes. but that real/sys time parsing isn't reliable.   

> After bumping the indexes up by 1, I get the correct real time reported
> as "STAT REAL_TIME".  And the system time is reported as "STAT
> CPU_TIME".
> 
> "STAT CPU_UTIL", however is computed in a completely different way based
> on numbers collected from /proc/stat.  If I'm, reading
> linux/Documentation/filesystems/proc.txt correctly, it is trying to
> return the (system time) / (total time).  The total time agrees with
> "STAT REAL_TIME".  However, the (system time) that it comes up with here
> is always considerably higher than "STAT CPU_TIME".

CPU_UTIL counts other background processes too.   The foreground process may just wait
when all work is done by pdflush.  I think CPU_UTIL is more useful than CPU_TIME.

> 
> User error is quite possible, as I am:
> 
> 1. Just getting to know mongo.
> 2. Not a perl guy.

i don't like perl too much, but seems Perl is perfect for the things you just did (the fixes).

> 3. There is obviously something mongo.pl doesn't like about my setup.
> 
> -Steve Bergman
> 
> P.S. In the 2 tests I've run, reiser4 is not doing all that badly
> against ext3 in OVERWRITE and MODIFY, though ext3 does come in faster.
> Reiser4 trails badly on APPEND however, ext3 (data=ordered, without
> htree) being some 2.5 - 4 times faster. 

-- 
Alex.

Re: reiser4 plugins

[Alex Zarochentsev]

On Fri, Aug 27, 2004 at 05:29:07PM -0500, Steve Bergman wrote:
> On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
> > >
> > If you ask real users, they say that reiser4 is fast, and their 
> > experience matches our benchmark.  You can criticize the benchmark if 
> > you want, but then you should run your own and publish it.
> > 
> 
> 
> As a "real" desktop user who just converted all his partitions from ext3
> to reiser4, I have not, as yet, noticed any startling performance
> increase.  Being slightly slightly irked to hear that the benchmark
> numbers that have been paraded around on Slashdot and the internet in
> general, at ext3's expense, have had reiser4's "bad" results surgically
> extracted, I am running my own benchmarks to get the real story on
> reiser4/ext3 mongo performance on my, rather average, desktop hardware.
> 
> I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.
> 
> Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
> argument at line 439.
> 
> I have done this to fix it:
> 
> 
> --- mongo.pl    2004-08-27 17:07:01.681723313 -0500
> +++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
> @@ -429,8 +429,8 @@
>         if ( -e ${ERR_FILE}) {
>             &DIE ("\nEXITED WITH FAIL\n");
>         }
> -       my $real = (split ' ', $time_output[0])[1];
> -       my $cpu  = (split ' ', $time_output[2])[1];
> +       my $real = (split ' ', $time_output[1])[1];
> +       my $cpu  = (split ' ', $time_output[3])[1];
>   
>         unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
>           LOG "@time_output";
> 
> 
> What it gets me is the "real" line of the "time" output for "STAT
> REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
> i.e. only system time is counted. I believe this was the intent of the
> original code, but want to verify before continuing.

it works on our test servers which are SuSE9.0/9.1.

I think there is a dependency on system utilities,  "Done" is not printed by
mongo.pl or any other program from the mongo package.

It would be nice to tell us what linux distro you are using and what mongo
phase fails. 

> Thanks,
> Steve Bergman

Thanks.

-- 
Alex.

Re: reiser4 plugins

[Redeeman]

On Fri, 2004-08-27 at 16:04 +0400, Nikita Danilov wrote:
> Hans Reiser writes:
>  > Christophe Saout wrote:
>  > 
>  > >
>  > >
>  > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
>  > >do something specific with a file that is completely transparent to the
>  > >VFS?
>  > >
>  > >  
>  > >
>  > To know what method to use, you must determine the pluginid, and then 
>  > find the method within that plugin for that vfs operation.
>  > 
>  > As for overhead, well, who eats whose dust in the benchmarks....?
> 
> Whoever sponsors the benchmark usually wins. Had you forgotten that
> mongo setup used by http://www.namesys.com/benchmarks.html was specially
> `tuned' to reach peak reiser4 performance? Remember why you decided to
> turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
> _second_ stalls with reiser4 under high io loads (large atom is
> obviously being flushed and everyone waits on it...). In my opinion, it
> is such things that are of utmost importance for real reiser4
> acceptance, not how to name `metas' sub-directory.
i am operating alot under heavy loads, often alot compiling, seti and
some other applications, and i must say, i never noticed this
> 
> Nikita.
> 
-- 
Redeeman <redeeman@metanurb.dk>

Re: reiser4 plugins

[Hans Reiser]

Christoph Hellwig wrote:

>
>  The problem is not that it
>doesn't work but that it's really hard to maintain.  
>
You really like to talk about what you know nothing of, yes?

Reiser4 is FAR easier to maintain than V3.  Ask anyone on reiserfs-dev.  
It was designed to reduce our software maintenance costs. 

Re: reiser4 plugins

[Hans Reiser]

Christoph Hellwig wrote:

>
>  The problem is not that it
>doesn't work but that it's really hard to maintain.  
>
You really like to talk about what you know nothing of, yes?

Reiser4 is FAR easier to maintain than V3.  Ask anyone on reiserfs-dev.  
It was designed to reduce our software maintenance costs. 

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Rik van Riel]

On Thu, 26 Aug 2004, Christophe Saout wrote:

> That's your opinion. reiser4 seems to work very well.

Have you tried /bin/cp, or a backup+restore ?

What happened to your file streams ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Rik van Riel]

On Thu, 26 Aug 2004, Christophe Saout wrote:

> That's your opinion. reiser4 seems to work very well.

Have you tried /bin/cp, or a backup+restore ?

What happened to your file streams ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Chris Mason]

On Thu, 2004-08-26 at 09:52, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Christophe Saout wrote:
> 
> > That's your opinion. reiser4 seems to work very well.
> 
> Have you tried /bin/cp, or a backup+restore ?
> 
> What happened to your file streams ?

Shrug, backup programs can be fixed.  The fact that we couldn't backup
16GB files before O_LARGEFILE wasn't a reason not to do it.

IMHO it's more important to ask the question 'is this important for
linux' and then we can figure out how to make it work.

-chris

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Am Donnerstag, den 26.08.2004, 09:52 -0400 schrieb Rik van Riel:

> > That's your opinion. reiser4 seems to work very well.
> 
> Have you tried /bin/cp, or a backup+restore ?
> 
> What happened to your file streams ?

1. I wasn't talking about reiser4 plugins, not about the
   file-as-a-directory thing or file streams
2. reiser4 doesn't currently have those
3. I don't really like file streams

:-)

But I'm using ACLs on a server and had to forward-port an rsync patch to
make full backups using rsync possible. That's not really better.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Am Donnerstag, den 26.08.2004, 09:52 -0400 schrieb Rik van Riel:

> > That's your opinion. reiser4 seems to work very well.
> 
> Have you tried /bin/cp, or a backup+restore ?
> 
> What happened to your file streams ?

1. I wasn't talking about reiser4 plugins, not about the
   file-as-a-directory thing or file streams
2. reiser4 doesn't currently have those
3. I don't really like file streams

:-)

But I'm using ACLs on a server and had to forward-port an rsync patch to
make full backups using rsync possible. That's not really better.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Thu, Aug 26, 2004 at 03:35:07PM +0200, Christophe Saout wrote:
> Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig:
> 
> > > First you say that that file-as-a-directory is crap then you say that it
> > > does belong into the filesystem?
> > 
> > I think you're talking about something different then me, I'm not
> > talking about the magic meta files but the VFS interface in general.
> > 
> > This VFS interface is an integral part of very filesystem, and it
> > doesn't make a whole lot to put it into a plugin.
> 
> Right. That's why these plugins are linked in uncoditionally. It doesn't
> work without them. Hence "plugins" is not a very good name.

its still plugins no matter what. they just emulate "conventional filesystem"
behavior for VFS, but at some point you might want just to scrap VFS..
if you'd like to.

And I honestly dont understand whats the other Christoph's worries are about.

Its got perfomance. Its there. It can emulate "conventional filesystem" behaviour,
 for legacy apps. Thouse two that are currently crippled by metas, we 
are so happy arguing about, will get fixed fast. Thats a point where you could
happily STOP and live with your happy "conventional filesystem emulation", yet
enjoying perfomance aspects (if perfomance hurts you, do not compile reiserfs,
just like one big-red distro does, nobody would not complain. except your hard-drive
mechanics sick of seeks).
 But most people would not stop, and its good.

Yes, I think it would be nice to have this infrastructure in VFS. Technically.
But its not possible, cause of "committee clusterfuck". Socially. Stupidly.




-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
> Yes, I think it would be nice to have this infrastructure in VFS. Technically.
> But its not possible, cause of "committee clusterfuck". Socially. Stupidly.

Please explain your problems.  I've not seen a single actually working
proposal yet.  Current reiser4 implementation apparently oopses and
breaks guaranteed fs semantics, it's not going in in either reiser4 or
the vfs.

Al has started a thread to hash out working semantics, but there's not been
a single namesys person involved.  Similar all of you have absolutely ignore
every single technical question or comment but started a flamefest
consisting mostly of personal attacs and "mine is longer" politics. 

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 01:22:55PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
> > Yes, I think it would be nice to have this infrastructure in VFS. Technically.
> > But its not possible, cause of "committee clusterfuck". Socially. Stupidly.
> 
> Please explain your problems.  I've not seen a single actually working
> proposal yet.  Current reiser4 implementation apparently oopses and

Not true. If true - send bugreports.

Its work for me couple of months. there were few hiccups, but they got fixed
quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS
limitations. Yes, i use it with LVM2 and stuff...

de facto its even more stable then reiserfs 3.6, which got issues with
CONFIG_PREEMPT.

> breaks guaranteed fs semantics, it's not going in in either reiser4 or
> the vfs.
A
> 
> Al has started a thread to hash out working semantics, but there's not been
> a single namesys person involved.  Similar all of you have absolutely ignore
Al message had a reply.
"namesys persons" is Hans Reiser.  Sufficient ?

> every single technical question or comment but started a flamefest
> consisting mostly of personal attacs and "mine is longer" politics. 
no need to comment on this.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote:
> Not true. If true - send bugreports.
> 
> Its work for me couple of months. there were few hiccups, but they got fixed
> quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS
> limitations. Yes, i use it with LVM2 and stuff...

See the mails from Christophe in this thread.

> > breaks guaranteed fs semantics, it's not going in in either reiser4 or
> > the vfs.
> A
> > 
> > Al has started a thread to hash out working semantics, but there's not been
> > a single namesys person involved.  Similar all of you have absolutely ignore
> Al message had a reply.
> "namesys persons" is Hans Reiser.  Sufficient ?

http://marc.theaimsgroup.com/?t=109355582600002&r=1&w=2

I can't see Hans anywhere.  And honestly Hans has been so out of touch
with VFS internals that some person actually understanding the issue
might be helpfull.  That would probably whoever has taken over Nikita's
position.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 01:51:04PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote:
> > Its work for me couple of months. there were few hiccups, but they got fixed
> > quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS
> > limitations. Yes, i use it with LVM2 and stuff...
> 
> See the mails from Christophe in this thread.

seen. noted. 

> > > breaks guaranteed fs semantics, it's not going in in either reiser4 or
> > > the vfs.
> > A
> > > 
> > > Al has started a thread to hash out working semantics, but there's not been
> > > a single namesys person involved.  Similar all of you have absolutely ignore
> > Al message had a reply.
> > "namesys persons" is Hans Reiser.  Sufficient ?
> 
> http://marc.theaimsgroup.com/?t=109355582600002&r=1&w=2
> 
> I can't see Hans anywhere.  And honestly Hans has been so out of touch
> with VFS internals that some person actually understanding the issue
> might be helpfull.  That would probably whoever has taken over Nikita's
> position.

I assume Vladimir Saveliev could play this out...


-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Markus Törnqvist]

On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote:
>Not true. If true - send bugreports.

Hate to say it, but apparently symlinking out of metas oopses, but that
should be trivial to fix?

>> Al has started a thread to hash out working semantics, but there's not been
>> a single namesys person involved.  Similar all of you have absolutely ignore
>Al message had a reply.
>"namesys persons" is Hans Reiser.  Sufficient ?

Not to be disrespectful, but Hans has sometimes taken personal shots on the
lists and that's not a productive approach. Not that he'd be the only one
around to do so.

Seems to me, as Christoph said, that some other Namesys guy than Hans should
participate in the Al/Linus branch of the thread with ideas.

Certainly the VFS may have to be extended right about now, so someone who
knows the exact details of Reiser4 should offer ideas and I'm not so sure
it's Hans. Looks like he had the ideas but not the implementation.

There's also the risk that Reiser4's implementation's don't really bend
into even a modified/extended VFS because of the plugins?
That's definitely something that has to be approached properly...

-- 
mjt

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Jan Harkes]

On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
> And I honestly dont understand whats the other Christoph's worries are about.

I honestly don't understand what all this 'grassroot' campaining is all
about. Very valid technical points have been made and mostly ignored.

The whole 'it breaks backups' noise is just that, noise.

- It breaks VFS locking rules.
- It pretends to provide a uniform way to store streams or metadata.
  However it only does so files and not for directories, symlinks,
  fifos, unix domain sockets, or device nodes, which makes it less than
  uniform.
- From what I saw in one part of the discussion, it allows for infinite
  depth recursion (file/metas/metas/metas/...). Some applications are
  going to have a lot of fun with that.

And finally,

- When reiserfs3 got merged, it introduced iget3 and read_inode2 in the
  VFS layer. Later on when I started to use them for Coda I almost
  immediately found serious consistency problems, resulting in the
  iget4_locked implementation in the 2.5 kernels.
  
  I don't think anyone ever fixed that race in reiser3. It should hit
  occasionally on concurrent lookups on SMP or preempt kernels. In 2.4,
  Coda needed a semaphore to prevent concurrency during the iget3 lookup
  until the new inode is actually initialized.

> Its got perfomance. Its there. It can emulate "conventional
> filesystem" behaviour, for legacy apps.

So does ramfs.

> Yes, I think it would be nice to have this infrastructure in VFS.
> Technically. But its not possible, cause of "committee clusterfuck".
> Socially. Stupidly.

It will get there, just consider it constructive criticism. If we can
help resolve or refute the technical issues, all the better. We might
even end up with improvements or extensions to the VFS or MM making life
easier for everyone.

Jan

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Nikita Danilov]

Jan Harkes writes:
 > On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
 > > And I honestly dont understand whats the other Christoph's worries are about.
 > 

[...]

 > - When reiserfs3 got merged, it introduced iget3 and read_inode2 in the
 >   VFS layer. Later on when I started to use them for Coda I almost
 >   immediately found serious consistency problems, resulting in the
 >   iget4_locked implementation in the 2.5 kernels.
 >   
 >   I don't think anyone ever fixed that race in reiser3. It should hit

Err... it was fixed.


 > Jan
 > 

Nikita.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 12:11:14PM -0400, Jan Harkes wrote:
> On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
> 
> It will get there, just consider it constructive criticism. If we can
> help resolve or refute the technical issues, all the better. We might
> even end up with improvements or extensions to the VFS or MM making life
> easier for everyone.

That would be nice. Some nice points too (metas/metas in just noice tough).

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> This VFS interface is an integral part of Фvery filesystem, and it

VFS never was "an integral part" of ANY filesystem. my dog knows it.
its just unified INTERFACE TO any filesystem (including reiser4).


P.S. I imagine, how much flamed it would be if reiser4 made any intensive
changes in linux VFS code...



-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> This VFS interface is an integral part of Фvery filesystem, and it

VFS never was "an integral part" of ANY filesystem. my dog knows it.
its just unified INTERFACE TO any filesystem (including reiser4).


P.S. I imagine, how much flamed it would be if reiser4 made any intensive
changes in linux VFS code...



-- 
"the liberation loophole will make it clear.."
lex lyamin
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> This VFS interface is an integral part of Фvery filesystem, and it

VFS never was "an integral part" of ANY filesystem. my dog knows it.
its just unified INTERFACE TO any filesystem (including reiser4).


P.S. I imagine, how much flamed it would be if reiser4 made any intensive
changes in linux VFS code...



-- 
"the liberation loophole will make it clear.."
lex lyamin
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote:
> Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> > This VFS interface is an integral part of ??very filesystem, and it
> 
> VFS never was "an integral part" of ANY filesystem. my dog knows it.
> its just unified INTERFACE TO any filesystem (including reiser4).

You's misquoting me.  IF you quoted the whole context it'd be pretty
sure that the part of the filesystem that intefaces with the VFS is
meant.

But one could even say VFS is integral part of a linux filesystem as
it does most of the work a filesystem driver does in other operating
systems.

> P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> changes in linux VFS code...

It really depends on how you sent them.  If you had a big patch without
explanations - sure.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 01:12:33PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote:
> > Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> > > This VFS interface is an integral part of ??very filesystem, and it
> > 
> > VFS never was "an integral part" of ANY filesystem. my dog knows it.
> > its just unified INTERFACE TO any filesystem (including reiser4).
> 
> You's misquoting me.  IF you quoted the whole context it'd be pretty
> sure that the part of the filesystem that intefaces with the VFS is
> meant.

No. Its not me "misquoting", its just someone sound plain ?incoherrent?.
Even if I overquoted reply with whole message, its still sound ?incohherent?.

> 
> But one could even say VFS is integral part of a linux filesystem as
> it does most of the work a filesystem driver does in other operating
> systems.

theres no "linux filesystem". there are "linux filesystems".
thanks god.

But I it would be really grate if you'll elaborate your sentence with
example of VFS functionality (lack of it) on said "other operating systems"
and if you'll define "most of work".


> 
> > P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> > changes in linux VFS code...
> 
> It really depends on how you sent them.  If you had a big patch without
> explanations - sure.
It would work with small tweaks, but you just can take a look at reiser4
code and you'll understand that it just could not be chopped in
"set of small patches" altough it could be documented better ofcourse,
but its really well commented already.

some times, some approaches to  some problems  just would not work.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote:
> > But one could even say VFS is integral part of a linux filesystem as
> > it does most of the work a filesystem driver does in other operating
> > systems.
> 
> theres no "linux filesystem". there are "linux filesystems".
> thanks god.

a linux filesystem, not the linux filesystem, please read again.

> But I it would be really grate if you'll elaborate your sentence with
> example of VFS functionality (lack of it) on said "other operating systems"
> and if you'll define "most of work".

most trivial example is namespace locking, in *BSD, Windows, SVR4 and
derivates it's done in the lowlevel filesystem.  In plan9, Linux and
soon DragonlyBSD it's done in the VFS. 

> > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> > > changes in linux VFS code...
> > 
> > It really depends on how you sent them.  If you had a big patch without
> > explanations - sure.
> It would work with small tweaks, but you just can take a look at reiser4
> code and you'll understand that it just could not be chopped in
> "set of small patches" altough it could be documented better ofcourse,
> but its really well commented already.
> 
> some times, some approaches to  some problems  just would not work.

You still haven't even bother explaining what you want to do.  It's hard
to argue against vague uncertainity.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 03:56:55PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote:
> > > But one could even say VFS is integral part of a linux filesystem as
> > > it does most of the work a filesystem driver does in other operating
> > > systems.
> > 
> > theres no "linux filesystem". there are "linux filesystems".
> > thanks god.
> 
> a linux filesystem, not the linux filesystem, please read again.
> 
> > But I it would be really grate if you'll elaborate your sentence with
> > example of VFS functionality (lack of it) on said "other operating systems"
> > and if you'll define "most of work".
> 
> most trivial example is namespace locking, in *BSD, Windows, SVR4 and
> derivates it's done in the lowlevel filesystem.  In plan9, Linux and
> soon DragonlyBSD it's done in the VFS. 

ok. good examples.

> 
> > > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> > > > changes in linux VFS code...
> > > 
> > > It really depends on how you sent them.  If you had a big patch without
> > > explanations - sure.
> > It would work with small tweaks, but you just can take a look at reiser4
> > code and you'll understand that it just could not be chopped in
> > "set of small patches" altough it could be documented better ofcourse,
> > but its really well commented already.
> > 
> > some times, some approaches to  some problems  just would not work.
> 
> You still haven't even bother explaining what you want to do.  It's hard
> to argue against vague uncertainity.

 o files as directories -  no oppinion on that.
                            
 o metafiles -   AFAIK it was product of Nikita Danilov just playing and fooling.

Probably solution that will satisfy you is to have "legacy (dumb?) mode" which
is leaving all this fancy stuff out of sight.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 11:23:50PM +0400, Alexander Lyamin wrote:
> Sat, Aug 28, 2004 at 03:56:55PM +0200, Christoph Hellwig wrote:
> > > 
> > > some times, some approaches to  some problems  just would not work.
> > 
> > You still haven't even bother explaining what you want to do.  It's hard
> > to argue against vague uncertainity.
> 
>  o files as directories -  no oppinion on that.
>                             
>  o metafiles -   AFAIK it was product of Nikita Danilov just playing and fooling.
> 
> Probably solution that will satisfy you is to have "legacy (dumb?) mode" which
> is leaving all this fancy stuff out of sight.

and enabled compile time or mount option...

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins

[Hans Reiser]

Alexander Lyamin wrote:

>
>                            
> o metafiles -   AFAIK it was product of Nikita Danilov just playing and fooling.
>  
>
No, it was not, it was an idea I had for so long I cannot remember when, 
maybe 1984, and I proposed it to darpa, and then nikita got told about 
it probably from being told to read the web page on our darpa project, 
and he took on the task of coding it.

Hans

Re: reiser4 plugins

[Hans Reiser]

Alexander Lyamin wrote:

>
>                            
> o metafiles -   AFAIK it was product of Nikita Danilov just playing and fooling.
>  
>
No, it was not, it was an idea I had for so long I cannot remember when, 
maybe 1984, and I proposed it to darpa, and then nikita got told about 
it probably from being told to read the web page on our darpa project, 
and he took on the task of coding it.

Hans

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Markus Törnqvist]

On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote:
>P.S. I imagine, how much flamed it would be if reiser4 made any intensive
>changes in linux VFS code...

Surely it would be flamefest galore, but thanks to Reiser4 there may be
some VFS changes to that direction, maybe completely from Reiser4.
Should keep some people at bay if the chances weren't so much replacing
VFS than extending it.

-- 
mjt

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Sat, 28 Aug 2004, Alexander Lyamin wrote:
> 
> VFS never was "an integral part" of ANY filesystem. my dog knows it.

That's not really true.

Name handling (dentry layer, mounting) is very much an integral part of
the filesystem. Almost everything else in the VFS is "helper functions",
ie a filesystem can choose to ignore buffer heads, page cache etc, but a 
filesystem really cannot ignore or override the VFS naming stuff.

(Arguably the page cache isn't part of the VFS layer at all, it's really a 
memory management thing, although it's so intertwined with the VFS helper 
functions that you can't really draw the line).

> its just unified INTERFACE TO any filesystem (including reiser4).

True, but that's not the whole truth. It's way more than just an
interface. It's a set of rules, and it's in many way the controlling
party.

		Linus

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 10:18:44AM -0700, Linus Torvalds wrote:
>
>
> On Sat, 28 Aug 2004, Alexander Lyamin wrote:
> >
> > VFS never was "an integral part" of ANY filesystem. my dog knows it.
>
> That's not really true.

Theres some truth. And its really arguably, but I think, I see your point.

And taking on the situation with reiser4 with "smartmost VFS" approach,
time should pass and with time and expirience on hands it will be evident what
should go in VFS. Now its may be too early.

Considering "amazing PR skills" of Hans Reiser it was the only viable way
to get this changes in VFS. Cause, ironically, mr. Hellwig that currently
demand it to be scrapped out or go in VFS would instakill Hans Reiser
(i know many people would:) if he only touched holy cow of VFS for any
reiser4 purpose.

As I told before, some technically right things just do not work in this
imperfect world. You have to count with social issues, and Hans Reiser
wont, he just avoid them whenever its possible. 

> Name handling (dentry layer, mounting) is very much an integral part of
> the filesystem. Almost everything else in the VFS is "helper functions",
> ie a filesystem can choose to ignore buffer heads, page cache etc, but a 
> filesystem really cannot ignore or override the VFS naming stuff.
> 
> (Arguably the page cache isn't part of the VFS layer at all, it's really a 
> memory management thing, although it's so intertwined with the VFS helper 
> functions that you can't really draw the line).
> 
> > its just unified INTERFACE TO any filesystem (including reiser4).
> 
> True, but that's not the whole truth. It's way more than just an
> interface. It's a set of rules, and it's in many way the controlling
> party.
> 
> 		Linus

"True, but thats not the whole truth."

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 11:03:50PM +0400, Alexander Lyamin wrote:
> And taking on the situation with reiser4 with "smartmost VFS" approach,
> time should pass and with time and expirience on hands it will be evident what
> should go in VFS. Now its may be too early.
> 
> Considering "amazing PR skills" of Hans Reiser it was the only viable way
> to get this changes in VFS. Cause, ironically, mr. Hellwig that currently
> demand it to be scrapped out or go in VFS would instakill Hans Reiser
> (i know many people would:) if he only touched holy cow of VFS for any
> reiser4 purpose.

Could you please stop talking such inproven crap?

Re: reiser4 plugins

[Hans Reiser]

I think it is reasonable to make the -nopseudos (turns off the metafiles 
) mount option mandatory, until the bugs are resolved.

Our testing did not find these metafile/VFS bugs because of the reason 
for all our bugs, we screwed up. 

There is a distinct difference between some persons and I, which is that 
some think all of reiser4 should be excluded until metafiles are 
implemented by VFS some long time from now, and I, in that I merely 
think buggy optional features should be turned off until they are 
fixed.  I, being renowned for my paranoia and asininity as I am, think 
these persons find it convenient as an excuse to keep us from competing, 
and I think that if we were slower there would be less hassle every time 
we try to get into the kernel. 

While reiser4 has some significant roughnesses remaining in its 
performance, I think the average user would find it performs better than 
other filesystems, and is stable enough for, say, a laptop, and I 
predict that by the time we have it stable enough for mission critical 
servers, all the roughness in various important corner cases will be 
gone.  

Persons benchmarking it with tarballs, please be sure to use tarballs 
created on reiser4, not ext2 tarballs, readdir order matters a lot for 
sorted directory filesystems.

Hans

Re: reiser4 plugins

[Hans Reiser]

I think it is reasonable to make the -nopseudos (turns off the metafiles 
) mount option mandatory, until the bugs are resolved.

Our testing did not find these metafile/VFS bugs because of the reason 
for all our bugs, we screwed up. 

There is a distinct difference between some persons and I, which is that 
some think all of reiser4 should be excluded until metafiles are 
implemented by VFS some long time from now, and I, in that I merely 
think buggy optional features should be turned off until they are 
fixed.  I, being renowned for my paranoia and asininity as I am, think 
these persons find it convenient as an excuse to keep us from competing, 
and I think that if we were slower there would be less hassle every time 
we try to get into the kernel. 

While reiser4 has some significant roughnesses remaining in its 
performance, I think the average user would find it performs better than 
other filesystems, and is stable enough for, say, a laptop, and I 
predict that by the time we have it stable enough for mission critical 
servers, all the roughness in various important corner cases will be 
gone.  

Persons benchmarking it with tarballs, please be sure to use tarballs 
created on reiser4, not ext2 tarballs, readdir order matters a lot for 
sorted directory filesystems.

Hans

Re: reiser4 plugins

[Herbert Poetzl]

On Sat, Aug 28, 2004 at 02:41:12PM -0700, Hans Reiser wrote:
> I think it is reasonable to make the -nopseudos (turns off the metafiles 
> ) mount option mandatory, until the bugs are resolved.
> 
> Our testing did not find these metafile/VFS bugs because of the reason 
> for all our bugs, we screwed up. 
> 
> There is a distinct difference between some persons and I, which is that 
> some think all of reiser4 should be excluded until metafiles are 
> implemented by VFS some long time from now, and I, in that I merely 
> think buggy optional features should be turned off until they are 
> fixed.  I, being renowned for my paranoia and asininity as I am, think 
> these persons find it convenient as an excuse to keep us from competing, 
> and I think that if we were slower there would be less hassle every time 
> we try to get into the kernel. 
> 
> While reiser4 has some significant roughnesses remaining in its 
> performance, I think the average user would find it performs better than 
> other filesystems, and is stable enough for, say, a laptop, and I 
> predict that by the time we have it stable enough for mission critical 
> servers, all the roughness in various important corner cases will be 
> gone.  
> 
> Persons benchmarking it with tarballs, please be sure to use tarballs 
> created on reiser4, not ext2 tarballs, readdir order matters a lot for 
> sorted directory filesystems.

hmm, so probably we have to wait until all
tar packagers moved to reiser4, so that the
available tar files are 'sorted properly' ...

best,
Herbert

> Hans
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reiser4 plugins

[Hans Reiser]

Herbert Poetzl wrote:

>
>hmm, so probably we have to wait until all
>tar packagers moved to reiser4, so that the
>available tar files are 'sorted properly' ...
>
>best,
>Herbert
>
>  
>
Or wait for a repacker, which will probably happen sooner. Distros that 
use reiser4 will probably tar in reiser4 order.

Re: reiser4 plugins

[Hans Reiser]

Herbert Poetzl wrote:

>
>hmm, so probably we have to wait until all
>tar packagers moved to reiser4, so that the
>available tar files are 'sorted properly' ...
>
>best,
>Herbert
>
>  
>
Or wait for a repacker, which will probably happen sooner. Distros that 
use reiser4 will probably tar in reiser4 order.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Sat, 28 Aug 2004, Alexander Lyamin wrote:
> 
> Considering "amazing PR skills" of Hans Reiser it was the only viable way
> to get this changes in VFS. Cause, ironically, mr. Hellwig that currently
> demand it to be scrapped out or go in VFS would instakill Hans Reiser
> (i know many people would:) if he only touched holy cow of VFS for any
> reiser4 purpose.

Hey, you don't have to try to convince me - I haven't been arguing against 
the reiser4 approach. I think it's a prototyping scheme, and quite frankly 
I don't think you _can_ get the name lookup right without real VFS support 
(races, namespaces etc are just not something you can solve in the 
filesystem). 

But the fact that name lookup does seriously need VFS support means that I 
don't think we want to merge reiser4 as-is. No way. That doesn't mean that 
I think reiser4 was _wrong_.

And I certainly agree that there are personality issues, and they are not 
all just Hans' ;)

			Linus

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig:

> > The reiser4 core doesn't know about inodes, directories or files. It's
> > the glue code between the VFS and the storage layer that does. It's
> > implemented as a plugin. This has nothing to do with semantic
> > enhancements yet. These should be removed for now and made a 2.7 topic.
> 
> Oh yes, it is.  As soon as you use different access methods on an
> overlapping set of objects you see exactly the problems I described.

Right. That shouldn't happen. Since an object is usally tied to exactly
one plugin this can't happen I think. At least as long the programmer
doesn't mess anything up.

> If they don't overlap there's no point for the plugins to start with,
> you'll better turn the core into a library that can be used by different
> projects then.

I think that's what Hans is trying to suggest. The core is theoretically
usable for anything. He made a filesystem of it.

You can see reiser4 as a library and the UNIX directory and file plugins
as the filesystem. That's why they are linked in uncoditionally because
it doesn't make sense otherwise. It's mainly the naming that seems to
confuse people.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig:

> > The reiser4 core doesn't know about inodes, directories or files. It's
> > the glue code between the VFS and the storage layer that does. It's
> > implemented as a plugin. This has nothing to do with semantic
> > enhancements yet. These should be removed for now and made a 2.7 topic.
> 
> Oh yes, it is.  As soon as you use different access methods on an
> overlapping set of objects you see exactly the problems I described.

Right. That shouldn't happen. Since an object is usally tied to exactly
one plugin this can't happen I think. At least as long the programmer
doesn't mess anything up.

> If they don't overlap there's no point for the plugins to start with,
> you'll better turn the core into a library that can be used by different
> projects then.

I think that's what Hans is trying to suggest. The core is theoretically
usable for anything. He made a filesystem of it.

You can see reiser4 as a library and the UNIX directory and file plugins
as the filesystem. That's why they are linked in uncoditionally because
it doesn't make sense otherwise. It's mainly the naming that seems to
confuse people.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig and Christophe Saout, I think you two are
miscommunicating a little.

I've never looked at reiser4 code, but I think I understand what you
are both saying, so I will explain.

Christoph Hellwig wrote:
> Here comes the problem.  All the access checking/race avoidance/loop
> creation avoiced, in short the posix+extension semantics are implemented
> in the Linux VFS layer.  If you want to allow a second access method
> (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
> inside reiser4.

That would be good.  Even better would be if VFS implemented the best
semantics that we agree is nice, after some experimenting with reiser4
lets us discover what works well.

> > The reiser4 core doesn't know about inodes, directories or files. It's
> > the glue code between the VFS and the storage layer that does. It's
> > implemented as a plugin. This has nothing to do with semantic
> > enhancements yet. These should be removed for now and made a 2.7 topic.
> 
> Oh yes, it is.  As soon as you use different access methods on an
> overlapping set of objects you see exactly the problems I described.
> 
> If they don't overlap there's no point for the plugins to start with,
> you'll better turn the core into a library that can be used by different
> projects then.

Firstly, the core library _is_ used by different projects.  That's the
commercial aim of reiser4: to be used by projects other than just as
Linux filesystem.  For example it might be used by a database, or a
portable application which needs to store structured data in a flat
file (with random access).  A good abstraction makes it more useful
than "just" a filesystem.

Secondly, the reiser4 design makes the "high-level plugin" interface a
good clean abstraction.  It's conceptually simple and versatile, so
simplifies the code and makes it easier to try new ideas as well.

I think the disagreement between you two comes down to the idea that a
generally useful API abstraction to file-like objects really should be
somewhere in the VFS, and not specific to a single filesystem.

Unfortunately, the problem is that reiser4 is the only filesystem
which is _technically capable_ of implementing that abstraction in a
practical way, apparently.  (I'm not sure if this is really true.
reiser4's object model is not the same as paths and inodes, but the
impedance mismatch doesn't seem huge.)

Perhaps in future some other filesystems could do it.  Then it would
make sense for the API to be common between them.

At the same time, some of the features which Hans Reiser would like to
offer users -- especially things like metadata indexing -- don't seem
to be very practical on top of the VFS abstraction.

These are conflicting goals.  Hence the argument.

VFS is very POSIX, which rules out some interesting things like
multiple views of an ordinary file (i.e. viewing a flat file and it's
structured form together), coherent files (i.e. metadata which is
calculated from the file contents and always guaranteed to match it),
and real time indexing (i.e. "grep", "locate" and "local-google" but
fancier, always up to date, and still fast).

Those features can be implemented poorly on top of POSIX, and hence
poorly on top of VFS.  We do that today.  It works, it's simple, but
it sucks.  Sometimes it goes wrong (index doesn't match files), it's
slow (locate isn't real-time, and if grep kept content indexes it
would still be too slow to verify them), and it's not really simple as
you have to use special tools for every file format these days, each
with their own unique interface.

I think we can tidy that up a lot, still using the VFS, but with some
extra hooks.  But it's not clear what hooks those would be, and how
the interesting features would be implemented by other POSIX-only
filesystems.  It'd be more useful: I want the fancy stuff to work over
NFS to a bog standard server, or over an old ext2 filesystem that
mustn't be upgraded, not just reiser4, even if I don't have the same
performance and it has to create lots of articial "metadata" files to
simulate the fancy stuff.

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

> Firstly, the core library _is_ used by different projects.  That's the
> commercial aim of reiser4: to be used by projects other than just as
> Linux filesystem.  For example it might be used by a database, or a
> portable application which needs to store structured data in a flat
> file (with random access).  A good abstraction makes it more useful
> than "just" a filesystem.

Which is okay as long as it still fits in the kernel.  Remember that
lots of projects had to adjust so far despite the commerical aims of
their creators.

> I think the disagreement between you two comes down to the idea that a
> generally useful API abstraction to file-like objects really should be
> somewhere in the VFS, and not specific to a single filesystem.

IT should, and in fact it actually is most of the part.  We can have
differnt operation vectors per object, but reiser4 is the only
filesystem in the tree not making use of that but provinding only a
single one and then dispatching internally.  Even XFS that carries
around and horrible internal dispatching layer as IRIX legacy gets
this right.

> Unfortunately, the problem is that reiser4 is the only filesystem
> which is _technically capable_ of implementing that abstraction in a
> practical way, apparently.  (I'm not sure if this is really true.
> reiser4's object model is not the same as paths and inodes, but the
> impedance mismatch doesn't seem huge.)

Umm, no.  In fact every filesystem does this.  There's not too many
objects with different namespace semantic: regular files and special
files vs directories vs symlinks basically, but as soon as you go to the
data patch you can have hundrets of different file_operations on a
single filesystems (for special files).

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > Unfortunately, the problem is that reiser4 is the only filesystem
> > which is _technically capable_ of implementing that abstraction in a
> > practical way, apparently.  (I'm not sure if this is really true.
> > reiser4's object model is not the same as paths and inodes, but the
> > impedance mismatch doesn't seem huge.)
> 
> Umm, no.  In fact every filesystem does this.  There's not too many
> objects with different namespace semantic: regular files and special
> files vs directories vs symlinks basically, but as soon as you go to the
> data patch you can have hundrets of different file_operations on a
> single filesystems (for special files).

Remember that reiser4 allows you to operate on little pieces of data,
glueing and rearranging them inside files (something like that).

No other filesystem has that capability, and it's a data model which
the fancy features (when they exist) will use.

You can map those pieces to underlying directories and files and
renames and unlinks, so that the fancy stuff works on other
filesystems, but it would be a useless model because those other
filesystems wouldn't be recognisably "ordinary" files any more.

For reiser4 to expose that model through a VFS interface, and the
fancy stuff to use it through the VFS interface, and for the fancy
stuff to work (even imperfectly) on other filesystems which don't
offer those operations, some kind of fall-back "store metadata and
fragment rearrangements in auxiliary files with special names" layer
would be requied.  That's a big job.

I think it's a good job to do (funding, anyone? :), and the right
place for that (imperfect but useful) fall-back layer is userspace
with perhaps minimal VFS support.  Another reason to put the fall-back
layer in userspace is that applications which depend on the fancy
stuff can still be portable to other OSes, working, but a bit
smoother, faster and more "integrated" on reiser4.

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 05:19:11PM +0100, Jamie Lokier wrote:
> Remember that reiser4 allows you to operate on little pieces of data,
> glueing and rearranging them inside files (something like that).
> 
> No other filesystem has that capability, and it's a data model which
> the fancy features (when they exist) will use.
> 
> You can map those pieces to underlying directories and files and
> renames and unlinks, so that the fancy stuff works on other
> filesystems, but it would be a useless model because those other
> filesystems wouldn't be recognisably "ordinary" files any more.
> 
> For reiser4 to expose that model through a VFS interface, and the
> fancy stuff to use it through the VFS interface, and for the fancy
> stuff to work (even imperfectly) on other filesystems which don't
> offer those operations, some kind of fall-back "store metadata and
> fragment rearrangements in auxiliary files with special names" layer
> would be requied.  That's a big job.

I seem to miss those files in the current implementation.  But again the
big question is how will the interface look?  If the smaller objects
interact with normal files (link/rename, contained in normal
directories), they will have to obey the same locking protocols.  If not
a filesystem could experiment with it of course.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> If the smaller objects interact with normal files (link/rename,
> contained in normal directories), they will have to obey the same
> locking protocols.  If not a filesystem could experiment with it of
> course.

I think the smaller objects will consist of special files, which are
magic attributes like the ones you currently see in reiser4, and
ordinary files (which might themselves be containers).

Link/rename/locking etc. should work as usual on the ordinary smaller
files, but probably aren't allowed at all on the special attribute
files, except where there's an appropriate meaning.  (For example
locking certain attributes might have an appropriate meaning).

It would make sense for the special attributes file to not identify as
"regular files" (S_IFREG), but there aren't many mode bits to play
with.  They should also be harder to find than ordinary smaller files.

-- Jamie

Re: silent semantic changes with reiser4

[Hans Reiser]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton:
>
>  
>
>>And describe the "plugin" system.  Why does the filesystem need such a
>>thing (other filesystems get their features via `patch -p1')?
>>    
>>
>
>Ok, let me try. (Hans, please don't shoot me if I don't get every detail
>right, I'm trying to simplify and translate)
>
>The reiser4 core is just a big and fast storage layer using a single
>tree. It is bound to a mount point and uses the page cache to manage its
>memory. The only thing it can do on its own is to flush dirty data to
>the disk when the VM wants to (memory pressure) or the VFS wants to
>(unmount, sync) or some "plugin" wants to.
>Additionally it's completely atomic (with respect to crashes, no
>isolation like in databases), comparable to data=journal but without the
>overhead of using a journal for everything.
>
>Up to this point there are no users of this "database" and it does not
>implement any of the VFS methods for a filesystem (except mount and
>unmount perhaps).
>
>All the functionality is provided to
>
^to^via
apologies for the correction but it might confuse some

> the plugins. You can insert,
>remove, lookup or modify key:object pairs (with an index into the
>object). The object
>
^object^item
perform that substitution everywhere in this email

> is a sequence of units. For files a unit would be 1
>byte and the index would be the byte offset. For directories the unit
>would be 1 entry and the index would be the filename.
>
>Now there are some plugins that define how the storage layout on the
>disk is (some kind of "backend" plugins).
>
>*And* there are plugins which are users of the "reiser4 client API" and
>implement the actual VFS methods.
>
>There's a UNIX directory plugin and a UNIX file plugin.
>
>Directories, inodes and file content are just key:object pairs and the
>plugins know how to operate on these.
>
>There's a new plugin in work that also implements UNIX file semantics
>but stores the data for that file encrypted and/or compressed.
>
>These plugins live between the VFS and the storage layer.
>
>Just like the filesystems live between the VFS and the block layer (Hans
>would say that filesystems are VFS plugins ;-)).
>
>  
>
>>And what are the licensing implications of plugins?  Are they derived
>>works?  Must they be GPL'ed?
>>    
>>
>
>I suppose yes, at least currently, since they can only be linked with
>reiser4, there's no module infrastructure.
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton:
>
>  
>
>>And describe the "plugin" system.  Why does the filesystem need such a
>>thing (other filesystems get their features via `patch -p1')?
>>    
>>
>
>Ok, let me try. (Hans, please don't shoot me if I don't get every detail
>right, I'm trying to simplify and translate)
>
>The reiser4 core is just a big and fast storage layer using a single
>tree. It is bound to a mount point and uses the page cache to manage its
>memory. The only thing it can do on its own is to flush dirty data to
>the disk when the VM wants to (memory pressure) or the VFS wants to
>(unmount, sync) or some "plugin" wants to.
>Additionally it's completely atomic (with respect to crashes, no
>isolation like in databases), comparable to data=journal but without the
>overhead of using a journal for everything.
>
>Up to this point there are no users of this "database" and it does not
>implement any of the VFS methods for a filesystem (except mount and
>unmount perhaps).
>
>All the functionality is provided to
>
^to^via
apologies for the correction but it might confuse some

> the plugins. You can insert,
>remove, lookup or modify key:object pairs (with an index into the
>object). The object
>
^object^item
perform that substitution everywhere in this email

> is a sequence of units. For files a unit would be 1
>byte and the index would be the byte offset. For directories the unit
>would be 1 entry and the index would be the filename.
>
>Now there are some plugins that define how the storage layout on the
>disk is (some kind of "backend" plugins).
>
>*And* there are plugins which are users of the "reiser4 client API" and
>implement the actual VFS methods.
>
>There's a UNIX directory plugin and a UNIX file plugin.
>
>Directories, inodes and file content are just key:object pairs and the
>plugins know how to operate on these.
>
>There's a new plugin in work that also implements UNIX file semantics
>but stores the data for that file encrypted and/or compressed.
>
>These plugins live between the VFS and the storage layer.
>
>Just like the filesystems live between the VFS and the block layer (Hans
>would say that filesystems are VFS plugins ;-)).
>
>  
>
>>And what are the licensing implications of plugins?  Are they derived
>>works?  Must they be GPL'ed?
>>    
>>
>
>I suppose yes, at least currently, since they can only be linked with
>reiser4, there's no module infrastructure.
>
>  
>

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:31:34AM -0700, Hans Reiser wrote:
> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> for Apple, and that means we need to put search engine and database 

Dou you know a nice thing?  We (as in the Linux Community) don't have to
compete with anyone.  Sure, we're usually trying to be better than
anyone else, but unlike companies under maret pressure we can wait until
something is ready.

Again, if you think you want your work in Linux cooperate with the Linux
Developers, and as you have seen in this thread there's a pretty broad
consensus that we want things working at the VFS (and actually working
of course)

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 11:34:07AM +0200, Christoph Hellwig wrote:
>
>Dou you know a nice thing?  We (as in the Linux Community) don't have to
>compete with anyone.  Sure, we're usually trying to be better than
>anyone else, but unlike companies under maret pressure we can wait until
>something is ready.

It won't get ready unless it's made ready, yes?

And what about the time when people will move away from Linux because
Linux has nothing but crappy file systems to offer?
Sure, it's not that we'd have to compete, but...

>Again, if you think you want your work in Linux cooperate with the Linux
>Developers, and as you have seen in this thread there's a pretty broad
>consensus that we want things working at the VFS (and actually working
>of course)

So move it into VFS. Neext! ;)

-- 
mjt

Re: silent semantic changes with reiser4

[Thomas Cataldo]

On Thu, 2004-08-26 at 11:34 +0200, Christoph Hellwig wrote:
> On Thu, Aug 26, 2004 at 01:31:34AM -0700, Hans Reiser wrote:
> > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> > for Apple, and that means we need to put search engine and database 
> 
> Dou you know a nice thing?  We (as in the Linux Community) don't have to
> compete with anyone.  Sure, we're usually trying to be better than
> anyone else, but unlike companies under maret pressure we can wait until
> something is ready.

By the way, this kind of features can be achieved easily in userspace
with, for example, ext3, beagle (a userspace fulltext indexing daemon)
and some (d|i)notify magic.

Re: silent semantic changes with reiser4

[Redeeman]

On Thu, 2004-08-26 at 01:31 -0700, Hans Reiser wrote:
> Andrew Morton wrote:
> 
> >Hans Reiser <reiser@namesys.com> wrote:
> >  
> >
> >>I had not intended to respond to this because I have nothing positive to 
> >>say, but Andrew said I needed to respond and suggested I should copy 
> >>Linus.
> >>    
> >>
> >
> >Yes, but I didn't say "flame Christoph and ignore the issues" ;)
> >  
> >
> Oh....;-)
> 
> >There are lots of little things to do with implementation, coding style,
> >module exports, deadlocks, what code goes where, etc.  These are all normal
> >daily kernel business and we should set them aside for now and concentrate
> >on the bigger issues.
> >  
> >
> Yes, you are right, but I am not sure Viro will go along with that..... ;-)
> 
> >And as I see it, there are two big issues:
> >
> >a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
> >   anticipate and 
> >
> >b) it does this within the context of just a single filesystem.
> >
> >I see three possible responses:
> >
> >a) accept the reiser4-only extensions as-is (possibly with post-review
> >   modifications, of course) or
> >
> >b) accept the reiser4-only extensions with a view to turning them into
> >   kernel-wide extensions at some time in the future, so all filesystems
> >   will offer the extensions (as much as poss) or
> >
> >c) reject the extensions.
> >
> >
> >My own order of preference is b) c) a). 
> >
> I don't object to b), though I think b) should wait for 2.7 and reiser4 
> should not.
> 
> > The fact that one filesystem will
> >offer features which other filesystems do not and cannot offer makes me
> >queasy for some reason.
> >  
> >
> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> for Apple, and that means we need to put search engine and database 
> functionality into the filesystem.  It takes 11 years of serious 
> research to build a clean storage layer able to handle doing that.  
> Reiser4 has done that, finally.  None of the other Linux filesystems 
> have.  The next major release of ReiserFS is going to be bursting with 
> semantic enhancements, because the prerequisites for them are in place 
> now.  None of the other Linux filesystems have those prerequisites.  
> They won't be able to keep up with the semantic enhancements.  This 
> metafiles and file-directories stuff is actually fairly trivial stuff.
> 
> Look guys, in 1993 I anticipated the battle would be here, and I build 
> the foundation for a defensive tower right at the spot MS and Apple are 
> now maneuvering towards.  Help me get the next level on the tower before 
> they get here.  It is one hell of a foundation, they won't be able to 
> shake it, their trees are not as powerful.  Don't move reiser4 into vfs, 
> use reiser4 as the vfs.  Don't write filesystems, write file plugins and 
> disk format plugins and all the other kinds of plugins, and you won't be 
> missing any expressive power that you really want....
> 
couldnt be more true

> Give Saveliev and I some credit.  10 years of hard work at an ivory 
> tower nobody thought mattered.  Now the battle leaves the browser and 
> swings our way.  Don't duplicate that infrastructure, use it. 
> 
> There is so much we could use help with if talented people like you 
> chose to contribute.
> 
> >b) means that at some time in the future we need to hoist the reiser4
> >extensions (at a conceptual level) (and probably with restrictions) up into
> >the VFS.  This will involve much thought, argument and work.
> >
> >
> >To get us started on this route it would really help me (and, probably,
> >others) if you could describe what these API extensions are in a very
> >simple way, without referring to incomprehehsible web pages,
> >
> what is not comprehensible....?
> 
> > and without
> >using terms which non-reiser4 people don't understand.
> >  
> >
> Well, I agree that there is value in defining things in more detail than 
> we have.
> 
> >It would be best if each extension was addressed in a separate email
> >thread.
> >
> >We also need to discuss what a reiser4 "module" is, what its capabilities
> >are, and what licensing implications they have.
> >
> >Then, we can look at each one and say "yup, that makes sense - we want
> >Linux to do that" and we can also think about how we would implement it at
> >the VFS level.
> >
> >If we follow the above route I believe we can make progress in a technical
> >direction and not get deadlocked on personal/political stuff.
> >
> >
> >Now, an alternative to all the above is to just merge reiser4 as-is, after
> >addressing all the lower-level coding issues.  And see what happens.  That
> >may be a thing which Linus wishes to do.  I'm easy.
> >
> >
> >  
> >
> 
> 
-- 
Redeeman <redeeman@metanurb.dk>

Re: silent semantic changes with reiser4

[Will Dyson]

Hans Reiser wrote:

> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> for Apple, and that means we need to put search engine and database 
> functionality into the filesystem.  It takes 11 years of serious 

Hans,

I'm very curious about your ideas on how to put search engine and 
database functionality into the filesystem. Although I've let the Befs 
filesystem driver largely stagnate over the past two years, from time to 
time I think about the problem of exporting to userspace the attribute 
indexes that BeFS keeps on disk.

In the original BeOS, they solved the problem by having the filesystem 
driver itself take a text query string and parse it, returning a list of 
inodes that match. The whole business of parsing a query string in the 
kernel (let alone in the filesystem driver) has always seemed ugly to 
me. However, the best alternative I've come up with is to simply export 
the index data as a special file (perhaps in sysfs?) and have userspace 
responsible for searching the index. That would probably work, but it 
wouldn't help other filesystems that implement even a different index 
format, much less a different form of extra searchability.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

Am Donnerstag, den 26.08.2004, 12:32 -0400 schrieb Will Dyson:

> > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> > for Apple, and that means we need to put search engine and database 
> > functionality into the filesystem.  It takes 11 years of serious 
> 
> I'm very curious about your ideas on how to put search engine and 
> database functionality into the filesystem.

I could imagine something like this.

There are some special folders or files or whatever under every file
where some userspace can put some metadata. A keyword perhaps.

echo linux > file.bla/keywords/topic

The filesystem might then automatically put these keywords into an index
and then provide a search mechanism elsewhere where it could ask "find
me all dentries with the keyword 'linux'." and it would return a list
like locate does. Only that it's in realtime and also works when moving
the file around (but not when copying with an unaware program for
obvious reasons).


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

Am Donnerstag, den 26.08.2004, 12:32 -0400 schrieb Will Dyson:

> > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> > for Apple, and that means we need to put search engine and database 
> > functionality into the filesystem.  It takes 11 years of serious 
> 
> I'm very curious about your ideas on how to put search engine and 
> database functionality into the filesystem.

I could imagine something like this.

There are some special folders or files or whatever under every file
where some userspace can put some metadata. A keyword perhaps.

echo linux > file.bla/keywords/topic

The filesystem might then automatically put these keywords into an index
and then provide a search mechanism elsewhere where it could ask "find
me all dentries with the keyword 'linux'." and it would return a list
like locate does. Only that it's in realtime and also works when moving
the file around (but not when copying with an unaware program for
obvious reasons).


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> echo linux > file.bla/keywords/topic
> 
> The filesystem might then automatically put these keywords into an index
> and then provide a search mechanism elsewhere where it could ask "find
> me all dentries with the keyword 'linux'." and it would return a list
> like locate does. Only that it's in realtime and also works when moving
> the file around (but not when copying with an unaware program for
> obvious reasons).

It could work even when copying with an unaware program - provided
the're a tool which knows to extract metadata from files where that
hasn't been done already, or where the file's been modified.

-- Jamie

Re: silent semantic changes with reiser4

[Hans Reiser]

Will Dyson wrote:

>
>
> In the original BeOS, they solved the problem by having the filesystem 
> driver itself take a text query string and parse it, returning a list 
> of inodes that match. The whole business of parsing a query string in 
> the kernel (let alone in the filesystem driver) has always seemed ugly 
> to me. 

Why?

> However, the best alternative I've come up with is to simply export 
> the index data as a special file (perhaps in sysfs?) and have 
> userspace responsible for searching the index.

That is the best implementation suggestion I've heard for splitting the 
filesystem into two parts, one in user space and one in kernel, but I 
still don't trust it to work well.

> That would probably work, but it wouldn't help other filesystems that 
> implement even a different index format, much less a different form of 
> extra searchability.
>
After reiser6 is implemented, we'll have a better idea of what parts 
need what from what other parts.  Until we get there, it needs to be 
kept together.  Frankly, I suspect that naming is not going to split so 
easily, but when it is done we'll be able to know.  The problems of 
putting too much functionality into the kernel are less than those of 
partitioning the filesystem code, and defining an inflexible API between 
the two parts.  Kernel interfaces are fairly inflexible because there 
will always be some moron who sidesteps the fs library to program 
directly to the kernel interface, gets screwed when the interface 
changes, has the nerve to complain about it, and works for some powerful 
distro.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

Will Dyson wrote:

>
>
> In the original BeOS, they solved the problem by having the filesystem 
> driver itself take a text query string and parse it, returning a list 
> of inodes that match. The whole business of parsing a query string in 
> the kernel (let alone in the filesystem driver) has always seemed ugly 
> to me. 

Why?

> However, the best alternative I've come up with is to simply export 
> the index data as a special file (perhaps in sysfs?) and have 
> userspace responsible for searching the index.

That is the best implementation suggestion I've heard for splitting the 
filesystem into two parts, one in user space and one in kernel, but I 
still don't trust it to work well.

> That would probably work, but it wouldn't help other filesystems that 
> implement even a different index format, much less a different form of 
> extra searchability.
>
After reiser6 is implemented, we'll have a better idea of what parts 
need what from what other parts.  Until we get there, it needs to be 
kept together.  Frankly, I suspect that naming is not going to split so 
easily, but when it is done we'll be able to know.  The problems of 
putting too much functionality into the kernel are less than those of 
partitioning the filesystem code, and defining an inflexible API between 
the two parts.  Kernel interfaces are fairly inflexible because there 
will always be some moron who sidesteps the fs library to program 
directly to the kernel interface, gets screwed when the interface 
changes, has the nerve to complain about it, and works for some powerful 
distro.

Hans

Re: silent semantic changes with reiser4

[Will Dyson]

Hans Reiser wrote:
> Will Dyson wrote:
>>
>> In the original BeOS, they solved the problem by having the filesystem 
>> driver itself take a text query string and parse it, returning a list 
>> of inodes that match. The whole business of parsing a query string in 
>> the kernel (let alone in the filesystem driver) has always seemed ugly 
>> to me. 
> 
> Why?

Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.

String parsing bloats the kernel with code that will be called rarely, 
and doing it inside the filesystem module makes for duplicate code if 
more than one filesystem does it. But a good common parser routine (or a 
kernel api that takes a pre-compiled parse tree) would reduce the bad taste.

The real objection I have is that it puts the kernel in the position of 
doing more work than it has to for the system to operate correctly and 
efficiently. The user wants to know what files match a certain set of 
criteria. The filesystem provides special features which can greatly 
accelerate some searches. Does it make more sense to move the 
functionality of /usr/bin/find into the kernel, or to export the index 
information so that an enhanced version of find can make use of it?

I don't think having find in the kernel would be worth the cost or 
complexity.

However, now that I've had to think about it while writing the above, it 
occurs to me that if we wanted to offer something like the beos's live 
queries, doing it in the kernel might be the way to go.

For the 99.9% of people who were not beos users back in the day, the 
"live query" was a way to register a query string with the kernel, so 
that you got notifications when the list of files matching the query 
changed. It was pretty cool, but this is really just blue-skying since I 
am not stepping up to write such a beast any time soon.

>> However, the best alternative I've come up with is to simply export 
>> the index data as a special file (perhaps in sysfs?) and have 
>> userspace responsible for searching the index.
> 
> That is the best implementation suggestion I've heard for splitting the 
> filesystem into two parts, one in user space and one in kernel, but I 
> still don't trust it to work well.

Why not? And if it really splits the filesystem into two parts depends 
on which of the two following statements you agree with more:

"Fast searching is a feature of the filesystem"
"Maintaining indexes is a feature of the filesystem"


PS.
The file-as-directory attribute/stream/whatever model is a much better 
fit for befs's internals than the xattr api. If the kernel supported it, 
I would enable support for it in befs.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Re: silent semantic changes with reiser4

[David Ford]

[-- Attachment #1: Type: text/plain, Size: 2800 bytes --]

Will Dyson wrote:

> Hans Reiser wrote:
>
>> Will Dyson wrote:
>>
>>>
>>> In the original BeOS, they solved the problem by having the 
>>> filesystem driver itself take a text query string and parse it, 
>>> returning a list of inodes that match. The whole business of parsing 
>>> a query string in the kernel (let alone in the filesystem driver) 
>>> has always seemed ugly to me. 
>>
>>
>> Why?
>
>
> Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.
>
> String parsing bloats the kernel with code that will be called rarely, 
> and doing it inside the filesystem module makes for duplicate code if 
> more than one filesystem does it. But a good common parser routine (or 
> a kernel api that takes a pre-compiled parse tree) would reduce the 
> bad taste.


Like say.. printk() ?  :)

> The real objection I have is that it puts the kernel in the position 
> of doing more work than it has to for the system to operate correctly 
> and efficiently. The user wants to know what files match a certain set 
> of criteria. The filesystem provides special features which can 
> greatly accelerate some searches. Does it make more sense to move the 
> functionality of /usr/bin/find into the kernel, or to export the index 
> information so that an enhanced version of find can make use of it?


I think it depends largely on where you see file systems going 5, 10, 20 
years from now.  Are they going to be decadent designs or will they be 
flush with meta data and other tidbits?  Where does the heuristic line 
sit when it becomes better to put such a function into the kernel v.s. 
user land?

> I don't think having find in the kernel would be worth the cost or 
> complexity.


Probably not.  But as I view it, file systems are slowly evolving into 
relational objects and the margin between what are files and directories 
versus content within said objects is shrinking.  More and more, people 
are using a relational database to store content that once resided in 
singular files.

Sure, it makes for a much fatter *FS system, but the user doesn't care 
too much, he simply enjoys the ability to manage, search, and cross 
reference his content much more easily.

And then comes the "it's a user land thing" comment again.  And again 
the response.  When does moving this from user land into kernel land 
become a better choice.

With the file systems we have today, it's really not much of a gain 
compared to the cost of the move.

Consider Reiser4 now.  More and more meta data is being used in regards 
to files.  That margin is shrinking.

BeOS broached it, Microsoft is building it.  Just because they did it 
doesn't mean we have to do it nor that it is right.

But it's food for thought.  Especially as more and more tidbits get 
associated with files.

-david


[-- Attachment #2: david+challenge-response.vcf --]
[-- Type: text/x-vcard, Size: 183 bytes --]

begin:vcard
fn:David Ford
n:Ford;David
email;internet:david@blue-labs.org
title:Industrial Geek
tel;home:Ask please
tel;cell:(203) 650-3611
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: silent semantic changes with reiser4

[David Ford]

[-- Attachment #1: Type: text/plain, Size: 2800 bytes --]

Will Dyson wrote:

> Hans Reiser wrote:
>
>> Will Dyson wrote:
>>
>>>
>>> In the original BeOS, they solved the problem by having the 
>>> filesystem driver itself take a text query string and parse it, 
>>> returning a list of inodes that match. The whole business of parsing 
>>> a query string in the kernel (let alone in the filesystem driver) 
>>> has always seemed ugly to me. 
>>
>>
>> Why?
>
>
> Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.
>
> String parsing bloats the kernel with code that will be called rarely, 
> and doing it inside the filesystem module makes for duplicate code if 
> more than one filesystem does it. But a good common parser routine (or 
> a kernel api that takes a pre-compiled parse tree) would reduce the 
> bad taste.


Like say.. printk() ?  :)

> The real objection I have is that it puts the kernel in the position 
> of doing more work than it has to for the system to operate correctly 
> and efficiently. The user wants to know what files match a certain set 
> of criteria. The filesystem provides special features which can 
> greatly accelerate some searches. Does it make more sense to move the 
> functionality of /usr/bin/find into the kernel, or to export the index 
> information so that an enhanced version of find can make use of it?


I think it depends largely on where you see file systems going 5, 10, 20 
years from now.  Are they going to be decadent designs or will they be 
flush with meta data and other tidbits?  Where does the heuristic line 
sit when it becomes better to put such a function into the kernel v.s. 
user land?

> I don't think having find in the kernel would be worth the cost or 
> complexity.


Probably not.  But as I view it, file systems are slowly evolving into 
relational objects and the margin between what are files and directories 
versus content within said objects is shrinking.  More and more, people 
are using a relational database to store content that once resided in 
singular files.

Sure, it makes for a much fatter *FS system, but the user doesn't care 
too much, he simply enjoys the ability to manage, search, and cross 
reference his content much more easily.

And then comes the "it's a user land thing" comment again.  And again 
the response.  When does moving this from user land into kernel land 
become a better choice.

With the file systems we have today, it's really not much of a gain 
compared to the cost of the move.

Consider Reiser4 now.  More and more meta data is being used in regards 
to files.  That margin is shrinking.

BeOS broached it, Microsoft is building it.  Just because they did it 
doesn't mean we have to do it nor that it is right.

But it's food for thought.  Especially as more and more tidbits get 
associated with files.

-david

Re: silent semantic changes with reiser4

[Hans Reiser]

David Ford wrote:

> Will Dyson wrote:
>
>> Hans Reiser wrote:
>>
>>> Will Dyson wrote:
>>>
>>
>>
>> String parsing bloats the kernel with code that will be called 
>> rarely, and doing it inside the filesystem module makes for duplicate 
>> code if more than one filesystem does it. But a good common parser 
>> routine (or a kernel api that takes a pre-compiled parse tree) would 
>> reduce the bad taste.
>
>
>
> Like say.. printk() ?  :)
>
or namei()....;-)

Seriously, what we are doing is making namei() more powerful.  That is 
all.  Currently namei() is very simple.  Evolution happens.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

I think there are two ways to analyze the code boundary issue.  One is 
"does it belong in the kernel?"  Another is, "does it belong in the 
filesystem. and if so should name resolution in a filesystem be split 
into two parts, one in kernel, and one in user space."  In ten years I 
might have the knowledge needed to make such a split, but I know for 
sure that I don't know how to do it today without regretting it 
tomorrow, and I don't really have confidence that I will ever be able to 
do it without losing performance.

Glad that BeFS finds the new model better.:)

Hans

Will Dyson wrote:

> Hans Reiser wrote:
>
>> Will Dyson wrote:
>>
>>>
>>> In the original BeOS, they solved the problem by having the 
>>> filesystem driver itself take a text query string and parse it, 
>>> returning a list of inodes that match. The whole business of parsing 
>>> a query string in the kernel (let alone in the filesystem driver) 
>>> has always seemed ugly to me. 
>>
>>
>> Why?
>
>
> Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.
>
> String parsing bloats the kernel with code that will be called rarely, 
> and doing it inside the filesystem module makes for duplicate code if 
> more than one filesystem does it. But a good common parser routine (or 
> a kernel api that takes a pre-compiled parse tree) would reduce the 
> bad taste.
>
> The real objection I have is that it puts the kernel in the position 
> of doing more work than it has to for the system to operate correctly 
> and efficiently. The user wants to know what files match a certain set 
> of criteria. The filesystem provides special features which can 
> greatly accelerate some searches. Does it make more sense to move the 
> functionality of /usr/bin/find into the kernel, or to export the index 
> information so that an enhanced version of find can make use of it?
>
> I don't think having find in the kernel would be worth the cost or 
> complexity.
>
> However, now that I've had to think about it while writing the above, 
> it occurs to me that if we wanted to offer something like the beos's 
> live queries, doing it in the kernel might be the way to go.
>
> For the 99.9% of people who were not beos users back in the day, the 
> "live query" was a way to register a query string with the kernel, so 
> that you got notifications when the list of files matching the query 
> changed. It was pretty cool, but this is really just blue-skying since 
> I am not stepping up to write such a beast any time soon.
>
>>> However, the best alternative I've come up with is to simply export 
>>> the index data as a special file (perhaps in sysfs?) and have 
>>> userspace responsible for searching the index.
>>
>>
>> That is the best implementation suggestion I've heard for splitting 
>> the filesystem into two parts, one in user space and one in kernel, 
>> but I still don't trust it to work well.
>
>
> Why not? And if it really splits the filesystem into two parts depends 
> on which of the two following statements you agree with more:
>
> "Fast searching is a feature of the filesystem"
> "Maintaining indexes is a feature of the filesystem"
>
>
> PS.
> The file-as-directory attribute/stream/whatever model is a much better 
> fit for befs's internals than the xattr api. If the kernel supported 
> it, I would enable support for it in befs.
>

Re: silent semantic changes with reiser4

[Hans Reiser]

I think there are two ways to analyze the code boundary issue.  One is 
"does it belong in the kernel?"  Another is, "does it belong in the 
filesystem. and if so should name resolution in a filesystem be split 
into two parts, one in kernel, and one in user space."  In ten years I 
might have the knowledge needed to make such a split, but I know for 
sure that I don't know how to do it today without regretting it 
tomorrow, and I don't really have confidence that I will ever be able to 
do it without losing performance.

Glad that BeFS finds the new model better.:)

Hans

Will Dyson wrote:

> Hans Reiser wrote:
>
>> Will Dyson wrote:
>>
>>>
>>> In the original BeOS, they solved the problem by having the 
>>> filesystem driver itself take a text query string and parse it, 
>>> returning a list of inodes that match. The whole business of parsing 
>>> a query string in the kernel (let alone in the filesystem driver) 
>>> has always seemed ugly to me. 
>>
>>
>> Why?
>
>
> Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.
>
> String parsing bloats the kernel with code that will be called rarely, 
> and doing it inside the filesystem module makes for duplicate code if 
> more than one filesystem does it. But a good common parser routine (or 
> a kernel api that takes a pre-compiled parse tree) would reduce the 
> bad taste.
>
> The real objection I have is that it puts the kernel in the position 
> of doing more work than it has to for the system to operate correctly 
> and efficiently. The user wants to know what files match a certain set 
> of criteria. The filesystem provides special features which can 
> greatly accelerate some searches. Does it make more sense to move the 
> functionality of /usr/bin/find into the kernel, or to export the index 
> information so that an enhanced version of find can make use of it?
>
> I don't think having find in the kernel would be worth the cost or 
> complexity.
>
> However, now that I've had to think about it while writing the above, 
> it occurs to me that if we wanted to offer something like the beos's 
> live queries, doing it in the kernel might be the way to go.
>
> For the 99.9% of people who were not beos users back in the day, the 
> "live query" was a way to register a query string with the kernel, so 
> that you got notifications when the list of files matching the query 
> changed. It was pretty cool, but this is really just blue-skying since 
> I am not stepping up to write such a beast any time soon.
>
>>> However, the best alternative I've come up with is to simply export 
>>> the index data as a special file (perhaps in sysfs?) and have 
>>> userspace responsible for searching the index.
>>
>>
>> That is the best implementation suggestion I've heard for splitting 
>> the filesystem into two parts, one in user space and one in kernel, 
>> but I still don't trust it to work well.
>
>
> Why not? And if it really splits the filesystem into two parts depends 
> on which of the two following statements you agree with more:
>
> "Fast searching is a feature of the filesystem"
> "Maintaining indexes is a feature of the filesystem"
>
>
> PS.
> The file-as-directory attribute/stream/whatever model is a much better 
> fit for befs's internals than the xattr api. If the kernel supported 
> it, I would enable support for it in befs.
>

Re: silent semantic changes with reiser4

[Will Dyson]

Hans Reiser wrote:
> I think there are two ways to analyze the code boundary issue.  One is 
> "does it belong in the kernel?"  Another is, "does it belong in the 
> filesystem. and if so should name resolution in a filesystem be split 
> into two parts, one in kernel, and one in user space."  In ten years I 
> might have the knowledge needed to make such a split, but I know for 
> sure that I don't know how to do it today without regretting it 
> tomorrow, and I don't really have confidence that I will ever be able to 
> do it without losing performance.

I don't see how exporting a set of indices on file attributes to 
userspace constitutes putting part of the name resolution into 
userspace. A file's name (or names, in the case of hardlinks) would 
still be determined entirely within the filesystem.

The more I think about it, the more I am convinced that "the index" is 
the correct primative to use for exposing any filesystem's fast 
searching features.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Separating Indexing and Searching (was silent semantic changes with reiser4)

[Alexander G. M. Smith]

Will Dyson wrote on Fri, 27 Aug 2004 14:20:29 -0400:
> Hans Reiser wrote:
> > Will Dyson wrote:
> >> In the original BeOS, they solved the problem by having the filesystem 
> >> driver itself take a text query string and parse it, returning a list 
> >> of inodes that match. The whole business of parsing a query string in 
> >> the kernel (let alone in the filesystem driver) has always seemed ugly 
> >> to me. 
> > 
> > Why?
> 
> Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.
> 
> String parsing bloats the kernel with code that will be called rarely, 
> and doing it inside the filesystem module makes for duplicate code if 
> more than one filesystem does it. But a good common parser routine (or a 
> kernel api that takes a pre-compiled parse tree) would reduce the bad taste.

Not only bloat, but time wasted reinventing the wheel.  For my experimental file system (http://members.rogers.com/agmsbeos/AGMSRAMFileSystem.readme - see the Nifty Features chapter), I had to figure out the BFS query language (it was documented but didn't have every little detail explained - some experimentation was needed), then write a tokenizer and parser for it, to convert it down to a parse tree that could be used during the query's life.  Good exercise for the programmer, but duplication of effort.

I also added a few new things to the query language, but because the parsing is done by each file system separately, my "improvements" won't show up in other file systems.

> For the 99.9% of people who were not beos users back in the day, the 
> "live query" was a way to register a query string with the kernel, so 
> that you got notifications when the list of files matching the query 
> changed. It was pretty cool, but this is really just blue-skying since I 
> am not stepping up to write such a beast any time soon.

Kind of like that.  For performance, the query string is converted into a parse tree and then all file operations that touch any attribute mentioned in the query get evaluated against the parse tree to see if the result (in or out of the query result set) has changed.  If it changes, then a kernel facility is used to broadcast a message describing the change and file involved to all parties monitoring that query.  Of course, before becoming live, the existing files that match the query are found by iterating over an index (choosing the appropriate index is an optimization trick).

> >> However, the best alternative I've come up with is to simply export 
> >> the index data as a special file (perhaps in sysfs?) and have 
> >> userspace responsible for searching the index.
> > 
> > That is the best implementation suggestion I've heard for splitting the 
> > filesystem into two parts, one in user space and one in kernel, but I 
> > still don't trust it to work well.
> 
> Why not? And if it really splits the filesystem into two parts depends 
> on which of the two following statements you agree with more:
> 
> "Fast searching is a feature of the filesystem"
> "Maintaining indexes is a feature of the filesystem"

I like that.  Put the indexing and a related attribute change update notification mechanism into the kernel.  User land libraries can build on that to implement whatever query language you want, thus saving work and having a common language for all file system varieties that support indices.

However, one of my (unfinished) experiments was to have magic directories that show query results as their contents.  One attribute of the directory (or even its name) would be the query string.  That way even old software (like "ls") could use queries!  Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level.

- Alex

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Hans Reiser]

Alexander G. M. Smith wrote:

>
>
>However, one of my (unfinished) experiments was to have magic directories that show query results as their contents.  One attribute of the directory (or even its name) would be the query string.  That way even old software (like "ls") could use queries!  Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level.
>  
>
Beautiful example of the sort of thing that I don't want to be locked 
out from doing by having the parser in user space.

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Hans Reiser]

Alexander G. M. Smith wrote:

>
>
>However, one of my (unfinished) experiments was to have magic directories that show query results as their contents.  One attribute of the directory (or even its name) would be the query string.  That way even old software (like "ls") could use queries!  Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level.
>
>- Alex
>
>
>  
>
Symlinks also.  Symlinks with powerful queries in them would require a 
parser in the kernel.  Thanks for helping me to distill my incoherent 
reasons for the parser being in the kernel.

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Jamie Lokier]

Hans Reiser wrote:
> Symlinks also.  Symlinks with powerful queries in them would require a 
> parser in the kernel.

No they wouldn't.  You can do symlinks with powerful queries in
userspace _today_ (as well as directories which list queries).

Making queries be up to date in real-time is still a problem, but that
has nothing to do with symlinks or directories, or where the parsers live.

> Thanks for helping me to distill my incoherent reasons for the
> parser being in the kernel.

This is one occasion where you're mistaken :)

There may be other reasons for parsing query strings in the kernel
(though I'm not convinced there are any), but this isn't one.

-- Jamie

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Will Dyson]

Alexander G. M. Smith wrote:

> Not only bloat, but time wasted reinventing the wheel.  For my 
> experimental file system 
> (http://members.rogers.com/agmsbeos/AGMSRAMFileSystem.readme - see 
> the Nifty Features chapter), I had to figure out the BFS query

I had a look. Cool stuff!

> I like that.  Put the indexing and a related attribute change update 
> notification mechanism into the kernel.  User land libraries can 
> build on that to implement whatever query language you want, thus 
> saving work and having a common language for all file system 
> varieties that support indices.

Yeah, thats basicly what I was thinking. Right now I'm trying to think
of any kind of filesystem that could support the general concept of 
"Fast Searching", but be somehow unable to export this capability as a 
set of indices on file attributes....

Ok. I give up. It seems like the right abstraction to me.

> However, one of my (unfinished) experiments was to have magic 
> directories that show query results as their contents.  One attribute
>  of the directory (or even its name) would be the query string.  That
>  way even old software (like "ls") could use queries!  Implementing 
> queries-as-directories might require moving some things back into the
>  kernel, or at least into some plug-in level.

That is a pretty awesome idea. It should be done in its own specialized
filesystem though. QueryFS. If the change-notification mechanism in the
kernel was robust enough (a big if), the rest could be done with a
userspace filesystem (via FUSE). Mount a query, cd into it, run grep,
write a little bash script... Very unixy.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Alexander G. M. Smith]

Will Dyson wrote on Mon, 30 Aug 2004 21:57:17 -0400:
> I had a look. Cool stuff!

Thanks!

> That is a pretty awesome idea. It should be done in its own specialized
> filesystem though. QueryFS. If the change-notification mechanism in the
> kernel was robust enough (a big if), the rest could be done with a
> userspace filesystem (via FUSE). Mount a query, cd into it, run grep,
> write a little bash script... Very unixy.

That brings up a good point, should the queries be allowed to run across
file systems or be restricted to just one?  BeOS was restricted to one
file system (the query evaluation code was file system specific) and
faked it for multiple ones at the user level by repeating the query on
each FS.  A separate Query FS wouldn't have that limit (or could fake
it at the kernel level), if the kernel/FSs exported their indices and
change notifications.

- Alex

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Will Dyson]

Alexander G. M. Smith wrote:

> That brings up a good point, should the queries be allowed to run across
> file systems or be restricted to just one?  BeOS was restricted to one
> file system (the query evaluation code was file system specific) and
> faked it for multiple ones at the user level by repeating the query on
> each FS.  A separate Query FS wouldn't have that limit (or could fake
> it at the kernel level), if the kernel/FSs exported their indices and
> change notifications.

I don't see any fundamental reason why filesystem boundaries couldn't be 
crossed. A separate Query FS could even fake it by scrubbing the disk 
with /usr/bin/find for filesystems that didn't export indices. Probably 
a bad idea, of course. But painfully slow execution is not a reason it 
_couldn't_ be done. Some people even like pain...

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Re: silent semantic changes with reiser4

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> Andrew Morton wrote:

[...]

> > The fact that one filesystem will offer features which other
> > filesystems do not and cannot offer makes me queasy for some reason.

Not me. As long as it is clearly experimental stuff that can be removed at
the head hacker's whim, that is. 2.7 material.

> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> for Apple,

Says who?

>            and that means we need to put search engine and database 
> functionality into the filesystem.

Please don't. Unix works and is extremely popular because it _doesn't_ try
to be smart (policy vs mechanism distinction, simple abstractions that can
be combined endlessly). If you add this to the filesystems, you either redo
_all_ userland to understand _one_ particular way of doing "smart things"
(which will turn out to be almost exactly the dumbest possible for some
uses, and then you are stuck), or get lots of shards from broken apps (and
users, and sysadmins).

>                                     It takes 11 years of serious 
> research to build a clean storage layer able to handle doing that.  

Great! Build an experimental OS showing how to use it, and through that see
if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

> Reiser4 has done that, finally.  None of the other Linux filesystems 
> have.

How come nobody before you in the almost 30 years of Unix has ever seen the
need for this indispensable feature?

>        The next major release of ReiserFS is going to be bursting with 
> semantic enhancements, because the prerequisites for them are in place 
> now.  None of the other Linux filesystems have those prerequisites.  

To me that would mean that ReiserFS has no place in Linux.

> They won't be able to keep up with the semantic enhancements.  This 
> metafiles and file-directories stuff is actually fairly trivial stuff.

Maybe. But the question of it being of any use aren't trivially answered
"yes". My gut feeling is that the answer is a resounding "no", but that's
just me. Direcories are directories, if you want to ship directory-like
stuff around, use directories (or tarfiles, or whatever). Just look what
happened to the much, much easier stuff of splitting SUID/SGID into
capabilities: Nothing at all whatsoever, because they have no decent place
to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
able to _define_ APIs to follow, but even so it isn't at all easy to do so.

> Look guys, in 1993 I anticipated the battle would be here, and I build 
> the foundation for a defensive tower right at the spot MS and Apple are 
> now maneuvering towards.

Why place a protective tower around the pit into which they are trying
desperately to throw themselves? ;-)

>                           Help me get the next level on the tower before 
> they get here.  It is one hell of a foundation, they won't be able to 
> shake it, their trees are not as powerful.  Don't move reiser4 into vfs, 
> use reiser4 as the vfs.  Don't write filesystems, write file plugins and 
> disk format plugins and all the other kinds of plugins, and you won't be 
> missing any expressive power that you really want....

Better write libraries handling whatever weird format you have in mind for
each use. Even applications. I don't expect to be able to use emacs to edit
a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
a poem. The current trend (which I heartily agree with) is to take stuff
_out_ of the kernel (for flexibility, access, development ease, even
performance).
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 02:12:32PM -0400, Horst von Brand wrote:

>> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
>> for Apple,
>Says who?

People will say it when people stop using Linux on servers because
they can integrate metadata easier in other operating systems ;)

>Please don't. Unix works and is extremely popular because it _doesn't_ try
>to be smart (policy vs mechanism distinction, simple abstractions that can
>be combined endlessly). If you add this to the filesystems, you either redo

Just more moving little pieces to be combined endlessly.

I think it looks like Reiser4 semantics should go into VFS and not break
legacy posix compatibility.

If you don't want to use a filesystem like that, don't.
Maybe the reason Linux hasn't developed this yet is the fact that Linux
supports n+1 file systems and above and beyond that many userspace ones.
So there has been enough work in maintaining and competing the old
ones and not coming up with new ones.

>_all_ userland to understand _one_ particular way of doing "smart things"
>(which will turn out to be almost exactly the dumbest possible for some
>uses, and then you are stuck), or get lots of shards from broken apps (and
>users, and sysadmins).

This is a question if implementation.

What if the files-as-dirs have a ..metas/mainstream[1] file and are handled
by legacy applications as directories? Nothing breaks, but may look a bit
ugly. Nothing one line of shell scripting would not cure.

OTOH, the user space progs are quickly patched to support all this.

>Great! Build an experimental OS showing how to use it, and through that see
>if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

Why should he?

Parts of Linux are already here and susceptible to experiments in
evolution. (Break out 2.7 for this one, the people want code! :)

>How come nobody before you in the almost 30 years of Unix has ever seen the
>need for this indispensable feature?

Lack of imagination?
Lack of need?
Lack of drive space?

And maybe someone did think among these lines back in the days, it's
not like the current dogma of file systems is the only one that has
ever been used.

>To me that would mean that ReiserFS has no place in Linux.

This is the type of argumentation I can not understand.
Do you not have the freedom not to use it?

And this is not personal, but please say Reiser4, as ReiserFS is coupled
with versions 3.x. They are quite different.

>Maybe. But the question of it being of any use aren't trivially answered
>"yes". My gut feeling is that the answer is a resounding "no", but that's

I think it's a pretty big yes, even if the technical details do need to
be clear before anything is set in stone. That's the reason for the
on-going discussion.

>just me. Direcories are directories, if you want to ship directory-like
>stuff around, use directories (or tarfiles, or whatever). Just look what

Without a common user-space API to deal in them. Oh, joy.
Make the common user-space API? Will never happen.

>Better write libraries handling whatever weird format you have in mind for
>each use. Even applications. I don't expect to be able to use emacs to edit
>a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>a poem. The current trend (which I heartily agree with) is to take stuff
>_out_ of the kernel (for flexibility, access, development ease, even
>performance).

I don't think anyone's talking about using emacs to edit a picture, but
emacs could be used to edit the picture's description.[2]
Maybe with some userspace-hook that exports the exif data via ..metas/
so the kernel won't have to parse it.

Linus wrote about why TCP is not in userspace and I think this is a similar
situation. Technically. But not everything has to be in kernel space
in Reiser4. Or that's what they say and I'm not sure where I stand on
an exif parser in the kernel...

On the social side, I may be wrong in this, but not many userspace 
filesystems have the possibility of becoming a de-facto standard. One 
might say that's because they suck, or they suck because they're 
implemented in userspace or they are just met with prejudice
for the same reason.

In the kernel something like this would have to be done right and people
would put effort into doing it right.

I for one would welcome that.

[1] Sorry, guys, just had to use that name..
[2] You can edit a picture in emacs already, it's just prone to break ;)

-- 
mjt

Re: silent semantic changes with reiser4

[mjt]

On Thu, Aug 26, 2004 at 02:12:32PM -0400, Horst von Brand wrote:

>> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
>> for Apple,
>Says who?

People will say it when people stop using Linux on servers because
they can integrate metadata easier in other operating systems ;)

>Please don't. Unix works and is extremely popular because it _doesn't_ try
>to be smart (policy vs mechanism distinction, simple abstractions that can
>be combined endlessly). If you add this to the filesystems, you either redo

Just more moving little pieces to be combined endlessly.

I think it looks like Reiser4 semantics should go into VFS and not break
legacy posix compatibility.

If you don't want to use a filesystem like that, don't.
Maybe the reason Linux hasn't developed this yet is the fact that Linux
supports n+1 file systems and above and beyond that many userspace ones.
So there has been enough work in maintaining and competing the old
ones and not coming up with new ones.

>_all_ userland to understand _one_ particular way of doing "smart things"
>(which will turn out to be almost exactly the dumbest possible for some
>uses, and then you are stuck), or get lots of shards from broken apps (and
>users, and sysadmins).

This is a question if implementation.

What if the files-as-dirs have a ..metas/mainstream[1] file and are handled
by legacy applications as directories? Nothing breaks, but may look a bit
ugly. Nothing one line of shell scripting would not cure.

OTOH, the user space progs are quickly patched to support all this.

>Great! Build an experimental OS showing how to use it, and through that see
>if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

Why should he?

Parts of Linux are already here and susceptible to experiments in
evolution. (Break out 2.7 for this one, the people want code! :)

>How come nobody before you in the almost 30 years of Unix has ever seen the
>need for this indispensable feature?

Lack of imagination?
Lack of need?
Lack of drive space?

And maybe someone did think among these lines back in the days, it's
not like the current dogma of file systems is the only one that has
ever been used.

>To me that would mean that ReiserFS has no place in Linux.

This is the type of argumentation I can not understand.
Do you not have the freedom not to use it?

And this is not personal, but please say Reiser4, as ReiserFS is coupled
with versions 3.x. They are quite different.

>Maybe. But the question of it being of any use aren't trivially answered
>"yes". My gut feeling is that the answer is a resounding "no", but that's

I think it's a pretty big yes, even if the technical details do need to
be clear before anything is set in stone. That's the reason for the
on-going discussion.

>just me. Direcories are directories, if you want to ship directory-like
>stuff around, use directories (or tarfiles, or whatever). Just look what

Without a common user-space API to deal in them. Oh, joy.
Make the common user-space API? Will never happen.

>Better write libraries handling whatever weird format you have in mind for
>each use. Even applications. I don't expect to be able to use emacs to edit
>a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>a poem. The current trend (which I heartily agree with) is to take stuff
>_out_ of the kernel (for flexibility, access, development ease, even
>performance).

I don't think anyone's talking about using emacs to edit a picture, but
emacs could be used to edit the picture's description.[2]
Maybe with some userspace-hook that exports the exif data via ..metas/
so the kernel won't have to parse it.

Linus wrote about why TCP is not in userspace and I think this is a similar
situation. Technically. But not everything has to be in kernel space
in Reiser4. Or that's what they say and I'm not sure where I stand on
an exif parser in the kernel...

On the social side, I may be wrong in this, but not many userspace 
filesystems have the possibility of becoming a de-facto standard. One 
might say that's because they suck, or they suck because they're 
implemented in userspace or they are just met with prejudice
for the same reason.

In the kernel something like this would have to be done right and people
would put effort into doing it right.

I for one would welcome that.

[1] Sorry, guys, just had to use that name..
[2] You can edit a picture in emacs already, it's just prone to break ;)

-- 
mjt

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 02:12:32PM -0400, Horst von Brand wrote:

>> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
>> for Apple,
>Says who?

People will say it when people stop using Linux on servers because
they can integrate metadata easier in other operating systems ;)

>Please don't. Unix works and is extremely popular because it _doesn't_ try
>to be smart (policy vs mechanism distinction, simple abstractions that can
>be combined endlessly). If you add this to the filesystems, you either redo

Just more moving little pieces to be combined endlessly.

I think it looks like Reiser4 semantics should go into VFS and not break
legacy posix compatibility.

If you don't want to use a filesystem like that, don't.
Maybe the reason Linux hasn't developed this yet is the fact that Linux
supports n+1 file systems and above and beyond that many userspace ones.
So there has been enough work in maintaining and competing the old
ones and not coming up with new ones.

>_all_ userland to understand _one_ particular way of doing "smart things"
>(which will turn out to be almost exactly the dumbest possible for some
>uses, and then you are stuck), or get lots of shards from broken apps (and
>users, and sysadmins).

This is a question if implementation.

What if the files-as-dirs have a ..metas/mainstream[1] file and are handled
by legacy applications as directories? Nothing breaks, but may look a bit
ugly. Nothing one line of shell scripting would not cure.

OTOH, the user space progs are quickly patched to support all this.

>Great! Build an experimental OS showing how to use it, and through that see
>if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

Why should he?

Parts of Linux are already here and susceptible to experiments in
evolution. (Break out 2.7 for this one, the people want code! :)

>How come nobody before you in the almost 30 years of Unix has ever seen the
>need for this indispensable feature?

Lack of imagination?
Lack of need?
Lack of drive space?

And maybe someone did think among these lines back in the days, it's
not like the current dogma of file systems is the only one that has
ever been used.

>To me that would mean that ReiserFS has no place in Linux.

This is the type of argumentation I can not understand.
Do you not have the freedom not to use it?

And this is not personal, but please say Reiser4, as ReiserFS is coupled
with versions 3.x. They are quite different.

>Maybe. But the question of it being of any use aren't trivially answered
>"yes". My gut feeling is that the answer is a resounding "no", but that's

I think it's a pretty big yes, even if the technical details do need to
be clear before anything is set in stone. That's the reason for the
on-going discussion.

>just me. Direcories are directories, if you want to ship directory-like
>stuff around, use directories (or tarfiles, or whatever). Just look what

Without a common user-space API to deal in them. Oh, joy.
Make the common user-space API? Will never happen.

>Better write libraries handling whatever weird format you have in mind for
>each use. Even applications. I don't expect to be able to use emacs to edit
>a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>a poem. The current trend (which I heartily agree with) is to take stuff
>_out_ of the kernel (for flexibility, access, development ease, even
>performance).

I don't think anyone's talking about using emacs to edit a picture, but
emacs could be used to edit the picture's description.[2]
Maybe with some userspace-hook that exports the exif data via ..metas/
so the kernel won't have to parse it.

Linus wrote about why TCP is not in userspace and I think this is a similar
situation. Technically. But not everything has to be in kernel space
in Reiser4. Or that's what they say and I'm not sure where I stand on
an exif parser in the kernel...

On the social side, I may be wrong in this, but not many userspace 
filesystems have the possibility of becoming a de-facto standard. One 
might say that's because they suck, or they suck because they're 
implemented in userspace or they are just met with prejudice
for the same reason.

In the kernel something like this would have to be done right and people
would put effort into doing it right.

I for one would welcome that.

[1] Sorry, guys, just had to use that name..
[2] You can edit a picture in emacs already, it's just prone to break ;)

-- 
mjt

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Fri, 27 Aug 2004, Markus Törnqvist wrote:
> 
> People will say it when people stop using Linux on servers because
> they can integrate metadata easier in other operating systems ;)

Heh. Considering that WinFS seems to be delayed yet more, I don't think 
that's a very strong argument.

Hell will freeze over before Microsoft does a filesystem right. Besides,
WinFS is likely almost in user mode anyway, ie mostly a library, rather
like the gnome people are already doing with nome storage.

So there's really no point in trying to push your agenda by trying to 
scare people with MS activities. Linux kernel developers do what's right 
because it is _right_, not because somebody else does it.

		Linus

Re[2]: silent semantic changes with reiser4

[Spam]

> On Fri, 27 Aug 2004, Markus Törnqvist wrote:
>> 
>> People will say it when people stop using Linux on servers because
>> they can integrate metadata easier in other operating systems ;)

> Heh. Considering that WinFS seems to be delayed yet more, I don't think
> that's a very strong argument.

> Hell will freeze over before Microsoft does a filesystem right. Besides,
> WinFS is likely almost in user mode anyway, ie mostly a library, rather
> like the gnome people are already doing with nome storage.

  Yes, it is mostly in user land. But, It is still integrated with the
  Windows  API.  This  means that normal programs that use the Windows
  API  will  be  able to use most of the new features (like displaying
  thumbnails, filtered views, etc when you do file operations)

  Besides,  NTFS  is  not  that  bad. It does have both encryption and
  compression  and a big set of ACL features that aren't in most Linux
  filesystems today.

> So there's really no point in trying to push your agenda by trying to
> scare people with MS activities. Linux kernel developers do what's right
> because it is _right_, not because somebody else does it.

  I  agree.  Things  should be done because they are good, not because
  you are threatened to.

  It  is  a shame really that so many people hate Microsoft (or Apple)
  so much they fail to see things that actually are good.

>         Linus

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote:
>
>Heh. Considering that WinFS seems to be delayed yet more, I don't think 
>that's a very strong argument.

But there's Mac OS and isn't Solaris doing something along these lines?

Microsoft isn't alone there ;)

>scare people with MS activities. Linux kernel developers do what's right 
>because it is _right_, not because somebody else does it.

Never really doubted that, mind you.
I just think this is both right and a competition situation.

-- 
mjt

Re: silent semantic changes with reiser4

[mjt]

On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote:
>
>Heh. Considering that WinFS seems to be delayed yet more, I don't think 
>that's a very strong argument.

But there's Mac OS and isn't Solaris doing something along these lines?

Microsoft isn't alone there ;)

>scare people with MS activities. Linux kernel developers do what's right 
>because it is _right_, not because somebody else does it.

Never really doubted that, mind you.
I just think this is both right and a competition situation.

-- 
mjt

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote:
>
>Heh. Considering that WinFS seems to be delayed yet more, I don't think 
>that's a very strong argument.

But there's Mac OS and isn't Solaris doing something along these lines?

Microsoft isn't alone there ;)

>scare people with MS activities. Linux kernel developers do what's right 
>because it is _right_, not because somebody else does it.

Never really doubted that, mind you.
I just think this is both right and a competition situation.

-- 
mjt

Re: silent semantic changes with reiser4

[Bill Huey]

On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote:
> Heh. Considering that WinFS seems to be delayed yet more, I don't think 
> that's a very strong argument.

Which is another argument, and bragging rights, to integrate something
like Reiser FS4. :)

> Hell will freeze over before Microsoft does a filesystem right. Besides,
> WinFS is likely almost in user mode anyway, ie mostly a library, rather
> like the gnome people are already doing with nome storage.

They're trying to push their low level storage model into the kernel
after they saw how slow their userspace version of that is. It makes
sense to have this kind of stuff directly apart of a lower level FS
later
 
> So there's really no point in trying to push your agenda by trying to 
> scare people with MS activities. Linux kernel developers do what's right 
> because it is _right_, not because somebody else does it.

I tried to stay out of this thread, but I couldn't resist after this
post. :) MS has other problems, namely, they're a large company trying
to basically replicated Apple's (NeXT) OS X stuff, but onl 10x larger
and slower than it is. It's really silly what's going on with MS
development as well as the greater open source X community. The sooner
folks realize that all rendering models are moving in the direction of
Apple's OS X, the sooner they can just flat out rip it off.

It's going to require both kernel and userspace changes for this to
happen.

bill

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Fri, 27 Aug 2004, Markus Törnqvist wrote:
>  
>
>>People will say it when people stop using Linux on servers because
>>they can integrate metadata easier in other operating systems ;)
>>    
>>
>
>Heh. Considering that WinFS seems to be delayed yet more, I don't think 
>that's a very strong argument.
>
>Hell will freeze over before Microsoft does a filesystem right. Besides,
>WinFS is likely almost in user mode anyway, ie mostly a library, rather
>like the gnome people are already doing with nome storage.
>
>So there's really no point in trying to push your agenda by trying to 
>scare people with MS activities. Linux kernel developers do what's right 
>because it is _right_, not because somebody else does it.
>
>		Linus
>
>
>  
>
Apple will get it right.  I promise it.  I have met Dominic, and he is 
very very sharp.  Look at the Tiger demos on their website.  Simple 
interface, looks nice to me.... 

The one area he might screw up is performance, but I don't care to count 
on that.

WinFS first tried to put it all in the FS, and then it became a user 
mode library almost certainly because they are making the standard 
mistakes the database guys make when they try to emulate file systems 
without changing the core balanced tree algorithms, and their 
performance sucked and they had to back off.  It took 11 years for me to 
get it right, and they aren't as crazy-err-persistent as I am.;-)

We might get lucky and have them produce another NTFS, but then again, 
when Microsoft focuses on a task, they do much better at it than they do 
most of the time, and they are focused on WinFS.  They have hired very 
sharp people.  We can hope that they don't know how to use them, but 
when they hire people like Gerard Salton for $1 million a year, there is 
just possibly a chance that they might try to get their money's worth 
out of him.

You should not be complacent about WinFS being delayed to 2007, because 
even if I get funding for enhanced ReiserFS semantics tomorrow we also 
can't get the job done before 2007.  This is big science, not writing a 
device driver.

Finally, how much harm will it be if we do it right and it is important 
and they fail?  Suppose I am wrong about them, and we create a powerful 
unifying namespace for Linux before any other OS does?  Is that so bad?

Creating a powerful namespace at the heart of Linux is the most 
important enhancement you can make to the OS. 

Finally the storage layer is good enough to support putting the 
relationship between keywords (actually keyobjects in my scheme....) and 
their documents directly into the FS without losing performance for 
traditional file system usage patterns, and I get to stop tweaking 
performance and go have fun with semantics in the next major release.

Hans

Re: silent semantic changes with reiser4

[Bill Huey]

On Sat, Aug 28, 2004 at 02:55:00AM -0700, Hans Reiser wrote:
> mistakes the database guys make when they try to emulate file systems 
> without changing the core balanced tree algorithms, and their 
> performance sucked and they had to back off.  It took 11 years for me to 
> get it right, and they aren't as crazy-err-persistent as I am.;-)
> 
> We might get lucky and have them produce another NTFS, but then again, 
> when Microsoft focuses on a task, they do much better at it than they do 
> most of the time, and they are focused on WinFS.  They have hired very 
> sharp people.  We can hope that they don't know how to use them, but 
> when they hire people like Gerard Salton for $1 million a year, there is 
> just possibly a chance that they might try to get their money's worth 
> out of him.

I realize this is a relatively late reply, but...

MS's problem is that they hire very smart folks, but the compartimentalize
them so they don't influence the overall architecture of the entire system.
I guarantee you that even if they did get all of this in place and working,
their arrogance in this matter will screw up how this will be properly used
and never expressed the power of a system like this.

Apple is another story. They're the only other folks that might get this right.

Your system is a dream of mine since I was dorking around with persistent
objects in Smalltalk systems. But getting other folks to understand
something like this and then use it in this way is very difficult and I
don't expect MS to be able to do this nor get .NET to compete with Java
for server side applications. I don't expect traditional Unix folks to
understand why something like this is important also.

It's not about a single technology as much as the system as a whole,
larger picture... That's were a system starts to show it's superiority.

That's my two cents on the matter.

bill

Re: silent semantic changes with reiser4

[Spam]

> Hans Reiser <reiser@namesys.com> said:
>> Andrew Morton wrote:

> [...]

>> > The fact that one filesystem will offer features which other
>> > filesystems do not and cannot offer makes me queasy for some reason.

> Not me. As long as it is clearly experimental stuff that can be removed at
> the head hacker's whim, that is. 2.7 material.

>> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> for Apple,

> Says who?

  At  least  I  as  a user see the benefits largely. The workstation /
  desktop  user has much more need for such features, than perhaps the
  normal Linux sysadmin.

>>            and that means we need to put search engine and database
>> functionality into the filesystem.

> Please don't. Unix works and is extremely popular because it _doesn't_ try
> to be smart (policy vs mechanism distinction, simple abstractions that can
> be combined endlessly). If you add this to the filesystems, you either redo
> _all_ userland to understand _one_ particular way of doing "smart things"
> (which will turn out to be almost exactly the dumbest possible for some
> uses, and then you are stuck), or get lots of shards from broken apps (and
> users, and sysadmins).

  Indeed.  But  you forget one thing. Combining lots of small userland
  tools can actually make things more complex - at least for the user.

  That said, I do not see that anyone has suggested to remove this way
  of  working with applications.

  Are the new and old features mutually exclusive?

>>                                     It takes 11 years of serious 
>> research to build a clean storage layer able to handle doing that.

> Great! Build an experimental OS showing how to use it, and through that see
> if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

  There  are  many  uses already. It could be encryption, compression,
  meta-data,  file streams, sorting, searching and a multitude of uses
  and ideas that haven't been discovered yet.

>> Reiser4 has done that, finally.  None of the other Linux filesystems
>> have.

> How come nobody before you in the almost 30 years of Unix has ever seen the
> need for this indispensable feature?

  I,  as  a  user, have for a long time. Do not drag everyone over the
  same  edge.  Users  have  different  needs.  We  should try to be as
  forthcoming as possible to provide the tools and options for them.

>>        The next major release of ReiserFS is going to be bursting with
>> semantic enhancements, because the prerequisites for them are in place
>> now.  None of the other Linux filesystems have those prerequisites.

> To me that would mean that ReiserFS has no place in Linux.

  Why  do  you say this? I thought Linux embraced advancements and new
  technologies.

>> They won't be able to keep up with the semantic enhancements.  This
>> metafiles and file-directories stuff is actually fairly trivial stuff.

> Maybe. But the question of it being of any use aren't trivially answered
> "yes". My gut feeling is that the answer is a resounding "no", but that's
> just me. Direcories are directories, if you want to ship directory-like
> stuff around, use directories (or tarfiles, or whatever). Just look what
> happened to the much, much easier stuff of splitting SUID/SGID into
> capabilities: Nothing at all whatsoever, because they have no decent place
> to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
> able to _define_ APIs to follow, but even so it isn't at all easy to do so.

  Exactly.  The  problem  I see in this discussion is the lack of just
  this - APIs. Because the lack of good APIs we cannot make any larger
  changes or bring new features because old applications will break.

  I think that Linux should make new API that will completely separate
  applications  from the storage media. This way we will not as easily
  break  applications  when new features are developed. This should be
  the main priority.

  Perhaps  over a transition period the old ways will be allowed while
  applications  are  ported  to  the new APIs. The change is certainly
  going  to be needed at one point or Linux won't be able to evolve as
  quickly as its users are demanding it.

  Application  breakage  has  happened  before between virtually every
  kernel change. It isn't new. Just take devfs and sysfs.


>> Look guys, in 1993 I anticipated the battle would be here, and I build
>> the foundation for a defensive tower right at the spot MS and Apple are
>> now maneuvering towards.

> Why place a protective tower around the pit into which they are trying
> desperately to throw themselves? ;-)

  It  is easy to say this. But the fact is that users can actually use
  these new features for good things.

>>                           Help me get the next level on the tower before
>> they get here.  It is one hell of a foundation, they won't be able to
>> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> disk format plugins and all the other kinds of plugins, and you won't be
>> missing any expressive power that you really want....

> Better write libraries handling whatever weird format you have in mind for
> each use. Even applications. I don't expect to be able to use emacs to edit
> a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
> a poem. The current trend (which I heartily agree with) is to take stuff
> _out_ of the kernel (for flexibility, access, development ease, even
> performance).

  If  I  want  to  be able to quickly find files by a certain author I
  could   search  the  meta-data fields (if they are used) with normal
  tools such as grep.

  Compression or encryption is another example.

  Would  you  rather  patch  every  application  there is for Linux to
  enable support for reading these files and directories?

  

Re: silent semantic changes with reiser4

[Spam]

> Hans Reiser <reiser@namesys.com> said:
>> Andrew Morton wrote:

> [...]

>> > The fact that one filesystem will offer features which other
>> > filesystems do not and cannot offer makes me queasy for some reason.

> Not me. As long as it is clearly experimental stuff that can be removed at
> the head hacker's whim, that is. 2.7 material.

>> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> for Apple,

> Says who?

  At  least  I  as  a user see the benefits largely. The workstation /
  desktop  user has much more need for such features, than perhaps the
  normal Linux sysadmin.

>>            and that means we need to put search engine and database
>> functionality into the filesystem.

> Please don't. Unix works and is extremely popular because it _doesn't_ try
> to be smart (policy vs mechanism distinction, simple abstractions that can
> be combined endlessly). If you add this to the filesystems, you either redo
> _all_ userland to understand _one_ particular way of doing "smart things"
> (which will turn out to be almost exactly the dumbest possible for some
> uses, and then you are stuck), or get lots of shards from broken apps (and
> users, and sysadmins).

  Indeed.  But  you forget one thing. Combining lots of small userland
  tools can actually make things more complex - at least for the user.

  That said, I do not see that anyone has suggested to remove this way
  of  working with applications.

  Are the new and old features mutually exclusive?

>>                                     It takes 11 years of serious 
>> research to build a clean storage layer able to handle doing that.

> Great! Build an experimental OS showing how to use it, and through that see
> if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

  There  are  many  uses already. It could be encryption, compression,
  meta-data,  file streams, sorting, searching and a multitude of uses
  and ideas that haven't been discovered yet.

>> Reiser4 has done that, finally.  None of the other Linux filesystems
>> have.

> How come nobody before you in the almost 30 years of Unix has ever seen the
> need for this indispensable feature?

  I,  as  a  user, have for a long time. Do not drag everyone over the
  same  edge.  Users  have  different  needs.  We  should try to be as
  forthcoming as possible to provide the tools and options for them.

>>        The next major release of ReiserFS is going to be bursting with
>> semantic enhancements, because the prerequisites for them are in place
>> now.  None of the other Linux filesystems have those prerequisites.

> To me that would mean that ReiserFS has no place in Linux.

  Why  do  you say this? I thought Linux embraced advancements and new
  technologies.

>> They won't be able to keep up with the semantic enhancements.  This
>> metafiles and file-directories stuff is actually fairly trivial stuff.

> Maybe. But the question of it being of any use aren't trivially answered
> "yes". My gut feeling is that the answer is a resounding "no", but that's
> just me. Direcories are directories, if you want to ship directory-like
> stuff around, use directories (or tarfiles, or whatever). Just look what
> happened to the much, much easier stuff of splitting SUID/SGID into
> capabilities: Nothing at all whatsoever, because they have no decent place
> to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
> able to _define_ APIs to follow, but even so it isn't at all easy to do so.

  Exactly.  The  problem  I see in this discussion is the lack of just
  this - APIs. Because the lack of good APIs we cannot make any larger
  changes or bring new features because old applications will break.

  I think that Linux should make new API that will completely separate
  applications  from the storage media. This way we will not as easily
  break  applications  when new features are developed. This should be
  the main priority.

  Perhaps  over a transition period the old ways will be allowed while
  applications  are  ported  to  the new APIs. The change is certainly
  going  to be needed at one point or Linux won't be able to evolve as
  quickly as its users are demanding it.

  Application  breakage  has  happened  before between virtually every
  kernel change. It isn't new. Just take devfs and sysfs.


>> Look guys, in 1993 I anticipated the battle would be here, and I build
>> the foundation for a defensive tower right at the spot MS and Apple are
>> now maneuvering towards.

> Why place a protective tower around the pit into which they are trying
> desperately to throw themselves? ;-)

  It  is easy to say this. But the fact is that users can actually use
  these new features for good things.

>>                           Help me get the next level on the tower before
>> they get here.  It is one hell of a foundation, they won't be able to
>> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> disk format plugins and all the other kinds of plugins, and you won't be
>> missing any expressive power that you really want....

> Better write libraries handling whatever weird format you have in mind for
> each use. Even applications. I don't expect to be able to use emacs to edit
> a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
> a poem. The current trend (which I heartily agree with) is to take stuff
> _out_ of the kernel (for flexibility, access, development ease, even
> performance).

  If  I  want  to  be able to quickly find files by a certain author I
  could   search  the  meta-data fields (if they are used) with normal
  tools such as grep.

  Compression or encryption is another example.

  Would  you  rather  patch  every  application  there is for Linux to
  enable support for reading these files and directories?

  

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>> 
>> > Says who?
>> 
>>   At  least  I  as  a user see the benefits largely. The workstation /
>>   desktop  user has much more need for such features, than perhaps the
>>   normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

  I am not saying it couldn't be done in userland. Most things can.

  The  problem with userland is that it doesn't provide generic access
  across  most  applications. For example GnomeVFS is limited to Gnome
  applications  only.  The  same  for  KDE.  to  be able to coordinate
  application developer on anything is just almost impossible.

  Still.  Why do you oppose plugins, streams and meta files? The could
  be   valuable   and  easy  to use tools for many purposes. One could
  be  would be advanced ACLs defined as a meta-file using XML format.
  

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

  I  am not sure what you refer to as "required features". Do you mean
  example of usage and contents for streams, plugins and meta files?

>> >>            and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>>   Indeed.  But  you forget one thing. Combining lots of small userland
>>   tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

  Difference being that it could be transparent in a way otherwise not
  possible.

>>   That said, I do not see that anyone has suggested to remove this way
>>   of  working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>>   Are the new and old features mutually exclusive?

> Yep.

>> >>                                     It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>> 
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>>   There  are  many  uses already.

> Where?

>>                                   It could be encryption, compression,
>>   meta-data,  file streams, sorting, searching and a multitude of uses
>>   and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally.  None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>>   I,  as  a  user, have for a long time. Do not drag everyone over the
>>   same  edge.  Users  have  different  needs.  We  should try to be as
>>   forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >>        The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now.  None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>>   Why  do  you say this? I thought Linux embraced advancements and new
>>   technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

  You  mean  in  Windows  and  MacOS  where much of this functionality
  already exists?

>> >> They won't be able to keep up with the semantic enhancements.  This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>>   Exactly.  The  problem  I see in this discussion is the lack of just
>>   this - APIs. Because the lack of good APIs we cannot make any larger
>>   changes or bring new features because old applications will break.


  I'll chop this up...
  
> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

  No   perhaps  not  by itself. But it certainly should be considered.
  These  two  OS'es  have  most  of  the market share. They have great
  resources  trying to make things that work for their users.

  Users  are  more  important  than you think. Linux and any OS can do
  what  it  wants, but if it wants to keep users then it has to listen
  to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

  This  is  a  perfectly  valid  reason.  If  it  is necessary to move
  something  into  the  core of Linux because it couldn't otherwise be
  done then it should seriously be considered.

>  will just add another incompatible way of "doing almost the same", this
> time in the kernel),

   People  on  this  list  are  actually trying to make it to be fully
   compatible with old legacy applications.

   If  you  hadn't noticed, Reiser4 does work. The meta files do work.
   The  discussion  is  wether parts of Reiser4 should be made generic
   and fit into the VFS.

>  and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

  You  paint up a doomsday picture here. Nothing of this magnitude has
  happened  and  won't. besides, what is wrong if sysadmins would have
  to learn something new?

>>   I think that Linux should make new API that will completely separate
>>   applications  from the storage media. This way we will not as easily
>>   break  applications  when new features are developed. This should be
>>   the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

  you  didn't  actually  comment  on  what  I  said. I mentioned API's
  nothing about directories and files. API being the keyword here.

  If  there  are  good API's that applications can use, for example to
  access filesystems, then it will be easier and much more possible to
  introduce  new  things into the filesystem without breaking existing
  applications.  One  example of preventing application breakage could
  be   to   have  a  versioning system. Applications could query which
  version is supported, or vice versa.

  From  what  I  understand  it  is  much  harder to do anything today
  because of the fixed ways. This is why we have this discussion.

> [...]

>>   Application  breakage  has  happened  before between virtually every
>>   kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

  Emacs  seem  to  work  fine  with  the  file-as-dir  on  my  Reiser4
  partition. And so does actually any other application I have used on
  it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>> 
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>> 
>>   It  is easy to say this. But the fact is that users can actually use
>>   these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

  Abuse depends on permissions and security policies which is your job
  as admin to look after.

  It  would  be  possible to limit, for example), the types of plugins
  that a user can either use or load.

  Do you give your users access to modprobe or tools like that?

>> >>                           Help me get the next level on the tower before
>> >> they get here.  It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>> 
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>>   If  I  want  to  be able to quickly find files by a certain author I
>>   could   search  the  meta-data fields (if they are used) with normal
>>   tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

  No  it  is  not  useless.  It  just  means  the  file  has  no  meta
  information. Just as it is today. No change.

>>   Compression or encryption is another example.
>> 
>>   Would  you  rather  patch  every  application  there is for Linux to
>>   enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

  So you can setup a directory that has some files compressed and some
  other  encrypted and yet they will be available to use with tar, cp,
  ls, mc, Nautilus, Emacs etc etc?

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>> 
>> > Says who?
>> 
>>   At  least  I  as  a user see the benefits largely. The workstation /
>>   desktop  user has much more need for such features, than perhaps the
>>   normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

  I am not saying it couldn't be done in userland. Most things can.

  The  problem with userland is that it doesn't provide generic access
  across  most  applications. For example GnomeVFS is limited to Gnome
  applications  only.  The  same  for  KDE.  to  be able to coordinate
  application developer on anything is just almost impossible.

  Still.  Why do you oppose plugins, streams and meta files? The could
  be   valuable   and  easy  to use tools for many purposes. One could
  be  would be advanced ACLs defined as a meta-file using XML format.
  

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

  I  am not sure what you refer to as "required features". Do you mean
  example of usage and contents for streams, plugins and meta files?

>> >>            and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>>   Indeed.  But  you forget one thing. Combining lots of small userland
>>   tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

  Difference being that it could be transparent in a way otherwise not
  possible.

>>   That said, I do not see that anyone has suggested to remove this way
>>   of  working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>>   Are the new and old features mutually exclusive?

> Yep.

>> >>                                     It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>> 
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>>   There  are  many  uses already.

> Where?

>>                                   It could be encryption, compression,
>>   meta-data,  file streams, sorting, searching and a multitude of uses
>>   and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally.  None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>>   I,  as  a  user, have for a long time. Do not drag everyone over the
>>   same  edge.  Users  have  different  needs.  We  should try to be as
>>   forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >>        The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now.  None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>>   Why  do  you say this? I thought Linux embraced advancements and new
>>   technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

  You  mean  in  Windows  and  MacOS  where much of this functionality
  already exists?

>> >> They won't be able to keep up with the semantic enhancements.  This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>>   Exactly.  The  problem  I see in this discussion is the lack of just
>>   this - APIs. Because the lack of good APIs we cannot make any larger
>>   changes or bring new features because old applications will break.


  I'll chop this up...
  
> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

  No   perhaps  not  by itself. But it certainly should be considered.
  These  two  OS'es  have  most  of  the market share. They have great
  resources  trying to make things that work for their users.

  Users  are  more  important  than you think. Linux and any OS can do
  what  it  wants, but if it wants to keep users then it has to listen
  to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

  This  is  a  perfectly  valid  reason.  If  it  is necessary to move
  something  into  the  core of Linux because it couldn't otherwise be
  done then it should seriously be considered.

>  will just add another incompatible way of "doing almost the same", this
> time in the kernel),

   People  on  this  list  are  actually trying to make it to be fully
   compatible with old legacy applications.

   If  you  hadn't noticed, Reiser4 does work. The meta files do work.
   The  discussion  is  wether parts of Reiser4 should be made generic
   and fit into the VFS.

>  and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

  You  paint up a doomsday picture here. Nothing of this magnitude has
  happened  and  won't. besides, what is wrong if sysadmins would have
  to learn something new?

>>   I think that Linux should make new API that will completely separate
>>   applications  from the storage media. This way we will not as easily
>>   break  applications  when new features are developed. This should be
>>   the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

  you  didn't  actually  comment  on  what  I  said. I mentioned API's
  nothing about directories and files. API being the keyword here.

  If  there  are  good API's that applications can use, for example to
  access filesystems, then it will be easier and much more possible to
  introduce  new  things into the filesystem without breaking existing
  applications.  One  example of preventing application breakage could
  be   to   have  a  versioning system. Applications could query which
  version is supported, or vice versa.

  From  what  I  understand  it  is  much  harder to do anything today
  because of the fixed ways. This is why we have this discussion.

> [...]

>>   Application  breakage  has  happened  before between virtually every
>>   kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

  Emacs  seem  to  work  fine  with  the  file-as-dir  on  my  Reiser4
  partition. And so does actually any other application I have used on
  it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>> 
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>> 
>>   It  is easy to say this. But the fact is that users can actually use
>>   these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

  Abuse depends on permissions and security policies which is your job
  as admin to look after.

  It  would  be  possible to limit, for example), the types of plugins
  that a user can either use or load.

  Do you give your users access to modprobe or tools like that?

>> >>                           Help me get the next level on the tower before
>> >> they get here.  It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>> 
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>>   If  I  want  to  be able to quickly find files by a certain author I
>>   could   search  the  meta-data fields (if they are used) with normal
>>   tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

  No  it  is  not  useless.  It  just  means  the  file  has  no  meta
  information. Just as it is today. No change.

>>   Compression or encryption is another example.
>> 
>>   Would  you  rather  patch  every  application  there is for Linux to
>>   enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

  So you can setup a directory that has some files compressed and some
  other  encrypted and yet they will be available to use with tar, cp,
  ls, mc, Nautilus, Emacs etc etc?

Re: silent semantic changes with reiser4

[Nikita Danilov]

Spam writes:
 > 
 > > Spam <spam@tnonline.net> said:
 > >> Horst von Brand <vonbrand@inf.utfsm.cl> said:

[...]

 > 
 > > Show benefits, show there are no downsides. Show that the same can't
 > > possibly be done in userland. Then I'll consider buying it.
 > 
 >   I am not saying it couldn't be done in userland. Most things can.
 > 
 >   The  problem with userland is that it doesn't provide generic access
 >   across  most  applications. For example GnomeVFS is limited to Gnome
 >   applications  only.  The  same  for  KDE.  to  be able to coordinate
 >   application developer on anything is just almost impossible.

Hmm... drag-and-drop also doesn't work constently over all
applications. Let's put it into kernel.

Seriously, kernel prformance is critical to the system, and to achieve
high performance all kernel code runs in single address space (actually,
in portion of address spaces of user processes shared by all
processes). All shared system state is located there. This means that
bug in a kernel affects whole system. This means that kernel should be
kept as simple as possible (and a bit more simple).

This is why only things that cannot be done efficiently in the user
level are put into kernel. And political agendas of various camps of
user-level developers change nothing here.

 > 
 >   Still.  Why do you oppose plugins, streams and meta files? The could
 >   be   valuable   and  easy  to use tools for many purposes. One could
 >   be  would be advanced ACLs defined as a meta-file using XML format.

POSIX has standard ACL API in C (well, "eternal draft" only), one can
use it to extract ACLs from kernel and convert it to any format to one's
heart content. Advantage of this is that when XML goes out of fashion (I
hope this wouldn't yet happen when you will read this message), kernel
API will remain intact.

 >   

Nikita.

Re: silent semantic changes with reiser4

[Nikita Danilov]

Spam writes:
 > 
 > > Spam <spam@tnonline.net> said:
 > >> Horst von Brand <vonbrand@inf.utfsm.cl> said:

[...]

 > 
 > > Show benefits, show there are no downsides. Show that the same can't
 > > possibly be done in userland. Then I'll consider buying it.
 > 
 >   I am not saying it couldn't be done in userland. Most things can.
 > 
 >   The  problem with userland is that it doesn't provide generic access
 >   across  most  applications. For example GnomeVFS is limited to Gnome
 >   applications  only.  The  same  for  KDE.  to  be able to coordinate
 >   application developer on anything is just almost impossible.

Hmm... drag-and-drop also doesn't work constently over all
applications. Let's put it into kernel.

Seriously, kernel prformance is critical to the system, and to achieve
high performance all kernel code runs in single address space (actually,
in portion of address spaces of user processes shared by all
processes). All shared system state is located there. This means that
bug in a kernel affects whole system. This means that kernel should be
kept as simple as possible (and a bit more simple).

This is why only things that cannot be done efficiently in the user
level are put into kernel. And political agendas of various camps of
user-level developers change nothing here.

 > 
 >   Still.  Why do you oppose plugins, streams and meta files? The could
 >   be   valuable   and  easy  to use tools for many purposes. One could
 >   be  would be advanced ACLs defined as a meta-file using XML format.

POSIX has standard ACL API in C (well, "eternal draft" only), one can
use it to extract ACLs from kernel and convert it to any format to one's
heart content. Advantage of this is that when XML goes out of fashion (I
hope this wouldn't yet happen when you will read this message), kernel
API will remain intact.

 >   

Nikita.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

[ This will probably go down in history as the most pointless message
  ever, I just told Nik on IRC I'd take it up on the list :) ]

On Sun, Aug 29, 2004 at 09:22:08PM +0400, Nikita Danilov wrote:
>Spam writes:

>Hmm... drag-and-drop also doesn't work constently over all
>applications. Let's put it into kernel.

What I'm after is that the file system should be able to store
any arbitrary metadata. No matter what the format is.

Maybe I was a bit harsh on XML in my previous email, but whatever.

>Seriously, kernel prformance is critical to the system, and to achieve
>high performance all kernel code runs in single address space (actually,
>in portion of address spaces of user processes shared by all
>processes). All shared system state is located there. This means that
>bug in a kernel affects whole system. This means that kernel should be
>kept as simple as possible (and a bit more simple).

I do agree here. But is it seriously such a big issue to implement
the files-as-dirs kind of thing?

Be it overloading MAY_READ so it works with chdir or implementing openat
or whatever it takes.

I don't think this should be much more, if any more, than the capability
to store this data. Which Reiser4 does. But moved from the Reiser4
level to the VFS level. Right?

>This is why only things that cannot be done efficiently in the user
>level are put into kernel. And political agendas of various camps of
>user-level developers change nothing here.

What I'd want to do is to be able to write a Reiser4 plugin that suits
my needs for something. Not that I'd even know how to, but that's a
matter of time.

Are there other means of doing this than Reiser? I mean Reiser4 plugin
functionality by other means.

>>   Still.  Why do you oppose plugins, streams and meta files? The could
>>   be   valuable   and  easy  to use tools for many purposes. One could
>>   be  would be advanced ACLs defined as a meta-file using XML format.
>POSIX has standard ACL API in C (well, "eternal draft" only), one can
>use it to extract ACLs from kernel and convert it to any format to one's
>heart content. Advantage of this is that when XML goes out of fashion (I
>hope this wouldn't yet happen when you will read this message), kernel
>API will remain intact.

Aa, this apparently is an argument against XML parsing in the kernel,
not the idea that if someone wanted ACLs, that are in XML format, and
could not use them (because the kernel doesn't parse), he should be
able to write such a plugin.

Besides, let's say the Reiser4 XML-ACL plugin has a parser that actually
DOES something. For argument's sake. Would it not be up to the individual
user to disable this plugin? I'd assume it's possible to write plugins
so that they do not affect anything but themselves, meaning this parser.

Of course the problem of copying a file with XML-ACLs to another Reiser4
fs without support for them should be thought of. But I guess this is
where you take the shortest route; the XML-ACL files are intact in the
file's metadata, but totally unusable?

-- 
mjt

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>> 
>> > Says who?
>> 
>>   At  least  I  as  a user see the benefits largely. The workstation /
>>   desktop  user has much more need for such features, than perhaps the
>>   normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

  I am not saying it couldn't be done in userland. Most things can.

  The  problem with userland is that it doesn't provide generic access
  across  most  applications. For example GnomeVFS is limited to Gnome
  applications  only.  The  same  for  KDE.  to  be able to coordinate
  application developer on anything is just almost impossible.

  Still.  Why do you oppose plugins, streams and meta files? The could
  be   valuable   and  easy  to use tools for many purposes. One could
  be  would be advanced ACLs defined as a meta-file using XML format.
  

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

  I  am not sure what you refer to as "required features". Do you mean
  example of usage and contents for streams, plugins and meta files?

>> >>            and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>>   Indeed.  But  you forget one thing. Combining lots of small userland
>>   tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

  Difference being that it could be transparent in a way otherwise not
  possible.

>>   That said, I do not see that anyone has suggested to remove this way
>>   of  working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>>   Are the new and old features mutually exclusive?

> Yep.

>> >>                                     It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>> 
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>>   There  are  many  uses already.

> Where?

>>                                   It could be encryption, compression,
>>   meta-data,  file streams, sorting, searching and a multitude of uses
>>   and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally.  None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>>   I,  as  a  user, have for a long time. Do not drag everyone over the
>>   same  edge.  Users  have  different  needs.  We  should try to be as
>>   forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >>        The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now.  None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>>   Why  do  you say this? I thought Linux embraced advancements and new
>>   technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

  You  mean  in  Windows  and  MacOS  where much of this functionality
  already exists?

>> >> They won't be able to keep up with the semantic enhancements.  This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>>   Exactly.  The  problem  I see in this discussion is the lack of just
>>   this - APIs. Because the lack of good APIs we cannot make any larger
>>   changes or bring new features because old applications will break.


  I'll chop this up...
  
> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

  No   perhaps  not  by itself. But it certainly should be considered.
  These  two  OS'es  have  most  of  the market share. They have great
  resources  trying to make things that work for their users.

  Users  are  more  important  than you think. Linux and any OS can do
  what  it  wants, but if it wants to keep users then it has to listen
  to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

  This  is  a  perfectly  valid  reason.  If  it  is necessary to move
  something  into  the  core of Linux because it couldn't otherwise be
  done then it should seriously be considered.

>  will just add another incompatible way of "doing almost the same", this
> time in the kernel),

   People  on  this  list  are  actually trying to make it to be fully
   compatible with old legacy applications.

   If  you  hadn't noticed, Reiser4 does work. The meta files do work.
   The  discussion  is  wether parts of Reiser4 should be made generic
   and fit into the VFS.

>  and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

  You  paint up a doomsday picture here. Nothing of this magnitude has
  happened  and  won't. besides, what is wrong if sysadmins would have
  to learn something new?

>>   I think that Linux should make new API that will completely separate
>>   applications  from the storage media. This way we will not as easily
>>   break  applications  when new features are developed. This should be
>>   the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

  you  didn't  actually  comment  on  what  I  said. I mentioned API's
  nothing about directories and files. API being the keyword here.

  If  there  are  good API's that applications can use, for example to
  access filesystems, then it will be easier and much more possible to
  introduce  new  things into the filesystem without breaking existing
  applications.  One  example of preventing application breakage could
  be   to   have  a  versioning system. Applications could query which
  version is supported, or vice versa.

  From  what  I  understand  it  is  much  harder to do anything today
  because of the fixed ways. This is why we have this discussion.

> [...]

>>   Application  breakage  has  happened  before between virtually every
>>   kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

  Emacs  seem  to  work  fine  with  the  file-as-dir  on  my  Reiser4
  partition. And so does actually any other application I have used on
  it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>> 
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>> 
>>   It  is easy to say this. But the fact is that users can actually use
>>   these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

  Abuse depends on permissions and security policies which is your job
  as admin to look after.

  It  would  be  possible to limit, for example), the types of plugins
  that a user can either use or load.

  Do you give your users access to modprobe or tools like that?

>> >>                           Help me get the next level on the tower before
>> >> they get here.  It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>> 
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>>   If  I  want  to  be able to quickly find files by a certain author I
>>   could   search  the  meta-data fields (if they are used) with normal
>>   tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

  No  it  is  not  useless.  It  just  means  the  file  has  no  meta
  information. Just as it is today. No change.

>>   Compression or encryption is another example.
>> 
>>   Would  you  rather  patch  every  application  there is for Linux to
>>   enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

  So you can setup a directory that has some files compressed and some
  other  encrypted and yet they will be available to use with tar, cp,
  ls, mc, Nautilus, Emacs etc etc?

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>> 
>> > Says who?
>> 
>>   At  least  I  as  a user see the benefits largely. The workstation /
>>   desktop  user has much more need for such features, than perhaps the
>>   normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

  I am not saying it couldn't be done in userland. Most things can.

  The  problem with userland is that it doesn't provide generic access
  across  most  applications. For example GnomeVFS is limited to Gnome
  applications  only.  The  same  for  KDE.  to  be able to coordinate
  application developer on anything is just almost impossible.

  Still.  Why do you oppose plugins, streams and meta files? The could
  be   valuable   and  easy  to use tools for many purposes. One could
  be  would be advanced ACLs defined as a meta-file using XML format.
  

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

  I  am not sure what you refer to as "required features". Do you mean
  example of usage and contents for streams, plugins and meta files?

>> >>            and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>>   Indeed.  But  you forget one thing. Combining lots of small userland
>>   tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

  Difference being that it could be transparent in a way otherwise not
  possible.

>>   That said, I do not see that anyone has suggested to remove this way
>>   of  working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>>   Are the new and old features mutually exclusive?

> Yep.

>> >>                                     It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>> 
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>>   There  are  many  uses already.

> Where?

>>                                   It could be encryption, compression,
>>   meta-data,  file streams, sorting, searching and a multitude of uses
>>   and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally.  None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>>   I,  as  a  user, have for a long time. Do not drag everyone over the
>>   same  edge.  Users  have  different  needs.  We  should try to be as
>>   forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >>        The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now.  None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>>   Why  do  you say this? I thought Linux embraced advancements and new
>>   technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

  You  mean  in  Windows  and  MacOS  where much of this functionality
  already exists?

>> >> They won't be able to keep up with the semantic enhancements.  This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>>   Exactly.  The  problem  I see in this discussion is the lack of just
>>   this - APIs. Because the lack of good APIs we cannot make any larger
>>   changes or bring new features because old applications will break.


  I'll chop this up...
  
> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

  No   perhaps  not  by itself. But it certainly should be considered.
  These  two  OS'es  have  most  of  the market share. They have great
  resources  trying to make things that work for their users.

  Users  are  more  important  than you think. Linux and any OS can do
  what  it  wants, but if it wants to keep users then it has to listen
  to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

  This  is  a  perfectly  valid  reason.  If  it  is necessary to move
  something  into  the  core of Linux because it couldn't otherwise be
  done then it should seriously be considered.

>  will just add another incompatible way of "doing almost the same", this
> time in the kernel),

   People  on  this  list  are  actually trying to make it to be fully
   compatible with old legacy applications.

   If  you  hadn't noticed, Reiser4 does work. The meta files do work.
   The  discussion  is  wether parts of Reiser4 should be made generic
   and fit into the VFS.

>  and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

  You  paint up a doomsday picture here. Nothing of this magnitude has
  happened  and  won't. besides, what is wrong if sysadmins would have
  to learn something new?

>>   I think that Linux should make new API that will completely separate
>>   applications  from the storage media. This way we will not as easily
>>   break  applications  when new features are developed. This should be
>>   the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

  you  didn't  actually  comment  on  what  I  said. I mentioned API's
  nothing about directories and files. API being the keyword here.

  If  there  are  good API's that applications can use, for example to
  access filesystems, then it will be easier and much more possible to
  introduce  new  things into the filesystem without breaking existing
  applications.  One  example of preventing application breakage could
  be   to   have  a  versioning system. Applications could query which
  version is supported, or vice versa.

  From  what  I  understand  it  is  much  harder to do anything today
  because of the fixed ways. This is why we have this discussion.

> [...]

>>   Application  breakage  has  happened  before between virtually every
>>   kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

  Emacs  seem  to  work  fine  with  the  file-as-dir  on  my  Reiser4
  partition. And so does actually any other application I have used on
  it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>> 
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>> 
>>   It  is easy to say this. But the fact is that users can actually use
>>   these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

  Abuse depends on permissions and security policies which is your job
  as admin to look after.

  It  would  be  possible to limit, for example), the types of plugins
  that a user can either use or load.

  Do you give your users access to modprobe or tools like that?

>> >>                           Help me get the next level on the tower before
>> >> they get here.  It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>> 
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>>   If  I  want  to  be able to quickly find files by a certain author I
>>   could   search  the  meta-data fields (if they are used) with normal
>>   tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

  No  it  is  not  useless.  It  just  means  the  file  has  no  meta
  information. Just as it is today. No change.

>>   Compression or encryption is another example.
>> 
>>   Would  you  rather  patch  every  application  there is for Linux to
>>   enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

  So you can setup a directory that has some files compressed and some
  other  encrypted and yet they will be available to use with tar, cp,
  ls, mc, Nautilus, Emacs etc etc?

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>> 
>> > Says who?
>> 
>>   At  least  I  as  a user see the benefits largely. The workstation /
>>   desktop  user has much more need for such features, than perhaps the
>>   normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

  I am not saying it couldn't be done in userland. Most things can.

  The  problem with userland is that it doesn't provide generic access
  across  most  applications. For example GnomeVFS is limited to Gnome
  applications  only.  The  same  for  KDE.  to  be able to coordinate
  application developer on anything is just almost impossible.

  Still.  Why do you oppose plugins, streams and meta files? The could
  be   valuable   and  easy  to use tools for many purposes. One could
  be  would be advanced ACLs defined as a meta-file using XML format.
  

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

  I  am not sure what you refer to as "required features". Do you mean
  example of usage and contents for streams, plugins and meta files?

>> >>            and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>>   Indeed.  But  you forget one thing. Combining lots of small userland
>>   tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

  Difference being that it could be transparent in a way otherwise not
  possible.

>>   That said, I do not see that anyone has suggested to remove this way
>>   of  working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>>   Are the new and old features mutually exclusive?

> Yep.

>> >>                                     It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>> 
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>>   There  are  many  uses already.

> Where?

>>                                   It could be encryption, compression,
>>   meta-data,  file streams, sorting, searching and a multitude of uses
>>   and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally.  None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>>   I,  as  a  user, have for a long time. Do not drag everyone over the
>>   same  edge.  Users  have  different  needs.  We  should try to be as
>>   forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >>        The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now.  None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>>   Why  do  you say this? I thought Linux embraced advancements and new
>>   technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

  You  mean  in  Windows  and  MacOS  where much of this functionality
  already exists?

>> >> They won't be able to keep up with the semantic enhancements.  This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>>   Exactly.  The  problem  I see in this discussion is the lack of just
>>   this - APIs. Because the lack of good APIs we cannot make any larger
>>   changes or bring new features because old applications will break.


  I'll chop this up...
  
> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

  No   perhaps  not  by itself. But it certainly should be considered.
  These  two  OS'es  have  most  of  the market share. They have great
  resources  trying to make things that work for their users.

  Users  are  more  important  than you think. Linux and any OS can do
  what  it  wants, but if it wants to keep users then it has to listen
  to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

  This  is  a  perfectly  valid  reason.  If  it  is necessary to move
  something  into  the  core of Linux because it couldn't otherwise be
  done then it should seriously be considered.

>  will just add another incompatible way of "doing almost the same", this
> time in the kernel),

   People  on  this  list  are  actually trying to make it to be fully
   compatible with old legacy applications.

   If  you  hadn't noticed, Reiser4 does work. The meta files do work.
   The  discussion  is  wether parts of Reiser4 should be made generic
   and fit into the VFS.

>  and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

  You  paint up a doomsday picture here. Nothing of this magnitude has
  happened  and  won't. besides, what is wrong if sysadmins would have
  to learn something new?

>>   I think that Linux should make new API that will completely separate
>>   applications  from the storage media. This way we will not as easily
>>   break  applications  when new features are developed. This should be
>>   the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

  you  didn't  actually  comment  on  what  I  said. I mentioned API's
  nothing about directories and files. API being the keyword here.

  If  there  are  good API's that applications can use, for example to
  access filesystems, then it will be easier and much more possible to
  introduce  new  things into the filesystem without breaking existing
  applications.  One  example of preventing application breakage could
  be   to   have  a  versioning system. Applications could query which
  version is supported, or vice versa.

  From  what  I  understand  it  is  much  harder to do anything today
  because of the fixed ways. This is why we have this discussion.

> [...]

>>   Application  breakage  has  happened  before between virtually every
>>   kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

  Emacs  seem  to  work  fine  with  the  file-as-dir  on  my  Reiser4
  partition. And so does actually any other application I have used on
  it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>> 
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>> 
>>   It  is easy to say this. But the fact is that users can actually use
>>   these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

  Abuse depends on permissions and security policies which is your job
  as admin to look after.

  It  would  be  possible to limit, for example), the types of plugins
  that a user can either use or load.

  Do you give your users access to modprobe or tools like that?

>> >>                           Help me get the next level on the tower before
>> >> they get here.  It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>> 
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>>   If  I  want  to  be able to quickly find files by a certain author I
>>   could   search  the  meta-data fields (if they are used) with normal
>>   tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

  No  it  is  not  useless.  It  just  means  the  file  has  no  meta
  information. Just as it is today. No change.

>>   Compression or encryption is another example.
>> 
>>   Would  you  rather  patch  every  application  there is for Linux to
>>   enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

  So you can setup a directory that has some files compressed and some
  other  encrypted and yet they will be available to use with tar, cp,
  ls, mc, Nautilus, Emacs etc etc?

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>> 
>> > Says who?
>> 
>>   At  least  I  as  a user see the benefits largely. The workstation /
>>   desktop  user has much more need for such features, than perhaps the
>>   normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

  I am not saying it couldn't be done in userland. Most things can.

  The  problem with userland is that it doesn't provide generic access
  across  most  applications. For example GnomeVFS is limited to Gnome
  applications  only.  The  same  for  KDE.  to  be able to coordinate
  application developer on anything is just almost impossible.

  Still.  Why do you oppose plugins, streams and meta files? The could
  be   valuable   and  easy  to use tools for many purposes. One could
  be  would be advanced ACLs defined as a meta-file using XML format.
  

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

  I  am not sure what you refer to as "required features". Do you mean
  example of usage and contents for streams, plugins and meta files?

>> >>            and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>>   Indeed.  But  you forget one thing. Combining lots of small userland
>>   tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

  Difference being that it could be transparent in a way otherwise not
  possible.

>>   That said, I do not see that anyone has suggested to remove this way
>>   of  working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>>   Are the new and old features mutually exclusive?

> Yep.

>> >>                                     It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>> 
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>>   There  are  many  uses already.

> Where?

>>                                   It could be encryption, compression,
>>   meta-data,  file streams, sorting, searching and a multitude of uses
>>   and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally.  None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>>   I,  as  a  user, have for a long time. Do not drag everyone over the
>>   same  edge.  Users  have  different  needs.  We  should try to be as
>>   forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >>        The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now.  None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>>   Why  do  you say this? I thought Linux embraced advancements and new
>>   technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

  You  mean  in  Windows  and  MacOS  where much of this functionality
  already exists?

>> >> They won't be able to keep up with the semantic enhancements.  This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>>   Exactly.  The  problem  I see in this discussion is the lack of just
>>   this - APIs. Because the lack of good APIs we cannot make any larger
>>   changes or bring new features because old applications will break.


  I'll chop this up...
  
> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

  No   perhaps  not  by itself. But it certainly should be considered.
  These  two  OS'es  have  most  of  the market share. They have great
  resources  trying to make things that work for their users.

  Users  are  more  important  than you think. Linux and any OS can do
  what  it  wants, but if it wants to keep users then it has to listen
  to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

  This  is  a  perfectly  valid  reason.  If  it  is necessary to move
  something  into  the  core of Linux because it couldn't otherwise be
  done then it should seriously be considered.

>  will just add another incompatible way of "doing almost the same", this
> time in the kernel),

   People  on  this  list  are  actually trying to make it to be fully
   compatible with old legacy applications.

   If  you  hadn't noticed, Reiser4 does work. The meta files do work.
   The  discussion  is  wether parts of Reiser4 should be made generic
   and fit into the VFS.

>  and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

  You  paint up a doomsday picture here. Nothing of this magnitude has
  happened  and  won't. besides, what is wrong if sysadmins would have
  to learn something new?

>>   I think that Linux should make new API that will completely separate
>>   applications  from the storage media. This way we will not as easily
>>   break  applications  when new features are developed. This should be
>>   the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

  you  didn't  actually  comment  on  what  I  said. I mentioned API's
  nothing about directories and files. API being the keyword here.

  If  there  are  good API's that applications can use, for example to
  access filesystems, then it will be easier and much more possible to
  introduce  new  things into the filesystem without breaking existing
  applications.  One  example of preventing application breakage could
  be   to   have  a  versioning system. Applications could query which
  version is supported, or vice versa.

  From  what  I  understand  it  is  much  harder to do anything today
  because of the fixed ways. This is why we have this discussion.

> [...]

>>   Application  breakage  has  happened  before between virtually every
>>   kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

  Emacs  seem  to  work  fine  with  the  file-as-dir  on  my  Reiser4
  partition. And so does actually any other application I have used on
  it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>> 
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>> 
>>   It  is easy to say this. But the fact is that users can actually use
>>   these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

  Abuse depends on permissions and security policies which is your job
  as admin to look after.

  It  would  be  possible to limit, for example), the types of plugins
  that a user can either use or load.

  Do you give your users access to modprobe or tools like that?

>> >>                           Help me get the next level on the tower before
>> >> they get here.  It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>> 
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>>   If  I  want  to  be able to quickly find files by a certain author I
>>   could   search  the  meta-data fields (if they are used) with normal
>>   tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

  No  it  is  not  useless.  It  just  means  the  file  has  no  meta
  information. Just as it is today. No change.

>>   Compression or encryption is another example.
>> 
>>   Would  you  rather  patch  every  application  there is for Linux to
>>   enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

  So you can setup a directory that has some files compressed and some
  other  encrypted and yet they will be available to use with tar, cp,
  ls, mc, Nautilus, Emacs etc etc?

Re: silent semantic changes with reiser4

[Jamie Lokier]

Horst von Brand wrote:
> Build an experimental OS showing how to use it, and through that see
> if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

Hey, if you can think of a way to build an experimental OS and get it
used in real, day-to-day, down-to-earth, practice - i.e. used by a
diverse audience of thousands of vocal groups doing different things
with it, please let me know.

Until you do we have to lobby for some things to be tried in Linux
before we know if they really work.

-- Jamie

Re: silent semantic changes with reiser4

[Martin J. Bligh]

> Don't move reiser4 into vfs, use reiser4 as the vfs. 

Ah ... that seems to be a good insight into what you're driving for.
That would seem to be good for you ... and bad for other filesystems.
So whilst I can see why you'd want it - but I don't think it's a good idea.
Competition is good.

Andrew wrote:

> And what are the licensing implications of plugins?  Are they derived
> works?  Must they be GPL'ed?

You replied:

>>And what are the licensing implications of plugins?  Are they derived
>>works?  
>
> Yes.

I find it odd that you cut out the last part of his question. So must
they be GPL'ed or not?

M.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Martin" == Martin J Bligh <mbligh@aracnet.com> writes:

Martin> Andrew wrote:

> And what are the licensing implications of plugins?  Are they derived
>> works?  Must they be GPL'ed?

Martin> [Hans] replied:

>>> And what are the licensing implications of plugins?  Are they
>>> derived works?
>> 
>> Yes.

Martin> I find it odd that you cut out the last part of his question. So
Martin> must they be GPL'ed or not?

If they are derived works of GPL'ed software, then they obviously must
be GPL'ed (or else they are illegal).  But if the authors obtain an
alternate license from Hans/Namesys, then they are bound by the terms
of that license.  (Of course, this case would most likely apply only to
Reiser4 running under a different operating system.)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: silent semantic changes with reiser4

[Hubert Chan]

>>>>> "Martin" == Martin J Bligh <mbligh@aracnet.com> writes:

Martin> Andrew wrote:

> And what are the licensing implications of plugins?  Are they derived
>> works?  Must they be GPL'ed?

Martin> [Hans] replied:

>>> And what are the licensing implications of plugins?  Are they
>>> derived works?
>> 
>> Yes.

Martin> I find it odd that you cut out the last part of his question. So
Martin> must they be GPL'ed or not?

If they are derived works of GPL'ed software, then they obviously must
be GPL'ed (or else they are illegal).  But if the authors obtain an
alternate license from Hans/Namesys, then they are bound by the terms
of that license.  (Of course, this case would most likely apply only to
Reiser4 running under a different operating system.)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

vfs2 (was Re: silent semantic changes with reiser4)

[Pavel Machek]

Hi!

> >Yes, but I didn't say "flame Christoph and ignore the issues" ;)
> > 
> >
> Oh....;-)
...
> enhancements.  This metafiles and file-directories stuff is actually 
> fairly trivial stuff.

Now I'm scared.

not as powerful.  Don't 
> move reiser4 into vfs, use reiser4 as the vfs.  Don't write 
> filesystems, write file plugins and disk format plugins and all the 
> other kinds of plugins, and you won't be missing any expressive power 
> that you really want....

So you want to replace vfs. Fine, but you should have told us before.

Also vfs replacement probably should not be hidden in fs/reiserfs4...


-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms