file as a directory

file as a directory

[Amit Gud]

Hi people,

 A straight forward question. Wouldn't adding a "file as a directory"
mechanism more logical in VFS itself, rather than having each fs (like
reiser4) to implement it seperately? My vision is to give archive-file
(.tar, .tar.gz, ...) support in the VFS itself, and of course
transparent to any fs and any user-land application. There are many
archive FSs around, but how feasible would it be to implement the
archive file support in the VFS at dentry-level? I'd be happy to share
my proposal.

AG
--
May the source be with you.

Re: file as a directory

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

hoi :)

On Mon, Nov 22, 2004 at 07:24:36PM +0530, Amit Gud wrote:
>  A straight forward question. Wouldn't adding a "file as a directory"
> mechanism more logical in VFS itself, rather than having each fs (like
> reiser4) to implement it seperately?

wouldn't it be better if such things would be implemented in a library?
use gnome-vfs, or try to get a vfs layer into libc.
That way you can even support different and old kernels and all
filesystems.

The kernel already provides all methods that are neccessary to do that.
So there is no need to implement it in the kernel.

-- 
Martin Waitz

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: file as a directory

[Al Viro]

On Mon, Nov 22, 2004 at 07:24:36PM +0530, Amit Gud wrote:
> I'd be happy to share my proposal.

So why don't you post your patches for review?

Re: file as a directory

[Helge Hafting]

Amit Gud wrote:

>Hi people,
>
> A straight forward question. Wouldn't adding a "file as a directory"
>mechanism more logical in VFS itself, rather than having each fs (like
>reiser4) to implement it seperately? My vision is to give archive-file
>  
>
Such support may happen for a few fs'es - people who
want this will then use those fses.  Those who don't
like the ideas will use others.

>(.tar, .tar.gz, ...) support in the VFS itself, and of course
>transparent to any fs and any user-land application. There are many
>archive FSs around, but how feasible would it be to implement the
>archive file support in the VFS at dentry-level? I'd be happy to share
>my proposal.
>
>  
>
You won't get .tar or .tar.gz support in the VFS, for a few simple reasons:
1. .tar and .tar.gz are complicated formats, and are therefore better
   left to userland.  You can get some of the same effect by using a shared
   library that redefines fopen() and fread() though.  It'll work fine for
   the vast majority of apps that happens to use the C library.

   It is hard to make a guaranteed bug-free decompressor that
   is efficient and works with a finite amount of memory.  The kernel
   needs all that - userland doesn't.

2. Both .tar and .gz  file formats may improve with time.  Getting a new
    version of tar og gunzip is easy enough - getting another compression
    algorithm into the kernel won't be that easy.

3. Writing into a tar.gz file is surprisingly difficult from the kernel 
side.
    Userland may create a new temp file when you add to a .tar.gz.
    Userland may assume that other processes aren't reading or writing
    the .tar.gz as it isupdated.  The kernel have no such luxuries.

I recommend looking at archived threads about file as directory,
you'll find many more arguments.  Currently there is one kind
of support for archive files - loop mounts over files containing
filesystem images.  These are not compressed though.

Helge Hafting

Re: file as a directory

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 1120 bytes --]

On Mon, 2004-11-22 at 15:37 +0100, Martin Waitz wrote:
> hoi :)
> 
> On Mon, Nov 22, 2004 at 07:24:36PM +0530, Amit Gud wrote:
> >  A straight forward question. Wouldn't adding a "file as a directory"
> > mechanism more logical in VFS itself, rather than having each fs (like
> > reiser4) to implement it seperately?
> 
> wouldn't it be better if such things would be implemented in a library?
> use gnome-vfs, or try to get a vfs layer into libc.
> That way you can even support different and old kernels and all
> filesystems.
> 
> The kernel already provides all methods that are neccessary to do that.
> So there is no need to implement it in the kernel.

There are already several things in filesystems that don't strictly
belong inside the kernel.  A filesystem could be implemented quite well
as a user-space daemon that sat on top of the block device and
communicated via sockets or shared memory just like an X server.

So, the argument that because it could be implemented in userspace that
it should be implemented in userspace is not automatically true.
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: file as a directory

[Tomas Carnecky]

Helge Hafting wrote:
> I recommend looking at archived threads about file as directory,
> you'll find many more arguments.  Currently there is one kind
> of support for archive files - loop mounts over files containing
> filesystem images.  These are not compressed though.

Isn't reiserfs trying to implement such things? I've read that in some 
next version of reiserfs one will be able to open /etc/passwd/[username] 
  and get the informations about [username], like UID, GID, home 
directory etc.

Still true? And when can we except such a version of reiserfs?

tom

Re: file as a directory

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

hoi :)

On Mon, Nov 22, 2004 at 08:34:02AM -0700, Zan Lynx wrote:
> > The kernel already provides all methods that are neccessary to do that.
> > So there is no need to implement it in the kernel.
> 
> There are already several things in filesystems that don't strictly
> belong inside the kernel.  A filesystem could be implemented quite well
> as a user-space daemon that sat on top of the block device and
> communicated via sockets or shared memory just like an X server.

this is quite different.
As you need to enforce security policies when accessing the block
device, you have to move the filesystem into its own daemon.
You cannot do it in a library.
It is irrelevant for the application weather the fs resides in a
separate daemon or in the kernel itself.

But support of different views on files is something different.
You can do that in a library, you only need an interface that is
capable of storing your data. The kernel already provides that
interface.

-- 
Martin Waitz

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: file as a directory

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Mon, 22 Nov 2004 19:24:36 +0530, Amit Gud said:

>  A straight forward question. Wouldn't adding a "file as a directory"
> mechanism more logical in VFS itself,

There was quite the flame-fest on the lkml a while back regarding
how the semantics of "file as a directory" should operate.  There's
a number of really nasty corner cases that you need to deal with.

Go back and re-read the whole flame-fest, understand all the points
raised, and let us know when you have a workable proposal.

(Hint - "file as directory" broke a number of programs that didn't
expect that a file *could* be a directory, when run on a reiser4
filesystem...)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: file as a directory

[Jan Engelhardt]

>>  A straight forward question. Wouldn't adding a "file as a directory"
>> mechanism more logical in VFS itself, rather than having each fs (like
>> reiser4) to implement it seperately?
>
>wouldn't it be better if such things would be implemented in a library?
>use gnome-vfs, or try to get a vfs layer into libc.
>That way you can even support different and old kernels and all
>filesystems.
>
>The kernel already provides all methods that are neccessary to do that.
>So there is no need to implement it in the kernel.


*cough* FUSE... *cough*




Jan Engelhardt
-- 
Gesellschaft für Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 Göttingen, www.gwdg.de

Re: file as a directory

[Jan Engelhardt]

>Go back and re-read the whole flame-fest, understand all the points
>raised, and let us know when you have a workable proposal.
>
>(Hint - "file as directory" broke a number of programs that didn't
>expect that a file *could* be a directory, when run on a reiser4
>filesystem...)

So let's keep it to reiser4 as to not wildly break programs running on other
filesystems used in "stable kernels". I'm saying that everybody who runs a R4
FS knows that apps might break, and thus is responsible for making them ready
for reiser4. (Or ask the prog's maintainer.)



Jan Engelhardt
-- 
Gesellschaft für Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 Göttingen, www.gwdg.de

Re: file as a directory

[Jan Engelhardt]

>Go back and re-read the whole flame-fest, understand all the points
>raised, and let us know when you have a workable proposal.
>
>(Hint - "file as directory" broke a number of programs that didn't
>expect that a file *could* be a directory, when run on a reiser4
>filesystem...)

So let's keep it to reiser4 as to not wildly break programs running on other
filesystems used in "stable kernels". I'm saying that everybody who runs a R4
FS knows that apps might break, and thus is responsible for making them ready
for reiser4. (Or ask the prog's maintainer.)



Jan Engelhardt
-- 
Gesellschaft für Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 Göttingen, www.gwdg.de

Re: file as a directory

[Hans Reiser]

Tomas Carnecky wrote:

> Helge Hafting wrote:
>
>> I recommend looking at archived threads about file as directory,
>> you'll find many more arguments.  Currently there is one kind
>> of support for archive files - loop mounts over files containing
>> filesystem images.  These are not compressed though.
>
>
> Isn't reiserfs trying to implement such things? I've read that in some 
> next version of reiserfs one will be able to open 
> /etc/passwd/[username]  and get the informations about [username], 
> like UID, GID, home directory etc.


>
> Still true? And when can we except such a version of reiserfs?
>
> tom
>
>
It was more that we said we would like to implement the functionality 
necessary for doing that (e.g. inheritance), not that we would 
specifically do /etc/passwd.  And that functionality will trickle in 
over time.

Hans

Re: file as a directory

[Hans Reiser]

Valdis.Kletnieks@vt.edu wrote:

>
>(Hint - "file as directory" broke a number of programs that didn't
>expect that a file *could* be a directory, when run on a reiser4
>filesystem...)
>  
>
It broke extraordinarily few.

Re: file as a directory

[Jan Engelhardt]

>>(Hint - "file as directory" broke a number of programs that didn't
>>expect that a file *could* be a directory, when run on a reiser4
>>filesystem...)
>
>It broke extraordinarily few.

(The fewer the better.)

That's good news, and frankly, I did not expect anything else. That's because
either programs definitely know that "it" is a file/directory because they just
mkdir'ed or so, or they implement correct error checks, e.g. the user just
created a directory and we check back (i.e. race protection).

What I am worried about is the opendir() libc call, which AFAIK does this:
  fd = open("directory", myflags | O_DIRECTORY)

OTOH, I'm not worried, because it should be the user's duty to check whether
directory really is one or not. Anything else is sloppy programming.
(Exception: taking argv[xx] from the user)


Cheers,
Jan Engelhardt
-- 
Gesellschaft für Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 Göttingen, www.gwdg.de

Re: file as a directory

[Amit Gud]

On Mon, 22 Nov 2004 16:04:28 +0100, Helge Hafting <helge.hafting@hist.no> wrote:

> You won't get .tar or .tar.gz support in the VFS, for a few simple reasons:
> 1. .tar and .tar.gz are complicated formats, and are therefore better
>    left to userland.  

Agreed that .tar.gz is a complicated format, but zlib is already in
the kernel. It _should_ simplify inflate and deflate of files. And as
compared to .gz format, .tar is much simpler, I guess.

> 
>    It is hard to make a guaranteed bug-free decompressor that
>    is efficient and works with a finite amount of memory.  The kernel
>    needs all that - userland doesn't.

I think, finite amount of memory is the concern of worry, not the rest
... if we could rely on zlib.
 
> 2. Both .tar and .gz  file formats may improve with time.  Getting a new
>     version of tar og gunzip is easy enough - getting another compression
>     algorithm into the kernel won't be that easy.

Doesn't zlib in the kernel gets updated as the formats change? If not,
.tar formats would be worth trying first as proof of concept.
 
AG
--
May the source be with you.

Re: file as a directory

[Dirk Steinberg]

On Monday 22 November 2004 19:52, Hans Reiser wrote:
> Valdis.Kletnieks@vt.edu wrote:
> >(Hint - "file as directory" broke a number of programs that didn't
> >expect that a file *could* be a directory, when run on a reiser4
> >filesystem...)
>
> It broke extraordinarily few.

I know from personal experience that it *does* break Acrobat Reader,
which, unfortunately, is binary-only and also a programm that I 
use quite often. For me this means I cannot use reiser4 (as root fs anyway)
without metas disabled.

How about making metas a mount option? Right now disabling metas 
requires patching the source.

/Dirk Steinberg

Re: file as a directory

[Markus Törnqvist]

On Tue, Nov 23, 2004 at 10:11:21AM +0100, Dirk Steinberg wrote:
>
>How about making metas a mount option? Right now disabling metas 
>requires patching the source.

Isn't there -o nopseudo already?

-- 
mjt

Re: file as a directory

[mjt]

On Tue, Nov 23, 2004 at 10:11:21AM +0100, Dirk Steinberg wrote:
>
>How about making metas a mount option? Right now disabling metas 
>requires patching the source.

Isn't there -o nopseudo already?

-- 
mjt

Re: file as a directory

[Amit Gud]

On Mon, 22 Nov 2004 20:05:25 +0100 (MET), Jan Engelhardt
<jengelh@linux01.gwdg.de> wrote:
> >>(Hint - "file as directory" broke a number of programs that didn't
> >>expect that a file *could* be a directory, when run on a reiser4
> >>filesystem...)
> >
> >It broke extraordinarily few.
> 
> (The fewer the better.)
> 
> That's good news, and frankly, I did not expect anything else. That's because
> either programs definitely know that "it" is a file/directory because they just
> mkdir'ed or so, or they implement correct error checks, e.g. the user just
> created a directory and we check back (i.e. race protection).
> 

Correct me if I'm wrong, but the best way I know whether a file should
be treated as directory or as a file (atleast how I've implemented it)
depends upon the context (how the file is accessed) in the user-space
and this context is reflected in the kernel space in the flags of the
struct nameidata. So ...

----
        /* check if the archive is a path component or if last
component with slash */
	flags = (nd->flags & LOOKUP_CONTINUE) || (nd->flags & LOOKUP_DIRECTORY);
        if(flags)
               /* directory */
        else
               /* file */

----

> What I am worried about is the opendir() libc call, which AFAIK does this:
>   fd = open("directory", myflags | O_DIRECTORY)
> 

No more worries! Am I missing something?

AG
--
May the source be with you.

Re: file as a directory

[Jan Engelhardt]

>Correct me if I'm wrong, but the best way I know whether a file should
>be treated as directory or as a file (atleast how I've implemented it)
>depends upon the context (how the file is accessed) in the user-space
>and this context is reflected in the kernel space in the flags of the
>struct nameidata. So ...

And there I see a problem! The open() call (kernel: sys_open) allows to open
both files and directories in the standard operation.
There is the O_DIRECTORY user-space flag, but which only says "it must be a
directory". So there's something missing to say "must be a file".

Hell will freeze over if a reiser4 "object" can be ANY type, blockdev,
chardev, symlink, <think something up>.


Jan Engelhardt
-- 
Gesellschaft für Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 Göttingen, www.gwdg.de

Re: file as a directory

[Amit Gud]

On Tue, 23 Nov 2004 15:00:37 +0100 (MET), Jan Engelhardt
<jengelh@linux01.gwdg.de> wrote:
> >Correct me if I'm wrong, but the best way I know whether a file should
> >be treated as directory or as a file (atleast how I've implemented it)
> >depends upon the context (how the file is accessed) in the user-space
> >and this context is reflected in the kernel space in the flags of the
> >struct nameidata. So ...
> 
> And there I see a problem! The open() call (kernel: sys_open) allows to open
> both files and directories in the standard operation.
> There is the O_DIRECTORY user-space flag, but which only says "it must be a
> directory". So there's something missing to say "must be a file".
> 
> Hell will freeze over if a reiser4 "object" can be ANY type, blockdev,
> chardev, symlink, <think something up>.
> 

Of course, I check before-hand if the file is archive (.tar, for now).
And then if the appropriate flag is set...treat it as a directory, or
else leave it. Again, if tar format looks as expected, support it or
else leave it.


AG
--
May the source be with you.

Re: file as a directory

[Hans Reiser]

Dirk Steinberg wrote:

>How about making metas a mount option? 
>
That was always the intention.  I thought it got implemented, sigh, my 
guys need smaller todo lists....

>Right now disabling metas 
>requires patching the source.
>
>/Dirk Steinberg
>  
>
You mean, enabling it requires changing a #define (if you are using the 
latest).  We changed that after bugs were found in the implementation 
that could cause crashes.

Hans

Re: file as a directory

[Peter Foldiak]

On Mon, 2004-11-22 at 18:48, Hans Reiser wrote:
> Tomas Carnecky wrote:
> > Helge Hafting wrote:
> >> I recommend looking at archived threads about file as directory,
> >> you'll find many more arguments.  Currently there is one kind
> >> of support for archive files - loop mounts over files containing
> >> filesystem images.  These are not compressed though.
> >
> > Isn't reiserfs trying to implement such things? I've read that in some 
> > next version of reiserfs one will be able to open 
> > /etc/passwd/[username]  and get the informations about [username], 
> > like UID, GID, home directory etc.
> > Still true? And when can we except such a version of reiserfs?
> > tom
> >
> It was more that we said we would like to implement the functionality 
> necessary for doing that (e.g. inheritance), not that we would 
> specifically do /etc/passwd.  And that functionality will trickle in 
> over time.
> 
> Hans


I think something like

/etc/passwd/[username]

would be a really nice extension. The idea is more general, it would
unify the namespace for file selection and part-of-file selection.

So if you have a file named "/home/peter/book", you should be able to
look at its Introduction as "/home/peter/book/Introduction" or chapter
3, paragraph 2 as
/home/peter/book/chapter[3]/paragraph[2]

(this may not be the ideal syntax, but something like this should be
good.)

In this case you could use 
/home/peter/book/chapter[3]/paragraph[2]
as a "real" file, read it, even edit it in a text editor. When you later
look at the whole book as /home/peter/book , you should see your
changes.

The idea is related to the W3C "XPath"
http://www.w3.org/TR/xpath
http://www.w3schools.com/xpath/
XPath only applies within XML files, but it is mostly a superset of
current Unix file naming, so it would be natural to unify the two
namespaces. The idea is that you would use the same syntax for selecting
file and part-of-file in a way that the user doesn't even need to know
at which level they "cross over" to selecting within the file.
In the above example, it may be that each chapter is stored in a
separate "real" file. We don't even need to know. In a sense, the whole
concept of a "file" becomes a bit blurred. (What we need to think about
is which fragments to store metadata for, and which fragments should
simply inherit their metadata from the parent.)
Hans' tree-based file system is really ideal for this kind of thing.
I am not sure how closely we would want to follow the XPath syntax (it
has some nice ideas), but it should then be easy to write an plugin
doing XPath really efficiently on top of this.(And it could be made to
work for non-XML files as well.)

I would really like to implement this for the next version of Hans' file
system.
 Peter

Re: file as a directory

[Helge Hafting]

Amit Gud wrote:

>On Mon, 22 Nov 2004 16:04:28 +0100, Helge Hafting <helge.hafting@hist.no> wrote:
>
>  
>
>>You won't get .tar or .tar.gz support in the VFS, for a few simple reasons:
>>1. .tar and .tar.gz are complicated formats, and are therefore better
>>   left to userland.  
>>    
>>
>
>Agreed that .tar.gz is a complicated format, but zlib is already in
>the kernel. It _should_ simplify inflate and deflate of files. And as
>compared to .gz format, .tar is much simpler, I guess.
>
>  
>
>>   It is hard to make a guaranteed bug-free decompressor that
>>   is efficient and works with a finite amount of memory.  The kernel
>>   needs all that - userland doesn't.
>>    
>>
>
>I think, finite amount of memory is the concern of worry, not the rest
>... if we could rely on zlib.
> 
>  
>
>>2. Both .tar and .gz  file formats may improve with time.  Getting a new
>>    version of tar og gunzip is easy enough - getting another compression
>>    algorithm into the kernel won't be that easy.
>>    
>>
>
>Doesn't zlib in the kernel gets updated as the formats change? If not,
>.tar formats would be worth trying first as proof of concept.
>
This is not so easy, as you have to audit the new version for
correctness.  It is not the end of the world if tar or gzip
occationally crashes on some corner case.   The kernel
must not do that though.

And then there is the much more complicated issues when
writing into such an archive.  You skipped that part, or
are you looking for a read-only solution only?

Helge Hafting

Re: file as a directory

[Amit Gud]

On Wed, 24 Nov 2004 11:32:13 +0100, Helge Hafting <helge.hafting@hist.no> wrote:
> Amit Gud wrote:
> 
> 
> 
> >On Mon, 22 Nov 2004 16:04:28 +0100, Helge Hafting <helge.hafting@hist.no> wrote:
> >
> >
> >
> >>You won't get .tar or .tar.gz support in the VFS, for a few simple reasons:
> >>1. .tar and .tar.gz are complicated formats, and are therefore better
> >>   left to userland.
> >>
> >>
> >
> >Agreed that .tar.gz is a complicated format, but zlib is already in
> >the kernel. It _should_ simplify inflate and deflate of files. And as
> >compared to .gz format, .tar is much simpler, I guess.
> >
> >
> >
> >>   It is hard to make a guaranteed bug-free decompressor that
> >>   is efficient and works with a finite amount of memory.  The kernel
> >>   needs all that - userland doesn't.
> >>
> >>
> >
> >I think, finite amount of memory is the concern of worry, not the rest
> >... if we could rely on zlib.
> >
> >
> >
> >>2. Both .tar and .gz  file formats may improve with time.  Getting a new
> >>    version of tar og gunzip is easy enough - getting another compression
> >>    algorithm into the kernel won't be that easy.
> >>
> >>
> >
> >Doesn't zlib in the kernel gets updated as the formats change? If not,
> >.tar formats would be worth trying first as proof of concept.
> >
> This is not so easy, as you have to audit the new version for
> correctness.  It is not the end of the world if tar or gzip
> occationally crashes on some corner case.   The kernel
> must not do that though.
> 

Yes, thats what I said in my last post...if the archive looks improper
forget it.

> And then there is the much more complicated issues when
> writing into such an archive.  You skipped that part, or
> are you looking for a read-only solution only?
> 

I'm coming up with something soon, along with the proof of
concept....to wrap up all scenarios....need some time ;)

AG

Re: file as a directory

[Jan Engelhardt]

>I think something like
>
>/etc/passwd/[username]
>
>would be a really nice extension. The idea is more general, it would
>unify the namespace for file selection and part-of-file selection.

Yeah, and where will you do that? (Possible answers are: kernel space, user
space).

I honestly vote against *these* kinds of plugins (i.e. reading .tar files,
/etc, and such). For one, it is to be done in kernel space, which means the
module code can not be swapped out. Debugging is more complex, segfaults will
kill the machine -- thus it's more open to blackhat hackers.

Also simply because it (the module code) would be a reinvention of wheel, it's
all been written before.




Jan Engelhardt
-- 
Gesellschaft für Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 Göttingen, www.gwdg.de

Re: file as a directory

[Paolo Ciarrocchi]

On 24 Nov 2004 09:16:03 +0000, Peter Foldiak
<peter.foldiak@st-andrews.ac.uk> wrote:
[...] 
> I would really like to implement this for the next version of Hans' file
> system.

I don't undersand how you want to use Xpath for not XML file.
I agree with you that the idea behind Xpath is cool but I fail to
unserstand how it can be applied to anything but XML

-- 
Paolo
Picasa users groups: www.picasa-users.tk
join the blog group: http://groups-beta.google.com/group/blog-users

Re: file as a directory

[Peter Foldiak]

On Wed, 2004-11-24 at 15:02, Paolo Ciarrocchi wrote:
> On 24 Nov 2004 09:16:03 +0000, Peter Foldiak
> <peter.foldiak@st-andrews.ac.uk> wrote:
> [...] 
> > I would really like to implement this for the next version of Hans' file
> > system.
> 
> I don't undersand how you want to use Xpath for not XML file.
> I agree with you that the idea behind Xpath is cool but I fail to
> unserstand how it can be applied to anything but XML

My message was mainly about XML, for which it is easy.
For non-XML, you need some other way of knowing the file format. The
example that originally came up in this thread was

/etc/passwd/[username]

In this case, the passwd file has a known format.
Other file types, like LaTex, html, jpeg also have (at least partially)
known formats. Some selection should be possible even for unknown
formats (e.g. byte range, line-range). There could also be some way of
specifying a new format but I don't know how to do this well. You could
give names (like filenames) to parts of files.
But I think the first step would be to concentrate on XML, and worry
about the rest later.   Peter

Re: file as a directory

[Christian Mayrhuber]

On Wednesday 24 November 2004 16:02, Paolo Ciarrocchi wrote:
> On 24 Nov 2004 09:16:03 +0000, Peter Foldiak
> <peter.foldiak@st-andrews.ac.uk> wrote:
> [...] 
> > I would really like to implement this for the next version of Hans' file
> > system.
> 
> I don't undersand how you want to use Xpath for not XML file.
> I agree with you that the idea behind Xpath is cool but I fail to
> unserstand how it can be applied to anything but XML
> 
> -- 
> Paolo
> Picasa users groups: www.picasa-users.tk
> join the blog group: http://groups-beta.google.com/group/blog-users
> 

Apache Cocoon uses so called generators to parse non-XML formats and produce a 
XML representation thereof. This XML can be addressed by XPath.
To store modifications back this XML needs to be serialized to the original
format. That's **very** fat and slow.

Maybe an automount with a special fuse filesystem could accomplish this.

For example:
# cd /etc/passwd/..metas/contents/

automounts /etc/passwd as "fuse-xpath-passwd" fs 
to /etc/passwd/..metas/contents/

Doing 'cat /etc/passwd/..metas/contents/shell[username = "joe"]' could work 
then.

Reiser4 would need a content-mount plugin that automounts
the respective file by means of a per file configureable
mount command. Something like
# cat /etc/passwd/..metas/plugins/content-mount
-t fuse-xpath-passwd -o ro

fuse-xpath-* filesystems would have to be written. These could be designed 
similiar to the apache cocoon approach of generators/serializers to
work with an intermediate XML representation of the file interior.

All the stuff besides mounting the fuse-xpath-fs's would happen in userspace.
I don't think that anyone can guarantee posix fs semantics by this approach.

-- 
lg, Chris

Re: file as a directory

[Peter Foldiak]

On Wed, 2004-11-24 at 16:11, Christian Mayrhuber wrote:
> On Wednesday 24 November 2004 16:02, Paolo Ciarrocchi wrote:
> > On 24 Nov 2004 09:16:03 +0000, Peter Foldiak
> > <peter.foldiak@st-andrews.ac.uk> wrote:
> > [...] 
> > > I would really like to implement this for the next version of Hans' file
> > > system.
> > 
> > I don't undersand how you want to use Xpath for not XML file.
> > I agree with you that the idea behind Xpath is cool but I fail to
> > unserstand how it can be applied to anything but XML
> > Paolo
> 
> Apache Cocoon uses so called generators to parse non-XML formats and produce a 
> XML representation thereof. This XML can be addressed by XPath.
> To store modifications back this XML needs to be serialized to the original
> format. That's **very** fat and slow.
> 
> Maybe an automount with a special fuse filesystem could accomplish this.
> 
> For example:
> # cd /etc/passwd/..metas/contents/
> 
> automounts /etc/passwd as "fuse-xpath-passwd" fs 
> to /etc/passwd/..metas/contents/
> 
> Doing 'cat /etc/passwd/..metas/contents/shell[username = "joe"]' could work 
> then.
> 
> Reiser4 would need a content-mount plugin that automounts
> the respective file by means of a per file configureable
> mount command. Something like
> # cat /etc/passwd/..metas/plugins/content-mount
> -t fuse-xpath-passwd -o ro
> 
> fuse-xpath-* filesystems would have to be written. These could be designed 
> similiar to the apache cocoon approach of generators/serializers to
> work with an intermediate XML representation of the file interior.
> 
> All the stuff besides mounting the fuse-xpath-fs's would happen in userspace.
> I don't think that anyone can guarantee posix fs semantics by this approach.

The problem with the
cat /etc/passwd/..metas/contents/shell[username = "joe"]
syntax is that it doesn't really achieve namespace unification.
As far as I understand the benefits of a unified namespace are due to
the user and the applications not having to know the details of what
they are dealing with. So, for instance, the nice thing about the
unification of files and devices in Unix is that an application (most
often) can treat a device in the same way as a file (or a pipe, etc.).
This is what gives it real flexibility.
The above syntax assumes you know exactly where the file "ends", and
where the parts of the file "begins" (indicated by the ..metas in your
path). Couldn't we get rid of ..metas from the path?

Also, what I am suggesting is not just to be able to select inside XML
files but also to extend XPath-like selection ABOVE the file level too,
to be used as if the whole file system was like a single big virtual XML
file.
 Peter

Re: file as a directory

[Pavel Machek]

Hi!

> Such support may happen for a few fs'es - people who
> want this will then use those fses.  Those who don't
> like the ideas will use others.
> 
> >(.tar, .tar.gz, ...) support in the VFS itself, and of course
> >transparent to any fs and any user-land application. There are many
> >archive FSs around, but how feasible would it be to implement the
> >archive file support in the VFS at dentry-level? I'd be happy to share
> >my proposal.
> >
> > 
> >
> You won't get .tar or .tar.gz support in the VFS, for a few simple reasons:
> 1. .tar and .tar.gz are complicated formats, and are therefore better
>   left to userland.  You can get some of the same effect by using a shared
>   library that redefines fopen() and fread() though.  It'll work fine for
>   the vast majority of apps that happens to use the C library.

It is not same effect -- with shared library you get no caching. And
that hurts a lot.

>   It is hard to make a guaranteed bug-free decompressor that
>   is efficient and works with a finite amount of memory.  The kernel
>   needs all that - userland doesn't.

If you have bug in decompressor, you are screwed, anyway, because you
get remote user exploit when mozilla gets the file from
web. Oops. [Ok, you at least do not get remote root exploit, but...]

								Pavel 
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: file as a directory

[Hans Reiser]

Peter Foldiak wrote:

>On Wed, 2004-11-24 at 15:02, Paolo Ciarrocchi wrote:
>  
>
>>On 24 Nov 2004 09:16:03 +0000, Peter Foldiak
>><peter.foldiak@st-andrews.ac.uk> wrote:
>>[...] 
>>    
>>
>>>I would really like to implement this for the next version of Hans' file
>>>system.
>>>      
>>>
>>I don't undersand how you want to use Xpath for not XML file.
>>I agree with you that the idea behind Xpath is cool but I fail to
>>unserstand how it can be applied to anything but XML
>>    
>>
>
>My message was mainly about XML, for which it is easy.
>For non-XML, you need some other way of knowing the file format. The
>example that originally came up in this thread was
>
>/etc/passwd/[username]
>
>In this case, the passwd file has a known format.
>Other file types, like LaTex, html, jpeg also have (at least partially)
>known formats. Some selection should be possible even for unknown
>formats (e.g. byte range, line-range). There could also be some way of
>specifying a new format but I don't know how to do this well. You could
>give names (like filenames) to parts of files.
>But I think the first step would be to concentrate on XML, and worry
>about the rest later.   Peter
>
>
>
>  
>
I think Peter is right.  It would be nice to have an interpreter for 
each of the common file formats, and  XML is just the biggest one.

Re: file as a directory

[Hans Reiser]

Peter Foldiak wrote:

>
>
>I would really like to implement this for the next version of Hans' file
>system.
> Peter
>
>
>
>  
>
It would be cool.

Re: file as a directory

[Hans Reiser]

Peter Foldiak wrote:

>
>The problem with the
>cat /etc/passwd/..metas/contents/shell[username = "joe"]
>syntax is that it doesn't really achieve namespace unification.
>
>
>
>
>  
>
For the case Peter cites, yes, it does add clutter to the pathname to 
say "..metas" (actually, it is "...." now in the current reiser4, not 
"..metas").  This is because you aren't looking for metafile 
information, you are looking for a subset and describing the subset, and 
that just requires a file-directory plugin that can handle the name of 
that subset and parse the file to find it.

Re: file as a directory

[Christian Mayrhuber]

On Friday 26 November 2004 19:19, Hans Reiser wrote:
> For the case Peter cites, yes, it does add clutter to the pathname to 
> say "..metas" (actually, it is "...." now in the current reiser4, not 
> "..metas").  This is because you aren't looking for metafile 
> information, you are looking for a subset and describing the subset, and 
> that just requires a file-directory plugin that can handle the name of 
> that subset and parse the file to find it.
> 

Regarding namespace unification + XPath:
For files: cat /etc/passwd/[. = "joe"] should work like in XPath.
But what to do with directories?
Would 'cat /etc/[. = "passwd"]' output the contents of the passwd file
or does it mean to output the file '[. = "passwd"]'?
If the first is the case then you have to prohibit filenames looking 
like '[foo bar]'.

If the shells wouldn't like * for themself, I'd suggest something like
cat /etc/*[. = "passwd"]
This means: list all contents and show the ones where filename = "passwd".

For the contents of /etc/passwd the following could become possible:
'cat /etc/passwd/*[. = "joe"]
'cat /etc/passwd/*[@shell = "/bin/tcsh"]
The XPath could behave similiar as if applied to the following XML:
<entries>
  <root passwd="x" shell="/bin/sh" .... />
  ...
  <joe passwd="x" shell="/bin/tcsh" uid="500" gid="500" .... />
</entries>
The output from the cat's above return the line of joe's entry:
joe:x:500:500:joe:/home/joe:/bin/tcsh

To change all tcsh entries to bash:
echo -n "/bin/bash" > /etc/passwd/*[@shell = "/bin/tcsh"]/@shell

I hope I'm not offending, but my impression is now that
XPath stuff fits better into some shell providing
a XPath view of the filesystem, than into the kernel.

--------------------------------------------------------------------

What about mapping the contents of files into "pure" posix namespace?
XML is basically a tree, too.
Notes: 
1) "...." below is the entry to reiser4 namespace.
2) # denotes a shell command
For example:

# cd /etc/passwd/
# ls -a *
. .. .... joe root
# cd joe
# ls
gid home passwd shell uid
# cat shell
/bin/tcsh
# cd ../....
# ls 
plugins 

I guess an implementation in reiser4 would require some
mime-type/file extension dispatcher plus a special
directory plugin for each mime-type.

-- 
lg, Chris

Re: file as a directory

[Peter Foldiak]

On Fri, 2004-11-26 at 21:13, Christian Mayrhuber wrote:
> Regarding namespace unification + XPath:
> For files: cat /etc/passwd/[. = "joe"] should work like in XPath.

I don't understand this. Why would you need the "."? And why the /
between passwd and [ ?
I would prefer (as was suggested earlier in the thread) that the syntax
for the entry for joe in the passwd file should be

/etc/passwd/joe

and the if you want to select joe's full name, then it should be

/etc/passwd/joe/fullname

and it would result in e.g. "Joe Smith" and

/etc/passwd/joe/shell

whould be the shell joe uses.

Of course something would have to tell the system that the passwd file
has one line for each user, and that fields within a line are separated
by ":"s and what the names of the fields are (e.g. the first field is
called "user"). And when you select inside the /etc/passwd file (or this
type of file) then the default interpretation of joe is that "user"
should be used to select the line.

So by default, /etc/passwd/joe should be equivalent to /etc/passwd[user
= "joe"]

But you should be able to select based on fullname too:

/etc/passwd[fullname = "Joe Smith"]

and

/etc/passwd[shell = "/bin/bash"]/user

should give you the user names of all users whose shell is /bin/bash,
right?

> But what to do with directories?
> Would 'cat /etc/[. = "passwd"]' output the contents of the passwd file
> or does it mean to output the file '[. = "passwd"]'?

I don't really see the point of this . = "passwd" syntax.

> If the first is the case then you have to prohibit filenames looking 
> like '[foo bar]'.

I think so.

> If the shells wouldn't like * for themself, I'd suggest something like
> cat /etc/*[. = "passwd"]
> This means: list all contents and show the ones where filename = "passwd".

but isn't that just /etc/passwd ? why complicate it?

> For the contents of /etc/passwd the following could become possible:
> 'cat /etc/passwd/*[. = "joe"]
> 'cat /etc/passwd/*[@shell = "/bin/tcsh"]

no, I think it should be 

cat /etc/passwd[shell = "bin/tcsh"]

why the *? and why the @? an attribute? why not simply a part of the
line with / ?

> To change all tcsh entries to bash:
> echo -n "/bin/bash" > /etc/passwd/*[@shell = "/bin/tcsh"]/@shell

I would say

echo -n "/bin/bash" > /etc/passwd[shell = "/bin/tcsh"]/shell


> What about mapping the contents of files into "pure" posix namespace?

Yes, that is exactly what I am suggesting, except that I would like to
extend the POSIX syntax by stealing some useful syntactic bits from
XPath.

> XML is basically a tree, too.
> Notes: 
> 1) "...." below is the entry to reiser4 namespace.
> 2) # denotes a shell command
> For example:
> 
> # cd /etc/passwd/
> # ls -a *
> . .. .... joe root
> # cd joe
> # ls
> gid home passwd shell uid

yes, but where is the username? that would be the first one listed here,
right?

> # cat shell
> /bin/tcsh
> # cd ../....
> # ls 
> plugins 
> 
> I guess an implementation in reiser4 would require some
> mime-type/file extension dispatcher plus a special
> directory plugin for each mime-type.

Re: file as a directory

[Tomasz Torcz]

On Wed, Nov 24, 2004 at 09:16:03AM +0000, Peter Foldiak wrote:
> So if you have a file named "/home/peter/book", you should be able to
> look at its Introduction as "/home/peter/book/Introduction" or chapter
> 3, paragraph 2 as
> /home/peter/book/chapter[3]/paragraph[2]
> 
> In this case you could use 
> /home/peter/book/chapter[3]/paragraph[2]
> as a "real" file, read it, even edit it in a text editor. When you later
> look at the whole book as /home/peter/book , you should see your
> changes.

 First, this kind of document storage needs extensive knowledge of
assocatied metadata and therefore is insuitable for kernel. It should be
done in userspace.
 Second, it is beeing done in userspace - http://www.gnome.org/~seth/storage/ .
No need to reinvent the wheel.

-- 
Tomasz Torcz               RIP is irrevelant. Spoofing is futile.
zdzichu@irc.-nie.spam-.pl     Your routes will be aggreggated. -- Alex Yuriev

Re: file as a directory

[Markus Törnqvist]

On Fri, Nov 26, 2004 at 10:19:57AM -0800, Hans Reiser wrote:

>For the case Peter cites, yes, it does add clutter to the pathname to 
>say "..metas" (actually, it is "...." now in the current reiser4, not 
>"..metas").  This is because you aren't looking for metafile 

"...." shound like something that could be an alias for ../..
so not much better than reserving the word "metas" from the namespace.

I guess I'll still go with ..metas here, as it's the best compromise
showed. Or maybe even ..meta (as there is no need for the plural imo)

Just re-opening a damned useless, old, tired and daft can of worms :P

-- 
mjt

Re: file as a directory

[mjt]

On Fri, Nov 26, 2004 at 10:19:57AM -0800, Hans Reiser wrote:

>For the case Peter cites, yes, it does add clutter to the pathname to 
>say "..metas" (actually, it is "...." now in the current reiser4, not 
>"..metas").  This is because you aren't looking for metafile 

"...." shound like something that could be an alias for ../..
so not much better than reserving the word "metas" from the namespace.

I guess I'll still go with ..metas here, as it's the best compromise
showed. Or maybe even ..meta (as there is no need for the plural imo)

Just re-opening a damned useless, old, tired and daft can of worms :P

-- 
mjt

Re: file as a directory

[Christian Mayrhuber]

On Saturday 27 November 2004 12:09, Peter Foldiak wrote:
> On Fri, 2004-11-26 at 21:13, Christian Mayrhuber wrote:
> > Regarding namespace unification + XPath:
> > For files: cat /etc/passwd/[. = "joe"] should work like in XPath.
> 
> I don't understand this. Why would you need the "."? And why the /
> between passwd and [ ?
Yes, I was confused by /etc/passwd/[username] in an earlier email.
I think we both mean basically the same.

> /etc/passwd/joe/shell
> 
> whould be the shell joe uses.
Yes.

> So by default, /etc/passwd/joe should be equivalent to /etc/passwd[user
> = "joe"]
Yes.
/etc/passwd/joe/shell would be equivalent to
/etc/passwd[shell = "/bin/bash"]/joe/shell if joe has bash as shell, right?

> 
> But you should be able to select based on fullname too:
> 
> /etc/passwd[fullname = "Joe Smith"]
Ok.
This means that any XPath like expression will need to return a directory 
entry representing a restricted view on the /etc/passwd contents.
The result of 'ls /etc/passwd[fullname = "Joe Smith"]' would be alike
'drwxr-xr-x  2 root root 48 2004-11-27 13:48 joe', right?

> 
> and
> 
> /etc/passwd[shell = "/bin/bash"]/user
> 
> should give you the user names of all users whose shell is /bin/bash,
> right?
I'm confused again.
I expected 'ls /etc/passwd[shell = "/bin/bash"]/user' to give you the
passwd entries of "user" and 'ls /etc/passwd[shell = "/bin/bash"]/' to
give the users that have a bash shell.


> > # cd /etc/passwd/
> > # ls -a *
> > . .. .... joe root
> > # cd joe
> > # ls
> > gid home passwd shell uid
> 
> yes, but where is the username? that would be the first one listed here,
> right?
joe is the username and a directory. There is no username entry in the
joe directory, because the username is already in the directory.
You can rename a user by renaming joe. 'mv joe newname'.

-- 
lg, Chris

Re: file as a directory

[Helge Hafting]

On Fri, Nov 26, 2004 at 12:09:37AM +0100, Pavel Machek wrote:
> Hi!
> > >
> > You won't get .tar or .tar.gz support in the VFS, for a few simple reasons:
> > 1. .tar and .tar.gz are complicated formats, and are therefore better
> >   left to userland.  You can get some of the same effect by using a shared
> >   library that redefines fopen() and fread() though.  It'll work fine for
> >   the vast majority of apps that happens to use the C library.
> 
> It is not same effect -- with shared library you get no caching. And
> that hurts a lot.
> 
The compressed file is still cached, and the library can cache
file contents in a shared mapping.  It does not have to
be a per-process thing.

> >   It is hard to make a guaranteed bug-free decompressor that
> >   is efficient and works with a finite amount of memory.  The kernel
> >   needs all that - userland doesn't.
> 
> If you have bug in decompressor, you are screwed, anyway, because you
> get remote user exploit when mozilla gets the file from
> web. Oops. [Ok, you at least do not get remote root exploit, but...]
> 
I don't worry about mozilla exploits - you get those from
nasty webpages as well.  I worried about a decompressor
cras (or random memory overwrite).  A userland implementation
will crash that particular userland process, with no ill effects on 
the rest of the system. 

A kernelside crash is much worse - it can hang the kernel and/or
mess up any process.  As for exploits - an in-kernel exploit
is even worse than a root exploit.  There are plenty
of thing even root can't do - at least not in 
straightforward ways.  The kernel has no limitations
whatsoever for what may go wrong.

Helge Hafting

Re: file as a directory

[Pavel Machek]

Hi!

> > > You won't get .tar or .tar.gz support in the VFS, for a few simple reasons:
> > > 1. .tar and .tar.gz are complicated formats, and are therefore better
> > >   left to userland.  You can get some of the same effect by using a shared
> > >   library that redefines fopen() and fread() though.  It'll work fine for
> > >   the vast majority of apps that happens to use the C library.
> > 
> > It is not same effect -- with shared library you get no caching. And
> > that hurts a lot.
> > 
> The compressed file is still cached, and the library can cache
> file contents in a shared mapping.  It does not have to
> be a per-process thing.

Okay, that way you can get it per-user but not system-global...

								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: file as a directory

[Hans Reiser]

Markus Törnqvist wrote:

>On Fri, Nov 26, 2004 at 10:19:57AM -0800, Hans Reiser wrote:
>
>  
>
>>For the case Peter cites, yes, it does add clutter to the pathname to 
>>say "..metas" (actually, it is "...." now in the current reiser4, not 
>>"..metas").  This is because you aren't looking for metafile 
>>    
>>
>
>"...." shound like something that could be an alias for ../..
>so not much better than reserving the word "metas" from the namespace.
>
>I guess I'll still go with ..metas here, as it's the best compromise
>showed. Or maybe even ..meta (as there is no need for the plural imo)
>
>Just re-opening a damned useless, old, tired and daft can of worms :P
>
>  
>
I agree that ..metas is much less likely to cause a namespace collision, 
but I also think that if we called it "john" it would not be a major 
problem, and since the issue is causing us political problems in getting 
into the kernel, "...." is more PR right (as it does not slight Finnish 
women named meta by suggesting they are too obscure to count), and so 
"...." wins.    "...." also has the advantage that it is elegant in 
extending the Unix convention, in that we already have a ".." and a "." 
and hidden files that start with ".".

Re: file as a directory

[Horst von Brand]

Christian Mayrhuber <christian.mayrhuber@gmx.net> said:
> On Saturday 27 November 2004 12:09, Peter Foldiak wrote:
> > On Fri, 2004-11-26 at 21:13, Christian Mayrhuber wrote:
> > > Regarding namespace unification + XPath:
> > > For files: cat /etc/passwd/[. = "joe"] should work like in XPath.
> > 
> > I don't understand this. Why would you need the "."? And why the /
> > between passwd and [ ?
> Yes, I was confused by /etc/passwd/[username] in an earlier email.
> I think we both mean basically the same.

Now think about files with other formats, for instance the (in)famous
sendmail.cf, or less structured stuff like you find in /etc/init.d/, or
just Postgres databases (with fun stuff like permissions on records and
fields)... or just people groping in /etc/passwd wanting to find the whole
entry (not just one field), or perhaps look at the 15th character of the
entry for John Doe.

This way lies utter madness (what format description should be applied to
what file this time around?). Plus shove all this garbage into the kernel?!
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: file as a directory

[Peter Foldiak]

Horst von Brand wrote:

>Now think about files with other formats, for instance the (in)famous
>sendmail.cf, or less structured stuff like you find in /etc/init.d/, or
>just Postgres databases (with fun stuff like permissions on records and
>fields)... or just people groping in /etc/passwd wanting to find the whole
>entry (not just one field), or perhaps look at the 15th character of the
>entry for John Doe.
>
>This way lies utter madness (what format description should be applied to
>what file this time around?). Plus shove all this garbage into the kernel?!
>  
>

I was suggesting this idea mainly form XML files, where the tags define 
the parts clearly.
In addition, I was suggesting that some of the XPath syntax (normally 
used for within-XML selection) could be extended above the file level 
into the file system.
The problems you mention are all related to non-XML file format issues, 
which was only a minor comment in parenthesis in my original mail. I am 
happy to do it only for XML to begin with (and if possible later see if 
it can be done for SOME non-XML formats).  Peter

Re: file as a directory

[Peter Foldiak]

Horst von Brand wrote:

>... or just people groping in /etc/passwd wanting to find the whole
>entry (not just one field),
>
the whole entry for joe would be
/etc/passwd/joe

> or perhaps look at the 15th character of the
>entry for John Doe.
>  
>
something like
/etc/passwd[fullname = "John Doe"]/character[15]
? what is so mad about that?  Peter

Re: file as a directory

[Kevin Fox]

Heh. So, you can have a filename that can contain XPath looking junk.
Now, what happens when you have an XML file that points to another XML
file using XPath? How do you separate the file name XPath from the XML
XPath?

On Mon, 2004-11-29 at 22:59 +0000, Peter Foldiak wrote:
> Horst von Brand wrote:
> 
> >Now think about files with other formats, for instance the (in)famous
> >sendmail.cf, or less structured stuff like you find in /etc/init.d/, or
> >just Postgres databases (with fun stuff like permissions on records and
> >fields)... or just people groping in /etc/passwd wanting to find the whole
> >entry (not just one field), or perhaps look at the 15th character of the
> >entry for John Doe.
> >
> >This way lies utter madness (what format description should be applied to
> >what file this time around?). Plus shove all this garbage into the kernel?!
> >  
> >
> 
> I was suggesting this idea mainly form XML files, where the tags define 
> the parts clearly.
> In addition, I was suggesting that some of the XPath syntax (normally 
> used for within-XML selection) could be extended above the file level 
> into the file system.
> The problems you mention are all related to non-XML file format issues, 
> which was only a minor comment in parenthesis in my original mail. I am 
> happy to do it only for XML to begin with (and if possible later see if 
> it can be done for SOME non-XML formats).  Peter
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: file as a directory

[prymitive]

Dnia Mon, 29 Nov 2004 18:20:35 -0300, Horst von Brand <vonbrand@inf.
utfsm.cl> napisa³:

> Now think about files with other formats, for instance the (in)
famous
> sendmail.cf, or less structured stuff like you find in /etc/init.
d/, or
> just Postgres databases (with fun stuff like permissions on 
records and
> fields)... or just people groping in /etc/passwd wanting to find 
the  
> whole
> entry (not just one field), or perhaps look at the 15th character 
of the
> entry for John Doe.
>
> This way lies utter madness (what format description should be 
applied to
> what file this time around?). Plus shove all this garbage into the 
 
> kernel?!

First time I was reading about /etc/passwd and stuff it was really 
directory as file, imagine You have a directory "/etc/passwd/" and 
inside files like "root", "john" and so, if You acces this directory 
by typing "cat /etc/passwd/" You would get list of files inside, but 
when You type "cat /etc/passwd" You get contest of all files inside 
this directory listed like one big file, the security bonus is that 
all files in /etc/passwd/ can have diffrent permissions, that was my 
firts impression of /etc/passwd hype and I think it would be quite 
good, putting *.tar, *.bz2, *.xml, *.whatever parsers in kernel is 
simply not.

£ukasz Mierzwa

Re: file as a directory

[Peter Foldiak]

On Mon, 2004-11-29 at 23:35, Kevin Fox wrote:
> Heh. So, you can have a filename that can contain XPath looking junk.
> Now, what happens when you have an XML file that points to another XML
> file using XPath? How do you separate the file name XPath from the XML
> XPath?

My suggestion was simply about unifying the namespace for selection in
the file system and selection within XML files using a syntax related to
(but not necessarily identical with) XPath.
 I was not suggesting you should do anything special with the content of
the XML files, even if the XML file contains an XPath reference.
(The latter could be interesting to think about as a separate issue
later, but it is certainly not part of my simpler suggestion.)
 Peter

Re: file as a directory

[Horst von Brand]

Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> said:
> Horst von Brand wrote:
> >Now think about files with other formats, for instance the (in)famous
> >sendmail.cf, or less structured stuff like you find in /etc/init.d/, or
> >just Postgres databases (with fun stuff like permissions on records and
> >fields)... or just people groping in /etc/passwd wanting to find the whole
> >entry (not just one field), or perhaps look at the 15th character of the
> >entry for John Doe.

> >This way lies utter madness (what format description should be applied to
> >what file this time around?). Plus shove all this garbage into the kernel?!

> I was suggesting this idea mainly form XML files, where the tags define 
> the parts clearly.

Use a XML parsing library then.

> In addition, I was suggesting that some of the XPath syntax (normally 
> used for within-XML selection) could be extended above the file level 
> into the file system.

Urgh.

> The problems you mention are all related to non-XML file format issues, 

Most (say 99,95%) files aren't XML; and if they are, the requisite parsing
is probably on hand already, so...

> which was only a minor comment in parenthesis in my original mail. I am 
> happy to do it only for XML to begin with (and if possible later see if 
> it can be done for SOME non-XML formats).

Please don't.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: file as a directory

[Peter Foldiak]

On Tue, 2004-11-30 at 14:51, Horst von Brand wrote:
> > I was suggesting this idea mainly form XML files, where the tags define 
> > the parts clearly.
> 
> Use a XML parsing library then.

But namespace unification is important, and to unify the namespace, you
have to use the same syntax. I guess you disagree with me on that. (If
not, how would you do it?)  Peter

Re: file as a directory

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 693 bytes --]

hoi :)

On Fri, Nov 26, 2004 at 10:13:57PM +0100, Christian Mayrhuber wrote:
> Regarding namespace unification + XPath:
> For files: cat /etc/passwd/[. = "joe"] should work like in XPath.
> But what to do with directories?
> Would 'cat /etc/[. = "passwd"]' output the contents of the passwd file
> or does it mean to output the file '[. = "passwd"]'?
> If the first is the case then you have to prohibit filenames looking 
> like '[foo bar]'.

perhaps we should create a XML/XPath shell and a replacement for the
textutils package instead of implementing all these utilities inside the
kernel.

Then convert /etc/passwd to /etc/passwd.xml and all is well.

-- 
Martin Waitz

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: file as a directory

[Kevin Fox]

On Tue, 2004-11-30 at 08:54 +0000, Peter Foldiak wrote:
> On Mon, 2004-11-29 at 23:35, Kevin Fox wrote:
> > Heh. So, you can have a filename that can contain XPath looking junk.
> > Now, what happens when you have an XML file that points to another XML
> > file using XPath? How do you separate the file name XPath from the XML
> > XPath?
> 
> My suggestion was simply about unifying the namespace for selection in
> the file system and selection within XML files using a syntax related to
> (but not necessarily identical with) XPath.

So long as its different enough that you can have current XPath
implementations be able to separate the XPath from the filename, it
should be fine.

Really simple example,
file#foo

Is #foo handled by XPath, or passed to the file system?

>  I was not suggesting you should do anything special with the content of
> the XML files, even if the XML file contains an XPath reference.
> (The latter could be interesting to think about as a separate issue
> later, but it is certainly not part of my simpler suggestion.)
>  Peter
> 

What I was suggesting is that if your not careful, it may not be
possible for current libraries to know the difference between its XPath
like stuff and whatever is done in the file system.

Kevin

Re: file as a directory

[Horst von Brand]

Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> said:
> On Tue, 2004-11-30 at 14:51, Horst von Brand wrote:
> > > I was suggesting this idea mainly form XML files, where the tags define 
> > > the parts clearly.

> > Use a XML parsing library then.

> But namespace unification is important,

Why? Directories are directories, files are files, file contents is file
contents. Mixing them up is a bad idea. Sure, you could build a filesystem
of sorts (perhaps more in the vein of persistent programming, or even data
base systems) where there simply is no distinction (because there are no
differences to show), but that is something different.

>                                         and to unify the namespace, you
> have to use the same syntax. I guess you disagree with me on that. (If
> not, how would you do it?)

I'd go one level up: Eliminate the distinctions that bother you, not try to
patch over them.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: file as a directory

[Jan Engelhardt]

>> [quote]
>[and another one]

Do you really want to put an XML parser and all the associated code
(read: McDonalds*) into the kernel?



Jan Engelhardt
-- 
* Referring to "Supersize me"

Re: file as a directory

[Hans Reiser]

Horst von Brand wrote:

>Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> said:
>  
>
>>On Tue, 2004-11-30 at 14:51, Horst von Brand wrote:
>>    
>>
>>>>I was suggesting this idea mainly form XML files, where the tags define 
>>>>the parts clearly.
>>>>        
>>>>
>
>  
>
>>>Use a XML parsing library then.
>>>      
>>>
>
>  
>
>>But namespace unification is important,
>>    
>>
>
>Why? Directories are directories, files are files, file contents is file
>contents. Mixing them up is a bad idea. Sure, you could build a filesystem
>of sorts (perhaps more in the vein of persistent programming, or even data
>base systems) where there simply is no distinction (because there are no
>differences to show), but that is something different.
>  
>
This is kind of like explaining to people around the office that they 
could ever possibly need a disk drive of more than 10mb back in 1982 or 
so.  I could not convince them then, Peter, you cannot convince this guy 
now, just spend the time coding it instead.  Peter, you expect people to 
understand the value of features they have never used.  Works for some 
of them.  Only some of them.

>  
>
>>                                        and to unify the namespace, you
>>have to use the same syntax. I guess you disagree with me on that. (If
>>not, how would you do it?)
>>    
>>
>
>I'd go one level up: Eliminate the distinctions that bother you, not try to
>patch over them.
>  
>
Are you saying you'd rewrite xml to put separate objects in separate 
files? 

Re: file as a directory

[Peter Foldiak]

On Tue, 2004-11-30 at 16:31, Horst von Brand wrote:
> > But namespace unification is important,
> 
> Why? Directories are directories, files are files, file contents is file
> contents. Mixing them up is a bad idea.

I disagree, I think it is a good idea.
Why is namespace unification important? Because you can use the same
tools on everything. Previously, each tool could handle one namespace.

A very simple example would be:
I want to count the words in the Appendix of my book.
If I can't select the appendix, my "wc" tool is useless (or very
difficult to use). On the other hand if I can say

wc ~/book/Appendix

it's fine. Hans Reiser would say that "namespaces are the roads and
waterways of the operating system" and "the value of an operating system
is proportional to the number of connections you can make". I think he
is right in that. And the authors of Unix knew it too, when they used
the same namespace for devices and files. They didn't say "files are
files and devices are devices". They said the difference should not
matter to the applications.
But there is still namespace fragmentation even in Unix, and this is
just one of them.

>  Sure, you could build a filesystem
> of sorts (perhaps more in the vein of persistent programming, or even data
> base systems) where there simply is no distinction (because there are no
> differences to show), but that is something different.
> 
> >                                         and to unify the namespace, you
> > have to use the same syntax. I guess you disagree with me on that. (If
> > not, how would you do it?)
> 
> I'd go one level up: Eliminate the distinctions that bother you, not try to
> patch over them.

But that is my point too. Peter

Re: file as a directory

[Jesse Pollard]

On Tuesday 30 November 2004 10:42, Jan Engelhardt wrote:
> >> [quote]
> >
> >[and another one]
>
> Do you really want to put an XML parser and all the associated code
> (read: McDonalds*) into the kernel?

AND assume the XML file is correct...

Re: file as a directory

[Jan Engelhardt]

>> >> [quote]
>> >[and another one]
>>
>> Do you really want to put an XML parser and all the associated code
>> (read: McDonalds*) into the kernel?
>
>AND assume the XML file is correct...

AND use newer compilers that supposedly create bigger code.

(I do not say that they do, and I neither say that they do not, I just read
about the backwards compat on kerneltraffic.)

Clearly, this is either a userspace job (-> FUSE, hint, hint) or nothing at
all. What if this madness was applied to any file{,-type} in the FS? You could
open() something and always got a shot (i.e. open(...) = ESUCCESS).

Having something like /etc/passwd/daemon spitting out the GECOS for user
"daemon" would make systems more open for stack smash attacks. Instead of
trying to squash a lot of shellcode just to read <your favorite file>, you just
need to use <your favorite file>/<your favorite entry> and *poof*. Even worse
when it's done with O_RDWR / O_WRONLY.

Hell, I do not even want to imagine

	unlink("unlinkthisfile");
	if(stat("unlinkthisfile", &sb) == 0) {
		BUG(); <--
	}

"<--" coming true just because that's the logic of some extension module.


Jan Engelhardt
-- 
ENOSPC

Re: file as a directory

[Horst von Brand]

Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> said:
> On Tue, 2004-11-30 at 16:31, Horst von Brand wrote:
> > > But namespace unification is important,

> > Why? Directories are directories, files are files, file contents is file
> > contents. Mixing them up is a bad idea.

> I disagree, I think it is a good idea.

Looks that way ;-)

> Why is namespace unification important? Because you can use the same
> tools on everything. Previously, each tool could handle one namespace.

Right.

> A very simple example would be:
> I want to count the words in the Appendix of my book.

I can do it... each chapter is a separate file ;-)

> If I can't select the appendix, my "wc" tool is useless (or very
> difficult to use). On the other hand if I can say
> 
> wc ~/book/Appendix

> it's fine.

Exactly what I'd do.

>            Hans Reiser would say that "namespaces are the roads and
> waterways of the operating system" and "the value of an operating system
> is proportional to the number of connections you can make". I think he
> is right in that.

I happen to agree.

>                   And the authors of Unix knew it too, when they used
> the same namespace for devices and files. They didn't say "files are
> files and devices are devices". They said the difference should not
> matter to the applications.

Right. And network connections, and pipes (connecting programs) are files
too.

> But there is still namespace fragmentation even in Unix, and this is
> just one of them.

You are right.

The next question is "Is the fragmentation required?", and then "How would
an OS without this fragmentation work?". Anwer to the first question is
"Not really, but...", answer to the second one is "It has been tried, got
nowhere" (The various IBM minicomputer OSes had no filesystem, just a
(relational) database for everything; didn't catch on. Multiple people have
worked on "persistent programming" (data floats around, programs munge it
and then go away), even APL and similar languages worked on that principle;
none did never get anywhere near "popular". There must be other examples
around).  You can't just go around pretending an element in an array is the
same as a device or a full database, recursively.  That way lies madness,
recursion _has_ to stop somewhere.

> >  Sure, you could build a filesystem
> > of sorts (perhaps more in the vein of persistent programming, or even data
> > base systems) where there simply is no distinction (because there are no
> > differences to show), but that is something different.

> > >                                         and to unify the namespace, you
> > > have to use the same syntax. I guess you disagree with me on that. (If
> > > not, how would you do it?)
> > 
> > I'd go one level up: Eliminate the distinctions that bother you, not try to
> > patch over them.
> 
> But that is my point too.

But the result _can't_ live inside the Unix worldview, it is quite at odds
with it on a fundamental level. I.e., build an environment that works that
way, don't go around trying to pretend things are what they aren't. 
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: file as a directory

[Dr. Giovanni A. Orlando]

Peter Foldiak wrote:

>On Tue, 2004-11-30 at 16:31, Horst von Brand wrote:
>  
>
>>>But namespace unification is important,
>>>      
>>>
>>Why? Directories are directories, files are files, file contents is file
>>contents. Mixing them up is a bad idea.
>>    
>>
>
>I disagree, I think it is a good idea.
>  
>
Hi,

    Please remember DOS.

    In DOS a directory is a file with a SPECIAL attrib: D_DIR.

    In Unix, is basically the same.

    There are nothing bad. The attrib specify that a file with the 
directory attrib
    may include additional files or directories.

Thanks,
Giovanni.

>Why is namespace unification important? Because you can use the same
>tools on everything. Previously, each tool could handle one namespace.
>
>A very simple example would be:
>I want to count the words in the Appendix of my book.
>If I can't select the appendix, my "wc" tool is useless (or very
>difficult to use). On the other hand if I can say
>
>wc ~/book/Appendix
>
>it's fine. Hans Reiser would say that "namespaces are the roads and
>waterways of the operating system" and "the value of an operating system
>is proportional to the number of connections you can make". I think he
>is right in that. And the authors of Unix knew it too, when they used
>the same namespace for devices and files. They didn't say "files are
>files and devices are devices". They said the difference should not
>matter to the applications.
>But there is still namespace fragmentation even in Unix, and this is
>just one of them.
>
>  
>
>> Sure, you could build a filesystem
>>of sorts (perhaps more in the vein of persistent programming, or even data
>>base systems) where there simply is no distinction (because there are no
>>differences to show), but that is something different.
>>
>>    
>>
>>>                                        and to unify the namespace, you
>>>have to use the same syntax. I guess you disagree with me on that. (If
>>>not, how would you do it?)
>>>      
>>>
>>I'd go one level up: Eliminate the distinctions that bother you, not try to
>>patch over them.
>>    
>>
>
>But that is my point too. Peter
>
>  
>


-- 


-- 
-- 

Check FT Websites ... http://www.futuretg.com  - ftp://ftp.futuretg.com
http://www.FTLinuxCourse.com
    http://www.FTLinuxCourse.com/Certification
http://www.rpmparadaise.org
http://GNULinuxUtilities.com
http://www.YourPersonalOperatingSystem.com

WorldWide Global Mobile: +39 393 665 4239

-- 

Re: file as a directory

[Amit Gud]

I cannot imagine viewing the entire filesystem as a single huge XML file. 

My suggestion is to add a framework, an infrastructure, in the VFS
wherein a simple plugin can be written to poke into the file as if it
were a directory. So with that framework in place, I can write a
plugin for archive support (treating the .tar files as directories),
Peter could write a plugin for poking into /etc/passwd (treating it as
a directory), and Jon Doe could write a plugin for sendmail.cf

like:
--
struct file_operations ops = {
   .read            = tar_readdir,
   .readdir        = tar_readdir,
   ......
};

register_file_type("tar", &ops);
--

How good would this be? 

AG
--
May the source be with you.

Re: file as a directory

[Jan Engelhardt]

>My suggestion is to add a framework, an infrastructure, in the VFS
>wherein a simple plugin can be written to poke into the file as if it
>were a directory. So with that framework in place, I can write a
>plugin for archive support (treating the .tar files as directories),
>Peter could write a plugin for poking into /etc/passwd (treating it as
>a directory), and Jon Doe could write a plugin for sendmail.cf

That's something I could live with, but how do you want to tag a file being
"tar" so that tar_ops is used instead of the "default file" ops?

You could not do so without an extra function, and once you use that extra
function to tag a certain file being "tar" -- you know that extensions are
kinda "worthless", and, especially, unrealiable -- you could also have used tar
-tvf.

Did I mention tar is not the perfect format? It's because it is lacking an
index and letting the kernel wade through a GB-sized tar file just to perform
and readdir (yet imagine reading the last file of it) would be a hell of
skipping. Keeping a non-persistent index in memory may solve the problem, but
hey, I also do not want to spend too much memory just for a single tar file.

>struct file_operations ops = {
>   .read            = tar_readdir,
>   .readdir        = tar_readdir,
>   ......
>};
>
>register_file_type("tar", &ops);




Jan Engelhardt
-- 
ENOSPC

Re: file as a directory

[Scott Young]

On Tue, 30 Nov 2004 19:39:15 +0100 (MET), Jan Engelhardt
<jengelh@linux01.gwdg.de> wrote:
> >My suggestion is to add a framework, an infrastructure, in the VFS
> >wherein a simple plugin can be written to poke into the file as if it
> >were a directory. So with that framework in place, I can write a
> >plugin for archive support (treating the .tar files as directories),
> >Peter could write a plugin for poking into /etc/passwd (treating it as
> >a directory), and Jon Doe could write a plugin for sendmail.cf

The biggest problem I see with adding the complicated stuff to VFS is
the bloat and risk to system stability.  However, some things cannot
be done in userspace, such as good caching.  How is one userspace
library supposed to keep a transparent cache of, for example, an index
for a tar file, not clutter up the on-disk representation of the
cache, effectively manage space utilization, and  be able to
efficiently detect changes to files in order to invalidate the cache? 
This would become orders of magnitude easier if a ubiquitous
filesystem interface were in use.  However, the only ubiquitous
filesystem interface is VFS, which shouldn't have to take all the code
bloat.

Maybe something crazy could work.  Let's take some concepts from the
Aspect Oriented Programming paradigm.  Whenever a program is loaded
into memory, calls in the program to the vfs interface are modified to
instead call new userspace functions that have all of the desired
functionality, and those userspace functions eventually call the real
system functions.  The kernel wouldn't have to take the bloat, plus it
would be able to do things the userspace libraries wouldn't be able to
do efficiently.  It's the best of both worlds, with a little insanity
thrown in (It'd be neat to see the loader bootstrap its own code to
weave in the caching of the pre-woven binaries).


> That's something I could live with, but how do you want to tag a file being
> "tar" so that tar_ops is used instead of the "default file" ops?
> 
> You could not do so without an extra function, and once you use that extra
> function to tag a certain file being "tar" -- you know that extensions are
> kinda "worthless", and, especially, unrealiable -- you could also have used tar
> -tvf.
> 
> Did I mention tar is not the perfect format? It's because it is lacking an
> index and letting the kernel wade through a GB-sized tar file just to perform
> and readdir (yet imagine reading the last file of it) would be a hell of
> skipping. Keeping a non-persistent index in memory may solve the problem, but
> hey, I also do not want to spend too much memory just for a single tar file.

It would also be nice to have an interface which can build, maintain,
and cache on the disk a persistent index into a tar file on the disk,
and then be able to delete this index when space is running low. 
Plus, this index could be generated by streaming the file through
memory, so you don't need to consume too much memory for a single
file.


> >struct file_operations ops = {
> >   .read            = tar_readdir,
> >   .readdir        = tar_readdir,
> >   ......
> >};
> >
> >register_file_type("tar", &ops);
> 
> Jan Engelhardt
> --
> ENOSPC

Re: file as a directory

[Amit Gud]

On Tue, 30 Nov 2004 19:39:15 +0100 (MET), Jan Engelhardt
<jengelh@linux01.gwdg.de> wrote:
> >My suggestion is to add a framework, an infrastructure, in the VFS
> >wherein a simple plugin can be written to poke into the file as if it
> >were a directory. So with that framework in place, I can write a
> >plugin for archive support (treating the .tar files as directories),
> >Peter could write a plugin for poking into /etc/passwd (treating it as
> >a directory), and Jon Doe could write a plugin for sendmail.cf
>
> That's something I could live with, but how do you want to tag a file being
> "tar" so that tar_ops is used instead of the "default file" ops?
>
> You could not do so without an extra function, and once you use that extra
> function to tag a certain file being "tar"

Yes an extra function, or few lines in vfs_readdir().

> -- you know that extensions are
> kinda "worthless", and, especially, unrealiable -- you could also have used tar
> -tvf.

I don't see how "worthless" is viewing certain files as directories.
It is worth to do 'vi /root/abc.tar.gz/README' to edit a file than to
inflate it yourself, make changes and deflate it again..deleting the
stuff inflated. And its on code's part to make it as reliable as
possible.

> Did I mention tar is not the perfect format? It's because it is lacking an
> index and letting the kernel wade through a GB-sized tar file just to perform
> and readdir (yet imagine reading the last file of it) would be a hell of
> skipping. Keeping a non-persistent index in memory may solve the problem, but
> hey, I also do not want to spend too much memory just for a single tar file.

There lies a problem with tar formats. So the tar plugin writer should
take this into account. But I see no problem with, for example,
/etc/passwd as a directory. tar is just one possible (and probably
debatable) plugin that can be introduced with the help of this
framework.


AG
--
May the source be with you.

Re: file as a directory

[Peter Foldiak]

Hans,
Following the recent discussions on the lists on "file as directory", I
re-read part of your http://www.namesys.com/v4/v4.html paper (in
sections near "Aggregating Files Can Improve The User Interface To
Them") and realised that you have discussed this issue in quite a lot of
detail already there (see copied below). (Strange though that nobody
referred to it in the discussion! People don't seem to be very good at
reading (and that seems to include me)).

It looks to me that the "file as directory" and "directory as file"
issues are almost identical. What you are saying is that you could
automatically aggregate files in a directory to look like a single file
(you give the example of the /etc/passwd file). I was saying that you
could make a file look like a directory.
Perhaps a better way to think about this is that instead of talking
about directories and files, we just talk about objects. Each object
would have file-type access methods and directory-type access methods.
This has some implications for the syntax. You (in v4.html) suggest
using the /.glued syntax:

/new_syntax_access_path/big_directory_of_small_files/.glued

I think the "object" philosophy would rather imply that the object (or
directory) should have a default glueing method, so when you access the
directory as a file (using read(), for instance)

/new_syntax_access_path/big_directory_of_small_files

would automatically give you the glued file (without having to add the
.glued !) and when you access it as a directory (using readdir(), for
instance), you would get the components listed as a directory.
(I am not sure whether the access method, e.g. read() vs. readdir() is
sufficient to distinguish the meaning. Another way may be putting a "/"
after the objectname to indicate that you want it as a directory.)

If we do this, the applications don't need to know whether they are
dealing with an object consisting of small files, aggregated, or whether
they are looking at a big file with some way of accessing their parts.
If an old application (or user) looks for the /etc/passwd file, it will
still get what it expects without having to know that the file is an
aggregate.

In fact, from the point of view of the file system, it doesn't make much
of a difference, in both cases they map names to keys. The only
difference (as far as I can see) is whether metadata is stored
separately for each component or only once for the objcts. 

If you store metadata only once, then a component could inherit metadata
(such as modification date) from the parent. There should be some way of
telling the file system which bits need their own metadata. But the way
of telling the file system this probably should not be mixed up with the
file naming.

There may be some complications with some parts of files being linked to
a different number of times, so if you remove a hard link from the whole
file, should a component linked from elsewhere be kept or deleted.

Do you think we could leave off the ./glued bit? Would it break too many
things?

 Peter

In http://www.namesys.com/v4/v4.html Hans Reiser wrote:
> ...
> Aggregating Files Can Improve The User Interface To Them
> Consider the use of emacs on a collection of a thousand small 8-32
byte
> files like you might have if you deconstructed /etc/passwd into small
> files with separable acls for every field. It is more convenient in
> screen real estate, buffer management, and other user interface
> considerations, to operate on them as an aggregation all placed into a
> single buffer rather than as a thousand 8-32 byte buffers.
> 
> 
> How Do We Write Modifications To An Aggregation
> Suppose we create a plugin that aggregates all of the files in a
> directory into a single stream. How does one handle writes to that
> aggregation that change the length of the components of that
> aggregation?
> 
> Richard Stallman pointed out to me that if we separate the aggregated
> files with delimiters, then emacs need not be changed at all to
acquire
> an effective interface for large numbers of small files accessed via
an
> aggregation plugin. If
> /new_syntax_access_path/big_directory_of_small_files/.glued is a
plugin
> that aggregates every file in big_directory_of_small_files with a
> delimiter separating every file within the aggregation, then one can
> simply type emacs
> /new_syntax_access_path/big_directory_of_small_files/.glued, and the
> filesystem has done all the work emacs needs to be effective at this.
> Not a line of emacs needs to be changed.
> 
> One needs to be able to choose different delimiting syntax for
different
> aggregation plugins so that one can, for say the passwd file,
aggregate
> subdirectories into lines, and files within those subdirectories into
> colon separate fields within the line. XML would benefit from yet
other
> delimiter construction rules. (We have been told by Philipp Guehring 
of
> LivingXML.NET  that ReiserFS is higher performance than any database
for
> storing XML, so this issue is not purely theoretical.)
> 
> 
> Aggregation Is Best Implemented As Inheritance
> In summary, to be able to achieve precision in security we need to
have
> inheritance with specifiable delimiters and we need whole file
> inheritance to support ACLs.
> 
> 
> One Plugin Using Delimiters That Resemble sys_reiser4() Syntax
> We provide the infrastructure for your constructing plugins that
> implement arbitrary processing of writes to inheriting files, but we
> also supply one generic inheriting file plugin that intentionally uses
> delimiters very close to the sys_reiser4() syntax. We will document
the
> syntax more fully when that code is working, for now syntax details
are
> in the comments in the file invert.c in the source code. ...

Re: file as a directory

[Jan Engelhardt]

>Following the recent discussions on the lists on "file as directory", I
[...]

Recently I discovered that calling "joe /etc" on FreeBSD opens the directory 
itself (rather than erroring out with EISDIR). The byte string looked like a 
number of "struct dirent"s. Did it break apps in BSD? I donot think so.

(Hell, I just voted _for_ the directory-as-a-file and vice-versa stuff, I'm 
never gonna regret it to myself :)

>would automatically give you the glued file (without having to add the
>.glued !) and when you access it as a directory (using readdir(), for
>instance), you would get the components listed as a directory.
>(I am not sure whether the access method, e.g. read() vs. readdir() is
>sufficient to distinguish the meaning. Another way may be putting a "/"
>after the objectname to indicate that you want it as a directory.)

I would not rely on that. Many apps strip a trailing slash (just to add it 
later again) or vice versa!

>If we do this, the applications don't need to know whether they are
>dealing with an object consisting of small files, aggregated, or whether
>they are looking at a big file with some way of accessing their parts.
>If an old application (or user) looks for the /etc/passwd file, it will
>still get what it expects without having to know that the file is an
>aggregate.

What will ls do? Consider this on a "normal" filesystem:

ls -l /etc/passwd
-rw-r--r--  1 root root 1614 Dec 12 23:57 /etc/passwd
ls -l /etc
[lots of files]


What will happen on v4 then?
ls -l /etc
-rw-r--r-- 1 root root 123456 Dec 01 23:45 /etc
ls -l /etc/passwd
-rw-r--r-- 1 root root 1234 Dec 01 23:45 root
-rw-r--r-- 1 root root 1234 Dec 01 23:45 jengelh
-rw-r--r-- 1 root root 1234 Dec 01 23:45 daemon
-rw-r--r-- 1 root root 1234 Dec 01 23:45 sys
-rw-r--r-- 1 root root 1234 Dec 01 23:45 stduser

It's because "ls" checks the type of the argument to decide what to do (unless
you add -d).


Jan Engelhardt
-- 
ENOSPC

Re: file as a directory

[Hans Reiser]

Peter, I think you are right, though it might be useful to have the 
default be dirname/..../glued and to allow users to link 
dirname/..../filebody to 
dirname/..../something_else_if_they_want_it_to_not_be_glued, and to have 
dirname/..../filebody or whatever it is linked to be what they get if 
they read the directory as a file.

Hans

Peter Foldiak wrote:

>Hans,
>Following the recent discussions on the lists on "file as directory", I
>re-read part of your http://www.namesys.com/v4/v4.html paper (in
>sections near "Aggregating Files Can Improve The User Interface To
>Them") and realised that you have discussed this issue in quite a lot of
>detail already there (see copied below). (Strange though that nobody
>referred to it in the discussion! People don't seem to be very good at
>reading (and that seems to include me)).
>
>It looks to me that the "file as directory" and "directory as file"
>issues are almost identical. What you are saying is that you could
>automatically aggregate files in a directory to look like a single file
>(you give the example of the /etc/passwd file). I was saying that you
>could make a file look like a directory.
>Perhaps a better way to think about this is that instead of talking
>about directories and files, we just talk about objects. Each object
>would have file-type access methods and directory-type access methods.
>This has some implications for the syntax. You (in v4.html) suggest
>using the /.glued syntax:
>
>/new_syntax_access_path/big_directory_of_small_files/.glued
>
>I think the "object" philosophy would rather imply that the object (or
>directory) should have a default glueing method, so when you access the
>directory as a file (using read(), for instance)
>
>/new_syntax_access_path/big_directory_of_small_files
>
>would automatically give you the glued file (without having to add the
>.glued !) and when you access it as a directory (using readdir(), for
>instance), you would get the components listed as a directory.
>(I am not sure whether the access method, e.g. read() vs. readdir() is
>sufficient to distinguish the meaning. Another way may be putting a "/"
>after the objectname to indicate that you want it as a directory.)
>
>If we do this, the applications don't need to know whether they are
>dealing with an object consisting of small files, aggregated, or whether
>they are looking at a big file with some way of accessing their parts.
>If an old application (or user) looks for the /etc/passwd file, it will
>still get what it expects without having to know that the file is an
>aggregate.
>
>In fact, from the point of view of the file system, it doesn't make much
>of a difference, in both cases they map names to keys. The only
>difference (as far as I can see) is whether metadata is stored
>separately for each component or only once for the objcts. 
>
>If you store metadata only once, then a component could inherit metadata
>(such as modification date) from the parent. There should be some way of
>telling the file system which bits need their own metadata. But the way
>of telling the file system this probably should not be mixed up with the
>file naming.
>
>There may be some complications with some parts of files being linked to
>a different number of times, so if you remove a hard link from the whole
>file, should a component linked from elsewhere be kept or deleted.
>
>Do you think we could leave off the ./glued bit? Would it break too many
>things?
>
> Peter
>
>In http://www.namesys.com/v4/v4.html Hans Reiser wrote:
>  
>
>>...
>>Aggregating Files Can Improve The User Interface To Them
>>Consider the use of emacs on a collection of a thousand small 8-32
>>    
>>
>byte
>  
>
>>files like you might have if you deconstructed /etc/passwd into small
>>files with separable acls for every field. It is more convenient in
>>screen real estate, buffer management, and other user interface
>>considerations, to operate on them as an aggregation all placed into a
>>single buffer rather than as a thousand 8-32 byte buffers.
>>
>>
>>How Do We Write Modifications To An Aggregation
>>Suppose we create a plugin that aggregates all of the files in a
>>directory into a single stream. How does one handle writes to that
>>aggregation that change the length of the components of that
>>aggregation?
>>
>>Richard Stallman pointed out to me that if we separate the aggregated
>>files with delimiters, then emacs need not be changed at all to
>>    
>>
>acquire
>  
>
>>an effective interface for large numbers of small files accessed via
>>    
>>
>an
>  
>
>>aggregation plugin. If
>>/new_syntax_access_path/big_directory_of_small_files/.glued is a
>>    
>>
>plugin
>  
>
>>that aggregates every file in big_directory_of_small_files with a
>>delimiter separating every file within the aggregation, then one can
>>simply type emacs
>>/new_syntax_access_path/big_directory_of_small_files/.glued, and the
>>filesystem has done all the work emacs needs to be effective at this.
>>Not a line of emacs needs to be changed.
>>
>>One needs to be able to choose different delimiting syntax for
>>    
>>
>different
>  
>
>>aggregation plugins so that one can, for say the passwd file,
>>    
>>
>aggregate
>  
>
>>subdirectories into lines, and files within those subdirectories into
>>colon separate fields within the line. XML would benefit from yet
>>    
>>
>other
>  
>
>>delimiter construction rules. (We have been told by Philipp Guehring 
>>    
>>
>of
>  
>
>>LivingXML.NET  that ReiserFS is higher performance than any database
>>    
>>
>for
>  
>
>>storing XML, so this issue is not purely theoretical.)
>>
>>
>>Aggregation Is Best Implemented As Inheritance
>>In summary, to be able to achieve precision in security we need to
>>    
>>
>have
>  
>
>>inheritance with specifiable delimiters and we need whole file
>>inheritance to support ACLs.
>>
>>
>>One Plugin Using Delimiters That Resemble sys_reiser4() Syntax
>>We provide the infrastructure for your constructing plugins that
>>implement arbitrary processing of writes to inheriting files, but we
>>also supply one generic inheriting file plugin that intentionally uses
>>delimiters very close to the sys_reiser4() syntax. We will document
>>    
>>
>the
>  
>
>>syntax more fully when that code is working, for now syntax details
>>    
>>
>are
>  
>
>>in the comments in the file invert.c in the source code. ...
>>    
>>
>
>
>
>  
>

Re: file as a directory

[Peter Foldiak]

On Tue, 2004-12-14 at 17:21, Jan Engelhardt wrote:
> Recently I discovered that calling "joe /etc" on FreeBSD opens the directory 
> itself (rather than erroring out with EISDIR). The byte string looked like a 
> number of "struct dirent"s. Did it break apps in BSD? I donot think so.

I don't imagine anyone would miss that behaviour, we can probably have
something more useful than a
cat: /etc: Is a directory
message.

> >(I am not sure whether the access method, e.g. read() vs. readdir() is
> >sufficient to distinguish the meaning. Another way may be putting a "/"
> >after the objectname to indicate that you want it as a directory.)
> 
> I would not rely on that. Many apps strip a trailing slash (just to add it 
> later again) or vice versa!

ok, I don't really care about the trailing slash. The reading method
(read() vs. readdir()) may be enough to distinguish the uses. The
trailing / is still a useful (and logical) visual notation though for
the different meanings, so something like ls could give both versions.

> What will ls do? 

list both /etc/passwd and /etc/passwd/ perhaps?

> Consider this on a "normal" filesystem:
> 
> ls -l /etc/passwd
> -rw-r--r--  1 root root 1614 Dec 12 23:57 /etc/passwd
> ls -l /etc
> [lots of files]
> 
> 
> What will happen on v4 then?
> ls -l /etc
> -rw-r--r-- 1 root root 123456 Dec 01 23:45 /etc
> ls -l /etc/passwd
> -rw-r--r-- 1 root root 1234 Dec 01 23:45 root
> -rw-r--r-- 1 root root 1234 Dec 01 23:45 jengelh
> -rw-r--r-- 1 root root 1234 Dec 01 23:45 daemon
> -rw-r--r-- 1 root root 1234 Dec 01 23:45 sys
> -rw-r--r-- 1 root root 1234 Dec 01 23:45 stduser
> 
> It's because "ls" checks the type of the argument to decide what to do (unless
> you add -d).
> 
> 
> Jan Engelhardt

Re: file as a directory

[Jan Engelhardt]

>I don't imagine anyone would miss that behaviour, we can probably have
>something more useful than a
>cat: /etc: Is a directory
>message.

But if /etc can also be a file, cat would not complain!

>ok, I don't really care about the trailing slash. The reading method
>(read() vs. readdir()) may be enough to distinguish the uses. The
>trailing / is still a useful (and logical) visual notation though for
>the different meanings, so something like ls could give both versions.

Yes, indeed the idea itself is good. `ls` uses it to follow symlinks that 
point to directories:

ls -l /U
lrwxrwxrwx  1 root root 35 Dec  6 22:03 /U -> 
/media/usb-0000000000004287:0:0:0p

ls -l /U/
drwxr-xr-x   2 root root  48 Dec  6 14:27 .
drwxr-xr-x  31 root root 824 Dec 14 17:51 ..
-rw-r--r--  1 root root 1614 Dec 12 23:57 tstpasswd

>> What will ls do? 
>list both /etc/passwd and /etc/passwd/ perhaps?

Well, and before ls is tweaked to do so?



Jan Engelhardt
-- 
ENOSPC

Re: file as a directory

[Horst von Brand]

Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> said:

[...]

> Perhaps a better way to think about this is that instead of talking
> about directories and files, we just talk about objects.

Then you have a collection of interrelated objects, i.e., a database.
Operating systems that work on databases (no filesystem) have been done,
and are a nice idea... but are far, far away from Unix.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: file as a directory

[Peter Foldiak]

On Tue, 2004-12-14 at 17:24, Hans Reiser wrote:
> Peter, I think you are right, though it might be useful to have the 
> default be dirname/..../glued and to allow users to link 
> dirname/..../filebody to 
> dirname/..../something_else_if_they_want_it_to_not_be_glued, and to have 
> dirname/..../filebody or whatever it is linked to be what they get if 
> they read the directory as a file.

Yes. I assume you mean that dirname in itself should always be
interpreted as dirname/..../glued, which by default would be a linked to
dirname/..../filebody, the latter being the file content, right?

Also, a pseudofile (e.g. dirname/..../structure ?) could be used to
specify how the files should be glued together. A simple question is,
for instance, what separators to use between the components, and what
ordering to use when putting the component objects together. (This
pseudofile could also determine more complicated ways of composing
objects.) 

The component objects themselves could be full objects, so they
themselves could have sub-components.
 Peter

Re: file as a directory

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter Foldiak wrote:
| On Tue, 2004-12-14 at 17:24, Hans Reiser wrote:
|
|>Peter, I think you are right, though it might be useful to have the
|>default be dirname/..../glued and to allow users to link
|>dirname/..../filebody to
|>dirname/..../something_else_if_they_want_it_to_not_be_glued, and to have
|>dirname/..../filebody or whatever it is linked to be what they get if
|>they read the directory as a file.
|
|
| Yes. I assume you mean that dirname in itself should always be
| interpreted as dirname/..../glued, which by default would be a linked to
| dirname/..../filebody, the latter being the file content, right?

I think he means the other way.
	cat foo
is the same as
	cat foo/.../filebody
where "filebody" is some sort of link to (maybe) foo/.../glued.

Understand that you want to allow "normal" files which are not "glued"
anything, but also "glued" files which are composed of sub-files.

| Also, a pseudofile (e.g. dirname/..../structure ?) could be used to
| specify how the files should be glued together. A simple question is,
| for instance, what separators to use between the components, and what
| ordering to use when putting the component objects together. (This
| pseudofile could also determine more complicated ways of composing
| objects.)

If the filesystem does caching, I'd rather have a type of executable
which, read normally, appears as a stream of its own standard output.
You'd get the actual file as something like bar/.../source.

This could be done with pipes and daemons, but it's not as easy to
manage and seems impossible to do as efficiently (with built-in caching,
etc.)

| The component objects themselves could be full objects, so they
| themselves could have sub-components.

Right.

Also, there should be an inverse. For instance, a file-as-directory type
object should have a "contents" object, usually a normal directory, but
which could conceivably be any type of object, including a code-ish
object which implements a filesystem. Accessing foo/ would be the same
as accessing foo/.../contents, only because "..." (or whatever we use
for meta-files) is outside the actual directory namespace,
foo/.../contents/... refers to the metas of object "contents", which are
different than the metas of object "foo".

These two steps essentially create userspace "plugins", and do away with
having to mount other kernel layers such as lufs (or whatever its
current implementation is). It does have one important implication, though:

It's important that "metas" or "..." or whatever we've decided on should
_not_ be mutable by a "userspace" plugin that I have described, nor
should any meta-files created by kernel plugins. There would be other
security implications, of course -- user should still not be able to
create files that are owned by other users and setuid. I'm not sure
where such checks should go, but we want mortal users to be able to add
whatever plugins they want, while super-users can feel safe using the
metas interface to manipulate user files.

I think the issues with directory-as-a-file were the same problems you
get when you allow hardlinked directories -- that you'd eventually have
to ditch reference counting for a garbage collector, which is hellishly
impractical. I don't have a solution to that, other than dropping
hardlinks entirely.

Hmm. Why do we need hardlinks? I forget.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQb/BxHgHNmZLgCUhAQJ4tA//ZVHoODSgxrIKgk2RtxGW4X8piQc0cGVq
KLwQmbCg/LRRld1VhPDikROMjZWrubzzsY5NujEmZFjdnA/m3mgO5DVsd6xtiGDJ
5B47aDTyygRWA6SONw6q+47me2lqFSBvZYOM9E468CNNKEExMys9R+sY82b1lbRi
iAesW49tWusZtonI66D1aK3a6ue9MEBjbZkGi+qR1DYK02+gc2SVbeDc7dbx4GrC
0GBfO92O+0OJKnYn3/e7wag57/cJYEOrDW2Dg9mOD2UOK+6fmJLf8lx6XIxI+QOp
YLp9dY4scac5PlC+z6DUPuv8rRGu7GUDJdffzNoG18MYQChZcE6e1lVWPpunMiqh
quWWr6wi09m5eoUq27w8fHS0UeJ/fn1+Ckr+4+zEZv3CluK0KMqPDYOcKfR5Hs/3
MYzrbVXFoaOZd59oMzBlikJoMbTikm0UNSQhaKlw3ZFH46V/lKrCzOjh3ZNQ3rqx
t0f0BrVgGKluX5cw5srzxOBjnOxw3OMtsuAlElHunNALeRn8xMFcc7tuVwVn0Wxv
hx48IUk3KwTrAomN0SS4m+a9MLUjtxXMkr/1hViJ8ofRRqXMFghXq/WTeIA4BLxM
9SnfZqHcldF3NKZq1bCQlv8nJFOPxwQE0Zt9gtFSBq6rndfexdO2XZtyH7pxm4FX
X/NK4MR5iSk=
=3BWI
-----END PGP SIGNATURE-----

Re: file as a directory

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
| Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> said:
|
| [...]
|
|
|>Perhaps a better way to think about this is that instead of talking
|>about directories and files, we just talk about objects.
|
|
| Then you have a collection of interrelated objects, i.e., a database.
| Operating systems that work on databases (no filesystem) have been done,
| and are a nice idea... but are far, far away from Unix.

Really? Because most Unix tools don't behave all that strangely when I
enable metadata. If Linux incrementally changes into something that is
not Unix, what is lost, besides the name?

Remember that a filesystem is a specialized database. It is specialized
for performance. If Hans can generalize it without losing that
performance, who can complain?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQb/C+3gHNmZLgCUhAQIjGg/+M17+XviIqi+uFy3RR6x8csoPne3KYBlL
cnjoDML4JZAXLn0YdDKAafCa1kt4xCbPYa/ZQjqw9RzUbnf6YEtdrvuVsLxsTC7t
yAHQDSFCpQ1ssCsy5S2ku80Srx6LkTf2R33F8BMr0hTVNg5eoDEL2RuOtT8tJ0g5
gMrdX04BIM4gxQEv2elvfNIm7cxyAv1aBvurcpb/VPckoaUChPQSgauuS6rJ0oYu
AmLhkLBZ+8NloQ2U4tKHdY8YrQWtLAzeyG+rYwIFxjPWRMUG54WYAVD+G2R6a9bN
nksqOtZ19lcCdjBOb5+5Wa3sOEOgwYRKySS4H/S/TTBI8cbm5+7tDwi30E4eiV9C
PQUi+yi3+18j+cp3Of+AhZsA99PgjJ9q7GKaC10SX5/gk0d2jq0TwADBR1v2EdbM
cABfKnaXWnS0RzqFARF/MZaZz++KWAbmsZwnTk7N9Tm2AfKSRjFb5wqsKbV01F+i
XltsPGnYj0f4DYKrSxQRxkhQxC5J9b2P69N9JhplvzcvAVQWIMvhPsU10i4vm7oN
GJi+ncuYzmdQQXB0Sxpar5uwdYj+xgcWcK35iX4mGG9E6CWythpKJQFq85u/KKsX
J37o0CcxY1eseV+5qgYEeGm0eLuoxllaKNW9yJzsX9Xf2qxUoS1mr0si2eFulsLh
z72g9LoNfyg=
=hIBh
-----END PGP SIGNATURE-----

Re: file as a directory

[Hans Reiser]

Horst von Brand wrote:

>Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> said:
>
>[...]
>
>  
>
>>Perhaps a better way to think about this is that instead of talking
>>about directories and files, we just talk about objects.
>>    
>>
>
>Then you have a collection of interrelated objects, i.e., a database.
>Operating systems that work on databases (no filesystem) have been done,
>and are a nice idea... but are far, far away from Unix.
>  
>
A journey of a thousand leagues begins with a single step.

Actually, databases are the wrong solution because they are relational, 
and what is needed is a semi-structured query language that is upwardly 
compatible with Unix hierarchical semantics, ala 
www.namesys.com/future_vision.html

Re: file as a directory

[Hans Reiser]

Peter Foldiak wrote:

>On Tue, 2004-12-14 at 17:24, Hans Reiser wrote:
>  
>
>>Peter, I think you are right, though it might be useful to have the 
>>default be dirname/..../glued and to allow users to link 
>>dirname/..../filebody to 
>>dirname/..../something_else_if_they_want_it_to_not_be_glued, and to have 
>>dirname/..../filebody or whatever it is linked to be what they get if 
>>they read the directory as a file.
>>    
>>
>
>Yes. I assume you mean that dirname in itself should always be
>interpreted as dirname/..../glued, which by default would be a linked to
>dirname/..../filebody, the latter being the file content, right?
>  
>
reversed:

dirname in itself should always be
interpreted as dirname/..../filebody, which by default would be a linked to
dirname/..../glued, 

>Also, a pseudofile (e.g. dirname/..../structure ?) could be used to
>specify how the files should be glued together. A simple question is,
>for instance, what separators to use between the components, and what
>ordering to use when putting the component objects together. (This
>pseudofile could also determine more complicated ways of composing
>objects.) 
>  
>
Could be cool.

>The component objects themselves could be full objects, so they
>themselves could have sub-components.
> Peter
>
>
>
>  
>

Re: file as a directory

[Hans Reiser]

David Masover wrote:

> Peter Foldiak wrote:
> | On Tue, 2004-12-14 at 17:24, Hans Reiser wrote:
> |
> |>Peter, I think you are right, though it might be useful to have the
> |>default be dirname/..../glued and to allow users to link
> |>dirname/..../filebody to
> |>dirname/..../something_else_if_they_want_it_to_not_be_glued, and to have
> |>dirname/..../filebody or whatever it is linked to be what they get if
> |>they read the directory as a file.
> |
> |
> | Yes. I assume you mean that dirname in itself should always be
> | interpreted as dirname/..../glued, which by default would be a linked to
> | dirname/..../filebody, the latter being the file content, right?
>
> I think he means the other way.
>     cat foo
> is the same as
>     cat foo/.../filebody
> where "filebody" is some sort of link to (maybe) foo/.../glued.
>
> Understand that you want to allow "normal" files which are not "glued"
> anything, but also "glued" files which are composed of sub-files.
>
> | Also, a pseudofile (e.g. dirname/..../structure ?) could be used to
> | specify how the files should be glued together. A simple question is,
> | for instance, what separators to use between the components, and what
> | ordering to use when putting the component objects together. (This
> | pseudofile could also determine more complicated ways of composing
> | objects.)
>
> If the filesystem does caching, I'd rather have a type of executable
> which, read normally, appears as a stream of its own standard output.
> You'd get the actual file as something like bar/.../source.

Interesting....

>
> This could be done with pipes and daemons, but it's not as easy to
> manage and seems impossible to do as efficiently (with built-in caching,
> etc.)

Explain the value of caching executable output more please.

>
> | The component objects themselves could be full objects, so they
> | themselves could have sub-components.
>
> Right.
>
> Also, there should be an inverse. For instance, a file-as-directory type
> object should have a "contents" object, usually a normal directory, but
> which could conceivably be any type of object, including a code-ish
> object which implements a filesystem. Accessing foo/ would be the same
> as accessing foo/.../contents, only because "..." (or whatever we use
> for meta-files) is outside the actual directory namespace,
> foo/.../contents/... refers to the metas of object "contents", which are
> different than the metas of object "foo".

Are you sure that having more than one "...." is needed?  Hmmm, 
interesting, must think about it.

>
> These two steps essentially create userspace "plugins", and do away with
> having to mount other kernel layers such as lufs (or whatever its
> current implementation is).

I don't follow this point above.

> It does have one important implication, though:
>
> It's important that "metas" or "..." or whatever we've decided on should
> _not_ be mutable by a "userspace" plugin that I have described, nor
> should any meta-files created by kernel plugins. There would be other
> security implications, of course -- user should still not be able to
> create files that are owned by other users and setuid. I'm not sure
> where such checks should go, but we want mortal users to be able to add
> whatever plugins they want, while super-users can feel safe using the
> metas interface to manipulate user files.
>
> I think the issues with directory-as-a-file were the same problems you
> get when you allow hardlinked directories -- that you'd eventually have
> to ditch reference counting for a garbage collector, which is hellishly
> impractical. I don't have a solution to that, other than dropping
> hardlinks entirely.

This issue is overblown.  Other fields have solved this problem. 

>
> Hmm. Why do we need hardlinks? I forget.

Re: file as a directory

[Hans Reiser]

David Masover wrote:

>
>
> Remember that a filesystem is a specialized database. It is specialized
> for performance. 

Yes, this is exactly right.  The reason earlier efforts to generalize 
filesystems failed was performance, and the reason performance reduced 
was a lack of hard work on the problem.

Re: file as a directory

[Peter Foldiak]

On Wed, 2004-12-15 at 04:47, David Masover wrote:
> Peter Foldiak wrote:
> | Also, a pseudofile (e.g. dirname/..../structure ?) could be used to
> | specify how the files should be glued together. A simple question is,
> | for instance, what separators to use between the components, and what
> | ordering to use when putting the component objects together. (This
> | pseudofile could also determine more complicated ways of composing
> | objects.)
> 
> If the filesystem does caching, I'd rather have a type of executable
> which, read normally, appears as a stream of its own standard output.
> You'd get the actual file as something like bar/.../source.

This sounds better and more general than my proposed ..../structure
file. So could you explain this in a bit more detail?
Would for instance the simplest (and default?) glueing code in your
bar/..../source file be

cat *

which just concatenates all the subcomponents in no particular order?


> This could be done with pipes and daemons, but it's not as easy to
> manage and seems impossible to do as efficiently (with built-in caching,
> etc.)

How would you do it?

> 
> | The component objects themselves could be full objects, so they
> | themselves could have sub-components.
> 
> Right.
> 
> Also, there should be an inverse. For instance, a file-as-directory type
> object should have a "contents" object, usually a normal directory, but
> which could conceivably be any type of object, including a code-ish
> object which implements a filesystem. Accessing foo/ would be the same
> as accessing foo/.../contents, only because "..." (or whatever we use
> for meta-files) is outside the actual directory namespace,
> foo/.../contents/... refers to the metas of object "contents", which are
> different than the metas of object "foo".
> 
> These two steps essentially create userspace "plugins", and do away with
> having to mount other kernel layers such as lufs (or whatever its
> current implementation is). It does have one important implication, though:
> 
> It's important that "metas" or "..." or whatever we've decided on should
> _not_ be mutable by a "userspace" plugin that I have described, nor
> should any meta-files created by kernel plugins. There would be other
> security implications, of course -- user should still not be able to
> create files that are owned by other users and setuid. I'm not sure
> where such checks should go, but we want mortal users to be able to add
> whatever plugins they want, while super-users can feel safe using the
> metas interface to manipulate user files.

Sounds really interesting.

Re: file as a directory

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> Horst von Brand wrote:
> >Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> said:

> >[...]

> >>Perhaps a better way to think about this is that instead of talking
> >>about directories and files, we just talk about objects.

> >Then you have a collection of interrelated objects, i.e., a database.
> >Operating systems that work on databases (no filesystem) have been done,
> >and are a nice idea... but are far, far away from Unix.

> A journey of a thousand leagues begins with a single step.

Right.  But you need to know where you are going, and why.

> Actually, databases are the wrong solution because they are relational, 

Says who?

> and what is needed is a semi-structured query language that is upwardly 
> compatible with Unix hierarchical semantics, ala 
> www.namesys.com/future_vision.html
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: file as a directory

[Hans Reiser]

Horst von Brand wrote:

>Hans Reiser <reiser@namesys.com> said:
>  
>
>>Horst von Brand wrote:
>>    
>>
>>>Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> said:
>>>      
>>>
>
>  
>
>>>[...]
>>>      
>>>
>
>  
>
>>>>Perhaps a better way to think about this is that instead of talking
>>>>about directories and files, we just talk about objects.
>>>>        
>>>>
>
>  
>
>>>Then you have a collection of interrelated objects, i.e., a database.
>>>Operating systems that work on databases (no filesystem) have been done,
>>>and are a nice idea... but are far, far away from Unix.
>>>      
>>>
>
>  
>
>>A journey of a thousand leagues begins with a single step.
>>    
>>
>
>Right.  But you need to know where you are going, and why.
>
>  
>
>>Actually, databases are the wrong solution because they are relational, 
>>    
>>
>
>Says who?
>  
>
Read the

www.namesys.com/future_vision.html

paper for why relational is the wrong model.

>  
>
>>and what is needed is a semi-structured query language that is upwardly 
>>compatible with Unix hierarchical semantics, ala 
>>www.namesys.com/future_vision.html
>>    
>>

Re: file as a directory

[Markus Törnqvist]

On Wed, Dec 15, 2004 at 08:57:27AM -0800, Hans Reiser wrote:

>www.namesys.com/future_vision.html

404.

http://www.namesys.com/whitepaper.html

-- 
mjt

Re: file as a directory

[mjt]

On Wed, Dec 15, 2004 at 08:57:27AM -0800, Hans Reiser wrote:

>www.namesys.com/future_vision.html

404.

http://www.namesys.com/whitepaper.html

-- 
mjt

Re: file as a directory

[Hans Reiser]

Markus Törnqvist wrote:

>On Wed, Dec 15, 2004 at 08:57:27AM -0800, Hans Reiser wrote:
>
>  
>
>>www.namesys.com/future_vision.html
>>    
>>
>
>404.
>
>http://www.namesys.com/whitepaper.html
>
>  
>
oops.  Thanks.

Re: file as a directory

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter Foldiak wrote:
| On Wed, 2004-12-15 at 04:47, David Masover wrote:
|
|>Peter Foldiak wrote:
|>| Also, a pseudofile (e.g. dirname/..../structure ?) could be used to
|>| specify how the files should be glued together. A simple question is,
|>| for instance, what separators to use between the components, and what
|>| ordering to use when putting the component objects together. (This
|>| pseudofile could also determine more complicated ways of composing
|>| objects.)
|>
|>If the filesystem does caching, I'd rather have a type of executable
|>which, read normally, appears as a stream of its own standard output.
|>You'd get the actual file as something like bar/.../source.
|
|
| This sounds better and more general than my proposed ..../structure
| file. So could you explain this in a bit more detail?
| Would for instance the simplest (and default?) glueing code in your
| bar/..../source file be
|
| cat *
|
| which just concatenates all the subcomponents in no particular order?

Sounds right.  Well, actually, you need a shebang, as this would be
called from kernel space.

Speaking of which, how much speed is lost by starting up a process?

The idea of caching is that running

cat *; cat *; cat *; cat *; cat *

is probably slower than

cat * > baz; cat baz; cat baz; cat baz; cat baz; cat baz

The kernel is the right place to do such caching.  It is not the right
place to implement almost any kind of "cool new kind of file" that Hans
wants us to develop.

|>This could be done with pipes and daemons, but it's not as easy to
|>manage and seems impossible to do as efficiently (with built-in caching,
|>etc.)
|
|
| How would you do it?

With two kernel reiser4 plugins -- one for files and one for
directories.  Others could come later, but I don't see this being so
important for things like symlinks.

Then, I'd let reiser4 cache things (in RAM and in free space on the
filesystem) which are generated through these plugins.

Directory plugins allow inheritence, too -- just have the plugin install
itself on all of its children.

Compression, encryption, etc. could all be done through the same mechanism.

[...]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQcDPO3gHNmZLgCUhAQIYhg//VeuJvEG2cbwANaka136xXDSy5blo71f0
ulgVBIe7XK8SMpLigNwVQQlTpmIncccu4ymOZG8egf4ex9d269Zv47TSmP+QkJEi
UU8Oky+oMFE/ntg5ZEoVxXe+oz2fCxzyYKwyjdWwHJZnylsHSaSsIpziaIZR+UXG
hcxZ9Cs3sLSlK0iYWM/R3dFBgRDMKR3/spz9gHGtyicLYbGvhzdHo6jSE1V8bqrg
44gt3XbB/SgKQTSeqBMr2/QCPpHKrFKqFhrsbVNzcjL7GftLOBeGRvVAWF9QIlUv
qDbIubfchr2KxYxE8qCsKTYqX5+VgDj0GDuwuymiq7fy4cIO0VIr3br1vJRWDUy3
VZeTbrQd3LJ6gSiTPN8+0CQQlpjEduS5EZfeIiygrQed1xHZBUbR59xV7KfIaHLz
lY06VKpwstmPjHueNx9T7T18aOjMVLWZbA+IWbRHdqLs0VqNltttVIz0e8WgFJHJ
3ISA0N+WNYCeR75XjWnD78e6vzhvTlGKLe2PjAdWDtuiSKTITV1HCKjSzMFL3JEd
CymOFhxddxx+aZrd8w1s0unjdGzkxdPfmpdd0XdBnB7vHYPpkw4ELacIqHt0OOTg
UYnBMem/UhLxisllfBOc0lunbxFynPOjsqYFTDTcHYHoZh0Kyel98m25uXeEr9VL
m2R6SMkl49k=
=oNG7
-----END PGP SIGNATURE-----

Re: file as a directory

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
[...]
| Explain the value of caching executable output more please.

If executables become "plugins" (see below), there is a performance loss
to be made up by caching.  For instance, if we have an executable which
calls "tar", which we attach to a tarball, we want to cache the
extracted files.  It's even more urgent when people start writing
perl/python/ruby executables to deal with system configuration files,
such as /etc/passwd.  It's allright for passwd to take half a second to
generate, so long as we cache that output, and thus only spend the half
second when passwd is modified, instead of every time a user logs in.
|
|>
|> | The component objects themselves could be full objects, so they
|> | themselves could have sub-components.
|>
|> Right.
|>
|> Also, there should be an inverse. For instance, a file-as-directory type
|> object should have a "contents" object, usually a normal directory, but
|> which could conceivably be any type of object, including a code-ish
|> object which implements a filesystem. Accessing foo/ would be the same
|> as accessing foo/.../contents, only because "..." (or whatever we use
|> for meta-files) is outside the actual directory namespace,
|> foo/.../contents/... refers to the metas of object "contents", which are
|> different than the metas of object "foo".
|
|
| Are you sure that having more than one "...." is needed?  Hmmm,
| interesting, must think about it.

Yes. foo is a file, foo/contents is another file (an executable).  But
maybe the executable has contents too -- with an extra "...", you can do
things like "foo/.../contents/.../contents", where the first "contents"
controls what you get from "ls foo/", and the second "contents" controls
what you get from "ls foo/.../contents/".

|>
|> These two steps essentially create userspace "plugins", and do away with
|> having to mount other kernel layers such as lufs (or whatever its
|> current implementation is).
|
|
| I don't follow this point above.

lufs is (or was) a kernel filesystem which talks to a user daemon.  The
daemon does all the work, so you don't have to do things like windows
emulation in the kernel in order to get captive-ntfs to work.

The reason we don't do that for the whole filesystem is performance --
after all, userland things are easier to write, debug, upgrade, and
install/use than kernel things.

If an executable can generate a directory listing or a whole directory
structure, some files, and so on, then we only need a few reiser4 kernel
plugins, and these userland executables can implement all the
functionality of most kernel plugins people have thought of so far.

The only problem is, it would be much slower than a kernel plugin.
Caching solves that.

Of course, someone still might find a situation where people really need
to create a new kernel plugin, and that would be allowed.  But I doubt
it would help that much, or we'd all be using TUX instead of Apache.

[...]

|> I think the issues with directory-as-a-file were the same problems you
|> get when you allow hardlinked directories -- that you'd eventually have
|> to ditch reference counting for a garbage collector, which is hellishly
|> impractical. I don't have a solution to that, other than dropping
|> hardlinks entirely.
|
|
| This issue is overblown.  Other fields have solved this problem.

Nevertheless, it's an issue that I don't have the skills to solve, and
one that must be solved if we are ever to implement these concepts.

If you have a solution, I'd love to hear it.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQcDT93gHNmZLgCUhAQLn6g/+KXD5StI9m9Tpij0i3uwqXAc+oAdz/dX5
YHSwU7YYXSdVPPqNqUNnU4t7i8M1p8Mba0Jpc2rPZk6Is67txBnphLX2F7c9jLm8
wfqfkjrfOM7/+5/EDQbdGVch0gZnuLJrxjGcpvukkwi8hkiMQnOtZuKuZwn1ieMW
UlTy5e/IhDPl5zW/Dy3rpoAOw7hszvPv6fjpFDujUENHhhUQmWYoiIbIaV3rdbhY
De+9AOkCRDEYNOmreCQQYmTkJD6w4C9cHzqmx7FrVx8vhxU3k9CPwoGRGGIrn42+
UPKAtUCAZu6Mya1v9v6QnPFaqnPgQatIJl4edg91bBzktpRZg6oR2iS9QnnAxc+b
LFJr95OKtdjBVVNuJGLRVCi06xoQsIjNWe5rOajAGHw/YHr7mHMZF315XKcp7FPa
F1df8FwdpQgq2xM6OdIfSFlBxk2poj21w5NqVyOZoEMunTyyDnK0YQkigsz2AQeN
lOeryopJ1sh9eHkQOsxOh/JdpYCVSkOfWiPgDKMst9mZbFF4yWwsot9JP1rNIUOC
EtGHdQzUOtKPkKWNy2+VXKxd8F+5EUpAtmOrdG2B7FT0wjS5Ao1pz6c7boMo1x9D
+9QxS2W1DdWjsCR9WhZhg0+eWc2wkf/dyVQIipf3JTm5vLOFJvJ/jfV3GRg0Ye7f
gtjTMcGxzAc=
=CIsH
-----END PGP SIGNATURE-----

Re: file as a directory

[Hans Reiser]

David Masover wrote:

>
>
> Speaking of which, how much speed is lost by starting up a process?
>
> The idea of caching is that running
>
> cat *; cat *; cat *; cat *; cat *
>
> is probably slower than
>
> cat * > baz; cat baz; cat baz; cat baz; cat baz; cat baz

Only for small files where the per file overhead of a read is significant.

Re: file as a directory

[Hans Reiser]

David Masover wrote:

> Hans Reiser wrote:
> [...]
> | Explain the value of caching executable output more please.
>
> If executables become "plugins" (see below), there is a performance loss
> to be made up by caching.  For instance, if we have an executable which
> calls "tar", which we attach to a tarball, we want to cache the
> extracted files.

This requires writing the plugin to have explicit caching of output.  It 
could be quite interesting actually.

>   It's even more urgent when people start writing
> perl/python/ruby executables to deal with system configuration files,
> such as /etc/passwd.  It's allright for passwd to take half a second to
> generate, so long as we cache that output, and thus only spend the half
> second when passwd is modified, instead of every time a user logs in.
> |
> |>
> |> | The component objects themselves could be full objects, so they
> |> | themselves could have sub-components.
> |>
> |> Right.
> |>
> |> Also, there should be an inverse. For instance, a file-as-directory 
> type
> |> object should have a "contents" object, usually a normal directory, but
> |> which could conceivably be any type of object, including a code-ish
> |> object which implements a filesystem. Accessing foo/ would be the same
> |> as accessing foo/.../contents, only because "..." (or whatever we use
> |> for meta-files) is outside the actual directory namespace,
> |> foo/.../contents/... refers to the metas of object "contents", 
> which are
> |> different than the metas of object "foo".
> |
> |
> | Are you sure that having more than one "...." is needed?  Hmmm,
> | interesting, must think about it.
>
> Yes. foo is a file, foo/contents is another file (an executable).  But
> maybe the executable has contents too -- with an extra "...", you can do
> things like "foo/.../contents/.../contents", where the first "contents"
> controls what you get from "ls foo/", and the second "contents" controls
> what you get from "ls foo/.../contents/".
>
> |>
> |> These two steps essentially create userspace "plugins", and do away 
> with
> |> having to mount other kernel layers such as lufs (or whatever its
> |> current implementation is).
> |
> |
> | I don't follow this point above.
>
> lufs is (or was) a kernel filesystem which talks to a user daemon.  The
> daemon does all the work, so you don't have to do things like windows
> emulation in the kernel in order to get captive-ntfs to work.
>
> The reason we don't do that for the whole filesystem is performance --
> after all, userland things are easier to write, debug, upgrade, and
> install/use than kernel things.
>
> If an executable can generate a directory listing or a whole directory
> structure, some files, and so on, then we only need a few reiser4 kernel
> plugins, and these userland executables can implement all the
> functionality of most kernel plugins people have thought of so far.
>
> The only problem is, it would be much slower than a kernel plugin.
> Caching solves that.
>
> Of course, someone still might find a situation where people really need
> to create a new kernel plugin, and that would be allowed.  But I doubt
> it would help that much, or we'd all be using TUX instead of Apache.
>
> [...]
>
> |> I think the issues with directory-as-a-file were the same problems you
> |> get when you allow hardlinked directories -- that you'd eventually have
> |> to ditch reference counting for a garbage collector, which is hellishly
> |> impractical. I don't have a solution to that, other than dropping
> |> hardlinks entirely.
> |
> |
> | This issue is overblown.  Other fields have solved this problem.
>
> Nevertheless, it's an issue that I don't have the skills to solve, and
> one that must be solved if we are ever to implement these concepts.
>
> If you have a solution, I'd love to hear it.

My solution is to tell Nate Diller that there is extensive literature on 
it, that it isn't really the big problem it is made out to be, and leave 
it to him to go read the literature and code it up after he finishes the 
required tasks of our current darpa contract.;-)

Re: file as a directory

[Peter Foldiak]

On Thu, 2004-12-16 at 18:48, Hans Reiser wrote:
> David Masover wrote:
> > Speaking of which, how much speed is lost by starting up a process?
> >
> > The idea of caching is that running
> >
> > cat *; cat *; cat *; cat *; cat *
> >
> > is probably slower than
> >
> > cat * > baz; cat baz; cat baz; cat baz; cat baz; cat baz
> 
> Only for small files where the per file overhead of a read is significant.

But if the glued "file" is a stream (or pipe?) you can't do everything
with it (e.g. seek() ) that you could do with a proper file, right?
You may want to do everything with it that you can do with a proper
file.

Re: file as a directory

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
[...]
| My solution is to tell Nate Diller that there is extensive literature on
| it, that it isn't really the big problem it is made out to be, and leave
| it to him to go read the literature and code it up after he finishes the
| required tasks of our current darpa contract.;-)

Can you point me to any such literature?  I'm just curious.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQcMCNHgHNmZLgCUhAQJJPA/+JFazzAVoF0sLhD1nemsYPAxzECyoat1D
sQC7QpAL4tlHZIBS9Ii4TiRtc6URYR2a28fMmllzh5uZLuCpUxW9piw/tsNNJVQQ
1hWbL5/CZaGGVSpNfPviByEJQ9Y/f6VPoRrzcKQl48lfstWC1tMACQnHUgruvlee
56Qg9lD50w6LSjKfet386mh3hNn7LAQ2SmYLxHJ33YkVsJLtu7naVh1/yrn+wXPR
YOj2rsckLnLCUoqei7JguOtZsZ4vlob6ArUxe30hwCQHOMGW5CMtnNE3rnSGHvj3
v4gQgpch6kasoBVdKXJGh3CkMkwFA871Ei0PtlRgs90ezBLT5FWWVOfg9Mn7eIi9
F9H7s0QXnIZTGA49otDZBTdq9MHs5/cqN1+E0j3GnxaW/z1VeLwu5/kjCc15kP3L
fqCKY+yiuUjaQLbodFtS8EzHHMcbRpWlb91qdCAJ6zQU46wQc+VnGKJ3upgHPw8A
SafuVkGgnsYU8nbY/KbjeIS3B8lIMmmzF0Ya2UN4Uvn552GnaDPy078UlOPLKSQU
EP+DOJ4LmH9ncHIcrdsyH0JxBIKs43XY8A77oeakyIfAQmE4kgLBdJMpHIh15vZ9
BXrqFOX2pREEwt5+jHpkzRAPkM0V85Dk5Fmj5XO0A5mMEhmGatUxvZSVlGBHRJ0r
zCnQBzLd5a4=
=jKLF
-----END PGP SIGNATURE-----

Re: file as a directory

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
| David Masover wrote:
|
|>
|>
|> Speaking of which, how much speed is lost by starting up a process?
|>
|> The idea of caching is that running
|>
|> cat *; cat *; cat *; cat *; cat *
|>
|> is probably slower than
|>
|> cat * > baz; cat baz; cat baz; cat baz; cat baz; cat baz
|
|
| Only for small files where the per file overhead of a read is significant.

That's potentially a common problem, and "cat *" is an overly-simplified
example.  Either you force the "plugin" to say whether it wants to be
cached or not, or you cache everything, because there are going to be
plugins like "tar -xjp" for which caching is a HUGE increase.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQcMDJHgHNmZLgCUhAQLDMxAAiW6qcBUDKepjkGGK30yyjpia7yzbRlI5
CLSwLnRFTX+oVoqIjYgjw6Cy+Y+kPAW8nL54Vnrvn/fjd05YV6SXIql/XqX5ZY1M
SXSD9LKnhxUEIH80Qcm8MdKtx0CrAPSIxVehpz7Clrv8pSClgtN2JIG1r4FsQHuB
E3Hasw11bHKZmg+NgYtNn1IDcgUxz2iTmw8xSwAlPtBZz8Enc4so4VVqZPOUYg+z
CTYOi84Qe8ecDGjhC6ZNvuEKsd8PgNl81MgEcJ5zcsnyCbNLpS2E8v1drsfVmXE5
8kbwelpc2q5ayEnNjeJgpIug8wH57cRWyc5KlAY733kZFTloG3LsnlWli8j9v0LT
sTcOtJMWa2zj7O7LE+1z/j6xMkjlPoT/QolMsAUv0SqloOihsfF3R71uolW3mrvM
VT8eQw5VUSHZyix/MqU0uDdvbOXsAoRvDL4xhdUAQv9/WbpGCNsB1PAiM693GFT2
A3VKiEcp0KMRUQZUxa5E1tBZh7WpwFvYkNTqxoN1fIDRKR6o/snAUDVb8UdTWxaW
T169coH6jS/4gEQVzsZ2PvWJ8gGpV5BjXM0C/04pgVLN4e610sJsoUXsha49wpob
bXgA8kw1wHp2C9fuA/NjNzyki8kzaTunkviu8ki/kaV7fEirVc0Prw78w7G2MkcC
j3enAzaZHVI=
=UFkt
-----END PGP SIGNATURE-----

Re: file as a directory

[Hans Reiser]

David Masover wrote:

> Hans Reiser wrote:
> [...]
> | My solution is to tell Nate Diller that there is extensive literature on
> | it, that it isn't really the big problem it is made out to be, and leave
> | it to him to go read the literature and code it up after he finishes the
> | required tasks of our current darpa contract.;-)
>
> Can you point me to any such literature?  I'm just curious.
>
Look in every field of cs except filesystems and kernels.;-)  Databases, 
garbage collectors, etc.  Specific reference, no, I didn't collect them, 
sorry, but alexander the befs driver guy knows more than I about this.

Re: file as a directory

[Hans Reiser]

David Masover wrote:

> Hans Reiser wrote:
> | David Masover wrote:
> |
> |>
> |>
> |> Speaking of which, how much speed is lost by starting up a process?
> |>
> |> The idea of caching is that running
> |>
> |> cat *; cat *; cat *; cat *; cat *
> |>
> |> is probably slower than
> |>
> |> cat * > baz; cat baz; cat baz; cat baz; cat baz; cat baz
> |
> |
> | Only for small files where the per file overhead of a read is 
> significant.
>
> That's potentially a common problem, and "cat *" is an overly-simplified
> example.  Either you force the "plugin" to say whether it wants to be
> cached or not, or you cache everything, because there are going to be
> plugins like "tar -xjp" for which caching is a HUGE increase.

Ok, I accept the argument.

Re: file as a directory

[Hans Reiser]

Peter Foldiak wrote:

>On Thu, 2004-12-16 at 18:48, Hans Reiser wrote:
>  
>
>>David Masover wrote:
>>    
>>
>>>Speaking of which, how much speed is lost by starting up a process?
>>>
>>>The idea of caching is that running
>>>
>>>cat *; cat *; cat *; cat *; cat *
>>>
>>>is probably slower than
>>>
>>>cat * > baz; cat baz; cat baz; cat baz; cat baz; cat baz
>>>      
>>>
>>Only for small files where the per file overhead of a read is significant.
>>    
>>
>
>But if the glued "file" is a stream (or pipe?) you can't do everything
>with it (e.g. seek() ) that you could do with a proper file, right?
>  
>
It does not need to be a pipe-like file.  Seek can be implemented for a 
composite (glued) file.

>You may want to do everything with it that you can do with a proper
>file.
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>

Re: file as a directory

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
| Peter Foldiak wrote:
|
|> On Thu, 2004-12-16 at 18:48, Hans Reiser wrote:
|>
|>
|>> David Masover wrote:
|>>
|>>
|>>> Speaking of which, how much speed is lost by starting up a process?
|>>>
|>>> The idea of caching is that running
|>>>
|>>> cat *; cat *; cat *; cat *; cat *
|>>>
|>>> is probably slower than
|>>>
|>>> cat * > baz; cat baz; cat baz; cat baz; cat baz; cat baz
|>>>
|>>
|>> Only for small files where the per file overhead of a read is
|>> significant.
|>>
|>
|>
|> But if the glued "file" is a stream (or pipe?) you can't do everything
|> with it (e.g. seek() ) that you could do with a proper file, right?
|>
|>
| It does not need to be a pipe-like file.  Seek can be implemented for a
| composite (glued) file.

Composite plugins are likely fairly simple (cat *) so they shouldn't be
required to implement seek on their own.  In OO terms, they should
"inherit" a more global facility, which would be based on cached pipes.

But definitely for files that are likely to be large, but still seekable
(zipfiles), you want the plugin to provide its own seek support.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQcN3zXgHNmZLgCUhAQKF1w/7B51wfpaIVi6ap+TVsZ+t3FLPTqyOF24W
lnqSUnM50zlJixQMKLzmZSOUGC/EsDZIVqShAUhniKAp8cvYNCE0HjqZl1b8S0HV
MggehXJJwSfl3/RHCbRXo1URXzSodGLyCsdEhi88viyhilj7+9uJRhxmOx1cO7uS
p/yS3LgpxEEjhDiu09MXnAHuXGTdKB9hQl63HxPwa72uM3Q5M7zCl0p+X+rH9xIj
HgdZMbmoCiA6QmRJzLBq/s0xRo385bfeTHMCwB3aIyYNFeTvJDoZP6IEbi9Ee0DH
Ns9rwJo+B2FPbHKHlmFxZkLVwmnQqr9WemYYybUGNOVdIamn87Ah6/ObvAmM+dQw
dnRF2YVPM9gNv0LSOS3nteNr6ldYzEZWGi24ScPpFrJpZAMBVNarX0zE4+S2Dc75
a733Z2pkOhHUvRPoDVXvUTKC5863cuWH7lgvTfT5eAANG/AANphi2vyFfgT5KWKt
gF+SnRkLiDSfjcqgts036vfpIziz/aM9dZxzgE+j8IjuGNchl/6q1OVt7+WOJOrq
grfOgVB26pFgYFD7ZjJSiZKHGRDJZXrUo4VnUoG43Dy4JZVlG4qpivMO50Ep0GD2
cVFw1/MKP0beAA1S1R5Fh0mLo0jNHlsxSD6XoKjUuL/xjnLpyN/AdmIhQq0mMujY
vYBBaDHYkYc=
=RMFf
-----END PGP SIGNATURE-----

Re: file as a directory

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> David Masover wrote:
> > Hans Reiser wrote:
> > [...]
> > | My solution is to tell Nate Diller that there is extensive literature on
> > | it, that it isn't really the big problem it is made out to be, and leave
> > | it to him to go read the literature and code it up after he finishes the
> > | required tasks of our current darpa contract.;-)

> > Can you point me to any such literature?  I'm just curious.

> Look in every field of cs except filesystems and kernels.;-)

Right. Smart people are found elsewhere only.

>                                                              Databases, 
> garbage collectors, etc.

Everthing stuff that works on the assumption that what they are working on
fits in RAM (or can overflow into swap space in a pinch), and that RAM is
fast (and even so they are infuriatingly slow). And disks are usually a few
thousand times larger than RAM (more stuff to shuffle around) and a million
times slower...

>                          Specific reference, no, I didn't collect them, 
> sorry, but alexander the befs driver guy knows more than I about this.

Furious handwaving doesn't make it true.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: file as a directory

[Hans Reiser]

Horst von Brand wrote:

>
>>>Can you point me to any such literature?  I'm just curious.
>>>      
>>>
>
>  
>
>>Look in every field of cs except filesystems and kernels.;-)
>>    
>>
>
>Right. Smart people are found elsewhere only.
>  
>
There is a huge pile of innovations by the database field that 
filesystems people do not yet make use of.  The merits of 
interdisciplinary study are understood by many.

>  
>
>>                                                             Databases, 
>>garbage collectors, etc.
>>    
>>
>
>Everthing stuff that works on the assumption that what they are working on
>fits in RAM (or can overflow into swap space in a pinch), and that RAM is
>fast (and even so they are infuriatingly slow). And disks are usually a few
>thousand times larger than RAM (more stuff to shuffle around) and a million
>times slower...
>
>  
>
>>                         Specific reference, no, I didn't collect them, 
>>sorry, but alexander the befs driver guy knows more than I about this.
>>    
>>
>
>Furious handwaving doesn't make it true.
>  
>
Ok, go talk to the befs driver guy, and you'll find out he has already 
done work on it.

If his work does not satisfy you, sorry, I do detailed research just 
before coding, not to satisfy some guy on lkml.

Re: file as a directory

[Alexander G. M. Smith]

Hans Reiser wrote on Mon, 20 Dec 2004 09:21:35 -0800:
> Ok, go talk to the befs driver guy, and you'll find out he has already 
> done work on it.

I just did some work in making a test file system with hard links and
bidirectional references (files know which directories (plural) they are in).
It uses a simple graph traversal upon delete to do garbage collection.
Compared to the sophisticated garbage collection algorithms I was reading
about at university a decade ago, it's nothing worth writing a paper about!
But still, there was enough new stuff to make it worth writing a text file:
http://web.ncf.ca/au829/BeOS/AGMSLinkExplanation.txt

And if you want, you can read the excessively commented source code,
including a large passage explaining the delete/rename algorithm, at:
http://web.ncf.ca/au829/BeOS/AGMSRAMFileSystem20040403.zip

Horst wrote:
>> Everthing stuff that works on the assumption that what they are working
>> on fits in RAM (or can overflow into swap space in a pinch), [...]

Admittedly my experiment was for a RAM disk.  It can be extended to a real
disk if the number of inodes in a graph traversal can fit into memory (or
otherwise can somehow be locked).  If that doesn't work, then the user
would have to delete individual files before removing a directory cycle.

- Alex

Re: file as a directory

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander G. M. Smith wrote:
[...]
|>>Everthing stuff that works on the assumption that what they are working
|>>on fits in RAM (or can overflow into swap space in a pinch), [...]
|
|
| Admittedly my experiment was for a RAM disk.  It can be extended to a real
| disk if the number of inodes in a graph traversal can fit into memory (or
| otherwise can somehow be locked).  If that doesn't work, then the user
| would have to delete individual files before removing a directory cycle.

Would it work in a typical setup?  My instinct is no.

Also, there's the consideration that applications want some memory, too.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQce1SHgHNmZLgCUhAQJ8Jg//VhW+qzwb12UN85JVDnxmN3Xzi4Dfyhay
Uvw2668X8iTbm2MWt10o0xOThdNcUM2tfuzCQi+qFx3pbYpDKFyoE3PdjmvYC0iA
MuHgb0GFULqCF5/DQT3mrLTrNO7P5njA6agVcHkcg6v1Uciuwjt+tDDMHNJLAe+o
wc7y02sA0i6XnmORjv4OK+KYJTnPl2bm9FtVq4RTYYG3bro823sxgzor9ldkuINE
6NPU0dx6ldp3g1yomn03wG03KciOqj5F5i1iILjOGwUhxigTAqSoJjyPbpA78Rt+
d3xA8i1Lqjbrow5lBVEdipNZYLTFniNIdW+E5+xXtzVMbOkVtnFansI66IMgr1SA
Y3OscmmT4rgdrKXxjY69Rib6bmfm3eDhaJsvPi6SRWuo9aH3yg/46TuBdj+7ZNnq
bAWanB+J3xdDXXTsjkRFOXXIDPCzAKEImDbp/PGKrpgk+Zuil7P3D8PUirCrKmv+
DiETgm8aKEU4eLsd8dhPu/NcOkXFH+zl2l2hNi8rLALSYnEFfbpo8wkXILym3Omb
CYJ9hw6TPPxABJZ927iNymvoUUf5Pm5i0w5N+GuDjYshew/feuKEKV+NjJ5X4e12
I68fPHO8VrdyGFRfX2i6ABVkCnGbjOWplpRY1JhGoPpBIMukvxbl7WJNbkT4u6pB
hUgGpPqWPy0=
=gKJo
-----END PGP SIGNATURE-----

Re: file as a directory

[Alexander G. M. Smith]

David Masover wrote on Mon, 20 Dec 2004 23:31:53 -0600:
> Would it work in a typical setup?  My instinct is no.

Depends on how big the directory loop and attached files are (basically all
the descendants of the directory you're trying to delete).  In most cases,
you don't have millions of directories in a loop.  With millions of files,
it might run out of RAM to keep track of locked inodes, making it impossible
to delete a directory in a loop that somewhere has millions of files as
descendants.  The awkward user work-around would be to move the files
somewhere else (files don't have descendants so that's feasible), delete
the directory, then move them back.

Of course, in a RAM disk, there can't be that many files :-)

- Alex

Re: file as a directory

[Horst von Brand]

"Alexander G. M. Smith" <agmsmith@rogers.com> said:
> Hans Reiser wrote on Mon, 20 Dec 2004 09:21:35 -0800:
> > Ok, go talk to the befs driver guy, and you'll find out he has already 
> > done work on it.

> I just did some work in making a test file system with hard links and
> bidirectional references (files know which directories (plural) they are in).
> It uses a simple graph traversal upon delete to do garbage collection.
> Compared to the sophisticated garbage collection algorithms I was reading
> about at university a decade ago, it's nothing worth writing a paper about!
> But still, there was enough new stuff to make it worth writing a text file:
> http://web.ncf.ca/au829/BeOS/AGMSLinkExplanation.txt

I.e., Stop the world while we traverse the whole thing to find out what (if
something) can be deleted now? An inode has a _huge_ size (or has a list of
parents hanging off it)?

> And if you want, you can read the excessively commented source code,
> including a large passage explaining the delete/rename algorithm, at:
> http://web.ncf.ca/au829/BeOS/AGMSRAMFileSystem20040403.zip

> Horst wrote:
> >> Everthing stuff that works on the assumption that what they are working
> >> on fits in RAM (or can overflow into swap space in a pinch), [...]

> Admittedly my experiment was for a RAM disk.  It can be extended to a real
> disk if the number of inodes in a graph traversal can fit into memory (or
> otherwise can somehow be locked).

Can't guarantee they fit into RAM; can't lock everything down in a _huge_
disk being used by lots of processes (the system would grind to a halt).

>                                     If that doesn't work, then the user
> would have to delete individual files before removing a directory cycle.

How does the user know that this is a directory cycle being removed? Also,
while I nuke a particular link into the cycle, somebody else could be
creating a new one...

Nice prototype. Can we now go back to real filesystems?
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: file as a directory

[Alexander G. M. Smith]

Horst von Brand wrote on Tue, 21 Dec 2004 13:29:46 -0300:
> How does the user know that this is a directory cycle being removed? Also,
> while I nuke a particular link into the cycle, somebody else could be
> creating a new one...

The error code would tell you.  If someone else is adding links while you're
trying to delete then the locking will either lock them out, or wait a moment
for them to finish, or fail if a deadlock happened.

> Nice prototype. Can we now go back to real filesystems?

I hope so!  Still, the prototype does help point out problems and possible
solutions - exploring the dark areas of file system knowledge.  Plus it's
useful for speeding up certain operations on BeOS, which doesn't have as
good disk caching (fixed 32MB cache size) as Linux.

- Alex

Re: file as a directory

[Peter Foldiak]

Back in November 2004, I suggested on the linux-kernel and reiserfs
lists that the Reiser4 architecture could allow us to abolish the
unnatural naming distinction between directories/files/parts-of-file
(i.e. to unify naming within-file-system and within-file naming) in an
efficient way.
I suggested that one way of doing that would be to extend XPath-like
selection syntax above the (XML) file level.
(See the archive of the discussion starting at
http://www.ussg.iu.edu/hypermail/linux/kernel/0411.3/0044.html
Wed Nov 24 2004 - 04:21:13 EST.)

ITworld now has an interesting article by Sean McGrath on a very similar
idea, mentioning the XML OASIS Open Document Format. What do you think?

 Peter Foldiak

Here it is:

--

ITworld

http://www.itworld.com/AppDev/1246/nls_ebizbooks050510/

Books/chapters and directories/files - dichotomies considered harmful
ITworld.com, Ebusiness in the Enterprise 5/9/05

Sean McGrath, ITworld.com

The distinction between a full book and a mere chapter of a book, is a
source of endless fascination for incurable information modellers like
me.

Obviously, at the logical level, the distinction is driven by the
content itself. A book is a complete unit of stuff. A chapter, is a
sub-division within the complete book. At the physical level, however,
technology starts to influence the book/chapter distinction. A chapter
boundary, for Microsoft Word users or Open Office users, is likely to be
influenced by how big the underlying file gets. Large files take longer
to load and get increasingly slower to work with in typical word
processing environments. Our decisions about where to draw the chapter
boundaries are influenced to some extent by technology limitations.

If the physical constraints are not allowed to dictate the boundaries
for chapters, then we can end up resorting to file naming conventions to
split the content into manageable chunks e.g. chapter1_a, chapter1_b and
so on. We might then decide to keep things clean by introducing a
subdirectory for each chapter, putting the sub-chapters tidily away in
their own little compartments.

All is well with the world. Or is it? This is where things get
interesting from an information management perspective. A full unit of
work - a book - has now been split into bits that are navigable through
a directory structure and bits that are navigable through an
application. The result? You can use off-the-shelf tools to navigate
your way through the directories. You can see the overall structure of
the book by simply looking at the directory structure as a hierarchy.
You can see that chapter 1 has a number of sub-chapters. However, that
is as far as you can go. To dig any further into the structure of
chapter 1, section A, you need to launch the editing application.

What a pity.

Why is it, that we have this hard and fast dichotomy between directory
structure and file structure? Why is it that file system exploring
utilities need to stop in their tracks when they hit things called
'files'?

As you have probably noticed, this artificial split can be breached in
certain circumstances, at least to some extent. Graphics file formats
are a good example. Many file system exploring tools know about, say,
JPEG files and can display thumbnails of their contents.

That is a start in the right direction but I think it needs to go a lot
further if the artificial directory/file distinction is to be
eradicated.

Let us go back to the book example. Let us use Microsoft's OLE
technology as an analogy. With OLE you can embed one thing in another.
So for example, you can embed an Excel spreadsheet into a Word document
file. Now, in your head, take that further. Imagine a world in which the
file system explorer is the top level application. It manages a single,
humungous file on the disk into which you embed documents, spreadsheets,
databases etc. Each think you embed into the explorer can itself embed
other things to any depth required.

In such a world, directories/files have merged into one abstraction. The
book author does not have to introduce artificial segmentation of the
book into separate entities. In such a world, filenames become something
of an oddity. What do you need filenames for? You would only really need
a filename at the point where you decided to exchange information
between systems A and B.

Moreover, once the package of data is pasted into System B's file system
explorer at some suitable point, the filename would be thrown away.

Sounds interesting wouldn't you say? So why don't we have systems that
work like that? There are, as ever, many reasons. One reason which was
an issue some years ago, is ceasing to be an issue very quickly now.
Obviously, in order to show the structure of a "file" a file system
explorer needs to look inside the file format. If the file format is
proprietary, then we can do nothing.

Enter XML-based file formats like the OASIS Open Document Format[1]. The
day is coming when file system explorers will be able to do for office
documents, what they currently do for JPEGs. That is a start in the
right direction. Eventually, I hope we will see the directory/file
distinction begin to melt away.

Technologies/applications that never quite made it to the mainstream
such as OpenDoc[2] and FrameMaker[3] with its powerful Book/Chapter
model, may yet have a second coming.

[1] http://www.oasis-open.org/committees/office/charter.php
[2] http://www.webopedia.com/TERM/O/OpenDoc.html
[3] http://www.adobe.com/products/framemaker/main.html

Sean McGrath is CTO of Propylon. He is an internationally acknowledged
authority on XML and related standards. He served as an invited expert
to the W3C's Expert Group that defined XML in 1998. He is the author of
three books on markup languages published by Prentice Hall. Visit his
site at: http://seanmcgrath.blogspot.com.

Re: file as a directory

[Hans Reiser]

I agree with the below in that sometimes you want to see a collection of
stuff as one file, and sometimes you want to see it as a tree, and that
file format browsers can be integrated into file system browsers to look
seamless to users.

A quibble: A name is just a means to select a file; he is completely
wrong to think that file browsers will eliminate filenames.

Hans

Peter Foldiak wrote:

>Back in November 2004, I suggested on the linux-kernel and reiserfs
>lists that the Reiser4 architecture could allow us to abolish the
>unnatural naming distinction between directories/files/parts-of-file
>(i.e. to unify naming within-file-system and within-file naming) in an
>efficient way.
>I suggested that one way of doing that would be to extend XPath-like
>selection syntax above the (XML) file level.
>(See the archive of the discussion starting at
>http://www.ussg.iu.edu/hypermail/linux/kernel/0411.3/0044.html
>Wed Nov 24 2004 - 04:21:13 EST.)
>
>ITworld now has an interesting article by Sean McGrath on a very similar
>idea, mentioning the XML OASIS Open Document Format. What do you think?
>
> Peter Foldiak
>
>Here it is:
>
>--
>
>ITworld
>
>http://www.itworld.com/AppDev/1246/nls_ebizbooks050510/
>
>Books/chapters and directories/files - dichotomies considered harmful
>ITworld.com, Ebusiness in the Enterprise 5/9/05
>
>Sean McGrath, ITworld.com
>
>The distinction between a full book and a mere chapter of a book, is a
>source of endless fascination for incurable information modellers like
>me.
>
>Obviously, at the logical level, the distinction is driven by the
>content itself. A book is a complete unit of stuff. A chapter, is a
>sub-division within the complete book. At the physical level, however,
>technology starts to influence the book/chapter distinction. A chapter
>boundary, for Microsoft Word users or Open Office users, is likely to be
>influenced by how big the underlying file gets. Large files take longer
>to load and get increasingly slower to work with in typical word
>processing environments. Our decisions about where to draw the chapter
>boundaries are influenced to some extent by technology limitations.
>
>If the physical constraints are not allowed to dictate the boundaries
>for chapters, then we can end up resorting to file naming conventions to
>split the content into manageable chunks e.g. chapter1_a, chapter1_b and
>so on. We might then decide to keep things clean by introducing a
>subdirectory for each chapter, putting the sub-chapters tidily away in
>their own little compartments.
>
>All is well with the world. Or is it? This is where things get
>interesting from an information management perspective. A full unit of
>work - a book - has now been split into bits that are navigable through
>a directory structure and bits that are navigable through an
>application. The result? You can use off-the-shelf tools to navigate
>your way through the directories. You can see the overall structure of
>the book by simply looking at the directory structure as a hierarchy.
>You can see that chapter 1 has a number of sub-chapters. However, that
>is as far as you can go. To dig any further into the structure of
>chapter 1, section A, you need to launch the editing application.
>
>What a pity.
>
>Why is it, that we have this hard and fast dichotomy between directory
>structure and file structure? Why is it that file system exploring
>utilities need to stop in their tracks when they hit things called
>'files'?
>
>As you have probably noticed, this artificial split can be breached in
>certain circumstances, at least to some extent. Graphics file formats
>are a good example. Many file system exploring tools know about, say,
>JPEG files and can display thumbnails of their contents.
>
>That is a start in the right direction but I think it needs to go a lot
>further if the artificial directory/file distinction is to be
>eradicated.
>
>Let us go back to the book example. Let us use Microsoft's OLE
>technology as an analogy. With OLE you can embed one thing in another.
>So for example, you can embed an Excel spreadsheet into a Word document
>file. Now, in your head, take that further. Imagine a world in which the
>file system explorer is the top level application. It manages a single,
>humungous file on the disk into which you embed documents, spreadsheets,
>databases etc. Each think you embed into the explorer can itself embed
>other things to any depth required.
>
>In such a world, directories/files have merged into one abstraction. The
>book author does not have to introduce artificial segmentation of the
>book into separate entities. In such a world, filenames become something
>of an oddity. What do you need filenames for? You would only really need
>a filename at the point where you decided to exchange information
>between systems A and B.
>
>Moreover, once the package of data is pasted into System B's file system
>explorer at some suitable point, the filename would be thrown away.
>
>Sounds interesting wouldn't you say? So why don't we have systems that
>work like that? There are, as ever, many reasons. One reason which was
>an issue some years ago, is ceasing to be an issue very quickly now.
>Obviously, in order to show the structure of a "file" a file system
>explorer needs to look inside the file format. If the file format is
>proprietary, then we can do nothing.
>
>Enter XML-based file formats like the OASIS Open Document Format[1]. The
>day is coming when file system explorers will be able to do for office
>documents, what they currently do for JPEGs. That is a start in the
>right direction. Eventually, I hope we will see the directory/file
>distinction begin to melt away.
>
>Technologies/applications that never quite made it to the mainstream
>such as OpenDoc[2] and FrameMaker[3] with its powerful Book/Chapter
>model, may yet have a second coming.
>
>[1] http://www.oasis-open.org/committees/office/charter.php
>[2] http://www.webopedia.com/TERM/O/OpenDoc.html
>[3] http://www.adobe.com/products/framemaker/main.html
>
>Sean McGrath is CTO of Propylon. He is an internationally acknowledged
>authority on XML and related standards. He served as an invited expert
>to the W3C's Expert Group that defined XML in 1998. He is the author of
>three books on markup languages published by Prentice Hall. Visit his
>site at: http://seanmcgrath.blogspot.com.
>
>
>
>
>  
>

Re: file as a directory

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

On Tue, 10 May 2005 10:39:23 BST, Peter Foldiak said:
> Back in November 2004, I suggested on the linux-kernel and reiserfs
> lists that the Reiser4 architecture could allow us to abolish the
> unnatural naming distinction between directories/files/parts-of-file
> (i.e. to unify naming within-file-system and within-file naming) in an
> efficient way.
> I suggested that one way of doing that would be to extend XPath-like
> selection syntax above the (XML) file level.

I believe the consensus was that this needs to happen at the VFS layer, not
the FS level.  The next step would be designing an API for this - what would
the VFS present to userspace, and in what way, and how would backward
combatability be maintained?

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: file as a directory

[Peter Foldiak]

On Tue, 2005-05-10 at 15:53, Hans Reiser wrote:
> I agree with the below in that sometimes you want to see a collection of
> stuff as one file, and sometimes you want to see it as a tree, and that
> file format browsers can be integrated into file system browsers to look
> seamless to users.
> 
> A quibble: A name is just a means to select a file; he is completely
> wrong to think that file browsers will eliminate filenames.

Yes, even if you think of the whole file system as a single "file", you
need a way to select the bit you need, and you will use names for that
(and whether you call that a filename, a file-part name or an object
name doesn't really matter).

It is interesting that both he and I gave the example of a book and
chapters, which is essentially a linear sequence, and the issue was just
the selection of a part of that sequence. It would also be interesting
to think about how you could map an arbitrary data structure more
complicated than a linear sequence (an "object") to disk. This brings up
issues of serialization and object databases....

Re: file as a directory

[Peter Foldiak]

On Tue, 2005-05-10 at 16:14, Valdis.Kletnieks@vt.edu wrote:
> On Tue, 10 May 2005 10:39:23 BST, Peter Foldiak said:
> > Back in November 2004, I suggested on the linux-kernel and reiserfs
> > lists that the Reiser4 architecture could allow us to abolish the
> > unnatural naming distinction between directories/files/parts-of-file
> > (i.e. to unify naming within-file-system and within-file naming) in an
> > efficient way.
> > I suggested that one way of doing that would be to extend XPath-like
> > selection syntax above the (XML) file level.
> 
> I believe the consensus was that this needs to happen at the VFS layer, not
> the FS level.  The next step would be designing an API for this - what would
> the VFS present to userspace, and in what way, and how would backward
> combatability be maintained?

But can it be done efficiently above the file system level??

As far as I understand, Reiser4 has this nice tree structure, which
means that the part of file selection could be done with almost no extra
effort, you just attach additional names to inside nodes of the tree, so
the same tree can be used to store the whole object, and part of the
same tree can be used to select the object part. Right?
If you do this above the file system level, I don't think it would have
such an efficient implementation. Or would it?  Peter

Re: file as a directory

[Sean McGrath]

Peter Foldiak wrote:

>On Tue, 2005-05-10 at 15:53, Hans Reiser wrote:
>  
>
>>I agree with the below in that sometimes you want to see a collection of
>>stuff as one file, and sometimes you want to see it as a tree, and that
>>file format browsers can be integrated into file system browsers to look
>>seamless to users.
>>
>>A quibble: A name is just a means to select a file; he is completely
>>wrong to think that file browsers will eliminate filenames.
>>    
>>
>
>Yes, even if you think of the whole file system as a single "file", you
>need a way to select the bit you need, and you will use names for that
>(and whether you call that a filename, a file-part name or an object
>name doesn't really matter).
>  
>
The thing that interests me most is the difference (if any) between 
giving a stream of bytes an opaque name e.g. "Chapter 1 of my book.sxw" 
versus giving a stream of bytes a query expression that can also be 
considered an opaque name e.g.
"/book/chapter[1] "

This is what the Russell/Frege descriptive theory of proper names 
applied to storage systems in a sense[1].

I've written about this stuff before on ITWorld (warning: chatty prose 
style ahead):

    Fractals, Self Similarity, and the Whimsical Boundaries of XML Documents
    http://www.itworld.com/nl/xml_prac/04252002/

    A study in XML culture and evolution
    http://www.itworld.com/nl/ebiz_ent/03252003/

[1] http://en.wikipedia.org/wiki/Proper_name

Sean

Re: file as a directory

[Hans Reiser]

Peter Foldiak wrote:

>On Tue, 2005-05-10 at 16:14, Valdis.Kletnieks@vt.edu wrote:
>  
>
>>On Tue, 10 May 2005 10:39:23 BST, Peter Foldiak said:
>>    
>>
>>>Back in November 2004, I suggested on the linux-kernel and reiserfs
>>>lists that the Reiser4 architecture could allow us to abolish the
>>>unnatural naming distinction between directories/files/parts-of-file
>>>(i.e. to unify naming within-file-system and within-file naming) in an
>>>efficient way.
>>>I suggested that one way of doing that would be to extend XPath-like
>>>selection syntax above the (XML) file level.
>>>      
>>>
>>I believe the consensus was that this needs to happen at the VFS layer, not
>>the FS level.  The next step would be designing an API for this - what would
>>the VFS present to userspace, and in what way, and how would backward
>>combatability be maintained?
>>    
>>
>
>But can it be done efficiently above the file system level??
>
>As far as I understand, Reiser4 has this nice tree structure, which
>means that the part of file selection could be done with almost no extra
>effort, you just attach additional names to inside nodes of the tree, so
>the same tree can be used to store the whole object, and part of the
>same tree can be used to select the object part. Right?
>If you do this above the file system level, I don't think it would have
>such an efficient implementation. Or would it?  Peter
>  
>
The tree structure Peter speaks of is a storage layer entity, and so I
think Peter's argument is not correct, but what Reiser4 also has is a
plugin architecture, and it would be much easier to code it if we use
the plugin architecture.

Hans

Re: file as a directory

[Hans Reiser]

Sean McGrath wrote:

> The thing that interests me most is the difference (if any) between
> giving a stream of bytes an opaque name e.g. "Chapter 1 of my
> book.sxw" versus giving a stream of bytes a query expression that can
> also be considered an opaque name e.g.
> "/book/chapter[1] "
>
What is an opaque name?

Re: file as a directory

[Sean McGrath]

Hans Reiser wrote:

>Sean McGrath wrote:
>
>  
>
>>The thing that interests me most is the difference (if any) between
>>giving a stream of bytes an opaque name e.g. "Chapter 1 of my
>>book.sxw" versus giving a stream of bytes a query expression that can
>>also be considered an opaque name e.g.
>>"/book/chapter[1] "
>>
>>    
>>
>What is an opaque name?
>
>
>  
>
By "opaque name" I mean a name that is purely a label. A name that 
cannot be interpreted as a query expression.

Sean

Re: file as a directory

[Hans Reiser]

Sean McGrath wrote:

> Hans Reiser wrote:
>
>> Sean McGrath wrote:
>>
>>  
>>
>>> The thing that interests me most is the difference (if any) between
>>> giving a stream of bytes an opaque name e.g. "Chapter 1 of my
>>> book.sxw" versus giving a stream of bytes a query expression that can
>>> also be considered an opaque name e.g.
>>> "/book/chapter[1] "
>>>
>>>   
>>
>> What is an opaque name?
>>
>>
>>  
>>
> By "opaque name" I mean a name that is purely a label. A name that
> cannot be interpreted as a query expression.

Isn't query just another name for name?

>
> Sean
>
>
>
>
>

Re: file as a directory

[Sean McGrath]

Hans Reiser wrote:

>Sean McGrath wrote:
>  
>
>>Hans Reiser wrote:
>>    
>>
>>>Sean McGrath wrote: 
>>>      
>>>
>>>>The thing that interests me most is the difference (if any) between
>>>>giving a stream of bytes an opaque name e.g. "Chapter 1 of my
>>>>book.sxw" versus giving a stream of bytes a query expression that can
>>>>also be considered an opaque name e.g.
>>>>"/book/chapter[1] "
>>>>
>>>>  
>>>>        
>>>>
>>>What is an opaque name?
>>>
>>>
>>> 
>>>
>>>      
>>>
>>By "opaque name" I mean a name that is purely a label. A name that
>>cannot be interpreted as a query expression.
>>    
>>
>
>Isn't query just another name for name?
>
>  
>
That is a major philosophical nugget :-)

I recommend Saul Kripke's Naming and Necessity:
    http://www.answers.com/topic/saul-kripke

Sean

Re: file as a directory

[Hans Reiser]

Sean McGrath wrote:

>
>>>> What is an opaque name?
>>>>
>>>>
>>>>
>>>>
>>>>     
>>>
>>> By "opaque name" I mean a name that is purely a label. A name that
>>> cannot be interpreted as a query expression.
>>>   
>>
>>
>> Isn't query just another name for name?
>>
>>  
>>
> That is a major philosophical nugget :-)
>
> I recommend Saul Kripke's Naming and Necessity:
>    http://www.answers.com/topic/saul-kripke
>
> Sean
>
>
>
>
I suggest considering your opaque names to be what reiser4 calls "keys",
that is, names that exist for the purpose of finding the object via the
storage layer.

Hans

Re: file as a directory

[Helge Hafting]

On Tue, May 10, 2005 at 04:38:48PM +0100, Peter Foldiak wrote:
> On Tue, 2005-05-10 at 16:14, Valdis.Kletnieks@vt.edu wrote:
> > On Tue, 10 May 2005 10:39:23 BST, Peter Foldiak said:
> > > Back in November 2004, I suggested on the linux-kernel and reiserfs
> > > lists that the Reiser4 architecture could allow us to abolish the
> > > unnatural naming distinction between directories/files/parts-of-file
> > > (i.e. to unify naming within-file-system and within-file naming) in an
> > > efficient way.
> > > I suggested that one way of doing that would be to extend XPath-like
> > > selection syntax above the (XML) file level.
> > 
> > I believe the consensus was that this needs to happen at the VFS layer, not
> > the FS level.  The next step would be designing an API for this - what would
> > the VFS present to userspace, and in what way, and how would backward
> > combatability be maintained?
> 
> But can it be done efficiently above the file system level??
> 
Anything that can be done at the fs level should be doable on the vfs level too.
That is simple to show in theory: You could make the VFS api identical to
the reiser4 api, and reiser4 should continue to work as efficiently as before.

> As far as I understand, Reiser4 has this nice tree structure, which
> means that the part of file selection could be done with almost no extra
> effort, you just attach additional names to inside nodes of the tree, so
> the same tree can be used to store the whole object, and part of the
> same tree can be used to select the object part. Right?
> If you do this above the file system level, I don't think it would have
> such an efficient implementation. Or would it?  Peter

I cannot see why reiser4 should suffer - but of course this might be hard to
implement for other filesystems.

Helge Hafting

Re: file as a directory

[Leo Comerford]

On 5/10/05, Peter Foldiak <Peter.Foldiak@st-andrews.ac.uk> wrote:
> On Tue, 2005-05-10 at 15:53, Hans Reiser wrote:
> > I agree with the below in that sometimes you want to see a collection of
> > stuff as one file, and sometimes you want to see it as a tree, and that
> > file format browsers can be integrated into file system browsers to look
> > seamless to users.
> >
> > A quibble: A name is just a means to select a file; he is completely
> > wrong to think that file browsers will eliminate filenames.
> 
> Yes, even if you think of the whole file system as a single "file", you
> need a way to select the bit you need, and you will use names for that
> (and whether you call that a filename, a file-part name or an object
> name doesn't really matter).
> 
> It is interesting that both he and I gave the example of a book and
> chapters, which is essentially a linear sequence, and the issue was just
> the selection of a part of that sequence. It would also be interesting
> to think about how you could map an arbitrary data structure more
> complicated than a linear sequence (an "object") to disk. This brings up
> issues of serialization and object databases....
> 
> 
Here's how you might go about it.

First, some necesary background. Some (not all) of this I've mentioned
before, mainly on reiserfs-list. I've marked the start of the
"compound object"-related material below.

The fundamental problem with reiser4-style metas is semantic. The core
Unix idiom is that pathnames assert predicates of non-directory files.
So for example "/etc/passwd" should be interpreted as "the file with
inode x is the password file", while "/bin" asserts "this is a binary
executable" of all of its non-directory descendants. In OO terms we
can see these as isAs: '/bin/touch' asserts that the file with that
name isA touch binary and, transitively, isA binary executable
('/bin'). (So pathnames from root are really better not thought of as
names at all, but never mind that now.)

Now in fact this idiom isn't as widely or consistently applied in *nix
as it could be if certain technical barriers were removed. But it's
strong enough to allow us to say that in general *nix directories
exist to provide metadata (in the form of pathnames) about their
non-directory descendants.

Under reiser4, we have two different metadata systems cohabiting in
the same namespace but remaining distinct. (If I want to say something
about a file, I might make a link to it, or I might put something in
its .... directory. Or maybe both.) One is the old Unix pathname
system, designed for finding a file given certain metadata (which is
the password file?). The other is metas, a replacement for stat() and
friends which like them is designed for finding some metadata for a
given file (who owns this file?). It uses the syntax and the API of
Unix filenames but a completely contrary semantics involving
non-directory files which exist to provide metadata about their parent
directories. Many of the difficulties of implementing metas are simply
symptoms of the underlying reality that metas are not namespace
integration but namespace duct-taping.

Now I think Hans is dead right when he says that namespace unification
is vital and that all you need are files and directories. Just
removing the problems that limit the usefulness of the "pathnames are
predicates" convention would achieve a lot. There are many pieces of
metadata that could readily be expressed as pathnames rather than
being stuffed away in stat blocks or ..metas files - ownership and
permissions data, for example.

One limitation to remove is the fact that it's impossible to (using
Sean's language) give a directory an "opaque name". An opaque
directory name would assert a predicate of the directory file itself
and not its descendants, whereas a normal directory name asserts a
predicate of the directory's non-directory descendants and not the
directory itself. So, for example, we might indicate the owner of a
directory by giving it the appropriate opaque name.

Probably the biggest barrier is the fact that it's nigh-impossible to
take a specific (non-directory) file and find its pathnames! We need
the ability to do this for any file, directory or otherwise, and for
all types of pathnames applied to the file.

But of course single-place predicates aren't enough to express all the
file metadata we might want. Sometimes we need relations too.

(**The answer begins here.** Apologies for the long set-up.)

Suppose I am using an improved *nix of the type I am proposing. There
is an image file on my computer named ~/photos/dessau-bauhaus . I have
a short description, stored in a text file, which I want to associate
with the image. I go the the directory

/(something)/description/

and create the following:

/(something)/description/aardvark:

/(something)/description/ is an ordinary directory, and
'/(something)/description' is an ordinary (non-opaque) name such as
directories have in the *nix of today. The only detail is that the
"pathnames are predicates" convention is stronger on this system than
on today's *nixes. So you can be sure that each of the (opaque)
descendants of /(something)/description/ isA
'/(something)/description' while /(something)/description/ itself is
not, unless it happens to be an opaque descendant of itself! (Again,
just as /bin/touch isA binary executable while /bin/ itself is not.)
Call /(something)/description/ a 'predicate-directory' or just a
directory.

/(something)/description/aardvark: (the colon is a delimiter) is
almost an ordinary directory. The calls for dealing with aardvark: are
probably almost exactly the same as those for handling an ordinary
directory; the internal representation of aardvark: may be identical
to that of an ordinary directory. The important difference is
semantic. aardvark: is a 'relation-directory': it expresses an
instance of a relation.

Consequently, '/(something)/description/aardvark' is not an ordinary
directory name. It's opaque: the predicate is asserted of the
directory itself and not its descendants.
/(something)/description/aardvark: isA
'/(something)/description/aardvark'. It therefore also isA
'/(something)/description', and so on transitively. By contrast, no
descendant of /something/description/aardvark: is (for example) a
'/(something)/description', except maybe through some other link. But
'/(something)/description/aardvark' is understood as asserting a file
type for aardvark, just as dessau-bauhaus might have another pathname
which would be understood as asserting that it is a jpeg.

Now I link /(whatever)/photos/dessau-bauhaus to aardvark: by the name

/(something)/description/aardvark:described

and I link the description to aardvark: by the name

/(something)/description/aardvark:description

So: aardvark: is an instance of the relation
'/(something)/description'. (Actually, it's also the only instance of
the more specialised relation '/(something)/description/aardvark', but
I haven't described the necessary tweak to get rid of that nuisance.)
In this instance of the relation, the file elsewhere named
'~/photos/dessau-bauhaus' has the role 'described', and the
description has taken the role 'description'. Now when one searches
for the pathnames of the file ~/photos/dessau-bauhaus,
'/(something)/description/aardvark:described' will be among them. A
person or program that knows the semantics of the
'/(something)/description' relation will be able to find the
description and know it as such. And if you don't know the semantics
you can still poke around inside the instance, like examining an XML
document whose schema you don't have.

So a relation-directory can express arbitrary relationships between
files. It's a bit like the relational model's weakly-typed sister,
where /(something)/description is a table, aardvark: is a row, and
~/photos/dessau-bauhaus is aardvark's entry in the column
/(something)/description/(the various relation-directories):described.
In OOese, /(something)/description is an association, aardvark: is one
of the links of that association, and (the various
relation-directories):described is a role name. (At least, that's the
Rumbaugh-Blaha-Premerlani-Eddy-and-Lorensen version; the UMLese may
vary.)

A relation-directory can have more than two children, and can have as
children predicate-directories (either opaquely or non-opaquely) or
other relation-directories.

But what about compound objects? Here's an example.

/(something)/concatenation/zebra:
/(something)/concatenation/zebra:1
/(something)/concatenation/zebra:2
/(something)/concatenation/zebra:3

This is of course the "concatenated file" example from the Reiser4
paper redone using a relation-directory. Just as with the earlier
description example, zebra is describing a relationship that exists
between its various member files. If we list the pathnames of
~/sometext, we might find that '/(something)/concatenation/zebra:2' is
one of its pathnames, which tells us that it is part 2 of some
specific concatenation. (Of course, we might also find pathnames
telling us that it is part 2 or 42 or some other concatenation, or a
description for ~/dessau-bauhaus, or very important, or whatever.) The
only difference is that in this case we find it useful to have the
relationship itself as an object in the filesystem. Easily done - just
treat the relationship's in-filesystem representation, the
relation-directory, as a file in its own right rather than just a
source of metadata about other files. In other words, just make some
links to the file otherwise known as
'/(something)/concatenation/zebra'. Call it '~/mybook', or
'/etc/passwd' if you're feeling brave. :) Now the real glory of this
system only reveals itself when you combine it with liberal use of
userspace mount() and (less importantly) file methods. That is the
synthesis of the dialectic between the Unix ideal of the file as the
atom of the filesystem and the world of compound documents, etc. - and
it's hoch-Unix. (Your slogan for the day: "bash Is My Object Browser".
Tomorrow: "mount() Is My Serialiser".) But note for now that if we
define 'atomic file' as 'just a simple sequence of bytes', we can
redefine 'file' as 'either an atomic file or a relationship between
files'.

To amplify, though, the "main", "real" purpose (so to speak) of
relation-directories is to express relationships between files. Doing
this properly just happens to give us compund objects for free. As
Rumbaugh, Blaha, Premerlani, Eddy and Lorensen say (in unison?),
"Aggregation is a special form of association, not an independent
concept." Beware the visual/spatial metaphors which subtly warp one's
understanding of the Unix file system. It's not a set of Russian dolls
or a maze of twisty little passages. In particular, files are not
physically inside directories, at any lelel of abstraction. aardvark:
just provides some metadata about ~/photos/dessau-bauhaus, and that's
all that ~/photos does too.

(Great, isn't it? The filesystem namespace: they're not names, and
it's not a space. :) )

On the other hand, garbage collection will be a significant hurdle,
for two reasons. One is cycles. The semantics of predicate-directories
mean that it's unnecessary to permit cycles containing only
predicate-directories, but if you're going to instances of, say the
singly-linked-list-node relation, then the need for cycles is
unavoidable. The other is more sophisticated needs for automatic
deletion. For example, we would probably need
/(something)/description/aardvark: to be marked for deletion as soon
as either of its children were unlinked.

Leo Comerford.

Re: file as a directory

[Alexander G. M. Smith]

Leo Comerford wrote on Mon, 16 May 2005 13:32:19 +0100:
> Probably the biggest barrier is the fact that it's nigh-impossible to
> take a specific (non-directory) file and find its pathnames! We need
> the ability to do this for any file, directory or otherwise, and for
> all types of pathnames applied to the file.

Not imposible with doubly linked directory entries, rather than the
traditional single links.  Been there, done that, the price is
performance, which we can now afford due to modern hardware?

> So a relation-directory can express arbitrary relationships between
> files. It's a bit like the relational model's weakly-typed sister,
> where /(something)/description is a table, aardvark: is a row, and
> ~/photos/dessau-bauhaus is aardvark's entry in the column
> /(something)/description/(the various relation-directories):described.
> In OOese, /(something)/description is an association, aardvark: is one
> of the links of that association, and (the various
> relation-directories):described is a role name. (At least, that's the
> Rumbaugh-Blaha-Premerlani-Eddy-and-Lorensen version; the UMLese may
> vary.)

As for the rest of your ideas, let me sum up what I think
you're talking about, please correct me if I got it wrong:

Group similar files together by linking them from a common parent
directory (files are allowed to have multiple parents) - which
serves mainly to give them an association.  You had a colon in
the directory name to mark that functionality, though it could
just be a regular directory in all but name.

Or is it about adding relations by attaching sub-files to give more
info about something?

So if I have a photograph of my aunt's cottage, I'd like to file it
in a directory for "Cottage" stuff and another one for "Aunt" things.
But when I want to add a descriptive text file, I'd rather have
it as a kind of attribute attached to the photo - making the
description a sub-file of the photo (treating the photo "file"
as a directory) so that it is preserved when I move the photo "file"
around.  The alternative of having a special description association
directory for all files that have the same description seems kind of
awkward (there's usually only one file per description).

Are both methods useful?  Yup.  What's the difference between
associations and properties?  Many-to-one and one-to-one?

Actually both can exist in the file system - when you do a directory
listing of a thing (the photo "file"), you get the subfile listing
(the text description file's name) and the parent directories listed
too (like ".." in regular directory listings, but you'd have ..photos
and ..aunt).

And then there are ReiserFS plugins, which might give you a magic
directory that when read for data, yields the concatenation of its
children's data contents.

> On the other hand, garbage collection will be a significant hurdle,
> for two reasons. One is cycles. [...]  The other is more
> sophisticated needs for automatic deletion.

Neither are impossible, we just have to pay the performance price
for the luxury features.

- Alex


P.S. Here's the example warped into an Unix style file system, with
dynamically generated fake symbolic links for things that wouldn't
fit (like multiple parent directories).  To get the world to accept
files as directories and other novelties, we may have to use
something like this for the transition from old to new:

Example:
total 2
drwxr-xr-x   1 agmsmith agmsmith        2 May 16 21:06 .
drwxrwxrwx   0 agmsmith agmsmith        7 May 16 21:06 ..
drwxr-xr-x   1 agmsmith agmsmith        1 May 16 21:09 Aunt
drwxr-xr-x   1 agmsmith agmsmith        1 May 16 21:08 Photos

Example/Aunt:
total 1
drwxr-xr-x   1 agmsmith agmsmith        1 May 16 21:09 .
drwxr-xr-x   1 agmsmith agmsmith        2 May 16 21:06 ..
lrwxrwxrwx   1 agmsmith agmsmith        1 May 16 21:09 NewBackStairs -> /RAMDisk/Example/Photos/NewBackStairs

Example/Photos:
total 1
drwxr-xr-x   1 agmsmith agmsmith        1 May 16 21:08 .
drwxr-xr-x   1 agmsmith agmsmith        2 May 16 21:06 ..
drwxr-xr-x   2 agmsmith agmsmith        2 May 16 21:08 NewBackStairs

Example/Photos/NewBackStairs:
total 202
drwxr-xr-x   2 agmsmith agmsmith        2 May 16 21:08 .
drwxr-xr-x   1 agmsmith agmsmith        1 May 16 21:08 ..
lrwxrwxrwx   0 agmsmith agmsmith        2 May 16 21:09 ... -> /RAMDisk/Example/Aunt
-rw-r--r--   1 agmsmith agmsmith       29 May 16 21:08 Description.txt
-rw-rw-rw-   1 agmsmith agmsmith   204641 May 14 15:40 ImageData.jpeg

Re: file as a directory

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander G. M. Smith wrote:
[...]
>>On the other hand, garbage collection will be a significant hurdle,
>>for two reasons. One is cycles. [...]  The other is more
>>sophisticated needs for automatic deletion.
> 
> 
> Neither are impossible, we just have to pay the performance price
> for the luxury features.

How much of a price do I pay?  And could that price be avioded by only
enabling those features for directories where I want them?  Could that
be done easily in Reiser4?

My mind isn't able to easily wrap itself around the point of having this
strange new kind of hard-links, but I'm willing to try it if there's no
performance hit for the rest of my system.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQop1aHgHNmZLgCUhAQJX6RAAgfnqnGGHRt+tzYNct9PfilJisPsOMwNV
EjH9dBvoU0e5WpNmFKAuwfzwscWZzLaPC/qyucKPWnr02AdroklGtId8DQdy9QWh
fYKc8/fFzDwwMyAu1r2vHNTbERBUX8GrKIMCeLRFnOLkgHdNDZ5vpn/bbWLZh0Tc
gMt6+qw8QU0qowWCoGtPadxR00C388Cc4QWiZAdaZhEF89gWX+Zhnwuhqtsni0aN
/h0gfXNArv3GgCefwG3WX+NDeikWLrZodpg9L3swJ2lWzVrikZ6yMET08NiG39Ih
iWLwrd5DCR8fIXir+jYeJBRCenoqx/LbqJmXkc7TengiddAob9WCFgcWYPbF4Mlv
vNBy/k9O+S8ds89GZwJAiaw/NO3YNuvf0I8SGo94KGlYhoVPPEq/lEZ2t0o5EiXn
nFyHYVhPSH9UG4ttGXDSuMiOikCbmeF1J9oLWhmnLoIXQ93kk6NOQYl4DgXGdOwD
QUM/xOPex69RW9RkNx8c4LygHssJeIAxxc6jhginWW+O/1Vgt6lfG1zlb22BXjls
n8Yi83gPn5d0fUGy+ss6xLIION6M9aWGpIxB/IMLjEV8Fwy+YR1qZqrSLJkVECZ1
oHtVq5A2uKDLr0xVZHdbxWyefFcSk73DkINBqDC6Vt8a+HCx/dqNHI3a1x2oaoR2
7ip1veAMiaI=
=DNyQ
-----END PGP SIGNATURE-----

Re: file as a directory

[Alexander G. M. Smith]

David Masover wrote on Tue, 17 May 2005 17:51:20 -0500:
> How much of a price do I pay?  And could that price be avioded by only
> enabling those features for directories where I want them?  Could that
> be done easily in Reiser4?

Interesting point.  A lot of those features require that file system
objects know which directories they are in (or whatever their parent
objects actually are, just think of them as directories for now).
That means extra pointers in each object back to all the parents it
belongs to.

Your idea leads to the idea of having both singly linked file system
objects and doubly linked ones.  Then when deleting items, the singly
linked subtrees don't need to have the full traversal to check for
cycles.  I'd leave that as a later optimization, if needed.

> My mind isn't able to easily wrap itself around the point of having this
> strange new kind of hard-links, but I'm willing to try it if there's no
> performance hit for the rest of my system.

The file creation and deletion time will be slower.  But hopefully not
by much, and that's about all that's affected.  The plus side is that
with the extra classifications (photos filed in both "Cottage" and "Aunt"
directories) you'd be able to find your data more easily!

- Alex

Re: file as a directory

[Leo Comerford]

> And then there are ReiserFS plugins, which might give you a magic
> directory that when read for data, yields the concatenation of its
> children's data contents.

Better, you could have a little custom filesystem which can take the
/(something)/concatenation/zebra: subgraph as its device and generate
a single file which is the concatenation of zebra:1, zebra:2, and so
on. (Remember that we can redefine 'file' as 'an atomic file or a link
between files'. So since the zebra: subgraph is a file, we can mount
it!) We could also have, for example, a pair of filesystems, one which
can mount an XML file and present it as an instance of an XML-document
association, and another which can mount such a link and present it as
a flat XML file.

Leo.

Re: file as a directory

[Leo Comerford]

On 5/17/05, Alexander G. M. Smith <agmsmith@rogers.com> wrote:

This is a bit of a shotgun reply, but I hope this answers your
questions and clarifies things. If not, please ask again and I'll try
to give a better answer. There are some other things I should add soon
anyway. (Not immediately, though - I'll be very busy over the next few
days.)

In the photo-with-description use-case, if you want to give the photo
a name, don't link to the relationship between the photo and the
description. Just link straight to the photo file itself, more/less as
you would do today. Remember, in my example, the jpeg file's original
name is '~/photos/dessau-bauhaus' . That links straight to the actual
jpeg binary file itself. (The name would be
'~/photos/dessau-bauhaus.jpeg', but in my example we're not using file
extensions to track file type anymore.) And after I've come along and
associated the dessau-bauhaus jpeg with a description,
~/photos/dessau-bauhaus *still* just links straight to the jpeg
binary. As far as the pathname '~/photos/dessau-bauhaus' is concerned,
'/(something)/description/aardvark:described' is just another pathname
that happens to link to the same file as it. Inspecting the pathnames
of the file ~/photos/dessau-bauhaus will reveal
'/(something)/description/aardvark:described' along with all the other
names it has.

The directory /(something)/description/aardvark: is the link between
the photo and the description. (To be clear, the colon isn't part of
any name-segment. It's a delimiter betweeen name-segments: it's the
relation-directory equivalent of the forward-slash.) In OO terms, the
directory /(something)/description/aardvark: isA
'/(something)/description' and it hasA 'description' and a
'described'. Being the child of /(something)/description/ gives the
relation-directory its file type: it's the pathname
'/(something)/description/(whatever)' that tells us to interpret the
relation-directory as a link between a description and the file it
describes. Actually aardvark: "already" has a file type of
relation-directory - the link from /(something)/description/ specifies
what type of relation-directory it is. Every relation-dir that is a
description-described link is (by definition) in
/(something)/description/ , regardless of which files they have as
their 'description' or 'described'. (This is the sense in which
/(something)/description/ is an "association" - in OOese the term
means a /type/ of link between objects.)

For example, let's suppose a different user, Bob, decides to attach
some descriptions too. He puts a description on the jpeg file
/home/bob/photos/petit-trianon ; he also decides he doesn't like my
description of /home/leo/photos/dessau-bauhaus and puts his own
description on it. So now we have:

/(something)/description/aardvark:
/(something)/description/aardvark:description   -- this is
/home/leo/dessau-bauhaus
/(something)/description/aardvark:described     -- this is my description of it
/(something)/description/manticore:
/(something)/description/manticore:description  -- this is
/home/leo/dessau-bauhaus again
/(something)/description/manticore:described    -- this is Bob's
description of it
/(something)/description/sheep:
/(something)/description/sheep:description        -- this is
/home/bob/petit-trianon
/(something)/description/sheep:described          -- this is Bob's
description of it

And now if I list the pathnames of ~/photos/dessau-bauhaus, I get both
'/(something)/description/aardvark:described' and
'/(something)/description/manticore:described' - telling me that there
are two different descriptions of the file. And everything still just
links directly to the jpeg file.

About the colon: there's nothing magic about the choice of character,
of course, but there does absolutely need to be a way to identify
links from relation-directories in pathnames. Programs (and humans)
need to be able to mechanically tell when (and where) a pathname
asserts part of a relation and when it asserts a predicate, just by
looking at it. Changing the delimiter is a good way to do it because
it highlights the important point: that the relationship between
aardvark and description is different to the relationship between a
(predicate-)directory and its child. (You could use file extensions as
a makeshift subsititute. How about .: ?)

> Are both methods useful?  Yup.  What's the difference between
> associations and properties?  Many-to-one and one-to-one?

The idea is that if you want to assert a single-place predicate of a
file, like "file x is important", you just use give the file an
approprate full path"name" ('~/important' or whatever). If you want to
assert a multi-place predicate - a relation - like "file x is more
important than file y" then you use a relation-directory. That goes
for every kind of multi-way relation/association you might want to
assert between files - one to one, one to many, many to many.
Actually, '/(something)/description' asserts a predicate just like
'~/important' does. But because it's a multi-place predicate, you
can't just link straight from it to the file(s) it applies to. You
need relation-directories to "break out" the different roles in each
instance of the predicate.

(In fact that isn't true in every case. The only reason you have a
separate description file in the photo-with-description example is
because the description is too big and clunky to fit in a file name
segment. In theory, what you really want is just to give the photo a
pathname that contains the description, but since that's not doable
you create a description file and assert a relation between that and
the photo. Back when I first made an earlier version of this proposal,
Hans had an alternative, possibly better, solution for cases like
this. (And it didn't involve subfiles!) But

a) it can probably be implemented using relations, and

b) it only substitutes for a link in the the special case where all
the other member files of the link only really exist to assert
metadata about one of the members. Many or most useful links aren't
like that - like the concatenation example.)

> Actually both can exist in the file system - when you do a directory
> listing of a thing (the photo "file"), you get the subfile listing
> (the text description file's name) and the parent directories listed
> too (like ".." in regular directory listings, but you'd have ..photos
> and ..aunt).

But if you have relation-directories and the ability to find the
pathnames of a given file, you can do everything you can do with
subfiles, just as nicely, and more besides. And if subfiles are
completely redundant and bad news anyway, we shouldn't have them.

Leo.

Re: file as a directory

[Bodo Eggert]

Martin Waitz wrote:
> On Mon, Nov 22, 2004 at 08:34:02AM -0700, Zan Lynx wrote:

>> There are already several things in filesystems that don't strictly
>> belong inside the kernel.  A filesystem could be implemented quite well
>> as a user-space daemon that sat on top of the block device and
>> communicated via sockets or shared memory just like an X server.
> 
> this is quite different.
> As you need to enforce security policies when accessing the block
> device, you have to move the filesystem into its own daemon.
> You cannot do it in a library.

Only the mapping from block-nr to uid/gid/perms needs to be in the kernel.
The rest can be done in userspace, but it would be ugly as hell.

> It is irrelevant for the application weather the fs resides in a
> separate daemon or in the kernel itself.

ACK.

> But support of different views on files is something different.
> You can do that in a library, you only need an interface that is
> capable of storing your data. The kernel already provides that
> interface.

If you want to allow users to set their default shell using some extension,
a simple userspace library will not do the job. You'll need a central
authority that is able to synchronize the access to the file, prevent
unauthorized modifications, do the caching etc.

I think the special file handlers should generally be daemons, but the
access should be controlled by kernel hooks, maybe something like
automounting a userspace filesystem. Simple meta-filesystems, e.g. those
that could be "emulated" using mount -oloop, may reside in the kernel
space, more complicated but common ones may be stored in the kernel
(placed on the initramfs?) to accelerate starting the helper daemon and
uncommon ones would be registered at runtime (maybe user-specific).

Having functions in the kernel to support those filesystems in the kernel
will help, e.g. a tgz helper daemon would need to allocate temporary
storage for accelerating access (e.g. file tree, cache) as well as means
to reassemble the tgz for operations on the whole file after write
operations occured. Meta-filesystems on fs without these features can
(off cause) be done, but I think they would be very slow or show
inconsitencies in certain situations.
-- 
Bug? That's not a bug, that's a feature.