Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Hans Reiser]

Brad Boyer wrote:

>Hans Reiser wrote:
>
>>I remember that I used to be a sysadmin with some NetApp boxes that have 
>>a .snapshot directory that is invisible, and has special qualities.
>>
>>It worked.  There were no namespace collision problems.  None.
>>
>>These things can be survived by users.;-)
>>
>
>Yes, these things can be survived, but speaking as someone who currently
>has a job involving multiple NetApp boxes, I can say that the .snapshot
>directory has some seriously annoying properties that break tar and
>other programs that expect things to look normal. The snapshots have
>saved my ass a few times, but they're still a pain to work with due
>to a few little quirks. In particular, the files in the snapshot keep
>the same inode number as the actual file. Just remember that clever
>solutions that almost fit the traditional model can have strange
>results over time.
>
>	Brad Boyer
>	flar@allandria.com
>
>
>

Can you detail the problem?

Hans

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[dmeyer]

In article <3C19DE41.6000507@namesys.com> you write:
> Brad Boyer wrote:
> >In particular, the files in the snapshot keep
> >the same inode number as the actual file. Just remember that clever
> >solutions that almost fit the traditional model can have strange
> >results over time.
> 
> Can you detail the problem?

Anything that uses something like file1.st_dev==file2.st_dev &&
file1.st_ino==file2.st_ino to decide if two filenames point to the
same file can get terribly confused.  For example,

$ ls -li .snapshot/hourly.0/.zshrc .zshrc
1411878 -rw-r--r--    1     1247 Mar 19  2001 .snapshot/hourly.0/.zshrc
1411878 -rw-r--r--    1     1248 Dec 14 09:51 .zshrc

Clearly, the file has been modified since the hourly.0 snapshot; however

$ cp .snapshot/hourly.0/.zshrc .zshrc
cp: `.snapshot/hourly.0/.zshrc' and `.zshrc' are the same file

you can't copy the snapshot on top of the current version, since they
have the same inode number.  A somewhat contrived example, perhaps,
but I have been bitten by something similar in the real world.  One of
the things I would like to be able to do with a snapshot is to open a
file in emacs, open a snapshot in another window, and compare the two
files with ediff.  And you can't; emacs treats the original and the
snapshot as if they were the same file - just like cp does - even
though the file contents are different.

-- 
Dave Meyer
dmeyer@dmeyer.net

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Brad Boyer]

Hans Reiser wrote:
> Brad Boyer wrote:
> >Yes, these things can be survived, but speaking as someone who currently
> >has a job involving multiple NetApp boxes, I can say that the .snapshot
> >directory has some seriously annoying properties that break tar and
> >other programs that expect things to look normal. The snapshots have
> >saved my ass a few times, but they're still a pain to work with due
> >to a few little quirks. In particular, the files in the snapshot keep
> >the same inode number as the actual file. Just remember that clever
> >solutions that almost fit the traditional model can have strange
> >results over time.
> 
> Can you detail the problem?
> 

The problem with the NetApp snapshots is that tar and cp and a few other
programs that check inode numbers get confused and think everything in
the snapshot is a hard link. So you can't copy a snapshot of a file back
over the original without copying it somewhere else first, and it's
painful to make an archive of the snapshots. We have data files on our
filers that get updated frequently, and any time I need to analyze the
same file over time, or restore an old file, it causes problems. I was
throwing it out more as an example of what sort of unexpected things
happen when you slightly change the way the filesystem works.

	Brad Boyer
	flar@allandria.com

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Bernd Eckenfels]

In article <20011214100638.A7268@jhereg.dmeyer.net> you wrote:
> Clearly, the file has been modified since the hourly.0 snapshot; however

What kind of snapshot software are u using here? lvm?

Greetings
Bernd

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Anton Altaparmakov]

At 15:27 13/12/01, Andrew Pimlott wrote:
>(Anton, does NTFS define any reserved filename characters, or only
>win32?)

It does. RTFS. (-8

 From ntfs-driver-tng/linux/fs/ntfs/layout.h:

/*
  * The maximum allowed length for a file name.
  */
#define MAXIMUM_FILE_NAME_LENGTH 255

/*
  * Possible namespaces for filenames in ntfs (8-bit).
  */
typedef enum {
         FILE_NAME_POSIX                 = 0x00,
                 /* This is the largest namespace. It is case sensitive and
                    allows all Unicode characters except for: '\0' and '/'.
                    Beware that in WinNT/2k files which eg have the same name
                    except for their case will not be distinguished by the
                    standard utilities and thus a "del filename" will delete
                    both "filename" and "fileName" without warning. */
         FILE_NAME_WIN32                 = 0x01,
                 /* The standard WinNT/2k NTFS long filenames. Case 
insensitive.
                    All Unicode chars except: '\0', '"', '*', '/', ':', '<',
                    '>', '?', '\' and '|'. Further, names cannot end with a '.'
                    or a space. */
         FILE_NAME_DOS                   = 0x02,
                 /* The standard DOS filenames (8.3 format). Uppercase only.
                    All 8-bit characters greater space, except: '"', '*', '+',
                    ',', '/', ':', ';', '<', '=', '>', '?' and '\'. */
         FILE_NAME_WIN32_AND_DOS         = 0x03,
                 /* 3 means that both the Win32 and the DOS filenames are
                    identical and hence have been saved in this single filename
                    record. */
} __attribute__ ((__packed__)) FILE_NAME_TYPE_FLAGS;

The whole of layout.h can be viewed here (link to view of CVS):

http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/linux-ntfs/ntfs-driver-tng/linux/fs/ntfs/layout.h?rev=1.6&content-type=text/vnd.viewcvs-markup

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Hans Reiser]

Andrew Pimlott wrote:

>On Thu, Dec 13, 2001 at 12:23:46PM +0300, Hans Reiser wrote:
>
>>Andrew Pimlott wrote:
>>
>>>First, I write a desktop application that wants to save an HTML file
>>>along with some other object that contains the name of the creating
>>>application.  The latter can go anywhere you want, except in the
>>>same stream as the HTML file.  The user has requested that the
>>>filename be /home/user/foo.html , and expects to be able to FTP this
>>>file to his ISP with a standard FTP program.  What calls does my
>>>application make to store the HTML and the application name?  If the
>>>answer is different depending on whether /home/user is NTFS or
>>>reiserfs4, explain both ways.
>>>
>>Are you sure that standard ftp will be able to handle extended 
>>attributes without modification?
>>
>
>No, the ftp program only needs to transfer the HTML part.
>
>>One approach is to create a plugin called ..archive that when read is a 
>>virtual file consisting of an archive of everything in the directory. 
>>
>
>Ok, does this mean that every directory in the filesystem (or in
>some part of it) will automatically have a node ..archive?
>Presumably, it will not appear in directory listings, but can be
>read but not written to?  Does this mean that a legacy application
>(pathological as it may be) that expects to be able to create a file
>called ..archive will fail?
>
I remember that I used to be a sysadmin with some NetApp boxes that have 
a .snapshot directory that is invisible, and has special qualities.

It worked.  There were no namespace collision problems.  None.

These things can be survived by users.;-)

Nothing I say should be construed to mean that I think that a particular 
name for a pseudo-file implemented by the default regular directory 
plugin is what should ship.  I am easy in such matters.  You can also 
get me to agree it should be modifiable, so that if Joe Sevenpack needs 
a file named ..archive, he can have it.

>
>
>Or do you mean that the application would explicitly create the node
>associated with this plugin?
>
Both.  If you want a file named '..glob' that does the same thing as
'..archive', go for it.  I am not necessarily committed to putting 
..archive in the default directory plugin (actually, I don't like that 
name, it should be something snappier, but I haven't thought of it).  I 
also am not funded to implement ..archive at the moment (I am funded to 
do inheritance though) .

>
>
>>It would be interesting I think to attach said plugin to standard 
>>directories by default along with several other standard plugins like 
>>..cat, etc.
>>
>
>Anyway, you didn't answer the part I really care about.  What calls
>does the application make to store the HTML and the "extended
>attribute"?  You can pick whatever conventions you want, just give
>me an example.
>

read, write, etc., on file.html/..joes_attribute, unless it is a 
particular attribute that has particular effects on the particular 
plugin for file.html, in which case it all depends on the plugin and the 
constraints imposed on joes_attribute.  It may be that modifying 
file.html modifies ..joes_attribute as a side-effect, plugins can do 
anything in response to a VFS operation.  You put the plugin into your 
kernel, you'd better be able to trust it....

>
>
>>>Second, I booted NT and created a directory in the NTFS filesystem
>>>called /foo .  In the directory, I created a file called bar.  I
>>>also created a named stream called bar, and an extended attribute
>>>called bar.  Now I boot Linux.  What calls do I make to see each of
>>>the three objects called bar?
>>>
>>You access /foo/bar, /foo/bar/,,bar, /foo/..bar by name.
>>
>
>How do I access the file called ..bar (created in NT) in the
>directory /foo?
>

If you have permission, you can:
cat /foo/..bar

Or you can use the efficient for small files API we are implementing, 
which I won't go into here.

>
>
>(Anton, does NTFS define any reserved filename characters, or only
>win32?)
>
>Andrew
>
>

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Andrew Pimlott]

On Thu, Dec 13, 2001 at 12:23:46PM +0300, Hans Reiser wrote:
> Andrew Pimlott wrote:
> >First, I write a desktop application that wants to save an HTML file
> >along with some other object that contains the name of the creating
> >application.  The latter can go anywhere you want, except in the
> >same stream as the HTML file.  The user has requested that the
> >filename be /home/user/foo.html , and expects to be able to FTP this
> >file to his ISP with a standard FTP program.  What calls does my
> >application make to store the HTML and the application name?  If the
> >answer is different depending on whether /home/user is NTFS or
> >reiserfs4, explain both ways.
> >
> Are you sure that standard ftp will be able to handle extended 
> attributes without modification?

No, the ftp program only needs to transfer the HTML part.

> One approach is to create a plugin called ..archive that when read is a 
> virtual file consisting of an archive of everything in the directory. 

Ok, does this mean that every directory in the filesystem (or in
some part of it) will automatically have a node ..archive?
Presumably, it will not appear in directory listings, but can be
read but not written to?  Does this mean that a legacy application
(pathological as it may be) that expects to be able to create a file
called ..archive will fail?

Or do you mean that the application would explicitly create the node
associated with this plugin?

> It would be interesting I think to attach said plugin to standard 
> directories by default along with several other standard plugins like 
> ..cat, etc.

Anyway, you didn't answer the part I really care about.  What calls
does the application make to store the HTML and the "extended
attribute"?  You can pick whatever conventions you want, just give
me an example.

> >Second, I booted NT and created a directory in the NTFS filesystem
> >called /foo .  In the directory, I created a file called bar.  I
> >also created a named stream called bar, and an extended attribute
> >called bar.  Now I boot Linux.  What calls do I make to see each of
> >the three objects called bar?
> >
> 
> You access /foo/bar, /foo/bar/,,bar, /foo/..bar by name.

How do I access the file called ..bar (created in NT) in the
directory /foo?

(Anton, does NTFS define any reserved filename characters, or only
win32?)

Andrew

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Hans Reiser]

Andrew Pimlott wrote:

>On Wed, Dec 12, 2001 at 12:21:49AM +0300, Hans Reiser wrote:
>
>>Naming conventions are easy.
>>
>
>Hans,
>
>While I look forward to your work, I think Anton points out some
>issues that you really should try to address now, only you have not
>understood them.  Can I take a crack at posing some concrete
>questions that manifest the issues?
>
>Let's imagine that we have a Linux system with an NTFS filesystem
>and a reiserfs4 filesystem.  You can make any tentative assumptions
>about reiserfs4 and new API's that you like, I just want to have an
>idea of how you envision the following working:
>
>First, I write a desktop application that wants to save an HTML file
>along with some other object that contains the name of the creating
>application.  The latter can go anywhere you want, except in the
>same stream as the HTML file.  The user has requested that the
>filename be /home/user/foo.html , and expects to be able to FTP this
>file to his ISP with a standard FTP program.  What calls does my
>application make to store the HTML and the application name?  If the
>answer is different depending on whether /home/user is NTFS or
>reiserfs4, explain both ways.
>
Are you sure that standard ftp will be able to handle extended 
attributes without modification?

One approach is to create a plugin called ..archive that when read is a 
virtual file consisting of an archive of everything in the directory. 
 It would be interesting I think to attach said plugin to standard 
directories by default along with several other standard plugins like 
..cat, etc.

>
>
>Second, I booted NT and created a directory in the NTFS filesystem
>called /foo .  In the directory, I created a file called bar.  I
>also created a named stream called bar, and an extended attribute
>called bar.  Now I boot Linux.  What calls do I make to see each of
>the three objects called bar?
>

You access /foo/bar, /foo/bar/,,bar, /foo/..bar by name.

>
>
>The heart of Anton's argument is that the UNIX filesystem name space
>is basically used up--there's just not much room to add new
>semantics.  The only obvious avenue for extension is, if /foo is not
>a directory, you can give some interpretation to /foo/bar .  But
>this doesn't help if /foo is a directory.  So something has to give,
>and we want to see what will give in reiserfs4.
>
>Andrew
>
>
Naming conventions are easy, but teaching user space is hard no matter 
whose scheme is used.

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Andrew Pimlott]

On Wed, Dec 12, 2001 at 12:21:49AM +0300, Hans Reiser wrote:
> Naming conventions are easy.

Hans,

While I look forward to your work, I think Anton points out some
issues that you really should try to address now, only you have not
understood them.  Can I take a crack at posing some concrete
questions that manifest the issues?

Let's imagine that we have a Linux system with an NTFS filesystem
and a reiserfs4 filesystem.  You can make any tentative assumptions
about reiserfs4 and new API's that you like, I just want to have an
idea of how you envision the following working:

First, I write a desktop application that wants to save an HTML file
along with some other object that contains the name of the creating
application.  The latter can go anywhere you want, except in the
same stream as the HTML file.  The user has requested that the
filename be /home/user/foo.html , and expects to be able to FTP this
file to his ISP with a standard FTP program.  What calls does my
application make to store the HTML and the application name?  If the
answer is different depending on whether /home/user is NTFS or
reiserfs4, explain both ways.

Second, I booted NT and created a directory in the NTFS filesystem
called /foo .  In the directory, I created a file called bar.  I
also created a named stream called bar, and an extended attribute
called bar.  Now I boot Linux.  What calls do I make to see each of
the three objects called bar?

The heart of Anton's argument is that the UNIX filesystem name space
is basically used up--there's just not much room to add new
semantics.  The only obvious avenue for extension is, if /foo is not
a directory, you can give some interpretation to /foo/bar .  But
this doesn't help if /foo is a directory.  So something has to give,
and we want to see what will give in reiserfs4.

Andrew

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Hans Reiser]

Anton Altaparmakov wrote:

> At 12:02 12/12/01, Hans Reiser wrote:
>
>> Anton Altaparmakov wrote:
>>
>>> On Wed, 12 Dec 2001, Hans Reiser wrote:
>>>
>>>> Anton Altaparmakov wrote:
>>>>
>>>>> Both MacOS and as of recently Windows do this kind of stuff, too, 
>>>>> and it
>>>>> can't be long before Linux goes the same way, provided file systems
>>>>> support the required features (i.e. EAs and/or named streams) so I
>>>>> disagree with you this is only a compatibility thing. It might 
>>>>> start out
>>>>> as one but it will find real world applications very quickly...
>>>>
>>>> I am not saying that the features of EAs are not useful, I am 
>>>> saying that I want to choose them individually for particular files.
>>>>
>>>> It could be so much better to have EDIBLE_PIZZA (example from 
>>>> previous email) instead of just PIZZA, sigh.
>>>
>>>
>>> I am not quite sure what you mean. Surely you can just have all 
>>> features
>>> available at all times/to all files and then you just use the ones you
>>> want, just ignoring/not using the rest. Why do you see the need for
>>> "selecting features of EAs individually for particular files"? It makes
>>> sense when buying EDIBLE_PIZZA but I don't see how that can be 
>>> transferred
>>> onto files. After all I can just have all pizza ingredients and only 
>>> put
>>> the ones I want on the pizza just ignoring the others.
>>
>> Inheriting stat data from the parent directory should be a feature 
>> available not just for streams, but for all files that want it. 
>> Efficient small file access to a 32 byte file should be a feature 
>> available to all files, not just EAs.  Not being listed in readdir 
>> should be a feature available to all files, not just EAs.  
>> Constraining what is written to them should be a feature available to 
>> all files, not just EAs, and arbitrary plugin based constraints 
>> should be possible.
>>
>> Is this more clear?
>
>
> Yes it is, thanks. And yes it makes sense. But this is talking about 
> files as a whole and has nothing to do with EAs as such (but it would 
> obviously apply to EAs, too under your proposed API). 

There would be no need for EAs if files as a whole could have these 
properties, as EAs would just be particular files with particular 
properties within a directory/file object.

>
>
> I will be looking forward to seeing this stuff implemented. (-:
>
> Anton
>
>

If Linux users get really unlucky, which seems likely, :-/, 2.6 will 
take as long as 2.4, in which case I think we will complete the task in 
plenty of time for 2.6, and we can ask Linus which implementation he 
prefers before he has committed to one in a stable release.

Hans

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Anton Altaparmakov]

At 12:02 12/12/01, Hans Reiser wrote:
>Anton Altaparmakov wrote:
>>On Wed, 12 Dec 2001, Hans Reiser wrote:
>>>Anton Altaparmakov wrote:
>>>>Both MacOS and as of recently Windows do this kind of stuff, too, and it
>>>>can't be long before Linux goes the same way, provided file systems
>>>>support the required features (i.e. EAs and/or named streams) so I
>>>>disagree with you this is only a compatibility thing. It might start out
>>>>as one but it will find real world applications very quickly...
>>>I am not saying that the features of EAs are not useful, I am saying 
>>>that I want to choose them individually for particular files.
>>>
>>>It could be so much better to have EDIBLE_PIZZA (example from previous 
>>>email) instead of just PIZZA, sigh.
>>
>>I am not quite sure what you mean. Surely you can just have all features
>>available at all times/to all files and then you just use the ones you
>>want, just ignoring/not using the rest. Why do you see the need for
>>"selecting features of EAs individually for particular files"? It makes
>>sense when buying EDIBLE_PIZZA but I don't see how that can be transferred
>>onto files. After all I can just have all pizza ingredients and only put
>>the ones I want on the pizza just ignoring the others.
>Inheriting stat data from the parent directory should be a feature 
>available not just for streams, but for all files that want it. Efficient 
>small file access to a 32 byte file should be a feature available to all 
>files, not just EAs.  Not being listed in readdir should be a feature 
>available to all files, not just EAs.  Constraining what is written to 
>them should be a feature available to all files, not just EAs, and 
>arbitrary plugin based constraints should be possible.
>
>Is this more clear?

Yes it is, thanks. And yes it makes sense. But this is talking about files 
as a whole and has nothing to do with EAs as such (but it would obviously 
apply to EAs, too under your proposed API).

I will be looking forward to seeing this stuff implemented. (-:

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Hans Reiser]

Anton Altaparmakov wrote:

>On Wed, 12 Dec 2001, Hans Reiser wrote:
>
>>Anton Altaparmakov wrote:
>>
>>>I was just stating a fact of how they are stored on NTFS, again something
>>>I have no power to change.
>>>
>>But does NTFS specificism/cripplism belong in VFS?
>>
>
>No, of course not. But the vfs needs to be able to cope with limitations
>of specific file systems (even if it is only by passing -Exyz into
>userspace).
>
We agree on this, I have no opposition to NTFS checking  size of files 
used to store EAs and rejecting any write more than 64k

>
>
>>>>Well, gosh, okay, maybe you want to prepend ',,' to streams and '..' to 
>>>>extended attributes.  I personally think Linux would only want to do so 
>>>>when used as a fileserver emulating NTFS/SAMBA.  There is no enhancement 
>>>>of user functionality from doing it for general purpose filesystems. 
>>>>
>>>Just wait until this functionality is available and watch all GUI things
>>>start to use it en masse! I don't doubt that GNOME/KDE/replace with your
>>>favourite window manager are going to hesitate to start putting in the
>>>icon, the name, and whatnot inside EAs or inside named streams the instant
>>>they are ubiquitously available and I think that makes a lot of sense too.
>>>No doubt I will get flamed for saying this but all flames go to
>>>/dev/null...
>>>
>>>Both MacOS and as of recently Windows do this kind of stuff, too, and it
>>>can't be long before Linux goes the same way, provided file systems
>>>support the required features (i.e. EAs and/or named streams) so I
>>>disagree with you this is only a compatibility thing. It might start out
>>>as one but it will find real world applications very quickly...
>>>
>>I am not saying that the features of EAs are not useful, I am saying 
>>that I want to choose them individually for particular files.
>>
>>It could be so much better to have EDIBLE_PIZZA (example from previous 
>>email) instead of just PIZZA, sigh.
>>
>
>I am not quite sure what you mean. Surely you can just have all features
>available at all times/to all files and then you just use the ones you
>want, just ignoring/not using the rest. Why do you see the need for
>"selecting features of EAs individually for particular files"? It makes
>sense when buying EDIBLE_PIZZA but I don't see how that can be transferred
>onto files. After all I can just have all pizza ingredients and only put
>the ones I want on the pizza just ignoring the others.
>
Inheriting stat data from the parent directory should be a feature 
available not just for streams, but for all files that want it. 
 Efficient small file access to a 32 byte file should be a feature 
available to all files, not just EAs.  Not being listed in readdir 
should be a feature available to all files, not just EAs.  Constraining 
what is written to them should be a feature available to all files, not 
just EAs, and arbitrary plugin based constraints should be possible.

Is this more clear?

Hans

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Anton Altaparmakov]

On Wed, 12 Dec 2001, Hans Reiser wrote:
> Anton Altaparmakov wrote:
> >I was just stating a fact of how they are stored on NTFS, again something
> >I have no power to change.
> >
> But does NTFS specificism/cripplism belong in VFS?

No, of course not. But the vfs needs to be able to cope with limitations
of specific file systems (even if it is only by passing -Exyz into
userspace).

> >>Well, gosh, okay, maybe you want to prepend ',,' to streams and '..' to 
> >>extended attributes.  I personally think Linux would only want to do so 
> >>when used as a fileserver emulating NTFS/SAMBA.  There is no enhancement 
> >>of user functionality from doing it for general purpose filesystems. 
> >
> >Just wait until this functionality is available and watch all GUI things
> >start to use it en masse! I don't doubt that GNOME/KDE/replace with your
> >favourite window manager are going to hesitate to start putting in the
> >icon, the name, and whatnot inside EAs or inside named streams the instant
> >they are ubiquitously available and I think that makes a lot of sense too.
> >No doubt I will get flamed for saying this but all flames go to
> >/dev/null...
> >
> >Both MacOS and as of recently Windows do this kind of stuff, too, and it
> >can't be long before Linux goes the same way, provided file systems
> >support the required features (i.e. EAs and/or named streams) so I
> >disagree with you this is only a compatibility thing. It might start out
> >as one but it will find real world applications very quickly...
> >
> I am not saying that the features of EAs are not useful, I am saying 
> that I want to choose them individually for particular files.
> 
> It could be so much better to have EDIBLE_PIZZA (example from previous 
> email) instead of just PIZZA, sigh.

I am not quite sure what you mean. Surely you can just have all features
available at all times/to all files and then you just use the ones you
want, just ignoring/not using the rest. Why do you see the need for
"selecting features of EAs individually for particular files"? It makes
sense when buying EDIBLE_PIZZA but I don't see how that can be transferred
onto files. After all I can just have all pizza ingredients and only put
the ones I want on the pizza just ignoring the others.

Um, I think we might be saying the same thing in different words...

> >>Programs will get written to use your API, and not work with reiserfs, 
> >>and will get written to use our API and not work with NTFS, and this is 
> >>bad....
> >
> >Now that is true. And yes, it is bad. However it will be up to the
> >community to decide which API to use and at the moment there are several
> >fs using the "bestbits" API and only reiserfs (?) the "reiserfs" one...
> >And we all know from our very own $Deity that we don't design software, we
> >just write things and let evolution decide which is better. (((-;
> >
> Fortunately he isn't entirely consistent on this point.:-)
> 
> I predict you guys will ship first and get a lot of usage, and then we 
> will ship later with more features,
> and the result will be a mess for users.  This is the usual evolutionary 
> design standards mess.  

Yes, your prediction will likely hold true IMO.

> Objectively, I understand it is highly reasonable for the Linux 
> community to assume that what we
> implement will be horrible until we finish it.  I would encourage it to 
> assume that someone else
> will eventually get orthogonalism right though, and I think it would be 
> worth waiting for it, because
> these are the sorts of design features that stick around for 30 years. 
>  I don't really expect that most folks will choose to wait though.

Me neither. People want it now, which pretty much limits the choice to
one of the things available and working now, plus some required cleanups
to satisfy all $Deities so the solution can be accepted in the kernel...

The one who comes first gets to populate the vacuum. Evolution at its
best. (-:

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Hans Reiser]

Anton Altaparmakov wrote:

>
>I was just stating a fact of how they are stored on NTFS, again something
>I have no power to change.
>
But does NTFS specificism/cripplism belong in VFS?  (I in no way blame 
you for NTFS's design:-) )

>>Well, gosh, okay, maybe you want to prepend ',,' to streams and '..' to 
>>extended attributes.  I personally think Linux would only want to do so 
>>when used as a fileserver emulating NTFS/SAMBA.  There is no enhancement 
>>of user functionality from doing it for general purpose filesystems. 
>>
>
>Just wait until this functionality is available and watch all GUI things
>start to use it en masse! I don't doubt that GNOME/KDE/replace with your
>favourite window manager are going to hesitate to start putting in the
>icon, the name, and whatnot inside EAs or inside named streams the instant
>they are ubiquitously available and I think that makes a lot of sense too.
>No doubt I will get flamed for saying this but all flames go to
>/dev/null...
>
>Both MacOS and as of recently Windows do this kind of stuff, too, and it
>can't be long before Linux goes the same way, provided file systems
>support the required features (i.e. EAs and/or named streams) so I
>disagree with you this is only a compatibility thing. It might start out
>as one but it will find real world applications very quickly...
>
I am not saying that the features of EAs are not useful, I am saying 
that I want to choose them
individually for particular files.

It could be so much better to have EDIBLE_PIZZA (example from previous 
email)
instead of just PIZZA, sigh.

>
>
>
>>>
>>Programs will get written to use your API, and not work with reiserfs, 
>>and will get written to use our API and not work with NTFS, and this is 
>>bad....
>>
>
>Now that is true. And yes, it is bad. However it will be up to the
>community to decide which API to use and at the moment there are several
>fs using the "bestbits" API and only reiserfs (?) the "reiserfs" one...
>And we all know from our very own $Deity that we don't design software, we
>just write things and let evolution decide which is better. (((-;
>
Fortunately he isn't entirely consistent on this point.:-)

I predict you guys will ship first and get a lot of usage, and then we 
will ship later with more features,
and the result will be a mess for users.  This is the usual evolutionary 
design standards mess.  
Objectively, I understand it is highly reasonable for the Linux 
community to assume that what we
implement will be horrible until we finish it.  I would encourage it to 
assume that someone else
will eventually get orthogonalism right though, and I think it would be 
worth waiting for it, because
these are the sorts of design features that stick around for 30 years. 
 I don't really expect that most
folks will choose to wait though.

Best to all,

Hans

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Anton Altaparmakov]

On Wed, 12 Dec 2001, Hans Reiser wrote:
> Anton Altaparmakov wrote:
> >> *Hans Reiser wrote:
> >>
> >> What is your intended functional difference between extended 
> >> attributes and streams?
> >>
> >> None?
> >
> >
> > Differences in NTFS:
> >
> > - maximum size (EA limited to 64kiB, named stream 2^63 bytes) 
> 
> These are desirable limits to preserve?  For sure?  If so, then a 
> particular plugin can be written to restrict files to 64k, though I 
> shake my head at the thought.

You cannot not preserve them. At least not on NTFS! That is what the NTFS
specifications state (as visible from $AttrDef system file). I don't have
the option to change this.

> > - locality of storage (all EAs are stored in one so they are quicker 
> > to access when you need to access multiple EAs) 
> 
> arbitrarily aggregating files is a useful feature, there is no reason to 
> rigidly offer and require the feature for EAs only.

I was just stating a fact of how they are stored on NTFS, again something
I have no power to change.

> > - name namespace (Unicode names for named streams vs ASCII for EAs) 
> 
> Namespaces can be changed for the children of directories.  Plan9 guys 
> have done such things, and it is cool.
> 
> > - potential ability to compress/encrypt (EAs cannot do this, named 
> > streams could possibly and they certainly can be sparse, too which EAs 
> > cannot be) 
> 
> Well, I suppose you could allow restricting some files to not have 
> compression and sparseness, though it isn't exciting to me.
> >
> > - named streams have creation/modification/access/etc times associated 
> > with them, EAs don't 
> 
> I thought streams shared the stat data of the parent file?  

Yes, they do. Sorry, I got mixed up on that one point. I was thinking of
hard links rather than named streams (attribute $FILE_NAME rather than
$DATA).

> > How is that for a start? 
> 
> Not one reason cited is convincing to me.
> >
> >> Ok, let's assume none until I get your response.  (I can respond more 
> >> specifically
> >> after you correct me.)  Let me further go out on a limb,and guess 
> >> that you intend
> >> that extended attributes are meta-information about the object, and 
> >> streams
> >> are contained within the object.
> >
> > Streams are only within the inode if they are tiny, otherwise they are 
> > stored indirect just like normal file data. What they contain is 
> > complete specific to the creator. Same is valid for EAs, with the 
> > exception that all EAs are stored as one "stream" (for lack of a 
> > better word). 
> 
> I miss the point of the implementation details cited above.

You said that EAs contain meta-info and streams are contained within the
object (not sure what you mean there but anyway) and I was saying that
that is not true, even if in somewhat unclear words.

> >> In this case, a naming convention is quite sufficient to distinguish 
> >> them.
> >
> > Still think so? 
> 
> Yes.
> 
> > I don't.
> >
> >> Extended attributes can have names of the form filenameA/..extone.
> >>
> >> Streams can have names of the form filenameA/streamone.
> >>
> >> In other words, all meta-information about an object should by 
> >> convention
> >> (and only by convention, because people should live free, and because
> >> there is not always an obvious distinction between meta and contained
> >> information) be preceded by '..'
> >>
> >> Note that readdir should return neither stream names nor extended 
> >> attribute names,
> >> and the use of 'hidden' directory entries accomplishes this (ala 
> >> .snapshot
> >> for WAFL).
> >> *
> >> **
> >>   **
> >
> >
> > All the below quotes refering to *Curtis are actually from me, IIRC...
> >
> >> *Curtis:
> >> You can't possibly have both using the same API since you would then 
> >> get name collision on filesystems where both named streams and EAs 
> >> are supported.
> >> *
> >>   **
> >> **
> >>
> >> *Name distinctions are what you use to avoid name collisions, see above.
> >> *
> >
> >
> > Ok, that would work, BUT:
> >
> > (Again this is me not Curtis...)
> >
> >> *Curtis:
> >> (And I haven't even mentioned EAs and named streams attached to 
> >> actual _real_ directories yet.)
> >>
> >> *
> >>   **
> >> *I don't understand this.
> >
> >
> > Ok, I will try to explain. An inode is the real thing, not a file. 
> 
> In reiserfs we say object, and consider files and directories (and 
> symlinks, etc.) to be objects.  We don't have on-disk inodes.   Inodes 
> are implementation layer not semantic layer.  We should be talking about 
> semantic layer here I think.
> 
> > An inode can by definition be a file or a directory (or a symlink, or 
> > special device file, etc).
> >
> > Any of these (i.e. any inode) can have both named streams AND EAs 
> > attached to them on NTFS. So say I have a directory named MyDir and it 
> > contains a named stream called MyStream1 and an EA called MyEA1 and 
> > two files, one called MyStream1 and one called "..MyEA1".
> >
> > Now with your scheme of naming things, looking up MyDir/MyStream1 
> > matches both the file MyStream1 that is in the directory MyDir and the 
> > named stream MyStream1 belonging to the directory MyDir. - How do 
> > you/does one distinguish the two in your scheme?!? I can only see it 
> > makind a big BANG here... 
> 
> Well, gosh, okay, maybe you want to prepend ',,' to streams and '..' to 
> extended attributes.  I personally think Linux would only want to do so 
> when used as a fileserver emulating NTFS/SAMBA.  There is no enhancement 
> of user functionality from doing it for general purpose filesystems. 

Just wait until this functionality is available and watch all GUI things
start to use it en masse! I don't doubt that GNOME/KDE/replace with your
favourite window manager are going to hesitate to start putting in the
icon, the name, and whatnot inside EAs or inside named streams the instant
they are ubiquitously available and I think that makes a lot of sense too.
No doubt I will get flamed for saying this but all flames go to
/dev/null...

Both MacOS and as of recently Windows do this kind of stuff, too, and it
can't be long before Linux goes the same way, provided file systems
support the required features (i.e. EAs and/or named streams) so I
disagree with you this is only a compatibility thing. It might start out
as one but it will find real world applications very quickly...

>  Feel free to substitute anything you like for ',,', the choice of 
> naming convention is not the point.  You could even use ':':-).
> 
> It is important though that you not require ',,', ':', or '..' to have 
> these special meanings for all Linux namespaces, I hope that is understood.

The problem with making this flexible is that how does a user space
application find out what the current separators are? Will you be
introducing a get_name_spaces_of_this_inode syscall that needs to be
called on each inode before accessing EAs? And what if someone changes it
halfway through while you are reading EAs? That way lie dragons IMHO.

The proposed EA API which accesses EAs as EAs and not as files doesn't
suffer from any such problems.

> > Similarly, looking up MyDir/..MyEA1 matches both the file named 
> > "..MyEA1" and the EA MyEA1. BANG!
> >
> > And add a named stream actually named "..MyEA1" to MyDir and you have 
> > total salad!
> >
> > See the problem now? 
> 
> No, see above.
> 
> > I certainly fail to see how your naming scheme is going to cope with 
> > this... Perhaps I am missing something? 
> 
> Naming conventions are easy.  See above.

How so? You are begging the question by just saying it is easy. Please
answer what you do when there is a clash, which is bound to happen
eventually especially if you make the name space prefixes flexible.

Does the user just experience undefined behaviour? That would be
unacceptable IMHO.

> > Now if you have distinct APIs for EAs you have no problems on that 
> > side and if you don't use the slash but say the colon (like Windows 
> > does) for named streams you get rid of the named streams in 
> > directories problem, too. But then you need to forbid the ":" as an 
> > accepted character in the file name just like Windows does which is 
> > probably a reason not to use that API either...
> >
> >> *
> >> **
> >>   **
> >>
> >> *Curtis:
> >> Let's face it: EAs exist. They are _not_ files/directories so the API
> >> *
> >>   **
> >> *Is this an argument?
> >>
> >> EA's do not exist in Linux, and they should never exist as something 
> >> that is more than a file. Since they do not exist, you might as well 
> >> improve the filesystems you port to Linux while porting them.  APIs 
> >> shape an OS over the long term, and if done wrong they burden 
> >> generations after you with crud.
> >
> >
> > Like Microsoft is going to let me change the NTFS specifications to 
> > modify how EAs and named streams are stored. Dream on!
> >
> > But perhaps we are talking past each other: I am talking on-disk 
> > format / specifications. 
> 
> 
> I am NOT talking about on-disk format, I am talking about APIs and 
> naming conventions.  On disk format is entirely FS specific.  Live free 
> (errr, no, you are doing NTFS, live confined;-) ).....
> 
> > These exist and no, we cannot change those at all. You can do that 
> > with reiserfs as it is yours but all of us supporting existing file 
> > systems owned by corporations like Microsoft, SGI, etc, have to live 
> > with the specifications.
> >
> >> *
> >> **
> >>   **
> >> *Curtis:
> >> should not make them appear as files/directories. - You have to 
> >> consider that there are a lot of filesystems out there which are 
> >> already developed and which need to be supported. - Not everyone has 
> >> their own filesystem which they can change/extend the 
> >> specifications/implementation of at will.
> >> *
> >>   **
> >> *
> >> Yes they do.  It is all GPL'd.  Even XFS.  Do the underlying 
> >> infrastructure
> >> the right way, and I bet you'll be surprised at how little need there 
> >> really
> >> is for ea's done the wrong way. A user space library can cover
> >> over it all (causing only the obsolete programs using it to suffer 
> >> while they
> >> wait to fade away).
> >
> >
> > ?!? GPL has nothing to do with on-disk format and I doubt Microsoft 
> > would agree that the ntfs on-disk layout is GPL. It's a trade secret! 
> > Why do you think ntfs developers have to spend half their life using 
> > disassemblers and hexeditors?!? 
> 
> Did you file comments in the various MS legal battles going on?  You 
> should.....  (I know, there is only a small chance it will have an 
> effect, but..... )  they should be required to give you the info, and if 
> you don't demand it I bet they won't be so required.  Did you notice how 
> they are restricting things to only persons with a viable business in 
> the opinion of MS?

They are restricting a lot of things. From what I have seen so far they
will end up not showing anything to anybody at all. This settlement is
complete garbage. But we are getting off topic...

> >> *
> >> What would have happened if set theory had not just sets and 
> >> elements, but sets, elements, extended-attributes, and streams, and 
> >> you could not use the same operators on streams that you use on 
> >> elements?  It would have been crap as a theoretical model.  It does 
> >> real damage when you add things that require different operators to 
> >> the set of primitives. Closure is extremely important to design.  
> >> Don't do this.
> > Since we are going into analogies: You don't use a hammer to affix a 
> > screw and neither do you use a screwdriver to affix a nail...at least 
> > I don't. I think you are trying to use a large sledge hammer to put 
> > together things which do not fit together thus breaking them in the 
> > process. To use your own words: Don't do this. (-; Each is distinct 
> > and should be treated as such. </me ducks>
> >
> Programs will get written to use your API, and not work with reiserfs, 
> and will get written to use our API and not work with NTFS, and this is 
> bad....

Now that is true. And yes, it is bad. However it will be up to the
community to decide which API to use and at the moment there are several
fs using the "bestbits" API and only reiserfs (?) the "reiserfs" one...
And we all know from our very own $Deity that we don't design software, we
just write things and let evolution decide which is better. (((-;

> Thanks for the FS driver by the way, it is very useful to us dual-booters.

Thanks. (-: It is indeed, I being one of the dual-booters as well. (-:

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Hans Reiser]

Anton Altaparmakov wrote:

> Hi Hans,
>
> At 12:02 11/12/01, Hans Reiser wrote:
>
>>  I respond below.
>>
>> I didn't see that email, probably because I was not on the cc list.
>>
>> Nathan Scott wrote:
>>
>>> hi Hans,
>>> On Sat, Dec 08, 2001 at 11:17:21PM +0300, Hans Reiser wrote:
>>>
>>>> Nathan Scott wrote:
>>>>
>>>>> In a way there's consensus wrt how to do POSIX ACLs on Linux
>>>>> now, as both the ext2/ext3 and XFS ACL projects will be using
>>>>> the same tools, libraries, etc.  In terms of other ACL types,
>>>>> I don't know of anyone actively working on any.
>>>>
>>>> We are taking a very different approach to EAs (and thus to ACLs) 
>>>> as described in brief at www.namesys.com/v4/v4.html.  We don't 
>>>> expect anyone to take us seriously on it before it works, but 
>>>> silence while coding does not equal consensus.;-)
>>>>
>>>> In essence, we think that if a file can't do what an EA can do, 
>>>> then you need to make files able to do more.
>>>
>>> We did read through your page awhile ago.  It wasn't clear to me
>>> how you were addressing Anton's questions here:
>>> http://marc.theaimsgroup.com/?l=linux-fsdevel&m=97260371413867&w=2
>>> (I couldn't find a reply in the archive, but may have missed it).
>>>
>>> We were concentrating on something that could be fs-independent,
>>> so the lack of answers there put us off a bit, and the dependence
>>> on a reiser4() syscall is pretty filesystem-specific too (I guess
>>> if your solution is intended to be a reiserfs-specific one, then
>>> the questions above are meaningless).
>>
>> Changing the name of the system call is not a biggie.  Our approach 
>> is to make
>> it work for reiserfs, then proselytize.  While we work, we let people 
>> know
>> what we are working on, and if they join in, great to have it work 
>> for more
>> than one FS.
>>
>>> I was curious on another thing also - in the section titled ``The 
>>> Usual Resolution Of These Flaws Is A One-Off Solution'',
>>> talking about security attributes interfaces, your page says:
>>>
>>>         "Linus said that we can have a system call to use as 
>>> our*experimental plaything in this. With what I have in mind for the
>>> API, one rather flexible system call is all we want..."
>>>
>>> How did you manage to get him to say that?  We were flamed for
>>> suggesting a syscall which multiplexed all extended attributes
>>> commands though the one interface (because its semantics were
>>> not clearly defined & it could be extended with new commands,
>>> like ioctl/quotactl/...), and we've also had no luck so far in
>>> getting either our original interface, nor any revised syscall
>>> interfaces (which aren't like that anymore) accepted by Linus.*
>>
>>
>> We expect to get flamed once we have a patch.;-)  When we
>> have something mature enough to be usable, I expect he'll find a lot 
>> that
>> could be made better.  He does that.;-)
>>
>> For us, there are semantic advantages to having a single system call. 
>> Probably
>> it will get a lot of argument once we have working code, and frankly 
>> I prefer
>> to have that argument only after it is something usable, and it is 
>> easy to see
>> the convenience of expression that comes from it.  We want to Linux 
>> to be
>> MORE expressive than BeOS in regards to files.
>>
>>> *
>>> many thanks.
>>> *
>>> **
>>
>>   **
>> *
>> Curtis Anderson wrote:
>>
>>> > The problem with streams-style attributes comes from stepping onto 
>>> the
>>> > slippery slope of trying to put too much generality into it.  I 
>>> chose the
>>> > block-access style of API so that there would be no temptation to 
>>> start
>>> > down that slope.
>>>
>>> I understand you right up until this.  I just don't get it.  If you 
>>> extend the functionality of files and directories so that attributes 
>>> are not needed, this is goodness, right?  I sure think it is the 
>>> right approach.  We should just decompose carefully what 
>>> functionality is provided by attributes that files and directories 
>>> lack, and one feature at a time add that capability to files and 
>>> directories as separate optional features.
>>
>
> I wrote:
>
>> No, it is _not_ goodness, IMHO. - If you did implement the API for 
>> attributes through files and directories, then what would you do with 
>> named streams?!?
>> *
>>   **
>> **
>>
>> *Hans Reiser wrote:
>>
>> What is your intended functional difference between extended 
>> attributes and streams?
>>
>> None?
>
>
> Differences in NTFS:
>
> - maximum size (EA limited to 64kiB, named stream 2^63 bytes) 

These are desirable limits to preserve?  For sure?  If so, then a 
particular plugin can be written to restrict files to 64k, though I 
shake my head at the thought.

>
> - locality of storage (all EAs are stored in one so they are quicker 
> to access when you need to access multiple EAs) 

arbitrarily aggregating files is a useful feature, there is no reason to 
rigidly offer and require the feature for EAs only.

>
> - name namespace (Unicode names for named streams vs ASCII for EAs) 

Namespaces can be changed for the children of directories.  Plan9 guys 
have done such things, and it is cool.

>
> - potential ability to compress/encrypt (EAs cannot do this, named 
> streams could possibly and they certainly can be sparse, too which EAs 
> cannot be) 

Well, I suppose you could allow restricting some files to not have 
compression and sparseness, though it isn't exciting to me.

>
> - named streams have creation/modification/access/etc times associated 
> with them, EAs don't 

I thought streams shared the stat data of the parent file?  

Regardless, files should be able to share/inherit stat data.

>
>
> How is that for a start? 

Not one reason cited is convincing to me.

>
>
>> Ok, let's assume none until I get your response.  (I can respond more 
>> specifically
>> after you correct me.)  Let me further go out on a limb,and guess 
>> that you intend
>> that extended attributes are meta-information about the object, and 
>> streams
>> are contained within the object.
>
>
> Streams are only within the inode if they are tiny, otherwise they are 
> stored indirect just like normal file data. What they contain is 
> complete specific to the creator. Same is valid for EAs, with the 
> exception that all EAs are stored as one "stream" (for lack of a 
> better word). 

I miss the point of the implementation details cited above.

>
>
>> In this case, a naming convention is quite sufficient to distinguish 
>> them.
>
>
> Still think so? 

Yes.

> I don't.
>
>> Extended attributes can have names of the form filenameA/..extone.
>>
>> Streams can have names of the form filenameA/streamone.
>>
>> In other words, all meta-information about an object should by 
>> convention
>> (and only by convention, because people should live free, and because
>> there is not always an obvious distinction between meta and contained
>> information) be preceded by '..'
>>
>> Note that readdir should return neither stream names nor extended 
>> attribute names,
>> and the use of 'hidden' directory entries accomplishes this (ala 
>> .snapshot
>> for WAFL).
>> *
>> **
>>   **
>
>
> All the below quotes refering to *Curtis are actually from me, IIRC...
>
>> *Curtis:
>> You can't possibly have both using the same API since you would then 
>> get name collision on filesystems where both named streams and EAs 
>> are supported.
>> *
>>   **
>> **
>>
>> *Name distinctions are what you use to avoid name collisions, see above.
>> *
>
>
> Ok, that would work, BUT:
>
> (Again this is me not Curtis...)
>
>> *Curtis:
>> (And I haven't even mentioned EAs and named streams attached to 
>> actual _real_ directories yet.)
>>
>> *
>>   **
>> *I don't understand this.
>
>
> Ok, I will try to explain. An inode is the real thing, not a file. 

In reiserfs we say object, and consider files and directories (and 
symlinks, etc.) to be objects.  We don't have on-disk inodes.   Inodes 
are implementation layer not semantic layer.  We should be talking about 
semantic layer here I think.

> An inode can by definition be a file or a directory (or a symlink, or 
> special device file, etc).
>
> Any of these (i.e. any inode) can have both named streams AND EAs 
> attached to them on NTFS. So say I have a directory named MyDir and it 
> contains a named stream called MyStream1 and an EA called MyEA1 and 
> two files, one called MyStream1 and one called "..MyEA1".
>
> Now with your scheme of naming things, looking up MyDir/MyStream1 
> matches both the file MyStream1 that is in the directory MyDir and the 
> named stream MyStream1 belonging to the directory MyDir. - How do 
> you/does one distinguish the two in your scheme?!? I can only see it 
> makind a big BANG here... 


Well, gosh, okay, maybe you want to prepend ',,' to streams and '..' to 
extended attributes.  I personally think Linux would only want to do so 
when used as a fileserver emulating NTFS/SAMBA.  There is no enhancement 
of user functionality from doing it for general purpose filesystems. 
 Feel free to substitute anything you like for ',,', the choice of 
naming convention is not the point.  You could even use ':':-).


It is important though that you not require ',,', ':', or '..' to have 
these special meanings for all Linux namespaces, I hope that is understood.

>
>
> Similarly, looking up MyDir/..MyEA1 matches both the file named 
> "..MyEA1" and the EA MyEA1. BANG!
>
> And add a named stream actually named "..MyEA1" to MyDir and you have 
> total salad!
>
> See the problem now? 

No, see above.

>
>
> I certainly fail to see how your naming scheme is going to cope with 
> this... Perhaps I am missing something? 

Naming conventions are easy.  See above.

>
>
> Now if you have distinct APIs for EAs you have no problems on that 
> side and if you don't use the slash but say the colon (like Windows 
> does) for named streams you get rid of the named streams in 
> directories problem, too. But then you need to forbid the ":" as an 
> accepted character in the file name just like Windows does which is 
> probably a reason not to use that API either...
>
>> *
>> **
>>   **
>>
>> *Curtis:
>> Let's face it: EAs exist. They are _not_ files/directories so the API
>> *
>>   **
>> *Is this an argument?
>>
>> EA's do not exist in Linux, and they should never exist as something 
>> that is more than a file. Since they do not exist, you might as well 
>> improve the filesystems you port to Linux while porting them.  APIs 
>> shape an OS over the long term, and if done wrong they burden 
>> generations after you with crud.
>
>
> Like Microsoft is going to let me change the NTFS specifications to 
> modify how EAs and named streams are stored. Dream on!
>
> But perhaps we are talking past each other: I am talking on-disk 
> format / specifications. 


I am NOT talking about on-disk format, I am talking about APIs and 
naming conventions.  On disk format is entirely FS specific.  Live free 
(errr, no, you are doing NTFS, live confined;-) ).....

> These exist and no, we cannot change those at all. You can do that 
> with reiserfs as it is yours but all of us supporting existing file 
> systems owned by corporations like Microsoft, SGI, etc, have to live 
> with the specifications.
>
>> *
>> **
>>   **
>> *Curtis:
>> should not make them appear as files/directories. - You have to 
>> consider that there are a lot of filesystems out there which are 
>> already developed and which need to be supported. - Not everyone has 
>> their own filesystem which they can change/extend the 
>> specifications/implementation of at will.
>> *
>>   **
>> *
>> Yes they do.  It is all GPL'd.  Even XFS.  Do the underlying 
>> infrastructure
>> the right way, and I bet you'll be surprised at how little need there 
>> really
>> is for ea's done the wrong way. A user space library can cover
>> over it all (causing only the obsolete programs using it to suffer 
>> while they
>> wait to fade away).
>
>
> ?!? GPL has nothing to do with on-disk format and I doubt Microsoft 
> would agree that the ntfs on-disk layout is GPL. It's a trade secret! 
> Why do you think ntfs developers have to spend half their life using 
> disassemblers and hexeditors?!? 

Did you file comments in the various MS legal battles going on?  You 
should.....  (I know, there is only a small chance it will have an 
effect, but..... )  they should be required to give you the info, and if 
you don't demand it I bet they won't be so required.  Did you notice how 
they are restricting things to only persons with a viable business in 
the opinion of MS?

>
>
>> *
>> What would have happened if set theory had not just sets and 
>> elements, but sets, elements, extended-attributes, and streams, and 
>> you could not use the same operators on streams that you use on 
>> elements?  It would have been crap as a theoretical model.  It does 
>> real damage when you add things that require different operators to 
>> the set of primitives. Closure is extremely important to design.  
>> Don't do this.
>
>
> Since we are going into analogies: You don't use a hammer to affix a 
> screw and neither do you use a screwdriver to affix a nail...at least 
> I don't. I think you are trying to use a large sledge hammer to put 
> together things which do not fit together thus breaking them in the 
> process. To use your own words: Don't do this. (-; Each is distinct 
> and should be treated as such. </me ducks>
>
> Best regards,
>
> Anton
>
>
Programs will get written to use your API, and not work with reiserfs, 
and will get written to use our API and not work with NTFS, and this is 
bad....

Thanks for the FS driver by the way, it is very useful to us dual-booters.

Hans

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Anton Altaparmakov]

Hi Hans,

At 12:02 11/12/01, Hans Reiser wrote:
>  I respond below.
>
>I didn't see that email, probably because I was not on the cc list.
>
>Nathan Scott wrote:
>>hi Hans,
>>On Sat, Dec 08, 2001 at 11:17:21PM +0300, Hans Reiser wrote:
>>>Nathan Scott wrote:
>>>>In a way there's consensus wrt how to do POSIX ACLs on Linux
>>>>now, as both the ext2/ext3 and XFS ACL projects will be using
>>>>the same tools, libraries, etc.  In terms of other ACL types,
>>>>I don't know of anyone actively working on any.
>>>We are taking a very different approach to EAs (and thus to ACLs) as 
>>>described in brief at www.namesys.com/v4/v4.html.  We don't expect 
>>>anyone to take us seriously on it before it works, but silence while 
>>>coding does not equal consensus.;-)
>>>
>>>In essence, we think that if a file can't do what an EA can do, then you 
>>>need to make files able to do more.
>>We did read through your page awhile ago.  It wasn't clear to me
>>how you were addressing Anton's questions here:
>>http://marc.theaimsgroup.com/?l=linux-fsdevel&m=97260371413867&w=2
>>(I couldn't find a reply in the archive, but may have missed it).
>>
>>We were concentrating on something that could be fs-independent,
>>so the lack of answers there put us off a bit, and the dependence
>>on a reiser4() syscall is pretty filesystem-specific too (I guess
>>if your solution is intended to be a reiserfs-specific one, then
>>the questions above are meaningless).
>Changing the name of the system call is not a biggie.  Our approach is to make
>it work for reiserfs, then proselytize.  While we work, we let people know
>what we are working on, and if they join in, great to have it work for more
>than one FS.
>
>>I was curious on another thing also - in the section titled ``The Usual 
>>Resolution Of These Flaws Is A One-Off Solution'',
>>talking about security attributes interfaces, your page says:
>>
>>         "Linus said that we can have a system call to use as 
>> our*experimental plaything in this. With what I have in mind for the
>>API, one rather flexible system call is all we want..."
>>
>>How did you manage to get him to say that?  We were flamed for
>>suggesting a syscall which multiplexed all extended attributes
>>commands though the one interface (because its semantics were
>>not clearly defined & it could be extended with new commands,
>>like ioctl/quotactl/...), and we've also had no luck so far in
>>getting either our original interface, nor any revised syscall
>>interfaces (which aren't like that anymore) accepted by Linus.*
>
>We expect to get flamed once we have a patch.;-)  When we
>have something mature enough to be usable, I expect he'll find a lot that
>could be made better.  He does that.;-)
>
>For us, there are semantic advantages to having a single system call. Probably
>it will get a lot of argument once we have working code, and frankly I prefer
>to have that argument only after it is something usable, and it is easy to see
>the convenience of expression that comes from it.  We want to Linux to be
>MORE expressive than BeOS in regards to files.
>
>>*
>>many thanks.
>>*
>>**
>   **
>*
>Curtis Anderson wrote:
>
>> > The problem with streams-style attributes comes from stepping onto the
>> > slippery slope of trying to put too much generality into it.  I chose the
>> > block-access style of API so that there would be no temptation to start
>> > down that slope.
>>
>>I understand you right up until this.  I just don't get it.  If you 
>>extend the functionality of files and directories so that attributes are 
>>not needed, this is goodness, right?  I sure think it is the right 
>>approach.  We should just decompose carefully what functionality is 
>>provided by attributes that files and directories lack, and one feature 
>>at a time add that capability to files and directories as separate 
>>optional features.

I wrote:

>No, it is _not_ goodness, IMHO. - If you did implement the API for 
>attributes through files and directories, then what would you do with 
>named streams?!?
>*
>   **
>**
>
>*Hans Reiser wrote:
>
>What is your intended functional difference between extended attributes 
>and streams?
>
>None?

Differences in NTFS:

- maximum size (EA limited to 64kiB, named stream 2^63 bytes)
- locality of storage (all EAs are stored in one so they are quicker to 
access when you need to access multiple EAs)
- name namespace (Unicode names for named streams vs ASCII for EAs)
- potential ability to compress/encrypt (EAs cannot do this, named streams 
could possibly and they certainly can be sparse, too which EAs cannot be)
- named streams have creation/modification/access/etc times associated with 
them, EAs don't

How is that for a start?

>Ok, let's assume none until I get your response.  (I can respond more 
>specifically
>after you correct me.)  Let me further go out on a limb,and guess that you 
>intend
>that extended attributes are meta-information about the object, and streams
>are contained within the object.

Streams are only within the inode if they are tiny, otherwise they are 
stored indirect just like normal file data. What they contain is complete 
specific to the creator. Same is valid for EAs, with the exception that all 
EAs are stored as one "stream" (for lack of a better word).

>In this case, a naming convention is quite sufficient to distinguish them.

Still think so? I don't.

>Extended attributes can have names of the form filenameA/..extone.
>
>Streams can have names of the form filenameA/streamone.
>
>In other words, all meta-information about an object should by convention
>(and only by convention, because people should live free, and because
>there is not always an obvious distinction between meta and contained
>information) be preceded by '..'
>
>Note that readdir should return neither stream names nor extended 
>attribute names,
>and the use of 'hidden' directory entries accomplishes this (ala .snapshot
>for WAFL).
>*
>**
>   **

All the below quotes refering to *Curtis are actually from me, IIRC...

>*Curtis:
>You can't possibly have both using the same API since you would then get 
>name collision on filesystems where both named streams and EAs are supported.
>*
>   **
>**
>
>*Name distinctions are what you use to avoid name collisions, see above.
>*

Ok, that would work, BUT:

(Again this is me not Curtis...)

>*Curtis:
>(And I haven't even mentioned EAs and named streams attached to actual 
>_real_ directories yet.)
>
>*
>   **
>*I don't understand this.

Ok, I will try to explain. An inode is the real thing, not a file. An inode 
can by definition be a file or a directory (or a symlink, or special device 
file, etc).

Any of these (i.e. any inode) can have both named streams AND EAs attached 
to them on NTFS. So say I have a directory named MyDir and it contains a 
named stream called MyStream1 and an EA called MyEA1 and two files, one 
called MyStream1 and one called "..MyEA1".

Now with your scheme of naming things, looking up MyDir/MyStream1 matches 
both the file MyStream1 that is in the directory MyDir and the named stream 
MyStream1 belonging to the directory MyDir. - How do you/does one 
distinguish the two in your scheme?!? I can only see it makind a big BANG 
here...

Similarly, looking up MyDir/..MyEA1 matches both the file named "..MyEA1" 
and the EA MyEA1. BANG!

And add a named stream actually named "..MyEA1" to MyDir and you have total 
salad!

See the problem now?

I certainly fail to see how your naming scheme is going to cope with 
this... Perhaps I am missing something?

Now if you have distinct APIs for EAs you have no problems on that side and 
if you don't use the slash but say the colon (like Windows does) for named 
streams you get rid of the named streams in directories problem, too. But 
then you need to forbid the ":" as an accepted character in the file name 
just like Windows does which is probably a reason not to use that API either...

>*
>**
>   **
>
>*Curtis:
>Let's face it: EAs exist. They are _not_ files/directories so the API
>*
>   **
>*Is this an argument?
>
>EA's do not exist in Linux, and they should never exist as something that 
>is more than a file. Since they do not exist, you might as well improve 
>the filesystems you port to Linux while porting them.  APIs shape an OS 
>over the long term, and if done wrong they burden generations after you 
>with crud.

Like Microsoft is going to let me change the NTFS specifications to modify 
how EAs and named streams are stored. Dream on!

But perhaps we are talking past each other: I am talking on-disk format / 
specifications. These exist and no, we cannot change those at all. You can 
do that with reiserfs as it is yours but all of us supporting existing file 
systems owned by corporations like Microsoft, SGI, etc, have to live with 
the specifications.

>*
>**
>   **
>*Curtis:
>should not make them appear as files/directories. - You have to consider 
>that there are a lot of filesystems out there which are already developed 
>and which need to be supported. - Not everyone has their own filesystem 
>which they can change/extend the specifications/implementation of at will.
>*
>   **
>*
>Yes they do.  It is all GPL'd.  Even XFS.  Do the underlying infrastructure
>the right way, and I bet you'll be surprised at how little need there really
>is for ea's done the wrong way. A user space library can cover
>over it all (causing only the obsolete programs using it to suffer while they
>wait to fade away).

?!? GPL has nothing to do with on-disk format and I doubt Microsoft would 
agree that the ntfs on-disk layout is GPL. It's a trade secret! Why do you 
think ntfs developers have to spend half their life using disassemblers and 
hexeditors?!?

>*
>What would have happened if set theory had not just sets and elements, but 
>sets, elements, extended-attributes, and streams, and you could not use 
>the same operators on streams that you use on elements?  It would have 
>been crap as a theoretical model.  It does real damage when you add things 
>that require different operators to the set of primitives. Closure is 
>extremely important to design.  Don't do this.

Since we are going into analogies: You don't use a hammer to affix a screw 
and neither do you use a screwdriver to affix a nail...at least I don't. I 
think you are trying to use a large sledge hammer to put together things 
which do not fit together thus breaking them in the process. To use your 
own words: Don't do this. (-; Each is distinct and should be treated as 
such. </me ducks>

Best regards,

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Hans Reiser]

  I respond below.

I didn't see that email, probably because I was not on the cc list.

Nathan Scott wrote:

>hi Hans,
>
>On Sat, Dec 08, 2001 at 11:17:21PM +0300, Hans Reiser wrote:
>
>>Nathan Scott wrote:
>>
>>>In a way there's consensus wrt how to do POSIX ACLs on Linux
>>>now, as both the ext2/ext3 and XFS ACL projects will be using
>>>the same tools, libraries, etc.  In terms of other ACL types,
>>>I don't know of anyone actively working on any.
>>>
>>We are taking a very different approach to EAs (and thus to ACLs) as 
>>described in brief at www.namesys.com/v4/v4.html.  We don't expect 
>>anyone to take us seriously on it before it works, but silence while 
>>coding does not equal consensus.;-)
>>
>>In essence, we think that if a file can't do what an EA can do, then you 
>>need to make files able to do more.
>>
>
>We did read through your page awhile ago.  It wasn't clear to me
>how you were addressing Anton's questions here:
>http://marc.theaimsgroup.com/?l=linux-fsdevel&m=97260371413867&w=2
>(I couldn't find a reply in the archive, but may have missed it).  
>
>We were concentrating on something that could be fs-independent,
>so the lack of answers there put us off a bit, and the dependence
>on a reiser4() syscall is pretty filesystem-specific too (I guess
>if your solution is intended to be a reiserfs-specific one, then
>the questions above are meaningless).
>
Changing the name of the system call is not a biggie.  Our approach is 
to make
it work for reiserfs, then proselytize.  While we work, we let people know
what we are working on, and if they join in, great to have it work for more
than one FS.

>
>
>I was curious on another thing also - in the section titled 
>``The Usual Resolution Of These Flaws Is A One-Off Solution'',
>talking about security attributes interfaces, your page says:
>
>	"Linus said that we can have a system call to use as our*experimental plaything in this. With what I have in mind for the
>API, one rather flexible system call is all we want..."
>
>How did you manage to get him to say that?  We were flamed for
>suggesting a syscall which multiplexed all extended attributes
>commands though the one interface (because its semantics were
>not clearly defined & it could be extended with new commands,
>like ioctl/quotactl/...), and we've also had no luck so far in
>getting either our original interface, nor any revised syscall
>interfaces (which aren't like that anymore) accepted by Linus.*
>

We expect to get flamed once we have a patch.;-)  When we
have something mature enough to be usable, I expect he'll find a lot that
could be made better.  He does that.;-)

For us, there are semantic advantages to having a single system call. 
 Probably
it will get a lot of argument once we have working code, and frankly I 
prefer
to have that argument only after it is something usable, and it is easy 
to see
the convenience of expression that comes from it.  We want to Linux to be
MORE expressive than BeOS in regards to files.

>*
>
>many thanks.
>
>*
>
> **

**

    **

*
Curtis Anderson wrote:

> > The problem with streams-style attributes comes from stepping onto the
> > slippery slope of trying to put too much generality into it.  I chose the
> > block-access style of API so that there would be no temptation to start
> > down that slope.
>
>I understand you right up until this.  I just don't get it.  If you extend 
>the functionality of files and directories so that attributes are not 
>needed, this is goodness, right?  I sure think it is the right 
>approach.  We should just decompose carefully what functionality is 
>provided by attributes that files and directories lack, and one feature at 
>a time add that capability to files and directories as separate optional 
>features.

No, it is _not_ goodness, IMHO. - If you did implement the API for 
attributes through files and directori
es, then what would you do with named 
streams?!?
*

    **

**

*Hans Reiser wrote:

What is your intended functional difference between extended attributes and streams?

None?

Ok, let's assume none until I get your response.  (I can respond more specifically
after you correct me.)  Let me further go out on a limb,and guess that you intend
that extended attributes are meta-information about the object, and streams
are contained within the object.

In this case, a naming convention is quite sufficient to distinguish them.

Extended attributes can have names of the form filenameA/..extone.

Streams can have names of the form filenameA/streamone.

In other words, all meta-information about an object should by convention
(and only by convention, because people should live free, and because
there is not always an obvious distinction between meta and contained
information) be preceded by '..'

Note that readdir should return neither stream names nor extended attribute names,
and the use of 'hidden' directory entries accomplishes this (ala .snapshot
for WAFL).
*

**

    **

*Curtis:
You can't possibly have both using the same API since you would then get 
name collision on filesystems where both named streams and EAs are 
supported. 

*

    **

**

*Name distinctions are what you use to avoid name collisions, see above.
*

**

    **

*Curtis:
(And I haven't even mentioned EAs and named streams attached to 
actual _real_ directories yet.)

*

    **

*I don't understand this.
*

**

    **

*Curtis:
Let's face it: EAs exist. They are _not_ files/directories so the API 

*

    **

*Is this an argument?

EA's do not exist in Linux, and they should never exist as something that
is more than a file. Since they do not
exist, you might as well improve the filesystems you port to Linux while
porting them.  APIs shape an OS over the long term, and if done wrong
they burden generations after you with crud.
*

**

    **

*Curtis:
should not make them appear as files/directories. - You have to consider 
that there are a lot of filesystems out there which are already developed 
and which need to be supported. - Not everyone has their own filesystem 
which they can change/extend the specifications/implementation of at will.
*

    **

*
Yes they do.  It is all GPL'd.  Even XFS.  Do the underlying infrastructure
the right way, and I bet you'll be surprised at how little need there really
is for ea's done the wrong way. A user space library can cover
over it all (causing only the obsolete programs using it to suffer while they
wait to fade away).
*



What would have happened if set theory had not just sets and elements, 
but sets, elements, extended-attributes, and streams, and you could not 
use the same operators on streams that you use on elements?  It would 
have been crap as a theoretical model.  It does real damage when you add 
things that require different operators to the set of primitives. 
 Closure is extremely important to design.  Don't do this.

Hans

    **

    **

*
*

reiser4 (was Re: [PATCH] Revised extended attributes interface)

[Nathan Scott]

hi Hans,

On Sat, Dec 08, 2001 at 11:17:21PM +0300, Hans Reiser wrote:
> Nathan Scott wrote:
> >
> >In a way there's consensus wrt how to do POSIX ACLs on Linux
> >now, as both the ext2/ext3 and XFS ACL projects will be using
> >the same tools, libraries, etc.  In terms of other ACL types,
> >I don't know of anyone actively working on any.
> >
> We are taking a very different approach to EAs (and thus to ACLs) as 
> described in brief at www.namesys.com/v4/v4.html.  We don't expect 
> anyone to take us seriously on it before it works, but silence while 
> coding does not equal consensus.;-)
> 
> In essence, we think that if a file can't do what an EA can do, then you 
> need to make files able to do more.

We did read through your page awhile ago.  It wasn't clear to me
how you were addressing Anton's questions here:
http://marc.theaimsgroup.com/?l=linux-fsdevel&m=97260371413867&w=2
(I couldn't find a reply in the archive, but may have missed it).  

We were concentrating on something that could be fs-independent,
so the lack of answers there put us off a bit, and the dependence
on a reiser4() syscall is pretty filesystem-specific too (I guess
if your solution is intended to be a reiserfs-specific one, then
the questions above are meaningless).

I was curious on another thing also - in the section titled 
``The Usual Resolution Of These Flaws Is A One-Off Solution'',
talking about security attributes interfaces, your page says:

	"Linus said that we can have a system call to use as our
experimental plaything in this. With what I have in mind for the
API, one rather flexible system call is all we want..."

How did you manage to get him to say that?  We were flamed for
suggesting a syscall which multiplexed all extended attributes
commands though the one interface (because its semantics were
not clearly defined & it could be extended with new commands,
like ioctl/quotactl/...), and we've also had no luck so far in
getting either our original interface, nor any revised syscall
interfaces (which aren't like that anymore) accepted by Linus.

many thanks.

-- 
Nathan