-mm -> 2.6.13 merge status

-mm -> 2.6.13 merge status

[Andrew Morton]

This summarises my current thinking on various patches which are presently
in -mm.  I cover large things and small-but-controversial things.  Anything
which isn't covered here (and that's a lot of material) is probably a "will
merge", unless it obviously isn't.

(If you reply to this email it would be a good idea to alter the Subject:
to reflect which feature you are discussing)



git-ocfs

    The OCFS2 filesystem.  OK by me, although I'm not sure it's had enough
    review.

sparsemem

    OK by me for a merge.  Need to poke arch maintainers first, check that
    they've looked at it sufficiently closely.

vm-early-zone-reclaim

    Needs some convincing benchmark numbers to back it up.  Otherwise OK.

avoiding-mmap-fragmentation

    Tricky.  Addresses vm area fragmentation issues due to recent
    optimisations to the free-area lookup code.  Will merge.

periodically-drain-non-local-pagesets

    Will merge

pcibus_to_node and users

    Will merge

CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.

    Will merge (will switch default to 1000 Hz later if that seems necessary)

dmi-*.patch

    Will merge.  I have a comment "The below break x440".  Maybe it got
    fixed.  We'll doubtless hear if not.

xen-*.patch

    These are little cleanups and abstractions which make a Xen merge
    easier.  May as well merge them.

CPU hotplug for x86 and x86_64

    Not really useful on current hardware, but these provide
    infrastructure which some power management patches need, and it seems
    sensible to make the reference architecture support hotplug.  Will merge.

swsusp-on-SMP

    Will merge.

cfq version 3

    Not sure.  Jens seems to be setting up a few git trees.  On hold.

RCUification of the key management code

    Don't know - dhowells seemed diffident last time we discussed this.

timers-fixes-improvements.patch

    SMP speedups for the core timer code.  It was bumpy, but this seems
    stable now.  Will merge.

kprobes-*

    Will merge

rapidio-*

    Will merge.

namespace*.patch

    Awaiting viro ack.

xtensa architecture

    Is xtensa now, or will it be in the future a sufficiently popular
    architecture to justify the cost of having this code in the tree?

    Heaven knows.  Will merge.

dlm-*.patch: Red Hat distributed lock manager

    Hard.  Right now it seems that no in-kernel projects will use this and
    only one out-of-kernel project will use it.  Shelve the problem until
    after Kernel Summit, where some light may be shed.

    Opinions are sought...

connector.patch

    Nice idea IMO, but there are still questions around the
    implementation.  More dialogue needed ;)

connector-add-a-fork-connector.patch

    OK, but needs connector.

inotify

    There are still concerns about the userspace API and internal
    implementation details.  More slogging needed.

pcmcia-*.patch

    Makes the pcmcia layer generate hotplug events and deprecates cardmgr.
    Will merge.

NUMA-aware slab allocator

    Seems stable now, but it needs some ifdef reduction work before
    merging, please.

CPU scheduler

    Will merge some of these patches.  We're still discussing which ones.

perfctr

    Not yet, but getting closer.  The PPC64 guys still need to sort out a
    few interface issues with Mikael.  We might be able to fit this into
    2.6.13 if people get a move on.

cachefs

    This is a ton of code which knows rather a lot about pagecache
    internals.  It allows the AFS client to cache file contents on a local
    blockdev.

    I don't think it's a justified addition for only AFS and I'd prefer to
    see it proven for NFS as well.

    Issues around add-page-becoming-writable-notification.patch need to
    be resolved.

cachefs-for-nfs

    A recent addition.  Needs review from NFS developers and considerably
    more testing.

    These things aren't looking likely for 2.6.13.

kexec and kdump

    I guess we should merge these.

    I'm still concerned that the various device shutdown problems will
    mean that the success rate for crashing kernels is not high enough for
    kdump to be considered a success.  In which case in six months time we'll
    hear rumours about vendors shipping wholly different crashdump
    implementations, which would be quite bad.

    But I think this has gone as far as it can go in -mm, so it's a bit of
    a punt.

reiser4

    Merge it, I guess.

    The patches still contain all the reiser4-specific namespace
    enhancements, only it is disabled, so it is effectively dead code.  Maybe
    we should ask that it actually be removed?

v9fs

    I'm not sure that this has a sufficiently high
    usefulness-to-maintenance-cost ratio.

fuse

    This is useful, but there are, AFAIK, two issues:

    - We're still deadlocked over some permission-checking hacks in there

    - It has an NFS server implementation which only works if the
      to-be-served file happens to be in dcache.

      It has been said that a userspace NFS server can be used to get
      full NFS server functionality with FUSE.  I think the half-assed kernel
      implementation should be done away with.

execute-in-place

    Will merge.  Have the embedded guys commented on the usefulness of
    this for execute-out-of-ROM?

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> fuse
> 
>     This is useful, but there are, AFAIK, two issues:
> 
>     - We're still deadlocked over some permission-checking hacks in there

Oh, god.  Let me try to explain this again:

  - This is a security issue with unprivileged mounts

  - Since no other filesystem currently offers secure unpivileged
    mounts in Linux, this is something "new"

  - Since it's something new, there's a big resistance to acceptance.
    I understand this, I only ask people, to please read
    Documentation/filesystems/fuse.txt, before arguing against it

  - IMO it's not a hack, and it's not something that can be solved
    otherwise (no, private namespaces are NOT a solution, they are
    mosty orthogonal to this).

So I welcome constructive discussion.  However bear in mind, that I
definitely don't want to disable unprivileged mounts.  For me that is
_the_ most important feature of FUSE.

>     - It has an NFS server implementation which only works if the
>       to-be-served file happens to be in dcache.

More preciesly it relies on icache.

>       It has been said that a userspace NFS server can be used to
>       get full NFS server functionality with FUSE.  I think the
>       half-assed kernel implementation should be done away with.

I won't shed many tears if you drop fuse-nfs-export.patch.  It would
at least give the userspace solution some boost.

However the patch is pretty small, and despite it's flaws, I know it's
used by a number of people.

Thanks,
Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Alan Cox]

On Maw, 2005-06-21 at 11:22, Miklos Szeredi wrote:
> So I welcome constructive discussion.  However bear in mind, that I
> definitely don't want to disable unprivileged mounts.  For me that is
> _the_ most important feature of FUSE.

If the choice was "merge FUSE without unpriv mounts for now" or "discard
fuse completely" which is preferable.

It seems to me (just IMHO) that it would be better to merge FUSE without
that feature and then spend the time getting that feature right _in
parallel_ with people using, breaking and reviewing FUSE a lot more.

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> > So I welcome constructive discussion.  However bear in mind, that I
> > definitely don't want to disable unprivileged mounts.  For me that is
> > _the_ most important feature of FUSE.
> 
> If the choice was "merge FUSE without unpriv mounts for now" or "discard
> fuse completely" which is preferable.

FUSE is doing fine outside mainline, so discard wouldn't be such a big
setback.  Including it without unpriv mounts would effectively fork
FUSE into an out-of-tree and an in-tree version, which is sure to
cause confusion.

So yes, I'd prefer not merging to merging without unpriv mounts.  But
it's GPL, so obviously I don't have any legal control over it.

> It seems to me (just IMHO) that it would be better to merge FUSE without
> that feature and then spend the time getting that feature right _in
> parallel_ with people using, breaking and reviewing FUSE a lot more.

The security measure in question is actually very simple (10 lines or
so).  So it's not the implementation that people have problems with
but the concept.  The concept itself is hard to swallow, because it
does something unexpected, but what it does is in fact very logical.

That's why I ask people to read the documentation, think about it and
_then_ argue.  Up till now the discussion with Christoph Hellwig about
this hasn't been on the level of rational arguments (and he's the only
definite naysayer).

Thanks,
Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Pavel Machek]

Hi!

> >     This is useful, but there are, AFAIK, two issues:
> > 
> >     - We're still deadlocked over some permission-checking hacks in there
> 
> Oh, god.  Let me try to explain this again:
> 
>   - This is a security issue with unprivileged mounts

Pretty please, just merge it without  unpriviledged mounts. I see they are
usefull,
but they are too strange for now. If user tries mounting themselves,
he gets -EPERM, and applies 10-liner from you. Does not look like "fork" or
anything serious.
				Pavel

-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: -mm -> 2.6.13 merge status (fuse)

[John Stoffel]

I'd like to see FUSE merged too, even without user mounts, if only to
get more motivated to actually play with this and see how it works.

John

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> I'd like to see FUSE merged too, even without user mounts, if only to
> get more motivated to actually play with this and see how it works.

You can try it out now.  Download from fuse.sf.net, ./configure; make;
make install.  It's not as if it was harder than compiling a kernel :)

Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> > >     This is useful, but there are, AFAIK, two issues:
> > > 
> > >     - We're still deadlocked over some permission-checking hacks in there
> > 
> > Oh, god.  Let me try to explain this again:
> > 
> >   - This is a security issue with unprivileged mounts
> 
> Pretty please, just merge it without unpriviledged mounts. I see
> they are usefull, but they are too strange for now.

An emotional argument again.  What's "strange" about it?

You have a choice of:

 1) believe me that the current solution is fine

 2) get down and try to understand the damn thing, and then come up
    with technical arguments for/against it

I know that 2) takes time and energy, and not a lot of people are
interested enough to go through it, but why on earth do you think it
will _ever_ be easier than now.

Thanks,
Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Pavel Machek]

Hi!

> > > >     This is useful, but there are, AFAIK, two issues:
> > > > 
> > > >     - We're still deadlocked over some permission-checking hacks in there
> > > 
> > > Oh, god.  Let me try to explain this again:
> > > 
> > >   - This is a security issue with unprivileged mounts
> > 
> > Pretty please, just merge it without unpriviledged mounts. I see
> > they are usefull, but they are too strange for now.
> 
> An emotional argument again.  What's "strange" about it?

Not so emotional argument...

System where users can mount their own filesystems should not be
called "Unix" any more. It introduces new mechanism, similar to
ptrace. It restricts root in ways not seen before. How is
updatedb/locate supposed to work on system with this? How is it going
to interact with backup tools? 

Add this to your A): "by tricking some interpretter to think script is
setuid".

> You have a choice of: 1) believe me that the current solution is
> fine

>  2) get down and try to understand the damn thing, and then come up
>     with technical arguments for/against it

Argument is "it is **** ugly".

Your fuse.txt explains why it is not security hole. It does not
explain why your interface is the best possible, and what alternative
ways of "not security hole" exist.

								Pavel
-- 
teflon -- maybe it is a trademark, but it should not be.

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> Not so emotional argument...
> 
> System where users can mount their own filesystems should not be
> called "Unix" any more.

It's not.  It's "Linux".  And anyway, sysadmin may set whatever
owner/group/permissions on '/dev/fuse' to disallow or selectively
allow users to be able to mount FUSE filesystems.

> It introduces new mechanism, similar to ptrace. It restricts root in
> ways not seen before.

Not true.  Root squash in NFS has similar effect.

> How is updatedb/locate supposed to work on system with this? How is
> it going to interact with backup tools?

I assure you, that it will cause no problems whatever.  These programs
are able to gracefully handle errors.

> Add this to your A): "by tricking some interpretter to think script is
> setuid".

How would you do that?

> > You have a choice of: 1) believe me that the current solution is
> > fine
> 
> >  2) get down and try to understand the damn thing, and then come up
> >     with technical arguments for/against it
> 
> Argument is "it is **** ugly".

Yeah, that's your opinion.  Mine is that it's f****** beautiful ;).

There are plenty of ugly things in Unix/Linux that you've become so
accustomed to, that they no longer seem ugly.  Think about the sticky
bit on directories for example.  That one was breaking assumptions
left and right when it got introduced, but people came to accept it,
because it's useful.

> Your fuse.txt explains why it is not security hole. It does not
> explain why your interface is the best possible, and what alternative
> ways of "not security hole" exist.

That's because I don't see any alternative.  The "preventing user from
tracing root" and "preventing access to user's filesysem by root" must
come together.  There's doesn't seem to be any other way.

BTW, thanks for reading through fuse.txt :)

Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Andrew Morton]

Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> > Not so emotional argument...
>  > 
>  > System where users can mount their own filesystems should not be
>  > called "Unix" any more.
> 
>  It's not.  It's "Linux".

It would be helpful if we could have a brief description of the feature
which you're discussing here.  We discussed this a couple of months back,
but I've forgotten most of it and it was off-list I think.

Doing `grep uid fs/fuse/*.c' gets us to the implementation, yes?

Which parts are controversial?

How _should_ we implement unprivileged mounts, if not this way?

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> It would be helpful if we could have a brief description of the feature
> which you're discussing here.  We discussed this a couple of months back,
> but I've forgotten most of it and it was off-list I think.
> 
> Doing `grep uid fs/fuse/*.c' gets us to the implementation, yes?
> 
> Which parts are controversial?

The controversial part is fuse_allow_task() called from
fuse_permission() and fuse_revalidate() (fs/fuse/dir.c).

This function (as explained by the header comment) disallows access to
the filesystem for any task, which the filesystem owner (the user who
did the mount) is not allowed to ptrace.

The rationale is that accessing the filesystem gives the filesystem
implementor ptrace like capabilities (detailed in
Documentation/filesystems/fuse.txt)

It is controversial, because obviously root owned tasks are not
ptrace-able by the user, and so these tasks will be denied access to
the user mounted filesystem (-EACCESS is returned on stat() or any
other file operation).

However nobody raised _any_ concrete technical problem associated with
this, and the 4 years of widespread use didn't turn up any either.  So
IMO it's "ugly" only in people's heads and not in reality.

> How _should_ we implement unprivileged mounts, if not this way?

A lot of the earlier discussion centered around making private
namespaces usable, and so segregating user mounts.  However because
suid/sgid programs can still gain elevated privileges in a private
namespace the above check is still necessary.

So private namespaces are alone not a solution.

I also suggested (and implemented a patch) which hides the user
mounts, based on the above check.  This would mean, that root and
other users would simply not see the user's mount instead of getting
an error.  This was also received with very little enthusiasm.

A further suggested alternative was to confine user mounts to a single
directory (e.g. '/mnt/usermounts').  This would solve the problem if
every daemon and suid/sgid program was tought to not enter this
directory.  This is quite impractical.  Besides this is a very
inflexible solution.

I'm still waiting for a practical alternative.  I don't consider
practical the opinion held by quite a few people:

   "let's integrate FUSE now, without unpriv mounts, maybe we'll find
    a better solution later"

I consider it highly unlikely, that any time will be better than now.

Thanks,
Miklos

  

Re: -mm -> 2.6.13 merge status (fuse)

[Andrew Morton]

Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> > It would be helpful if we could have a brief description of the feature
> > which you're discussing here.  We discussed this a couple of months back,
> > but I've forgotten most of it and it was off-list I think.
> > 
> > Doing `grep uid fs/fuse/*.c' gets us to the implementation, yes?
> > 
> > Which parts are controversial?
> 
> The controversial part is fuse_allow_task() called from
> fuse_permission() and fuse_revalidate() (fs/fuse/dir.c).
> 
> This function (as explained by the header comment) disallows access to
> the filesystem for any task, which the filesystem owner (the user who
> did the mount) is not allowed to ptrace.

That's fairly weird.  Overloading ptraceability is awkward, but also the
*direction* is wrong.  It's saying "if I can ptrace you, you can read my
data".  I'd have expected to see "if you can ptrace me, you can access my
data".

> The rationale is that accessing the filesystem gives the filesystem
> implementor ptrace like capabilities (detailed in
> Documentation/filesystems/fuse.txt)

hrm.  Makes some sense.

> It is controversial, because obviously root owned tasks are not
> ptrace-able by the user, and so these tasks will be denied access to
> the user mounted filesystem (-EACCESS is returned on stat() or any
> other file operation).
> 
> However nobody raised _any_ concrete technical problem associated with
> this, and the 4 years of widespread use didn't turn up any either.  So
> IMO it's "ugly" only in people's heads and not in reality.

It's ugly ;)

But the problem you're addressing here largely revolves around the fact that
the filesystem implementation is a userspace process which is potentially
owned by a different user.  So you need to prevent the mount owner from
peeking at the fs user's activity.  That problem is unique to FUSE and so a
solution within fuse is appropriate.

This security feature doesn't sounds terribly important to me.  So the fuse
server can find out what files I'm looking at.  But I've already
deliberately given the fuse server the ability to ptrace my process?

Can we enhance private namespaces so they can squash setuid/setgid?  If so,
is that adequate?

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> But the problem you're addressing here largely revolves around the fact that
> the filesystem implementation is a userspace process which is potentially
> owned by a different user.  So you need to prevent the mount owner from
> peeking at the fs user's activity.  That problem is unique to FUSE and so a
> solution within fuse is appropriate.

It's in fact not so unique to FUSE.  It would equally well apply to
v9fs or even samba, since both want to allow unprvileged mounts, and
synthetic (or at least user-controlled) file serving.

> This security feature doesn't sounds terribly important to me.  So the fuse
> server can find out what files I'm looking at.  But I've already
> deliberately given the fuse server the ability to ptrace my process?

If it's deliberate, than OK. 

However with suid/sgid, this is not a deliberate action of the user
under whose capabilities the process runs.  Neither in the case, when
it's a daemon doing some recursive directory traversal.

And it's not just peeking at the filesystem access patterns.  A much
more dangerous aspect is controlling _when_ an operation returns
(e.g. delaying it forever), and _what_ it returns (e.g. huge
files/directories).

Of course, this is only truly relevant for systems with untrusted
users.  But I do want to make FUSE work securely in those cases too.

For the single user system, the sysadmin can turn this feature off,
and be done with it.

> Can we enhance private namespaces so they can squash setuid/setgid?  If so,
> is that adequate?

We could.  But that would again be overly restrictive.  The goal is to
make the use of FUSE filesystems for users as simple as possible.  If
the user has to manage multiple namespaces, each with it's own
restrictions, it's becoming a very un-user-friendly environment.

Thanks,
Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Andrew Morton]

Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> > Can we enhance private namespaces so they can squash setuid/setgid?  If so,
>  > is that adequate?
> 
>  We could.  But that would again be overly restrictive.  The goal is to
>  make the use of FUSE filesystems for users as simple as possible.  If
>  the user has to manage multiple namespaces, each with it's own
>  restrictions, it's becoming a very un-user-friendly environment.

I'd have thought that it would be possible to offer the same user interface
as you currently have with private namespaces.  Hide any complexity in the
userspace tools?  Where's the problem?

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> >  We could.  But that would again be overly restrictive.  The goal is to
> >  make the use of FUSE filesystems for users as simple as possible.  If
> >  the user has to manage multiple namespaces, each with it's own
> >  restrictions, it's becoming a very un-user-friendly environment.
> 
> I'd have thought that it would be possible to offer the same user interface
> as you currently have with private namespaces.  Hide any complexity in the
> userspace tools?  Where's the problem?

Sorry, I don't get it.

You mean implement the permission checking in the userspace
filesystem?  That doesn't work, since it's running with the privileges
of the filesystem owner, and as such obviously cannot be trusted with
checking for access by other users.

Thanks,
Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Andrew Morton]

Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> > >  We could.  But that would again be overly restrictive.  The goal is to
> > >  make the use of FUSE filesystems for users as simple as possible.  If
> > >  the user has to manage multiple namespaces, each with it's own
> > >  restrictions, it's becoming a very un-user-friendly environment.
> > 
> > I'd have thought that it would be possible to offer the same user interface
> > as you currently have with private namespaces.  Hide any complexity in the
> > userspace tools?  Where's the problem?
> 
> Sorry, I don't get it.

I'm asking you to expand on what the problems would be if we were to
enhance the namespace code as suggested.  What's the "very un-user-friendly
environment", and why cannot it be made more friendly with appropriate
support tools?

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> > > >  We could.  But that would again be overly restrictive.  The goal is to
> > > >  make the use of FUSE filesystems for users as simple as possible.  If
> > > >  the user has to manage multiple namespaces, each with it's own
> > > >  restrictions, it's becoming a very un-user-friendly environment.
> > > 
> > > I'd have thought that it would be possible to offer the same user interface
> > > as you currently have with private namespaces.  Hide any complexity in the
> > > userspace tools?  Where's the problem?
> > 
> > Sorry, I don't get it.
> 
> I'm asking you to expand on what the problems would be if we were to
> enhance the namespace code as suggested.

OK, what I was thinking, is that the user could create a new
namespace, that has all the filesystems remounted 'nosuid'.  This
wouldn't need any new kernel infrastructure, just a suid-root program
(e.g. newns_nosuid), that would do a clone(CLONE_NEWNS), then
recursively remount everything 'nosuid' in the new namespace.  Then
restore the user's privileges, and exec a shell.

In this namespace the user could mount things to his heart's content.
He could mount over system directories or even the root directory,
without being able to do any harm.

This is very nice, but a bit inpractical, since now all the other
programs of the user, his desktop environment, login shells etc. Won't
be able to see the userspace filesystems mounted in the private
namespace.

Of course he can enter the private namespace immediately after login,
but then he won't be able to send mail for example, because 'sendmail'
could be suid-mail or whatever.

I think the following must be true, for a namespace environment to be
useful:

 1) All the processes should be able to access the same files
    (including new mounts)

 2) User should be able to run suid/sgid programs


If the suid/sgid programs want to access the contents of a userspace
filesystem, that's tough.  You can't have that, because it's insecure.
But experience shows that this isn't a problem.  The exception is the
'mount' program itself, and that is why I'm slowly working towards
making sys_mount() and sys_umount() unprivileged.

Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Andrew Morton]

Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> > > > >  We could.  But that would again be overly restrictive.  The goal is to
>  > > > >  make the use of FUSE filesystems for users as simple as possible.  If
>  > > > >  the user has to manage multiple namespaces, each with it's own
>  > > > >  restrictions, it's becoming a very un-user-friendly environment.
>  > > > 
>  > > > I'd have thought that it would be possible to offer the same user interface
>  > > > as you currently have with private namespaces.  Hide any complexity in the
>  > > > userspace tools?  Where's the problem?
>  > > 
>  > > Sorry, I don't get it.
>  > 
>  > I'm asking you to expand on what the problems would be if we were to
>  > enhance the namespace code as suggested.
> 
>  OK, what I was thinking, is that the user could create a new
>  namespace, that has all the filesystems remounted 'nosuid'.  This
>  wouldn't need any new kernel infrastructure, just a suid-root program
>  (e.g. newns_nosuid), that would do a clone(CLONE_NEWNS), then
>  recursively remount everything 'nosuid' in the new namespace.  Then
>  restore the user's privileges, and exec a shell.
> 
>  In this namespace the user could mount things to his heart's content.
>  He could mount over system directories or even the root directory,
>  without being able to do any harm.
> 
>  This is very nice, but a bit inpractical, since now all the other
>  programs of the user, his desktop environment, login shells etc. Won't
>  be able to see the userspace filesystems mounted in the private
>  namespace.

Yup, that's useless.  That makes the whole CLONE_NEWNS idea unworkable,
yes?

Have we exhausted the alternatives?

(If, as you say, v9fs and samba (did you mean smbfs/cifs?) want
unprivileged mounts, shouldn't the code which you have there be moved out
of fs/fuse/ and into fs/?)

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> Yup, that's useless.  That makes the whole CLONE_NEWNS idea unworkable,
> yes?

For solving this problem, yes.

> Have we exhausted the alternatives?

I believe yes.  At least to date no workable alternative has been
suggested.

> (If, as you say, v9fs and samba (did you mean smbfs/cifs?) want
> unprivileged mounts, shouldn't the code which you have there be
> moved out of fs/fuse/ and into fs/?)

Probably.  But since the code is so small, and there's no other user
yet, I didn't want to take the initiative.

Thanks,
Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Eric Van Hensbergen]

> >
> > I'm asking you to expand on what the problems would be if we were to
> > enhance the namespace code as suggested.
> 
> OK, what I was thinking, is that the user could create a new
> namespace, that has all the filesystems remounted 'nosuid'.  This
> wouldn't need any new kernel infrastructure, just a suid-root program
> (e.g. newns_nosuid), that would do a clone(CLONE_NEWNS), then
> recursively remount everything 'nosuid' in the new namespace.  Then
> restore the user's privileges, and exec a shell.
>

I'm confused why everything has to be remounted nosuid.  I understand
enforcing synthetics to be mounted nosuid, but not the rest of the
file systems.  I thought all the problems revolving around the private
namespace solution where the FUSE team's desire to have per-user
namespace and/or per-session namespace versus per-process namespace.

         -eric

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> > > I'm asking you to expand on what the problems would be if we were to
> > > enhance the namespace code as suggested.
> > 
> > OK, what I was thinking, is that the user could create a new
> > namespace, that has all the filesystems remounted 'nosuid'.  This
> > wouldn't need any new kernel infrastructure, just a suid-root program
> > (e.g. newns_nosuid), that would do a clone(CLONE_NEWNS), then
> > recursively remount everything 'nosuid' in the new namespace.  Then
> > restore the user's privileges, and exec a shell.
> >
> 
> I'm confused why everything has to be remounted nosuid.  I understand
> enforcing synthetics to be mounted nosuid, but not the rest of the
> file systems.

It's related to the problem of a suid program accessing synthetic
filesystem, and filesystem doing something bad to suid program (make
it hang, supply bogus data ...).  This can be solved by "squashing"
suid for the whole namespace (basically the Plan 9 solution).
Unfortunately this is not really practical in Linux/Unix.

> I thought all the problems revolving around the private namespace
> solution where the FUSE team's desire to have per-user namespace
> and/or per-session namespace versus per-process namespace.

That's a different aspect of this thing.  Private namespaces cannot do
anything about the above problem.

Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Eric Van Hensbergen]

On 6/22/05, Miklos Szeredi <miklos@szeredi.hu> wrote:
> >
> > I'm confused why everything has to be remounted nosuid.  I understand
> > enforcing synthetics to be mounted nosuid, but not the rest of the
> > file systems.
> 
> It's related to the problem of a suid program accessing synthetic
> filesystem, and filesystem doing something bad to suid program (make
> it hang, supply bogus data ...).  This can be solved by "squashing"
> suid for the whole namespace (basically the Plan 9 solution).
> Unfortunately this is not really practical in Linux/Unix.
> 

Just to make sure I understand you - if I don't squash suid for the
entire name space, a user could mount a malicious synthetic (even with
NOSUID) and then launch an SUID app from an inherited mount which
would then traverse to the malicious synthetic.

That's a nasty case I hadn't considered before -- however, what's the
potential damage there?  The user could hold up progress of the SUID
app that they launched, but that wouldn't necessarily impede system
progress since system-critical suid apps wouldn't be typically
launched by a user.  I suppose there is the possibility that if
multiple instances of such an SUID app share a global lock you could
get into trouble -- do we have any concrete example apps that would
exhibit this kind of behavior?

Are there other vunerabilities that I'm missing?

      -eric

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> > It's related to the problem of a suid program accessing synthetic
> > filesystem, and filesystem doing something bad to suid program (make
> > it hang, supply bogus data ...).  This can be solved by "squashing"
> > suid for the whole namespace (basically the Plan 9 solution).
> > Unfortunately this is not really practical in Linux/Unix.
> > 
> 
> Just to make sure I understand you - if I don't squash suid for the
> entire name space, a user could mount a malicious synthetic (even with
> NOSUID) and then launch an SUID app from an inherited mount which
> would then traverse to the malicious synthetic.

Yes. 

> That's a nasty case I hadn't considered before -- however, what's the
> potential damage there?  The user could hold up progress of the SUID
> app that they launched, but that wouldn't necessarily impede system
> progress since system-critical suid apps wouldn't be typically
> launched by a user.  I suppose there is the possibility that if
> multiple instances of such an SUID app share a global lock you could
> get into trouble -- do we have any concrete example apps that would
> exhibit this kind of behavior?

I don't know any.  But with 'sudo' the potential set of SUID apps is
basically infinite, so you'd have a hard time proving that this sort
of situation won't arise.

> Are there other vunerabilities that I'm missing?

Another theoretical possibility is that you make the SUID app consume
some resource by feeding it a large-file/deep-directory/etc that quota
would otherwise prevent (you can't do quota on a synthetic filesystem,
without the filesystem's cooperation).

Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Pavel Machek]

Hi!

> > Can we enhance private namespaces so they can squash setuid/setgid?  If so,
> > is that adequate?
> 
> We could.  But that would again be overly restrictive.  The goal is to
> make the use of FUSE filesystems for users as simple as possible.  If
> the user has to manage multiple namespaces, each with it's own
> restrictions, it's becoming a very un-user-friendly environment.

Actually I think this solution is way less ugly. We have precent: if
task is ptraced, suid bits on anything it execs are ignored.

I don't think user interface issues belong to the kernel (and I do not
think different namespaces are that bad for the user; various chroots
and ld_preload hacks already work that way)
								Pavel
-- 
teflon -- maybe it is a trademark, but it should not be.

Re: -mm -> 2.6.13 merge status (fuse)

[Theodore Ts'o]

On Wed, Jun 22, 2005 at 09:16:34AM +0200, Miklos Szeredi wrote:
> The controversial part is fuse_allow_task() called from
> fuse_permission() and fuse_revalidate() (fs/fuse/dir.c).
> 
> This function (as explained by the header comment) disallows access to
> the filesystem for any task, which the filesystem owner (the user who
> did the mount) is not allowed to ptrace.
> 
> The rationale is that accessing the filesystem gives the filesystem
> implementor ptrace like capabilities (detailed in
> Documentation/filesystems/fuse.txt)
> 
> It is controversial, because obviously root owned tasks are not
> ptrace-able by the user, and so these tasks will be denied access to
> the user mounted filesystem (-EACCESS is returned on stat() or any
> other file operation).

I don't think this should be objectionable, since we already have
times when root-owned tasks can get EACCESS when accessing some
filesystem.  This can happen with any distributed filesystem that
enforces real security --- whether it be nfs-root-squash, or the
Andrew Filesystem, or NFSv4.  Root can get "permission denied" when
some other userid with appropriate credentials would be allowed to
access the file/directory.

On the other hand, sometimes a root process, or some other user's
process, might _want_ to give permission to allow a trusted FUSE
filesystem the potential to monkey with it (return potentially
untrusted information, or stop it entirely), in exchange for access to
the filesystem.  So it would be nice if there was some way that a
process could tell the kernel that it is willing to give permission to
allow certain FUSE filesystems to potentially affect it.  Say, via a
fnctl() call, perhaps.

					- Ted

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> I don't think this should be objectionable, since we already have
> times when root-owned tasks can get EACCESS when accessing some
> filesystem.  This can happen with any distributed filesystem that
> enforces real security --- whether it be nfs-root-squash, or the
> Andrew Filesystem, or NFSv4.  Root can get "permission denied" when
> some other userid with appropriate credentials would be allowed to
> access the file/directory.

Right.

> On the other hand, sometimes a root process, or some other user's
> process, might _want_ to give permission to allow a trusted FUSE
> filesystem the potential to monkey with it (return potentially
> untrusted information, or stop it entirely), in exchange for access to
> the filesystem.  So it would be nice if there was some way that a
> process could tell the kernel that it is willing to give permission to
> allow certain FUSE filesystems to potentially affect it.  Say, via a
> fnctl() call, perhaps.

Hmm.  'su' works for root.  

How do you think fcntl() could be used?  I think a task flag settable
via prctl() would be more appropriate.

Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Theodore Ts'o]

On Wed, Jun 22, 2005 at 07:48:50PM +0200, Miklos Szeredi wrote:
> > On the other hand, sometimes a root process, or some other user's
> > process, might _want_ to give permission to allow a trusted FUSE
> > filesystem the potential to monkey with it (return potentially
> > untrusted information, or stop it entirely), in exchange for access to
> > the filesystem.  So it would be nice if there was some way that a
> > process could tell the kernel that it is willing to give permission to
> > allow certain FUSE filesystems to potentially affect it.  Say, via a
> > fnctl() call, perhaps.
> 
> Hmm.  'su' works for root.  

??  Not sure what you're saying.

> How do you think fcntl() could be used?  I think a task flag settable
> via prctl() would be more appropriate.

The problem with a task flag is that a root process might not want to
give permission for _all_ user-mountable filesystem, but only certain
ones.  So the idea is that the process should be able to specify
specific mountpoints as being "ok" for the process to access.

					- Ted

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> > > On the other hand, sometimes a root process, or some other user's
> > > process, might _want_ to give permission to allow a trusted FUSE
> > > filesystem the potential to monkey with it (return potentially
> > > untrusted information, or stop it entirely), in exchange for access to
> > > the filesystem.  So it would be nice if there was some way that a
> > > process could tell the kernel that it is willing to give permission to
> > > allow certain FUSE filesystems to potentially affect it.  Say, via a
> > > fnctl() call, perhaps.
> > 
> > Hmm.  'su' works for root.  
> 
> ??  Not sure what you're saying.

If user X mounts a filesystem, and root wants to access it, it can do
'su X' and see the otherwise inaccesible files.

> > How do you think fcntl() could be used?  I think a task flag settable
> > via prctl() would be more appropriate.
> 
> The problem with a task flag is that a root process might not want to
> give permission for _all_ user-mountable filesystem, but only certain
> ones.  So the idea is that the process should be able to specify
> specific mountpoints as being "ok" for the process to access.

OK, so you're saying, it should be a per-mountpoint flag, and not a
per-process one.

Then the solution is simple I think.  Root can just remount the
filesystem with the 'allow_other' flag (that's currently not possible,
because the remount_fs operation is not yet implemented).

Thanks,
Miklos

Re: -mm -> 2.6.13 merge status (fuse)

[Theodore Ts'o]

On Fri, Jun 24, 2005 at 07:49:50AM +0200, Miklos Szeredi wrote:
> > ??  Not sure what you're saying.
> 
> If user X mounts a filesystem, and root wants to access it, it can do
> 'su X' and see the otherwise inaccesible files.

That's rather awkward, and will be painful in certain applications
that are trying to scan multiple filesystems, possibly run by
different users X, Y, and Z.

> > > How do you think fcntl() could be used?  I think a task flag settable
> > > via prctl() would be more appropriate.
> > 
> > The problem with a task flag is that a root process might not want to
> > give permission for _all_ user-mountable filesystem, but only certain
> > ones.  So the idea is that the process should be able to specify
> > specific mountpoints as being "ok" for the process to access.
> 
> OK, so you're saying, it should be a per-mountpoint flag, and not a
> per-process one.

No, I'm saying that it should be both per-mountpoint and per-process.
Hence something like fnctl.  Problem is though fnctl() works on a file
descriptor, and with FUSE running you wouldn't be able to even get a
file descriptor opened on the mountpoint.  

> Then the solution is simple I think.  Root can just remount the
> filesystem with the 'allow_other' flag (that's currently not possible,
> because the remount_fs operation is not yet implemented).

Globally remounting the filesystem allow_other isn't the right thing,
since applications that don't know what to expect will be taken by
surprise.  Hence my suggestion to do something which is both
per-mountpoint and per-process.

						- Ted

Re: -mm -> 2.6.13 merge status (fuse)

[Gábor Lénárt]

EHLO,

On Tue, Jun 21, 2005 at 11:39:14PM -0700, Andrew Morton wrote:
> >  > System where users can mount their own filesystems should not be
> >  > called "Unix" any more.
> > 
> >  It's not.  It's "Linux".
> 
> It would be helpful if we could have a brief description of the feature
> which you're discussing here.  We discussed this a couple of months back,
> but I've forgotten most of it and it was off-list I think.

<offtopic>

Excuse me, I'm far from being a filesystem/vfs expert ... However I've
got the idea about the merging fuse/reiser4 that some guys keep complaining
about the quite strange behaviour of these stuffs: when I write 'strange'
I mean strange from the view point of some standard unix ideas about filesystems
 (and anything related to filesystems like permission checking, namespaces
etc) and how they should be implemented and handled.

This reminds me articles about comparing Linux and BSDs. BSD guys claims
that BSD distros _ARE_ unices but Linux is not. It's out of scope to waste
mails about these flames like this (it's question of view point as almost
always) however there IS some lesson here. BSD systems are somewhat (well,
not counting the interesting ideas of DragonFly BSD) conservative to
implement new stuffs. I'm about stuffs like filesystem transactions, API
exported to the user space to be able to do things like deleting data from
the begining of the file (there is API call to truncate - from the end of
the file ...) and such (what a quite braindead idea to rewrite eg a 10Gbyte
long file just for inserting one byte to somewhere in the middle of the file
- in 2005 ...). The only thing explains where the later is not present in
most OSes is because of historical reasons and not technical ones. And if even
Linux does not want to open toward extended filesystem abilities which common
open source system will? I guess none.

I can inmagine that vendors of some closed source systems will exploit
the hole in the area of outdated filesystem concept of our current world
and when it becomes reality it's to late. Maybe.

Please forgive for my possible offtopic mail here but I could not resist :)

</offtopic>

-- 
- Gábor

Re: -mm -> 2.6.13 merge status (fuse)

[Pavel Machek]

Hi!

> > > You have a choice of: 1) believe me that the current solution is
> > > fine
> > 
> > >  2) get down and try to understand the damn thing, and then come up
> > >     with technical arguments for/against it
> > 
> > Argument is "it is **** ugly".
> 
> Yeah, that's your opinion.  Mine is that it's f****** beautiful ;).
> 
> There are plenty of ugly things in Unix/Linux that you've become so
> accustomed to, that they no longer seem ugly.  Think about the sticky
> bit on directories for example.  That one was breaking assumptions
> left and right when it got introduced, but people came to accept it,
> because it's useful.

Just for the record, I still consider sticky bit "slightly" ugly and
nfs root squash "very" ugly.

> > Your fuse.txt explains why it is not security hole. It does not
> > explain why your interface is the best possible, and what alternative
> > ways of "not security hole" exist.
> 
> That's because I don't see any alternative.  The "preventing user from
> tracing root" and "preventing access to user's filesysem by root" must
> come together.  There's doesn't seem to be any other way.

It is clear that we can't allow root (or anyone else) to access that
filesystem. Infinite namespace is nice trap.

> BTW, thanks for reading through fuse.txt :)

You are welcome ;-).
								Pavel

-- 
teflon -- maybe it is a trademark, but it should not be.

Re: -mm -> 2.6.13 merge status (fuse)

[Matthias Urlichs]

Hi, Pavel Machek wrote:

> How is it going
> to interact with backup tools?

I admit to severe dislike for any backup tool which crosses file system
borders without permission, and therefore am not going to cry if something
breaks because of this.

Ditto updatedb/locate -- it's a remote file system. Tools should
treat it as such.

NB: I am assuming that root can stat() the mountpoint and that it's not
possible to delay this call arbitrarily. If not, the debate IMHO should be
whether to special-case this situation.

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
The early worm has a death wish.

Re: -mm -> 2.6.13 merge status (fuse)

[Eric Van Hensbergen]

On 6/21/05, Pavel Machek <pavel@ucw.cz> wrote:
> >
> > An emotional argument again.  What's "strange" about it?
> 
> Not so emotional argument...
> 
> System where users can mount their own filesystems should not be
> called "Unix" any more. It introduces new mechanism, similar to
> ptrace.

I think that's a rather severe statement.   I don't see allowing user
mounts damaging standard UNIX system semantics as long as certain
rules are followed.   After all, user-mounts and private name spaces
are what the original authors of UNIX went on to develop.

> It restricts root in ways not seen before. How is
> updatedb/locate supposed to work on system with this? How is it going
> to interact with backup tools?
>

I think requiring user-mounts to be in private name spaces solves many
of these issues -- you don't have to worry about system daemons and/or
other users wandering into synthetic file systems.

         -eric

Re: -mm -> 2.6.13 merge status (fuse)

[Pavel Machek]

Hi!

> > > An emotional argument again.  What's "strange" about it?
> > 
> > Not so emotional argument...
> > 
> > System where users can mount their own filesystems should not be
> > called "Unix" any more. It introduces new mechanism, similar to
> > ptrace.
> 
> I think that's a rather severe statement.   I don't see allowing user
> mounts damaging standard UNIX system semantics as long as certain
> rules are followed.   After all, user-mounts and private name spaces
> are what the original authors of UNIX went on to develop.

Well, but notice how it is called "Plan 9", not "Unix" :-). The
"certain rules" are rather tricky to enforce... btw any ideas how
Plan 9 solves problems around user-mounts? Does it allow it at all?

								Pavel
-- 
teflon -- maybe it is a trademark, but it should not be.

Re: -mm -> 2.6.13 merge status (fuse)

[Eric Van Hensbergen]

On 6/22/05, Pavel Machek <pavel@ucw.cz> wrote:
> >
> > I think that's a rather severe statement.   I don't see allowing user
> > mounts damaging standard UNIX system semantics as long as certain
> > rules are followed.   After all, user-mounts and private name spaces
> > are what the original authors of UNIX went on to develop.
> 
> Well, but notice how it is called "Plan 9", not "Unix" :-). The
> "certain rules" are rather tricky to enforce... btw any ideas how
> Plan 9 solves problems around user-mounts? Does it allow it at all?
> 

I agree the rules can be tricky, but I think we can start with
something ultra-conservative (but not as ultra-conservative as no user
mounts) and work from there.

Plan 9 does indeed allow (in fact in many ways it depends on) user
mounts.  There are a number of answers for how it gets around some of
the sticky issues -- some of those may be applicable in UNIX-style
systems, while others most likely don't apply.

The ones that don't apply:
(a) Plan 9 has no root user, nor does it have a concept of
setuid/setgid -- this solves a lot of problems, but isn't practical in
UNIX environments.
(b) Plan 9 cluster environments typically have user-controller
terminals and shared centralized resources (cpu servers, file servers,
etc.)  Users have more or less complete control over their local
workstation (which includes their name space) -- and they can import
resources/services from shared servers (where they have less/no
control).  Critical services, such as authentication, are hosted on a
remote server protected from normal users.  While this sort of
environment can be set up using UNIX systems -- its probably not
practical for most folks.

The ones that do apply:
(c) Private Name Space - by default, every new process in Plan 9 gets
a private name space inherited from its parent.  This means that any
user mounts will be in their own private name space and not interfere
with system tasks or other users.  As I've stated, I think this has
direct application in UNIX environments.  We may not go to the extreme
of making CLONENS the default for new threads, but I think it should
be easy enough to enforce unprivileged mounts be in a non-global name
space.
(d) For situations in which sharing a name space is desirable, the
name space is re-exported through a global handle (/srv in Plan 9's
case) so that a user can mount instances of his private name space in
other threads -- or he can set permissions on the handle so it may be
used by other users.  This re-export is done using the 9P protocol
(although if re-exporting to the same system it short-circuits to just
function calls within the OS).  The 9P layer adds a certain level of
security to just allowing name spaces to be accessed/mounted directly
- of course it also adds a bit of overhead.  We are currently looking
at this sort of implementation in the v9fs 2.1 release.

Given that (a) & (b) aren't practical in UNIX environments, I think a
reasonable set of restrictions (some of which were suggested by
Miklos) can be used to maintain integrity:
1) only allow user's to mount/bind on directories/files where they
have unconditional write access.
2) enforce NOSUID mount options on user-mounts

If you combine these two restrictions with only allowing unprivileged
mounts in private name space I think you get 90% there.  The only
thing left to resolve is the best way to allow sharing private name
spaces between threads/users -- and I still view this as more of
extended functionality than a hard-requirement.

I'm giving a talk at OLS overviewing my perspective on the barriers
and solutions.  My motivation was to try and get some
dialog/compromise/closure on some of the user-space mount and private
name space issues.  Hopefully some of the stake-holders will attend.

            -eric

Re: -mm -> 2.6.13 merge status (fuse)

[Eric Van Hensbergen]

On 6/22/05, Eric Van Hensbergen <ericvh@gmail.com> wrote:
    ...
> 
> If you combine these two restrictions with only allowing unprivileged
> mounts in private name space I think you get 90% there.  The only
> thing left to resolve is the best way to allow sharing private name
> spaces between threads/users -- and I still view this as more of
> extended functionality than a hard-requirement.
> 

Reviewing my notes, there were a few subtle restrictions I forgot
(most of which originally suggested by Miklos):

(a) User's can't mount file system types not deemed "safe" (via  flag
in the file system type) -- this should help mitigate user's
exploiting bugs in existing file systems to interfere with the system.
 Judging when a file system type is "safe" is a nasty kettle of fish
though...
(b) Enforce NODEV along with NOSUID so that user-based synthetics
can't have device inodes with compromised permissions, etc.

       -eric

Re: -mm -> 2.6.13 merge status (fuse)

[Alan Cox]

> 1) only allow user's to mount/bind on directories/files where they
> have unconditional write access.

Like say /tmp. Build a bizarre behaving /tmp and I can do funky stuff
with some third party suid apps. Its a good start but you probably want
a stronger policy and one enforced by the user space side not kernel (eg
"Below ~")

> 2) enforce NOSUID mount options on user-mounts

2 is unneccessarily crude. Just enforce suid owner/owner group. 

Alan

Re: -mm -> 2.6.13 merge status (fuse)

[Eric Van Hensbergen]

On 6/22/05, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> > 1) only allow user's to mount/bind on directories/files where they
> > have unconditional write access.
> 
> Like say /tmp. Build a bizarre behaving /tmp and I can do funky stuff
> with some third party suid apps. Its a good start but you probably want
> a stronger policy and one enforced by the user space side not kernel (eg
> "Below ~")
>

Well in the original discussions Miklos had classified directories
that had the sticky bit set (such as /tmp) as out-of-bounds for
user-mounts.  However, its a point well taken.   I had originally
proposed having some sort of a policy file (sort of like an extended
fstab with regular expressions) to give more granular control over
where users could and couldn't mount (along with what types of
devices, network servers, and file systems they could mount from). 
However, this leans more towards the "super-mount" suid-application
which I think many found undesirable.  An alternative would be some
way for the kernel to consult with an application about different
mount policies.  I don't know what the right thing is here.
 
> > 2) enforce NOSUID mount options on user-mounts
> 
> 2 is unneccessarily crude. Just enforce suid owner/owner group.
> 

I'm dense this morning, not sure what you mean here.
 
    -eric

Re: -mm -> 2.6.13 merge status (fuse)

[Avuton Olrich]

On 6/21/05, Miklos Szeredi <miklos@szeredi.hu> wrote:
> I won't shed many tears if you drop fuse-nfs-export.patch.  It would
> at least give the userspace solution some boost.
> 
> However the patch is pretty small, and despite it's flaws, I know it's
> used by a number of people.

Why not leave it up to the user as an option, for the time being at
least. Does this somehow break things?

thanks,
avuton

-- 
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: -mm -> 2.6.13 merge status (fuse)

[Miklos Szeredi]

> > I won't shed many tears if you drop fuse-nfs-export.patch.  It would
> > at least give the userspace solution some boost.
> > 
> > However the patch is pretty small, and despite it's flaws, I know it's
> > used by a number of people.
> 
> Why not leave it up to the user as an option, for the time being at
> least.

Making it a config option could make sense, yes.

> Does this somehow break things?

You mean outside NFS export?  No, it's completely harmless. 

NFS export itself is slightly broken (random ESTALE errors), but it's
still useful.

Thanks,
Miklos

Re: -mm -> 2.6.13 merge status

[Andrey Panin]

[-- Attachment #1: Type: text/plain, Size: 6529 bytes --]

On 171, 06 20, 2005 at 11:54:58 -0700, Andrew Morton wrote:
> 
> This summarises my current thinking on various patches which are presently
> in -mm.  I cover large things and small-but-controversial things.  Anything
> which isn't covered here (and that's a lot of material) is probably a "will
> merge", unless it obviously isn't.
> 
> (If you reply to this email it would be a good idea to alter the Subject:
> to reflect which feature you are discussing)
> 
> 
> 
> git-ocfs
> 
>     The OCFS2 filesystem.  OK by me, although I'm not sure it's had enough
>     review.
> 
> sparsemem
> 
>     OK by me for a merge.  Need to poke arch maintainers first, check that
>     they've looked at it sufficiently closely.
> 
> vm-early-zone-reclaim
> 
>     Needs some convincing benchmark numbers to back it up.  Otherwise OK.
> 
> avoiding-mmap-fragmentation
> 
>     Tricky.  Addresses vm area fragmentation issues due to recent
>     optimisations to the free-area lookup code.  Will merge.
> 
> periodically-drain-non-local-pagesets
> 
>     Will merge
> 
> pcibus_to_node and users
> 
>     Will merge
> 
> CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.
> 
>     Will merge (will switch default to 1000 Hz later if that seems necessary)
> 
> dmi-*.patch
> 
>     Will merge.  I have a comment "The below break x440".  Maybe it got
>     fixed.  We'll doubtless hear if not.

Fixed, patch merged in -mm as dmi-move-acpi-sleep-quirk-fix.patch

http://marc.theaimsgroup.com/?l=linux-kernel&m=111829134708641&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=111832375203467&w=2

> xen-*.patch
> 
>     These are little cleanups and abstractions which make a Xen merge
>     easier.  May as well merge them.
> 
> CPU hotplug for x86 and x86_64
> 
>     Not really useful on current hardware, but these provide
>     infrastructure which some power management patches need, and it seems
>     sensible to make the reference architecture support hotplug.  Will merge.
> 
> swsusp-on-SMP
> 
>     Will merge.
> 
> cfq version 3
> 
>     Not sure.  Jens seems to be setting up a few git trees.  On hold.
> 
> RCUification of the key management code
> 
>     Don't know - dhowells seemed diffident last time we discussed this.
> 
> timers-fixes-improvements.patch
> 
>     SMP speedups for the core timer code.  It was bumpy, but this seems
>     stable now.  Will merge.
> 
> kprobes-*
> 
>     Will merge
> 
> rapidio-*
> 
>     Will merge.
> 
> namespace*.patch
> 
>     Awaiting viro ack.
> 
> xtensa architecture
> 
>     Is xtensa now, or will it be in the future a sufficiently popular
>     architecture to justify the cost of having this code in the tree?
> 
>     Heaven knows.  Will merge.
> 
> dlm-*.patch: Red Hat distributed lock manager
> 
>     Hard.  Right now it seems that no in-kernel projects will use this and
>     only one out-of-kernel project will use it.  Shelve the problem until
>     after Kernel Summit, where some light may be shed.
> 
>     Opinions are sought...
> 
> connector.patch
> 
>     Nice idea IMO, but there are still questions around the
>     implementation.  More dialogue needed ;)
> 
> connector-add-a-fork-connector.patch
> 
>     OK, but needs connector.
> 
> inotify
> 
>     There are still concerns about the userspace API and internal
>     implementation details.  More slogging needed.
> 
> pcmcia-*.patch
> 
>     Makes the pcmcia layer generate hotplug events and deprecates cardmgr.
>     Will merge.
> 
> NUMA-aware slab allocator
> 
>     Seems stable now, but it needs some ifdef reduction work before
>     merging, please.
> 
> CPU scheduler
> 
>     Will merge some of these patches.  We're still discussing which ones.
> 
> perfctr
> 
>     Not yet, but getting closer.  The PPC64 guys still need to sort out a
>     few interface issues with Mikael.  We might be able to fit this into
>     2.6.13 if people get a move on.
> 
> cachefs
> 
>     This is a ton of code which knows rather a lot about pagecache
>     internals.  It allows the AFS client to cache file contents on a local
>     blockdev.
> 
>     I don't think it's a justified addition for only AFS and I'd prefer to
>     see it proven for NFS as well.
> 
>     Issues around add-page-becoming-writable-notification.patch need to
>     be resolved.
> 
> cachefs-for-nfs
> 
>     A recent addition.  Needs review from NFS developers and considerably
>     more testing.
> 
>     These things aren't looking likely for 2.6.13.
> 
> kexec and kdump
> 
>     I guess we should merge these.
> 
>     I'm still concerned that the various device shutdown problems will
>     mean that the success rate for crashing kernels is not high enough for
>     kdump to be considered a success.  In which case in six months time we'll
>     hear rumours about vendors shipping wholly different crashdump
>     implementations, which would be quite bad.
> 
>     But I think this has gone as far as it can go in -mm, so it's a bit of
>     a punt.
> 
> reiser4
> 
>     Merge it, I guess.
> 
>     The patches still contain all the reiser4-specific namespace
>     enhancements, only it is disabled, so it is effectively dead code.  Maybe
>     we should ask that it actually be removed?
> 
> v9fs
> 
>     I'm not sure that this has a sufficiently high
>     usefulness-to-maintenance-cost ratio.
> 
> fuse
> 
>     This is useful, but there are, AFAIK, two issues:
> 
>     - We're still deadlocked over some permission-checking hacks in there
> 
>     - It has an NFS server implementation which only works if the
>       to-be-served file happens to be in dcache.
> 
>       It has been said that a userspace NFS server can be used to get
>       full NFS server functionality with FUSE.  I think the half-assed kernel
>       implementation should be done away with.
> 
> execute-in-place
> 
>     Will merge.  Have the embedded guys commented on the usefulness of
>     this for execute-out-of-ROM?
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Andrey Panin		| Linux and UNIX system administrator
pazke@donpac.ru		| PGP key: wwwkeys.pgp.net

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: -mm -> 2.6.13 merge status

[Alan Cox]

On Maw, 2005-06-21 at 07:54, Andrew Morton wrote:
> CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.
>     Will merge (will switch default to 1000 Hz later if that seems necessary)

This has been in Fedora for a while. DaveJ can probably give you more
info. From own testing 100Hz is how far down you want to go to avoid
random clock slew on laptops and to see power improvements.

Re: -mm -> 2.6.13 merge status

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Tue, 2005-06-21 at 13:35 +0100, Alan Cox wrote:
> On Maw, 2005-06-21 at 07:54, Andrew Morton wrote:
> > CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.
> >     Will merge (will switch default to 1000 Hz later if that seems necessary)
> 
> This has been in Fedora for a while. DaveJ can probably give you more
> info. From own testing 100Hz is how far down you want to go to avoid
> random clock slew on laptops and to see power improvements.

actually 250Hz is a not so fun value. 300 is a lot nicer (multiple of
both 50Hz and 60Hz and thus covers most TV standards)


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: -mm -> 2.6.13 merge status

[Erik Slagter]

On Tue, 2005-06-21 at 09:07 -0400, Arjan van de Ven wrote:
> On Tue, 2005-06-21 at 13:35 +0100, Alan Cox wrote:
> > On Maw, 2005-06-21 at 07:54, Andrew Morton wrote:
> > > CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.
> > >     Will merge (will switch default to 1000 Hz later if that seems necessary)
> > 
> > This has been in Fedora for a while. DaveJ can probably give you more
> > info. From own testing 100Hz is how far down you want to go to avoid
> > random clock slew on laptops and to see power improvements.
> 
> actually 250Hz is a not so fun value. 300 is a lot nicer (multiple of
> both 50Hz and 60Hz and thus covers most TV standards)

Sorry, the ITU-R "M" standard specifies 30000/1001 frames (60000/1001
fields) per second, that's close to 30/60 but not the same. Now please
please don't make use run HZ = (60000/1001 * 5) or similar ;-)

BTW it seems to me that power management using C2/C3 states will work
much more efficiently with a lower HZ because the chipset/processor will
be a larger percentage of the time in c2/c3 mode, right?

Re: -mm -> 2.6.13 merge status

[Carsten Otte]

On 6/21/05, Andrew Morton <akpm@osdl.org> wrote:
> execute-in-place
> 
>     Will merge.  Have the embedded guys commented on the usefulness of
>     this for execute-out-of-ROM?
Allright. Going to push our test-team to run their tests on the xip
code that is in -mm.

Re: -mm -> 2.6.13 merge status

[Jörn Engel]

On Mon, 20 June 2005 23:54:58 -0700, Andrew Morton wrote:
> 
> execute-in-place
> 
>     Will merge.  Have the embedded guys commented on the usefulness of
>     this for execute-out-of-ROM?

It looks fairly useful, but XIP for NOR flashes still needs additional
work.  No objections from my side.

Jörn

-- 
Optimizations always bust things, because all optimizations are, in
the long haul, a form of cheating, and cheaters eventually get caught.
-- Larry Wall 

Re: -mm -> 2.6.13 merge status (Suspend-to-disk)

[Nigel Cunningham]

Hi.

(Marcelo: Copied for issue at the bottom).

On Tue, 2005-06-21 at 16:54, Andrew Morton wrote:
> CPU hotplug for x86 and x86_64
> 
>     Not really useful on current hardware, but these provide
>     infrastructure which some power management patches need, and it seems
>     sensible to make the reference architecture support hotplug.  Will merge.

Yay. I'm not going to use it yet, but know Pavel wants it for the next
one.

> swsusp-on-SMP
> 
>     Will merge.
> 
> kexec and kdump
> 
>     I guess we should merge these.
> 
>     I'm still concerned that the various device shutdown problems will
>     mean that the success rate for crashing kernels is not high enough for
>     kdump to be considered a success.  In which case in six months time we'll
>     hear rumours about vendors shipping wholly different crashdump
>     implementations, which would be quite bad.
> 
>     But I think this has gone as far as it can go in -mm, so it's a bit of
>     a punt.

No potential clashes with suspend code, I assume?

> execute-in-place
> 
>     Will merge.  Have the embedded guys commented on the usefulness of
>     this for execute-out-of-ROM?

Switch roles for a mo and put my Cyclades hat on. Probably not useful to
us at the moment, at least in the case of the products I work on.
Marcelo?

Regards,

Nigel

Re: -mm -> 2.6.13 merge status (Suspend-to-disk)

[Marcelo Tosatti]

> > execute-in-place
> > 
> >     Will merge.  Have the embedded guys commented on the usefulness of
> >     this for execute-out-of-ROM?
> 
> Switch roles for a mo and put my Cyclades hat on. Probably not useful to
> us at the moment, at least in the case of the products I work on.
> Marcelo?

Well yes, its definately very useful for embedded folks where RAM is a 
precious resource (not our case at the moment).

I'm not aware of any users of this XIP implementation, maybe Tim Bird or 
Russell have reviewed/tested it? 

It went through filesystem folks reviewing (and I'm pretty sure akpm knows
about that already)...

Hope to be helpful.

Re: -mm -> 2.6.13 merge status (Suspend-to-disk)

[Pavel Machek]

Hi!

> > kexec and kdump
> > 
> >     I guess we should merge these.
> > 
...
> No potential clashes with suspend code, I assume?
> 

I test suspend in -mm series from time to time, and it seems ok;
so this one should be safe.
				Pavel
-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: v9fs (-mm -> 2.6.13 merge status)

[Eric Van Hensbergen]

On 6/21/05, Andrew Morton <akpm@osdl.org> wrote:
> 
> v9fs
> 
>     I'm not sure that this has a sufficiently high
>     usefulness-to-maintenance-cost ratio.
> 
 
I think v9fs/9P has some unique aspects which differentiate it from
the other distributed system protocols integrated into Linux:
a) it presents a unified distributed resource sharing protocol.  It
will be able to distribute devices, file systems, system services, and
application interfaces.
b) it provides non-caching RPC-style access to synthetic file systems
which could be used with in-kernel file systems such as sysfs or with
user-space synthetics such as those provided by FUSE
c) its implementation supports transport independence enabling easy
support for different interconnects (shared memory, Xen device
channels, RDMA, Infiniband, etc.)

v9fs-2.0 has a somewhat limited audience at the moment - but now that
the initial implementation is more or less complete we are working to
build applications on top of it (and provide a better server).  It's
being integrated into cluster projects at LANL and being looked at wrt
virtualization I/O at IBM.  Its our hope that these improvements and
cluster applications will motivate more wide-spread use of the v9fs
module.

     -eric

Re: v9fs (-mm -> 2.6.13 merge status)

[Uriel]

On Tue, Jun 21, 2005 at 08:51:27AM -0500, Eric Van Hensbergen wrote:
> On 6/21/05, Andrew Morton <akpm@osdl.org> wrote:
> > 
> > v9fs
> > 
> >     I'm not sure that this has a sufficiently high
> >     usefulness-to-maintenance-cost ratio.

The 9P protocol implemented by v9fs is the result of over a decade of 
research in distributed systems at Bell Labs by the original Unix team,
and it has various implementations for other operating systems that have
been used in production systems for many years.

9P is designed to be portable across systems and transport protocols,
it's network transparent, and it gives us interoperativity with
Inferno(which can run hosted under Linux already), Plan 9, and p9p, and
implementations for *BSD and other systems are in the works.

9P has the potential to become the standard protocol for distributed
resources and I don't think any of the alternatives come anywhere near
being as well designed, well proven and encompassing.

uriel

Re: v9fs (-mm -> 2.6.13 merge status)

[Martin Atkins]

Uriel <uriell <at> binarydream.org> writes:
>..
> The 9P protocol implemented by v9fs is the result of over a decade of 
> research in distributed systems at Bell Labs by the original Unix team,

I would second that. There are several other filesystems already merged or
under discussion that are superficially similar - fuse, coda, and even nfs!
However, v9fs is the only one that is (all of)
1) suitable for synthetic filesystems (resources),
   as well as 'normal' filesystems
2) equally good for local and remote filesystems
3) OS/machine independent "on-the-wire"
4) (network) transport independent
5) has a significant history of deployment

If there are few Linux applications using v9fs as yet (but there is plan9ports),
then one must admit that there is something of a chicken-and-eggs
situation involved!

Martin

Re: Fwd: v9fs (-mm -> 2.6.13 merge status)

[Ronald G. Minnich]

On Tue, 21 Jun 2005, Eric Van Hensbergen wrote:

> On 6/21/05, Andrew Morton <akpm@osdl.org> wrote:
> >
> > v9fs
> >
> >     I'm not sure that this has a sufficiently high
> >     usefulness-to-maintenance-cost ratio.
> >

I got pointed at this discussion. Here are my $.02 on why we at LANL are
interested in v9fs.

We build clusters on the order of 2000 machines at present, with larger 
systems coming along. The system which we use to run these clusters is 
bproc. While bproc has proven to be very powerful to date, it does have 
its limits:
- requires homogenous system
- the network protocols it uses, while simple, are somewhat ad-hoc 
  (as is common in this type of system)
- if you are on a bproc system as user x, using 25% of the system, 
  you still see 100% of the processes. This is a bit of a security issue. 

We have a desire to build single-system-image looking clusters along the 
bproc model, but at the same time compose those clusters of, e.g., 
Opterons and G5s. This mixing is highly desirable for compoutations that 
have phases, some of which belong on one type of a machine, and some on 
another. 

We are going to use v9fs as the glue for our next-generation cluster 
software, called 'xcpu'. Xcpu has been implemented on Plan 9 and works 
there. I have ported xcpu to Linux, using v9fs as the client side and Russ 
Cox's plan9ports server to write servers. 

xcpu presents a remote execution service as a 9p server. xcpu has been
tested across architectures and it works very well. By summer 2006, we
hope to have cut over our bproc systems to xcpu.

That's one use for v9fs. We also plan to use v9fs to provide us with 
servers for global /proc, monitoring, and control systems for our 
clusters. 

The global /proc is interesting. bproc provides a global /proc, but it is 
incomplete; entries for, e.g., exe and maps are not filled in. bproc also 
caches part of the /proc, but the rules about what is cached and what the 
timeouts are, are set in the kernel module and not easily changed. We are 
going to have an "aggregating" user level 9p server based on 
Mirtchovskis's aggrfs, which will both aggregate all the cluster nodes, 
and have caching rules that make sense in clusters of 1000s of node (for 
example, it is ok to cache /proc/x/status; there is no need to cache 
/proc/x/maps, and you probably don't want to anyway). 

A neat capability is that if we give a user, e.g., 25% of the cluster, we
can tailor that user's name space so that they only see their procs and
the 25% of the cluster they own. This is good for security, but also good
for convenience: most users don't really care that some other user is on
75% of the cluster. Global pid spaces are neat in theory, messy in
practice at large scale. I want my global pid space to be global to *me*,
meaning I see the global space of the nodes I care about.  The sysadmin,
of course, wants to see everything. All this is possible. V9fs, along with
Linux private name spaces, will allow us to provide this model: users can 
see some or all of the global pid space, depending on need; users can be 
constrained to only see part of the global pid space, depending on other 
issues. 

9p will also replace the Supermon protocol, allowing people to easily view 
status information in a file system. 

In addition to the cluster usage, there is also grid usage. The 9grid, 
composed of plan 9 systems, is connected by 9p servers. Linux systems can 
join the 9grid with no problem, once Linux has v9fs. 

Were v9fs just a file system, I would not really be interested in it one 
way or another; we have NFS, after all. But v9fs is really the key piece 
of a new model of cluster services we are building at LANL. 9p will be the 
glue, and v9fs will be the needed client side for hooking 9p servers into 
the file system name space. 

I'm hoping we can see v9fs in the kernel someday.

thanks

ron

Re: -mm -> 2.6.13 merge status

[Martin Hicks]

On Mon, Jun 20, 2005 at 11:54:58PM -0700, Andrew Morton wrote:
> 
> vm-early-zone-reclaim
> 
>     Needs some convincing benchmark numbers to back it up.  Otherwise OK.

The only benchmarks I have for this were included in my last mail to
linux-mm:

http://marc.theaimsgroup.com/?l=linux-mm&m=111763597218177&w=2

Are they convincing?  Well, the patch doesn't seem to make the memory
thrashing case much worse ("make -j" kernbench run) which is a good
thing since the VM is trying to reclaim much earlier.

In the same e-mail I mention that there is a fairly good performance
gain in the optimal case, where processes are tied to a single node and
the node's memory is filled with page cache.  With zone reclaim turned
on the "make -j8" kernel build runs in 700 seconds;  735 seconds with
no reclaim.

mh

-- 
Martin Hicks                Wild Open Source Inc.
mort@wildopensource.com     613-266-2296

Re: -mm -> 2.6.13 merge status

[Andrew Morton]

Martin Hicks <mort@wildopensource.com> wrote:
>
> On Mon, Jun 20, 2005 at 11:54:58PM -0700, Andrew Morton wrote:
>  > 
>  > vm-early-zone-reclaim
>  > 
>  >     Needs some convincing benchmark numbers to back it up.  Otherwise OK.
> 
>  The only benchmarks I have for this were included in my last mail to
>  linux-mm:
> 
>  http://marc.theaimsgroup.com/?l=linux-mm&m=111763597218177&w=2
> 
>  Are they convincing?  Well, the patch doesn't seem to make the memory
>  thrashing case much worse ("make -j" kernbench run) which is a good
>  thing since the VM is trying to reclaim much earlier.
> 
>  In the same e-mail I mention that there is a fairly good performance
>  gain in the optimal case, where processes are tied to a single node and
>  the node's memory is filled with page cache.  With zone reclaim turned
>  on the "make -j8" kernel build runs in 700 seconds;  735 seconds with
>  no reclaim.

Ah, OK, I failed to capture that info.  (I always have to move the info in
the [patch 0/n] email into the first real patch, and this time I didn't)

Thanks.

Re: -mm -> 2.6.13 merge status

[Martin Hicks]

On Tue, Jun 21, 2005 at 12:54:57PM -0700, Andrew Morton wrote:
> 
> Ah, OK, I failed to capture that info.  (I always have to move the info in
> the [patch 0/n] email into the first real patch, and this time I didn't)

Oops.  I'll try to remember to stick the benchmark info into one of the
real patches next time.

mh

-- 
Martin Hicks                Wild Open Source Inc.
mort@wildopensource.com     613-266-2296

Re: -mm -> 2.6.13 merge status (wireless)

[Pavel Machek]

Hi!

> This summarises my current thinking on various patches which are presently
> in -mm.  I cover large things and small-but-controversial things.  Anything
> which isn't covered here (and that's a lot of material) is probably a "will
> merge", unless it obviously isn't.

I'd like to ask about 802.11 stack and ipw2100 in particular... Is it
"small enough that it did not need mentioning"?
Working wireless in mainline would be great...
				Pavel
-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: -mm -> 2.6.13 merge status (wireless)

[Andrew Morton]

Pavel Machek <pavel@ucw.cz> wrote:
>
> Hi!
> 
> > This summarises my current thinking on various patches which are presently
> > in -mm.  I cover large things and small-but-controversial things.  Anything
> > which isn't covered here (and that's a lot of material) is probably a "will
> > merge", unless it obviously isn't.
> 
> I'd like to ask about 802.11 stack and ipw2100 in particular... Is it
> "small enough that it did not need mentioning"?
> Working wireless in mainline would be great...

That's up to Jeff.

Re: -mm -> 2.6.13 merge status (wireless)

[Jeff Garzik]

Andrew Morton wrote:
> Pavel Machek <pavel@ucw.cz> wrote:
> 
>>Hi!
>>
>>
>>>This summarises my current thinking on various patches which are presently
>>>in -mm.  I cover large things and small-but-controversial things.  Anything
>>>which isn't covered here (and that's a lot of material) is probably a "will
>>>merge", unless it obviously isn't.
>>
>>I'd like to ask about 802.11 stack and ipw2100 in particular... Is it
>>"small enough that it did not need mentioning"?
>>Working wireless in mainline would be great...
> 
> 
> That's up to Jeff.

802.11 stack is still too ipw-specific.

Someone needs to get together another driver using 802.11 stack (such as 
HostAP, the original location of much of the code).

So, the merge criteria is:  something other than ipw uses it.

Otherwise, it'll never be generic...

	Jeff, who has several SuSE wireless patches to merge still

Re: -mm -> 2.6.13 merge status

[Jeff Garzik]

Andrew Morton wrote:
> git-ocfs
> 
>     The OCFS2 filesystem.  OK by me, although I'm not sure it's had enough
>     review.

Every time I come up with a complaint about ocfs2, the Oracle guys 
manage to shoot me down.  I think it's OK to merge.


> sparsemem
> 
>     OK by me for a merge.  Need to poke arch maintainers first, check that
>     they've looked at it sufficiently closely.

seems sane, though there are some whitespace niggles that should be 
cleaned up


> rapidio-*
> 
>     Will merge.

send through netdev, as is proper


> dlm-*.patch: Red Hat distributed lock manager
> 
>     Hard.  Right now it seems that no in-kernel projects will use this and
>     only one out-of-kernel project will use it.  Shelve the problem until
>     after Kernel Summit, where some light may be shed.
> 
>     Opinions are sought...

I hate to say it, since its my own employer, but I agree:  wait for 
in-kernel users, at the very least.


> connector.patch
> 
>     Nice idea IMO, but there are still questions around the
>     implementation.  More dialogue needed ;)
> 
> connector-add-a-fork-connector.patch
> 
>     OK, but needs connector.

I don't like connector


> inotify
> 
>     There are still concerns about the userspace API and internal
>     implementation details.  More slogging needed.

We should ask hpa what he needs for kernel.org.  Ideally kernel.org 
probably wants <something> that facilitates listening to <something> for 
a list of files being changed.  That would greatly speed up the robots, 
and possibly rsync-like activities too.


> pcmcia-*.patch
> 
>     Makes the pcmcia layer generate hotplug events and deprecates cardmgr.
>     Will merge.

Testing?  The goal behind the patch is certainly good, but I worry about 
exposure.


> cachefs
> 
>     This is a ton of code which knows rather a lot about pagecache
>     internals.  It allows the AFS client to cache file contents on a local
>     blockdev.
> 
>     I don't think it's a justified addition for only AFS and I'd prefer to
>     see it proven for NFS as well.
> 
>     Issues around add-page-becoming-writable-notification.patch need to
>     be resolved.
> 
> cachefs-for-nfs
> 
>     A recent addition.  Needs review from NFS developers and considerably
>     more testing.
> 
>     These things aren't looking likely for 2.6.13.

If I could vote more than once, I would!  I really like cachefs, and 
have been pushing for its inclusion for a while.


> kexec and kdump
> 
>     I guess we should merge these.
> 
>     I'm still concerned that the various device shutdown problems will
>     mean that the success rate for crashing kernels is not high enough for
>     kdump to be considered a success.  In which case in six months time we'll
>     hear rumours about vendors shipping wholly different crashdump
>     implementations, which would be quite bad.
> 
>     But I think this has gone as far as it can go in -mm, so it's a bit of
>     a punt.

I'm not particularly pleased with these, and indeed vendors ARE shipping 
other crashdump methods.


> reiser4
> 
>     Merge it, I guess.
> 
>     The patches still contain all the reiser4-specific namespace
>     enhancements, only it is disabled, so it is effectively dead code.  Maybe
>     we should ask that it actually be removed?

The plugin stuff is crap.  This is not a filesystem but a filesystem + 
new layer.  IMO considered in that light, it duplicates functionality 
elsewhere.


> v9fs
> 
>     I'm not sure that this has a sufficiently high
>     usefulness-to-maintenance-cost ratio.

agreed (though I think 9P is neat)


>       It has been said that a userspace NFS server can be used to get
>       full NFS server functionality with FUSE.  I think the half-assed kernel
>       implementation should be done away with.

"It has been said" -- its true.  A userspace NFS server can do 100% of 
userspace FS functionality.

	Jeff

Re: -mm -> 2.6.13 merge status

[Robert Love]

On Tue, 2005-06-21 at 11:26 -0400, Jeff Garzik wrote:

> > inotify
> > 
> >     There are still concerns about the userspace API and internal
> >     implementation details.  More slogging needed.
> 
> We should ask hpa what he needs for kernel.org.  Ideally kernel.org 
> probably wants <something> that facilitates listening to <something> for 
> a list of files being changed.  That would greatly speed up the robots, 
> and possibly rsync-like activities too.

I've talked to some people who've hooked inotify into rsync
successfully.  Cool hack.

	Robert Love

Re: -mm -> 2.6.13 merge status

[Christoph Lameter]

On Tue, 21 Jun 2005, Robert Love wrote:

> > We should ask hpa what he needs for kernel.org.  Ideally kernel.org 
> > probably wants <something> that facilitates listening to <something> for 
> > a list of files being changed.  That would greatly speed up the robots, 
> > and possibly rsync-like activities too.
> 
> I've talked to some people who've hooked inotify into rsync
> successfully.  Cool hack.

I noticed that select() is not working on real files. Could inotify 
be used to fix select()?

Re: -mm -> 2.6.13 merge status

[Robert Love]

On Tue, 2005-06-21 at 12:22 -0700, Christoph Lameter wrote:

> I noticed that select() is not working on real files. Could inotify 
> be used to fix select()?

Select the system call?  It should work fine.   ;-)

Who is confused?

	Robert Love

Re: -mm -> 2.6.13 merge status

[Christoph Lameter]

On Tue, 21 Jun 2005, Robert Love wrote:

> On Tue, 2005-06-21 at 12:22 -0700, Christoph Lameter wrote:
> 
> > I noticed that select() is not working on real files. Could inotify 
> > be used to fix select()?
> 
> Select the system call?  It should work fine.   ;-)

Hmmm. I just wrote an app that uses select to do essentially a "tail" 
waiting for new content in a log file. The file descriptors for real disk 
files are always ready even if there is no content available for the 
application.

The file is positioned at the end of the file after open via lseek.
select tells me that data is available but the read() returns zero bytes.

The current fix on the app level is to checking if useful work was 
done as a result of "READY" file descriptors. If the read() operations
do not return any data then the app will simply sleep for a couple of 
seconds. So the app degenerates to a kind of poll mode if disk files are 
used.

Re: -mm -> 2.6.13 merge status

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

On Tue, 2005-06-21 at 15:38 -0400, Robert Love wrote:
> On Tue, 2005-06-21 at 12:22 -0700, Christoph Lameter wrote:
> 
> > I noticed that select() is not working on real files. Could inotify 
> > be used to fix select()?
> 
> Select the system call?  It should work fine.   ;-)
> 
> Who is confused?
> 
> 	Robert Love

Sounds interesting.  tail -f could use it.  Instead of sleep 1, seek to
current position, read to eof; just select() for read on the file and
sleep in select() until someone else writes to that file.  

I've never tried doing that.  It might work, for all I know.
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: -mm -> 2.6.13 merge status

[Christoph Lameter]

On Tue, 21 Jun 2005, Zan Lynx wrote:

> I've never tried doing that.  It might work, for all I know.

I was told that Linux cannot do this. It always returns the filehandles as 
ready for disk files.

Re: -mm -> 2.6.13 merge status

[Robert Love]

On Tue, 2005-06-21 at 13:06 -0700, Christoph Lameter wrote:
> On Tue, 21 Jun 2005, Zan Lynx wrote:
> 
> > I've never tried doing that.  It might work, for all I know.
> 
> I was told that Linux cannot do this. It always returns the filehandles as 
> ready for disk files.

Inotify would definitely work.

	Robert Love

Re: -mm -> 2.6.13 merge status

[Christoph Lameter]

On Tue, 21 Jun 2005, Robert Love wrote:

> > I was told that Linux cannot do this. It always returns the filehandles as 
> > ready for disk files.
> 
> Inotify would definitely work.

Well we could use it in kernel to make select() work correctly. For disk 
files set up a notification for write and then only return from select if 
new data is available.

Re: -mm -> 2.6.13 merge status

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]

On Tue, 2005-06-21 at 13:10 -0700, Christoph Lameter wrote:
> On Tue, 21 Jun 2005, Robert Love wrote:
> 
> > > I was told that Linux cannot do this. It always returns the filehandles as 
> > > ready for disk files.
> > 
> > Inotify would definitely work.
> 
> Well we could use it in kernel to make select() work correctly. For disk 
> files set up a notification for write and then only return from select if 
> new data is available.

You could do it inside glibc.
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: -mm -> 2.6.13 merge status

[Matthias Urlichs]

Hi, Christoph Lameter wrote:

> Well we could use it in kernel to make select() work correctly.

select() already works correctly. It answers the "will I not block if I
call read()/write() on this" question, and since you never block on files
(assuming infinite disk speed ;-) select() will always return True on it.

You can't change this, it's in POSIX.

... or maybe I misunderstood your comment.

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
"I don't really miss God
 but i sure miss Santa Claus!"
      [Courtney Love]

Re: -mm -> 2.6.13 merge status

[Alan Cox]

On Maw, 2005-06-21 at 21:06, Christoph Lameter wrote:
> On Tue, 21 Jun 2005, Zan Lynx wrote:
> 
> > I've never tried doing that.  It might work, for all I know.
> 
> I was told that Linux cannot do this. It always returns the filehandles as 
> ready for disk files.

That is because disk files are always ready - select/poll are for waits
for data (or space) to become available not for events in the sense of
inotify.
That said there *is* scope in the poll() API [but not select()] to add a
new kind of poll notification type.

Re: -mm -> 2.6.13 merge status

[Stephen Hemminger]

On Tue, 21 Jun 2005 14:02:09 -0600
Zan Lynx <zlynx@acm.org> wrote:

> On Tue, 2005-06-21 at 15:38 -0400, Robert Love wrote:
> > On Tue, 2005-06-21 at 12:22 -0700, Christoph Lameter wrote:
> > 
> > > I noticed that select() is not working on real files. Could inotify 
> > > be used to fix select()?
> > 
> > Select the system call?  It should work fine.   ;-)
> > 
> > Who is confused?
> > 
> > 	Robert Love
> 
> Sounds interesting.  tail -f could use it.  Instead of sleep 1, seek to
> current position, read to eof; just select() for read on the file and
> sleep in select() until someone else writes to that file.  
> 
> I've never tried doing that.  It might work, for all I know.
> -- 
> Zan Lynx <zlynx@acm.org>


Posix requires select() of regular files always return true:
	http://www.opengroup.org/onlinepubs/009695399/functions/select.html

	File descriptors associated with regular files shall always select true for ready to read, 
	ready to write, and error conditions.

-- 
Stephen Hemminger	<shemminger@osdl.org>

Re: -mm -> 2.6.13 merge status

[Jeff Garzik]

Christoph Lameter wrote:
> On Tue, 21 Jun 2005, Robert Love wrote:
> 
> 
>>>We should ask hpa what he needs for kernel.org.  Ideally kernel.org 
>>>probably wants <something> that facilitates listening to <something> for 
>>>a list of files being changed.  That would greatly speed up the robots, 
>>>and possibly rsync-like activities too.
>>
>>I've talked to some people who've hooked inotify into rsync
>>successfully.  Cool hack.
> 
> 
> I noticed that select() is not working on real files. Could inotify 
> be used to fix select()?

Non-blocking file I/O is an open issue.

AIO is probably a better path.

	Jeff

Re: -mm -> 2.6.13 merge status

[Christoph Lameter]

On Tue, 21 Jun 2005, Jeff Garzik wrote:

> Non-blocking file I/O is an open issue.
> 
> AIO is probably a better path.

AIO is requiring you to poll and check if I/O is complete. select() does 
not require any polling and just needs to be made to work the way it was 
intended to.

Re: -mm -> 2.6.13 merge status

[Jeff Garzik]

Christoph Lameter wrote:
> On Tue, 21 Jun 2005, Jeff Garzik wrote:
> 
> 
>>Non-blocking file I/O is an open issue.
>>
>>AIO is probably a better path.
> 
> 
> AIO is requiring you to poll and check if I/O is complete. select() does 

Incorrect.  The entire point of AIO is that its an async callback 
system, when the I/O is complete...  just like the kernel's internal I/O 
request queue system.

	Jeff

Re: -mm -> 2.6.13 merge status

[Christoph Lameter]

On Tue, 21 Jun 2005, Jeff Garzik wrote:

> > AIO is requiring you to poll and check if I/O is complete. select() does 
> 
> Incorrect.  The entire point of AIO is that its an async callback system, when
> the I/O is complete...  just like the kernel's internal I/O request queue
> system.

Hmmm.. Okay it may work like dnotify. You get some signal and 
then its up to you to figure out what was going on. Traditionally select() 
does that all for you and tells you which stream got input.

Re: -mm -> 2.6.13 merge status

[Matt Porter]

On Tue, Jun 21, 2005 at 11:26:44AM -0400, Jeff Garzik wrote:
> Andrew Morton wrote:
> > rapidio-*
> > 
> >     Will merge.
> 
> send through netdev, as is proper

rapidio-support-net-driver.patch is the only netdev portion.

-Matt

Re: -mm -> 2.6.13 merge status

[randy_dunlap]

On Tue, 21 Jun 2005 11:26:44 -0400 Jeff Garzik wrote:

| Andrew Morton wrote:
| 
| > connector.patch
| > 
| >     Nice idea IMO, but there are still questions around the
| >     implementation.  More dialogue needed ;)
| > 
| > connector-add-a-fork-connector.patch
| > 
| >     OK, but needs connector.
| 
| I don't like connector

can you be more specific, like you did with reiser4?


| > kexec and kdump
| > 
| >     I guess we should merge these.
| > 
| >     I'm still concerned that the various device shutdown problems will
| >     mean that the success rate for crashing kernels is not high enough for
| >     kdump to be considered a success.  In which case in six months time we'll
| >     hear rumours about vendors shipping wholly different crashdump
| >     implementations, which would be quite bad.
| > 
| >     But I think this has gone as far as it can go in -mm, so it's a bit of
| >     a punt.
| 
| I'm not particularly pleased with these, and indeed vendors ARE shipping 
| other crashdump methods.

any specifics on the "not particularly pleased" part?

| > reiser4
| > 
| >     Merge it, I guess.
| > 
| >     The patches still contain all the reiser4-specific namespace
| >     enhancements, only it is disabled, so it is effectively dead code.  Maybe
| >     we should ask that it actually be removed?
| 
| The plugin stuff is crap.  This is not a filesystem but a filesystem + 
| new layer.  IMO considered in that light, it duplicates functionality 
| elsewhere.

I don't think that r4 is just a filesystem either, but you know more
about that than I do.


thanks,
---
~Randy

Re: -mm -> 2.6.13 merge status

[Hans Reiser]

Jeff Garzik wrote:

>
>
>> reiser4
>>
>>    
>
>
> The plugin stuff is crap.  This is not a filesystem but a filesystem +
> new layer.  IMO considered in that light, it duplicates functionality
> elsewhere.


What functionality where?  Please remember that this is per file, per
item, per node, per attribute, per disk format, per bitmap, per super
block, etc.,  abstracting, not per filesystem abstracting.

Plugins allow a number of things:

1) They allow us to never pay the cost to change the disk format again,
no matter how much we add in future years.  This really matters: the
prohibitive cost of disk format changes are the number one impediment to
filesystem improvements, and the primary reason why most filesystems
ossify after time has past.

2) They allow us to easily structure code for reuse.   If we want to
create a new kind of file that is like some other kind of file except
for one thing, we just write the one thing, and then easily reuse all
the other code, and create a new plugin id. 

The use of plugins forced all the programmers to think about reusability
at every layer of design.  V3 of reiserfs is way too hard to work on and
modify.  If you ask one of the team to code something for V3 instead of
V4, they quietly groan at the thought.  It is just so much easier to do
in V4.

When I asked one of our team to completely change the key assignment
algorithm for V4 (which controls what things get packed near what in the
tree), he complained that it would take 6 weeks to do it.  Under V3 it
would have taken 3 months.  It took him 3 days, and now to change it
again would take him 3 hours I think.  Oh, by the way, the change
boosted performance dramatically.

If you take the time to become familiar with coding within our plugin
layer, I think you will find yourself wanting the same to exist for
other filesystems.  Of course, no other filesystem needs to be impacted
by our plugin layer, and there is no way reiser4 could easily be
rewritten to exist without it (it would be like requiring that the
kernel not use function calls and only use gotos). 

Reiser4's plugin layer has as its explicit objective making it possible
for the weekend hacker to add something useful to reiser4 and send it in
for inclusion.  We want to democratize filesystem innovation so that
random bright guys who usually work on something other than filesystems
can act on their bright ideas without spending 3 years in the filesystem
field to do it.  I believe that most great filesystem innovations are
envisioned by persons not working on filesystems, and go nowhere because
of the especially high cost of entry into our field.

I am not as bright as other filesystem developers, and so we must tinker
with three ideas and keep one of them.  The speed of tinkering is
crucial to us, and the plugin layer increases that speed several fold.

Please reconsider your view. 

Re: -mm -> 2.6.13 merge status

[Christoph Hellwig]

Hans, we had this discussion before.  And the consensus was pretty simple:
the disk structure plugins are your business and fine to keep.  The
higher-level pluging that just add another useless abstraction below
file_operation/inode_operation/etc.  are not.  keep the former and kill
the latter and you're one step further.

reiser4 plugins

[Hans Reiser]

Christoph,

Reiser4 users love the plugin concept, and all audiences which have
listened to a presentation on plugins have been quite positive about
it.  Many users think it is the best thing about reiser4.  Can you
articulate why you are opposed to plugins in more detail?  Perhaps you
are simply not as familiar with it as the audiences I have presented
to.  Perhaps persons on our mailing list can comment.....

In particular, what is wrong with having a plugin id associated with
every file, storing the pluginid on disk in permanent storage in the
stat data, and having that plugin id define the set of methods that
implement the vfs operations associated with a particular file, rather
than defining VFS methods only at filesystem granularity?

What is wrong with having an encryption plugin implemented in this
manner?  What is wrong with being able to have some files implemented
using a compression plugin, and others in the same filesystem not.

What is wrong with having one file in the FS use a write only plugin, in
which the encrypion key is changed with every append in a forward but
not backward computable manner, and in order to read a file you must
either have a key that is stored on another computer or be reading what
was written after the moment of cracking root?

What is wrong with having a set of critical data files use a CRC
checking file plugin?

What we have hurts no one but us.  I have never seen an audience for one
of my talks that thought it hurt us..... most audiences think it is
great.  

Let us tinker with our FS, and you tinker with yours, and so long as
what we do does not affect your FS, let the users choose.

In the end, somebody will write a new fs that steals the good ideas from
both of us, and obsoletes us both.  They can only do this though if we
are left to be both free to implement differing filesystem designs.

I do not tell you how to design XFS, why are you making my life unpleasant?

Christoph Hellwig wrote:

>Hans, we had this discussion before.  And the consensus was pretty simple:
>the disk structure plugins are your business and fine to keep.  The
>higher-level pluging that just add another useless abstraction below
>file_operation/inode_operation/etc.  are not.  keep the former and kill
>the latter and you're one step further.
>
>
>  
>

Re: reiser4 plugins

[Jeff Garzik]

Hans Reiser wrote:
> Christoph,
> 
> Reiser4 users love the plugin concept, and all audiences which have
> listened to a presentation on plugins have been quite positive about
> it.  Many users think it is the best thing about reiser4.  Can you
> articulate why you are opposed to plugins in more detail?  Perhaps you
> are simply not as familiar with it as the audiences I have presented
> to.  Perhaps persons on our mailing list can comment.....
> 
> In particular, what is wrong with having a plugin id associated with
> every file, storing the pluginid on disk in permanent storage in the
> stat data, and having that plugin id define the set of methods that
> implement the vfs operations associated with a particular file, rather
> than defining VFS methods only at filesystem granularity?

You're basically implementing another VFS layer inside of reiser4, which 
is a big layering violation.

This sort of feature should -not- be done at the low-level filesystem level.

What happens if people decide plugins are a good idea, and they want 
them in ext3?  We need massive surgery to extract the guts from reiser4.

	Jeff

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeff Garzik wrote:
> Hans Reiser wrote:
> 
>> Christoph,
>>
>> Reiser4 users love the plugin concept, and all audiences which have
>> listened to a presentation on plugins have been quite positive about
>> it.  Many users think it is the best thing about reiser4.  Can you
>> articulate why you are opposed to plugins in more detail?  Perhaps you
>> are simply not as familiar with it as the audiences I have presented
>> to.  Perhaps persons on our mailing list can comment.....
>>
>> In particular, what is wrong with having a plugin id associated with
>> every file, storing the pluginid on disk in permanent storage in the
>> stat data, and having that plugin id define the set of methods that
>> implement the vfs operations associated with a particular file, rather
>> than defining VFS methods only at filesystem granularity?
> 
> 
> You're basically implementing another VFS layer inside of reiser4, which
> is a big layering violation.

There's been sloppy code in the kernel before.  I remember one bit in
particular which was commented "Fuck me gently with a chainsaw."  If I
remember correctly, this had all of the PCI ids and the names and
manufacturers of the corresponding devices -- in a data structure -- in
C source code.  It was something like one massive array definition, or
maybe it was a structure.  I don't remember exactly, but...

The point is, this was in the kernel for quite awhile, and it was so
ugly that someone would rather be fucked with a chainsaw.  If something
that bad can make it in the kernel and stay for awhile because it
worked, and no one wanted to replace it -- nowdays, that database is
kept in userland as some nice text format -- then I vote for putting
Reiser4 in the kernel now, and ironing the sloppiness ("violation") out
later.  It runs now.

> This sort of feature should -not- be done at the low-level filesystem
> level.

I agree there, too.  In fact, some people have suggested that all
"legacy" (read: non-reiser) filesystems should be implemented as Reiser4
plugins, effectively killing VFS.*

So, Reiser4 may eventually take over VFS and be the only Linux
filesystem, but if so, it will have to do it much more slowly.  Take the
good ideas -- things like plugins -- and make them at least look like
incremental updates to the current VFS, and make them available to all
filesystems.

Eventually, this would result in a full merge of Reiser and Linux, such
that the only thing left of "Reiser4" are a few plugins -- things like
storage plugins and maybe some more exotic things like fibration that I
don't really understand.

> What happens if people decide plugins are a good idea, and they want
> them in ext3?  We need massive surgery to extract the guts from reiser4.

And here is the crucial point.  Reiser4 is usable and useful NOW, not
after it has undergone massive surgery, and Namesys is bankrupt, and
users have given up and moved on to XFS.  But the massive surgery should
happen eventually, partly to make all filesystems better (see below),
and partly to make the transition easier and more palatable for those
who don't work for Namesys.





* Imagine ext3 as a storage-level plugin for reiser4.  You get one
benefit immediately -- lazy allocation.  Lazy allocation is nice both
for fragmentation and for busy systems writing and nuking a lot of
temporary files.  Imagine a file which is written and then deleted
before it hits disk -- it should never touch the disk, regardless of the
underlying storage layout.

Other benefits are subtler but inevitable.  Right now, it would be an
understatement to say that there's duplication of effort between ext3,
xfs, and reiser4.  And yet, I don't think there are many core design
decisions that influence my decision as to which I pick up.  I just want
it to be fast, stable, and have a stable on-disk format so I don't have
to reformat too often.  I honestly don't care about how XFS is
segmenting the disk, or Reiser4 packs really well, or ext3 can be read
as ext2 and flushes to disk every five seconds.  These are the kinds of
things which should be set to sane defaults, tunable for enterprise
users, but are not really enough to warrant completely separate code bases.

I would imagine that it wouldn't be too long after this before an
uber-fs rose, something which combined enough of the strong points of
all the existing Linux filesystems that no one would use anything else.
 But, Linux still needs support for FAT32, ISO9660, UDF, and all the
other filesystems which won't go away as easily as XFS and ext3, and it
would be nice if these could all share as much code as possible.


I don't know if storage plugins really work that way, but they should.

I think.  I don't work here.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrjoNHgHNmZLgCUhAQIYYw/7BWZ0gVvy0ln5tRo405yUoRHJ/jVFBGyP
5pQ610ECMZORVWRO1qP/NXbvGZwDjEggM5iIeahsGqnBWzNGDsB6RslMUZAxzCYy
iAovi0881zoARf3dmhKrDgbGkvNLPTx+/ypa20oRcHLnyI+NAjerUxNuvGc79PJn
Ybm9JhX6tQsqGIKjZye9uNDHCifLqzA1gdxucPwWo9sz4ymzM9FgsMGvb+IxrU50
2a2G2K6AHcH+nkomEHw3xgY3PmUZUy0s2hQDSkJx6dCido7fGZwwykaSXm4IZs9s
xZqBxKx32G/LDnDV3W8gpj8ZisUE+58kefRbIjo4Ml6IzgXvQqpRjaQOuz8JoKDX
9KUV43tcZkPpK+neIYPQYpXCrdMQqB5+ISpbNKuz/Z/abkcVw1042sy0EKM+/VnD
n3Jr7PpSyk0lfCyADr+33qnqPQRFq02gQTohg9FheoMthhV01aaeGW5ExmTM/Wwa
6Dmv/qnn2oNImi+Ebz5u3Lbnz7lL3MVpRL4aoMmEOVUB3xhehnxesf//yacBmwj9
M/3KVae9epwX4K6yi8qQXzH4160IBFHpWUxBLc9LnOZlHQuZ+Fz3BIrbKQBvmaHT
7lrwi0Os6TmiBGMSFd6OUWHcYs4p97aMw30NG33fCL6e8X6oNVFwwnJXZpBPN1Mr
+lrsVAywKEI=
=rHdK
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Jeff Garzik]

David Masover wrote:
> There's been sloppy code in the kernel before.  I remember one bit in
> particular which was commented "Fuck me gently with a chainsaw."  If I
> remember correctly, this had all of the PCI ids and the names and
> manufacturers of the corresponding devices -- in a data structure -- in
> C source code.  It was something like one massive array definition, or
> maybe it was a structure.  I don't remember exactly, but...
> 
> The point is, this was in the kernel for quite awhile, and it was so
> ugly that someone would rather be fucked with a chainsaw.  If something
> that bad can make it in the kernel and stay for awhile because it
> worked, and no one wanted to replace it -- nowdays, that database is
> kept in userland as some nice text format -- then I vote for putting
> Reiser4 in the kernel now, and ironing the sloppiness ("violation") out
> later.  It runs now.

Existence of ugly code is not an excuse to add more.

We have to maintain said ugly code for decades.  Maintainability is a 
big deal when you deal with the timeframes we deal with.


> So, Reiser4 may eventually take over VFS and be the only Linux
> filesystem, but if so, it will have to do it much more slowly.  Take the
> good ideas -- things like plugins -- and make them at least look like
> incremental updates to the current VFS, and make them available to all
> filesystems.

This is how all Linux development is done.

The code evolves over time.

You have just described the path ReiserFS needs to take, in order to get 
this stuff into the kernel, in fact.


> And here is the crucial point.  Reiser4 is usable and useful NOW, not
> after it has undergone massive surgery, and Namesys is bankrupt, and
> users have given up and moved on to XFS.  But the massive surgery should
> happen eventually, partly to make all filesystems better (see below),
> and partly to make the transition easier and more palatable for those
> who don't work for Namesys.

We care about technical merit, not some random company's financial 
solvancy.  Reiser4 has layering violations, and doesn't work with the 
current security layer.  Those are two biggies.

There is no entitlement to be merged in the upstream kernel.  If people 
don't like how the Linux kernel is managed, they are free to maintain 
their own fork.  If its better, people will use that instead.


> I would imagine that it wouldn't be too long after this before an
> uber-fs rose, something which combined enough of the strong points of
> all the existing Linux filesystems that no one would use anything else.
>  But, Linux still needs support for FAT32, ISO9660, UDF, and all the
> other filesystems which won't go away as easily as XFS and ext3, and it
> would be nice if these could all share as much code as possible.
> 
> 
> I don't know if storage plugins really work that way, but they should.

No, they shouldn't.


> I think.  I don't work here.

Obviously.

	Jeff

Re: reiser4 plugins

[Hans Reiser]

Jeff Garzik wrote:

>> after it has undergone massive surgery, and Namesys is bankrupt, and
>> users have given up and moved on to XFS.  But the massive surgery should
>> happen eventually, partly to make all filesystems better (see below),
>> and partly to make the transition easier and more palatable for those
>> who don't work for Namesys.
>
> And here is the crucial point.  Reiser4 is usable and useful NOW, not
>
> We care about technical merit, not some random company's financial
> solvancy. 
>
>
>
Hmm,   that must be why Daniel Robbins, the creator of the distro with
the technically superior Gentoo approach, is still with us, yes?

If you cared about technical merit, you would discuss benchmarks, yes?

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeff Garzik wrote:
> David Masover wrote:
> 
>> There's been sloppy code in the kernel before.  I remember one bit in
>> particular which was commented "Fuck me gently with a chainsaw."  If I
>> remember correctly, this had all of the PCI ids and the names and
>> manufacturers of the corresponding devices -- in a data structure -- in
>> C source code.  It was something like one massive array definition, or
>> maybe it was a structure.  I don't remember exactly, but...
>>
>> The point is, this was in the kernel for quite awhile, and it was so
>> ugly that someone would rather be fucked with a chainsaw.  If something
>> that bad can make it in the kernel and stay for awhile because it
>> worked, and no one wanted to replace it -- nowdays, that database is
>> kept in userland as some nice text format -- then I vote for putting
>> Reiser4 in the kernel now, and ironing the sloppiness ("violation") out
>> later.  It runs now.
> 
> 
> Existence of ugly code is not an excuse to add more.

Maybe not, but I'm making a point.  I'm sure that, at the time, people
wanted something that ran.  They wanted lspci to work.  It was generally
assumed that it would be cleaned up later, and it was.  Too much later,
but it happened, eventually.

I've been reading a bit of history, and the reason Linux got so popular
in the first place was the tendency to include stuff that worked and
provided a feature people wanted, even if it was ugly.  The philosophy
would be:  choose a good implementation over an ugly one, but choose an
ugly one over nothing at all.

> We have to maintain said ugly code for decades.  Maintainability is a
> big deal when you deal with the timeframes we deal with.

Maintainability is like optimization.  The maintainability of a
non-working program is irrelevant.  You'd be right if we already had
plugins-in-the-VFS.  We don't.  The most maintainable solution for
plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
- -- because it is the _only_ one that actually exists right now.

>> So, Reiser4 may eventually take over VFS and be the only Linux
>> filesystem, but if so, it will have to do it much more slowly.  Take the
>> good ideas -- things like plugins -- and make them at least look like
>> incremental updates to the current VFS, and make them available to all
>> filesystems.
> 
> 
> This is how all Linux development is done.
> 
> The code evolves over time.
> 
> You have just described the path ReiserFS needs to take, in order to get
> this stuff into the kernel, in fact.

This is the path it needs to take, long term, for the sanity of
everyone.  But I don't think it can get there without being included,
short term.  People will stomp all over any attempts to change the VFS
as "unproven" and "unneccesary".  Do you think it will be any easier to
get this stuff into the kernel that way?

Better, I think, to drop it in, as-is, and move stuff incrementally from
the FS to the VFS.  That way, there's at least something intermediate
for people to use, test, and hammer/beat down, and for maintainers to
get more comfortable with the idea and the logistics of this beast in
their kernel.

>> And here is the crucial point.  Reiser4 is usable and useful NOW, not
>> after it has undergone massive surgery, and Namesys is bankrupt, and
>> users have given up and moved on to XFS.  But the massive surgery should
>> happen eventually, partly to make all filesystems better (see below),
>> and partly to make the transition easier and more palatable for those
>> who don't work for Namesys.
> 
> 
> We care about technical merit, not some random company's financial
> solvancy.  Reiser4 has layering violations, and doesn't work with the
> current security layer.  Those are two biggies.

Ah, so you mean to say, you care about how well something fits the
current model.  That has little to do with technical merit.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrkqoHgHNmZLgCUhAQIKixAAnTJTJJpuCLatFp8syccjNnE7WdHlO/zx
G1bWuSthCnb7uaVb8buDeAlpArzttoguKKum0NE0khz/FjKw4YUXH4AEsVYGlZaO
nBYpw0MVyypNP+hZhEuo1T826frEOb6Z40Y1WZCpYwAZCs9EQQm7TeYSMjhD17Ew
PehYwUFUmnv1S7CpYNQvuboKh/1wuUQb6R2thjuCGJVkif8Mn2U20Fhk1/HIgnIr
OHoCD8ZgwoqBDPKQ6V26dUX+ZHzQVJX1j/IgLnnitJ9E4quIHNs33lq4S99DWta6
uDS4hkHgFMRemh37sA0FUMeitFsrwNb2b2f0o/X8MpDJmwbdrdg9kxvwfHqqgaz+
Enj0rBXO8E+5a4STTk4L2LaSR2+knSEFdj53MYYq4ABL07hEbJp2cNFKh5AFXvg0
wg5WoHt4HhhOeuftIG9Twv5tHIC5qoM57ae9yZzj783G9ZnXy0xBefUmH+pWVQsp
H1IpKIR4a0l8gV1AkJa6BUyAyzDDObFzmqcZ61W15Y2relD9Ow2qzVqMxroB78uJ
O+on741BecGtohB5xdfth9rwDY6JPkDug3C6zHzxSAkGGEnWIn6O8CzcGrGqS0Ta
EmB4LGw/fZqGcEYOOErqMC6GuImv2JbjtBOx8nAxM2OhGXFoDiD9FQaDaxWw9zjj
ROODOhm0aTA=
=ivqd
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Nikita Danilov]

David Masover writes:

[...]

 > 
 > Maintainability is like optimization.  The maintainability of a
 > non-working program is irrelevant.  You'd be right if we already had
 > plugins-in-the-VFS.  We don't.  The most maintainable solution for
 > plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
 > - -- because it is the _only_ one that actually exists right now.

But it is not so. There _are_ plugins-in-the-VFS. VFS operates on opaque
objects (inodes, dentries, file system types) through interfaces:
{inode,address_space,dentry,sb,etc.}_operations. Every file system
back-end if free to implement whatever number of these interfaces. And
the do this already: check the sources; even ext2 does this: e.g.,
ext2_fast_symlink_inode_operations and ext2_symlink_inode_operations.

This is exactly what upper level reiser4 plugins are for.

I guess that one of Christoph Hellwig's complaints is that reiser4
introduces another layer of abstraction to implement something that
already exists.

Nikita.

Re: reiser4 plugins

[Vladimir Saveliev]

Hello

On Wed, 2005-06-22 at 18:28, Nikita Danilov wrote:
> David Masover writes:
> 
> [...]
> 
>  > 
>  > Maintainability is like optimization.  The maintainability of a
>  > non-working program is irrelevant.  You'd be right if we already had
>  > plugins-in-the-VFS.  We don't.  The most maintainable solution for
>  > plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
>  > - -- because it is the _only_ one that actually exists right now.
> 
> But it is not so. There _are_ plugins-in-the-VFS. VFS operates on opaque
> objects (inodes, dentries, file system types) through interfaces:
> {inode,address_space,dentry,sb,etc.}_operations. Every file system
> back-end if free to implement whatever number of these interfaces. And
> the do this already: check the sources; even ext2 does this: e.g.,
> ext2_fast_symlink_inode_operations and ext2_symlink_inode_operations.
> 

imho, this is something different. Ext2 decides itself how to manage a
symlink depending on length of string the symlink is to store.
Reiser4 plugins are to allow a user to define himself which operations
file is to be managed with.

> This is exactly what upper level reiser4 plugins are for.

> I guess that one of Christoph Hellwig's complaints is that reiser4
> introduces another layer of abstraction to implement something that
> already exists.
> 

I do not think that it exists already.
You can have standart type of files and that is all.

Linux filesystem is not supposed to provide anything but plain
regular/directory/symlinks/sockets/block device/char device/fifo files.

Existing file API does not allow create anything but that.
Merging reiser4 with object plugins would make it necessary to modify
VFS layer so that files of arbitrary types could be created.

Re: reiser4 plugins

[Nikita Danilov]

Vladimir Saveliev writes:
 > Hello
 > 
 > On Wed, 2005-06-22 at 18:28, Nikita Danilov wrote:
 > > David Masover writes:
 > > 
 > > [...]
 > > 
 > >  > 
 > >  > Maintainability is like optimization.  The maintainability of a
 > >  > non-working program is irrelevant.  You'd be right if we already had
 > >  > plugins-in-the-VFS.  We don't.  The most maintainable solution for
 > >  > plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
 > >  > - -- because it is the _only_ one that actually exists right now.
 > > 
 > > But it is not so. There _are_ plugins-in-the-VFS. VFS operates on opaque
 > > objects (inodes, dentries, file system types) through interfaces:
 > > {inode,address_space,dentry,sb,etc.}_operations. Every file system
 > > back-end if free to implement whatever number of these interfaces. And
 > > the do this already: check the sources; even ext2 does this: e.g.,
 > > ext2_fast_symlink_inode_operations and ext2_symlink_inode_operations.
 > > 
 > 
 > imho, this is something different. Ext2 decides itself how to manage a
 > symlink depending on length of string the symlink is to store.
 > Reiser4 plugins are to allow a user to define himself which operations
 > file is to be managed with.

I fail to see this as an important distinction:

 - ext2    decides what "plugin" to use based on parameters supplied to
 the file creation system call;

 - reiser4 decides what "plugin" to use based on parameters supplied to
 the file creation system call.

 > 
 > > This is exactly what upper level reiser4 plugins are for.
 > 
 > > I guess that one of Christoph Hellwig's complaints is that reiser4
 > > introduces another layer of abstraction to implement something that
 > > already exists.
 > > 
 > 
 > I do not think that it exists already.
 > You can have standart type of files and that is all.
 > 
 > Linux filesystem is not supposed to provide anything but plain
 > regular/directory/symlinks/sockets/block device/char device/fifo files.

If you are talking about S_IFMT bits they are just a relic to keep user
level happy (yes, VFS does few S_ISFOO checks here and there, but they
all can be replaced with checks for appropriate operations being
non-NULL). Nothing prevents file system from rolling forward whatever
exotic objects it wants.

 > 
 > Existing file API does not allow create anything but that.
 > Merging reiser4 with object plugins would make it necessary to modify
 > VFS layer so that files of arbitrary types could be created.
 > 

This is important, but it doesn't depend on whether file system has
additional layer if indirection below VFS operations. If file system
wants to support extended object types, additional parameters have to be
somehow communicated from the user level during object creation. But:

 - this is necessary even if file system has no "plugins" below VFS
 operations;

 - this doesn't necessary means changing VFS. For example, these
 parameters can be set on parent directory (this is how ACLs were
 integrated into POSIX).

 >

Nikita.

Re: reiser4 plugins

[Christoph Hellwig]

On Wed, Jun 22, 2005 at 04:08:49AM -0500, David Masover wrote:
> I've been reading a bit of history, and the reason Linux got so popular
> in the first place was the tendency to include stuff that worked and
> provided a feature people wanted, even if it was ugly.  The philosophy
> would be:  choose a good implementation over an ugly one, but choose an
> ugly one over nothing at all.

And things change over time.  Back in those days the linux codebase was
small and it was easy to change things all over the place.  These times
our codebase is huge, and people that know enough parts of the kernel to
do big changes are overloaded with work.  Thus we have to set our
acceptance criteria a lot higher now - else we'd be totally lost with
the current size of the project already.

> > We have to maintain said ugly code for decades.  Maintainability is a
> > big deal when you deal with the timeframes we deal with.
> 
> Maintainability is like optimization.  The maintainability of a
> non-working program is irrelevant.  You'd be right if we already had
> plugins-in-the-VFS.  We don't.  The most maintainable solution for
> plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
> - -- because it is the _only_ one that actually exists right now.

We do have plugins in the VFS, every filesystem is a set of a few of them
and some gluecode.

<skipping a lot stuff>

David and Hans, I've read through my backlog a lot now, and I must say
it's pretty pointless - you're discussing lots of highlevel what if and
don't actually care about something as boring as actual technical details.

Hans has lots of very skillfull technical people like zam and vs, and maybe
he should give them some freedom to sort out technical issues for a basic
reiser4 merge, and one that is done we can turn back to discussion of
advanced features and their implementation, hopefully with a few more
arguments on both sides and a real technical discussion.

Re: reiser4 plugins

[Alexander Zarochentsev]

On Sunday 26 June 2005 21:02, Christoph Hellwig wrote:
> On Wed, Jun 22, 2005 at 04:08:49AM -0500, David Masover wrote:
> > I've been reading a bit of history, and the reason Linux got so popular
> > in the first place was the tendency to include stuff that worked and
> > provided a feature people wanted, even if it was ugly.  The philosophy
> > would be:  choose a good implementation over an ugly one, but choose an
> > ugly one over nothing at all.
>
> And things change over time.  Back in those days the linux codebase was
> small and it was easy to change things all over the place.  These times
> our codebase is huge, and people that know enough parts of the kernel to
> do big changes are overloaded with work.  Thus we have to set our
> acceptance criteria a lot higher now - else we'd be totally lost with
> the current size of the project already.
>
> > > We have to maintain said ugly code for decades.  Maintainability is a
> > > big deal when you deal with the timeframes we deal with.
> >
> > Maintainability is like optimization.  The maintainability of a
> > non-working program is irrelevant.  You'd be right if we already had
> > plugins-in-the-VFS.  We don't.  The most maintainable solution for
> > plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
> > - -- because it is the _only_ one that actually exists right now.
>
> We do have plugins in the VFS, every filesystem is a set of a few of them
> and some gluecode.
>
> <skipping a lot stuff>
>
> David and Hans, I've read through my backlog a lot now, and I must say
> it's pretty pointless - you're discussing lots of highlevel what if and
> don't actually care about something as boring as actual technical details.
>
> Hans has lots of very skillfull technical people like zam and vs, and maybe
> he should give them some freedom to sort out technical issues for a basic
> reiser4 merge, and one that is done we can turn back to discussion of
> advanced features and their implementation, hopefully with a few more
> arguments on both sides and a real technical discussion.

Unfortunately, this is not only a technical discussion...  it is about linux 
development model too.

Well, about the plugins. We can clean reiser4<->VFS interface up by setting 
per-vfs-object inode/dentry/super ops instead using of our own dispatcher.  
So different reiser4 inodes/files will have different i_ops/f_ops.  That 
would be only visible-in-VFS part of reiser4 object plugins.   

Would the help to solve "reiser4 plugins" question?  It is just as other FS do 
-- procfs  has seq_file and sysconfig interfaces below the VFS and l-k people 
do not complain each day about layering violation ;-)  Procfs is taken as an 
example because it deals with objects of different types, actually anybody 
may create own procfs objects more or less general way.

I don't belive that you want to see all reiser4-specific things as item 
plugins, disk format plugins in the VFS.

Thanks,
Alex.

Re: reiser4 plugins

[Christoph Hellwig]

On Mon, Jun 27, 2005 at 01:30:06PM +0400, Alexander Zarochentsev wrote:
> -- procfs  has seq_file and sysconfig interfaces below the VFS and l-k people 
> do not complain each day about layering violation ;-)  Procfs is taken as an 
> example because it deals with objects of different types, actually anybody 
> may create own procfs objects more or less general way.

seq_file actually works at the file_operations level, that's exactly
what I'm telling you to do.  The old sub-callbacks are on their way out.

> I don't belive that you want to see all reiser4-specific things as item 
> plugins, disk format plugins in the VFS.

If you'd read the previous discussions you'd see that no one complained
about disk format plugins.

Re: reiser4 plugins

[Alexander Zarochentsev]

On Monday 27 June 2005 13:42, Christoph Hellwig wrote:
> On Mon, Jun 27, 2005 at 01:30:06PM +0400, Alexander Zarochentsev wrote:
> > -- procfs  has seq_file and sysconfig interfaces below the VFS and l-k
> > people do not complain each day about layering violation ;-)  Procfs is
> > taken as an example because it deals with objects of different types,
> > actually anybody may create own procfs objects more or less general way.
>
> seq_file actually works at the file_operations level, that's exactly
> what I'm telling you to do.  The old sub-callbacks are on their way out.

not exactly.  I meant that seq_file has its own VFS-like thing struct 
seq_operations.

So I may assume that having own objects and their operations is allowed.  The 
complains are about adding unnecessary level of indirection in the trivial 
reiser4 wrappers as reiser4_write().   

> > I don't belive that you want to see all reiser4-specific things as item
> > plugins, disk format plugins in the VFS.
>
> If you'd read the previous discussions you'd see that no one complained
> about disk format plugins.

Re: reiser4 plugins

[Christoph Hellwig]

On Mon, Jun 27, 2005 at 03:28:49PM +0400, Alexander Zarochentsev wrote:
> not exactly.  I meant that seq_file has its own VFS-like thing struct 
> seq_operations.

It's not that VFS-like, it's more a set of callback than actual methods.
But yes, if you're actually doing work at a significant lower layer
adding abstractions make sense.  Note that seq_file.c while not beeing
the VFS is also a generic library that you can use with any filesystem
if you want to implement sequential synthetic files.

Re: reiser4 plugins

[Horst von Brand]

Alexander Zarochentsev <zam@namesys.com> wrote:
> On Sunday 26 June 2005 21:02, Christoph Hellwig wrote:

[...]

> > David and Hans, I've read through my backlog a lot now, and I must say
> > it's pretty pointless - you're discussing lots of highlevel what if and
> > don't actually care about something as boring as actual technical details.
> >
> > Hans has lots of very skillfull technical people like zam and vs, and maybe
> > he should give them some freedom to sort out technical issues for a basic
> > reiser4 merge, and one that is done we can turn back to discussion of
> > advanced features and their implementation, hopefully with a few more
> > arguments on both sides and a real technical discussion.

> Unfortunately, this is not only a technical discussion...  it is about linux 
> development model too.

Then better separate the two, keeping the technical discussion on LKML and
taking the development model stuff elsewhere.

> Well, about the plugins.

First step: Axe them. They are (at most) /configuration options/, that will
have to get fixed meanings, available to all ReiserFS filesystems purely
for compatibility reasons. Sure, do get wild on new options in your own
experimental trees, and migrate what survives into the standard version
(and then it'll be ReiserFS 4.1, 4.2, ..., 4.25, ...).

>                          We can clean reiser4<->VFS interface up by setting 
> per-vfs-object inode/dentry/super ops instead using of our own dispatcher.  

Sounds reasonable.

> So different reiser4 inodes/files will have different i_ops/f_ops.  That 
> would be only visible-in-VFS part of reiser4 object plugins.   

How would that work out from the userland (system call) perspective? How
does that get handled from on-disk format?

> Would the help to solve "reiser4 plugins" question?  It is just as other
> FS do -- procfs has seq_file and sysconfig interfaces below the VFS and
> l-k people do not complain each day about layering violation ;-) Procfs
> is taken as an example because it deals with objects of different types,
> actually anybody may create own procfs objects more or less general way.

But procfs /is/ quite special, as it is supposed to be a window into the
kernel, not real files. Some layering violation is unavoidable there.

> I don't belive that you want to see all reiser4-specific things as item 
> plugins, disk format plugins in the VFS.

Only what makes sense. Plus many of those will probably have to go. Decide
on /one/ way of doing things, even if not perfect for all uses. Everything
else is useless bloat.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Sean]

On Tue, June 28, 2005 10:01 am, Horst von Brand said:
>> I don't belive that you want to see all reiser4-specific things as item
>> plugins, disk format plugins in the VFS.
>
> Only what makes sense. Plus many of those will probably have to go. Decide
> on /one/ way of doing things, even if not perfect for all uses. Everything
> else is useless bloat.

It doesn't seem to be a problem as long as loadable plugins are GPL.  How
is it useless bloat?  It doesn't seem much different from having loadable
modules in the security system.  Just don't enable Reiser4 in your kernel
if you don't want to use it.

Sean

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sean wrote:
> On Tue, June 28, 2005 10:01 am, Horst von Brand said:
> 
>>>I don't belive that you want to see all reiser4-specific things as item
>>>plugins, disk format plugins in the VFS.
>>
>>Only what makes sense. Plus many of those will probably have to go. Decide
>>on /one/ way of doing things, even if not perfect for all uses. Everything
>>else is useless bloat.
> 
> 
> It doesn't seem to be a problem as long as loadable plugins are GPL.  How

"Loadable" how?

reiser4 plugins must be compiled in, either directly to the kernel, or
part of reiser4.ko

The only thing "loadable" about them is that some can be turned on or
off for individual files.

> is it useless bloat?  It doesn't seem much different from having loadable
> modules in the security system.  Just don't enable Reiser4 in your kernel
> if you don't want to use it.

This is not getting us anywhere.  It seems to be an issue of overall
design and neatness at this point.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsGkwXgHNmZLgCUhAQKUlw/6AxzaT+5R7lzhy4d/W55KY7m+8QJA90Ut
3m+E/hiSKykolERURKUoRPEK+jly8DUxLS+UpA5ZQwxzkR4YipMcoW5i3lHyJ9GB
Hf9lBtGu0TwCF8W9Qh1Fk/7mwQcvyz6ATUYOYqvSA5Wxfr8BDFdcJUqTwMRCOxrA
Fh5oOSBXEhqnSElfvurBkzMNurOmNnkpeZt8rM6qXqeYRRPrmIWmFGvkufoyb/q7
N2wdwtwoubWokl8FLWHQmH0f4pJqJUNXpDQvw67zKi24zOUq8ZZ7xH7EE8vNI9s9
ppVVsrLiAFE2/GeDRIdbQNSuClliq7XZV4AMHkorik7Qe0nIeR4hBr9hYI41l/Ie
G3A9yP/69XsOodUWMZB/asxj3zMEhDynz26t98C2S858eRA7EJLcJ+IU1HnxHv7N
KUcZLUyMP6DPDcm3r2FtjvwDwneQwk3zBqFE30DgAERZTgZ7RTglVuyBhETryk3m
ZIf6bjuwnqn2/TFC7Yqwf9tzHshx91ywkfpOXwc/3IcrKKpfY74Mp4Cj/LkRemoz
LnE6DdpjuNpMmwTwV0Vb5FKkfeO2nFkLZ96z8HphNmAAINh7gP3bpUMaODc8fLDG
HtpzloYmBa4Yvn9JX+AwFuQStupPmScwa62Gcyv0Zm5EQ6fB1pqY1HdhpLHvXehT
l0i+8OO74B0=
=NI4M
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

Jeff Garzik wrote:

> We have to maintain said ugly code for decades.

No you don't, I do.

>> filesystem, but if so, it will have to do it much more slowly.  Take the
>> good ideas -- things like plugins -- and make them at least look like
>> incremental updates to the current VFS, and make them available to all
>> filesystems.
>
> So, Reiser4 may eventually take over VFS and be the only Linux
>
> This is how all Linux development is done.
>
> The code evolves over time.
>
> You have just described the path ReiserFS needs to take, in order to
> get this stuff into the kernel, in fact.

You missed his point.  He is saying ext3 code should migrate towards
becoming reiser4 plugins, and reiser4 should be merged now so that the
migration can get started.

I don't really care whether ext3 uses our plugin model.  (If it does,
fine, please credit me for it though, and consider maybe a thank you.  I
am happy to thank the XFS team for the idea of delayed allocation....)

I am entitled to get some advantage from being the first on the block. 
Other fs maintainers don't want me to have that entitlement, so they add
delays to slow us down.  It is odd how Hellwig no longer describes
himself as XFS associated.  Did SGI and him

> There is no entitlement to be merged in the upstream kernel.  

True, but what is interesting is that you don't think the faster
filesystem should be merged, and the others should catch up to it later
if they can, instead you think it should be delayed until all of its
benefits can be transmitted to the other filesystems first so that
everyone can be equal. 

Re: reiser4 plugins

[Pekka Enberg]

Hi Hans,

Jeff Garzik wrote:
> > We have to maintain said ugly code for decades.

On 6/23/05, Hans Reiser <reiser@namesys.com> wrote:
> No you don't, I do.

Lots of people work on the kernel. If you wish to keep reiser4
maintenance to yourself, you probably need to keep it as a separate
patch. Please consider submitting the non-controversial bits for
merging first and then continue with the rest. It does not make a lot
of sense reviewing the current patchkit as many parts of it won't be
merged as is (e.g. the plugin stuff).

                          Pekka

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
> Jeff Garzik wrote:
> 
> 
>>We have to maintain said ugly code for decades.
> 
> 
> No you don't, I do.
> 
> 
>>>filesystem, but if so, it will have to do it much more slowly.  Take the
>>>good ideas -- things like plugins -- and make them at least look like
>>>incremental updates to the current VFS, and make them available to all
>>>filesystems.
>>
>>So, Reiser4 may eventually take over VFS and be the only Linux
>>
>>This is how all Linux development is done.
>>
>>The code evolves over time.
>>
>>You have just described the path ReiserFS needs to take, in order to
>>get this stuff into the kernel, in fact.
> 
> 
> You missed his point.  He is saying ext3 code should migrate towards
> becoming reiser4 plugins, and reiser4 should be merged now so that the
> migration can get started.

Sort of.

I think ext3 would be nice as a reiser4 plugin.  Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself
(even more than it already has), the ext3 people may find themselves
wanting some of the more generic optimizations.

But, I don't think that will realistically happen at all.

Instead, what will probably happen is that once Reiser4 is in the
mainstream kernel, it will become more popular and noticable.  Other
FSes will take note.  ext3 people may decide they want
file-as-directory, and vfat people may decide they want cryptocompress,
and so on.  In order to do this, they can work with Namesys to port
whatever feature they need at the time to the standard VFS.

Eventually, with all the features ported, we end up with a situation
where there may be no meaningful difference between a filesystem and a
low-level reiser4 plugin.

At that point, we both win.  ext3 will effectively be reiser4 plugins,
and reiser4 will replace VFS, but incrementally and sanely, without
breaking old filesystems or depriving them of features.  This is indeed
the way to evolve the code slowly over time.

This way also has the advantage of avoiding yet another, even more
vicious flamefest if all the reiser4 functionality had to be ported to
VFS all at once.  Can any of you honestly say that you would be more
willing to accept a brand-new, relatively unproven filesystem with tons
of changes to the VFS than without?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrrDBHgHNmZLgCUhAQJN/A/6A4SqpsunKmj6NOKxqyMxkLVdMjxSuusa
V9XZIL6K0dDsF1FmvxsSVnl3so8u93sRCkxrBs3WQs9JqsA3f8O5VUvsdfqas/Fb
ZO+ht5n0r1BVfm1WJHHmR9xGm+jL2TMwVlVrcrrnBQLthq+9efggn5r9BXkOUPgz
57VMh6sjB9ioigmc2Wf6BsMlbBJpCz7EB91pcIrNzHdFOEWyuHm9mAP6ui2sppfK
bGaDlZcU4BzNUM8beakMkFuy7JCTFQj/O7s3G42Tqpi/4Ol1Vk7F2tgPkpxB0lHm
b/hCmCPbTHheRsDhT7gSHOzWrwA5pwKlT5nO4EWGq7DxZo2dgoYsa/2fP32uYPIX
r3vUfygk7MR59DKMi4LLcDy2OzvrtbwTkzgOVrw82NCi5J5/pu/gIHH1aHzkXM6T
YRd6G39+wmhN0n/MP3WAT7o3rbH4nuPiG8ZyZQyVTII9YRaOGwNLSjJKe1p9JFHY
Y+6bqK3h7LQrX4MfTSIu8k3m0iIt806uHP3GlV+HHKx/N/4HSxt6ikikI3jURO4z
XBr23/Uo3YNK+IuxQtPw2mKcdqqKhFq5OBmLkVFIzJyuh8KR7cfnFdl7luCTjFGL
KjtgZ39SwtjOUWKWJslkcrNwESqplMTwAaB6GPHiJRKbilaCM9GXm/teI0ZRCl+o
gBD0AdEBA9s=
=NJM0
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:
> Hans Reiser wrote:
> > Jeff Garzik wrote:

[...]

> > You missed his point.  He is saying ext3 code should migrate towards
> > becoming reiser4 plugins, and reiser4 should be merged now so that the
> > migration can get started.

> Sort of.
> 
> I think ext3 would be nice as a reiser4 plugin.

What for? It works just fine as it stands, AFAICS.

>                                                 Not everyone will want
> to reformat at once, but as the reiser4 code matures and proves itself
> (even more than it already has),

I for one have seen mainly people with wild claims that it will make their
machines much faster, and coming back later asking how they can recover
their thrashed partitions...

>                                  the ext3 people may find themselves
> wanting some of the more generic optimizations.

They'll filch them in due time, don't worry.

> But, I don't think that will realistically happen at all.
> 
> Instead, what will probably happen is that once Reiser4 is in the
> mainstream kernel, it will become more popular and noticable.  Other
> FSes will take note.  ext3 people may decide they want
> file-as-directory,

That idea is even much older than Linux itself, and no other (Unix)
filesystem has implemented it. Ever. Wonder why...

>                    and vfat people may decide they want cryptocompress,

I'm sure they don't, because it is mostly for Windows and assorted devices
(pendrives, digital cameras, ...) compatibility.

> and so on.  In order to do this, they can work with Namesys to port
> whatever feature they need at the time to the standard VFS.

I'm sure they are grateful for the offer.

> Eventually, with all the features ported, we end up with a situation
> where there may be no meaningful difference between a filesystem and a
> low-level reiser4 plugin.

Could very well take decades, if ever.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Adrian Ulrich]

> >                                                 Not everyone will want
> > to reformat at once, but as the reiser4 code matures and proves itself
> > (even more than it already has),
> 
> I for one have seen mainly people with wild claims that it will make their
> machines much faster, and coming back later asking how they can recover
> their thrashed partitions...

Then please show us some Links/Message-IDs to such postings.
I'd like to read them.

>From my POV:
 I've been using Reiser4 for almost everything (Rootfs / External
 Harddrives) for about ~8 Months without any data loss..

 Powerloss, unpluging the Disk while writing, full filesystem,
 heavy use : No problems with reiser4.. It *is* stable.



 -- Adrian

Re: reiser4 plugins

[Michael Dreher]

>>>                                                 Not everyone will want
>>> to reformat at once, but as the reiser4 code matures and proves itself
>>> (even more than it already has),
>>
>> I for one have seen mainly people with wild claims that it will make
>> their machines much faster, and coming back later asking how they can
>> recover their thrashed partitions...
>
> Then please show us some Links/Message-IDs to such postings.
> I'd like to read them.

Here you are....

The following happened to me with reiserfs as it was shipped with
suse 9.1:

------------------------------------------------------------------
dreher@euler03:~/mytex/konstanz/wohnung> ls
auto          makler2.aux  makler2.log  makler2.tex  makler.aux  makler.log  
swk.eps      unilogo.eps
briefkpf.tex  makler2.dvi  makler2.ps   makler3.tex  makler.dvi  makler.tex  
unikopf.tex
dreher@euler03:~/mytex/konstanz/wohnung> rm *.aux *.log
rm: cannot remove `makler2.log': No such file or directory
dreher@euler03:~/mytex/konstanz/wohnung> ls
auto  briefkpf.tex  makler2.dvi  makler2.ps  makler2.tex  makler3.tex  
makler.dvi  makler.tex  swk.eps  unikopf.tex  unilogo.eps
dreher@euler03:~/mytex/konstanz/wohnung> uname -a
Linux euler03 2.6.5-7.108-smp #1 SMP Wed Aug 25 13:34:40 UTC 2004 i686 i686 
i386 GNU/Linux
dreher@euler03:~/mytex/konstanz/wohnung> date
Tue Sep 21 13:15:45 CEST 2004
----------------------------------------------------------

Note the line "rm: cannot remove `makler2.log': No such file or directory"

There was no data loss, but such a bug should not happen.
I never had similar experiences with ext3.

Unfortunately, I cannot reproduce this behavior. 

>  Powerloss, unpluging the Disk while writing, full filesystem,
>  heavy use : No problems with reiser4.. It *is* stable.

My impression: reiser3 is not 100% stable, but quite stable, 
written by someone who asks for "review by benchmark".

Michael

Re: reiser4 plugins

[M.]

I think we are talking about reiser4, not reiser3..

Michele

On 6/23/05, Michael Dreher <michael.dreher@uni-konstanz.de> wrote:
> >>>                                                 Not everyone will want
> >>> to reformat at once, but as the reiser4 code matures and proves itself
> >>> (even more than it already has),
> >>
> >> I for one have seen mainly people with wild claims that it will make
> >> their machines much faster, and coming back later asking how they can
> >> recover their thrashed partitions...
> >
> > Then please show us some Links/Message-IDs to such postings.
> > I'd like to read them.
> 
> Here you are....
> 
> The following happened to me with reiserfs as it was shipped with
> suse 9.1:
> 
> ------------------------------------------------------------------
> dreher@euler03:~/mytex/konstanz/wohnung> ls
> auto          makler2.aux  makler2.log  makler2.tex  makler.aux  makler.log
> swk.eps      unilogo.eps
> briefkpf.tex  makler2.dvi  makler2.ps   makler3.tex  makler.dvi  makler.tex
> unikopf.tex
> dreher@euler03:~/mytex/konstanz/wohnung> rm *.aux *.log
> rm: cannot remove `makler2.log': No such file or directory
> dreher@euler03:~/mytex/konstanz/wohnung> ls
> auto  briefkpf.tex  makler2.dvi  makler2.ps  makler2.tex  makler3.tex
> makler.dvi  makler.tex  swk.eps  unikopf.tex  unilogo.eps
> dreher@euler03:~/mytex/konstanz/wohnung> uname -a
> Linux euler03 2.6.5-7.108-smp #1 SMP Wed Aug 25 13:34:40 UTC 2004 i686 i686
> i386 GNU/Linux
> dreher@euler03:~/mytex/konstanz/wohnung> date
> Tue Sep 21 13:15:45 CEST 2004
> ----------------------------------------------------------
> 
> Note the line "rm: cannot remove `makler2.log': No such file or directory"
> 
> There was no data loss, but such a bug should not happen.
> I never had similar experiences with ext3.
> 
> Unfortunately, I cannot reproduce this behavior.
> 
> >  Powerloss, unpluging the Disk while writing, full filesystem,
> >  heavy use : No problems with reiser4.. It *is* stable.
> 
> My impression: reiser3 is not 100% stable, but quite stable,
> written by someone who asks for "review by benchmark".
> 
> Michael
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: reiser4 plugins

[Alan Cox]

On Iau, 2005-06-23 at 21:49, Michael Dreher wrote:
> My impression: reiser3 is not 100% stable, but quite stable, 
> written by someone who asks for "review by benchmark".

Review by uniprocessor benchmark perhaps. However Reiser4 is new code.
The original extfs on Linux was not very good either while ext2 was
excellent. It seems inappropriate to technically review one fs based on
the other. Looking at the authors maintenance record I think is
important but every single star Linux kernel contributors first major
contribution was generally not very good.

Alan

Re: reiser4 plugins

[Hans Reiser]

One, you were using V3 not V4.

Two, this bug you mention is probably not an fs bug.  rm first creates a
list of names, and then removes them. 

reiser@linux:~/scratch> touch fufu
reiser@linux:~/scratch> touch fifu
reiser@linux:~/scratch> rm *fu fi*
rm: cannot remove `fifu': No such file or directory

Your file either somehow got removed before rm got to it, or rm somehow
got to it twice.  Vitaly, can you look at the error handling by rm and
see if it can get to things twice when it hits an error or if you can
otherwise figure this out?  If I remember right, I have hit this myself
for non-reiserfs filesystems, and I never investigated it.

Michael Dreher wrote:

>>>>                                                Not everyone will want
>>>>to reformat at once, but as the reiser4 code matures and proves itself
>>>>(even more than it already has),
>>>>        
>>>>
>>>I for one have seen mainly people with wild claims that it will make
>>>their machines much faster, and coming back later asking how they can
>>>recover their thrashed partitions...
>>>      
>>>
>>Then please show us some Links/Message-IDs to such postings.
>>I'd like to read them.
>>    
>>
>
>Here you are....
>
>The following happened to me with reiserfs as it was shipped with
>suse 9.1:
>
>------------------------------------------------------------------
>dreher@euler03:~/mytex/konstanz/wohnung> ls
>auto          makler2.aux  makler2.log  makler2.tex  makler.aux  makler.log  
>swk.eps      unilogo.eps
>briefkpf.tex  makler2.dvi  makler2.ps   makler3.tex  makler.dvi  makler.tex  
>unikopf.tex
>dreher@euler03:~/mytex/konstanz/wohnung> rm *.aux *.log
>rm: cannot remove `makler2.log': No such file or directory
>dreher@euler03:~/mytex/konstanz/wohnung> ls
>auto  briefkpf.tex  makler2.dvi  makler2.ps  makler2.tex  makler3.tex  
>makler.dvi  makler.tex  swk.eps  unikopf.tex  unilogo.eps
>dreher@euler03:~/mytex/konstanz/wohnung> uname -a
>Linux euler03 2.6.5-7.108-smp #1 SMP Wed Aug 25 13:34:40 UTC 2004 i686 i686 
>i386 GNU/Linux
>dreher@euler03:~/mytex/konstanz/wohnung> date
>Tue Sep 21 13:15:45 CEST 2004
>----------------------------------------------------------
>
>Note the line "rm: cannot remove `makler2.log': No such file or directory"
>
>There was no data loss, but such a bug should not happen.
>I never had similar experiences with ext3.
>
>Unfortunately, I cannot reproduce this behavior. 
>
>  
>
>> Powerloss, unpluging the Disk while writing, full filesystem,
>> heavy use : No problems with reiser4.. It *is* stable.
>>    
>>
>
>My impression: reiser3 is not 100% stable, but quite stable, 
>written by someone who asks for "review by benchmark".
>
>Michael
>
>
>  
>

Re: reiser4 plugins

[Bernd Eckenfels]

In article <42BB4C81.6070500@namesys.com> you wrote:
> reiser@linux:~/scratch> touch fufu
> reiser@linux:~/scratch> touch fifu
> reiser@linux:~/scratch> rm *fu fi*
> rm: cannot remove `fifu': No such file or directory

# echo rm *fu fi*
rm fifu fufu fifu

shell magic :)

Bernd

Re: reiser4 plugins

[zhilla]

Bernd Eckenfels wrote:

 >> reiser@linux:~/scratch> touch fufu
 >> reiser@linux:~/scratch> touch fifu
 >> reiser@linux:~/scratch> rm *fu fi*
 >> rm: cannot remove `fifu': No such file or directory
 >
 > # echo rm *fu fi*
 > rm fifu fufu fifu
 > shell magic :)


may i add that you guys are a bunch of stoners :)

Re: reiser4 plugins

[Vitaly Fertman]

On Friday 24 June 2005 03:57, Hans Reiser wrote:
> One, you were using V3 not V4.
> 
> Two, this bug you mention is probably not an fs bug.  rm first creates a
> list of names, and then removes them. 

shell creates a list of names, rm simply handles the given set 
of names and tries to remove a file twice if specified twice. 

the problem could be in the shell, or what is more likely in patterns 
that overlap each other, like in the example of 'rm *fu fi*'. 
we should probably investigate a reproducable example.

> reiser@linux:~/scratch> touch fufu
> reiser@linux:~/scratch> touch fifu
> reiser@linux:~/scratch> rm *fu fi*
> rm: cannot remove `fifu': No such file or directory
> 
> Your file either somehow got removed before rm got to it, or rm somehow
> got to it twice.  Vitaly, can you look at the error handling by rm and
> see if it can get to things twice when it hits an error or if you can

once per every given name.

> otherwise figure this out?  If I remember right, I have hit this myself
> for non-reiserfs filesystems, and I never investigated it.
> 
> Michael Dreher wrote:
> 
> >>>>                                                Not everyone will want
> >>>>to reformat at once, but as the reiser4 code matures and proves itself
> >>>>(even more than it already has),
> >>>>        
> >>>>
> >>>I for one have seen mainly people with wild claims that it will make
> >>>their machines much faster, and coming back later asking how they can
> >>>recover their thrashed partitions...
> >>>      
> >>>
> >>Then please show us some Links/Message-IDs to such postings.
> >>I'd like to read them.
> >>    
> >>
> >
> >Here you are....
> >
> >The following happened to me with reiserfs as it was shipped with
> >suse 9.1:
> >
> >------------------------------------------------------------------
> >dreher@euler03:~/mytex/konstanz/wohnung> ls
> >auto          makler2.aux  makler2.log  makler2.tex  makler.aux  makler.log  
> >swk.eps      unilogo.eps
> >briefkpf.tex  makler2.dvi  makler2.ps   makler3.tex  makler.dvi  makler.tex  
> >unikopf.tex
> >dreher@euler03:~/mytex/konstanz/wohnung> rm *.aux *.log
> >rm: cannot remove `makler2.log': No such file or directory
> >dreher@euler03:~/mytex/konstanz/wohnung> ls
> >auto  briefkpf.tex  makler2.dvi  makler2.ps  makler2.tex  makler3.tex  
> >makler.dvi  makler.tex  swk.eps  unikopf.tex  unilogo.eps
> >dreher@euler03:~/mytex/konstanz/wohnung> uname -a
> >Linux euler03 2.6.5-7.108-smp #1 SMP Wed Aug 25 13:34:40 UTC 2004 i686 i686 
> >i386 GNU/Linux
> >dreher@euler03:~/mytex/konstanz/wohnung> date
> >Tue Sep 21 13:15:45 CEST 2004
> >----------------------------------------------------------
> >
> >Note the line "rm: cannot remove `makler2.log': No such file or directory"
> >
> >There was no data loss, but such a bug should not happen.
> >I never had similar experiences with ext3.
> >
> >Unfortunately, I cannot reproduce this behavior. 
> >
> >  
> >
> >> Powerloss, unpluging the Disk while writing, full filesystem,
> >> heavy use : No problems with reiser4.. It *is* stable.
> >>    
> >>
> >
> >My impression: reiser3 is not 100% stable, but quite stable, 
> >written by someone who asks for "review by benchmark".
> >
> >Michael
> >
> >
> >  
> >
> 
> 
> 

-- 
Thanks,
Vitaly Fertman

Re: reiser4 plugins

[Avuton Olrich]

On 6/23/05, Adrian Ulrich <reiser4@blinkenlights.ch> wrote:
> From my POV:
>  I've been using Reiser4 for almost everything (Rootfs / External
>  Harddrives) for about ~8 Months without any data loss..
> 
>  Powerloss, unpluging the Disk while writing, full filesystem,
>  heavy use : No problems with reiser4.. It *is* stable.

*From users who use it* I have heard nothing but love for reiser4.
It's amazing how quickly people seem to be dismissive about what
reiser4 has to offer when they more than likely haven't taken it for a
spin at all. All I hear about is 'we can't let something ugly go into
the stable kernel' then in the same day I looked into some of the
config options...

CONFIG_WDC_ALI15X3:
*snip*
This allows for UltraDMA support for WDC drives that ignore CRC
checking. You are a fool for enabling this option, but there have been
requests.
*snip*

How many have requested that reiser4 make it into the kernel? I'd
imagine many more then requested this IDE feature. And it's an
*option*. Please work something out on this.

avuton

-- 
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: reiser4 plugins

[Dan Oglesby]

Avuton Olrich wrote:
> On 6/23/05, Adrian Ulrich <reiser4@blinkenlights.ch> wrote:
> 
>>From my POV:
>> I've been using Reiser4 for almost everything (Rootfs / External
>> Harddrives) for about ~8 Months without any data loss..
>>
>> Powerloss, unpluging the Disk while writing, full filesystem,
>> heavy use : No problems with reiser4.. It *is* stable.
> 
> 
> *From users who use it* I have heard nothing but love for reiser4.
> It's amazing how quickly people seem to be dismissive about what
> reiser4 has to offer when they more than likely haven't taken it for a
> spin at all. All I hear about is 'we can't let something ugly go into
> the stable kernel' then in the same day I looked into some of the
> config options...
> 
> CONFIG_WDC_ALI15X3:
> *snip*
> This allows for UltraDMA support for WDC drives that ignore CRC
> checking. You are a fool for enabling this option, but there have been
> requests.
> *snip*
> 
> How many have requested that reiser4 make it into the kernel? I'd
> imagine many more then requested this IDE feature. And it's an
> *option*. Please work something out on this.
> 
> avuton
> 

As a user who uses ReiserFS v3, I have nothing but love for ReiserFS.

The speed and reliability are great right now, I'm really looking 
forward to using ReiserFS v4.

--Dan

Re: reiser4 plugins

[Jim Crilly]

On 06/23/05 02:29:21PM -0700, Avuton Olrich wrote:
> On 6/23/05, Adrian Ulrich <reiser4@blinkenlights.ch> wrote:
> > From my POV:
> >  I've been using Reiser4 for almost everything (Rootfs / External
> >  Harddrives) for about ~8 Months without any data loss..
> > 
> >  Powerloss, unpluging the Disk while writing, full filesystem,
> >  heavy use : No problems with reiser4.. It *is* stable.
> 
> *From users who use it* I have heard nothing but love for reiser4.
> It's amazing how quickly people seem to be dismissive about what
> reiser4 has to offer when they more than likely haven't taken it for a
> spin at all. All I hear about is 'we can't let something ugly go into
> the stable kernel' then in the same day I looked into some of the
> config options...
> 
> CONFIG_WDC_ALI15X3:
> *snip*
> This allows for UltraDMA support for WDC drives that ignore CRC
> checking. You are a fool for enabling this option, but there have been
> requests.
> *snip*

That's hardly a good example, that config option, while obviously
questionable, only added 2 #ifdef blocks and affects 1 function that's 20
lines long.

> 
> How many have requested that reiser4 make it into the kernel? I'd
> imagine many more then requested this IDE feature. And it's an
> *option*. Please work something out on this.
> 
> avuton

Jim.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
> David Masover <ninja@slaphack.com> wrote:
> 
>>Hans Reiser wrote:
>>
>>>Jeff Garzik wrote:
> 
> 
> [...]
> 
> 
>>>You missed his point.  He is saying ext3 code should migrate towards
>>>becoming reiser4 plugins, and reiser4 should be merged now so that the
>>>migration can get started.
> 
> 
>>Sort of.
>>
>>I think ext3 would be nice as a reiser4 plugin.
> 
> 
> What for? It works just fine as it stands, AFAICS.

So does DOS.  Do you use DOS?  I don't even use DOS to run DOS programs.

"Ain't broke" is the battle cry of stagnation.

But, there are some things Reiser does better and faster than ext3, even
if you don't count file-as-directory and other toys.  There's nothing
ext3 does better than Reiser, unless you count the compatibility with
random bootloaders and low-level tools.

>>                                                Not everyone will want
>>to reformat at once, but as the reiser4 code matures and proves itself
>>(even more than it already has),
> 
> 
> I for one have seen mainly people with wild claims that it will make their
> machines much faster, and coming back later asking how they can recover
> their thrashed partitions...

You know how many I've had thrashed on Reiser4?  Two.  The first one was
with a VERY early alpha/beta, and the second one was when I dropped a
laptop and the disk failed.

And it does make certain things faster.  For one thing, "emerge sync" on
Gentoo is twice to four times as fast, and /usr/portage is 75% as big,
as on ReiserFS (3).

>>                                 the ext3 people may find themselves
>>wanting some of the more generic optimizations.
> 
> 
> They'll filch them in due time, don't worry.

Duplication of effort.  With plugins, we can optimize the upper layers
of ALL filesystems, regardless of the lower layers, in such a way that
it is optional.  I'm sure it's far easier to write a Reiser storage
plugin than a brand new FS.

Eventually, once competition is only based on storage format, we could
end up with just one format.  Just one filesystem!  (except for
fat/ntfs/iso/udf/network...)  And in the open source world, sometimes a
single product is a good thing.

>>But, I don't think that will realistically happen at all.
>>
>>Instead, what will probably happen is that once Reiser4 is in the
>>mainstream kernel, it will become more popular and noticable.  Other
>>FSes will take note.  ext3 people may decide they want
>>file-as-directory,
> 
> 
> That idea is even much older than Linux itself, and no other (Unix)
> filesystem has implemented it. Ever. Wonder why...
> 
> 
>>                   and vfat people may decide they want cryptocompress,
> 
> 
> I'm sure they don't, because it is mostly for Windows and assorted devices
> (pendrives, digital cameras, ...) compatibility.

I, for one, would like to use a pendrive and have certain files be
encrypted transparently, only for use on Linux, but others be ready to
transfer to a Windows box.

>>Eventually, with all the features ported, we end up with a situation
>>where there may be no meaningful difference between a filesystem and a
>>low-level reiser4 plugin.
> 
> 
> Could very well take decades, if ever.

How would you do it, in a way that doesn't take decades?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrsx6HgHNmZLgCUhAQKXYw//Z9bSh74AWr2o1n+EM0nCHUNV78xgeuey
ZHtzUT8rCv2KQ+2fMe5EnRUaRYTKvFnnGccH4vu/OAArqKqt/RgP3NP8UZALsZbB
MMoEHZSX/E0BFlKKiBjPgvwfnnY9ZYF0GPkU5L97dj1K0dEQMndOZoYDV07H4TnN
/1VkytnsXuhm5nqRhPd89rDbvQtXpzHiDjVNPfT+J6M6JFw8jfYQZ0Fo1T9dVKMg
qibVneJj+onHVBmBBqGTZ0Ane5VmG0h0a2ZwsslQPkf03DAgprliykr40NCECdli
OdaS73qYPlYRRl1nuw84g2KOLXbTnSGmW1t4qt/tyI6t3TOEn9FqY7YzwvbWIVLP
GMkJG1htAQefGNcEXx+15xAHJi6/0DiSoJu+P+ie+yG8pkG943936AxEgs89pTpC
z8i/GV9Uq+S6BgA+RJuxpk6U8rQ0YHMhAiK83p+s7vdUsGIKomUmSbn7a6DC7cZt
aGkqxYyaoV2MlHUMUebSv5F82JUgx6rPuc8SQW1wvVIVdNA7QhlYYsPa5vFFj34C
scxN9vNGyxWGu60yKx7LSYRkB9/prJytKvpezVKRkn8pnKnl4AHodKioSPxt1iHC
BvNsLL8Kn8FBe/HG98d1v8vwTqe0Y8KgBMRe/J73u/OzU82lh2V3YbXNaW+DLVGz
MOaHYEzfoNg=
=yFsv
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Alan Cox]

On Iau, 2005-06-23 at 23:04, David Masover wrote:
> > What for? It works just fine as it stands, AFAICS.
> 
> So does DOS.  Do you use DOS?  I don't even use DOS to run DOS programs.

False argument. So does the pen, so do hinges on doors. Do you still
have hinges on your doors - probably.

> "Ain't broke" is the battle cry of stagnation.

Its also the battle cry of everyone over the age of 20 who also has a
real job to do 8)

> But, there are some things Reiser does better and faster than ext3, even
> if you don't count file-as-directory and other toys.  There's nothing
> ext3 does better than Reiser, unless you count the compatibility with
> random bootloaders and low-level tools.

Certainly compared with reiser3 you've missed a few out including
resilience to disk errors (nearly nil on reiser3), and SMP scaling.

> You know how many I've had thrashed on Reiser4?  Two.  The first one was
> with a VERY early alpha/beta, and the second one was when I dropped a
> laptop and the disk failed.

Entirely or bad blocks ? The latter should have a minimal cost on a well
designed fs.

> Duplication of effort.  With plugins, we can optimize the upper layers
> of ALL filesystems, regardless of the lower layers, in such a way that

In which case the features belong in the VFS as all those with
experience and kernel contributions have been arguing.

Re: reiser4 plugins

[Hans Reiser]

Alan Cox wrote:

> SMP scaling.

Reiser4 should do much better at this, as it was designed for it.  I
wish we had a nice hunking multiprocessor to verify that and work
through the inevitable unintended sources of bottlenecks though.

>  
>
>>You know how many I've had thrashed on Reiser4?  Two.  The first one was
>>with a VERY early alpha/beta, and the second one was when I dropped a
>>laptop and the disk failed.
>>    
>>
>
>Entirely or bad blocks ? The latter should have a minimal cost on a well
>designed fs.
>
>  
>
>>Duplication of effort.  With plugins, we can optimize the upper layers
>>of ALL filesystems, regardless of the lower layers, in such a way that
>>    
>>
>
>In which case the features belong in the VFS as all those with
>experience and kernel contributions have been arguing.
>  
>
So you fundamentally reject the prototype it in one fs and then abstract
it to others development model?

Re: reiser4 plugins

[Jeff Garzik]

Hans Reiser wrote:
> Alan Cox wrote:
>>>Duplication of effort.  With plugins, we can optimize the upper layers
>>>of ALL filesystems, regardless of the lower layers, in such a way that

>>In which case the features belong in the VFS as all those with
>>experience and kernel contributions have been arguing.

> So you fundamentally reject the prototype it in one fs and then abstract
> it to others development model?


For similar reasons, we don't let hardware vendors implement software 
RAID inside the hardware driver.  It's not appropriate at that level.

	Jeff

Re: reiser4 plugins

[Lincoln Dale]

Hans Reiser wrote:

>So you fundamentally reject the prototype it in one fs and then abstract
>it to others development model?
>  
>

Hans,
after many years here now, one would have thought you would have "got" 
this part of Linux: kernel development & code that gets into the kernel 
only does so by getting past the benevolent dictators.
instead, it seems that every time there is ReiserFS to be merged (and we 
can go back in history a number of years here..), it always seems to 
come as a great shock that your code won't be merged 'as-is' without 
peer review & comment.

don't feel that you're being singled out here.  you aren't.  there isn't 
any anti-Hans-and-his-filesystem conspiracy here.
there are plenty of examples on where this has happened in Linux 
previously in other parts of the tree.
EVMS is a great example of similar things - a proposal to include kernel 
code to do various volume-mgmt functions - which was basically 
accomplishing the same goal as that of LVM/LVM2 and MD drivers (& DM 
framework).
the EVMS team are a great act to follow -  see 
http://lwn.net/Articles/14714/ - they showed high levels of professional 
conduct and made what was essentially a 'hard' but 'correct' decision in 
reworking EVMS to use the same DM infrastructure as LVM2.
there are countless other examples at various times - various 
'competing' IPv6 projects, IPSec, various "hardware" (software) RAID 
controllers, various IP offload schemes et al.

why does Reiserfs have to be any different?

you know that VFS is the mechanism in Linux.  you know (i hope..) how it 
works.  it isn't so hard to see how many of the Reiser4 "plug-ins" could 
be tied into VFS calls.
OR, if they cannot TODAY, propose how VFS _COULD_ be made to do this.

the key here is trust.  and trust is a two-way street.

the irony of this whole thread is that history is repeating itself.  see 
http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0519.html
kernel developers pushed back on you 3 years ago - in 2001 - what has 
really changed?

*an observation*

cheers,

lincoln.

Re: reiser4 plugins

[Jeff Garzik]

Lincoln Dale wrote:
> the EVMS team are a great act to follow -  see 
> http://lwn.net/Articles/14714/ - they showed high levels of professional 
> conduct and made what was essentially a 'hard' but 'correct' decision in 
> reworking EVMS to use the same DM infrastructure as LVM2.

I just want to heartily second this.  The EVMS team made the hard but 
ultimately best choice, and everybody benefited.  They deserve all the 
kudos they get.

	Jeff

Re: reiser4 plugins

[Hans Reiser]

Lincoln Dale wrote:

>
> the irony of this whole thread is that history is repeating itself. 
> see http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0519.html
> kernel developers pushed back on you 3 years ago - in 2001 - what has
> really changed?

It is exactly the same, but rather than dwell on that, I'll just remind
that I have sent out two technical emails which talk only about the
issue of plugins and pluginids, and whether plugins are classes rather
than just instances, and whether the classes really would benefit from
being instantiated into VFS at the cost of keeping the same info in two
places, and I got no answer on them.  Zam pointed out that our plugins
do more than just VFS operations, and got no response on that or his
other points.

Regarding trust, Christophe comes out the gate using the words "useless
abstraction layer" that happens to be a core feature of our design,
demanding we cut it out, and not really understanding it or recognizing
any functionality it provides, and you can't really expect me to respect
this, can you?

Now, if his target is reduced to whether we can eliminate a function
indirection, and whether we can review the code together and see if it
is easy to extend plugins and pluginids to other filesystems by finding
places to make it more generic and accepting of per filesystem plugins,
especially if it is not tied to going into 2.6.13, well, that is the
conversation I would have liked to have had.

Re: reiser4 plugins

[Lincoln Dale]

Hans Reiser wrote:

>Lincoln Dale wrote:
>
>  
>
>>the irony of this whole thread is that history is repeating itself. 
>>see http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0519.html
>>kernel developers pushed back on you 3 years ago - in 2001 - what has
>>really changed?
>>    
>>
>
>It is exactly the same, but rather than dwell on that, I'll just remind
>that I have sent out two technical emails which talk only about the
>issue of plugins and pluginids, and whether plugins are classes rather
>than just instances, and whether the classes really would benefit from
>being instantiated into VFS at the cost of keeping the same info in two
>places, and I got no answer on them.  Zam pointed out that our plugins
>do more than just VFS operations, and got no response on that or his
>other points.
>
>Regarding trust, Christophe comes out the gate using the words "useless
>abstraction layer" that happens to be a core feature of our design,
>demanding we cut it out, and not really understanding it or recognizing
>any functionality it provides, and you can't really expect me to respect
>this, can you?
>  
>
heh.  one example i didn't mention in the myriad of EVMS, IPv6, IPSec et 
al was iSCSI.
Christoph was (rightly so) very strong in pushing back on things that 
needed to be fixed/changed/addressed in linux-iscsi, a project near & 
dear to much of the paid work i do.

while i'm sure Christoph could probably be more tactful at times, his 
views on technical matters in the kernel are respected.
the key here is to not take it personally - but instead just understand 
that this is the framework that you have to work in to get it into the 
mainline kernel.

if you don't want to go down that path, you're free to do so.  its open 
source, you can keep your own linux-kernel fork.
but if you want to get your code into mainline, i don't think its so 
much a case of l-k folks telling you how to make your code work under 
VFS.  the onus is on you to say WHY your code and plugins could never be 
put into VFS.

if anything, you should be thankful that Al hasn't yet commented. :)
 

>Now, if his target is reduced to whether we can eliminate a function
>indirection, and whether we can review the code together and see if it
>is easy to extend plugins and pluginids to other filesystems by finding
>places to make it more generic and accepting of per filesystem plugins,
>especially if it is not tied to going into 2.6.13, well, that is the
>conversation I would have liked to have had.
>
>  
>
fantastic - some common ground.
any reason WHY there has to be an abstraction of 'pluginid' when in 
theory VFS operations can already provide the necessary abstraction on a 
per-object basis?

Nikita basically said as much in Message-ID: 
<17081.30107.751071.983835@gargle.gargle.HOWL> earlier in this thread:
    "But it is not so. There _are_ plugins-in-the-VFS. VFS operates on 
opaque
     objects (inodes, dentries, file system types) through interfaces:
     {inode,address_space,dentry,sb,etc.}_operations. Every file system
     back-end if free to implement whatever number of these interfaces. And
     the do this already: check the sources; even ext2 does this: e.g.,
     ext2_fast_symlink_inode_operations and ext2_symlink_inode_operations.

     This is exactly what upper level reiser4 plugins are for.

     I guess that one of Christoph Hellwig's complaints is that reiser4
     introduces another layer of abstraction to implement something that
     already exists."

i never saw any reason why there can't be specific VFS operations put 
together for specific inodes if the policy for that inode dictates any 
special operation (e.g. compression).



cheers,

lincoln.

Re: reiser4 plugins

[Hans Reiser]

Lincoln Dale wrote:

>
>> Now, if his target is reduced to whether we can eliminate a function
>> indirection, and whether we can review the code together and see if it
>> is easy to extend plugins and pluginids to other filesystems by finding
>> places to make it more generic and accepting of per filesystem plugins,
>> especially if it is not tied to going into 2.6.13, well, that is the
>> conversation I would have liked to have had.
>>
>>  
>>
> fantastic - some common ground.
> any reason WHY there has to be an abstraction of 'pluginid' when in
> theory VFS operations can already provide the necessary abstraction on
> a per-object basis?

VFS supplies instances, plugins are classes.  If a language can
instantiate an object, that does not eliminate the value of being able
to create classes.

Does it make sense to you now?

Re: reiser4 plugins

[Lincoln Dale]

Hans Reiser wrote:

>>>ow, if his target is reduced to whether we can eliminate a function
>>>indirection, and whether we can review the code together and see if it
>>>is easy to extend plugins and pluginids to other filesystems by finding
>>>places to make it more generic and accepting of per filesystem plugins,
>>>especially if it is not tied to going into 2.6.13, well, that is the
>>>conversation I would have liked to have had.
>>>
>>fantastic - some common ground.
>>any reason WHY there has to be an abstraction of 'pluginid' when in
>>theory VFS operations can already provide the necessary abstraction on
>>a per-object basis?
>>    
>>
>VFS supplies instances, plugins are classes.  If a language can
>instantiate an object, that does not eliminate the value of being able
>to create classes.
>
>Does it make sense to you now?
>  
>
you've lost me . . .

regardless, it isn't /me/ that you need to convince.  how about a 
posting to l-k on "why Reiser4 cannot use VFS infrastructure for 
[crypto,compression,blahblah] plugins" - ideally, for each plugin.


cheers,

lincoln.

Re: reiser4 plugins

[Horst von Brand]

Hans Reiser <reiser@namesys.com> wrote:
> Lincoln Dale wrote:
> >> Now, if his target is reduced to whether we can eliminate a function
> >> indirection, and whether we can review the code together and see if it
> >> is easy to extend plugins and pluginids to other filesystems by finding
> >> places to make it more generic and accepting of per filesystem plugins,
> >> especially if it is not tied to going into 2.6.13, well, that is the
> >> conversation I would have liked to have had.

> > fantastic - some common ground.
> > any reason WHY there has to be an abstraction of 'pluginid' when in
> > theory VFS operations can already provide the necessary abstraction on
> > a per-object basis?

> VFS supplies instances, plugins are classes.  If a language can
> instantiate an object, that does not eliminate the value of being able
> to create classes.

In OOP speak, VFS is an abstract class, each individual filesystem derives
from this class giving a concrete class, a specific on-disk (or whereever)
filesystem is an object of its (concrete) class. The rest of the kernel (as
a client) doesn't care for the concrete classes, it speaks only (or mostly)
in terms of the abstract class (VFS). And concrete filesystems in turn use
the generic block layer.

> Does it make sense to you now?

No. Sounds jumbled up and backwards. And I don't see how "languages" could
even enter the picture here.

Again: Is there any (sane) way the /existing/ VFS can be used to express
what you want? What are the /minimal/ changes to VFS so each extension can
be catered for in an uniform way across filesystems (anything else destroys
the core idea of having a VFS in the first place!)? What are the required
changes in the clients of VFS (i.e., changes to the core kernel)? What
would be the impact on existing filesystems that /don't/ use new
functionality (this is a lot harder, because it impacts many independent
pieces of code)? What would be required for an existing filesystem to
incorporate it?

When all this is answered, go over the implementation details. But that is
far off still.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Tyson Whitehead]

[-- Attachment #1: Type: text/plain, Size: 5253 bytes --]

On June 24, 2005 11:32, Horst von Brand wrote:
> Hans Reiser <reiser@namesys.com> wrote:
> >
> > VFS supplies instances, plugins are classes.  If a language can
> > instantiate an object, that does not eliminate the value of being able
> > to create classes.
>
> In OOP speak, VFS is an abstract class, each individual filesystem derives
> from this class giving a concrete class, a specific on-disk (or whereever)
> filesystem is an object of its (concrete) class. The rest of the kernel (as
> a client) doesn't care for the concrete classes, it speaks only (or mostly)
> in terms of the abstract class (VFS). And concrete filesystems in turn use
> the generic block layer.
>
> > Does it make sense to you now?
>
> No. Sounds jumbled up and backwards. And I don't see how "languages" could
> even enter the picture here.

There seems to be a bit of confusion by the OO speak (I was initially confused 
as well -- guess I've spend too much time in broken OO languages recently).  
*grin*  Thinking of interfaces, classes, and objects as a bunch of 1:M 
relationships make it much clearer.

If you start at the bottom, an object (instantiation of a class) has the 
per-object stuff (usually data).  A class (instantiation of a meta-class or 
interface) has the shared object stuff (methods/functions that work on the 
data and static/shared data).  An interface has the shared class stuff (info 
on what the methods/functions and static/shared data are).

A class is a realization of an interface.  An object is a realization of a 
class.  The standard C thing is to have only objects be dynamic at runtime.  
A slightly less common (but very flexible) thing to do is to make the classes 
dynamic at runtime.  This introduces the following C -> OO talk:

-Instances of structures full of data are objects.
-Instances of structure full of function pointers are classes.
-Instances of structure that are a big mix of both are the crossbred 
offsprings of programmers committing OO maintenance suicide by violating all 
the layering (or a savy programmers cleverly saving of one level of 
indirection by caching an object's class' function pointers in the object -- 
depends on your point of view *grin*).

In the Reiser4 FS I believe it goes like this (apologies Hans if I'm wrong).  
The objects are files, directories, etc.  The classes are the plugins.  
Associated with each object is a number (plugin id/class pointer) identifying 
the set of functions (class/plugin) that provides the functions/methods to 
manipulate that object (ids are required because the objects [files, 
directories, etc.] live on the device while the plugins live in memory).

The good thing is that it is very easy to provide new plugins (instances of 
the interface to manipulate the objects).  To create a new new plugin (a new 
class) you just fill in a bunch of function pointer fields in a structure 
(possibly mixing and matching existing functions) and give it a plugin id.  
To use it, you just link appropriate objects (files for file plugins, 
directories for directory plugins, etc.) back to it via the plugin id.

The VFS also has a bunch of OO stuff.  Looking at "include/linux/fs.h", the 
inode structure is mostly data with a few pointers to structures full of 
functions pointers.  Instances of it are most definitely objects.  Instances 
of the associated structures full of functions pointers (inode_operations, 
file_operations, etc.) are most definitely classes (they give an implement of 
their respective interfaces for that inode).  Likewise, instances of 
super_block are objects, while instances of super_operations (and friends) 
are classes.  In short, in Reiser4 FS talk, much of the VFS is done with 
plugins (a plugin being an instance of one of the *_operations structures).

The current merge issue seems to be that apparently (I haven't actually looked 
myself) lots of the VFS plugins that Reiser4 exports are just thunks.  This 
might imply that there are Reiser4 class/plugin interfaces that correspond 
very closely to the VFS class/plugin interfaces (the *_operations).  In this 
case it might make good sense to lift those plugins from Reiser4 to VFS 
plugins.

(If the VFS class/plugin interfaces are subsets of some Reiser4 class/plugin 
interfaces, you could also probably do the lifting.  The additional Reiser4 
class/plugin interface parts could be added to the split off and added to the 
VFS objects via the generic pointer field [i.e., use it to attach the 
additional data and class/plugin pointers to the objects].  As this adds some 
pointer pain for the additional parts you may not really be gaining anything.  
Another possibly approach might be to just extend each of the VFS object 
structures by making them the first entry in more flushed out Reiser4 
specific structures [the Reiser4 code better be doing the allocating in this 
case].  *grin*)

Later!  -T

-- 
 Tyson Whitehead  (-twhitehe@uwo.ca -- WSC-)
 Computer Engineer                          Dept. of Applied Mathematics,
 Graduate Student- Applied Mathematics      University of Western Ontario,
 GnuPG Key ID# 0x8A2AB5D8                   London, Ontario, Canada

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Timothy Webster]

--- Lincoln Dale <ltd@cisco.com> wrote:
snip

> 
> Nikita basically said as much in Message-ID: 
> <17081.30107.751071.983835@gargle.gargle.HOWL>
> earlier in this thread:
>     "But it is not so. There _are_
> plugins-in-the-VFS. VFS operates on 
> opaque
>      objects (inodes, dentries, file system types)
> through interfaces:
>     
> {inode,address_space,dentry,sb,etc.}_operations.
> Every file system
>      back-end if free to implement whatever number
> of these interfaces. And
>      the do this already: check the sources; even
> ext2 does this: e.g.,
>      ext2_fast_symlink_inode_operations and
> ext2_symlink_inode_operations.
> 
>      This is exactly what upper level reiser4
> plugins are for.
> 
>      I guess that one of Christoph Hellwig's
> complaints is that reiser4
>      introduces another layer of abstraction to
> implement something that
>      already exists."


What reiserfs4 brings is file based plugins. Which is
extremely useful and powerful. I don't want to see
this go away.

I think it is the task of the linux community to
generalize the vfs layer and not lock out reiserfs4
until that is done. reiserfs4 wants to keep a plugin
id for each and every file. An additional filesystem
layer is the traditional solution to achieve advanced
features, but not an optimal solution in my opinion.
Yes gnome, kde and perhaps cifs do it. But if instead
they used file plugins a lot more could be shared.

Blush, I am not a file system expert

-Tim

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: reiser4 plugins

[Horst von Brand]

Timothy Webster <tdwebste2@yahoo.com> wrote:

[...]

> I think it is the task of the linux community to
> generalize the vfs layer and not lock out reiserfs4
> until that is done.

No... the task of the Linux community is to make a better kernel.

You will have to do the work to convince it that the changes give a better
kernel if included. That means the work to find out what they object to,
solve the problem (if it can be solved), and try again.

>                     reiserfs4 wants to keep a plugin
> id for each and every file.

Good for you. Nobody else has shown any interest in that.

>                             An additional filesystem
> layer is the traditional solution to achieve advanced
> features, but not an optimal solution in my opinion.

Right. And what is said here is to /use/ said layer (VFS) or see how it can
be changed to cater for your needs.

> Yes gnome, kde and perhaps cifs do it.

Gnome and KDE are userspace utilities, designed to run on several operating
systems that do not have such filesystem plugins at all, plus for the
foreseable future the mayority of Linux (file)systems won't have them
either; so they probably won't ever use it for portability's sake. CIFS is
a second class citizen AFAIU, as it exists for compatibility with legacy
systems only. So none of the above is in any way compelling.

>                                        But if instead
> they used file plugins a lot more could be shared.

> Blush, I am not a file system expert

/me neither ;-)
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Fwd: reiser4 plugins

[Perry Kundert]

On 6/24/05, Horst von Brand <vonbrand@inf.utfsm.cl> wrote:
> Timothy Webster <tdwebste2@yahoo.com> wrote:
>
> [...]
>
> > I think it is the task of the linux community to
> > generalize the vfs layer and not lock out reiserfs4
> > until that is done.
>
> No... the task of the Linux community is to make a better kernel.

    In general, isn't it better to first include modules providing
divergent but possibly interesting functionality (such as Reiser4) as
an "optional" or "experimental" component, and then slowly re-factor
desirable functionality into higher level facilities like the VFS?

    One of the strengths of Linux (so far), is that it hasn't tried
too hard to "guess" what functionality might be the next "killer"
kernel feature; if something is truly interesting, it will eventually
find its way more deeply into the kernel.  If not, it stays in an
"optional" driver.

>
> You will have to do the work to convince it that the changes give a better
> kernel if included. That means the work to find out what they object to,
> solve the problem (if it can be solved), and try again.
>
> >                     reiserfs4 wants to keep a plugin
> > id for each and every file.
>
> Good for you. Nobody else has shown any interest in that.

    The jffs2 filesystem may perhaps vary its functionality and
"on-flash" format wildly, depending on whether you use it with NOR or
NAND flash.  Not many people have shown interest in that; however, it
is still an important feature to some (ie. ME).

    This is an implicit analog of a plugin; Why force the VFS to take
note of how jffs2 happens to handle its formatting for NAND and NOR
flash?  Just let it use the correct "plugin"!

    Virtually every non-trivial filesystem has the concept of varying
its behaviour, some implicitly (jffs2 NAND or NOR?), some explicitly
(NFS mount options).  Should we force them ALL to select ALL format
variability via the VFS?

    So what if reiser4 wants to have a plugin that converts all
english files to swahili or pig latin as an on-disk format, and it
wants to call it a "plugin".  So long as it doesn't interfere with the
correctness of other filesystems, why prevent it from merging?

    I believe that it would be a GREATER error to force VFS changes
supporting the generic idea of a "plugin", before proving that the
concept has more general applicability.  See if reiser4 lives or dies
first, as an optional experimental filesystem.  No one is forced to
use it.  In a couple of years, if it becomes an ignored "backwater"
filesystem (like, say, coda?), then remove it from the mainline
kernel.  No one will  notice...

>
> >                             An additional filesystem
> > layer is the traditional solution to achieve advanced
> > features, but not an optimal solution in my opinion.
>
> Right. And what is said here is to /use/ said layer (VFS) or see how it can
> be changed to cater for your needs.

    If reiser4 wants to provide a usable interface (internally), to
select behaviour that is traditionally selected by choosing a
completely separate filesystem, then why not let it?  It harms *no
one* -- forcing changes to the VFS before it is proven that this kind
of functionality is even generally desirable would be a HUGE error!

>
> > Yes gnome, kde and perhaps cifs do it.
>
> Gnome and KDE are userspace utilities, designed to run on several operating
> systems that do not have such filesystem plugins at all, plus for the
> foreseable future the mayority of Linux (file)systems won't have them
> either; so they probably won't ever use it for portability's sake. CIFS is
> a second class citizen AFAIU, as it exists for compatibility with legacy
> systems only. So none of the above is in any way compelling.
>
> >                                        But if instead
> > they used file plugins a lot more could be shared.

    The ext2 and ext3 filesystems support the same basic on-disk
format, as everyone knows.  However, the selection of ext3 provides an
on-disk journal, which (if the filesystem is then mounted as ext2), is
ignored.  You select the "journalling plugin" by mounting the fs as
ext3.  You turn it off by mounting it as ext2.

    Reiser4 provides a way to do precisely this kind of selection of
variable on-disk formats by specifying a plugin.

   I ask you -- if everyone in kernel-land is so convinced that you
should always select varying on-disk formats via the VFS, then *why*
hasn't ext2/ext3 been merged into a single filesystem?  Surely the
"journalling" plugin of this filesystem is a prime candidate for
selection via the VFS?



    Here's the deal.

    So Hans can be a butt-head.  Get over it.  Big projects attract
big egos.  This is a good thing!

    Reiser4 provides potentially very interesting functionality, some
of which may or may not  (over time) be interesting to other FS
implementers.  At that time, it will be "abstracted" into the VFS.
*AFTER* it has been proven safe and useful to do so!

    Until then, including Reiser4 in the main-line kernel is
*completely* consistent with the "do no harm" philosophy that has let
other filesystems and drivers have be included.  No one running a
"stable" server is going to let the "experimental" Reiser4 filesystem
code anywhere near their kernel -- just like all other "experimental"
code!

    Others (like me), will use the 'md' drivers to provide the stable,
error-free media that ALL filesystems, including reiser4, assume,
'rsync' to provide for backups since it is clearly marked as
"experimental", and continue to benefit from the improved atomicity
and speed that reiser4 provides -- just as I have been doing for the
last 18 months!

--
    -pjk

"If you will not fight when your victory will be sure and not too
costly, you may come to the moment when you will have to fight with
all the odds against you and only a precarious chance for survival.
There may even be a worse case. You may have to fight when there is no
hope of victory, because it is better to perish than to live as
slaves." -- Winston Churchill

Re: reiser4 plugins

[Al Viro]

On Thu, Jun 23, 2005 at 11:49:51PM -0700, Hans Reiser wrote:
> Regarding trust, Christophe comes out the gate using the words "useless
> abstraction layer" that happens to be a core feature of our design,
> demanding we cut it out, and not really understanding it or recognizing
> any functionality it provides, and you can't really expect me to respect
> this, can you?
> 
> Now, if his target is reduced to whether we can eliminate a function
> indirection, and whether we can review the code together and see if it
> is easy to extend plugins and pluginids to other filesystems by finding
> places to make it more generic and accepting of per filesystem plugins,
> especially if it is not tied to going into 2.6.13, well, that is the
> conversation I would have liked to have had.

Have I missed the posting with analysis of changes in locking scheme
and update of proof of correctness?  If so, please give the message ID.

_That_ had been the major showstopper for any merges, IIRC.

Re: reiser4 plugins

[Hans Reiser]

Al Viro wrote:

>Have I missed the posting with analysis of changes in locking scheme
>and update of proof of correctness?  If so, please give the message ID.
>
>_That_ had been the major showstopper for any merges, IIRC.
>  
>
Ah, the prince of helpfulness has arrived.

Yes, as I remember, last time with V3 you announced that there were race
conditions that we needed to fix if V3 was to be merged, you would not
tell us what they were when asked, Linus merged us anyway, you never did
tell us what they were, later vs fixed some race conditions but I have
no idea if they were the same ones you found, oh well, getting rid of
bugs never was your objective was it?  Does V3 still have those race
conditions you spoke of?

Proof of correctness, is that where we check and see if the filesystems
all mount/unmount before checking in code changes to the stable release
branch?  Oh dear, that was unkind of me.

Ok, sure, define what you want in the way of a proof of correctness and
an analysis.  Is this a new VFS tradition?  Is it documented anywhere? 
Are there tools for it?  Probably I should ask Zam if we already did it
too.....  Zam?

Hans

Re: reiser4 plugins

[Al Viro]

On Fri, Jun 24, 2005 at 02:03:33AM -0700, Hans Reiser wrote:
> Al Viro wrote:
> 
> >Have I missed the posting with analysis of changes in locking scheme
> >and update of proof of correctness?  If so, please give the message ID.
> >
> >_That_ had been the major showstopper for any merges, IIRC.
> >  
> >
> Ah, the prince of helpfulness has arrived.
> 
> Yes, as I remember,

Kindly put some efforts into remembering the thread that contains e.g.
http://marc.theaimsgroup.com/?l=linux-kernel&m=109347094309283&w=2

If that work (summary: introduction of hybrid objects invalidates the
existing locking scheme for directories and that had lead at least to
several user-exploitable deadlocks described in details in the same
thread; current proof of correctness is in the tree, see
Documentation/filesystems/Directory-Locking.txt and at the very least
it needs to be updated) had been done - please, give the message ID
of posting with such update.  If not - please, arrange getting it done.

Re: reiser4 plugins

[Al Viro]

On Fri, Jun 24, 2005 at 03:45:23PM +0100, Al Viro wrote:
> On Fri, Jun 24, 2005 at 02:03:33AM -0700, Hans Reiser wrote:
> > Al Viro wrote:
> > 
> > >Have I missed the posting with analysis of changes in locking scheme
> > >and update of proof of correctness?  If so, please give the message ID.
> > >
> > >_That_ had been the major showstopper for any merges, IIRC.
> > >  
> > >
> > Ah, the prince of helpfulness has arrived.
> > 
> > Yes, as I remember,
> 
> Kindly put some efforts into remembering the thread that contains e.g.
> http://marc.theaimsgroup.com/?l=linux-kernel&m=109347094309283&w=2
> 
> If that work (summary: introduction of hybrid objects invalidates the
> existing locking scheme for directories and that had lead at least to
> several user-exploitable deadlocks described in details in the same
> thread; current proof of correctness is in the tree, see
> Documentation/filesystems/Directory-Locking.txt and at the very least

	gaaack...  Sorry - should've proofread better -
s/Directory-Locking.txt/directory-locking/ in the above.
> it needs to be updated) had been done - please, give the message ID
> of posting with such update.  If not - please, arrange getting it done.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: reiser4 plugins

[Hans Reiser]

Al Viro wrote:

>On Fri, Jun 24, 2005 at 02:03:33AM -0700, Hans Reiser wrote:
>  
>
>>Al Viro wrote:
>>
>>    
>>
>>>Have I missed the posting with analysis of changes in locking scheme
>>>and update of proof of correctness?  If so, please give the message ID.
>>>
>>>_That_ had been the major showstopper for any merges, IIRC.
>>> 
>>>
>>>      
>>>
>>Ah, the prince of helpfulness has arrived.
>>
>>Yes, as I remember,
>>    
>>
>
>Kindly put some efforts into remembering the thread that contains e.g.
>http://marc.theaimsgroup.com/?l=linux-kernel&m=109347094309283&w=2
>
>If that work (summary: introduction of hybrid objects invalidates the
>existing locking scheme for directories and that had lead at least to
>several user-exploitable deadlocks described in details in the same
>thread; current proof of correctness is in the tree, see
>Documentation/filesystems/Directory-Locking.txt and at the very least
>it needs to be updated) had been done - please, give the message ID
>of posting with such update.  If not - please, arrange getting it done.
>
>
>  
>
We disabled metafiles until we can fix this.  With no metafiles, the
issue does not exist.

Thanks for your assistance with finding that problem.

Hans

Re: reiser4 plugins

[Alan Cox]

On Gwe, 2005-06-24 at 02:12, Hans Reiser wrote:
> >In which case the features belong in the VFS as all those with
> >experience and kernel contributions have been arguing.
> So you fundamentally reject the prototype it in one fs and then abstract
> it to others development model?

More fundamentally - prototype things *out* of the main kernel. If
everyone was doing their prototyping in kernel Andrew Morton would by
now be a team of about 25

Alan

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 2587 bytes --]

On Fri, Jun 24, 2005 at 11:41:21AM +0100, Alan Cox wrote:
>
>More fundamentally - prototype things *out* of the main kernel. If
>everyone was doing their prototyping in kernel Andrew Morton would by
>now be a team of about 25

This is going semi off-topic but how then do you justify regression
that's apparently confessed? :)

It appears -mm is more of a prototype tree than a development tree;
I remember Con Kolivas cursing (with SMP nice support) that the
interface is too lively to keep supporting one patch. Unfortunately
I can't find the original post I'm thinking about but
http://lkml.org/lkml/2005/5/16/68 says essentially the same thing.

There's also my all-time favorite, http://lkml.org/lkml/2005/3/14/4

The lack of QA seems appalling here, and I'm sure Reiser has had
to do more of that for DARPA than most linux kernel hackers around.

What I'm saying here is isn't it a bit hypocritic to ramble on that
we need real assurances for this to work, community proof and bugfixes
on the list mean nothing, when other core systems seem to be much
looser in this respect?

Sure, "other people merge design-breakers and bugs" is NOT a justification
for merging Reiser4, of course, but Reiser4's track record has vastly
cleaned itself up.

Most bug reports come from broken hardware or unsupported patches or
old code. Just have Namesys swear they won't introduce design changes
until the userland utils are available, and won't do it at all after
EXPERIMENTAL has been removed.
They already did this with changes that require mkfs :>

Here's a real suggestion, for LKML et al, dropping the sarcastic mode.

0. Namesys addresses the code beautification reqs mentioned here earlier.

1. Merge Reiser4 sans metas into 2.6.13.
2. Namesys can have a separate metas patch for testing.
3. Gradually merge Reiser4 architecture into VFS and if this really
   requires a 2.7, as (iirc) Valdis Kletnieks suggested, make it so.
   Might do the rest of the kernel some good too.

The above is a lame compromise; I'd much rather have the meta system
in, as is dropping the stupid name, ..metas/ is better, and dropping "meta
files can have meta data, ie. endless recursion in ..metas/

Then it can be merged into the VFS as needed, but if there truly are issues 
left, I realize it can't probably be fixed fast enough.

Having metas on by default probably leads to a lot of bug reports,
which get fixed, and at the same time makes VFS-merging easier,
when metas-related bug issues are fixed in VFS.

My two grumpy cents.

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Nick Piggin]

Markus Törnqvist wrote:

 > I can't find the original post I'm thinking about but
 > http://lkml.org/lkml/2005/5/16/68 says essentially the same thing.

The scheduler is being improved for better behaviour on complex
topologies like multi core + NUMA and multi level NUMA systems.
If Con's work had gone in first, then conversely these improvements
would have had to wait.

> There's also my all-time favorite, http://lkml.org/lkml/2005/3/14/4
> 

What's wrong with that? The slowdown is due to the workload
becoming disk bound. The reasons are still not entirely clear,
but I don't think it is a recent (ie. 2.6) regression (or even
a regression at all IIRC).

> The lack of QA seems appalling here, and I'm sure Reiser has had
> to do more of that for DARPA than most linux kernel hackers around.
> 

And what QA would you have preferred?

I think if you are resorting to bringing up all time favourite
blunders when trying to justify Reiser4 being included, then
that is a sign right there that something is fundamentally wrong
(if not with the code, then with your line of thought0

And note my email has nothing to do with any *real* argument for
or against R4.

Thanks,
Nick

Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 3508 bytes --]

On Mon, Jun 27, 2005 at 07:46:15PM +1000, Nick Piggin wrote:
>
>The scheduler is being improved for better behaviour on complex
>topologies like multi core + NUMA and multi level NUMA systems.
>If Con's work had gone in first, then conversely these improvements
>would have had to wait.

Or be merged in later.

The problem is, why do the interfaces have to live so much that
examples like this come to my ears (or eyes ;) all the time?

I hate to say this without digging out any URLs, but one friend
of mine says he has a very hard time doing any networking code
because it's too labile. Maybe that's being embettered for something
else too?

Or the other friend who curses that the networking code is just
crap and basically has to rewrite the code to get it working.
Yes, I've tried to get these guys to submit their code, but they
argue back that no one wants to see it.

>>There's also my all-time favorite, http://lkml.org/lkml/2005/3/14/4
>
>What's wrong with that? The slowdown is due to the workload
>becoming disk bound. The reasons are still not entirely clear,
>but I don't think it is a recent (ie. 2.6) regression (or even
>a regression at all IIRC).

It's not that easy.

So let's say the scheduler slowdown is a temporary situation to
adapt it to multicore+NUMA.
I assume that's what you mean, by having the kernels on par
with 2.6.10 and the above paragraph.

Why on earth did the slowdown ever get merged and we have to wait
for it to be on par with some older version?

Maybe the multicore+NUMA guys don't think it's a regression, hell,
it may be better for them at the cost of the embedded (or desktop
or whoever) guys.

Still my initial reaction is "if a patch slows things down, revert
it. If it didn't do anything, keep reverting until you have the
original situation."
It's also my reaction after the initial one :)

Sure, 2.4 and others did also have issues, so you couldn't automatically
assume each and every new kernel would be better than the last,
but it seems nowadays it's also hard to predict even the trend.

But to be fair, Linux is not a disaster yet and you guys are doing
good work, despite things looking very shaky at times and
my temper being more and more easily flared nowadays at signs
of trouble :)

>I think if you are resorting to bringing up all time favourite
>blunders when trying to justify Reiser4 being included, then
>that is a sign right there that something is fundamentally wrong
>(if not with the code, then with your line of thought0

Myeah, I know, this is not helping the Reiser4 cause _as_such_.

What I'm saying is, Reiser4 could be merged as it looks like a
lot of other stuff is merged too, which may not be very
tested.

Reiser4 at least is tested.

We can skip the parts that appear to suck.

>And note my email has nothing to do with any *real* argument for
>or against R4.

I think this could be further split into an argument about
the development model :P

Having a separate dev tree would keep the majority happy,
as things are "good enough for most people" and after all
has been evened out in the dev tree, a new stable tree can
be launched.

This particular new file system should not be too drastic a
thing to merge into a stable kernel, without the instability-
inducing factors, and I think in the old model, all this would
have been fleshed out in the dev tree automagically.

Please do correct me if I'm wrong :)

Thanks!

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Nick Piggin]

Markus Törnqvist wrote:
> On Mon, Jun 27, 2005 at 07:46:15PM +1000, Nick Piggin wrote:
> 
>>The scheduler is being improved for better behaviour on complex
>>topologies like multi core + NUMA and multi level NUMA systems.
>>If Con's work had gone in first, then conversely these improvements
>>would have had to wait.
> 
> 
> Or be merged in later.
> 

It _was_ merged later. Con was talking about Andrew's -mm tree.

> The problem is, why do the interfaces have to live so much that
> examples like this come to my ears (or eyes ;) all the time?
> 

The scheduler interfaces are probably among the most stable
in the kernel. Con was talking about internal implementation.

> I hate to say this without digging out any URLs, but one friend
> of mine says he has a very hard time doing any networking code
> because it's too labile. Maybe that's being embettered for something
> else too?
> 

It seems there are plenty of people who can do networking code
just fine.

> Or the other friend who curses that the networking code is just
> crap and basically has to rewrite the code to get it working.
> Yes, I've tried to get these guys to submit their code, but they
> argue back that no one wants to see it.
> 

I get the feeling your friend is making up some tall tales if he
says he rewrote the networking code to be much better than it is
now. But I can guarantee him that if he has it will be of great
interest to the network developers.

> 
>>>There's also my all-time favorite, http://lkml.org/lkml/2005/3/14/4
>>
>>What's wrong with that? The slowdown is due to the workload
>>becoming disk bound. The reasons are still not entirely clear,
>>but I don't think it is a recent (ie. 2.6) regression (or even
>>a regression at all IIRC).
> 
> 
> It's not that easy.
> 
> So let's say the scheduler slowdown is a temporary situation to
> adapt it to multicore+NUMA.
> I assume that's what you mean, by having the kernels on par
> with 2.6.10 and the above paragraph.

I'm pretty sure it isn't a scheduler slowdown, but rather the
system is getting disk bound. I think the reporter has
performance back to 2.6.10 levels - it is actually something I
still have to follow up on, but it is difficult because the
reporter isn't able to share his code, although he's otherwise
very helpful.

> 
> Why on earth did the slowdown ever get merged and we have to wait
> for it to be on par with some older version?
> 

If slowdowns do get merged, it is generally because they either
haven't been reported until being merged or they make an accepted
tradeoff. Not that they haven't been tested.

Let's just get this straight, no amount of QA other than releasing
a kernel and asking people to use it is going to find all the bugs
that will be encountered.

> 
> Please do correct me if I'm wrong :)
> 

I mostly agree with you apart from your specific examples I think
we could do things better, and most of us have made some big blunders,
However there is just no way that bringing any of that up will help
Reiser4 being merged.

Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 5986 bytes --]

On Tue, Jun 28, 2005 at 10:23:19AM +1000, Nick Piggin wrote:

>The scheduler interfaces are probably among the most stable
>in the kernel. Con was talking about internal implementation.

Hmm, I'm by no means a kernel developer, but looking at the
patches, it looked like the functions were just renamed and
maybe something else was changed.

I may be totally wrong here, and I apologize in advance if I am,
but it seems to me like there would be no real need in changing
some syntax?

>I get the feeling your friend is making up some tall tales if he
>says he rewrote the networking code to be much better than it is
>now. But I can guarantee him that if he has it will be of great
>interest to the network developers.

Yeah, I just talked this over with him.

[
 IF THIS IS BULLSHIT, LET ME KNOW AND I'LL CORRECT HIM ;)
 POINTERS TO RELEVANT DOCUMENTATION ALSO WELCOME ;)
]

His grudge was that conntrack was used everywhere, not like
in 2.4, with a slow lookup in proc, that made it unusable
for large nat systems.

They wrote some reimplementation of the lookup part, which
they didn't release.

Then they figured out how to capture the packets before
reaching conntrack and doing some mojo on them and reinjecting
them somewhere later or bypassing conntrack altogether.

However, his opinions were even less motivated than mine,
it seems, as he didn't have anything except some ipfilter
archives he read, where they cursed the Linux way of
doing things.

I haven't looked all too deeply into the network part, though,
as I'm starting to get lazy and like to have a separate,
dedicated piece of hardware do the network stuff for me...

So maybe I should just retract my comments and look into
every possible subsystem's development and see for myself,
which I don't have time for :)

>I'm pretty sure it isn't a scheduler slowdown, but rather the
>system is getting disk bound. I think the reporter has
>performance back to 2.6.10 levels - it is actually something I
>still have to follow up on, but it is difficult because the
>reporter isn't able to share his code, although he's otherwise
>very helpful.

It's just that for cases like these testing would be great.

I'm not really sure how to do it, though, as it would require
a lot of hardware and time and someone who can write a decent
test suite.

If anyone has links to good test suites, I'm all ears :)

So, any chances of something being messed up in the disk
part of the code?

...Too bad I just heard a big slowdown-in-disk-i/o curse
was caused by dma falling off, for whatever reason (bug?)
so that can't be used as forensics either :P

>If slowdowns do get merged, it is generally because they either
>haven't been reported until being merged or they make an accepted
>tradeoff. Not that they haven't been tested.
>
>Let's just get this straight, no amount of QA other than releasing
>a kernel and asking people to use it is going to find all the bugs
>that will be encountered.

Would you think a separate dev tree would help here?

I've also talked long with a guy from one of our major customers
who says he's been losing faith in Linux over the past year, as
his boxes are going oops or panicking every once in a while.

The problem is that their system is too redundantly set up for
a single oops or panic to matter much, so I haven't convinced them
to report bugs. They'd rather boot the boxes and curse.

He just figured it odd that this started happening after the new
dev model, but the timelines are too granular for a real correlation.
Just a strong gut feeling, and that's the kind of FUD that makes
people look to *BSD, and a gut feeling that may have some merit
to it, although it could never be proven. So don't take that
as a flame or anything, just an observation :P

Sure it must be difficult developing the kernel, if it were easier,
I'd probably be at it too, but people don't like not being well
informed of accepted tradeoffs, as they might not accept said
tradeoff, and tracking what caused it may be too much.

A lot of people out there don't like vendor kernels either,
with 65536 backported patches and the 65537th voids your warranty.

It's also probably very difficult to do analytical QA before
submitting code, as bugs ALWAYS run through.

But my gut tells me this -mm thing may not be the optimal way,
as people are more scared of -mm, where some things may or
may not work, than a separate dev tree, of which kerneltrap
or whoever can post "this is severely broken, test if you like"
and "this seems to be going mostly strong now, test it on your
everyday desktop"

A lot of complaints seem to come from people who use 
patchsets with half-assed backports and experimental code,
having a separate tree may drive these people to use
the experimental stuff knowing what it is, instead of
believing their patch set is as good as vanilla.

>I mostly agree with you apart from your specific examples I think
>we could do things better, and most of us have made some big blunders,
>However there is just no way that bringing any of that up will help
>Reiser4 being merged.

Myeah, if I try to keep it a bit more on topic, I'm just saying that
Reiser4 (sans metas) has been heavily tested, and now that the
Namesys guys are apparently beautifying the code, we should be
good with everything.

Still, this is an instinct, but it seems like the bigger VFS
overhauls would be easier to do in an official dev tree with
a big note saying "THIS WILL BE BROKEN FOR A MONTH."

If this was the case, so bad that people couldn't develop 
anything with the dev tree, it would probably be easier to 
maintain a rollback-vfs-changes patch than constantly keeping 
in sync with -mm.

That method could be used for every system undergoing extensive
work that shows up in other systems.

The difference may not be big, but it may be enough.

Thanks!

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Jesper Juhl]

On 6/27/05, Markus   Törnqvist <mjt@nysv.org> wrote:
[...]
> 
> I hate to say this without digging out any URLs, but one friend
> of mine says he has a very hard time doing any networking code
> because it's too labile. Maybe that's being embettered for something
> else too?
> 
> Or the other friend who curses that the networking code is just
> crap and basically has to rewrite the code to get it working.
> Yes, I've tried to get these guys to submit their code, but they
> argue back that no one wants to see it.
> 
[...]

I'm pretty damn sure the relevant maintainers (and a bunch of people
on LKML and the netdev lists in general) would like to see patches
that improves their code.
If these friends of yours are sitting on patches that make massive
improvements to the code and they are not submitting patches then they
are not exactely helping (and I'd suspect them to just be full of BS)
- they should get their code merged instead of having to maintain it
themselves out-of-tree for ever - let the rest of us bennefit as well.
It's my experience that if you can explain the problems your patch fix
and explain well why the fix is sane, then getting your patches merged
doesn't have to be hard.


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: reiser4 plugins

[Alan Cox]

On Llu, 2005-06-27 at 10:18, Markus Törnqvist wrote:
> Sure, "other people merge design-breakers and bugs" is NOT a justification
> for merging Reiser4, of course, but Reiser4's track record has vastly
> cleaned itself up.

The discussion is about merging from -mm, not into -mm. The merge into
-mm isn't at all under discussion so I don't the relevance of the
comparison.

> 0. Namesys addresses the code beautification reqs mentioned here earlier.
> 
> 1. Merge Reiser4 sans metas into 2.6.13.
> 2. Namesys can have a separate metas patch for testing.
> 3. Gradually merge Reiser4 architecture into VFS and if this really
>    requires a 2.7, as (iirc) Valdis Kletnieks suggested, make it so.
>    Might do the rest of the kernel some good too.

0 and 1 look like the first right steps to take to me as well. That will
allow people to use reiser4 and give it a good hammering see what comes
out and how it benches in real life. 

Alan

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alan Cox wrote:
> On Iau, 2005-06-23 at 23:04, David Masover wrote:
> 
>>>What for? It works just fine as it stands, AFAICS.
>>
>>So does DOS.  Do you use DOS?  I don't even use DOS to run DOS programs.
> 
> 
> False argument. So does the pen, so do hinges on doors. Do you still
> have hinges on your doors - probably.

Indeed.  Because there's nothing better -- not because I "like it the
way it is".

>>"Ain't broke" is the battle cry of stagnation.
> 
> 
> Its also the battle cry of everyone over the age of 20 who also has a
> real job to do 8)

You caught me.  I'm not over 20.  But I have a real job, with a company
that understands the difference between "ain't broke" and "works well".

>>But, there are some things Reiser does better and faster than ext3, even
>>if you don't count file-as-directory and other toys.  There's nothing
>>ext3 does better than Reiser, unless you count the compatibility with
>>random bootloaders and low-level tools.
> 
> 
> Certainly compared with reiser3 you've missed a few out including
> resilience to disk errors (nearly nil on reiser3), and SMP scaling.

Actually, I was talking about reiser4.  And Hans corrected me on that...

Although resilience to disk errors isn't a design decision.  That's what
SMART and new hard drives are for.  And if you're stubborn enough to
keep the same FS around, there's dm-bbr.

I think Hans (or someone) decided that when hardware stops working, it's
not the job of the FS to compensate, it's the job of lower layers, or
better, the job of the admin to replace the disk and restore from backups.

>>You know how many I've had thrashed on Reiser4?  Two.  The first one was
>>with a VERY early alpha/beta, and the second one was when I dropped a
>>laptop and the disk failed.
> 
> 
> Entirely or bad blocks ? The latter should have a minimal cost on a well
> designed fs.

I was able to recover from bad blocks, though of course no Reiser that I
know of has had bad block relocation built in...  But I got all my files
off of it, fortunately.

But the disk did fail, completely, later.  Lots of loud clicking.

>>Duplication of effort.  With plugins, we can optimize the upper layers
>>of ALL filesystems, regardless of the lower layers, in such a way that
> 
> 
> In which case the features belong in the VFS as all those with
> experience and kernel contributions have been arguing.

No one's arguing that.  What we're arguing is that there does seem to be
a bit of prejudice when fuck-me-with-a-chainsaw and YOU ARE A FOOL FOR
ENABLING THIS all got in, and with at least one of those, there doesn't
seem to be any intention of changing it -- all of that, and even with an
expressed intention to fix the aesthetical problems with Reiser4 later,
we can't get the working version in now.

More infuriatingly, I, at least, have a distinct feeling that once all
the issues are fixed, an entirely different crowd of benevolent
dictators will come around and say that we can't get in because we
change the VFS.  At least some people on this list have said things to
that effect.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrt7MngHNmZLgCUhAQKdXA/+PDpOzZYTVXgF2n4qiFyrmjFeQ6h0n8i6
c/hXx+QUU0Hw5mjq31+jf2vNpDCKQxcE/HTLdJlRfw8az+xklVOfxzEHf9yV41tv
mVKMRJYBhzk2mEvKEDNtnw47SQPBAKo9BtJvl7gOEofiPK/f2K/cy8yMUrow1E9D
02PNT0XX8ysoe86Dqip35+sphczkQN8gilXyUQujNe8edEdkW7PBhbJn92zBQag2
JxA194bquxRyhW78T3tKAEN6/tTPgZYJNy202KC619zzLlK3TslwjjfOQILdRb2i
NNkaSQBdYDK70BiFs5Ri7ZbfJHenY6mgGv7yG0vjGF6zjoXtVNsKGrYt9KBL6E1D
392ayxOlWCvBoG9n9sAUzHzcQxmU1lP6OHcO9xWrL6ySD7Fzv4rCtM0uo7gXOzNB
aDl5vK7q+ysEjOXZJWT0ikj5ndATCv0Ry8wnt1uL/uktOuaE0egwsouU0jgCDgx1
8Ib8KX/B6bhCy13WkYLPTb6Yg0k+ph0BUyONEcN5cxJIcqfcEt/4Un4MYM+CjGck
KLYrrZDclZr8p/paWdNqx1dI9NIBn+F3u78OcWY3NIGfZPh1jmSdG3LRt9DZo5bZ
ua2UlAiAWnEDHNLQyMH2zcji31DMqIUwmQ5bX9isBcxt8LORX+IKxoLdUICoY8JS
Ii5kNWJAjf4=
=Xsdv
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:
> Alan Cox wrote:
> > On Iau, 2005-06-23 at 23:04, David Masover wrote:

[...]

> >>>What for? It works just fine as it stands, AFAICS.

> >>So does DOS.  Do you use DOS?  I don't even use DOS to run DOS programs.

> > False argument. So does the pen, so do hinges on doors. Do you still
> > have hinges on your doors - probably.

> Indeed.  Because there's nothing better -- not because I "like it the
> way it is".

There being nothing better means nobody has ever been able to come up with
a better way. Most of the time at least.

> >>"Ain't broke" is the battle cry of stagnation.

> > Its also the battle cry of everyone over the age of 20 who also has a
> > real job to do 8)

> You caught me.  I'm not over 20.  But I have a real job, with a company
> that understands the difference between "ain't broke" and "works well".

"Doesn't work well" /is/ the definition of "broken". Modulo how irritating
the "not working well" is...

> >>But, there are some things Reiser does better and faster than ext3, even
> >>if you don't count file-as-directory and other toys.  There's nothing
> >>ext3 does better than Reiser, unless you count the compatibility with
> >>random bootloaders and low-level tools.

> > Certainly compared with reiser3 you've missed a few out including
> > resilience to disk errors (nearly nil on reiser3), and SMP scaling.

> Actually, I was talking about reiser4.  And Hans corrected me on that...
> 
> Although resilience to disk errors isn't a design decision.  That's what
> SMART and new hard drives are for.  And if you're stubborn enough to
> keep the same FS around, there's dm-bbr.

> I think Hans (or someone) decided that when hardware stops working, it's
> not the job of the FS to compensate, it's the job of lower layers, or
> better, the job of the admin to replace the disk and restore from
> backups.

Handling other people's data this way is just reckless irresponsibility.
Sure, you can get high performance if you just forego some of your basic
responsibilities.

> >>You know how many I've had thrashed on Reiser4?  Two.  The first one was
> >>with a VERY early alpha/beta, and the second one was when I dropped a
> >>laptop and the disk failed.

> > Entirely or bad blocks ? The latter should have a minimal cost on a well
> > designed fs.

> I was able to recover from bad blocks, though of course no Reiser that I
> know of has had bad block relocation built in...  But I got all my files
> off of it, fortunately.

And if the bad blocks had been on files? 

[...]

> > In which case the features belong in the VFS as all those with
> > experience and kernel contributions have been arguing.

> No one's arguing that.  What we're arguing is that there does seem to be
> a bit of prejudice when fuck-me-with-a-chainsaw and YOU ARE A FOOL FOR
> ENABLING THIS all got in, and with at least one of those, there doesn't
> seem to be any intention of changing it -- all of that, and even with an
> expressed intention to fix the aesthetical problems with Reiser4 later,
> we can't get the working version in now.
> 
> More infuriatingly, I, at least, have a distinct feeling that once all
> the issues are fixed, an entirely different crowd of benevolent
> dictators will come around and say that we can't get in because we
> change the VFS.  At least some people on this list have said things to
> that effect.

The dictators here have changed over time, sure. Their general attitude has
been the same all the way through, since the list started. And you might
not like it that way, but I am sure Linux is of the current high quality
exactly because any core code that gets into the kernel (and a new
filesystem that wants to be central is clearly of this kind) is checked by
multiple people with (sometimes widely) diverging backgrounds, interests,
and views. Linus himself has stated more than once that his principal job
isn't to integrate code into the kernel, but leaving stuff out.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
> David Masover <ninja@slaphack.com> wrote:
> 
>>Alan Cox wrote:
>>
>>>On Iau, 2005-06-23 at 23:04, David Masover wrote:
> 
> 
> [...]

[...]

>>>>"Ain't broke" is the battle cry of stagnation.
> 
> 
>>>Its also the battle cry of everyone over the age of 20 who also has a
>>>real job to do 8)
> 
> 
>>You caught me.  I'm not over 20.  But I have a real job, with a company
>>that understands the difference between "ain't broke" and "works well".
> 
> 
> "Doesn't work well" /is/ the definition of "broken". Modulo how irritating
> the "not working well" is...

Well, I agree with you there, but to most people, "broken" = "Doesn't
work at all."  Stagnation happens when people cling to the old way of
doing something until they are actually forced to change.

One example of this is energy.  Why do we still burn coal and oil?  Even
lithium ion batteries still give a 200-300 mile range on one electric
car I was looking at.  There is enough wind in Iowa alone to power half
the country, and if we were to put up a bunch of solar panels in Nevada,
we'd be done.  With the money we would save on not buying oil, we could
finish the fuel-cell technologies being worked on, and eventually, the
US would get its energy entirely from clean sources.

For that matter, we'd easily save enough money, not to mention the
public health benefits (no more smog), to pay for the transition
(building hydrogen pumping stations and such, buying new cars...)

But people don't like to change.  So we burn oil and make smog and choke
on our own stupidity.

That debate really belongs off-list, by the way.  All flames on this
topic, hit "reply", not "reply-to-all".

>>>>But, there are some things Reiser does better and faster than ext3, even
>>>>if you don't count file-as-directory and other toys.  There's nothing
>>>>ext3 does better than Reiser, unless you count the compatibility with
>>>>random bootloaders and low-level tools.
> 
> 
>>>Certainly compared with reiser3 you've missed a few out including
>>>resilience to disk errors (nearly nil on reiser3), and SMP scaling.
> 
> 
>>Actually, I was talking about reiser4.  And Hans corrected me on that...
>>
>>Although resilience to disk errors isn't a design decision.  That's what
>>SMART and new hard drives are for.  And if you're stubborn enough to
>>keep the same FS around, there's dm-bbr.
> 
> 
>>I think Hans (or someone) decided that when hardware stops working, it's
>>not the job of the FS to compensate, it's the job of lower layers, or
>>better, the job of the admin to replace the disk and restore from
>>backups.
> 
> 
> Handling other people's data this way is just reckless irresponsibility.
> Sure, you can get high performance if you just forego some of your basic
> responsibilities.

Who's responsibility?

It's the responsibility of the user to back up their files.  Sure, this
time it was just a little dust, but maybe next time it'll be coffee
spilled on the box.  Or maybe a power surge.  Maybe someone just gets
mad enough and kicks the computer very hard.

Reiser4 has been absolutely immune to power loss for me, and quite
stable in the face of some corruption, but as soon as I see bad blocks,
I throw out the disk, I don't try to make the FS deal with it.  And
that's all that's missing right now -- a way to use a disk which has
some bad blocks -- and that's implemented by dm-bbr if you're that stubborn.

Ultimately, disks are cheap.  Cheaper than data loss.

>>>>You know how many I've had thrashed on Reiser4?  Two.  The first one was
>>>>with a VERY early alpha/beta, and the second one was when I dropped a
>>>>laptop and the disk failed.
> 
> 
>>>Entirely or bad blocks ? The latter should have a minimal cost on a well
>>>designed fs.
> 
> 
>>I was able to recover from bad blocks, though of course no Reiser that I
>>know of has had bad block relocation built in...  But I got all my files
>>off of it, fortunately.
> 
> 
> And if the bad blocks had been on files? 

Then no FS would save me.

> [...]
> 
> 
>>>In which case the features belong in the VFS as all those with
>>>experience and kernel contributions have been arguing.

[...]

>>More infuriatingly, I, at least, have a distinct feeling that once all
>>the issues are fixed, an entirely different crowd of benevolent
>>dictators will come around and say that we can't get in because we
>>change the VFS.  At least some people on this list have said things to
>>that effect.
> 
> 
> The dictators here have changed over time, sure. Their general attitude has
> been the same all the way through, since the list started. And you might
> not like it that way, but I am sure Linux is of the current high quality
> exactly because any core code that gets into the kernel (and a new
> filesystem that wants to be central is clearly of this kind) is checked by
> multiple people with (sometimes widely) diverging backgrounds, interests,
> and views. Linus himself has stated more than once that his principal job
> isn't to integrate code into the kernel, but leaving stuff out.

I mean, half of the benevolent dictators at any given time don't seem to
like reiser4 for some reason or other.  I get the feeling that if we
satisfy that half, the way in which we do it will irritate the other
half.  But I don't know, and I'm going to leave the more specific
debates soon.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzRmHgHNmZLgCUhAQIOyw//bL4sUYQyGSwjKnKZxTznzNnchn0rpNsx
PnxrQEvETtkqFlidToJPKDEP3H32tE5W3SeONhIprbQVOK9BagI5Ojm+AhM8pXXa
8K0NJmedlnjMJp5yAxbB+dIwPQ8IN8cG7cdeZCi4ij76yKiDFEqZHXrpm/bvXcbB
jzzOF86XPVfexGOUQkwDpR0gmFeY+zIp3+7TGKwG17xHYOej7KpyQctYB7e1me9t
MKFll/2USUAglvtdHQ6on2T9slmBTRMny4OFJXKnBTo6rILBTvo6sxC3hG2lv1lN
RYXHOd/AV/JQHzhZ1f1vRAn9VxdsUkMgvcOq3k3+ofUP86b1VqxsqGP4Mv3S8uka
hghTqBh7ivOr5llCwUHudDIXjRzNw1UBwi6UBC+YB0QQzR3vzc3Q0rvkD1v6hoV+
Li+sX0GhgjaIDzTE+e1rJOAlTAIxeQ9dRMDVBOR5b9HVtkxZ20SSdagDSeHz9JPH
Lv0JmqkvYgVf703a3wt56FpTIMd7XvY6dUSR9u3k1xUGQg2jeJA0gZ3tySSogKbT
xick+z+m1THTIyA5tXA1JRS1+GNAc4yCL4mHPcUKVWPsKfxaIDB4/+6eCll7Ctxz
XzP6s8nv4HkPAsWe1iLvJmuUXdiradC87MJiDGCKM6mbNZ0hFGnKa8zG8nI4C/Ua
qHz+cXVkgi8=
=QFCz
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

On Thu, Jun 23, 2005 at 11:34:50PM -0400, Horst von Brand wrote:
>David Masover <ninja@slaphack.com> wrote:

>> I think Hans (or someone) decided that when hardware stops working, it's
>> not the job of the FS to compensate, it's the job of lower layers, or
>> better, the job of the admin to replace the disk and restore from
>> backups.
>Handling other people's data this way is just reckless irresponsibility.
>Sure, you can get high performance if you just forego some of your basic
>responsibilities.

Your honest-to-bog opinion is that the FS vendor is responsible for
the admin not taking backups or the hardware vendor shipping crap?

*still trying to understand how that can be*

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Theodore Ts'o]

On Mon, Jun 27, 2005 at 12:21:38PM +0300, Markus   T?rnqvist wrote:
> On Thu, Jun 23, 2005 at 11:34:50PM -0400, Horst von Brand wrote:
> >David Masover <ninja@slaphack.com> wrote:
> 
> >> I think Hans (or someone) decided that when hardware stops working, it's
> >> not the job of the FS to compensate, it's the job of lower layers, or
> >> better, the job of the admin to replace the disk and restore from
> >> backups.
> >Handling other people's data this way is just reckless irresponsibility.
> >Sure, you can get high performance if you just forego some of your basic
> >responsibilities.
> 
> Your honest-to-bog opinion is that the FS vendor is responsible for
> the admin not taking backups or the hardware vendor shipping crap?
> 
> *still trying to understand how that can be*

Most Linux users are using PC-class hardware.  And Ted's First Law of
PC-Class Hardware is: "Most of it is crap".  And then there's Ted's
Second Law, "Too many system administrators don't do backups".  This
is because most system admins are users who've never been trained to
be a sysadmin, or who haven't (yet) had weeks or months of works
disappear after a hardware failure.  

So it's a matter of matching the filesystem to the needs of the user.
If you have a filesystem which is blazingly fast, but which at the
slightest sign of trouble, trashes your data, versus one which is fast
but perhaps not-so-fast as the other filesystem, but which is much
more reliable, which would you choose?  

XFS has similar issues where it assumes that hardware has powerfail
interrupts, and that the OS can use said powerfail interrupt to stop
DMA's in its tracks on an power failure, so that you don't have
garbage written to key filesystem data structures when the memory
starts suffering from the dropping voltage on the power bus faster
than the DMA engine or the disk drives.  So XFS is a great filesystem
--- but you'd better be running it on a UPS, or on a system which has
power fail interrupts and an OS that knows what to do.  Ext3, because
it does physical block journalling, does not suffer from this problem.
(Yes, Resierfs uses logical journalling as well, so it suffers from
the same problem.)

So perhaps it's not the job of the FS vendor to be responsible for
crap hardware or lazy sysadmins that don't do backups.  But a system
administrator who knows that he doesn't do backups frequently enough,
or is running on cheap, crap hardware, would be wise to consider
carefully which filesystem he/she wants to use given the systems
configuration and his backup habits.  

Me, I'll go for the robust filesystem, just on general principles.  As
a friend from the large-scale enterprise storage world once put it,
"Performance is Job 2.  Robustness is Job #1."  (Of course, if you
want to put your fragile filesystem on a multi-million dollar
enterprise storage system such as an IBM Shark or an EMC Symmetrix
box, I'm sure IBM or EMC will be happy to sell you one.  :-)

							- Ted

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Theodore Ts'o wrote:
> On Mon, Jun 27, 2005 at 12:21:38PM +0300, Markus   T?rnqvist wrote:
> 
>>On Thu, Jun 23, 2005 at 11:34:50PM -0400, Horst von Brand wrote:
>>
>>>David Masover <ninja@slaphack.com> wrote:
>>
>>>>I think Hans (or someone) decided that when hardware stops working, it's
>>>>not the job of the FS to compensate, it's the job of lower layers, or
>>>>better, the job of the admin to replace the disk and restore from
>>>>backups.
>>>
>>>Handling other people's data this way is just reckless irresponsibility.
>>>Sure, you can get high performance if you just forego some of your basic
>>>responsibilities.
>>
>>Your honest-to-bog opinion is that the FS vendor is responsible for
>>the admin not taking backups or the hardware vendor shipping crap?
>>
>>*still trying to understand how that can be*
> 
> 
> Most Linux users are using PC-class hardware.  And Ted's First Law of
> PC-Class Hardware is: "Most of it is crap".  And then there's Ted's
> Second Law, "Too many system administrators don't do backups".

Not our problem anymore.  This is like saying that we should run all
filesystems in synchronous mode because some users will grab stuff out
of the computer without unmounting it -- even more a problem now that we
have SATA, which supports hot-swapping.

Too many system administrators don't do backups?  Some day the building
will burn down and they will wish they had.

> So it's a matter of matching the filesystem to the needs of the user.

My needs are a filesystem which doesn't assume I'm an idiot who can't
make backups.

> If you have a filesystem which is blazingly fast, but which at the
> slightest sign of trouble, trashes your data, versus one which is fast
> but perhaps not-so-fast as the other filesystem, but which is much
> more reliable, which would you choose?  

Hypothetically?  I choose the faster one.  Failures happen only rarely,
and when they happen, there's no telling how small or large they will
be, therefore I keep regular backups, and as soon as I see my hardware
starts to fail, I grab what I can off of it, merge that with the latest
backup, and buy new hardware.

> XFS has similar issues where it assumes that hardware has powerfail
> interrupts, and that the OS can use said powerfail interrupt to stop
> DMA's in its tracks on an power failure, so that you don't have
> garbage written to key filesystem data structures when the memory
> starts suffering from the dropping voltage on the power bus faster
> than the DMA engine or the disk drives.  So XFS is a great filesystem
> --- but you'd better be running it on a UPS, or on a system which has
> power fail interrupts and an OS that knows what to do.

XFS, Reiser3, and Reiser4 all pass the pull-the-plug test.  Maybe I just
haven't pushed them hard enough?

> Ext3, because
> it does physical block journalling, does not suffer from this problem.
> (Yes, Resierfs uses logical journalling as well, so it suffers from
> the same problem.)

I don't actually understand the difference here.  Are we talking about
metadata journalling vs. data journalling?

> So perhaps it's not the job of the FS vendor to be responsible for
> crap hardware or lazy sysadmins that don't do backups.

Good!  Glad we agree!

> But a system
> administrator who knows that he doesn't do backups frequently enough,
> or is running on cheap, crap hardware, would be wise to consider
> carefully which filesystem he/she wants to use given the systems
> configuration and his backup habits.  

Given a choice between changing filesystems or getting a Streamload
account, I choose Streamload.  (streamload.com)

Given a choice between reformatting or editing cron, I'd edit cron.

> Me, I'll go for the robust filesystem, just on general principles.  As
> a friend from the large-scale enterprise storage world once put it,
> "Performance is Job 2.  Robustness is Job #1."  (Of course, if you
> want to put your fragile filesystem on a multi-million dollar
> enterprise storage system such as an IBM Shark or an EMC Symmetrix
> box, I'm sure IBM or EMC will be happy to sell you one.  :-)

Job 1 is done by either that system or by good backups.  Don't overkill
Job 1 at the expense of Job 2.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsAY5XgHNmZLgCUhAQJq6g/9G2GEK0R1kujoqn3tjsTIbgC23iu8byYV
d6NHSrmmDV8035oo5iCgjSXbaTlIEyOQQqSB78fBC27eHbGctEZJIpSQfHrWlJu6
UJvvaEyl+sR/Mzq8sButKeSUv/T8RMCBKyhdfSgm28lDM/kt9OGZcrf1P1ChUSeK
ZnaToh4SAReRhGzq247o+qa2rW5IuIlpKeYGMZhiOB/tXGC3IDMtUOE9IBkn8KVd
TtDN4f74PuDJ9VX8K/tZ9DDtKebOJ1wKvIQ/BwyNXt5g36vwj2UwK6GkT2ZJ3bd6
2XLL24LpIBcVU16bk5WPkADRKe0W7S6AseUj7ggjAK0OL8z889vQ3NWJ/rmT/Wx7
OQi58QBwL89OZGSM2qZFJ09CLEH42GXD5jjTrKt4URudYzoIa45VZxD9YYhzeu5e
Gxxj4r2BM8bf/AHczCU/tEQwb6hfqxrq+SImWd1uIsozjeiwInHBC56x2qUDjFB/
kiyO1LVmVzrq+6frjPAg6n/i0zuPfF24SeeHX02UOx0jlNLkq2D4Qu1wDB5XlVTK
4xrN83Uy+dHSuDBvnf2tWTqmBju2YG6zmOl1xWKzi3/AVV6mfg1ur0gX0Q6GiZVa
5Nqp618Cs00ZtSk759eAPQRu7Ico0vTHhA0+rTUhApLYMboGaRN5W8OtQu2utHz8
ALX7pD3DrfE=
=wdYo
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Theodore Ts'o]

On Mon, Jun 27, 2005 at 10:19:01AM -0500, David Masover wrote:
> > XFS has similar issues where it assumes that hardware has powerfail
> > interrupts, and that the OS can use said powerfail interrupt to stop
> > DMA's in its tracks on an power failure, so that you don't have
> > garbage written to key filesystem data structures when the memory
> > starts suffering from the dropping voltage on the power bus faster
> > than the DMA engine or the disk drives.  So XFS is a great filesystem
> > --- but you'd better be running it on a UPS, or on a system which has
> > power fail interrupts and an OS that knows what to do.
> 
> XFS, Reiser3, and Reiser4 all pass the pull-the-plug test.  Maybe I just
> haven't pushed them hard enough?

Try doing huge amounts of metadata updates while doing the
pull-the-plug test.  The problem comes when the disk drive is writing
into the inode table or other filesystem metadata at the precise
moment when power goes down.  If the disk is quiescent, you won't see
the issue.   

XFS is known to have this problem --- the problem was first told to me
by an SGI engineer, and they solved this problem in hardware, by
adding a power fail interrupt and larger capacitors to the power
supply, and then modifying Irix to frantically run around shutting
down all outstanding DMA transfer during the grace period provided by
the enlarged capacitors.  Unfortunately, PC-class hardware don't have
power fail interrupts, so....

I have seen this problem reported on ext2 filesystems, so I know it
will happen on at least some, and probably most, PC-class hardware.
Fortunately switching to ext3 solves the problem, since metadata
updates are first written to the journal first, and the recovery
replays the full block, which papers over the problem.  Unfortunately,
a filesystem which uses logical journalling doesn't have a complete
copy of the metadata block in the log (only the logical changes are
recorded in the journal).  This is more compact, and will result in
better numbers in artificial benchmarks that emphasize metadata
updates (which is not the case in most real-world workloads in my
experience).  But it the drawback is that filesystems that journal
logical updates instead of physical blocks are vulnerable to this
particular lossage mode.

> Given a choice between changing filesystems or getting a Streamload
> account, I choose Streamload.  (streamload.com)

*If* you can afford the upload bandwidth to backup over the network,
and *if* you don't mind these gems in their legal T's and C's:

	Streamload cannot warrant and does not guarantee, and You
	should not expect, that all of Your private communications and
	other personal information will never be disclosed in ways not
	otherwise described in this Privacy Policy.

	As-Is and As-Available. Neither Streamload nor any User, or
	their respective agents, warrants that the service, Private
	Content or Public Content, or Your access thereto, will be
	uninterrupted or error free, and Streamload's services and
	both Public Content and Private Content are provided on an "as
	is, as available" basis. Streamload has the right to make
	changes to its services without notice to You. Neither
	Streamload nor any User warrants that Public Content or
	Private Content will be free of viruses or similar
	contamination or destructive features. You expressly agree to
	assume any and all risk as to the use, quality, performance,
	accuracy or completeness of any Private Content, Public
	Content, or Streamload's services.

And of course, while Streamload is pretty cheap for a minimal account,
if you have say, a several hundred gigabytes of data which is lost
when your filesystem implodes, they have you over a barrel and charge
$$$ in order for you to recover it.  But if you think it's this works
for you, please, go right ahead.  Just don't come whining to us
afterwards if you get screwed.

						- Ted

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Theodore Ts'o wrote:
> On Mon, Jun 27, 2005 at 10:19:01AM -0500, David Masover wrote:
[...]
>>Given a choice between changing filesystems or getting a Streamload
>>account, I choose Streamload.  (streamload.com)
> 
> 
> *If* you can afford the upload bandwidth to backup over the network,
> and *if* you don't mind these gems in their legal T's and C's:
> 
> 	Streamload cannot warrant and does not guarantee, and You
> 	should not expect, that all of Your private communications and
> 	other personal information will never be disclosed in ways not
> 	otherwise described in this Privacy Policy.

gpg.  Was in my upload script to begin with.  I keep my key written many
times on a single hidden CD.  So long as the isofs can be read, at least
one of the copies should be usable.

> 	As-Is and As-Available. Neither Streamload nor any User, or
> 	their respective agents, warrants that the service, Private
> 	Content or Public Content, or Your access thereto, will be
> 	uninterrupted or error free, and Streamload's services and
> 	both Public Content and Private Content are provided on an "as
> 	is, as available" basis. Streamload has the right to make
> 	changes to its services without notice to You. Neither

They don't have any billing information on me.  If they charge me for
something, I'll cancel my account.

> 	Streamload nor any User warrants that Public Content or
> 	Private Content will be free of viruses or similar
> 	contamination or destructive features. You expressly agree to
> 	assume any and all risk as to the use, quality, performance,
> 	accuracy or completeness of any Private Content, Public
> 	Content, or Streamload's services.

They are giving you unlimited storage for free.  Even with all of that,
it's a damn good deal.

> And of course, while Streamload is pretty cheap for a minimal account,
> if you have say, a several hundred gigabytes of data which is lost
> when your filesystem implodes, they have you over a barrel and charge
> $$$ in order for you to recover it.  But if you think it's this works
> for you, please, go right ahead.  Just don't come whining to us
> afterwards if you get screwed.

I don't have several hundred gigabytes of dynamic data, and neither does
the average desktop user.  The several hundred gigabytes of static data
I have would be backed up on DVDs anyway.

Now, if we're talking about a sysadmin, then of course you need a better
backup solution -- no matter what FS you use.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsBrxngHNmZLgCUhAQLihw//Qz+m971rPW90Fw9We2pqK49pstfsdk4Z
n7kimE7kptdcQX0ndnBa1UJ/1bTdIsaGrL+OFTkqLVKtNfitt8DuhdrbXlgm9gvJ
k8mwiopXIcfPA7BZ7vxzudttX/35tRfV2ubosagqWZjnpRgBLE3mEdDRCAJb1Z7+
YkynyekwzLPNo4qxWjhazDLqkBvGz4gntQrAwMoOYFlgpQZHlcDAABdRgd1rv+EA
paf/ZD90sGm5ZSodwURA8rUKIhRGqv+a5icJQ/PFWl8Cyjtg61w6toWjeK52tE+t
1AezDvjYy7JBItV3BlScgcCze6lKlAEHXDgjUZzKBllh4+h8b46IR6qjvpEk1YeX
jOeJpUwIv/YPtRgfGbUBgVyd5wBWqD4+W7a2qn+GIr6sQT176X+fNFzs6RKy+QlZ
Ty6yXGejFoydOhbg8zuhZvbEfJKuZCAv+1JkUijFZI6uVZj4Yychc04+qRybg+SQ
yY+52KbHyeeH3yQSQrGNa3ngEh84LJjoUpvYXGhQg+ex7kDuQAYlqdUAd7NIQIFj
LFvGzkVT/8SLAOi9D7lXB0g52+W4Atl7a3nYpFhIkm2AthNjUkSFhZxyu6wmhFrj
5SPReftDmp4MVQeC4FVZfB72muY4zcO/WYSVmpGn2P89lHEgD4uw8gqQ0XAi5xe6
Gwdb2XcczCo=
=+R+m
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Theodore Ts'o]

On Mon, Jun 27, 2005 at 04:12:38PM -0500, David Masover wrote:
> > 	Streamload cannot warrant and does not guarantee, and You
> > 	should not expect, that all of Your private communications and
> > 	other personal information will never be disclosed in ways not
> > 	otherwise described in this Privacy Policy.
> 
> gpg.  Was in my upload script to begin with.  I keep my key written many
> times on a single hidden CD.  So long as the isofs can be read, at least
> one of the copies should be usable.
> 
> They don't have any billing information on me.  If they charge me for
> something, I'll cancel my account.

If you are using the free service, and are encrypting the data, you
are explicitly violating their terms of service, and they can delete
your data at any time, once they notice.

						- Ted

Re: reiser4 plugins

[Dmitry Torokhov]

On 6/30/05, Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, Jun 27, 2005 at 04:12:38PM -0500, David Masover wrote:
> > >     Streamload cannot warrant and does not guarantee, and You
> > >     should not expect, that all of Your private communications and
> > >     other personal information will never be disclosed in ways not
> > >     otherwise described in this Privacy Policy.
> >
> > gpg.  Was in my upload script to begin with.  I keep my key written many
> > times on a single hidden CD.  So long as the isofs can be read, at least
> > one of the copies should be usable.
> >
> > They don't have any billing information on me.  If they charge me for
> > something, I'll cancel my account.
> 
> If you are using the free service, and are encrypting the data, you
> are explicitly violating their terms of service, and they can delete
> your data at any time, once they notice.
> 

Does not look like it:

3c. No encryption and/or steganography for the purpose of
circumventing Streamload's rules.
... For example, if you'd like to encrypt something for an extra sense
of security and privacy, please feel free to do so. However, when
these tools are used solely for the purpose of circumventing
Streamload's rules, as determined by the sole discretion of
Streamload, the files will be deleted.

-- 
Dmitry

backup (was Re: reiser4 plugins)

[David Masover]

Dmitry Torokhov wrote:
> On 6/30/05, Theodore Ts'o <tytso@mit.edu> wrote:
> 
>>On Mon, Jun 27, 2005 at 04:12:38PM -0500, David Masover wrote:
>>
>>>>    Streamload cannot warrant and does not guarantee, and You
>>>>    should not expect, that all of Your private communications and
>>>>    other personal information will never be disclosed in ways not
>>>>    otherwise described in this Privacy Policy.
>>>
>>>gpg.  Was in my upload script to begin with.  I keep my key written many
>>>times on a single hidden CD.  So long as the isofs can be read, at least
>>>one of the copies should be usable.
>>>
>>>They don't have any billing information on me.  If they charge me for
>>>something, I'll cancel my account.
>>
>>If you are using the free service, and are encrypting the data, you
>>are explicitly violating their terms of service, and they can delete
>>your data at any time, once they notice.
>>
> 
> 
> Does not look like it:
> 
> 3c. No encryption and/or steganography for the purpose of
> circumventing Streamload's rules.
> ... For example, if you'd like to encrypt something for an extra sense
> of security and privacy, please feel free to do so. However, when
> these tools are used solely for the purpose of circumventing
> Streamload's rules, as determined by the sole discretion of
> Streamload, the files will be deleted.

Trasnlation:  They really, honestly, have no way at all of knowing if my 
GPG-encrypted 450 meg tarball violates their terms.  They have legally 
given themselves the right to delete it just because they think it might 
violate their terms, but it wouldn't make sense for them to delete a 
file called "backup.tar.bz2.gpg" when there are no doubt a lot of other 
encrypted files on their servers.  It wouldn't make sense because they 
would either have to delete such files completely at random, or they'd 
have to delete all of them at once.

Further translation:  This was probably put here to keep the lawyers 
happy.  They don't want to be held liable for illegal stuff on their 
servers, for example, so they have at least spelled out, in legalese, 
that illegal stuff (and probably a few other categories) is not allowed 
on their servers, and that encyrpting it doesn't magically make it legal.

This kind of stuff makes lawyers happy, even if it *practically* does 
nothing at all.

Regardless, I feel reasonably safe with a backup there.  If I ever start 
to feel unsafe, I can always back up the most critical stuff to gmail, 
and create a few permanent copies on DVDs.  But I feel considerably 
safer with Streamload than with Gmail, because using Gmail for my own 
custom backup app directly violates their terms of service.

I think we should officially kill this particular thread.

Re: backup (was Re: reiser4 plugins)

[Theodore Ts'o]

Rule #3 from http://www.streamload.com/About/Legal_eng.asp?page=id73#
is pretty clear about what applies if you have a trial account (which
seems to be what you have since you say you'll cancel your account if
they charge you anything):

  3. Do not circumvent Freeloader download restrictions.
  Any attempt to circumvent Freeloader trial account restrictions will
  result in a permanent banishment from the Streamload community. This
  includes (3a) files with an invalid or disguised file format; (3b)
  encryption; (3c) steganography and/or (3d) creating multiple
  freeloader accounts.

You can interpret that whatever way you like, but if you're that
cavalier with your data, hey, I'm not sure _I'd_ trust your judgement
about which filesystem to trust.  :-)

						- Ted

Re: backup (was Re: reiser4 plugins)

[David Masover]

Theodore Ts'o wrote:
> Rule #3 from http://www.streamload.com/About/Legal_eng.asp?page=id73#
> is pretty clear about what applies if you have a trial account (which
> seems to be what you have since you say you'll cancel your account if
> they charge you anything):
> 
>   3. Do not circumvent Freeloader download restrictions.
>   Any attempt to circumvent Freeloader trial account restrictions will
>   result in a permanent banishment from the Streamload community. This
>   includes (3a) files with an invalid or disguised file format; (3b)
>   encryption; (3c) steganography and/or (3d) creating multiple
>   freeloader accounts.
> 
> You can interpret that whatever way you like, but if you're that
> cavalier with your data, hey, I'm not sure _I'd_ trust your judgement
> about which filesystem to trust.  :-)

Streamload is one possibility.  There are others -- gmail and such. 
This is just what can be had *for*free*.

For that matter, the uploading process doesn't require a user to agree 
to those terms at all.  There's a module in CPAN that allows an upload, 
and never asks me to agree to any terms, nor does it ask me for an 
account password.

I don't have multiple accounts, I don't *use* the encryption to attempt 
to circumvent the download restrictions, especially considering that 
said download restrictions don't even come into play until I need a 
restore.  By the time I do need a restore, I'm willing to pay money, as 
100 megs a month is certainly not going to be enough.

Re: reiser4 plugins

[Hans Reiser]

Dmitry Torokhov wrote:

>
>>If you are using the free service, and are encrypting the data, you
>>are explicitly violating their terms of service, and they can delete
>>your data at any time, once they notice.
>>
>>    
>>
>
>Does not look like it:
>
>3c. No encryption and/or steganography for the purpose of
>circumventing Streamload's rules.
>... For example, if you'd like to encrypt something for an extra sense
>of security and privacy, please feel free to do so. However, when
>these tools are used solely for the purpose of circumventing
>Streamload's rules, as determined by the sole discretion of
>Streamload, the files will be deleted.
>
>  
>
Oops, sorry, got that wrong, ok, it is more that they provide this
service and the terms are at their convenience not anyone else's.

Re: reiser4 plugins

[Hans Reiser]

Wow.  Good that you noticed it.  It sounds like they have the business
model of an intelligence agency.....

Hans

Theodore Ts'o wrote:

>If you are using the free service, and are encrypting the data, you
>are explicitly violating their terms of service, and they can delete
>your data at any time, once they notice.
>
>						- Ted
>  
>

Re: reiser4 plugins

[Hans Reiser]

Steve, there is a remark about XFS below which you are going to be more
expert on.

Theodore Ts'o wrote:

>Most Linux users are using PC-class hardware.  And Ted's First Law of
>PC-Class Hardware is: "Most of it is crap".  And then there's Ted's
>Second Law, "Too many system administrators don't do backups".  This
>is because most system admins are users who've never been trained to
>be a sysadmin, or who haven't (yet) had weeks or months of works
>disappear after a hardware failure.  
>  
>
The above I agree with.  Your words below though are irresponsible.

I get users who tell me that ext* is crap and fsck.ext2 corrupted their
filesystem and thats why they use ReiserFS.  A difference between us is
that I tell them that with all the major linux filesystems (I include
XFS and JFS in this)  it is by this time far more likely to be hardware
that caused corruption than the filesystem software, whereas I guess you
tell them something else.

Describing us as trashing data at the slightest sign of trouble is
irresponsible.  We do block journaling not logical journaling, but there
is absolutely nothing inherently wrong with logical journaling, and no
reason why it could not be just as robust if that is what XFS uses, so I
suspect the rest of your remarks about XFS are unlikely to be sound.

I would say much more, but I am trying to avoid flames this week.

Ted, please try to consider that maybe your competitors also do a fairly
good job at the mundane but important aspects of filesystem work, and
lets all avoid FUDing each others work.  Both ext* and XFS are excellent
filesystems, and Linux is lucky to have 3 of the 4 best filesystems
available for it.

It will be interesting to see though whether Dominic Giampaolo beats all
three of us in the next five years.  He probably won't have the
performance, but he sure might have the semantic features, and he is
very bright.  If MS ever makes good use of the talent they have been
hiring, we'll have a hard time there also.  We may get lucky on that
though.....;-)

>So it's a matter of matching the filesystem to the needs of the user.
>If you have a filesystem which is blazingly fast, but which at the
>slightest sign of trouble, trashes your data, versus one which is fast
>but perhaps not-so-fast as the other filesystem, but which is much
>more reliable, which would you choose?  
>
>XFS has similar issues where it assumes that hardware has powerfail
>interrupts, and that the OS can use said powerfail interrupt to stop
>DMA's in its tracks on an power failure, so that you don't have
>garbage written to key filesystem data structures when the memory
>starts suffering from the dropping voltage on the power bus faster
>than the DMA engine or the disk drives.  So XFS is a great filesystem
>--- but you'd better be running it on a UPS, or on a system which has
>power fail interrupts and an OS that knows what to do.  Ext3, because
>it does physical block journalling, does not suffer from this problem.
>(Yes, Resierfs uses logical journalling as well, so it suffers from
>the same problem.)
>
>So perhaps it's not the job of the FS vendor to be responsible for
>crap hardware or lazy sysadmins that don't do backups.  But a system
>administrator who knows that he doesn't do backups frequently enough,
>or is running on cheap, crap hardware, would be wise to consider
>carefully which filesystem he/she wants to use given the systems
>configuration and his backup habits.  
>
>Me, I'll go for the robust filesystem, just on general principles.  As
>a friend from the large-scale enterprise storage world once put it,
>"Performance is Job 2.  Robustness is Job #1."  (Of course, if you
>want to put your fragile filesystem on a multi-million dollar
>enterprise storage system such as an IBM Shark or an EMC Symmetrix
>box, I'm sure IBM or EMC will be happy to sell you one.  :-)
>
>							- Ted
>
>
>  
>

Re: reiser4 plugins

[Steve Lord]

Hans Reiser wrote:
> Steve, there is a remark about XFS below which you are going to be more
> expert on.
> 
> Theodore Ts'o wrote:
> 

>>
>>XFS has similar issues where it assumes that hardware has powerfail
>>interrupts, and that the OS can use said powerfail interrupt to stop
>>DMA's in its tracks on an power failure, so that you don't have
>>garbage written to key filesystem data structures when the memory
>>starts suffering from the dropping voltage on the power bus faster
>>than the DMA engine or the disk drives.  So XFS is a great filesystem
>>--- but you'd better be running it on a UPS, or on a system which has
>>power fail interrupts and an OS that knows what to do.  Ext3, because
>>it does physical block journalling, does not suffer from this problem.
>>(Yes, Resierfs uses logical journalling as well, so it suffers from
>>the same problem.)
>>

I presume Ted is referring to problems guaranteeing the integrity of
the journal at recovery time. I am coming into this without all the
available context, so I may be barking up the wrong tree.... In
particular, I am not sure how journaling whole blocks protects
you from this.

The xfs journal protects itself against partial writes, to a certain
degree. The header of a journal write (inside a 512 byte sector)
contains an array of words which are swapped out from the start of
each following 512 byte sector of the journal write. The following
sectors then each have the log sequence number (LSN) of the write inserted
in place of that data.

During recovery, we find the most recent LSN via a binary chop
search, this gives us an associated tail LSN. A scan backwards
from the head LSN is then done - this covers the total possible
amount of in flight data (maximum log buffers x maximum log buffer
size). If any of the sectors has the wrong LSN in the first word,
then it an all following data is discarded from replay. Of course,
we will also not replay any journal entry for which we do not find
the transaction commit record.

Now, this protects against some failure cases, it assumes that
sector writes are atomic, they either happen or they do not
happen. If sector writes are not atomic and one end can be
good with the other is bad, then a partial sector is possibly
going to get replayed. There have been discussions about doing
this with the head and tail of each sector, or using a checksum
instead.

XFS on linux has had power cycle crash testing, but there is no
way you can cover all possible hardware configurations, and I
seem to recall some hardware never recovered from this testing,
by that I mean the PC did not survive the continual power cycling
and went up in smoke.

Steve

Re: reiser4 plugins

[Theodore Ts'o]

On Mon, Jun 27, 2005 at 03:18:30PM -0500, Steve Lord wrote:
> I presume Ted is referring to problems guaranteeing the integrity of
> the journal at recovery time. I am coming into this without all the
> available context, so I may be barking up the wrong tree.... In
> particular, I am not sure how journaling whole blocks protects
> you from this.

Actually, I was talking about the problem what happens when power
fails while DMA'ing to the disk, and memory, which is more sensitive
to voltage drops than the rest of the system, starts sending garbage
to the bus, which the disk then faithfully writes to the inode table.

As I recall, you were the one who told me about this problem, and how
it was fixed in Irix by using a powerfail interrupt to abort DMA
transfers, as well as giving me a program which tests for this
condition (basically it writes known test pattern to the disk, and
then you do an unclean shutdown, and you look to see if garbage is
written to the disk instead of one of the known test patterns).  If it
wasn't you, it must have been Jim Mostek --- but I could have sworn it
was you.....

						- Ted

Re: reiser4 plugins

[Hans Reiser]

Theodore Ts'o wrote:

>On Mon, Jun 27, 2005 at 03:18:30PM -0500, Steve Lord wrote:
>  
>
>>I presume Ted is referring to problems guaranteeing the integrity of
>>the journal at recovery time. I am coming into this without all the
>>available context, so I may be barking up the wrong tree.... In
>>particular, I am not sure how journaling whole blocks protects
>>you from this.
>>    
>>
>
>Actually, I was talking about the problem what happens when power
>fails while DMA'ing to the disk, and memory, which is more sensitive
>to voltage drops than the rest of the system, starts sending garbage
>to the bus, which the disk then faithfully writes to the inode table.
>
>As I recall, you were the one who told me about this problem, and how
>it was fixed in Irix by using a powerfail interrupt to abort DMA
>transfers, as well as giving me a program which tests for this
>condition (basically it writes known test pattern to the disk, and
>then you do an unclean shutdown, and you look to see if garbage is
>written to the disk instead of one of the known test patterns).  If it
>wasn't you, it must have been Jim Mostek --- but I could have sworn it
>was you.....
>
>						- Ted
>
>
>  
>
Maybe a more positive approach would be to say, hey, I got this program
from the XFS guys that tests for DMA problems on power failure, maybe
all the Linux filesystem developers should give it a try?

Shall we try that style of interaction?

Ted, if this corruption hits an ext3 directory structure, and that
corrupted directory structure commits, what happens?  With ReiserFS
(Chris please comment here) it would have to corrupt the directory in
the journal, return successful IO, then after that point not corrupt the
commit block and have enough power to write the commit block.  How is
that different for XFS or ext3?

Hans

Re: reiser4 plugins

[Steve Lord]

Theodore Ts'o wrote:
> On Mon, Jun 27, 2005 at 03:18:30PM -0500, Steve Lord wrote:
> 
>>I presume Ted is referring to problems guaranteeing the integrity of
>>the journal at recovery time. I am coming into this without all the
>>available context, so I may be barking up the wrong tree.... In
>>particular, I am not sure how journaling whole blocks protects
>>you from this.
> 
> 
> Actually, I was talking about the problem what happens when power
> fails while DMA'ing to the disk, and memory, which is more sensitive
> to voltage drops than the rest of the system, starts sending garbage
> to the bus, which the disk then faithfully writes to the inode table.
> 
> As I recall, you were the one who told me about this problem, and how
> it was fixed in Irix by using a powerfail interrupt to abort DMA
> transfers, as well as giving me a program which tests for this
> condition (basically it writes known test pattern to the disk, and
> then you do an unclean shutdown, and you look to see if garbage is
> written to the disk instead of one of the known test patterns).  If it
> wasn't you, it must have been Jim Mostek --- but I could have sworn it
> was you.....
> 
> 						- Ted
> 

Your memory is better than mine, not sure about the test program, but
there was at one point a scenario like that on Irix, and I quite
probably did mention it to you. That was certainly not PC hardware,
but more likely a large Irix system with multiple power busses spread
around a large amount of hardware. I presume you are saying that the fact
that ext3 journals complete inode blocks rather that subsections of inodes
means that, should a similar trashing of metadata occur during power down,
a journal replay would fix it?

I see no way short of hardware fixes of avoiding the general problem of
a system failing in an ugly manner like this. Unless you write everything
to disk twice (i.e. journal all data), you can still end up with a
legitimate set of metadata, and the master copy of your employee
database full of nasty little bits of corruption.

Steve

Re: reiser4 plugins

[Prakash Punnoor]

[-- Attachment #1: Type: text/plain, Size: 3268 bytes --]

Steve Lord schrieb:

> I see no way short of hardware fixes of avoiding the general problem of
> a system failing in an ugly manner like this. Unless you write everything
> to disk twice (i.e. journal all data), you can still end up with a
> legitimate set of metadata, and the master copy of your employee
> database full of nasty little bits of corruption.

If I as a user may add my own little experience with different fs on my
computer, where I tend to use kernels or features that cause my machine to
(hard) lock up:

2.4.2x kernel and reiserfs (V3): This was the only one which really managed to
smoke my partition...

2.6.x-test-y till 2.6.x stable kernel and reiserfs (V3): I did a short
intermezzo with xfs but at that time it felt too slow to bear, so I gave
reiserfs another chance. And here it was very robust. No lock-up did serious
damage to the data. Only currently accessed files were partly overwritten with
@^@^@^ and so on...

After a year or so of heavy usage the formerly fast reiserfs partition became
more and more slow till I got sick of it. So I tried JFS and forced a hard
lock. Data was OK, but (probably error in gentoo scripts) fsck wasn't
automatically called (or failed, I don't remember) and I had to do it by hand,
which was a major annoyance - I thought then.

So I gave ext3 a try. Very robust, but at the same time slooow. I couldn't
bear it after some months. So I gave xfs another try. Yes, now it felt much
better. Still not that fast as reiserfs, IIRC, but better than the first time
I tried. I am still having xfs on / and it works pretty well, and is rather
robust against hard locks with about the same amount of data losing as
reiserfs. But what annoys me very much, is that I have to run xfs_repair by
hand and by booting from another partition. Even after a hard lock, the
partition mounts w/o problems and everything seems OK, but it only seems like
that. In fact after some hours/days of use, you'll notice oddities, like files
or directories which cannot be removed and things like that. After running
xfs_repair everything is back in order.

In between I gave an alpha (or rather several alphas) of reiser4 a try - but
not on /, just on /usr. Well, I wouldn't say it was extraordinary fast. In
fact it felt slower than reiserfs V3, but much more space efficient. And to my
surprise it was very robust as well. Hard-locks were no problem. Only annoying
then was, that unmounting regularly produced oops but later versions corrected
that. But nevertheless it didn't survive, as like V3, with time V4 became
slower and slower. In this case no year was needed, but just one month or
alike. So end of test...but in fact I'll give V4 another go in the near future.


All in all I can say that every fs I tested was able to not smoke all of my
data, even using an instable machine - but only ext3, reiser v3 and v4 were
non-annoying. But xfs is majorly annoying in that respect. I hope that new
versions will be able to keep consistency w/o having to run xfs_repair every
time after a lock-up...

But what I don't understand is, that sometimes even files, which were only
opened for reading, got overwritten with @^@^@ after a lock-up. I don't
understand the logics here, how this could happen.

Thx for your time,

Prakash

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Hans Reiser]

Prakash Punnoor wrote:

>
>. But nevertheless it didn't survive, as like V3, with time V4 became
>slower and slower. In this case no year was needed, but just one month or
>alike. So end of test...but in fact I'll give V4 another go in the near future.
>  
>
Interesting that it got slower with time.  It sounds like our online
repacker is much needed.  It will be a priority for after the kernel merge.

Re: reiser4 plugins

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

On Mon, 2005-06-27 at 17:40 -0700, Hans Reiser wrote:
> Prakash Punnoor wrote:
> 
> >
> >. But nevertheless it didn't survive, as like V3, with time V4 became
> >slower and slower. In this case no year was needed, but just one month or
> >alike. So end of test...but in fact I'll give V4 another go in the near future.
> >  
> >
> Interesting that it got slower with time.  It sounds like our online
> repacker is much needed.  It will be a priority for after the kernel merge.

I told y'all that last year on the reiserfs-list, with benchmarks :-)
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Jim Crilly]

On 06/28/05 01:06:54AM +0200, Prakash Punnoor wrote:
> 
> So I gave ext3 a try. Very robust, but at the same time slooow. I couldn't
> bear it after some months. So I gave xfs another try. Yes, now it felt much
> better. Still not that fast as reiserfs, IIRC, but better than the first time
> I tried. I am still having xfs on / and it works pretty well, and is rather
> robust against hard locks with about the same amount of data losing as
> reiserfs. But what annoys me very much, is that I have to run xfs_repair by
> hand and by booting from another partition. Even after a hard lock, the
> partition mounts w/o problems and everything seems OK, but it only seems like
> that. In fact after some hours/days of use, you'll notice oddities, like files
> or directories which cannot be removed and things like that. After running
> xfs_repair everything is back in order.

I don't know what was going on with your systems, but I've been using XFS
since the original 1.0 Linux release from SGI and I'd guess that I've had to run
xfs_repair less than 10 times and most of them were on Alpha and Sparc64
before issues with those arches got ironed out.

Right now I have XFS on both Alpha and Sparc64 and I haven't had
any issues on either box. Infact the Sparc64 box's XFS filesystem was
converted from reiser3 because the some part of the filesystem got
mysteriously corrupted in such a way that reiserfsck and the reiser3 driver
both thought it was fine but accessing a certain file would cause a lockup.

I really hope the reiser4 userland tools are a lot better than the reiser3
tools, that's an area that reiserfs has lagged behind hugely IMO.

> 
> In between I gave an alpha (or rather several alphas) of reiser4 a try - but
> not on /, just on /usr. Well, I wouldn't say it was extraordinary fast. In
> fact it felt slower than reiserfs V3, but much more space efficient. And to my
> surprise it was very robust as well. Hard-locks were no problem. Only annoying
> then was, that unmounting regularly produced oops but later versions corrected
> that. But nevertheless it didn't survive, as like V3, with time V4 became
> slower and slower. In this case no year was needed, but just one month or
> alike. So end of test...but in fact I'll give V4 another go in the near future.
> 
> 
> All in all I can say that every fs I tested was able to not smoke all of my
> data, even using an instable machine - but only ext3, reiser v3 and v4 were
> non-annoying. But xfs is majorly annoying in that respect. I hope that new
> versions will be able to keep consistency w/o having to run xfs_repair every
> time after a lock-up...
> 
> But what I don't understand is, that sometimes even files, which were only
> opened for reading, got overwritten with @^@^@ after a lock-up. I don't
> understand the logics here, how this could happen.
> 
> Thx for your time,
> 
> Prakash

Jim.

Re: reiser4 plugins

[Prakash Punnoor]

[-- Attachment #1: Type: text/plain, Size: 1542 bytes --]

Jim Crilly schrieb:
> On 06/28/05 01:06:54AM +0200, Prakash Punnoor wrote:
> 
>>So I gave ext3 a try. Very robust, but at the same time slooow. I couldn't
>>bear it after some months. So I gave xfs another try. Yes, now it felt much
>>better. Still not that fast as reiserfs, IIRC, but better than the first time
>>I tried. I am still having xfs on / and it works pretty well, and is rather
>>robust against hard locks with about the same amount of data losing as
>>reiserfs. But what annoys me very much, is that I have to run xfs_repair by
>>hand and by booting from another partition. Even after a hard lock, the
>>partition mounts w/o problems and everything seems OK, but it only seems like
>>that. In fact after some hours/days of use, you'll notice oddities, like files
>>or directories which cannot be removed and things like that. After running
>>xfs_repair everything is back in order.
> 
> 
> I don't know what was going on with your systems, but I've been using XFS
> since the original 1.0 Linux release from SGI and I'd guess that I've had to run
> xfs_repair less than 10 times and most of them were on Alpha and Sparc64
> before issues with those arches got ironed out.

Perhaps it is due to the fact that I use xfs on software RAID-0 and both HDs
have 8MB cache write-back enabled? So, all in all 16MB needs to be commited
on/before lock-up, maybe too much for xfs? (This situation was no prob for
ext3, though. Thinking again, I never used reiser V3 or V4 on the RAID-0, so
my comparison might not have been fair.)

Prakash

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Jim Crilly]

On 06/28/05 06:03:25AM +0200, Prakash Punnoor wrote:
> Jim Crilly schrieb:
> > On 06/28/05 01:06:54AM +0200, Prakash Punnoor wrote:
> > 
> >>So I gave ext3 a try. Very robust, but at the same time slooow. I couldn't
> >>bear it after some months. So I gave xfs another try. Yes, now it felt much
> >>better. Still not that fast as reiserfs, IIRC, but better than the first time
> >>I tried. I am still having xfs on / and it works pretty well, and is rather
> >>robust against hard locks with about the same amount of data losing as
> >>reiserfs. But what annoys me very much, is that I have to run xfs_repair by
> >>hand and by booting from another partition. Even after a hard lock, the
> >>partition mounts w/o problems and everything seems OK, but it only seems like
> >>that. In fact after some hours/days of use, you'll notice oddities, like files
> >>or directories which cannot be removed and things like that. After running
> >>xfs_repair everything is back in order.
> > 
> > 
> > I don't know what was going on with your systems, but I've been using XFS
> > since the original 1.0 Linux release from SGI and I'd guess that I've had to run
> > xfs_repair less than 10 times and most of them were on Alpha and Sparc64
> > before issues with those arches got ironed out.
> 
> Perhaps it is due to the fact that I use xfs on software RAID-0 and both HDs
> have 8MB cache write-back enabled? So, all in all 16MB needs to be commited
> on/before lock-up, maybe too much for xfs? (This situation was no prob for
> ext3, though. Thinking again, I never used reiser V3 or V4 on the RAID-0, so
> my comparison might not have been fair.)

Maybe, I've never used XFS on software RAID-0. The Sparc64 I mentioned has
it's XFS filesystem on software RAID-1 and that's introduced no problems, but 
I've seen way too many drives die to risk running RAID-0.

> 
> Prakash

Jim.

RE: reiser4 plugins

[Al Boldi]

Hans Reiser wrote:
> Steve, there is a remark about XFS below which you are going to be 
> more expert on.
> 
> Theodore Ts'o wrote:
> 
>>
>>XFS has similar issues where it assumes that hardware has powerfail 
>>interrupts, and that the OS can use said powerfail interrupt to stop 
>>DMA's in its tracks on an power failure, so that you don't have 
>>garbage written to key filesystem data structures when the memory 
>>starts suffering from the dropping voltage on the power bus faster 
>>than the DMA engine or the disk drives.  So XFS is a great filesystem
>>--- but you'd better be running it on a UPS, or on a system which has 
>>power fail interrupts and an OS that knows what to do.  Ext3, because 
>>it does physical block journalling, does not suffer from this problem.
>>(Yes, Resierfs uses logical journalling as well, so it suffers from 
>>the same problem.)
>>

True now, not so around 2.4.20 when XFS was rock-solid. I think they tried
to improve on performance and broke something. I wish they would fix that
because it forced me back to ext3, as in consistency over performance any
time.

Re: reiser4 plugins

[Theodore Ts'o]

On Mon, Jun 27, 2005 at 12:46:23PM -0700, Hans Reiser wrote:
> A difference between us is
> that I tell them that with all the major linux filesystems (I include
> XFS and JFS in this)  it is by this time far more likely to be hardware
> that caused corruption than the filesystem software, whereas I guess you
> tell them something else.

Oh, I agree with this, and I do tell people that.  The question though
is how the filesystem recovers from said hardware-caused corruption
once it does happen.  You've admitted that reiserfs3 has less than
optimal recovery characteristics from hardware-induced corruptions if
said filesystem contains reiserfs filesystem images; that would be an
example of a filesystem not being as robust as it could be.  (It'll be
interesting to see if SuSE will support reiserfsv3 in combination with
the Xen hypervisor or other virtualization systems, which makes use of
filesystem images.)  Another example would be DMA'ing garbage into the
hard drive after a power failure --- how does a filesystem respond to
this eventuality?

You probably hear more stories people who got unlucky with
hardware-induced corruptions with ext2/3, and I probably hear more
from users who have sworn off of reiserfs just because are sample sets
are somewhat biased.  Such are the dangers of relying on anecdotal
evidence.

However, logically speaking, if a filesystem is designed such that in
certain cases, the fsck program has to brute-force search every single
disk block looking for data structures that _look_ like they might be
part of the filesystem data, well, that's always going to be more
error prone than one where the filesystem metadata is in
easily-predicted locations.  It sounds like you've added some more
checks in reiser4, and that's definitely a good thing.  Time will tell
whether they are sufficient or not.

Regards,

						- Ted

Re: reiser4 merging action list

[Hans Reiser]

Andrew asked me to put together a list of things that need to be done
before merging:

    * VFS will dispatch directly to the method of the plugin for the
*_operations methods.  This requires duplicating to all plugin methods
the common code currently used by all reiser4 plugins for a given
method.  It has the desirable side effect of making the methods more
fully self-contained, which is somethng I had wanted two years ago and
was a little sad to not get, and the cost of duplicating some code. 
Since not all plugin methods are *_operations, it means we have two
structures with duplicated data, and duplicate data that must be in sync
at all times is classical badness in programming technique (see Codd and
normalization).                                 vs owns this task

    * review all sparse complaints, and revise as appropriate. 

    * panic and code beauty: everyone agrees that having function, file,
and line added to reiser4_panic output hurts nothing (I hope).  Everyone
agrees that restarting the machine without an error message seems like a
useless option to allow.   Much else was argued, not sure if anything
was a consensus view.  Various detail improvements were suggested by
Pecca, and I agreed with half of them.


   * metafiles should be disabled until we can present code that works
right.  Half the list thinks we cannot solve the cycles problem ever. 
Disable metafiles and postpone problem until working code, or the
failure to produce it, makes it possible to do more than rant at each
other.  This is currently already done in the -mm patches, but is
mentioned lest someone think it forgotten.

   * update the locking documentation

Probably I forget something.

Best,

Hans

Re: reiser4 merging action list

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
> 
> Andrew asked me to put together a list of things that need to be done
> before merging:

Thanks.

As I said to Hans, if we can get a list of bullet-point actions nailed down
and agreed to then we have an uncontroversial path to happiness and a
merge.  Let's get down and concentrate on technical specifics.

Hans, please maintain this list and republish it as we work through things.

>     * VFS will dispatch directly to the method of the plugin for the
> *_operations methods.  This requires duplicating to all plugin methods
> the common code currently used by all reiser4 plugins for a given
> method.  It has the desirable side effect of making the methods more
> fully self-contained, which is somethng I had wanted two years ago and
> was a little sad to not get, and the cost of duplicating some code. 
> Since not all plugin methods are *_operations, it means we have two
> structures with duplicated data, and duplicate data that must be in sync
> at all times is classical badness in programming technique (see Codd and
> normalization).                                 vs owns this task
> 
>     * review all sparse complaints, and revise as appropriate. 
> 
>     * panic and code beauty: everyone agrees that having function, file,
> and line added to reiser4_panic output hurts nothing (I hope).  Everyone
> agrees that restarting the machine without an error message seems like a
> useless option to allow.   Much else was argued, not sure if anything
> was a consensus view.  Various detail improvements were suggested by
> Pecca, and I agreed with half of them.
> 
> 
>    * metafiles should be disabled until we can present code that works
> right.  Half the list thinks we cannot solve the cycles problem ever. 
> Disable metafiles and postpone problem until working code, or the
> failure to produce it, makes it possible to do more than rant at each
> other.  This is currently already done in the -mm patches, but is
> mentioned lest someone think it forgotten.
> 
>    * update the locking documentation
> 

There's also the custom list, hash and debug code.  We should either

a) remove them or

b) generify them and submit as standalone works or

c) justify them as custom-to-reiser4 and leave them as-is.

Re: reiser4 merging action list

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>Andrew asked me to put together a list of things that need to be done
>>before merging:
>>    
>>
>
>Thanks.
>
>As I said to Hans, if we can get a list of bullet-point actions nailed down
>and agreed to then we have an uncontroversial path to happiness and a
>merge.  Let's get down and concentrate on technical specifics.
>
>Hans, please maintain this list and republish it as we work through things.
>
>  
>
>>    * VFS will dispatch directly to the method of the plugin for the
>>*_operations methods.  This requires duplicating to all plugin methods
>>the common code currently used by all reiser4 plugins for a given
>>method.  It has the desirable side effect of making the methods more
>>fully self-contained, which is somethng I had wanted two years ago and
>>was a little sad to not get, and the cost of duplicating some code. 
>>Since not all plugin methods are *_operations, it means we have two
>>structures with duplicated data, and duplicate data that must be in sync
>>at all times is classical badness in programming technique (see Codd and
>>normalization).                                 vs owns this task
>>
>>    * review all sparse complaints, and revise as appropriate. 
>>
>>    * panic and code beauty: everyone agrees that having function, file,
>>and line added to reiser4_panic output hurts nothing (I hope).  Everyone
>>agrees that restarting the machine without an error message seems like a
>>useless option to allow.   Much else was argued, not sure if anything
>>was a consensus view.  Various detail improvements were suggested by
>>Pecca, and I agreed with half of them.
>>
>>
>>   * metafiles should be disabled until we can present code that works
>>right.  Half the list thinks we cannot solve the cycles problem ever. 
>>Disable metafiles and postpone problem until working code, or the
>>failure to produce it, makes it possible to do more than rant at each
>>other.  This is currently already done in the -mm patches, but is
>>mentioned lest someone think it forgotten.
>>
>>   * update the locking documentation
>>
>>    
>>
>
>There's also the custom list, hash and debug code.  We should either
>
>a) remove them or
>
>b) generify them and submit as standalone works or
>
>c) justify them as custom-to-reiser4 and leave them as-is.
>
>
>
>
>  
>
either b) or c) is ok with me for the list code.  The debug code should
be c) I think.

Probably vs can offer a more detailed and accurate opinion,

Hans

Re: reiser4 merging action list

[Pekka Enberg]

Andrew Morton wrote:
> > There's also the custom list, hash and debug code.  We should either
> >
> > a) remove them or
> >
> > b) generify them and submit as standalone works or
> >
> > c) justify them as custom-to-reiser4 and leave them as-is.

On 6/29/05, Hans Reiser <reiser@namesys.com> wrote:
> either b) or c) is ok with me for the list code.  The debug code should
> be c) I think.
> 
> Probably vs can offer a more detailed and accurate opinion,

I completely agree that the current state of the generic hashing
facilities is somewhat poor but I fail to see why you can't use
<linux/list.h>.

As for the debugging code, I would love to see that turned into
something generic (every subsystem has their own now) but it is
definitely not something that should stop you from merging.

                                Pekka

Re: reiser4 merging action list

[Hans Reiser]

Pekka Enberg wrote:

>Andrew Morton wrote:
>  
>
>>>There's also the custom list, hash and debug code.  We should either
>>>
>>>a) remove them or
>>>
>>>b) generify them and submit as standalone works or
>>>
>>>c) justify them as custom-to-reiser4 and leave them as-is.
>>>      
>>>
>
>On 6/29/05, Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>either b) or c) is ok with me for the list code.  The debug code should
>>be c) I think.
>>
>>Probably vs can offer a more detailed and accurate opinion,
>>    
>>
>
>I completely agree that the current state of the generic hashing
>facilities is somewhat poor but I fail to see why you can't use
><linux/list.h>.
>  
>
I'll let vs and maybe nikita comment.

>As for the debugging code, I would love to see that turned into
>something generic (every subsystem has their own now) but it is
>definitely not something that should stop you from merging.
>
>                                Pekka
>
>
>  
>
If I encourage you to make a patch, is that ok of me? 

Re: reiser4 merging action list

[Christoph Hellwig]

On Mon, Jun 27, 2005 at 04:00:01PM -0700, Hans Reiser wrote:
> 
> Andrew asked me to put together a list of things that need to be done
> before merging:

...

> Probably I forget something.

I've started to do a very basic look over the tree and there's a few
more things that spring to mind:

 - cpp abuse.  There's quite a lot of really odd macros - the typesafe_list,
   typesafe_hash stuff is mentioned already, but there's really horrible
   stuff like _INIT_ and _DONE_ bits in init_super.c, and the wrappers for
   plugin method invocations.
 - endianess handling.  The d* types are a lovely attempt to make sure
   you're not missing endianess conversions.  We have a more general
   way to ensure that now using sparse, that's the __le* / __be* types.
   Try running sparse -Wbitwise to find places you'll need that annotations
   (after the normal sparse warnings are fixed, else you'll have a hard
   time seeing them I guess) - after that the single element struct thing
   can go away.  Also you're defining a CPU_IN_DISK_ORDER macro that's
   never used..
 - lease use kthread_* instead of kernel_thread() + lots of opencoding

Re: reiser4 merging action list

[Adrian Bunk]

On Mon, Jun 27, 2005 at 04:00:01PM -0700, Hans Reiser wrote:
> 
> Andrew asked me to put together a list of things that need to be done
> before merging:
>...
> Probably I forget something.


  * remove the dependency on !4KSTACKS


> Best,
> 
> Hans

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: reiser4 plugins

[Hans Reiser]

What happens when the ext3 inode tables get hit by sector damage? Like
us, ext3 has data munging if you hit the wrong thing, its just that our
sources of data munging are different in the details.  Details matter
though, and so we are improving with each release, and V4 does have some
real improvements.  I hope the next major release will have atomic
transactions that don't have that niggling 5% left undone that prevents
them from being fully there, and then reiser4 can seriously advance the
field of reliability in filesystems rather than just playing catchup.

Good luck with ext3,

Hans

Theodore Ts'o wrote:

>However, logically speaking, if a filesystem is designed such that in
>certain cases, the fsck program has to brute-force search every single
>disk block looking for data structures that _look_ like they might be
>part of the filesystem data, well, that's always going to be more
>error prone than one where the filesystem metadata is in
>easily-predicted locations.  It sounds like you've added some more
>checks in reiser4, and that's definitely a good thing.  Time will tell
>whether they are sufficient or not.
>
>Regards,
>
>						- Ted
>
>
>  
>

Re: reiser4 plugins

[Horst von Brand]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

Markus   Törnqvist <mjt@nysv.org> wrote:
> On Thu, Jun 23, 2005 at 11:34:50PM -0400, Horst von Brand wrote:
> >David Masover <ninja@slaphack.com> wrote:
> >> I think Hans (or someone) decided that when hardware stops working, it's
> >> not the job of the FS to compensate, it's the job of lower layers, or
> >> better, the job of the admin to replace the disk and restore from
> >> backups.

> >Handling other people's data this way is just reckless irresponsibility.
> >Sure, you can get high performance if you just forego some of your basic
> >responsibilities.

> Your honest-to-bog opinion is that the FS vendor is responsible for
> the admin not taking backups or the hardware vendor shipping crap?

No. But just relying on perfect hardware and concientious sysadmins is
reckless. Hardware /is/ flaky, sysadmins /are/ (sometimes) lazy (and
besides, today they are increasingly just plain Joe Sixpack users). Also,
backing up a few hundred GiB is /not/ fun, and then keeping track of all
the backups is messy.

Also, I'm not claiming that they are /solely/ responsible, but not having
the filesystem fall apart utterly every time some bug breaths on it /is/ a
requirement.

> *still trying to understand how that can be*

You haven't been around too much yet, do you?
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 1661 bytes --]

On Tue, Jun 28, 2005 at 09:44:59AM -0400, Horst von Brand wrote:
>
>No. But just relying on perfect hardware and concientious sysadmins is
>reckless. Hardware /is/ flaky, sysadmins /are/ (sometimes) lazy (and
>besides, today they are increasingly just plain Joe Sixpack users). Also,
>backing up a few hundred GiB is /not/ fun, and then keeping track of all
>the backups is messy.

Even home users have started to set up raid mirrors at home now that
disk space is cheap. That's a step in the right direction, I
suppose, with hardware never being good.

Taking backups in an environment where you need a few hundred GiB
backed up is not that difficult.

Get a separate, redundant box with a big tape changer and drop
periodical backups off at your bank's vault.

Get a separate, very reduntant box, with a truckload of proven 
drives in a separate raid box and run your stuff there.

Get both of the above.

If Joe Sixpack loses his mp3 collection, I don't really care,
nor should anyone else. Anything important enough to care
about is easy enough to back up. Always.

Arrogance? Maybe.

>Also, I'm not claiming that they are /solely/ responsible, but not having
>the filesystem fall apart utterly every time some bug breaths on it /is/ a
>requirement.

Reiserfs does not fall apart utterly every time some bug breaths on it.

>> *still trying to understand how that can be*
>You haven't been around too much yet, do you?

Rather I take backups, buy better hardware and understand there's a
risk involved.

Computers as a complete set can't be trusted, you can only make
the best accomodations you can.

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Jake Maciejewski]

On Tue, 2005-06-28 at 23:47 +0300, Markus Törnqvist wrote:
> On Tue, Jun 28, 2005 at 09:44:59AM -0400, Horst von Brand wrote:
> >
> >No. But just relying on perfect hardware and concientious sysadmins is
> >reckless. Hardware /is/ flaky, sysadmins /are/ (sometimes) lazy (and
> >besides, today they are increasingly just plain Joe Sixpack users). Also,
> >backing up a few hundred GiB is /not/ fun, and then keeping track of all
> >the backups is messy.
> 
> Even home users have started to set up raid mirrors at home now that
> disk space is cheap. That's a step in the right direction, I
> suppose, with hardware never being good.

I've lost more data to my own recklessness and stupidity than filesystem
corruption and hardware failure combined. RAID isn't really a good
solution. My policy for cheap storage (currently the only variety I own)
is when I buy a new hard drive, use the old drive for backups. The old
drive is always large enough to hold everything I'd miss and then some.
Home users should be capable of doing the same, assuming Windows has
something similar to rsync.

> 
> Taking backups in an environment where you need a few hundred GiB
> backed up is not that difficult.
> 
> Get a separate, redundant box with a big tape changer and drop
> periodical backups off at your bank's vault.
> 
> Get a separate, very reduntant box, with a truckload of proven 
> drives in a separate raid box and run your stuff there.
> 
> Get both of the above.
> 
> If Joe Sixpack loses his mp3 collection, I don't really care,
> nor should anyone else. Anything important enough to care
> about is easy enough to back up. Always.
> 
> Arrogance? Maybe.
> 
> >Also, I'm not claiming that they are /solely/ responsible, but not having
> >the filesystem fall apart utterly every time some bug breaths on it /is/ a
> >requirement.
> 
> Reiserfs does not fall apart utterly every time some bug breaths on it.
> 
> >> *still trying to understand how that can be*
> >You haven't been around too much yet, do you?
> 
> Rather I take backups, buy better hardware and understand there's a
> risk involved.
> 
> Computers as a complete set can't be trusted, you can only make
> the best accomodations you can.
> 
-- 
Jake Maciejewski <maciejej@msoe.edu>

Re: reiser4 plugins

[Alan Cox]

On Gwe, 2005-06-24 at 04:17, David Masover wrote:
> > False argument. So does the pen, so do hinges on doors. Do you still
> > have hinges on your doors - probably.
> 
> Indeed.  Because there's nothing better -- not because I "like it the
> way it is".

I chose hinges carefully - there are better technologies than the
generic hinge and there have been for many many years (plus cool stuff
like magnetic doors). You of course aren't a door geek (nor am I I'd
point out) but a computing one...

> Although resilience to disk errors isn't a design decision.  That's what
> SMART and new hard drives are for.  And if you're stubborn enough to
> keep the same FS around, there's dm-bbr.

Its very much a file system design consideration. If you get a small
amount of corruption or lost blocks which is the usual drive failure
case you want a high probability of getting the data back. BSD style
bitmap/inode table file systems based on FFS concepts (UFS, ext2, ext3
etc) are very good in that respect and your loss is usually minimal.

> I think Hans (or someone) decided that when hardware stops working, it's
> not the job of the FS to compensate, it's the job of lower layers, or
> better, the job of the admin to replace the disk and restore from backups.

Most desktop users today don't have backups because there is no credible
backup technology for 500Gb of data. They may have partial backups. Some
things the fs can't deal with - if the disk goes boom then thats a lower
level concern. Also certain bits like writing to spare blocks on a
problem write are indeed handled drive level nowdays.

> > Entirely or bad blocks ? The latter should have a minimal cost on a well
> > designed fs.
> 
> I was able to recover from bad blocks, though of course no Reiser that I
> know of has had bad block relocation built in...  But I got all my files
> off of it, fortunately.

Thats a good sign. reiser3 was very fragile when blocks of disk went for
a walk. If you got most of your data back thats a positive sign, not
statistically a valid sample but a positive sign

> the issues are fixed, an entirely different crowd of benevolent
> dictators will come around and say that we can't get in because we
> change the VFS.  At least some people on this list have said things to
> that effect.

There are four important issues I see here

1. It must work
2. It must be clean code that follows the kernel style
3. It must not break other stuff
4. It needs a maintainer who won't get bored 12 months later and only
support reiser5

#3 is the VFS stuff and getting the VFS locking wrong or unclean is a
*very* big deal because you'll cause corruption to non reiser4 users.

Alan

Re: reiser4 plugins

[Hans Reiser]

Alan Cox wrote:

>Thats a good sign. reiser3 was very fragile when blocks of disk went for
>a walk. If you got most of your data back thats a positive sign, not
>statistically a valid sample but a positive sign
>  
>
Alan, this is FUD.   Our V3 fsck was written after everything else was,
for lack of staffing reasons (why write an fsck before you have an FS
worth using).  As a result, there was a long period where the fsck code
was unstable.  It is reliable now. 

People often think that our tree makes fsck less robust.  Actually fsck
can throw the entire internal tree away and rebuild from leaf nodes, and
frankly that makes things pretty robust. 

There is an area where we suffered from writing fsck last.  When there
are two leaf nodes with the same key range AND the bitmap cannot be
trusted to tell us which is the valid one, we don't know which is the
most recent, and pick arbitrarily.  Also, if you store a backup of V3,
and you don't compress it, and you wipe out the bitmap blocks and need
to use fsck, we don't know what blocks are backup image and what blocks
are the fs.  We advise users to never store a V3 backup on V3 without
compressing it.  Additionally, if you create a filesystem, use it a lot,
and then mkfs on top of it, and use the new filesystem, and then wipe
out the bitmap blocks so that we cannot tell what was in the original fs
and what is in the current fs, things get merged together.  Also,
anytime you wipe out the bitmap blocks, you can get files that were
deleted.  This turns out to be a feature for many users who delete a
file, regret it, email us, we usually get it back just fine by ignoring
the bitmap blocks and recovering everything we can find.

In V4 we fixed all this.  In V4 we started by asking Vitaly our fsck guy
what node format changes would make his code easier and better.  We put
timestamps in every formatted node, and then because we don't trust
clocks we also put an mkfsid.  Thus, we can distinguish the current and
the previously made filesystem even if the clock is moved backwards.  We
can distinguish backups of other filesystems that were not compressed
from the real filesystem.  Timestamps allows fixing everything that I
described above.

What is more important, to fix V3 as described above would require a
disk format change that makes it prohibitive to roll out in a minor
release, and in V4, to change the disk format just requires adding a
plugin which could be done in a minor release.  Not only did we fix the
problem in V4, we fixed the meta problem so that any future disk format
problems are now no biggie to fix.

>4. It needs a maintainer who won't get bored 12 months later and only
>support reiser5
>  
>
Alan, Chris and Jeff support V3.  Whenever they hit a bug that I think
someone else should fix, or fail to get around to something fast enough
(rare), I have vs do it.  What more do you want?

What Chris and Jeff do that really irks me is add FEATURES not bugfixes
to V3.  As a result, almost all bugs for the last 2 years in V3 have
been theirs, and mostly due to features being added.  It also annoys me
that they don't send every patch to a second person to test before
sending it in and getting it accepted, like everyone else at Namesys
does.  What can I say, despite these gripes I am lucky to have them helping.

Features (with some exceptions) belong in major releases, not in bugfix
releases.

Namesys follows an industry standard process of putting features in
major releases and only bugfixes in the stable incremental releases. 
When we, or anyone else in the industry, do this, there are always users
who complain that their feature that they need right now has been put
into the major release and not released to them right now in the
incremental releases.  They usually use the phrase "you aren't
supporting the (insert name of stable branch), you have abandoned us!"

Now Alan, as I understand it, it is usually YOU who is wishing that
Linux would follow something closer to the release methodology that I
just described.  Maybe we have more commonality than you realize?

Hans

PS

Of course, with V4 we will have an interesting issue in that the plugins
make it so easy to add features that there will be a temptation to let
things dribble out one plugin at a time, with releases being more
releases of plugins than releases of an entire filesystem.  I haven't
really thought through how to handle that best.  I think I want a user
to be able to define that he wants the mission critical server version
of reiser4, and have that be an utterly stable body of code with the
latest plugins excluded from it.

Re: reiser4 plugins

[Olivier Galibert]

On Fri, Jun 24, 2005 at 12:21:18PM -0700, Hans Reiser wrote:
> Alan, this is FUD.   Our V3 fsck was written after everything else was,
> for lack of staffing reasons (why write an fsck before you have an FS
> worth using).  As a result, there was a long period where the fsck code
> was unstable.  It is reliable now. 

Was that more or less that about a year ago?

  OG.

Re: reiser4 plugins

[Theodore Ts'o]

On Fri, Jun 24, 2005 at 12:21:18PM -0700, Hans Reiser wrote:
> There is an area where we suffered from writing fsck last.  When there
> are two leaf nodes with the same key range AND the bitmap cannot be
> trusted to tell us which is the valid one, we don't know which is the
> most recent, and pick arbitrarily.  Also, if you store a backup of V3,
> and you don't compress it, and you wipe out the bitmap blocks and need
> to use fsck, we don't know what blocks are backup image and what blocks
> are the fs.  We advise users to never store a V3 backup on V3 without
> compressing it.

Unfortunately, there are plenty of reasons why you might want to store
a filesystem image on disk besides for backup purposes:

	* Regression tests --- I have some 70+ small filesystem images
		used for e2fsck's regression test suite.  (I am always
		amazed how many filesystem fsck programs don't have
		regression test suites.)
	* Initial ram-disk images
	* Image files for qemu or user-mode-linux

... and probably many more.  None of these are safe to store on a
reiserfs3 filesystem if you're worried about fsck being robust after a
disk failure.

Funny thing.  When I tell system administrators who have been around
the block more than a few times about this particular "feature" of
reiserfs3, they usually very quickly decide that it's time to switch
to another filesystem.....

						- Ted

Re: reiser4 plugins

[Hans Reiser]

fsck is better in V4 than it is in V3. Users should move from V3 to V4,
as V3 is obsolete. I agree on that Ted.

I also agree that Ted did a great job with fsck.ext*.

V3 was where we learned. There are performance problems in V3 that I
could only fix by writing V4. The balancing code for V3 was extremely
difficult to modify because it understood all the internals of the item
structures, and that is why we based V4 on plugins. V3 had a time where
it was really useful, and the time when it was the only metadata
journaling filesystem for Linux was its high point (thanks Chris), but
its usefulness is leaving us very soon now with V4. In 12 months, after
V4 has been pounded on by a few million users, few will want to make new
installs onto V3 instead of V4.

It would be nice if we could concentrate on speeding that transition
instead of flaming each other.

I would like to thank the ext* team, especially Ted, for a great
filesystem that I used for many years while developing V3, that was
crucial to the early success of Linux, and that is still useful to a
great many.

Hans

Re: reiser4 plugins

[Alan Cox]

On Gwe, 2005-06-24 at 20:21, Hans Reiser wrote:
> Alan, this is FUD.   Our V3 fsck was written after everything else was,
> for lack of staffing reasons (why write an fsck before you have an FS
> worth using).  As a result, there was a long period where the fsck code
> was unstable.  It is reliable now. 
> 
> People often think that our tree makes fsck less robust.  Actually fsck
> can throw the entire internal tree away and rebuild from leaf nodes, and
> frankly that makes things pretty robust. 

I did a series of tests well after resier3 had fsck that consisted of
modelling the behaviour of systems under error state. I modelled random
bit errors, bit errors at a fixed offset (class ram failure), sector 4
byte slip (known IDE fail case) and sectors going away.

Reiserfs didn't handle it anything like as gracefully as ext2. Its a
pretty easy experiment to write the code for and the results are
interesting.

Re: reiser4 plugins

[Grzegorz Kulewski]

On Sun, 26 Jun 2005, Alan Cox wrote:

> On Gwe, 2005-06-24 at 20:21, Hans Reiser wrote:
>> Alan, this is FUD.   Our V3 fsck was written after everything else was,
>> for lack of staffing reasons (why write an fsck before you have an FS
>> worth using).  As a result, there was a long period where the fsck code
>> was unstable.  It is reliable now.
>>
>> People often think that our tree makes fsck less robust.  Actually fsck
>> can throw the entire internal tree away and rebuild from leaf nodes, and
>> frankly that makes things pretty robust.
>
> I did a series of tests well after resier3 had fsck that consisted of
> modelling the behaviour of systems under error state. I modelled random
> bit errors, bit errors at a fixed offset (class ram failure), sector 4
> byte slip (known IDE fail case) and sectors going away.
>
> Reiserfs didn't handle it anything like as gracefully as ext2. Its a
> pretty easy experiment to write the code for and the results are
> interesting.

Maybe but I once checked some other error scenario. I generated (by 
mistake of course) dm table that lineary connected 3 times the same 
partition (instead of 3 different partitions). Both Reiser4 and Reiserfs3 
gave a lot of errors while trying to use such device. Ext3 did not give 
single error and was hapily droping my data,

I agree that this is not very useful test case for disk problems but it 
shows that, at least, checks for trouble in Reiser4 are miles before those 
in Ext2/3. If only Reiser4 could print a note what I done wrong... ;-)


Grzegorz Kulewski

torturing filesystems [was Re: reiser4 plugins]

[Pavel Machek]

Hi!

> > Alan, this is FUD.   Our V3 fsck was written after everything else was,
> > for lack of staffing reasons (why write an fsck before you have an FS
> > worth using).  As a result, there was a long period where the fsck code
> > was unstable.  It is reliable now. 
> > 
> > People often think that our tree makes fsck less robust.  Actually fsck
> > can throw the entire internal tree away and rebuild from leaf nodes, and
> > frankly that makes things pretty robust. 
> 
> I did a series of tests well after resier3 had fsck that consisted of
> modelling the behaviour of systems under error state. I modelled random
> bit errors, bit errors at a fixed offset (class ram failure), sector 4
> byte slip (known IDE fail case) and sectors going away.

Do you have that code somewhere? Best I could do was overwriting
random parts of partition with random data, you seem to have "more
interesting" failures...
							Pavel
-- 
teflon -- maybe it is a trademark, but it should not be.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alan Cox wrote:
> On Gwe, 2005-06-24 at 04:17, David Masover wrote:
> 
>>>False argument. So does the pen, so do hinges on doors. Do you still
>>>have hinges on your doors - probably.
>>
>>Indeed.  Because there's nothing better -- not because I "like it the
>>way it is".
> 
> 
> I chose hinges carefully - there are better technologies than the
> generic hinge and there have been for many many years (plus cool stuff
> like magnetic doors). You of course aren't a door geek (nor am I I'd
> point out) but a computing one...

Right, but even if I was a door geek, changing hinges costs more than
changing code.  Now, if I were building a new house and I happened to
know about other alternatives, I might not still be using hinges.  Also,
I leave enough doors open enough of the time that it's not a serious
impact on my life.

Now, OS design is a place which does affect my productivity, so I do
care about the design, and while it's hard to change that design, it
doesn't actually involve buying stuff.

>>I think Hans (or someone) decided that when hardware stops working, it's
>>not the job of the FS to compensate, it's the job of lower layers, or
>>better, the job of the admin to replace the disk and restore from backups.
> 
> 
> Most desktop users today don't have backups because there is no credible
> backup technology for 500Gb of data. They may have partial backups. Some

Bandwidth is getting faster.  And I just found a nice site for backups
called streamload.com.  They don't seem to support rsync, and allow only
100 meg downloads, but unlimited uploads.

Few desktop users today really need to backup more than 50 megs of data.

> things the fs can't deal with - if the disk goes boom then thats a lower
> level concern. Also certain bits like writing to spare blocks on a
> problem write are indeed handled drive level nowdays.

Right.  And putting them in the FS is unneccesary bloat if you've got
another mechanism for dealing with it.  Anyone with 500 gigs of data can
afford to do a little RAID, or at least some burned DVDs.

DVDs are cheap nowdays:
http://www.newegg.com/Product/Product.asp?Item=N82E16817502002

Ok, you might need two of those.  Still, it's not much, unless that's
500 gigs of data which is actually changing pretty rapidly.  Most users
I know seem to burn CDs or DVDs of static things (like music) and only
keep games and office-type files, plus copies of the stuff they've
burned, on their hard drives.  So, backup for most people I know = not
throwing away game install disks + cron job to send < 50 megs of data to
Streamload.

I agree it's nice to have a more corruption-proof filesystem.
Convenient.  But not absolutely necessary.

>>the issues are fixed, an entirely different crowd of benevolent
>>dictators will come around and say that we can't get in because we
>>change the VFS.  At least some people on this list have said things to
>>that effect.
> 
> 
> There are four important issues I see here
> 
> 1. It must work
> 2. It must be clean code that follows the kernel style
> 3. It must not break other stuff
> 4. It needs a maintainer who won't get bored 12 months later and only
> support reiser5
> 
> #3 is the VFS stuff and getting the VFS locking wrong or unclean is a
> *very* big deal because you'll cause corruption to non reiser4 users.

It's #2 and #3 that I'm worried about.  #4 is a little unfair, and I can
verify that #1 is satisfied.

And, by "worried about", I mean I'm worried about the attitude of other
people.  As far as I'm concerned, Reiser4 is mergeable now.  But I'm not
exactly an authority.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzMMXgHNmZLgCUhAQLqMQ/+LeGfZlmMt8pog6StIbWj3ZtfSXXnOAtu
Os6jMMITPL7QlGfD8ren6nExtS/zxAfua6eQmoc9Lk4qPSFGJsSj8s3APE6p+igr
YLHSD6yKMW+sEwz+9d/jLI22R+Ct6x2AUGeYKbmJ4GnYYSeDMsJieeG/OJJscQFG
ETjSDWYQM8Ba7YgiJBWfJFYfD9LuM2wE1yUY5zmqVlT1hKSEB6rKEpx+J4MpAj6H
u19DCDoluVqTI/4XIFDjpHkwsNfYElCDe6dbdtgeHlDIjcgX8PRu2ZEVAhDjwG3H
wlLc1V6lE/qznf4kUgWg3XAc6P2MbJAJd6S7xg5ifSaEzYE7sYwPrXAaPOh+BpNV
P01T5eqsSoIbDQb1q1w686EVdQKXSXms2W21IFctHi60FBwfBIfLNwJ8I590MIhG
SkhBj/LAnZrFDztf8Z16oBy0zNrVKLQBGkd3FlD8YDv8J7II266yv+aOVEvNp0Sw
5g8RnlxnUqnl1JCo7TRHLe3pknPXms9JISd/wHvjIPuxgOBSVw0nPDde+T/mdlOP
28ef3MDEDwc2cBe3xPLj1nXYjrLvw6zGrSR22IRYDrdZgqPtKjrUEj+0T6pmmHW/
NAH8Ck9zROvadnaB9/st4JJT74axRtwI1HFb4SMfo23vQ6za7RNIHKjDMVsAjtqV
fmgsvO2QUo8=
=MlUY
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Jeff Garzik]

David Masover wrote:
> And, by "worried about", I mean I'm worried about the attitude of other
> people.  As far as I'm concerned, Reiser4 is mergeable now.  But I'm not
> exactly an authority.


Then why not listen to authorities, all of whom are saying "it's not 
ready yet"

	Jeff

Re: reiser4 plugins

[Hans Reiser]

Jeff Garzik wrote:

> Then why not listen to authorities, all of whom are saying "it's not
> ready yet"

What exactly is not ready Jeff?  As I see it, this thread is 95%
posturing, and almost no technical reasons for not merging.  These
"authorities"seem perfectly content with echoing the words of someone
who skimmed the code and shot his mouth off without understanding it,
and now these "authorities" just echo him because they like him and
assume he must be right.

Actually, David's understanding of reiser4 is far deeper than those of
anyone claiming to be an "authority".  He understands what a plugin is. 
Wow.  He does not have to be told that VFS does not currently support a
notion of pluginids, so it is not duplicative.  Wow.  He does not have
to be told that plugins are analogous to classes and what VFS currently
has is analogous to instances.  Wow.  He gets it that reiser4 plugins
disturb absolutely nothing in the rest of the kernel because we use the
standard VFS interface.  Wow.  Does any of the rest of that echo chamber
of authority understand those things?  Seems to me that these
"authorities" are just as capable of shooting their mouths off before
doing their homework as anyone else is..... 

David has used reiser4.  Have you?  Maybe you guys should try it.  Maybe
you should all have less faith in your ability to skim code and
understand it instantly.  Maybe you guys should actually read the
technical parts of this thread rather than the flame parts only?

Maybe our community's social traditions are a bit lacking?  Maybe saying
that everyone else goes through this just means that changing the way we
do things is needed even more?

Hans

Re: reiser4 plugins

[Jeff Garzik]

Hans Reiser wrote:
> Jeff Garzik wrote:
> 
> 
>>Then why not listen to authorities, all of whom are saying "it's not
>>ready yet"
> 
> 
> What exactly is not ready Jeff?  As I see it, this thread is 95%
> posturing, and almost no technical reasons for not merging.  These
> "authorities"seem perfectly content with echoing the words of someone
> who skimmed the code and shot his mouth off without understanding it,
> and now these "authorities" just echo him because they like him and
> assume he must be right.

I've already listed two rather large technical reasons.

When a vendor submits code, they know their [hardware | filesystem] 
best, but Linux kernel developers know the kernel code and kernel 
direction best.

There needs to be a meeting of the minds.  Vendors that stick to "I know 
best" line of reasoning don't get very far in Linux.

	Jeff

Re: reiser4 plugins

[Hans Reiser]

Jeff Garzik wrote:

>
> I've already listed two rather large technical reasons.

They are?

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
[...]
> David has used reiser4.  Have you?  Maybe you guys should try it.  Maybe
> you should all have less faith in your ability to skim code and
> understand it instantly.  Maybe you guys should actually read the
> technical parts of this thread rather than the flame parts only?

To be fair, I use it, but I haven't read much of it.  I mostly read the
whitepaper, and then read it again, several times, to make sure I groked
it.  Then I read the Future Vision paper, once, and completely failed to
understand it.  Now that I understand Spotlight, I might go back to it.

This is why I stay out of the deeply technical stuff, and frequently
insist that I don't know what I'm doing and don't work here.

> Maybe our community's social traditions are a bit lacking?  Maybe saying
> that everyone else goes through this just means that changing the way we
> do things is needed even more?

The social traditions aren't uniform.  In fact, if we're referring to
all of open source, go hang out on irc.freenode.net#gentoo for a great
community, whether or not you like the distro.  If you want developers,
there's not too many RTFM's and "I was coding bytecode engines when you
were in dipers" attitudes in #perl, either.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrziPXgHNmZLgCUhAQKJiA//aIvOMGjwvorsU8YhIhR9jpqITFzR16/y
YeXef0o7vX5UC6ws/eP3EQbl46/NbT8mth0l3x0OgXIkTha3edCPMtNUBuJ/ISad
nYY17mNPCr5NWzyh5aL7Cw1DYBIVrwQ8dcVg7VbmnbutPyTxKnFxugABExk9MASQ
+1n1FVmKRRl4cjte3qERnFN9crPxkLnGWWT5UneGo9tnxKCqd3VBcJQ23PYHsskF
dyYmh9/YJaPMskFRK9oQzw9TAM740rwZvOOrQGO84fQkYnrgjrVxis3O6JYbVYzP
tiQNC7kSE2HQoMwFratL+tTj2mpx2+Akmhb/XT738tDSNvpSIBhy2eGqXU2p61MF
sZiz0kNiSpW2e2kUGYiDKtey9o7bTlQJJ1KmzLtVKqOhPgMpRkyR8PG7X9Sz0P//
zLaETf02VHt+1stN9xKum9zYI3Ba0aOt3fhA+1RqBSszV9JdTYVA8/zekGSTYezx
neEGJGMkZCRISY2hWQTmHYNVG55ie3cxGgKQdYRxM7wO6AMQID1074UMIqE2aPn5
qjGdtkLXtiLQ33gg7v5uAQeysyQxBZSEBI+uQSH3guXIjOX9XItMaqqygKNXB7ak
zsZw8qguDVNHIAyPlHVBVHkCUEZ0hstdfYUU5gyMY+mLP0eq/HgSp5XEReX6RU0U
3KSielBMshs=
=ipn3
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

> The social traditions aren't uniform.  In fact, if we're referring to
> all of open source, go hang out on irc.freenode.net#gentoo for a great
> community, whether or not you like the distro.  If you want developers,
> there's not too many RTFM's and "I was coding bytecode engines when you
> were in dipers" attitudes in #perl, either.

I would develop for BSD instead if they were anywhere near Linux in
marketshare.  These attitudes, when the BSD guys had them, were why I
chose Linux over BSD to develop for.  Now Linux has the market share,
and they act like the old BSD guys used to.   Sad.  Somebody else will
come along and challenge Linux someday.   I wonder if Apple is a better
social environment for developers these days than Linux?  It would be
fun to work with Steve Jobs, he has such a sense of vision and a delight
in new things.  He hires good people too; Dominic Giampaolo is really
sharp. 

Hans

Re: reiser4 plugins

[Bob R. Taylor]

On Fri, 2005-06-24 at 23:15, Hans Reiser wrote:

> I would develop for BSD instead if they were anywhere near Linux in
> marketshare.  These attitudes, when the BSD guys had them, were why I
> chose Linux over BSD to develop for.  Now Linux has the market share,
> and they act like the old BSD guys used to.   Sad.  Somebody else will
> come along and challenge Linux someday.   I wonder if Apple is a better
> social environment for developers these days than Linux?  It would be
> fun to work with Steve Jobs, he has such a sense of vision and a delight
> in new things.  He hires good people too; Dominic Giampaolo is really
> sharp. 

I hate to jump in here but this is just too much!

Hans, I remember well the bellyaching you did when you insisted on the
Linux developers merging Reiserfs V1 into the kernel. Did you not learn
from *that* experience? You have taken *some* criticism here which may
*not* have been flames at you until you flamed *first*. You have been
told by at least 2 developers here what you must do *first* before
consideration of merging V4. As of this writing YOU HAVE NOT ANSWERED!
You obviously have not even read all articles posted here. When Jeff
Garzik said "I've already listed two rather large technical reasons."
You answered "They are?". You just continue your insane bickering, now
followed by picking up your marbles and leaving. Grow up.

BTW, I could care less whether V4 gets merged or not as I am *not* a
developer.

Bob
-- 
Bob R. Taylor <brtaylor@sanfelipe.com.mx>

Re: reiser4 plugins

[Diego Calleja]

El Fri, 24 Jun 2005 21:31:02 -0700,
Hans Reiser <reiser@namesys.com> escribió:


> What exactly is not ready Jeff?  As I see it, this thread is 95%
> posturing, and almost no technical reasons for not merging.  These
> "authorities"seem perfectly content with echoing the words of someone
> who skimmed the code and shot his mouth off without understanding it,
> and now these "authorities" just echo him because they like him and
> assume he must be right.


I'm not one of those "authorities" but I doubt reiser4 will be merged at all
with that attitude.

Re: reiser4 plugins

[Matthew Frost]

--- Diego Calleja <diegocg@gmail.com> wrote:

> El Fri, 24 Jun 2005 21:31:02 -0700,
> Hans Reiser <reiser@namesys.com> escribió:
> 
> 
> > What exactly is not ready Jeff?  As I see it, this thread is 95%
> > posturing, and almost no technical reasons for not merging.  These
> > "authorities"seem perfectly content with echoing the words of someone
> > who skimmed the code and shot his mouth off without understanding it,
> > and now these "authorities" just echo him because they like him and
> > assume he must be right.
> 
> 
> I'm not one of those "authorities" but I doubt reiser4 will be merged
> at all with that attitude.


Spot on.  Somebody hasn't read his Linus.  It seems to me that anytime I
hear whining about how unfair the system is, how arbitrary patch
rejection is, the flaw is either a lack of social competence, or failure
to follow the submission guidelines.  Lately the number three reason,
"Linus forgot," happens less and less.  And linux developers are offering
to help Reiser development fix what's wrong!  That's above and beyond.  

If Hans wants to take his ball and go home, let it be understood that it
is not because of a failure in the linux process.  

People that don't want flames, drop out of flame wars and do constructive
work.  People that want to talk ideology, start flame wars and perpetuate
them.  If you're having a code argument and you can go four posts without
referring to actual code, you're no longer having a code argument. 
Non-demonstrable merit is not merit.  Benchmarks can be necessary -- but
are never alone sufficient -- merit to merge to mainline.

Hans: consider that enough linux people like your stuff enough to have
this argument.  Performance has long since ceased to be at issue.  Linux
is a thing in itself.  If Reiser4 is a thing in itself, and not merely a
means to a further release point, you will care enough about it to merge
it usably and maintainably into the kernel.  Should you obsolesce 4 as
you have done to 3, someone else can care enough about it, and do
something with it.  Features are added to Reiser3 because it is a thing
in itself now that it is in the kernel.  Its users care.  Its
maintainers, even if they still work for Namesys, care.  Listen next time
you hear someone talk about "vendor-abandoned projects" continuing to
function through community development.

Alan has said, "Free software is always late."  Free software follows the
itch.  If Hans wants linux to have Reiser4, Reiser development will work
with linux development to become compliant and explain/adapt any
additional functionality.  Linux does not abjectly need Reiser4 in-tree. 
Reiser4 is ultimately Hans's scratch for Hans's itch.  It will be the
more effective and widely usable once it is stable in-tree, but the
mainline kernel developers aren't the ones with the itch.  Protocols
exist in social systems as a means to constructively enlist the help of
people outside the problem.  The protocols here are well established. 
This is a patch meritocracy.  Failure to comply further increases the
lateness of Reiser4 in mainline kernels.  Ego involvement is noise.

My observations being what they are, may I suggest that this thread die,
in favor of more eyes and in-depth current code analysis help in "Re:
reiser4 merging action list"?  Higher utility function; higher S:N. 
Flames can always go down with the ship.

Frosty
Not a programmer, merely literate.

No, I don't usually respond to trolls, but this thread, from this man,
itches me.  It's not worthy of his brilliance, which may be his tragic
flaw.

Re: reiser4 plugins

[Jesper Krogh]

["Followup-To:" header set to gmane.linux.kernel.]
I gmane.linux.kernel, skrev David Masover:
> > Most desktop users today don't have backups because there is no credible
> > backup technology for 500Gb of data. They may have partial backups. Some
> 
>  Bandwidth is getting faster.  And I just found a nice site for backups
>  called streamload.com.  They don't seem to support rsync, and allow only
>  100 meg downloads, but unlimited uploads.
> 
>  Few desktop users today really need to backup more than 50 megs of data.

That gives tedious manual work.. and btw, won't save you if you from
loosing stuff from when the backup was made until now. 

> > things the fs can't deal with - if the disk goes boom then thats a lower
> > level concern. Also certain bits like writing to spare blocks on a
> > problem write are indeed handled drive level nowdays.
> 
>  Right.  And putting them in the FS is unneccesary bloat if you've got
>  another mechanism for dealing with it.  Anyone with 500 gigs of data can
>  afford to do a little RAID, or at least some burned DVDs.
> 
>  DVDs are cheap nowdays:
>  http://www.newegg.com/Product/Product.asp?Item=N82E16817502002

Again lots of manual work.. I actually have a DAT-station.. but I'm not
getting it used. People have DVD-burners, but many don't get time to do
a backup anyway. A Copy-On-Write feature in the filesystem would save
the average dataloss situation todag (for home users). Where they
accidentally deletes stuff. 

>  Streamload.

Why, when it could be quick and transparent. And Linux is used many
places where you cant let data out-of-the-house of where bandwidth
"sucks". The waste-space in my diskdrives increases everyday .. and i
fill up with a tar-ball of the system every now-and-then, but it would
definately be better suited and more effecient (save me more times) done
directly in the filesystem. 

>  I agree it's nice to have a more corruption-proof filesystem.
>  Convenient.  But not absolutely necessary.

Thats called raid, we have that allready. But raid won't help for and: 
rm /etc/passwd, a Copy-On-Write filesystem (not-snapshot) would. Used on
a mirrored raiddisk, with enough space on the disk, it would actually
guard you from allmost anything but getting the computer stolen. 

Totally unrelated to reiser4 but a feature that would be nice to have in
"any" filesystem. 

Jesper
-- 
./Jesper Krogh, jesper@krogh.cc

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jesper Krogh wrote:
> ["Followup-To:" header set to gmane.linux.kernel.]
> I gmane.linux.kernel, skrev David Masover:
> 
>>>Most desktop users today don't have backups because there is no credible
>>>backup technology for 500Gb of data. They may have partial backups. Some
>>
>> Bandwidth is getting faster.  And I just found a nice site for backups
>> called streamload.com.  They don't seem to support rsync, and allow only
>> 100 meg downloads, but unlimited uploads.
>>
>> Few desktop users today really need to backup more than 50 megs of data.
> 
> 
> That gives tedious manual work.. and btw, won't save you if you from
> loosing stuff from when the backup was made until now. 

Manual?

Try scripting.  For me, that's a tar command involving /home, /etc, and
about one or two other files, with a few excludes, like /home/shared/video.

>>>things the fs can't deal with - if the disk goes boom then thats a lower
>>>level concern. Also certain bits like writing to spare blocks on a
>>>problem write are indeed handled drive level nowdays.
>>
>> Right.  And putting them in the FS is unneccesary bloat if you've got
>> another mechanism for dealing with it.  Anyone with 500 gigs of data can
>> afford to do a little RAID, or at least some burned DVDs.
>>
>> DVDs are cheap nowdays:
>> http://www.newegg.com/Product/Product.asp?Item=N82E16817502002
> 
> 
> Again lots of manual work.. I actually have a DAT-station.. but I'm not
> getting it used. People have DVD-burners, but many don't get time to do
> a backup anyway. A Copy-On-Write feature in the filesystem would save
> the average dataloss situation todag (for home users). Where they
> accidentally deletes stuff. 

A lot of the people I know keep stuff on their DVDs, like movies and
music, so they can carry them around.  And the rest of it is the 50 megs.

>> Streamload.
> 
> 
> Why, when it could be quick and transparent. And Linux is used many
> places where you cant let data out-of-the-house of where bandwidth
> "sucks". The waste-space in my diskdrives increases everyday .. and i
> fill up with a tar-ball of the system every now-and-then, but it would
> definately be better suited and more effecient (save me more times) done
> directly in the filesystem. 

Streamload is quick and transparent for me.  I put files on the
fileserver, it tars them up and uploads them via Streamload's perl client.

>> I agree it's nice to have a more corruption-proof filesystem.
>> Convenient.  But not absolutely necessary.
> 
> 
> Thats called raid, we have that allready. But raid won't help for and: 
> rm /etc/passwd, a Copy-On-Write filesystem (not-snapshot) would. Used on
> a mirrored raiddisk, with enough space on the disk, it would actually
> guard you from allmost anything but getting the computer stolen. 
> 
> Totally unrelated to reiser4 but a feature that would be nice to have in
> "any" filesystem. 

Not totally unrelated.  COW has been discussed.  I don't remember if the
low-level stuff was done, but the main complaint was a lack of a copy
system call.

And RAID is an argument for Reiser4's attitude that it's not the job of
the FS to be corruption-proof.

Still, it's far easier to avoid deleting stuff than to avoid disk
failure.  First priority is to get the stuff OFF the machine.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr0LxXgHNmZLgCUhAQKKBBAAiAicaEKY42s0WctOcLg/m/ciVtphQ1jY
rChlThrsRCl4xAE45GgAu4P1ZdEYGwI1574W9z2j8EpbdnghX8tBHFUDSG1/K3+f
8Ud0zyMaI46k+D/IzkXS1ENYDj1PGmPAuVbM2pAa3JW0UOMzvKRsSADewxAW7OdY
V9fazSYu6l7Sn64XJxpZmXs9fkElXkqaNu/2N5d6rdH8hMmLhxs8HcsbAaJ7ch87
lz2RMruQ4Keh6H6HTHHvmoYog6XakwkD0pOY0efFZolO/EZFnmIlS9VHRIyb6mls
2xKPABDh9Qq0qQxpATgmXnVI9oh9qgrp4wqQ8nTQfEnhL5fvfBTXzJgfjOJiqXUy
vX4K/PP+9wzQNoqhTj4g11JBT8ilemsA4U+gbr82qSvvkOXrHkgGTQe6wgUyo6GZ
R8Ui7/jmAvNw+RvfL/p0s0s3e/EimUC7ASvN64z6z77wqNH0uUfUwRmD9SL8TbTf
jPdFvcd65F84T4gXphT4Dhwjs4fVjZUq3BqB+zMl3lKJP6pJfc5bqpjvqc2Rg1Yv
jfpvk3gihG7YN68IgZV+9IHJG6d5P05gi/YMqu75JDkqTXe2VznOXXQp4d58PhO0
ZKJnvAfr/xKrnuwgDUjgfaHThgquyMiC95hMo4IlSEfUquKPFGY4c9k0VKQHduqc
lcZOnfkvar0=
=olqO
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Alan Cox]

On Sad, 2005-06-25 at 04:14, David Masover wrote:
> Right, but even if I was a door geek, changing hinges costs more than
> changing code.  Now, if I were building a new house and I happened to

Probably not, programmers are expensive 8)

> DVDs are cheap nowdays:
> http://www.newegg.com/Product/Product.asp?Item=N82E16817502002

> Ok, you might need two of those.  Still, it's not much, unless that's

At least four because the media decay patterns are not well understood.
So you need raid DVD too 8)

> > 1. It must work
> > 2. It must be clean code that follows the kernel style
> > 3. It must not break other stuff
> > 4. It needs a maintainer who won't get bored 12 months later and only
> > support reiser5

> It's #2 and #3 that I'm worried about.  #4 is a little unfair, and I can
> verify that #1 is satisfied.

Please review the 2.5 history of reiserfs3 if you believe so

Re: reiser4 plugins

[Olivier Galibert]

On Thu, Jun 23, 2005 at 10:17:06PM -0500, David Masover wrote:
> I was able to recover from bad blocks, though of course no Reiser that I
> know of has had bad block relocation built in...  But I got all my files
> off of it, fortunately.

My experience shows that you've been very, very lucky.  I hope r4 is
better in that regard.

If you want to try with r3, take a well-used partition[1] and copy it
at block level to another partition or a file.  Then zero some random
spans of blocks in the copy and reiserfsck --rebuild-tree it.  My
experience is that you'll usually get the files names and directory
tree but their contents will have been scattered all over the place.

  OG.

[1] I suspect a minimum of fragmentation is in order to see the
problem

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Olivier Galibert wrote:
> On Thu, Jun 23, 2005 at 10:17:06PM -0500, David Masover wrote:
> 
>>I was able to recover from bad blocks, though of course no Reiser that I
>>know of has had bad block relocation built in...  But I got all my files
>>off of it, fortunately.
> 
> 
> My experience shows that you've been very, very lucky.  I hope r4 is
> better in that regard.

I was on Reiser4.  And as Hans is loudly protesting, v3 seems to be
stable now, including being somewhat resilient to disk breakage.

But I do better backups now.

> If you want to try with r3, take a well-used partition[1] and copy it
> at block level to another partition or a file.  Then zero some random
> spans of blocks in the copy and reiserfsck --rebuild-tree it.  My
> experience is that you'll usually get the files names and directory
> tree but their contents will have been scattered all over the place.

Sometime I'll try that.  I do remember mounting read-only *before* I
tried the --rebuild-tree, because I didn't want to give it a chance to
screw up more that it already was.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzG3XgHNmZLgCUhAQKayQ//Z4R/1lKkjaiHfIrIpVXmLrGfcGUuVXLG
FeQb+50orlA6pfaHDJ5JkJKKyHnTsnNSvn+VHdxqvz25/ijWDoewe368syXSggOA
JkSUSvdQlcKn+dnpT8tPFY8OU3Ve87zb53fAI9J145DRLpXXmWjW5/R1OUzlPnmr
+nn2n32UvbSyUj8VAM9ECtJ3+HaxXKm0ASbQLKgLoLmJ5bO/u2xml7rGtB1kdaCr
xJIxugAaWnA/sytoK2vIsdnH4QFQqbuqW4bdHa3ziORS1w/DQ08PMFtxu/3ChKVQ
5d0cGtMsZkVIz+k4zE+3RD2QXZVLRvZCTwPqQciitvXCRqz3d9Wc73D+fC8FwLKx
t7aARQKprPDA/bgg4SPAoJSVxZMTJDATuJ5f6XrpDB/FVUmHYQfqNLhOD/Vl5IAA
9aZCPjDXMknClydasw/SiRu3DUiOIKYlXXUN1Tnsx4l9lbVJpca25TBem+0Val2P
IsruPiJ8L3XztRNwQJOY4RGH1CMobPnwEA4F9PFKqVF+wwJS5sNNqbkoD7XMrOT6
UYn3KC0DixgVCkl75nP6ORs8F9LjIuDtFd/+KNQC+AlcU5MX+BeaJD1QhT5G/1qd
GjRcQaN+95jVxLVjkqnb552dQ7+CMt/DzszFDtfKlJepobflVMtX1hoZ0phudAgR
PqjDS3tIROk=
=lkfJ
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Pavel Machek]

Hi!

> > I was able to recover from bad blocks, though of course no Reiser that I
> > know of has had bad block relocation built in...  But I got all my files
> > off of it, fortunately.
> 
> My experience shows that you've been very, very lucky.  I hope r4 is
> better in that regard.
> 
> If you want to try with r3, take a well-used partition[1] and copy it
> at block level to another partition or a file.  Then zero some random
> spans of blocks in the copy and reiserfsck --rebuild-tree it.  My
> experience is that you'll usually get the files names and directory
> tree but their contents will have been scattered all over the place.

Actually it would be nice to have a script to automate this, and test
various filesystems for robustness :-). I done something similar, and
it uncovered few bugs in ext2fsck and vfatfsck; perhaps it is time to
try it on other filesystems, too?
								Pavel

#!/bin/bash
#
# fscktest
#
# Usage: 
#	 Make sure output is logged somewhere
#        First, run fscktest -p as root
#	 Then you can run fscktest as normal user...
#

prepare() {
	SIZE=100000
	echo "Creating file..."
	cat /dev/zero | head -c $[$SIZE*1024] > test
	echo "Making filesystem..."
	/sbin/mkfs.$FS test
	echo "Mounting..."
	mount test -o loop /mnt || exit "Cant mount"
	echo "Copying files..."
	cp -a /bin /mnt
	cp -a /usr/bin /mnt
	cp -a /usr/src/linux /mnt
	echo "Syncing..."
	sync
	echo "Unmounting..."
	umount /mnt
	echo "Moving..."
	mv test fsck.okay
	echo "All done."
}

FS=ext2
if [ .$1 == .-p ]; then
	prepare
	exit
	fi
RUN=0
while true; do
	RUN=$[$RUN+1]
	echo "Run #$RUN"
	echo Preparing...
	cat fsck.okay > fsck.damaged
	echo Damaging...
	dd if=/dev/urandom of=fsck.damaged count=10240 seek=7 conv=notrunc
	cp fsck.damaged fsck.test
	echo First check...
	/sbin/fsck.$FS -fy fsck.damaged
	RESULT=$?
	if [ $RESULT != 1 -a $RESULT != 2 -a $RESULT != 0 ]; then
		echo "Fsck failed in bad way (result = $RESULT)"
		exit
		fi
	echo Second check...
	/sbin/fsck.$FS -fy fsck.damaged
	RESULT=$?
	if [ $RESULT != 0 ]; then
		echo "Fsck lied about its success (result = $RESULT)"
		exit
		fi
	done

-- 
teflon -- maybe it is a trademark, but it should not be.

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:


>
>
> I was able to recover from bad blocks, though of course no Reiser that I
> know of has had bad block relocation built in...  

there was a patch somewhere.  Vitaly, please comment.

Re: reiser4 plugins

[Vitaly Fertman]

On Friday 24 June 2005 23:46, Hans Reiser wrote:
> David Masover wrote:
> 
> 
> >
> >
> > I was able to recover from bad blocks, though of course no Reiser that I
> > know of has had bad block relocation built in...  
> 
> there was a patch somewhere.  Vitaly, please comment.

http://www.namesys.com/bad-block-handling.html describes 
how reiserfs handles bad blocks.

-- 
Thanks,
Vitaly Fertman

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vitaly Fertman wrote:
> On Friday 24 June 2005 23:46, Hans Reiser wrote:
> 
>>David Masover wrote:
>>
>>
>>
>>>
>>>I was able to recover from bad blocks, though of course no Reiser that I
>>>know of has had bad block relocation built in...  
>>
>>there was a patch somewhere.  Vitaly, please comment.
> 
> 
> http://www.namesys.com/bad-block-handling.html describes 
> how reiserfs handles bad blocks.

Anything like this for v4?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsBqhHgHNmZLgCUhAQLnqQ/8CNGlg/b2rS5TJi6Qzbi5E3eQMDXpU6GO
bQZCrZ60dhPGI0z5AqM+kpd4tfMqvF9pW83Pa4xPN9Snvs4RxJt79JutRss5JByF
ZrBdwt2Whq0Hmv5MxbIlCKof92thg3AL7HVmuKVgApAy85cOLH3C0CTMS00KjHqn
oVDO6gAYlZxJXd/Z+VqCILRTorcTJQXi7NaZy0ptSYPZlna/1JL+0JUOKOAQ8U++
8xK/ETwMwnWRd0/yZfrjE7Y1hU1bFlM5QAHKoRHnXfXEqXNm5eg+idsfCbrv80uw
+QuQy46Xl0puumcB1G6kp+WLboQe+u4fCn85osiWo8OzVcU4Q4hkqCjV6kOo6WH5
VYn/W3kEgdEpgxnkuDEG4uxiootzpUPUVLpVsK2GSqo5pdiNz63IGvt40WYQOnkc
I6/YPDTlHdmrFbxpASY05gMdtHYDEkvI7cw8MBeaK35Fr7S5VvCeRkEFI48CZHTw
Yz+wljq6v22tPmkdKZByXqIY+8A5hgSf2eBXGG4HBL9ogBFbj+IkueqhRQ9Z023J
EL7qYVlO0EDmpxb+8MbDYvugMXk6PlzebZLNu4Zw7S0sbkZFUhdkCQS3bnn22kJ4
UxYfsxy7fTUl5Rbdk+TzPbUX39U/fO9seZi6euEt6Dc3ak4g9lVnnzIo8GkwPBNF
7KRZCgN3wCw=
=7bqs
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Vitaly Fertman]

On Tuesday 28 June 2005 01:07, David Masover wrote:
> Vitaly Fertman wrote:
> > On Friday 24 June 2005 23:46, Hans Reiser wrote:
> >
> >>David Masover wrote:
> >>
> >>
> >>
> >>>
> >>>I was able to recover from bad blocks, though of course no Reiser that I
> >>>know of has had bad block relocation built in...
> >>
> >>there was a patch somewhere.  Vitaly, please comment.
> >
> >
> > http://www.namesys.com/bad-block-handling.html describes
> > how reiserfs handles bad blocks.
> 
> Anything like this for v4?
 
in todo for v4, not implemented yet.

-- 
Thanks,
Vitaly Fertman

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vitaly Fertman wrote:
> On Tuesday 28 June 2005 01:07, David Masover wrote:
> 
>>Vitaly Fertman wrote:
>>
>>>On Friday 24 June 2005 23:46, Hans Reiser wrote:
>>>
>>>
>>>>David Masover wrote:
>>>>
>>>>
>>>>
>>>>
>>>>>I was able to recover from bad blocks, though of course no Reiser that I
>>>>>know of has had bad block relocation built in...
>>>>
>>>>there was a patch somewhere.  Vitaly, please comment.
>>>
>>>
>>>http://www.namesys.com/bad-block-handling.html describes
>>>how reiserfs handles bad blocks.
>>
>>Anything like this for v4?
> 
>  
> in todo for v4, not implemented yet.

Is it significantly different that there'd be another doc I can read?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsGhjXgHNmZLgCUhAQJaeg/7BeQVRKc0m0AEKiwp6B/N/R6/32JcIvk9
0jZq3A6wYsSxmRSDfId+FVoHApAoZldOtw5CZ7W3TwUJBxzGwFZ7IYM4mQaZ5eRa
7CdAoX9LrG85sY/2M9qypE5cW6BQ/0DJFPDOiree/+6mNomg8uOUJ6FeO93j2JW3
jdPrWqsmYBPSDo4vKha+xoAZpCH/sMLwDDBUIFfC2iz4B4yTxYNfEX+mJ4T2mEls
3YqSiTMkgnFbsVTdUiIjqVJY+jJ9ALhDKdc/hkNNrNir3Iib6aNbg9J9J5cDgNHU
zzvNyp6WsCgQjtCvfMojVewKAMu94eg1sXt6ESNz4dKbdPBSdo+lBnATntnMjFCN
7nS9K/kKcTKwSVQjQdyEhbm51C9SXYLxj4CMcru9ZyVtdlyzCLhBda1FvaxWbatJ
iGCh8tg43Xkd1N4H+LlTQpJrfbkJBXu4U9oYIuNSQuolMfz6IbJqcfNzV0hm9drp
lGLODFSFEWFqHUgPqLymk8/pBW9xP4XYwwWFcY8x0Dw4UxpWRB43EZ8Nydg4PBEk
SrLnq1FNSP+QLHZvWwF+8vpWj1jyeiBU2hFpt/p9crU5uy4OWGbKEh8GuEHFgp9P
BdKzp4iwTFwgjKR/uM3JnCfN4ct0yyyBZVMNf0N8w2Idu4OkBiS2edg/70A8/OqV
U0JVANH5qxc=
=sd/Q
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

>
>
> But, there are some things Reiser does better and faster than ext3, even
> if you don't count file-as-directory and other toys.  There's nothing
> ext3 does better than Reiser, unless you count the compatibility with
> random bootloaders and low-level tools.

In fairness, there are some things that I am aware of.  In Reiser4 fsync
performance needs optimizing, truly random modifications and anything
else that write twice logging is best for needs optimizing, workloads
dominated by dirtying more mmap'd pages than can fit into RAM need
optimizing.

>
> >>                                               Not everyone will want
> >>to reformat at once, but as the reiser4 code matures and proves itself
> >>(even more than it already has),
>
>
> >I for one have seen mainly people with wild claims that it will make
> their
> >machines much faster, and coming back later asking how they can recover
> >their thrashed partitions...
>
>
> You know how many I've had thrashed on Reiser4?  Two.  The first one was
> with a VERY early alpha/beta, and the second one was when I dropped a
> laptop and the disk failed.
>
> And it does make certain things faster.  For one thing, "emerge sync" on
> Gentoo is twice to four times as fast, and /usr/portage is 75% as big,
> as on ReiserFS (3).

Wow.  You know, Nikita complains about my benchmarks, but what I usually
hear from users is that their performance when they use their favorite
app is mostly similar to my benchmarks.

I would be really happy if our compression code was working before we
went in.  Half the space and thus twice the speed to disk.  Edward could
use some help with it.  If only we could work on that plus fsync plus
mmap dirtying plus random modification optimized journaling rather than
these flamefests. 

Hans

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:
> Horst von Brand wrote:
> > David Masover <ninja@slaphack.com> wrote:
> >>Hans Reiser wrote:
> >>>Jeff Garzik wrote:

> > [...]

> >>>You missed his point.  He is saying ext3 code should migrate towards
> >>>becoming reiser4 plugins, and reiser4 should be merged now so that the
> >>>migration can get started.

> >>Sort of.
> >>
> >>I think ext3 would be nice as a reiser4 plugin.

> > What for? It works just fine as it stands, AFAICS.

> So does DOS.

I'm sorry?

>              Do you use DOS?

Good riddance, no! Not for something like 15 years.

>                               I don't even use DOS to run DOS programs.

Haven't seen one in quite some time, to be honest. Oops, sorry, I just
lied. Booted it from floppy to run a program to whack the BIOS password on
a PC some two weeks ago. Only use in a long time.

> "Ain't broke" is the battle cry of stagnation.

I see it as the battle cry of those that are looking for /real/ problems to
solve.

> But, there are some things Reiser does better and faster than ext3,

Yes, I've heard it is supposed to be faster on huge directories, and
doesn't run out of inodes. And it is more efficient spacewise on small
files. But then again, space is extremely cheap today...

And again, on a list around here I've seen several cries for help with
completely hosed filesystems, all ReiserFS. No solution has ever come
forth.

>                                                                     even
> if you don't count file-as-directory and other toys.  There's nothing
> ext3 does better than Reiser, unless you count the compatibility with
> random bootloaders and low-level tools.

For me, those are quite critical...

> >>                                                Not everyone will want
> >>to reformat at once, but as the reiser4 code matures and proves itself
> >>(even more than it already has),

> > I for one have seen mainly people with wild claims that it will make their
> > machines much faster, and coming back later asking how they can recover
> > their thrashed partitions...

> You know how many I've had thrashed on Reiser4?  Two.  The first one was
> with a VERY early alpha/beta, and the second one was when I dropped a
> laptop and the disk failed.

OK. Know how many I thrashed with ext2/3? I remember 3, could have been as
many as 5. One was due to a failed disk, another one because of DMA to a
disk causing slow corruption. Another one I believe was due to a odd kernel
compiled with a snapshot gcc a long time back, plus power loss at the wrong
time. And that is from using ext2/3 since they were in early beta. On
several machines at the same time, over years. I'd have to say that there
isn't much of a difference.

> And it does make certain things faster.  For one thing, "emerge sync" on
> Gentoo is twice to four times as fast, and /usr/portage is 75% as big,
> as on ReiserFS (3).

That can't all be due to on-disk format.

> >>                                 the ext3 people may find themselves
> >>wanting some of the more generic optimizations.

> > They'll filch them in due time, don't worry.

> Duplication of effort.  With plugins, we can optimize the upper layers
> of ALL filesystems, regardless of the lower layers, in such a way that
> it is optional.

Generic optimizations how, if they need VFS support?!

>                  I'm sure it's far easier to write a Reiser storage
> plugin than a brand new FS.

Comparing apples and oranges tells you what?

> Eventually, once competition is only based on storage format, we could
> end up with just one format.  Just one filesystem!  (except for
> fat/ntfs/iso/udf/network...)  And in the open source world, sometimes a
> single product is a good thing.

Sorry, I don't think this will come to pass in our lifetime, if ever. There
are different requirements, and the way to cater to them is different
solutions. That has always been what Linux (as opposed to the propietary
and even some open source systems) is all about...

> >>But, I don't think that will realistically happen at all.

> >>Instead, what will probably happen is that once Reiser4 is in the
> >>mainstream kernel, it will become more popular and noticable.  Other
> >>FSes will take note.  ext3 people may decide they want
> >>file-as-directory,

> > That idea is even much older than Linux itself, and no other (Unix)
> > filesystem has implemented it. Ever. Wonder why...

> >>                   and vfat people may decide they want cryptocompress,

> > I'm sure they don't, because it is mostly for Windows and assorted devices
> > (pendrives, digital cameras, ...) compatibility.

> I, for one, would like to use a pendrive and have certain files be
> encrypted transparently, only for use on Linux, but others be ready to
> transfer to a Windows box.

And that would surely break Windows compatibility (because you have to keep
the data on what to encrypt one the filesystem itself). Besides, having
pgp, or gpg, or crypt, or my own whacky encryption proggie do the job in
/userland/, and not shoving into the kernel and so down the next user's
throat, is in the end a largeish part of what Unix is all about for me.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Hubert Chan]

On Thu, 23 Jun 2005 22:41:01 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:

> David Masover <ninja@slaphack.com> wrote:
>> I, for one, would like to use a pendrive and have certain files be
>> encrypted transparently, only for use on Linux, but others be ready
>> to transfer to a Windows box.

> And that would surely break Windows compatibility (because you have to
> keep the data on what to encrypt one the filesystem itself).

Easily solved, using the same method OS/2 added Extended Attribute
support to FAT.  Of course, things may break if you try to manipulate
the encrypted file under Windows (e.g. deleting those files would leave
some garbage blocks behind), but breakage should be minor and relatively
easy to fix.

> Besides, having pgp, or gpg, or crypt, or my own whacky encryption
> proggie do the job in /userland/, and not shoving into the kernel and
> so down the next user's throat, is in the end a largeish part of what
> Unix is all about for me.

Meh.  It's nice to have encrypting to be transparent.  Unless your
editor supports gpg/crypt/etc., you'll need to decrypt the file to a
temporary area, edit the file, and then re-encrypt.  Much easier to make
a mistake and lose data, or leave traces of the cleartext data.

Besides, the encryption would be optional, and hence not being shoved
down the user's throat.  If you don't like it, don't use it.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
> David Masover <ninja@slaphack.com> wrote:
> 
>>Horst von Brand wrote:
>>
>>>David Masover <ninja@slaphack.com> wrote:
>>>
>>>>Hans Reiser wrote:
>>>>
>>>>>Jeff Garzik wrote:
> 
> 
>>>[...]
> 
> 
>>>>>You missed his point.  He is saying ext3 code should migrate towards
>>>>>becoming reiser4 plugins, and reiser4 should be merged now so that the
>>>>>migration can get started.
> 
> 
>>>>Sort of.
>>>>
>>>>I think ext3 would be nice as a reiser4 plugin.
> 
> 
>>>What for? It works just fine as it stands, AFAICS.
> 
> 
>>So does DOS.
> 
> 
> I'm sorry?
> 
> 
>>             Do you use DOS?
> 
> 
> Good riddance, no! Not for something like 15 years.

[...]

>>"Ain't broke" is the battle cry of stagnation.
> 
> 
> I see it as the battle cry of those that are looking for /real/ problems to
> solve.

I'll refer you to my other rant about stagnation and oil.

And, listen to yourself.  "Good riddance, no, I don't use DOS" but
"Ain't broke is the battle cry of those looking for real problems to solve."

You can solve real problems in DOS, it's usable, it ain't broke, there
are even some decent games (doom) and windowing systems (win3.1 and
others) for it.

But Linux is better.  DOS ain't broke, but Linux is better.  So maybe
VFS ain't broke, but plugins would be better.  I guess we'll only know
if we let Reiser4 merge...

>>But, there are some things Reiser does better and faster than ext3,
> 
> 
> Yes, I've heard it is supposed to be faster on huge directories, and
> doesn't run out of inodes. And it is more efficient spacewise on small
> files. But then again, space is extremely cheap today...

Speed isn't.  CPU is, but not disk speed.  And packing stuff more
efficiently, without actually compressing it, will give you some of that
speed.

Also, space is not so cheap that I won't take 25% more.

> And again, on a list around here I've seen several cries for help with
> completely hosed filesystems, all ReiserFS. No solution has ever come
> forth.

I haven't been counting, but I've seen a number of solutions fly around
reiserfs-list for people with reiser4 problems.

>>                                                                    even
>>if you don't count file-as-directory and other toys.  There's nothing
>>ext3 does better than Reiser, unless you count the compatibility with
>>random bootloaders and low-level tools.
> 
> 
> For me, those are quite critical...

There have been patches...

>>>>                                               Not everyone will want
>>>>to reformat at once, but as the reiser4 code matures and proves itself
>>>>(even more than it already has),
> 
> 
>>>I for one have seen mainly people with wild claims that it will make their
>>>machines much faster, and coming back later asking how they can recover
>>>their thrashed partitions...
> 
> 
>>You know how many I've had thrashed on Reiser4?  Two.  The first one was
>>with a VERY early alpha/beta, and the second one was when I dropped a
>>laptop and the disk failed.
> 
> 
> OK. Know how many I thrashed with ext2/3? I remember 3, could have been as
> many as 5. One was due to a failed disk, another one because of DMA to a
> disk causing slow corruption. Another one I believe was due to a odd kernel
> compiled with a snapshot gcc a long time back, plus power loss at the wrong
> time. And that is from using ext2/3 since they were in early beta. On
> several machines at the same time, over years. I'd have to say that there
> isn't much of a difference.
> 
> 
>>And it does make certain things faster.  For one thing, "emerge sync" on
>>Gentoo is twice to four times as fast, and /usr/portage is 75% as big,
>>as on ReiserFS (3).
> 
> 
> That can't all be due to on-disk format.

It's not compression.  What do you think it is?

To be fair, it's gotten a bit slower recently.  Wish that repacker was
done.  I think it's actually gotten just a little slower than it ever
was on v3, but v3 was awhile ago for me, and Gentoo has been growing a
lot.  /usr/portage is probably twice as big now as it was then.

If you want, I can do some unscientific benchmarks.

>>>>                                the ext3 people may find themselves
>>>>wanting some of the more generic optimizations.
> 
> 
>>>They'll filch them in due time, don't worry.
> 
> 
>>Duplication of effort.  With plugins, we can optimize the upper layers
>>of ALL filesystems, regardless of the lower layers, in such a way that
>>it is optional.
> 
> 
> Generic optimizations how, if they need VFS support?!

This was about a hypothetical ext3 format as a reiser4 storage plugin.
I'm not sure how this ties into the VFS stuff.

>>                 I'm sure it's far easier to write a Reiser storage
>>plugin than a brand new FS.
> 
> 
> Comparing apples and oranges tells you what?

That I'd rather have oranges?

A lot of what people like about ext3 is its stability and fairly
universally accepted format.  A lot of what people like about XFS is its
stability and speed, mainly with large files.  A lot of what people like
about Reiser4 (as it is today) is its speed, with large and especially
with small files.

Those are broad and somewhat uneducated statements, but I doubt most
people care what FS they are using if the stability and performance is
what they want.  In that case, why have so much duplicated effort
between different filesystems -- even between ISO and UDF and Reiser and
XFS -- when most of what's really different is the on-disk format and
the optimization?

So, in this hypothetical situation where ext3 is a reiser4 plugin,
suddenly all the ext3 developers are trying to improve the speed and
reliability of reiser4, which benefits both ext3 and reiser4, instead of
just ext3.

>>Eventually, once competition is only based on storage format, we could
>>end up with just one format.  Just one filesystem!  (except for
>>fat/ntfs/iso/udf/network...)  And in the open source world, sometimes a
>>single product is a good thing.
> 
> 
> Sorry, I don't think this will come to pass in our lifetime, if ever. There
> are different requirements, and the way to cater to them is different
> solutions. That has always been what Linux (as opposed to the propietary
> and even some open source systems) is all about...

egrep and fgrep are the same program.  A lot of the code is shared, I'm
sure.  But you change some settings and they suddenly solve entirely
different problems.

You don't need a wholly different solution if you can adapt an existing
one.  That's what plugins are all about -- adapting one solution to any
number of requirements, so that people can focus on making that one
solution better.

But, you may be right about it not happening in our lifetime, as I said:

>>>>But, I don't think that will realistically happen at all.

People are stubborn.

[...]

>>>>                  and vfat people may decide they want cryptocompress,
> 
> 
>>>I'm sure they don't, because it is mostly for Windows and assorted devices
>>>(pendrives, digital cameras, ...) compatibility.
> 
> 
>>I, for one, would like to use a pendrive and have certain files be
>>encrypted transparently, only for use on Linux, but others be ready to
>>transfer to a Windows box.
> 
> 
> And that would surely break Windows compatibility (because you have to keep
> the data on what to encrypt one the filesystem itself). Besides, having

Aside from what someone else already said about this, why not just have
support for accessing, say, a .gpg file as transparently decrypted?  You
don't even need file-as-directory, just create a file called foo which
is really the decrypted version of foo.gpg.  No need to change the
format, just the filesystem.

> pgp, or gpg, or crypt, or my own whacky encryption proggie do the job in
> /userland/, and not shoving into the kernel and so down the next user's
> throat, is in the end a largeish part of what Unix is all about for me.

Do you use ipv6?  I don't.  And it's not shoved down my throat, either,
although it's in the kernel.  I simply disable it, and you would (I'm
sure) disable the crypto stuff.

Plus, as someone else said, it's much easier to do
$ vim /some/encrypted/file
than
$ gpg --decrypt /some/encrypted/file > /some/decrypted/file
$ vim /some/decrypted/file
$ gpg --encrypt /some/decrypted/file > /some/encrypted/file
$ shred /some/decrypted/file

Not to mention, shred doesn't work on modern filesystems, so you need to
either patch vim (and any other program you might want to use), create
an actual ramdisk, or deactivate swap and mount a tmpfs.



On doing stuff in userland:  I am all for moving tons of stuff to
userland, but not until someone does a microkernel right, if it's even
possible.  And even then, I'd probably still use Linux and argue for
some stuff in the kernel, because Linux has developers.  Developers.
Developers.  Developers.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzZOngHNmZLgCUhAQLk/w//dwfRaR5mtrznECpLMtpfNTm/n0EK2gqC
ABYClsvcyKPzFtRTYnhZmBmVvAIueJpaReRmrR3QryESoeD6zftrxpuyvgcxxFRA
BUIZcXKtqDF7ILmwCgSN0kB3ltD859q0E9ceJz5/lQZJwIJ2JN1huDM9xsOtw4sW
5HZEwSd6C46HArHD5fnnljppCgxV9ozVDfVzWs2GHU9r9cYZ+y9FU/kWIiuw78Ub
fSSF7UqXd+QP0PWkkC7bAA9EtePT4Xd875WX7etgUmJRqVbf6erzPNq34b8V4r0r
6o2QFd5j4ZjpUvAeUzlMivSwFl10Yta+I9E3MHkYIm/86pX3GmPY5W24YfkDKsdS
Y1odr3HYHt16QIJF3MPO3724PsSax+c9vmF7ANU+2QadtyQFrU0aasKxehTPnp9M
PjShkWpWLGCG2YyesM3lq2HzydbljBraFUfzK9LKKoTqBk0jd6kcUrP8869NPo94
/stcyLSa/1gSAsSX+97Gs8CpQZ9JeJp7cMyDdrbR68h0xAg+UGK7tDez5j7VY9kc
cSNo+T+W8OBiRPfz7Zg39ltHIeaPT/8THPKRmOLHWXK2duW+RmZBTDPK22Cwbv83
0f9RgRtm1q1FYG1q4gH373V/hjmvZIXGrN+/I5mU6P43iVrLuz/7MIWAGkQyXS3M
nOCoZV7WiHY=
=XUC1
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 3628 bytes --]

On Fri, 24 Jun 2005 23:10:35 CDT, David Masover said:

> But Linux is better.  DOS ain't broke, but Linux is better.  So maybe
> VFS ain't broke, but plugins would be better.  I guess we'll only know
> if we let Reiser4 merge...

No, we'll only know if we merge something that does plugins at the VFS
level in a well-designed way.

> This was about a hypothetical ext3 format as a reiser4 storage plugin.
> I'm not sure how this ties into the VFS stuff.

Very poorly.  There's only two interpretations of "ext3 as a reiser4 plugin"
that make *any* sense.  The first is that reiser4 is totally violating the VFS
layer boundary, and the second is that reiser4 is trying to be an all-singing
all-dancing wankfest.  Later on, you say:

> A lot of what people like about ext3 is its stability and fairly
> universally accepted format.  A lot of what people like about XFS is its
> stability and speed, mainly with large files.  A lot of what people like
> about Reiser4 (as it is today) is its speed, with large and especially
> with small files.

Now *think* for a moment - how does a hypothetical Reiser4 using ext3 format
gain any speed advantage with small files, when the speed advantage is based
on using a format other than ext3?

As I said, either it's violating the VFS boundary, or it's busy wanking.

The Reiser4 proponents would be well served to disavow that particular
hypothetical example - I have yet to see *anything* that does more damage
to the Reiser4 cause.

> So, in this hypothetical situation where ext3 is a reiser4 plugin,
> suddenly all the ext3 developers are trying to improve the speed and
> reliability of reiser4, which benefits both ext3 and reiser4, instead of
> just ext3.

Or we can do what *should* be done, which is:

a) Put the crack pipe down.

b) Tell reiser4 to get its grubby little paws off the VFS if it ever intends
to have a chance of being merged in mainline.

c) Have a *separate* project to improve the speed/reliability/function of
the VFS layer, which is the only way that your vision of having the ext3 and
reiser developers cooperating will ever happen.

Yes, the VFS could probably use an overhaul.  But that *will* happen like this:

1) A patch is submitted and passes review to change the VFS.
2) If appropriate, a patch for reiser4 (if it gets merged) is also submitted
(possibly by the same people) to be the first user of the new API/functionality.

There's a *reason* why we see patch streams that look like:

Patch 1/3: Add moby_foo_init function to nautical core.
Patch 2/3: Modify white_whale driver to use moby_foo_init
Patch 3/3: Modify captain_ahab driver to use moby_foo_init

> Aside from what someone else already said about this, why not just have
> support for accessing, say, a .gpg file as transparently decrypted?  You
> don't even need file-as-directory, just create a file called foo which
> is really the decrypted version of foo.gpg.  No need to change the
> format, just the filesystem.

I don't think this is what they mean by "Linux gives you enough rope to
shoot yourself in the foot with"...

> Plus, as someone else said, it's much easier to do
> $ vim /some/encrypted/file
> than
> $ gpg --decrypt /some/encrypted/file > /some/decrypted/file
> $ vim /some/decrypted/file
> $ gpg --encrypt /some/decrypted/file > /some/encrypted/file
> $ shred /some/decrypted/file

You've totally failed to understand that the whole *point* of PGP is that 'vim
/some/encrypted/file' *isnt* easy to do.  A better example might be the various
crypto-loop-ish variants or Microsoft's EFS, where the key management model is
more tractable to this sort of automation.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Fri, 24 Jun 2005 23:10:35 CDT, David Masover said:
> 
> 
>>But Linux is better.  DOS ain't broke, but Linux is better.  So maybe
>>VFS ain't broke, but plugins would be better.  I guess we'll only know
>>if we let Reiser4 merge...
> 
> 
> No, we'll only know if we merge something that does plugins at the VFS
> level in a well-designed way.

Which will never happen, the way you see it.  (see below.)

Or maybe you could see this in action if you look at the *current* code,
not the hypothetical future code where reiser4 plugins are removed from
the FS, which is then merged, then plugins are added to VFS, then back
into reiser...   (see below)

>>This was about a hypothetical ext3 format as a reiser4 storage plugin.
>>I'm not sure how this ties into the VFS stuff.
> 
> 
> Very poorly.  There's only two interpretations of "ext3 as a reiser4 plugin"
> that make *any* sense.  The first is that reiser4 is totally violating the VFS
> layer boundary, and the second is that reiser4 is trying to be an all-singing
> all-dancing wankfest.  Later on, you say:

You'll have to be more specific and less insulting.

Actually, the way I'd like to see that is if we

>>A lot of what people like about ext3 is its stability and fairly
>>universally accepted format.  A lot of what people like about XFS is its
>>stability and speed, mainly with large files.  A lot of what people like
>>about Reiser4 (as it is today) is its speed, with large and especially
>>with small files.
> 
> 
> Now *think* for a moment - how does a hypothetical Reiser4 using ext3 format
> gain any speed advantage with small files, when the speed advantage is based
> on using a format other than ext3?

It doesn't.  It gains in other ways.  Transparent cryptocompress,
oddball other plugins, not to mention some speed optimizations that
happen in RAM.  If you do a ton of work with a dataset that stays in
RAM, Reiser probably performs as well or better than a ramdisk, because
changes that get overwritten while still in RAM never actually touch the
disk.  Reiser also doesn't fragment as quickly as ext3, and I don't
think that has anything to do with its format.

> The Reiser4 proponents would be well served to disavow that particular
> hypothetical example - I have yet to see *anything* that does more damage
> to the Reiser4 cause.

It is my example.  I don't speak for Namesys.  I don't work at Namesys.

If it does "damage", that is only in the eyes of those not paying attention.

>>So, in this hypothetical situation where ext3 is a reiser4 plugin,
>>suddenly all the ext3 developers are trying to improve the speed and
>>reliability of reiser4, which benefits both ext3 and reiser4, instead of
>>just ext3.
> 
> 
> Or we can do what *should* be done, which is:
> 
> a) Put the crack pipe down.

That was unneccesary.  Come back when you can debate technical reasons,
not imaginary personal ones.

> b) Tell reiser4 to get its grubby little paws off the VFS if it ever intends
> to have a chance of being merged in mainline.

You are saying Reiser isn't in because it shouldn't touch VFS.  Every
single other person in this thread says Reiser isn't in because it
*should* touch VFS.

> c) Have a *separate* project to improve the speed/reliability/function of
> the VFS layer, which is the only way that your vision of having the ext3 and
> reiser developers cooperating will ever happen.

Why does it have to be a separate project if it is already *done* as
part of Reiser4?  Or is the name Reiser just cursed that way?

> Yes, the VFS could probably use an overhaul.  But that *will* happen like this:
> 
> 1) A patch is submitted and passes review to change the VFS.
> 2) If appropriate, a patch for reiser4 (if it gets merged) is also submitted
> (possibly by the same people) to be the first user of the new API/functionality.

The FS that gets merged ahead of time without plugins would no longer be
Reiser4.  Go read the whitepaper, or tell me why I'm wrong, but even
symlinks are implemented as plugins.

> There's a *reason* why we see patch streams that look like:
> 
> Patch 1/3: Add moby_foo_init function to nautical core.
> Patch 2/3: Modify white_whale driver to use moby_foo_init
> Patch 3/3: Modify captain_ahab driver to use moby_foo_init

I'm not arguing that at all.  But if you've got an entirely new driver,
why not do:

Patch 1/2:  Add white_whale driver, which also adds moby_foo_init to
nautical core.
Patch 2/2:  Modify captain_ahab driver to use moby_foo_init

>>Plus, as someone else said, it's much easier to do
>>$ vim /some/encrypted/file
>>than
>>$ gpg --decrypt /some/encrypted/file > /some/decrypted/file
>>$ vim /some/decrypted/file
>>$ gpg --encrypt /some/decrypted/file > /some/encrypted/file
>>$ shred /some/decrypted/file
> 
> 
> You've totally failed to understand that the whole *point* of PGP is that 'vim
> /some/encrypted/file' *isnt* easy to do.  A better example might be the various
> crypto-loop-ish variants or Microsoft's EFS, where the key management model is
> more tractable to this sort of automation.

Actually, plugins are just as easy or easier than crypto-loop or
dm-crypt.  And why shouldn't my crypto be easy?  Most users are insecure
in all kinds of ways because of that attitude -- security is HARD, so I
won't do it.  If security is transparent, just enter a password and go,
then more people would do it.  If more people are secure in more ways,
there are less DDOS attacks and the world is a better place.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr2jd3gHNmZLgCUhAQIeihAAgifgKxYw2FECNKLsspQ7sMOmBaT1jcev
5CJdDCiDjiMEMMcsh1h749tZX38YTjqL/w8VQQ18ys/2QqRERbjNEn/lGWLjO4Hs
UwFFv/usVMv3RmRModH62P2j7QAxWhjTcfWYeIexdh20whMJLJA+BpIhnoKZ6Xjq
WQMbdc9X/NW/lS09zkthWUju8JXtjTjD5rsMF3yQ47Scijec2i67ATsnzNxdXB5/
fZmjOfWKkahHh8qdO/IXX/6nvyEEzmVThr1EskCjUwoRYUSfb9bLQMkntjvEWSDh
mDIBbwAUg88k3dM947uS55iV3QGoVkgDRvF0cpWxeaOEP+FcKmF9gPiaW0wm5jYw
n60tw7P48FVchBrAPZTwvq5po78Evj3bB09NmS3i68in56rhRI3hTB7KNR/vIbS1
9dKvYuilV0C+thGD+sfE81qqFyP/O66sjvpxRi/YhjL8hU8N3x2f5tTIkKli7Y2C
bZSxCiLiDRzKYAgCq0Rr+mrn/TZb5Ld5W8Ys062qF5l00M0I6cDO0sPiH72RexM7
PMM5tWNBDqVacUWxNYReULC5O7Q6c8FYTWcZz/zC1boLTeGpvlI8uVkbxP6K/grY
j1u7vHbjgAqsztYhA8qmw3nl5vHtXCGHCRFQJ36g14Qt3atFcMgvfE5yphWyC1P9
hYZm8EWvMqc=
=GvIJ
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 4924 bytes --]

On Sat, 25 Jun 2005 13:33:27 CDT, David Masover said:
> > Now *think* for a moment - how does a hypothetical Reiser4 using ext3 format
> > gain any speed advantage with small files, when the speed advantage is based
> > on using a format other than ext3?

> happen in RAM.  If you do a ton of work with a dataset that stays in
> RAM, Reiser probably performs as well or better than a ramdisk, because
> changes that get overwritten while still in RAM never actually touch the
> disk.

At that point, since the actual buffer management is being done at the VFS
level (see fs/buffer.c and friends) what you're really comparing is the speed
of journalling metadata - at which point you need to be *very* careful to
specify exactly what configuration you're talking about.  If you don't believe
me, investigate why mounting a filesystem with 'noatime,nodiratime' can make a
dramatic difference totally independent of the underlying filesystem, but the
actual amount of gain is dependent on format (hint - how far do the heads have
to move to record 3 atime updates against 3 random inodes on an ext2, an ext3,
and a VFAT filesystem, assuming no other disk activity), and why
ext3 has 3 different modes data=ordered/writeback/journal.

>        Reiser also doesn't fragment as quickly as ext3, and I don't
> think that has anything to do with its format.

Care to explain why it's not format-dependent? 

> > b) Tell reiser4 to get its grubby little paws off the VFS if it ever intends
> > to have a chance of being merged in mainline.
> 
> You are saying Reiser isn't in because it shouldn't touch VFS.  Every
> single other person in this thread says Reiser isn't in because it
> *should* touch VFS.

Hmm.. let's see.. I said Reiser isn't in because it shouldn't be screwing with
the VFS, and said stuff should be done separate from the Reiser4 filesystem.

Everybody else said that to achieve all the goals that Hans wants will require
changes to the VFS, and the way reiser4 gets there isn't acceptable.

Seems like myself and everybody else are saying this needs to be factored into
2 pieces, a FS piece and a VFS piece, and moved forward separately.

> > c) Have a *separate* project to improve the speed/reliability/function of
> > the VFS layer, which is the only way that your vision of having the ext3 and
> > reiser developers cooperating will ever happen.
> 
> Why does it have to be a separate project if it is already *done* as
> part of Reiser4?  Or is the name Reiser just cursed that way?

Because it's done *as a part of reiser4*, and not as a separately reviewed
change to the VFS.

> The FS that gets merged ahead of time without plugins would no longer be
> Reiser4.  Go read the whitepaper, or tell me why I'm wrong, but even
> symlinks are implemented as plugins.

Which is another way of saying Reiser4 can't be merged in its present form.

> I'm not arguing that at all.  But if you've got an entirely new driver,
> why not do:
> 
> Patch 1/2:  Add white_whale driver, which also adds moby_foo_init to
> nautical core.

You don't do this because the rule is "one patch, one logical change", and
"which also" implies more than one change.

> Actually, plugins are just as easy or easier than crypto-loop or
> dm-crypt.  And why shouldn't my crypto be easy?  Most users are insecure
> in all kinds of ways because of that attitude -- security is HARD, so I

There's a vast distinction between "easy for implementors" and "easy for
users".  Jaari Russo's loop-aes stuff does a wonderful job of being "easy for
users" - just say "mount", answer the passphrase, and you're good to go.  The
underlying arguing about the crypto involved is complicated enough keep
professional crypto jocks busy for years (is the watermark attack Jaari is
concerned about a real threat?  You tell me. ;)

Meanwhile, PGP was designed to be used in an environment where you could do
this:  "Today's secret plans are AES256 encrypted.  The key is the next key in
your one-time-pad book, XOR'ed with your 128-bit secret key - do it in your head".
(And yes, you can easily memorize a 16-digit hex number and learn to do an XOR
with another 16-digit number, if failing to do so means you could end up dead).

This is inconvenient for the user, but intractable for an attacker to create a
scenario where they can just 'vi /each/decrypted/file' ;)

> won't do it.  If security is transparent, just enter a password and go,
> then more people would do it.

"Just enter a password and go, then more people would do it".

Two words: "phishing e-mail".

Why does phishing work at all?  Because it's simple for the user, and the
user isn't aware of the totally busticated underlying security model of
SMTP (namely, there isn't one) and the mostly busticated security model of
most browsers (the misleading concept that if an SSL site is identified
by the SSL cert, that this implies the site can be trusted, and other similar
misfeatures).


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[Hans Reiser]

Valdis.Kletnieks@vt.edu wrote:

> Hmm.. let's see.. I said Reiser isn't in because it shouldn't be
> screwing with
>
>the VFS, and said stuff should be done separate from the Reiser4 filesystem.
>  
>
We don't touch a line of VFS code.  We look like a normal fs at the
interface.

Shall we send in a file labeled reiservfs.c containing the plugin layer?
That will really test whether the Reiser name is cursed that way, yes?

There has been no response to the technical email asking for what
exactly it is that is duplicative, and what exactly it is that ought to
be changed in how the code works, as opposed to what file the code is
placed in, or who is considered its maintainer.    There has been no
response to the questions about whether the difference between class and
instance makes our layer non-duplicative.

Perhaps no response was possible?

Hans

Re: reiser4 plugins

[Lincoln Dale]

Hans Reiser wrote:

>There has been no response to the technical email asking for what
>exactly it is that is duplicative, and what exactly it is that ought to
>be changed in how the code works, as opposed to what file the code is
>placed in, or who is considered its maintainer.    There has been no
>response to the questions about whether the difference between class and
>instance makes our layer non-duplicative.
>
>Perhaps no response was possible?
>  
>
Hans,

the l-k community have asked YOU may times.  any lack of response isn't 
because of the kernel cabal .. its because YOU are refusing to entertain 
any notion that what Reiser4 has implemented is unpalatable to the 
kernel community.

you can threaten all you want to take your code and move it to BSD.  or 
fork the kernel.  whatever.
but if you want to get your work into the mainline kernel, you need to 
provide answers to the question that keeps being asked of you - and 
which you patently keep ignoring time & time again.

in short,  as per Message-ID: <42BBC710.8010906@cisco.com>:
    posting to l-k on "why Reiser4 cannot use VFS infrastructure for
    [crypto,compression,blahblah] plugins" - ideally, for each plugin.

or again, in Message-ID: <42BBB1FA.7070400@cisco.com>:
    [..] but instead just understand that this is the framework that you 
have to
    work in to get it into the mainline kernel.
    if you don't want to go down that path, you're free to do so.
    its open source, you can keep your own linux-kernel fork.
    but if you want to get your code into mainline, i don't think its
    so much a case of l-k folks telling you how to make your code
    work under VFS.  the onus is on you to say WHY your code
    and plugins could never be put into VFS.

or further back in Message-ID: <42BB7083.2070107@cisco.com>
    you know that VFS is the mechanism in Linux.  you know
    (i hope..) how it works.  it isn't so hard to see how many
    of the Reiser4 "plug-ins" could be tied into VFS calls.
    OR, if they cannot TODAY, propose how VFS _COULD_ be
    made to do this.


NB. it doesn't matter what David thinks.  this is linux-kernel, not 
linux-users.


cheers,

lincoln.

>  
>

Re: reiser4 plugins

[Gregory Maxwell]

On 6/26/05, Lincoln Dale <ltd@cisco.com> wrote:
> the l-k community have asked YOU may times.  any lack of response isn't
> because of the kernel cabal .. its because YOU are refusing to entertain
> any notion that what Reiser4 has implemented is unpalatable to the
> kernel community.

A lot of this is based on misconceptions, for example in recent times
reiser4 is faulted for layering violations.. But it doesn't have them,
it neither duplicates nor modifies the VFS.

It has also been requested that reiser4 be changed to move some of
it's operations above the VFS... not only would that not make sense
for the currently provided functions, but merging was put off
previously because of changes to the VFS.... now that it doesn't
change the VFS we are asking hans to push it off until it does??

It's a filesysem for gods sake. Hans and his team have worked hard to
minimize its impact and they are still willing to accept more
guidance, even if their patience has started to run a little thin.  
The acceptance of reiser4 into the mainline shouldn't be any larger
deal than any other filesystem, but yet it is...

Re: reiser4 plugins

[Lincoln Dale]

Gregory Maxwell wrote:

>On 6/26/05, Lincoln Dale <ltd@cisco.com> wrote:
>  
>
>>the l-k community have asked YOU may times.  any lack of response isn't
>>because of the kernel cabal .. its because YOU are refusing to entertain
>>any notion that what Reiser4 has implemented is unpalatable to the
>>kernel community.
>>    
>>
>
>A lot of this is based on misconceptions, for example in recent times
>reiser4 is faulted for layering violations.. But it doesn't have them,
>it neither duplicates nor modifies the VFS.
>
>It has also been requested that reiser4 be changed to move some of
>it's operations above the VFS... not only would that not make sense
>for the currently provided functions, but merging was put off
>previously because of changes to the VFS.... now that it doesn't
>change the VFS we are asking hans to push it off until it does??
>  
>
<sigh>

it has NEVER been a case of Reiser4 not being merged because "it 
required changes to VFS".

the whole point of VFS is to provide a standard API for data to/from 
individual filesystems.
over the course of history, VFS itself hasn't been a static thing - it 
has had to adapt and change as a result of the needs of filesystems.
but it hasn't ever been a case of individual filesystems doing 
'proprietary' things (i.e. there isn't a sys_ext3() system call) - where 
it has made sense to do things at a VFS layer, VFS itself has been 
adapted to handle those things.

a semi-recent example of this is extended attributes.
it is with some irony that on my desktop i make use of the excellent 
open-source desktop search tool 'beagle'.
it, however, uses extended attributes for storing things - and Reiser4's 
EAs are incompatible with the "standard" EAs such that Reiser4 is 
incompatible with beagle.

this is the WHOLE point of standardization .. i don't think its that 
Reiser4's EAs offer any more or less capabilities than standard EAs - 
BUT they haven't used the standard mechanisms available for implementing 
them, such for Beagle to work on Reiser4, there now needs to be logic 
added to Beagle to do so.

lets take this a step further.  what about compression?  do we accept 
that each filesystem can implement its own proprietary compression via 
its own API - and now we need individual user-space tools to understand 
each of these APIs?
how about encryption?
... and so-on.
suddenly every user app out there needs to have specialized knowledge of 
each type of filesystem.

Hans should be applauded for the 'plug-in' concept and showing how it 
can be used.  however, from an implementation stand-point, it really 
shouldn't come as any great surprise that numerous kernel developers are 
pushing back saying 'layering violation' and "why can't this be done at 
the VFS layer".

none of this is rocket-science.  its just plain common sense.

>It's a filesysem for gods sake. Hans and his team have worked hard to
>minimize its impact and they are still willing to accept more
>guidance,
>
i don't see any acceptance at this point.  simply lots of hot air that 
smells like marketing & PR.


cheers,

lincoln.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lincoln Dale wrote:

[...]

> this is the WHOLE point of standardization .. i don't think its that
> Reiser4's EAs offer any more or less capabilities than standard EAs -

They do.  Reiser4's EAs can look like any other object -- files,
folders, symlinks, whatever.  This is important, especially for
transparency.

For one thing, can I access a Beagle search as a folder?

> BUT they haven't used the standard mechanisms available for implementing
> them, such for Beagle to work on Reiser4, there now needs to be logic
> added to Beagle to do so.

Well, ideally, I'd like to see people stop bickering, come up with
something better than sys_reiser4, add an emulation layer for xattrs and
mark them obsolete.

But I don't speak for Namesys.

> lets take this a step further.  what about compression?  do we accept
> that each filesystem can implement its own proprietary compression via
> its own API - and now we need individual user-space tools to understand

No, that's the beauty of these "EAs" in Reiser4.  The API is standard
write(2) commands.  sys_reiser4 supposedly implements an interface to
make this scale better, but otherwise have the same semantics.  And who
said anything about proprietary compression?  I think we were planning
on the kernel's zlib, though we might have been planning to make it a
bit more seekable...

> each of these APIs?

So, the API becomes something like:

cat crypto/inflated/foo		# transparently decompressed
cat crypto/raw/foo.gz		# raw, gzip-compressed

Another possibility, if you like file-as-a-directory:

cat foo.gz			# raw
cat foo.gz/inflated		# decompressed

One could easily imagine things like these two potentially equivalent
commands:

cp foo bar.zip/
zip bar foo

The whole point is to have less userland tools, not more.  I'm not
saying we move zip into the kernel, just that the user now has one less
command to remember.

But, back to reality.  file-as-directory probably won't happen, at least
not for awhile, so imagine more along the lines of my first example.

> how about encryption?

About the same, only now you have a key file that you write to in order
to unlock the decrypted files.

> ... and so-on.
> suddenly every user app out there needs to have specialized knowledge of
> each type of filesystem.

Not really.

More like, every app that cares to has generalized knowledge, if that.

> none of this is rocket-science.  its just plain common sense.

I could say the same of my stuff, but lots of people seem to disagree
with me, or at least fail to see it.  I guess I can say the same of your
comments.

>> It's a filesysem for gods sake. Hans and his team have worked hard to
>> minimize its impact and they are still willing to accept more
>> guidance,
>>
> i don't see any acceptance at this point.  simply lots of hot air that
> smells like marketing & PR.

They do keep asking for specifically what they need to do to put this
stuff in VFS.  Or am I wrong?

Or maybe it should be obvious to them?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr5dtngHNmZLgCUhAQIBGxAAiLK6EyHnLRhEA+rUIDCwacM4K89wlE7X
+dcw3xv3Pc9tZZqVVAd7Y27whEzjmNOwfGkvPkzPk/ATQditnyt+7xHcuXpqORNU
j7zHc5zS8MGDRU8Re4MXTO6jCXDgtTwQHjcdg4i8KYWLMPT7LpO+DHY/mZyQEgpD
kZGE4WJePA+aNlHAzySW9u/atnwp5hSRvmfuF4zzN8ng5tf8SSMvbfoCyjYSue8l
N6jvcGnt+yItmbHVaij0IdHUw1/9/u6b3Q0Ut39NBk8fUKXJcASHmKtjwLTAoWW+
hiYVhLdZQGkWo2d6XdzdNY2OgE3kWVnLBqrOuTo7zCjMojvWIrEGE/x3Yh/E6Hs8
cAPVRebG5yUQBxJk1lcDeJOBozutIpCVyTzwBnKU1nz3KArqanU51oT++3cTjVha
1tBnaLS4RLcdy8UD1ewS+VHj61VSlcBjv2abCrYw4DC0anUFrYUSciNjx3tdYJRx
o7l/pEn7UYPpGaXgHyBdVDIRlRNdOoTRZp3aIY2Z2v6/jyi3TeufMUjGtpuQHl2k
BuYm7tV4l1Ec/QZLM+PbAyVU9qqlz9BuHlI1U7z1p3gYeAzz0guAeDHfi1l95sUn
l+bFCOfmXi3qRAxVZyidczqeOtGtCed3nIUH+1z+siuFzH3jecjzUpGWKcGxzVlc
jkUS+tlihfg=
=C4UP
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Lincoln Dale]

David Masover wrote:

>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Lincoln Dale wrote:
>
>[...]
>
>  
>
>>this is the WHOLE point of standardization .. i don't think its that
>>Reiser4's EAs offer any more or less capabilities than standard EAs -
>>    
>>
>
>They do.  Reiser4's EAs can look like any other object -- files,
>folders, symlinks, whatever.  This is important, especially for
>transparency.
>  
>
it was accepted not so long ago that 'file-as-directory' and 'EA' are 
two different things, predominantly because existing tools and apps 
won't necessarily "do the right thing<tm>".

this has been discussed to death previously.  oh what a surprise.  
http://lwn.net/Articles/100271/


cheers,

lincoln.


cheers,

lincoln.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lincoln Dale wrote:
> 
> 
> David Masover wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Lincoln Dale wrote:
>>
>> [...]
>>
>>  
>>
>>> this is the WHOLE point of standardization .. i don't think its that
>>> Reiser4's EAs offer any more or less capabilities than standard EAs -
>>>   
>>
>>
>> They do.  Reiser4's EAs can look like any other object -- files,
>> folders, symlinks, whatever.  This is important, especially for
>> transparency.
>>  
>>
> it was accepted not so long ago that 'file-as-directory' and 'EA' are
> two different things, predominantly because existing tools and apps
> won't necessarily "do the right thing<tm>".

They are different, but not quite -- what's the word -- discrete?
Rather, file-as-directory is one logical conclusion of EA, and one
possible representation of EA.

Anyway, Reiser's EAs currently (I think) could appear as files, but not
necessarily the file-as-directory madness we were talking about then.
They could also be accessed by openat.  And they can also be
hierarchical, which I would think makes them cleaner and faster.

> this has been discussed to death previously.  oh what a surprise. 
> http://lwn.net/Articles/100271/

I was there.  That was fun to watch, actually, once it got over my head.

The three points he lists are only relevent if you try to do
file-as-directory.  Right now, I like the idea of file attributes as
showing up in separate magic directories, or just creating magic
directories first and filling them with files.  Symlinks could help
clean up messiness, and allow you to do things like encrypt /etc/passwd
- -- symlink to /etc/crypto/passwd/decrypted or something.  Yes, I know
about shadow, and there are still some paranoid reasons to encrypt passwd.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr521XgHNmZLgCUhAQKSSA/9HK8JiG6wW40jQDUXsyKPz+l9UvoaNuLs
hqIXo3sdQj1CWAuwa1BKY16w91tJBN3uX75jLgwR36ix4A1jXQ5v1sRjvfkLKR/7
3a51v9UsRAhAisiqWFGWYTrbXgAh+S5B51xHuXv3qTs/wqC8sxuJtAHwuCldPaBr
cR1WzPtGvgd+ESqx5avllZIfNCy9tcH3P5fUsaIFCCm30E6u9PVO6xBOHylFWZKS
Pywv+wGUTxbgCFSmLG07/zhwVF94fAHPIWXQXQGmhPrGf3Wtt7VTMiRpkpyONyso
eoY1hFiwyh2YMrIPxdL0Uo+mcDvErWFFZyTcCGqIMp6x+QSe0Ww9Ie90afZPvvS0
Q4DmdST2JEHEinal5MCiqr3S83wanv3+h9ywTCEkTcl3mEK1iwtc4dwmvRNVHfkx
f34CAxM1rBfu++kd3xgL+Kb/ao4LOCDhVL3XY6pNDglX6Y+Kk5dRSJjOm4kAU2fB
j66uGOUZOiCCzSMLvVrCAPnNXmUavLKUXDKXKL2gTCYM6TL+7RkAXEMrqu5YxdZM
ihIbfmGc7FzItbQCDZhhozG51IMkLCotU9U9CaotnhfUaosibmwEnmeY8FWY75ba
MOs+VH3UJUZObRBwmBHSX4pwg5sDhSjyqPR05M26A5xz8nGh4kukD/E6QZYCB9EP
lVdDus4iJUM=
=hsOJ
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 3314 bytes --]

On Sun, 26 Jun 2005 02:48:06 CDT, David Masover said:

> Lincoln Dale wrote:

> > this is the WHOLE point of standardization .. i don't think its that
> > Reiser4's EAs offer any more or less capabilities than standard EAs -
> 
> They do.  Reiser4's EAs can look like any other object -- files,
> folders, symlinks, whatever.  This is important, especially for
> transparency.

No, you want them to look like the same objects that {get|set}xattr() manage
currently.  You don't want programs to have to guess what an EA looks like
this week, with this user's combination of plugins that's different from
everybody else's.

> > lets take this a step further.  what about compression?  do we accept
> > that each filesystem can implement its own proprietary compression via
> > its own API - and now we need individual user-space tools to understand
> 
> No, that's the beauty of these "EAs" in Reiser4.  The API is standard
> write(2) commands.  sys_reiser4 supposedly implements an interface to
> make this scale better, but otherwise have the same semantics.  And who
> said anything about proprietary compression?  I think we were planning
> on the kernel's zlib, though we might have been planning to make it a
> bit more seekable...
> 
> > each of these APIs?
> 
> So, the API becomes something like:
> 
> cat crypto/inflated/foo		# transparently decompressed
> cat crypto/raw/foo.gz		# raw, gzip-compressed

And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly?

Now throw some .bz2 and .zip files into the mix... ;)

> Another possibility, if you like file-as-a-directory:
> 
> cat foo.gz			# raw
> cat foo.gz/inflated		# decompressed
> 
> One could easily imagine things like these two potentially equivalent
> commands:
> 
> cp foo bar.zip/
> zip bar foo

Unless of course the user had done 'mkdir sorted.by.city.zip' to make
a directory of files containing data sorted by USPS Zip code.

And what happens if the user has a file 'bar' that's not a ZIP file,
and a directory 'bar.zip' isn't a view into 'bar'?

Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a
directory 'linux-2.6.12', what is under the directory is *NOT* the same
data as what's in the .bz2 - I've done 'make oldconfig' and a few builds
and some variable amount of patching, usually with rejects, and I *don't*
want that .bz2 being updated during all this (hint - what's my next command
after 'rm -rf linux-2.6.12' likely to be, and why, and  what expectations
do I have when I do it?)

You want to think this sort of thing through *really* thoroughly, because
there's a *lot* of things, both users and programs, that have expectations
about The Way Things Work.

> The whole point is to have less userland tools, not more.  I'm not
> saying we move zip into the kernel, just that the user now has one less
> command to remember.

But now instead of having to remember the one meme "I can manage any
compressed-archive format that's stored in a file, and put other files in it,
and all I need is the appropriate userspace tool", they have to remember "the
cp trick works for .zip and .tar, but I'll get a "not a directory" error if I
try it with a .hqx file, and that other file format may or may not work,
because I can't remember if this kernel has a working out-of-tree module for
this kernel...."


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Sun, 26 Jun 2005 02:48:06 CDT, David Masover said:
> 
> 
>>Lincoln Dale wrote:
> 
> 
>>>this is the WHOLE point of standardization .. i don't think its that
>>>Reiser4's EAs offer any more or less capabilities than standard EAs -
>>
>>They do.  Reiser4's EAs can look like any other object -- files,
>>folders, symlinks, whatever.  This is important, especially for
>>transparency.
> 
> 
> No, you want them to look like the same objects that {get|set}xattr() manage
> currently.  You don't want programs to have to guess what an EA looks like
> this week, with this user's combination of plugins that's different from
> everybody else's.

"Plugins" is a bad word.  This user's combination of plugins is most
likely identical to other users', it's just which ones are enabled, and
which aren't?  If they are all included, I assume they play nice.

And just because they are called "plugins" doesn't mean the EA looks
different every week.

I cannot stress this enough.  In Reiser4, EVERYTHING is some sort of
plugin.  We now have a standard disk format plugin, and a standard set
of plugins providing UNIX filesystem-like things (vanilla files,
directories, symlinks, device nodes, named pipes...) which has been
stable for some time.

I see no problems with providing at least as consistent an EA interface
as any other FS, only more extensible because it's not flat.

>>>lets take this a step further.  what about compression?  do we accept
>>>that each filesystem can implement its own proprietary compression via
>>>its own API - and now we need individual user-space tools to understand
>>
>>No, that's the beauty of these "EAs" in Reiser4.  The API is standard
>>write(2) commands.  sys_reiser4 supposedly implements an interface to
>>make this scale better, but otherwise have the same semantics.  And who
>>said anything about proprietary compression?  I think we were planning
>>on the kernel's zlib, though we might have been planning to make it a
>>bit more seekable...
>>
>>
>>>each of these APIs?
>>
>>So, the API becomes something like:
>>
>>cat crypto/inflated/foo		# transparently decompressed
>>cat crypto/raw/foo.gz		# raw, gzip-compressed
> 
> 
> And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly?
> 
> Now throw some .bz2 and .zip files into the mix... ;)

Interface is the same.  Only, zip files aren't just compression, so
maybe the interface changes a little there.

Point is, now you have a standard interface for any program to access
any simple lossless compression, transparently.

>>Another possibility, if you like file-as-a-directory:
>>
>>cat foo.gz			# raw
>>cat foo.gz/inflated		# decompressed
>>
>>One could easily imagine things like these two potentially equivalent
>>commands:
>>
>>cp foo bar.zip/
>>zip bar foo
> 
> 
> Unless of course the user had done 'mkdir sorted.by.city.zip' to make
> a directory of files containing data sorted by USPS Zip code.

What's this got to do with anything?

> And what happens if the user has a file 'bar' that's not a ZIP file,
> and a directory 'bar.zip' isn't a view into 'bar'?

In file-as-a-directory (which is probably NOT happening soon), bar.zip
is both the actual zipfile and the view inside, depending on whether you
try to open() it directly or peek inside it as a directory.

I've used that interface for some simpler things like file permissions.
 I could do things like "echo 0 > some_file/metas/uid".  The interface
is gone for now, because it broke some things, I'm told, but nothing
serious for me while I was using it.  I don't think the problem is
insurmountable.

However, let's not discuss this now.  I do NOT want to start another
"silent semantic changes with reiser4" thread.  File-as-directory is not
happening this time, so don't worry about it -- this time.

> Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a
> directory 'linux-2.6.12', what is under the directory is *NOT* the same
> data as what's in the .bz2 - I've done 'make oldconfig' and a few builds
> and some variable amount of patching, usually with rejects, and I *don't*
> want that .bz2 being updated during all this (hint - what's my next command
> after 'rm -rf linux-2.6.12' likely to be, and why, and  what expectations
> do I have when I do it?)

You're misunderstanding.  man zip.
$ zip bar foo
creates/modifies a file named "bar.zip", not "bar", which contains the
file "foo".

> You want to think this sort of thing through *really* thoroughly, because
> there's a *lot* of things, both users and programs, that have expectations
> about The Way Things Work.

Or, I can avoid those issues altogether, and simply delegate this kind
of stuff to user-created-but-magic directories.  For instance, I could
have a directory called "/foo" which contains encrypted files, and
"/foo/decrypted" which has transparently decrypted representations of them.

>>The whole point is to have less userland tools, not more.  I'm not
>>saying we move zip into the kernel, just that the user now has one less
>>command to remember.
> 
> 
> But now instead of having to remember the one meme "I can manage any
> compressed-archive format that's stored in a file, and put other files in it,
> and all I need is the appropriate userspace tool", they have to remember "the
> cp trick works for .zip and .tar, but I'll get a "not a directory" error if I
> try it with a .hqx file, and that other file format may or may not work,
> because I can't remember if this kernel has a working out-of-tree module for
> this kernel...."

True, for now.  But for me, it's not hard at all to try "cp" first, and
after it fails, Google and install the required tool.  After all, most
users already have to do that with GUI interfaces, this just takes the
GUI way of doing things (like in file-roller) and makes it perform
better and work with all programs.  Not necessarily all formats, but all
programs.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr8IzngHNmZLgCUhAQKS9A/+K+Nkq0sLynA0ltiA7o+BnA4cRSLYnu0t
LdVI+yvYMUocvZt1HcIuU1LN2y3//Xm7hh7Lsyf3moNCIFMkDUN/xuzUcmfpUs5W
NpyYPS4M3h8bX7nAMI9SmJ2KPoQNJxiEMToi5cYkQvafAeAlR2xw+z9nQ7Cv2PAi
PCKROYSbZ1MdyCVLDPYkIpk9Z9EnOZ7zE+nbMaippqAnwfoqMoe8OMBKt1qxBROS
tSPn4h1+JQZTs5OY18Cg/km7J6SM3SmHEWtsicil9/AVMFoOCpUXkeuiBLgVZ7Vk
hh79SLjmuAOgU5hTBhEwvtEjB5uGqbjRtp+quIWNO4tmHqPuSv+TP0YaOgMTqbf7
9Cr+iq/YrWMDCA3fYrnq/TiSCNK/YHGNlQG4rZtnKDXUFUZRuItMhbRaiYYfv/Cl
wQjBrknKci4xFNB9gIOoykS9Vs3JDdZc8oj26rFhHGqkGE+xo6b0z1EaZMvBfes9
L+VqU6B7YnyDWnhDbK6TiLZ/Qg83SigTyoLORVIDzyo3k2SVA8kmy8NgdA9NlmIT
ko/g0+7zHtu64LuEtQEdL/FULnI/YcMtxQDjYuhCqtUo/x4y3hi5685P00iZDfdp
RpuZnOT/esiw/RLashgGRK3xsVik9R+WX2psrEXp7HvNYvr5X4BpFovyVYeTxSnj
jV201bzTv6A=
=z+xB
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 4149 bytes --]

On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said:

> "Plugins" is a bad word.  This user's combination of plugins is most
> likely identical to other users', it's just which ones are enabled, and
> which aren't?  If they are all included, I assume they play nice.

Which ones are enabled. Exactly.

> And just because they are called "plugins" doesn't mean the EA looks
> different every week.

They do if the one enabled this week is "make EAs look like symlinks", and
last week's was "make EAs look like folders".

(Don't blame me, *you're* the one that said "EAs can look like any other object"..)


> > And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly
?
> > 
> > Now throw some .bz2 and .zip files into the mix... ;)
> 
> Interface is the same.  Only, zip files aren't just compression, so
> maybe the interface changes a little there.

Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you.

> Point is, now you have a standard interface for any program to access
> any simple lossless compression, transparently.
> 
> >>Another possibility, if you like file-as-a-directory:
> >>
> >>cat foo.gz			# raw
> >>cat foo.gz/inflated		# decompressed
> >>
> >>One could easily imagine things like these two potentially equivalent
> >>commands:
> >>
> >>cp foo bar.zip/
> >>zip bar foo

> > Unless of course the user had done 'mkdir sorted.by.city.zip' to make
> > a directory of files containing data sorted by USPS Zip code.
> 
> What's this got to do with anything?

It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*,
the way Unix-style systems have done for 3 decades, and suddenly my system is
running like a pig because the kernel decided that it's a .zip file.

> > And what happens if the user has a file 'bar' that's not a ZIP file,
> > and a directory 'bar.zip' isn't a view into 'bar'?
> 
> In file-as-a-directory (which is probably NOT happening soon), bar.zip
> is both the actual zipfile and the view inside, depending on whether you
> try to open() it directly or peek inside it as a directory.

Ahem.  "bar.zip' is a *DIRECTORY*. I said 'mkdir bar.zip' - why is it not
acting like a directory?
 
> However, let's not discuss this now.  I do NOT want to start another
> "silent semantic changes with reiser4" thread.  File-as-directory is not
> happening this time, so don't worry about it -- this time.

Fish or cut bait.  You are the one who started handwaving the 'file-as-directory'.
If you don't want it discussed, don't mention it.

> > Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a
> > directory 'linux-2.6.12', what is under the directory is *NOT* the same
> > data as what's in the .bz2 - I've done 'make oldconfig' and a few builds
> > and some variable amount of patching, usually with rejects, and I *don't*
> > want that .bz2 being updated during all this (hint - what's my next command
> > after 'rm -rf linux-2.6.12' likely to be, and why, and  what expectations
> > do I have when I do it?)
> 
> You're misunderstanding.  man zip.
> $ zip bar foo
> creates/modifies a file named "bar.zip", not "bar", which contains the
> file "foo".

No. *YOU* are misunderstanding.  I have a directory 'linux-2.6.12', and
I have a file 'linux-2.6.12.tar.bz2', and I do *NOT* want directory operations
to be silently converted into "let's scribble into the middle of this tar file
and then compress it".  (Hint - work out how long a kernel 'make' would take
if you were doing it inside a .tar.bz2).

> > You want to think this sort of thing through *really* thoroughly, because
> > there's a *lot* of things, both users and programs, that have expectations
> > about The Way Things Work.
> 
> Or, I can avoid those issues altogether, and simply delegate this kind
> of stuff to user-created-but-magic directories.  For instance, I could
> have a directory called "/foo" which contains encrypted files, and
> "/foo/decrypted" which has transparently decrypted representations of them.

So rather than everything working in a funky manner, a program gets to guess
how funky, and in what direction, a given magical directory is....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said:
> 
> 
>>"Plugins" is a bad word.  This user's combination of plugins is most
>>likely identical to other users', it's just which ones are enabled, and
>>which aren't?  If they are all included, I assume they play nice.
> 
> 
> Which ones are enabled. Exactly.

I doubt there will be duplicate plugins, once things settle down.  So
you just have the program demand certain ones to be enabled.

>>And just because they are called "plugins" doesn't mean the EA looks
>>different every week.
> 
> 
> They do if the one enabled this week is "make EAs look like symlinks", and
> last week's was "make EAs look like folders".

We could just as easily do that with other extended attributes.  Sure,
the details would be different -- maybe we just randomly add a XA onto
the beginning of every string.

> (Don't blame me, *you're* the one that said "EAs can look like any other object"..)

Doesn't mean we can't guarentee a certain kind in a certain
configuration will always be available for a certain plugin, once it's
been accepted.
> 
> 
> 
>>>And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly
> 
> ?
> 
>>>Now throw some .bz2 and .zip files into the mix... ;)
>>
>>Interface is the same.  Only, zip files aren't just compression, so
>>maybe the interface changes a little there.
> 
> 
> Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you.

In that example (shouldn't have used the name "crypto", but oh well), it
should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is
the gzip'ed file and foo is the transparently compressed/decompressed
file.  Basically, these are equivalent:

$ zcat crypto/raw/foo.gz
$ cat crypto/inflated/foo

>>Point is, now you have a standard interface for any program to access
>>any simple lossless compression, transparently.
>>
>>
>>>>Another possibility, if you like file-as-a-directory:
>>>>
>>>>cat foo.gz			# raw
>>>>cat foo.gz/inflated		# decompressed
>>>>
>>>>One could easily imagine things like these two potentially equivalent
>>>>commands:
>>>>
>>>>cp foo bar.zip/
>>>>zip bar foo
> 
> 
>>>Unless of course the user had done 'mkdir sorted.by.city.zip' to make
>>>a directory of files containing data sorted by USPS Zip code.
>>
>>What's this got to do with anything?
> 
> 
> It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*,
> the way Unix-style systems have done for 3 decades, and suddenly my system is
> running like a pig because the kernel decided that it's a .zip file.

The kernel does not decide that.  You do.  And it doesn't automatically
decide that every time you create a file.  You have to use some
interface to trigger the plugins.

Originally, this was file-as-a-directory.  Now?  I'm not sure, I guess
we could use a separate delimiter.  Something like:

foo	# file
...foo	# directory containing xattrs of file, list of plugins used.

foo/	# directory
foo/...	# directory containing xattrs of file, list of plugins used.

I guess I need a new name for this approach.  That's three possible ways
of doing this?

>>>And what happens if the user has a file 'bar' that's not a ZIP file,
>>>and a directory 'bar.zip' isn't a view into 'bar'?
>>
>>In file-as-a-directory (which is probably NOT happening soon), bar.zip
>>is both the actual zipfile and the view inside, depending on whether you
>>try to open() it directly or peek inside it as a directory.
> 
> 
> Ahem.  "bar.zip' is a *DIRECTORY*. I said 'mkdir bar.zip' - why is it not
> acting like a directory?

If you said "mkdir", it would act like a directory.

More likely than foo.zip/bar would be foo.zip/.../contents/bar.  Which
would also work for tarballs.  But would not automatically compress
anything you didn't want it to.

>>However, let's not discuss this now.  I do NOT want to start another
>>"silent semantic changes with reiser4" thread.  File-as-directory is not
>>happening this time, so don't worry about it -- this time.
> 
> 
> Fish or cut bait.  You are the one who started handwaving the 'file-as-directory'.
> If you don't want it discussed, don't mention it.

I do want it discussed.  I'm not sure it's a good idea now.  But looks
like you got me...

>>>Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a
>>>directory 'linux-2.6.12', what is under the directory is *NOT* the same
>>>data as what's in the .bz2 - I've done 'make oldconfig' and a few builds
>>>and some variable amount of patching, usually with rejects, and I *don't*
>>>want that .bz2 being updated during all this (hint - what's my next command
>>>after 'rm -rf linux-2.6.12' likely to be, and why, and  what expectations
>>>do I have when I do it?)
>>
>>You're misunderstanding.  man zip.
>>$ zip bar foo
>>creates/modifies a file named "bar.zip", not "bar", which contains the
>>file "foo".
> 
> 
> No. *YOU* are misunderstanding.  I have a directory 'linux-2.6.12', and
> I have a file 'linux-2.6.12.tar.bz2', and I do *NOT* want directory operations
> to be silently converted into "let's scribble into the middle of this tar file
> and then compress it".  (Hint - work out how long a kernel 'make' would take
> if you were doing it inside a .tar.bz2).

I remember discussing that, actually.  It wouldn't automatically do this
if you didn't want it to, but it would be nice if, say, it was something
truly seekable like linux-2.6.12.zip, and linux-2.6.12 was a
user-created symlink to linux-2.6.12.zip/.../contents, and we had a nice
caching system...

This is nice because then you get exactly the same performance during
"make" as you would with "unzip && make", only better, because files you
don't ever use (lots of arch, for instance) are not unpacked.

This is all something that's been discussed before.  I think the
consensus was that it sounded cool, but not useful enough by itself to
justify file-as-a-directory.

You can always go re-read "Silent Semantic Changes..."

>>>You want to think this sort of thing through *really* thoroughly, because
>>>there's a *lot* of things, both users and programs, that have expectations
>>>about The Way Things Work.
>>
>>Or, I can avoid those issues altogether, and simply delegate this kind
>>of stuff to user-created-but-magic directories.  For instance, I could
>>have a directory called "/foo" which contains encrypted files, and
>>"/foo/decrypted" which has transparently decrypted representations of them.
> 
> 
> So rather than everything working in a funky manner, a program gets to guess
> how funky, and in what direction, a given magical directory is....

It doesn't have to guess, it can know if it needs to.  Although this
probably wreaks havok on traditional backup systems.  You'd have to
implement a somewhat new backup system, but it's not impossible, and
certainly doesn't require that your new backup system know about every
plugin -- just make sure plugins know about backup.

Of course, file-as-directory avoids all that -- you just have to patch
programs to know that if foo/ exists or foo responds to opendir, foo is
not necessarily a directory.  You probably just drop the trailing / and
use stat().

Both (all three?) appraches have cans-of-worms and workarounds.  But I
think both (all three?) are doable.  I know user-created-but-magic is
doable.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr8txHgHNmZLgCUhAQKOxA/+KP1lmUgmrbEH+kPKriSFF3qFfFBgbeJh
iebHVaOY4AMa99a3yHy8iA5ZyE5N2b3CEueoq44ygYs5gqeft1IF14SJweNk/6or
IgGt+OsAkqDMdbbpKZdDGo8lZK0bJq/Ni+PMzx44sMGg4vWIHGAQfDGZ9FYRmrRT
fLNprUrkUsIzVEDHcuYjrZfPHvqr4HK8KXa5IJIKj6WkCYj5gxkQlH3g9zEaWXlN
Mfsqr6uioK34vKEgQw9HvpDpMZaB5NpZ7RWr5vEP3G3AV120VNKIEqU09Pex/IJD
oiOaroR7y1OZUaLuTSyut+4MGjHJ7k6TNr8H/CFdRfpLs7mP5tsJG42rzAhqTdyP
2D2m7sgpEN8ALkGd+IU3ihLOVQCrSq2T0jpr6tFhDe5uTenP3fXBeGgu8ynRzPFD
kkpyZhT094UWKoq8n+IcHQrIcba4j6X/jDJt7SDx8IwfZaPoMpcBGQH1//Q+lTuC
q5qM7olI5OzcSr4AhDld7OOjqO0pLXRfH5y8hy8mgWEiGgomK+N5OFTqXrQsuB/g
BopfBpqHRRYVpI1VkAutmTpyNjm3dQjrHmRWXX2sFe9lxExpWo9VX4By9ytv9EXQ
UFfSBF9TVe+EkZ4XxO+JMCcahNUsBh45rRXY3wVpH1qPwsOVp8FxYcOmuzQSTha9
mgOxWbRfqkQ=
=jrll
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

> Valdis.Kletnieks@vt.edu wrote:
>
> >On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said:
>
>
> >>"Plugins" is a bad word.  This user's combination of plugins is most
> >>likely identical to other users', it's just which ones are enabled, and
> >>which aren't?  If they are all included, I assume they play nice.
>
>
> >Which ones are enabled. Exactly.

Reiser4 plugins are not per user, but per kernel.  They are compiled
in.  The model is intended to ease the development process, nothing
more.  Apologies if the naming suggests more.

Hans

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
> David Masover wrote:
> 
> 
>>Valdis.Kletnieks@vt.edu wrote:
>>
>>
>>>On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said:
>>
>>
>>>>"Plugins" is a bad word.  This user's combination of plugins is most
>>>>likely identical to other users', it's just which ones are enabled, and
>>>>which aren't?  If they are all included, I assume they play nice.
>>
>>
>>>Which ones are enabled. Exactly.
> 
> 
> Reiser4 plugins are not per user, but per kernel.  They are compiled
> in.  The model is intended to ease the development process, nothing
> more.  Apologies if the naming suggests more.

But, to avoid confusion, the inclusion of a crytocompress plugin in a
given kernel doesn't mean that all files accessed from that kernel are
encrypted and compressed.  It just means that you can pick an individual
file and set it to be transparently encrypted/compressed.

That is what I meant by "enabled".  Not per-user, but per-file.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr9FcHgHNmZLgCUhAQJ2+BAAgnqAcEg41VXTUmpUUZWMduWeJKKYDE16
QvWore6MK/FWoU8hqFMddTzJVpzKNH8NoZ1Jfm2nEC51nndckwtBDsuIlMb1CGC+
I+wCrpe1kjXTz8sTs++CwDg29YojHP1/Wa5MwnUgTEBkZxacPI7r0VXyt1oO+f/N
Eu6Jpgl4IN3ba52/PS+dXCQrmOJDhLeHAb+l4rtRsK+LASJitqmGmpf4PRwEejM8
dWZyjEYdiAcMx6mkeB/lho5IIcp1Xi5QBACf6S8SHXvHxRW50cKeouAISR25Ttk2
Oa1vKhWPGo4IBREjK7fzgj6GwfpIBnUfE25ZRjrBRupWumwekaCAY91JOoRkdRI+
Lw70OZhqqIKQ/O46xqhyCrroP/6is5ZxLyIRe1q3qsQfsWUfHBOONRUBdtaTQlaa
3OWzAU49cxn4Jv9S9UEYEyKx9ggqJ5hd94wZpXPq4GogWt4S8cYK3d4ZtRIyXEBr
qOrLoJoTF9WeT254u+uq5gLP5kxq7Z7J51aMXbGF+4luuGJPq/50Y4hbapAMQWFA
v0Z8YNWoOnOJgYji/+u8qJhsfzBdi/dmWlhhy9Te1e1b/1+zHcsbCslgsIrG2spk
3uF1GOv5NorO2n7RhierhkUrkUsLLlFqf0vPmgMqJ9DG4h6wl3bOJkYShBKTYQB+
ldYcxERKMZ4=
=jv9S
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

On Sun, 26 Jun 2005 19:16:48 CDT, David Masover said:

> But, to avoid confusion, the inclusion of a crytocompress plugin in a
> given kernel doesn't mean that all files accessed from that kernel are
> encrypted and compressed.  It just means that you can pick an individual
> file and set it to be transparently encrypted/compressed.
> 
> That is what I meant by "enabled".  Not per-user, but per-file.

Doing key management in a secure manner is going to be *fun*. :)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

> Hans Reiser wrote:
>
> >David Masover wrote:
>
>
> >>Valdis.Kletnieks@vt.edu wrote:
> >>
> >>
> >>>On Sun, 26 Jun 2005 14:58:07 CDT, David Masover said:
> >>
> >>
> >>>>"Plugins" is a bad word.  This user's combination of plugins is most
> >>>>likely identical to other users', it's just which ones are
> enabled, and
> >>>>which aren't?  If they are all included, I assume they play nice.
> >>
> >>
> >>>Which ones are enabled. Exactly.
>
>
> >Reiser4 plugins are not per user, but per kernel.  They are compiled
> >in.  The model is intended to ease the development process, nothing
> >more.  Apologies if the naming suggests more.
>
>
> But, to avoid confusion, the inclusion of a crytocompress plugin in a
> given kernel doesn't mean that all files accessed from that kernel are
> encrypted and compressed.  It just means that you can pick an individual
> file and set it to be transparently encrypted/compressed.
>
> That is what I meant by "enabled".  Not per-user, but per-file.
>
You are correct.

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:
> Hans Reiser wrote:

[...]

> > Reiser4 plugins are not per user, but per kernel.  They are compiled
> > in.  The model is intended to ease the development process, nothing
> > more.  Apologies if the naming suggests more.

What do you gain by this? It is just a kernel configuration option of
sorts? Just name mangling of existing mechanisms for no good reason at all?

> But, to avoid confusion, the inclusion of a crytocompress plugin in a
> given kernel doesn't mean that all files accessed from that kernel are
> encrypted and compressed.  It just means that you can pick an individual
> file and set it to be transparently encrypted/compressed.
> 
> That is what I meant by "enabled".  Not per-user, but per-file.

Wonderful! I carefully "transparently encrypt" my secret files, so
/everybody/ can read them! Now /that/ is progress!
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Gregory Maxwell]

On 6/27/05, Horst von Brand <vonbrand@inf.utfsm.cl> wrote:
> Wonderful! I carefully "transparently encrypt" my secret files, so
> /everybody/ can read them! Now /that/ is progress!

All of this side feature argument is completely offtopic for the
inclusion of reiser4, but oh well.

In any case, the real use for encrypted files (vs encrypted
partitions) would be for doing things like tying keying into the login
process so that your files are only accessible while you are logged
in.  This would be a very nice feature on a multiuser system.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
> David Masover <ninja@slaphack.com> wrote:
> 
>>Hans Reiser wrote:
> 
> 
> [...]
> 
> 
>>>Reiser4 plugins are not per user, but per kernel.  They are compiled
>>>in.  The model is intended to ease the development process, nothing
>>>more.  Apologies if the naming suggests more.
> 
> 
> What do you gain by this? It is just a kernel configuration option of
> sorts? Just name mangling of existing mechanisms for no good reason at all?

I don't know why it's named "plugins", but the mechanism itself gains a
lot.  For one thing, certain new "plugins" for new kinds of files
(cryptocompressed files, for instance) can be almost as easy as new
plugins for (say) XMMS.

Still, Hans, it's high time to either rename these "plugins" or start
talking about bytecode engines...

>>But, to avoid confusion, the inclusion of a crytocompress plugin in a
>>given kernel doesn't mean that all files accessed from that kernel are
>>encrypted and compressed.  It just means that you can pick an individual
>>file and set it to be transparently encrypted/compressed.
>>
>>That is what I meant by "enabled".  Not per-user, but per-file.
> 
> 
> Wonderful! I carefully "transparently encrypt" my secret files, so
> /everybody/ can read them! Now /that/ is progress!

Explain to me how this is worse than how you currently encrypt your
secret files?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+bCHgHNmZLgCUhAQKnag/8DF1PjfQwEYaJWUOOY0roqn6b+/L8X/X1
kgFE3SyPJkNRuY6X/J3EQUht5v/EBEbk2ZE/oTWB+HGEeUv1juUmG7z2rWCvi8sY
tXPG28iqOAjwvZa+2aq8o4ptW0+gaXVIbrNGEaWTXPgbxdxtoVMOAakBJl3FUchK
pwZWJD+msuQuPFc42zQ3Zdvp+4/NhhC6qcSpZ9DGXHO9uDJDDdl3Xgf9+qOD+W7Y
Bz4sGfNphIbJjydHs0DdcZPh49MhS+qd48Qw1WFskalF9dhZhHFq3MKIBFbg8bA8
MX9yY5XHrNPJqJXF96YtcJV9nZcXh4qEz81/JC7V+VYm/BME20a0jtUt5bMtGRjj
kC8FxzVvamOmeXnzRnU2DKIlGlfTUmK1u6wQpMsoiwPZr9WH7WRxiP/tHPVYQbBc
gPpg7POOz3M6Q3guZMGTxeYu7kHJm8s/zJVwXm04fz+03hrkab1q/w4cT0SaX4Pv
naipO7nkdr5qoRzF2sBSZAsnauxGa0ViR0TP715FN+oX0ZRiSQcgEGc+zXz6qPb5
bSz29jG1iGkrepB6LU88gMZiyxc2qK1SHpZlKjn9iPKqpxpwEtpQToUBhRsAlFBv
9Z6KxrQYrUYfJ/Z9ktIUuvsvN3zbnnc7tmZGYNisqm03dq61kguAwOV66INSlYDj
gFaw23Z9X58=
=xsgf
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 2841 bytes --]

On Sun, 26 Jun 2005 17:35:48 CDT, David Masover said:

> > Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you.
> 
> In that example (shouldn't have used the name "crypto", but oh well), it
> should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is
> the gzip'ed file and foo is the transparently compressed/decompressed
> file.  Basically, these are equivalent:
> 
> $ zcat crypto/raw/foo.gz
> $ cat crypto/inflated/foo

I'm *quite* aware of what your preconceived notions think it *should* be.

Maybe the two examples I asked for have *real-world* meanings that you should
be allowing for.  Like, for instance, on a mail server, where the A/V software
may need to unzip a file 5 or 6 times to find out if there's malicious content.

Or seeing if it's a ".zip bomb", where a small .zip will decompress to gigabytes.

Or I'm testing a new compression algorithm, to see if multiple compressions help
(yes, I know that it *shouldn't* help - but I've seen real-world cases where the
algorithm could only look at a 4K or 8K window at a time, and if you hit a *very*
long run of duplicate 4K segments, a second compression would compress all the
identical or near-identical "this is a 4K chunk" tokens...)


> > It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*,
> > the way Unix-style systems have done for 3 decades, and suddenly my system is
> > running like a pig because the kernel decided that it's a .zip file.
> 
> The kernel does not decide that.  You do.  And it doesn't automatically
> decide that every time you create a file.  You have to use some
> interface to trigger the plugins.

Oh, I'm waiting for the fun the first time somebody deploys a plugin that
has similar semantics to 'chmod g+s dirname/' ;)

> I guess I need a new name for this approach.  That's three possible ways
> of doing this?

I *said* you need to think this through in detail, didn't I? ;)
 
> I remember discussing that, actually.  It wouldn't automatically do this
> if you didn't want it to, but it would be nice if, say, it was something
> truly seekable like linux-2.6.12.zip, and linux-2.6.12 was a
> user-created symlink to linux-2.6.12.zip/.../contents, and we had a nice
> caching system...

I think you're highly deluded as to just how much or little performance gain
this will get you. Model what happens with a kernel 'make' on a 256M machine
with and without all that zipping and compressing, and assume that a constant
48M is available for caching of the linux-2.6.12/ tree.

> This is nice because then you get exactly the same performance during
> "make" as you would with "unzip && make", only better, because files you
> don't ever use (lots of arch, for instance) are not unpacked.

Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Sun, 26 Jun 2005 17:35:48 CDT, David Masover said:
> 
> 
>>>Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you.
>>
>>In that example (shouldn't have used the name "crypto", but oh well), it
>>should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is
>>the gzip'ed file and foo is the transparently compressed/decompressed
>>file.  Basically, these are equivalent:
>>
>>$ zcat crypto/raw/foo.gz
>>$ cat crypto/inflated/foo
> 
> 
> I'm *quite* aware of what your preconceived notions think it *should* be.
> 
> Maybe the two examples I asked for have *real-world* meanings that you should
> be allowing for.  Like, for instance, on a mail server, where the A/V software
> may need to unzip a file 5 or 6 times to find out if there's malicious content.
> 
> Or seeing if it's a ".zip bomb", where a small .zip will decompress to gigabytes.
> 
> Or I'm testing a new compression algorithm, to see if multiple compressions help
> (yes, I know that it *shouldn't* help - but I've seen real-world cases where the
> algorithm could only look at a 4K or 8K window at a time, and if you hit a *very*
> long run of duplicate 4K segments, a second compression would compress all the
> identical or near-identical "this is a 4K chunk" tokens...)
> 
> 
> 
>>>It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*,
>>>the way Unix-style systems have done for 3 decades, and suddenly my system is
>>>running like a pig because the kernel decided that it's a .zip file.
>>
>>The kernel does not decide that.  You do.  And it doesn't automatically
>>decide that every time you create a file.  You have to use some
>>interface to trigger the plugins.
> 
> 
> Oh, I'm waiting for the fun the first time somebody deploys a plugin that
> has similar semantics to 'chmod g+s dirname/' ;)
> 
> 
>>I guess I need a new name for this approach.  That's three possible ways
>>of doing this?
> 
> 
> I *said* you need to think this through in detail, didn't I? ;)
>  
> 
>>I remember discussing that, actually.  It wouldn't automatically do this
>>if you didn't want it to, but it would be nice if, say, it was something
>>truly seekable like linux-2.6.12.zip, and linux-2.6.12 was a
>>user-created symlink to linux-2.6.12.zip/.../contents, and we had a nice
>>caching system...
> 
> 
> I think you're highly deluded as to just how much or little performance gain
> this will get you. Model what happens with a kernel 'make' on a 256M machine
> with and without all that zipping and compressing, and assume that a constant
> 48M is available for caching of the linux-2.6.12/ tree.

Ignoring Hans' point, there is still a performance gain.

Assume we can do on-disk caching, similar to fscache/cachefs for nfs.
Now, benchmark:

$ unzip linux-2.6.12.zip && make -C linux-2.6.12

versus the hypothetical

$ make -C linux-2.6.12.zip/.../contents

This is an automatic performance gain, in theory, because the second
command is identical to unzipping just the parts you need into
linux-2.6.12, then running "make".  The one disadvantage is that because
the unzipping is done on demand, it only really performs well if you can
keep the "zip" binary cached.  Even on most embedded systems, 54K of RAM
is really not much to ask these days.

Also, once you've run "make" once, you get to run it as many times as
you like, and so long as the on-disk cache of the zipfile is still there
and valid, you never have the overhead of unzipping again.

Of course, this probably saves only a minute or two at most per kernel
compile.  But that doesn't mean there aren't real-world situations where
this kind of architecture would be much more beneficial.

>>This is nice because then you get exactly the same performance during
>>"make" as you would with "unzip && make", only better, because files you
>>don't ever use (lots of arch, for instance) are not unpacked.
> 
> 
> Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives.

So what?  I don't intend to convince anyone based on how much
slower/faster their kernel compiles.  It's meant to illustrate the
principle of the thing.

Besides, your point was that you could not run make inside of a kernel
tarball/zipfile.  Nobody ever suggested that you would actually want to.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr9mfHgHNmZLgCUhAQJi0g//RGxFXWi8Om4EnKsZHcI0J7X3G6T9pj2a
nwkWwjLdyg6jykdt3a5MTELBgOM2uT87k7CeO7TasA/T1ZkZ/y2Yw7x50YIgrb7j
W1u5N/vfDLw3C2Nd6O2fe/b4ygReyBB6HQtNTw/k+XxDswxtEp0mcZHpNxt+W9B4
s0naezawRjF51P4ISqa6HoRo7vZUkXv+3CwuZinC5m8KnW2Us8Ww5uDjtNBLpJGR
zs79w24zaj6VSHjF8lO6CuMKQLjSelMDXKDEkFHaybJgz7AckkcZg5c6VDTrc3/t
m8HM5oyHWfRqVeQt9cMdLdcBZnhdbspSwQaNQkdkrZx1IX96mPoMNDUwk1B/TIi7
++iqpkDYeOdg+DWzGPVpwQ+5LQC+7m8vRHv5dROIM6T89TnlUck2clBiPovzSP+8
KMR1R6F7qBPxJMkPcy2MNOVMkjN+VLSOCzOeOXVEUkNYdXjrJB5h3XIN5MyRR7C/
pRmjB2crzPTUz2yBatP+QUFNUMadikMqj44sEc/ie8iHbo9pfxQC/wY4M4VkJcf2
03spe2e8M+k3txj63O9TmJYgobkjx+d+cJ5Zm7DbKJmGlVaAGqmCXeNjxhTtBSwU
yP2Jrz6Awu+nDxFxMAsN4GP17/0/aXdBhIYLPGyAJ9/HV7KHENIqIjnvD4e3bXnU
shBrM+G1N5E=
=BXCi
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Kyle Moffett]

On Jun 26, 2005, at 22:37:48, David Masover wrote:
> $ make -C linux-2.6.12.zip/.../contents

I've yet to see how such automatic unzipping does not inherently  
introduce
system insecurity into _existing_ applications by changing POSIX and  
UNIX
semantics.

When I do this in my suid perl script:

open my $fh, '<', $ARGV[0];

I do _not_ mean this:
system '/bin/zip', 'x', $path1, $path2;

Neither do I want the kernel to unzip it, because that just  
introduces the
kernel to a whole series of normally application-level vulnerabilities.
I've seen situations where a specially doctored tarball can expand to a
result file over 1000x the size of the original.  Likewise, there  
have been
cases where crafted tarballs have taken advantage of buffer overflows.

Can you illustrate for me with precise, clear, and unambiguous arguments
how this can avoid all possible pitfalls, especially in cases where it's
likely to break existing working privileged apps?  Such extended  
operations,
including the automatic encryption/decryption and most other non- 
standard
filesystem features (Basically the whole '...' directory), should  
probably
be left out of any patch submitted for inclusion until they can be  
_proven_
(or at least thoroughly checked) not to have undesirable results.

Cheers,
Kyle Moffett

--
Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best  
answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't.  
That's why
I draw cartoons. It's my life."
-- Charles Shultz

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kyle Moffett wrote:
> On Jun 26, 2005, at 22:37:48, David Masover wrote:
> 
>> $ make -C linux-2.6.12.zip/.../contents
> 
> 
> I've yet to see how such automatic unzipping does not inherently  introduce
> system insecurity into _existing_ applications by changing POSIX and  UNIX
> semantics.
> 
> When I do this in my suid perl script:
> 
> open my $fh, '<', $ARGV[0];
> 
> I do _not_ mean this:
> system '/bin/zip', 'x', $path1, $path2;
> 
> Neither do I want the kernel to unzip it, because that just  introduces the
> kernel to a whole series of normally application-level vulnerabilities.

That just means the zip plugin has to be more carefully written than I
thought.  It would have to be sandboxed and limited to prevent buffer
overruns and zip bombs.

I think that you could create a situation where an untrusted zip could
be explored, and the worst side effect would be that the files you see
inside the untrusted zip wouldn't be what you expect.  But that's no
surprise -- if the zip is really malicious, there's no amount of
sandboxing that would give you valid files anyway.

I was probably taking this too lightly, though.

Remember that zip, at least, would not be in the kernel.  I think what
is going in the kernel is gzip and lzo, and it's being done so
transparently that you never actually see the compressed version.

> Can you illustrate for me with precise, clear, and unambiguous arguments

That will take some time.  And some thinking.

> how this can avoid all possible pitfalls,

Especially if you want perfection.

> including the automatic encryption/decryption and most other non- standard
> filesystem features (Basically the whole '...' directory), should  probably
> be left out of any patch submitted for inclusion until they can be 
> _proven_
> (or at least thoroughly checked) not to have undesirable results.

I am doing that out of habit.  Actually, it probably ends up more as:
  .../foo.zip/
instead of
  foo.zip/.../

But, it is left out.  At least that interface.

Now, the cryptocompress as it currently stands does not involve ... and
does not introduce any new security holes in the way that you are
describing.  There might be some issues with key management (someone
hinted vaguely at that), but nothing insurmountable.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr9xZ3gHNmZLgCUhAQKmOA/9EGus4fGXKnBPoK4Rpd9K/6tgy0A7QFIO
EeF50BSgl7M5sot9Vp28Dg1DA9X8gXHf/BxWIUse2doEdrbYKy3HFNd4ZChPFXCS
j4ZtJo/eGYQntIFZ+exNJG2gDeprSBUH9AnMxF9xBfG8CxBdl6WL1dx8d7kc7ip/
UYGiu+9WmC9LanEb0ezhsO8I0KBvjx73Q3FTbD9N0cMIzK1Drv7p9r9rhswsoIzg
eZnKrT2Z0u8BASbd0GfCAT3DH3Zn6zHf6Zk9FOaPaqcLwWlXbELaTbFvBp+2rpnH
9j3NwKHTtCKX5Z2BOQqtiEDEE8QInuDlcEV2K/y4g9YM1mMw3TNoXswaZrbPWWeF
RD/YyzknaDhfgQdz9kQ2bfHJM7//Y9IUJZp/3NmWhvhc2+QBhXBBoTb8UEnRl7tK
D9UIgsDFDVHlLcO9KPKokjyf3fRL57dd0ictHFORvicVIK6NFSfFP9LY/ZsmUXF9
Fbqwwuu8/5n9z5j9IyIqf5m7XJkHcZCeLFDkn89VS4ZM8W1aB0g3yRIlhSEDDkVt
0nZZRzvIlRHCPoPMZuoFhWSngi50ZAYXlRicKrHQERP8f7ECkB1JvY2rLSFqM+FX
BGUZpXxgHe/zgg806ACNbfdnny5WgZ7qXl/IXD9svQa3lIgxY6We8ACj8Oi6c7eu
5ooBcT7AmRE=
=pBA8
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Kyle Moffett]

On Jun 26, 2005, at 23:24:23, David Masover wrote:
>> Neither do I want the kernel to unzip it, because that just   
>> introduces the
>> kernel to a whole series of normally application-level  
>> vulnerabilities.
>
> That just means the zip plugin has to be more carefully written than I
> thought.  It would have to be sandboxed and limited to prevent buffer
> overruns and zip bombs.



> Remember that zip, at least, would not be in the kernel.  I think what
> is going in the kernel is gzip and lzo, and it's being done so
> transparently that you never actually see the compressed version.


>> Can you illustrate for me with precise, clear, and unambiguous  
>> arguments
>
> That will take some time.  And some thinking.

Precisely.  Come back when you have a good sturdy set of arguments.  See
the FUSE project (Also discussed in this thread), for better ideas on  
how
to add strange filesystem semantics.  NOTE:  Even the FUSE project,  
which
is in _userspace_ (as opposed to the massive kernelspace mess of  
reiser4),
is taking a lot of flak for changing UNIX semantics, so I see no reason
why Reiser4 should be special.

>> how this can avoid all possible pitfalls,
>
> Especially if you want perfection.

It doesn't need perfection, it just needs to convince a large segment of
the developers on this list (and especially linus).

>> including the automatic encryption/decryption and most other non-  
>> standard
>> filesystem features (Basically the whole '...' directory), should   
>> probably
>> be left out of any patch submitted for inclusion until they can be
>> _proven_ (or at least thoroughly checked) not to have undesirable  
>> results.
>
> I am doing that out of habit.  Actually, it probably ends up more as:
>   .../foo.zip/
> instead of
>   foo.zip/.../
>
> But, it is left out.  At least that interface.

Ok, good.  That's one of the first issues.  A _lot_ of applications  
would get
themselves thoroughly confused at that '...' interface, not to  
mention a lot
of sysadmins :-D.

> Now, the cryptocompress as it currently stands does not involve ...  
> and
> does not introduce any new security holes in the way that you are
> describing.

Good.  I think that we can all agree that, in the event  
cryptocompress is
included, it would be a nice feature to have in all filesystems.   
Therefore,
we should attempt to come up with a clean interface with which it  
could be
added _inside_ the VFS.

> There might be some issues with key management (someone
> hinted vaguely at that), but nothing insurmountable.

Likewise, this should be handled in a common VFS interface that all FSs
can use.  There already exists a key management system that would not be
particularly difficult to interface with, but it would take thought and
discussion.

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to  
make it so simple that there are obviously no deficiencies. And the  
other way is to make it so complicated that there are no obvious  
deficiencies.
  -- C.A.R. Hoare

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kyle Moffett wrote:
> On Jun 26, 2005, at 23:24:23, David Masover wrote:
> 
>>> Neither do I want the kernel to unzip it, because that just  
>>> introduces the
>>> kernel to a whole series of normally application-level  vulnerabilities.
>>
>>
>> That just means the zip plugin has to be more carefully written than I
>> thought.  It would have to be sandboxed and limited to prevent buffer
>> overruns and zip bombs.
> 
> 
> 
> 
>> Remember that zip, at least, would not be in the kernel.  I think what
>> is going in the kernel is gzip and lzo, and it's being done so
>> transparently that you never actually see the compressed version.
> 
> 
> 
>>> Can you illustrate for me with precise, clear, and unambiguous 
>>> arguments
>>
>>
>> That will take some time.  And some thinking.
> 
> 
> Precisely.  Come back when you have a good sturdy set of arguments.  See
> the FUSE project (Also discussed in this thread), for better ideas on  how
> to add strange filesystem semantics.

If I didn't care about performance.  I will read up on how FUSE works,
though, to see if there's anything in the _kernel_ code that would be
useful.

> NOTE:  Even the FUSE project,  which
> is in _userspace_ (as opposed to the massive kernelspace mess of  reiser4),
> is taking a lot of flak for changing UNIX semantics, so I see no reason
> why Reiser4 should be special.

Right.  Kernel people hate change.  Got it.

No, seriously, I have to respect the history involved, which is why I'm
not pretending to know more than I do.

> Ok, good.  That's one of the first issues.  A _lot_ of applications 
> would get
> themselves thoroughly confused at that '...' interface, not to  mention
> a lot
> of sysadmins :-D.

Not as much as you think.  Unless a lot of applications can't handle
opening or saving files that have '...' in the pathname?

The sysadmin problem doesn't worry me so much.

>> Now, the cryptocompress as it currently stands does not involve ...  and
>> does not introduce any new security holes in the way that you are
>> describing.
> 
> 
> Good.  I think that we can all agree that, in the event  cryptocompress is
> included, it would be a nice feature to have in all filesystems.  
> Therefore,
> we should attempt to come up with a clean interface with which it  could be
> added _inside_ the VFS.

Agreed.  I don't know enough about VFS to propose one, though.  I think
others are working on that, finally.

>> There might be some issues with key management (someone
>> hinted vaguely at that), but nothing insurmountable.
> 
> 
> Likewise, this should be handled in a common VFS interface that all FSs
> can use.  There already exists a key management system that would not be
> particularly difficult to interface with, but it would take thought and
> discussion.

Not too much, I hope.

Although being able to use different keys on a per-file basis would be
nice.  I'm not sure if that would work in the existing system.  Not to
mention that keys would also be handlable with '...', if/when it happens.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsBtWXgHNmZLgCUhAQI5gw/8DXFk/v3faGbAtv0vdeMmy5aZ4MSpr3pp
+rjjb8Wx6twCZelmPQgwTBtSQec9XUEipW3ucHVeP5Y2LM8hZHfbxEpHCzDydlH0
AppxZfHp/FBlqROSYajcc4qX062QQ7Wc/VfePmvVKcC7Plo1n1UnF30RRFLR67/r
SOwb4/GPDnYyY0SQQt8XmuFmu1ngRB5M/kFgGOkiGLOsRiPzx4zVnYsxnOp8vH/q
l0SxCsjev4+dL0TRnFBhx/uw7GerZJhqjjB9DZZgMIXILXt8oTHj1ubEXD0WRQxr
iwiXruO2rO3vFbJVXzdlhFnUj98ViQR9nALa+CwT779Zus9hx8/KAxScPXEZPzVz
jdF7y7GUvBL9vR+t5TTJjlpSqyfiyYKEw038/li/uBB1ThXtjTx4uL0QLzWkaaOh
mfdX5RFwyinrsmscBaYgUPHwqLTjgcIWwqSfmGQ9RvYbMkHy9ZOzHTP9chJIeD5F
cuJPluPEOMpeQduF/S7sG/Y00eQaMx0nBxtDprkwonnhy3Qw2jSSgsXqEIG+x45P
3qjiK7/4cZSkN/Q/VkObEwD0GE2FD31yfqSkrkGRMkhOg22YWBVqrwvmcm2RyQk9
h9S9C3HzNXGAuW9HfCmWLqg1yAQCilkTdGRpc38unBa364nRlBzii7PkA3XLol3S
zChIED+O964=
=shbE
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Kyle Moffett]

On Jun 27, 2005, at 17:19:21, David Masover wrote:
>> Precisely.  Come back when you have a good sturdy set of  
>> arguments.  See
>> the FUSE project (Also discussed in this thread), for better ideas  
>> on  how
>> to add strange filesystem semantics.
>
> If I didn't care about performance.  I will read up on how FUSE works,
> though, to see if there's anything in the _kernel_ code that would be
> useful.

A number of projects do their performance critical things in userspace,
including real-time audio and video editing.  The kernel is *NOT* for
performance-critical things, unless they cannot be done efficiently in
userspace.  It is for _security_critical_ and _stability_critical_  
things,
and even then, some of those are pushed to userspace as well.

>> NOTE:  Even the FUSE project,  which
>> is in _userspace_ (as opposed to the massive kernelspace mess of   
>> reiser4),
>> is taking a lot of flak for changing UNIX semantics, so I see no  
>> reason
>> why Reiser4 should be special.
>
> Right.  Kernel people hate change.  Got it.
>
> No, seriously, I have to respect the history involved, which is why  
> I'm
> not pretending to know more than I do.

There is a good reason to hate change (at least without thorough  
evidence
to back it up).  Even small changes can break things in subtle ways.   
Big
changes can tend to wreak kernel-global (and even userspace-global)  
havoc.

>> Ok, good.  That's one of the first issues.  A _lot_ of applications
>> would get themselves thoroughly confused at that '...' interface, not
>> to  mention a lot of sysadmins :-D.
>>
>
> Not as much as you think.  Unless a lot of applications can't handle
> opening or saving files that have '...' in the pathname?

If those files are as special as you lead me to believe, even a simple
"tar -cjvf foo.tar.bz2 foo" would break horribly, because it would  
tar up
all sorts of strange special files that shouldn't be included.  Worse,
when I untar that archive on the filesystem, it will overwrite those  
files,
which probably should be overwritten even less than they should be  
stored
in an archive.

> The sysadmin problem doesn't worry me so much.

Personally, I know several sysadmins who, never having heard of Reiser4
but using it anyways, would get scared when all of the '...' directories
started showing up, thinking that some cracker had gotten a module  
loaded
in their kernel.  If you can, please avoid overloading existing  
semantics
in filesystems.  You can either claim that your filesystem is POSIXy,
(similar to ext3, xfs, etc), or you can claim that it's not  
compatible by
design (AFS, Coda) or you can claim that it's a completely virtualized
interface (procfs, sysfs, CKRM fs, etc) and doesn't care about POSIX in
any case.  Reiser4 definitely isn't the third, but neither is it  
either of
the first two, because it would break standard operations.

> Agreed.  I don't know enough about VFS to propose one, though.  I  
> think
> others are working on that, finally.

When the VFS work gets done, this discussion can move on, otherwise, we
are stuck at an impasse.

> Not too much, I hope.
>
> Although being able to use different keys on a per-file basis would be
> nice.  I'm not sure if that would work in the existing system.  Not to
> mention that keys would also be handlable with '...', if/when it  
> happens.

I think the '...' is just a bad idea in general, because it breaks tar
and such.  An interface like this might be simpler to design and use:

# mkdir /attr
# mount -t attrfs attrfs /attr

/attr/fd/$FD_NUM              == '...' directory for a filedescriptor
/attr/fs/$DEV_NUM/$INODE_NUM  == '...' directory for an inode

It would be usable from a shell with a simple "getattrpath" command
which returns "fs/$DEV_NUM/$INODE_NUM" by stat-ing any given path.

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to  
make it so simple that there are obviously no deficiencies. And the  
other way is to make it so complicated that there are no obvious  
deficiencies.
  -- C.A.R. Hoare

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kyle Moffett wrote:
> On Jun 27, 2005, at 17:19:21, David Masover wrote:
> 
>>> Precisely.  Come back when you have a good sturdy set of  arguments. 
>>> See
>>> the FUSE project (Also discussed in this thread), for better ideas 
>>> on  how
>>> to add strange filesystem semantics.
>>
>>
>> If I didn't care about performance.  I will read up on how FUSE works,
>> though, to see if there's anything in the _kernel_ code that would be
>> useful.
> 
> 
> A number of projects do their performance critical things in userspace,
> including real-time audio and video editing.  The kernel is *NOT* for
> performance-critical things, unless they cannot be done efficiently in
> userspace.

I don't think transparent compress-on-flush can be done efficiently in
userspace.

> It is for _security_critical_ and _stability_critical_  things,
> and even then, some of those are pushed to userspace as well.

There's more to it than that.  It is also for certain interfaces that
makes more sense in kernel space, or are there for historical reasons.
For instance, a filesystem could be done in userspace, with a userspace
library, but even FUSE uses the kernel for its interface, rather than
(say) glibc.

>>> NOTE:  Even the FUSE project,  which
>>> is in _userspace_ (as opposed to the massive kernelspace mess of  
>>> reiser4),
>>> is taking a lot of flak for changing UNIX semantics, so I see no  reason
>>> why Reiser4 should be special.
>>
>>
>> Right.  Kernel people hate change.  Got it.
>>
>> No, seriously, I have to respect the history involved, which is why  I'm
>> not pretending to know more than I do.
> 
> There is a good reason to hate change (at least without thorough  evidence
> to back it up).  Even small changes can break things in subtle ways.   Big
> changes can tend to wreak kernel-global (and even userspace-global)  havoc.

Right on all points.  Just remember that some change is good.  Why do we
have ALSA now?  Everything a user can do with ALSA, they can do with
OSS, AFAIK.

>>> Ok, good.  That's one of the first issues.  A _lot_ of applications
>>> would get themselves thoroughly confused at that '...' interface, not
>>> to  mention a lot of sysadmins :-D.
>>>
>>
>> Not as much as you think.  Unless a lot of applications can't handle
>> opening or saving files that have '...' in the pathname?
> 
> 
> If those files are as special as you lead me to believe, even a simple
> "tar -cjvf foo.tar.bz2 foo" would break horribly, because it would  tar up
> all sorts of strange special files that shouldn't be included.  Worse,
> when I untar that archive on the filesystem, it will overwrite those 
> files,
> which probably should be overwritten even less than they should be  stored
> in an archive.

Point well taken, although sometimes that's exactly what you want.  You
no longer need the -p flag to Tar that way, although you would want its
opposite, so that you can ignore '...'
> 
>> The sysadmin problem doesn't worry me so much.
> 
> 
> Personally, I know several sysadmins who, never having heard of Reiser4
> but using it anyways, would get scared when all of the '...' directories
> started showing up, thinking that some cracker had gotten a module  loaded
> in their kernel.

I'm not sure if the '...' or '..metas' directory is currently available
in Reiser4, but at one point it was the default, then it got changed to
a mount option which is off by default.

> If you can, please avoid overloading existing  semantics
> in filesystems.  You can either claim that your filesystem is POSIXy,
> (similar to ext3, xfs, etc), or you can claim that it's not  compatible by
> design (AFS, Coda) or you can claim that it's a completely virtualized
> interface (procfs, sysfs, CKRM fs, etc) and doesn't care about POSIX in
> any case.  Reiser4 definitely isn't the third, but neither is it  either of
> the first two, because it would break standard operations.

Reiser4 is attempting to extend POSIX without breaking it too much.

>> Agreed.  I don't know enough about VFS to propose one, though.  I  think
>> others are working on that, finally.
> 
> When the VFS work gets done, this discussion can move on, otherwise, we
> are stuck at an impasse.

Not my department.  It seems that people are agreeing on what needs to
be done, though.

>> Not too much, I hope.
>>
>> Although being able to use different keys on a per-file basis would be
>> nice.  I'm not sure if that would work in the existing system.  Not to
>> mention that keys would also be handlable with '...', if/when it 
>> happens.
> 
> 
> I think the '...' is just a bad idea in general, because it breaks tar
> and such.  An interface like this might be simpler to design and use:
> 
> # mkdir /attr
> # mount -t attrfs attrfs /attr
> 
> /attr/fd/$FD_NUM              == '...' directory for a filedescriptor
> /attr/fs/$DEV_NUM/$INODE_NUM  == '...' directory for an inode
> 
> It would be usable from a shell with a simple "getattrpath" command
> which returns "fs/$DEV_NUM/$INODE_NUM" by stat-ing any given path.

Still pretty annoying, but maybe a good idea, especially if the shell
gets extended to support it.  Not sure I like using inode numbers,
though -- I like the idea of being able to symlink to stuff inside the
meta-file dir.

Actually, I like this.  Give me some time to let it percolate.

Hans, thoughts?  Seems to be namespace fragmentation, but seems usable,
less breakage, and so on.  And should it be /attr or /meta?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsCLXXgHNmZLgCUhAQIXjQ/+OQc+KqDl+ybutM6Ba2KmaiwAJTdpIi40
DAO6n/vRkdO87r0vLU+g+3kpWyTqkIMrvGmbvm2dle/d+Q5yZc3fc3Lmg9tuwXyO
GDMA/Lhx/1ckVX3EtcQ8PUomhcCvCFebR/y8IuNgEEthsHcBHPaZD32xkke9Q7u7
0zUbFCry5mgiN3EuSGMy5q4ZvvETUteK/xoEcTGAhh3K3JMTp0N8dHGR8XkGb+Z9
qFG3mYu8g2uRAsZ6hFScQ/rf4j3fu9012Q0SxvEZoTLTNJjFnZhSRqyNSY4i24iV
OVYfstJrnpEWEAeEn29sTk0Pp5cHc8dh/Ure0MCOVHZUnMVD9xy259+3ivavePPn
N27Z7dMmqvf4bKdjqa35tK7bao78Of8EkGeW3v61VJe3qsG45SwpiLmYG/uEvtIw
u/9e/RlWQYyNS9k8eIxY6rjcrPjL4HErFiHmcw36wUmjyDqNYDlKtfsq0+KDGfOe
7I1G7TQc7K3uJu41XEqsG3IVQpY3o6bW8m7WyicRoOg3Qco+g/cjBBERmfAZh65W
t+t7tzk4p+UZfvSiqSfkRnm2CITrqFC0VW2c1pk42BmY0rJPfzRmVU5oBGlH1Avd
j75UFJi8qwm4d2S1InX8bxPoLaZpSjQGPAbpcKpZaaiM5mB3pfj62UVQV5hks9lt
QNzi7/gMmto=
=I2R/
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hubert Chan]

On Mon, 27 Jun 2005 18:27:26 -0500, David Masover <ninja@slaphack.com> said:

> Kyle Moffett wrote:
>> I think the '...' is just a bad idea in general, because it breaks
>> tar and such.  An interface like this might be simpler to design and
>> use:
>> 
>> # mkdir /attr
>> # mount -t attrfs attrfs /attr
>> 
>> /attr/fd/$FD_NUM == '...' directory for a filedescriptor
>> /attr/fs/$DEV_NUM/$INODE_NUM == '...' directory for an inode

The most obvious (at least obvious for me) complaint about this is that
the attributes are separate from the file.  (In other words, it's a bit
ugly. ;-) ) So if you want to backup a file, along with all its
(extended) attributes (or substreams, or ... etc. ...), you need to
backup the file, and find the appropriate /attr/fs/$DEV_NUM/$INODE_NUM
and back that up.  If I want to edit an attribute, I need to find
$DEV_NUM and $INODE_NUM (which I have no idea how to do), rather than
just "edit foo/.../extended_attribute".  (Or using your "getattrpath"
command, it would be a two-step process -- run getattrpath, then run
editor -- rather than a one-step process.  Of course, this is only
mildly annoying.)

It also exposes a difference between attributes and regular files.
e.g. can I add attributes to an attribute?  (say, I have a thumbnail
attribute for the file ~/foo, and I want to add a mime-type attribute to
that thumbnail attribute.)  If you want to allow it, you have to be
careful not to run into a big loop generating an infinite number of
inodes in the attrfs. (e.g.
/attr/fs/$(getattrpath /attr/fs/$(getattrpath ~/foo)/thumbnail)/mimetype
-- attrfs shouldn't generate the inode for that until
/attr/fs/$(getattrpath~/foo)/thumbnail is accessed, and maybe not even
then...)

That said, I prefer that interface over xattr or openat.

>> It would be usable from a shell with a simple "getattrpath" command
>> which returns "fs/$DEV_NUM/$INODE_NUM" by stat-ing any given path.

> Still pretty annoying, but maybe a good idea, especially if the shell
> gets extended to support it.  Not sure I like using inode numbers,
> though -- I like the idea of being able to symlink to stuff inside the
> meta-file dir.

The advantage of using inodes rather than pathnames is that it is robust
with respect to file renaming/moving.  It also allows things like
adding attributes to symlinks (since the inode number for a symlink is
different from the inode number for the file that it points to).

You can still symlink.  It just takes a little more effort to figure out
what $DEV_NUM/$INODE_NUM to use.

> Actually, I like this.  Give me some time to let it percolate.

> Hans, thoughts?  Seems to be namespace fragmentation, but seems
> usable, less breakage, and so on.  And should it be /attr or /meta?

For the mount point, it doesn't matter; it's up to the user.  It's the
attrfs or metafs or ???fs that matters (but which will greatly influence
whether people user /attr or /meta).

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Kyle Moffett]

On Jun 27, 2005, at 22:21:35, Hubert Chan wrote:
> On Mon, 27 Jun 2005 18:27:26 -0500, David Masover  
> <ninja@slaphack.com> said:
>> Kyle Moffett wrote:
>>
>>> I think the '...' is just a bad idea in general, because it breaks
>>> tar and such.  An interface like this might be simpler to design and
>>> use:
>>>
>>> # mkdir /attr
>>> # mount -t attrfs attrfs /attr
>>>
>>> /attr/fd/$FD_NUM == '...' directory for a filedescriptor
>>> /attr/fs/$DEV_NUM/$INODE_NUM == '...' directory for an inode
>
> The most obvious (at least obvious for me) complaint about this is  
> that
> the attributes are separate from the file.  (In other words, it's a  
> bit
> ugly. ;-) ) So if you want to backup a file, along with all its
> (extended) attributes (or substreams, or ... etc. ...), you need to
> backup the file, and find the appropriate /attr/fs/$DEV_NUM/$INODE_NUM
> and back that up.  If I want to edit an attribute, I need to find
> $DEV_NUM and $INODE_NUM (which I have no idea how to do), rather than
> just "edit foo/.../extended_attribute".  (Or using your "getattrpath"
> command, it would be a two-step process -- run getattrpath, then run
> editor -- rather than a one-step process.  Of course, this is only
> mildly annoying.)

These are not really "attributes" so much as they are "metadata", for
example, a "contents" subdirectory, if one existed, would be based on
the original file, and therefore non-unique, but would be looked up
based on information about the original file.

> It also exposes a difference between attributes and regular files.
> e.g. can I add attributes to an attribute?  (say, I have a thumbnail
> attribute for the file ~/foo, and I want to add a mime-type  
> attribute to
> that thumbnail attribute.)  If you want to allow it, you have to be
> careful not to run into a big loop generating an infinite number of
> inodes in the attrfs. (e.g.
> /attr/fs/$(getattrpath /attr/fs/$(getattrpath ~/foo)/thumbnail)/ 
> mimetype
> -- attrfs shouldn't generate the inode for that until
> /attr/fs/$(getattrpath~/foo)/thumbnail is accessed, and maybe not even
> then...)

Agreed.  I discuss this more below

> That said, I prefer that interface over xattr or openat.

Absolutely.  On the other hand, that's not to say it can't be improved.

>
>>> It would be usable from a shell with a simple "getattrpath" command
>>> which returns "fs/$DEV_NUM/$INODE_NUM" by stat-ing any given path.
>
>> Still pretty annoying, but maybe a good idea, especially if the shell
>> gets extended to support it.  Not sure I like using inode numbers,
>> though -- I like the idea of being able to symlink to stuff inside  
>> the
>> meta-file dir.
>>
>
> The advantage of using inodes rather than pathnames is that it is  
> robust
> with respect to file renaming/moving.  It also allows things like
> adding attributes to symlinks (since the inode number for a symlink is
> different from the inode number for the file that it points to).
>
> You can still symlink.  It just takes a little more effort to  
> figure out
> what $DEV_NUM/$INODE_NUM to use.

Also, unlike a symlink, if the path doesn't change and the file does, it
will break, also, if the file is removed and another created  
elsewhere, it
will be redirected improperly.  Perhaps a new symlink syntax is  
needed to
allow attribute specification (Ick, more changes :-\).

>
>> Hans, thoughts?  Seems to be namespace fragmentation, but seems
>> usable, less breakage, and so on.  And should it be /attr or /meta?
>
> For the mount point, it doesn't matter; it's up to the user.  It's the
> attrfs or metafs or ???fs that matters (but which will greatly  
> influence
> whether people user /attr or /meta).

"meta" seems the more descriptive name.  There should also probably be a
somewhat standardized location for this, such that programs can  
locate it
without much trouble.  This mechanism would be usable from all FSs, and
could be built into the VFS.  Also, it would allow one to access the  
meta
data of meta data (if supported by the filesystem, and possibly only
through the file descriptor lookup, due to numbering limitations.)

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to  
make it so simple that there are obviously no deficiencies. And the  
other way is to make it so complicated that there are no obvious  
deficiencies.
  -- C.A.R. Hoare

Re: reiser4 plugins

[Hubert Chan]

On Mon, 27 Jun 2005 22:59:24 -0400, Kyle Moffett <mrmacman_g4@mac.com> said:

> On Jun 27, 2005, at 22:21:35, Hubert Chan wrote:
>> On Mon, 27 Jun 2005 18:27:26 -0500, David Masover
>> <ninja@slaphack.com> said:
>>> Kyle Moffett wrote:

[...]

>>>> /attr/fd/$FD_NUM == '...' directory for a filedescriptor
>>>> /attr/fs/$DEV_NUM/$INODE_NUM == '...' directory for an inode

[...]

> These are not really "attributes" so much as they are "metadata", for
> example, a "contents" subdirectory, if one existed, would be based on
> the original file, and therefore non-unique, but would be looked up
> based on information about the original file.

I think for most people on the reiser-fs list, the '...' directory
represents an interface to many things including
- automatic aggregating/tar/untar/compress
- a different interface to stat data
- adding extended attributes/substreams/acls/etc.
- anything else you might imagine
(I think this is your understanding too?  Just double-checking.)
So some of that stuff would be separate from the file.  (Separate in the
sense that it's not generated from the file's binary data.)

Personally, I don't like it all being in one directory, but it's not
that important.

>> You can still symlink.  It just takes a little more effort to figure
>> out what $DEV_NUM/$INODE_NUM to use.

> Also, unlike a symlink, if the path doesn't change and the file does,
> it will break, also, if the file is removed and another created
> elsewhere, it will be redirected improperly.  Perhaps a new symlink
> syntax is needed to allow attribute specification (Ick, more changes
> :-\).

I think those breakages are acceptable.  (IMHO) In other words, I think
it would not occur very often without the user being aware of it, and
should very rarely result in catastrophic effects.

One other minor annoyance is it isn't easy to go backwards from the
... directory to the file.  e.g. if I have a symlink that points to
/attr/fs/2/92036, I don't know what file's attributes that refers to.
Hopefully I'm sane enough to give the symlink a descriptive enough
name...

>>> Hans, thoughts?  Seems to be namespace fragmentation, but seems
>>> usable, less breakage, and so on.  And should it be /attr or /meta?
>>  For the mount point, it doesn't matter; it's up to the user.  It's
>> the attrfs or metafs or ???fs that matters (but which will greatly
>> influence whether people user /attr or /meta).

> "meta" seems the more descriptive name.  There should also probably be
> a somewhat standardized location for this, such that programs can
> locate it without much trouble.

Agreed.  Ultimately, it's the user's decision where to put it, but
probably 99.99999% of all people will put it in the same place.  Just
like you could put procfs or sysfs somewhere other than /proc or /sys if
you really wanted to, but nobody does that.

> This mechanism would be usable from all FSs, and could be built into
> the VFS.  Also, it would allow one to access the meta data of meta
> data (if supported by the filesystem, and possibly only through the
> file descriptor lookup, due to numbering limitations.)

One other issue is that the attrfs/metafs needs to communicate with the
other filesystem somehow.  It needs to know if the filesystem can handle
storing of extended attributes/substreams/etc. so that it knows whether
or not to allow those interfaces.  In my
/attr/fs/$(getattrpath /attr/fs/$(getattrpath ~/foo)/thumbnail)/mimetype
example, it needs to be smart enough to store that in ~/foo's
filesystem.  etc.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Kyle Moffett]

On Jun 27, 2005, at 23:45:00, Hubert Chan wrote:
> On Mon, 27 Jun 2005 22:59:24 -0400, Kyle Moffett  
> <mrmacman_g4@mac.com> said:
>> /attr/fd/$FD_NUM == '...' directory for a filedescriptor
>> /attr/fs/$DEV_NUM/$INODE_NUM == '...' directory for an inode
>
> [...]
>
>> These are not really "attributes" so much as they are "metadata", for
>> example, a "contents" subdirectory, if one existed, would be based on
>> the original file, and therefore non-unique, but would be looked up
>> based on information about the original file.
>
> I think for most people on the reiser-fs list, the '...' directory
> represents an interface to many things including
> - automatic aggregating/tar/untar/compress
> - a different interface to stat data
> - adding extended attributes/substreams/acls/etc.
> - anything else you might imagine
> (I think this is your understanding too?  Just double-checking.)
> So some of that stuff would be separate from the file.  (Separate  
> in the
> sense that it's not generated from the file's binary data.)

Ok.  Those things should probably be divided up.  Stuff like POSIX
extended attributes and such that have existing interfaces should use
those.  Other types of data should chose other interfaces that make
the most sense for that type of data.  I think that the /meta fs
should probably only be used when the data is generated from the
existing file or directory, and perhaps a few other cases.

> Personally, I don't like it all being in one directory, but it's not
> that important.

I agree with the first, but I think that the implementation and usage
confusion will be one of the barriers preventing '...' or metafs merge.

>> Also, unlike a symlink, if the path doesn't change and the file does,
>> it will break, also, if the file is removed and another created
>> elsewhere, it will be redirected improperly.  Perhaps a new symlink
>> syntax is needed to allow attribute specification (Ick, more changes
>> :-\).
>
> I think those breakages are acceptable.  (IMHO) In other words, I  
> think
> it would not occur very often without the user being aware of it, and
> should very rarely result in catastrophic effects.

Agreed.  On the other hand, perhaps filesystems that support such meta
data should internally assign the appropriate numbering to make it work
nicely.

> One other minor annoyance is it isn't easy to go backwards from the
> ... directory to the file.  e.g. if I have a symlink that points to
> /attr/fs/2/92036, I don't know what file's attributes that refers to.
> Hopefully I'm sane enough to give the symlink a descriptive enough
> name...

I don't see this occurring often enough to be a major problem, and in
any case inodes are not path-exclusive (think hardlinks).  If they have
the filedescriptor open, however, they could use /proc/$PID/* to figure
out where the file is.

>> "meta" seems the more descriptive name.  There should also  
>> probably be
>> a somewhat standardized location for this, such that programs can
>> locate it without much trouble.
>
> Agreed.  Ultimately, it's the user's decision where to put it, but
> probably 99.99999% of all people will put it in the same place.  Just
> like you could put procfs or sysfs somewhere other than /proc or / 
> sys if
> you really wanted to, but nobody does that.

Yeah.  I was referring more to calling it "metafs" than "attrfs",
because it seems it would be more than just attributes.

> One other issue is that the attrfs/metafs needs to communicate with  
> the
> other filesystem somehow.  It needs to know if the filesystem can  
> handle
> storing of extended attributes/substreams/etc. so that it knows  
> whether
> or not to allow those interfaces.

Those would be tied to file descriptor (/meta/fd) or filesystem device
ID (/meta/fs/$DEV), and would therefore be accessible from metafs in a
similar way to how filedescriptor path info is available from procfs.

> In my
> /attr/fs/$(getattrpath /attr/fs/$(getattrpath ~/foo)/thumbnail)/ 
> mimetype
> example, it needs to be smart enough to store that in ~/foo's
> filesystem.  etc.

Agreed.  On another note, it's nice to see the flamewar has died out
and several technical discussions are taking place on various  
levels :-D.

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to  
make it so simple that there are obviously no deficiencies. And the  
other way is to make it so complicated that there are no obvious  
deficiencies.
  -- C.A.R. Hoare

Re: reiser4 plugins

[Hubert Chan]

Agree with most of the stuff you wrote.

On Tue, 28 Jun 2005 00:38:38 -0400, Kyle Moffett <mrmacman_g4@mac.com> said:

> On Jun 27, 2005, at 23:45:00, Hubert Chan wrote:

>> I think for most people on the reiser-fs list, the '...' directory
>> represents an interface to many things including
>> - automatic aggregating/tar/untar/compress
>> - a different interface to stat data
>> - adding extended attributes/substreams/acls/etc.
>> - anything else you might imagine

> Ok.  Those things should probably be divided up.  Stuff like POSIX
> extended attributes and such that have existing interfaces should use
> those.

One of the claimed advantages of the '...' interface is that everything
is editable as a file.  So if someone wanted to edit the description
extended attribute for foo.txt, he would just run
"[editor] foo.txt/.../description" (or
"[editor] /meta/fs/$(getattrpath foo.txt)/description"), instead of
needing to use some special purpose editor.  It works well for things
like being able to use Gimp to edit a thumbnail or icon attribute.

Of course, it shouldn't be too hard to provide both interfaces, as long
as you get locking and caching right...

The Reiser4 file-as-dir interface is also supposed to be more flexible
than POSIX extended attributes.  For example adding attributes to
attributes (e.g. adding a mimetype attribute to a thumbnail attribute).
The point of it is to present many different things (extended
attributes, file manipulation magic like automatic compress, etc.)
through a single interface that everyone knows how to use (i.e. regular
files) so that people can use regular programs to edit or manipulate
them.

The inspiration, I think, was the MacOS X/NeXTSTEP bundle format.  For
example, MacOS X/NeXTSTEP .app file is actually a directory that behaves
much like an executable file (double-clicking a .app file in the Finder
launches the application, instead of opening the directory).  However,
it is in reality a directory that contains many things that could be
thought of as extended attributes (such as the application icon,
information about the application, etc.).  Since the application icon is
a real file, it can be edited by normal graphics editors (not like
Windows programs, where you need a special icon editor).  And since it's
inside the .app directory, it's attached to the application (not like
Linux, where the program is in /usr/bin, and the icon is in
/usr/share/pixmaps), so it makes package management easier (to delete an
application, just delete the .app file -- don't need to look in
/usr/share/pixmaps for the icon and delete it).

> Other types of data should chose other interfaces that make the most
> sense for that type of data.  I think that the /meta fs should
> probably only be used when the data is generated from the existing
> file or directory, and perhaps a few other cases.

[...]

>> One other minor annoyance is it isn't easy to go backwards from the
>> ... directory to the file.  e.g. if I have a symlink that points to
>> /attr/fs/2/92036, I don't know what file's attributes that refers to.
>> Hopefully I'm sane enough to give the symlink a descriptive enough
>> name...

> I don't see this occurring often enough to be a major problem,

I don't see that this should be a major problem either, but I thought it
was worth bringing up.

> and in any case inodes are not path-exclusive (think hardlinks).  If
> they have the filedescriptor open, however, they could use
> /proc/$PID/* to figure out where the file is.

Although I guess if they have a file descriptor open, they probably have
the original filename lying around somewhere...

[...]

> On another note, it's nice to see the flamewar has died out and
> several technical discussions are taking place on various levels :-D.

Agreed! :-D

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Kyle Moffett]

On Jun 28, 2005, at 01:30:08, Hubert Chan wrote:
> On Tue, 28 Jun 2005 00:38:38 -0400, Kyle Moffett  
> <mrmacman_g4@mac.com> said:
>> Ok.  Those things should probably be divided up.  Stuff like POSIX
>> extended attributes and such that have existing interfaces should use
>> those.
>
> One of the claimed advantages of the '...' interface is that  
> everything
> is editable as a file.  So if someone wanted to edit the description
> extended attribute for foo.txt, he would just run
> "[editor] foo.txt/.../description" (or
> "[editor] /meta/fs/$(getattrpath foo.txt)/description"), instead of
> needing to use some special purpose editor.  It works well for things
> like being able to use Gimp to edit a thumbnail or icon attribute.

I don't disagree with the thumbnail/icon/description, but things like
POSIX acls and extended attributes have _existing_ interfaces which
should be used.  I don't deny them the right to add other interfaces
later, but such should be a secondary or tertiary patch, after the
rest of the stuff is in.  In any case, if we were to provide an
interface by which one could $EDITOR the POSIX ACLs, it should be
done in the VFS where all filesystems can share it.

> The inspiration, I think, was the MacOS X/NeXTSTEP bundle format.  For
> example, MacOS X/NeXTSTEP .app file is actually a directory that  
> behaves
> much like an executable file (double-clicking a .app file in the  
> Finder
> launches the application, instead of opening the directory).  However,
> it is in reality a directory that contains many things that could be
> thought of as extended attributes (such as the application icon,
> information about the application, etc.).  Since the application  
> icon is
> a real file, it can be edited by normal graphics editors (not like
> Windows programs, where you need a special icon editor).  And since  
> it's
> inside the .app directory, it's attached to the application (not like
> Linux, where the program is in /usr/bin, and the icon is in
> /usr/share/pixmaps), so it makes package management easier (to  
> delete an
> application, just delete the .app file -- don't need to look in
> /usr/share/pixmaps for the icon and delete it).

The key difference here is that Mac OS X does all of the bundle mess
in userspace where it belongs. :-D  (I know, I use it daily)  I think
that part of Reiser4 needs more review than it's been given at present.

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to  
make it so simple that there are obviously no deficiencies. And the  
other way is to make it so complicated that there are no obvious  
deficiencies.
  -- C.A.R. Hoare

Re: reiser4 plugins

[Hubert Chan]

On Tue, 28 Jun 2005 02:01:12 -0400, Kyle Moffett <mrmacman_g4@mac.com> said:

> On Jun 28, 2005, at 01:30:08, Hubert Chan wrote:
>> On Tue, 28 Jun 2005 00:38:38 -0400, Kyle Moffett
>> <mrmacman_g4@mac.com> said:
>>> Ok.  Those things should probably be divided up.  Stuff like POSIX
>>> extended attributes and such that have existing interfaces should
>>> use those.
>>  One of the claimed advantages of the '...' interface is that
>> everything is editable as a file.  So if someone wanted to edit the
>> description extended attribute for foo.txt, he would just run
>> "[editor] foo.txt/.../description" (or "[editor]
>> /meta/fs/$(getattrpath foo.txt)/description"), instead of needing to
>> use some special purpose editor.  It works well for things like being
>> able to use Gimp to edit a thumbnail or icon attribute.

> I don't disagree with the thumbnail/icon/description, but things like
> POSIX acls and extended attributes have _existing_ interfaces which
> should be used.

OK.  I agree with that.  Of course, Reiser4 can always present both
interfaces, just like it presented two interfaces to the stat data --
the regular interface and the metas (now '...') interface -- before
file-as-dir got disabled by default.

> I don't deny them the right to add other interfaces later, but such
> should be a secondary or tertiary patch, after the rest of the stuff
> is in.  In any case, if we were to provide an interface by which one
> could $EDITOR the POSIX ACLs, it should be done in the VFS where all
> filesystems can share it.

I don't know if VFS is the right place for it, but I agree that it would
be good to make it accessible to all filesystems.

>> The inspiration, I think, was the MacOS X/NeXTSTEP bundle format.
>> For example, MacOS X/NeXTSTEP .app file is actually a directory that
>> behaves much like an executable file (double-clicking a .app file in
>> the Finder launches the application, instead of opening the
>> directory).  However, it is in reality a directory that contains many
>> things that could be thought of as extended attributes (such as the
>> application icon, information about the application, etc.).  Since
>> the application icon is a real file, it can be edited by normal
>> graphics editors (not like Windows programs, where you need a special
>> icon editor).  And since it's inside the .app directory, it's
>> attached to the application (not like Linux, where the program is in
>> /usr/bin, and the icon is in /usr/share/pixmaps), so it makes package
>> management easier (to delete an application, just delete the .app
>> file -- don't need to look in /usr/share/pixmaps for the icon and
>> delete it).

> The key difference here is that Mac OS X does all of the bundle mess
> in userspace where it belongs. :-D (I know, I use it daily)

Yes.  It's handled by NSWorkspace which is approximately equivalent to
this sort of thing being handled by GnomeVFS or the KDE equivalent.  Of
course the problem with handling it in userspace is that behaviour isn't
uniform -- applications that don't use NSWorkspace (e.g. some
command-line utils, programs ported over from UNIX, etc.) won't have the
same behaviour.  Whether or not that is an actual problem seems to be
debatable.  (I don't use MacOS X, but I've done some programming in
GNUstep.)

Another problem is that it only works with bundle files.  e.g. I can't
add an icon to a regular txt file.  Tiger now supports xattrs, which you
could use for that functionality, but then we run into the problem again
of not being able to edit it with regular applications.

> I think that part of Reiser4 needs more review than it's been given at
> present.

Looking forward to the flamewar that happens when Namesys decides that
file-as-dir is ready to be turned on again. ;-)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hubert Chan wrote:
> On Tue, 28 Jun 2005 02:01:12 -0400, Kyle Moffett <mrmacman_g4@mac.com> said:
> 
> 
>>On Jun 28, 2005, at 01:30:08, Hubert Chan wrote:
>>
>>>On Tue, 28 Jun 2005 00:38:38 -0400, Kyle Moffett
>>><mrmacman_g4@mac.com> said:
>>>
>>>>Ok.  Those things should probably be divided up.  Stuff like POSIX
>>>>extended attributes and such that have existing interfaces should
>>>>use those.
>>>
>>> One of the claimed advantages of the '...' interface is that
>>>everything is editable as a file.  So if someone wanted to edit the
>>>description extended attribute for foo.txt, he would just run
>>>"[editor] foo.txt/.../description" (or "[editor]
>>>/meta/fs/$(getattrpath foo.txt)/description"), instead of needing to
>>>use some special purpose editor.  It works well for things like being
>>>able to use Gimp to edit a thumbnail or icon attribute.
> 
> 
>>I don't disagree with the thumbnail/icon/description, but things like
>>POSIX acls and extended attributes have _existing_ interfaces which
>>should be used.

Any existing interface should be supported, but Reiser4 seems to want to
replace all existing interfaces except a direct open() and a proposed
new system call which opens lots of small files at once, efficiently.

The two approaches are not mutually exclusive, though.

> OK.  I agree with that.  Of course, Reiser4 can always present both
> interfaces, just like it presented two interfaces to the stat data --
> the regular interface and the metas (now '...') interface -- before
> file-as-dir got disabled by default.

The proposal I like right now is /meta, as a separate FS, which provides
access to what would be the '...' interface.  Check the archives.

>>I don't deny them the right to add other interfaces later, but such
>>should be a secondary or tertiary patch, after the rest of the stuff
>>is in.  In any case, if we were to provide an interface by which one
>>could $EDITOR the POSIX ACLs, it should be done in the VFS where all
>>filesystems can share it.
> 
> 
> I don't know if VFS is the right place for it, but I agree that it would
> be good to make it accessible to all filesystems.

You put it in /meta, which is available to all filesystems.

>>I think that part of Reiser4 needs more review than it's been given at
>>present.
> 
> 
> Looking forward to the flamewar that happens when Namesys decides that
> file-as-dir is ready to be turned on again. ;-)

Namesys still hasn't commented directly on the /meta proposal, have
they?  That would avoid the flamewar altogether.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsGl6HgHNmZLgCUhAQK+OQ//Zhj9DPyXRaASbTmnj2oPcWYWHzY5+M8O
n0e7PsFPtVpb6mduDPYu91WTGYooPnicVOBBQezWUOVFYMN4SWxyVunTpDYbvTGn
xVwSLTjb5/AZ7dIuklVfverN72nwmLURCJjGZ41iqyylHJT/nCufYZHhQIPa15Pu
o0IzYlQahS3uxuYVJubDOms0G/K2pDuoFGuJJkk+JTPCbf9+2UEqjiUrL3FTqeG+
5RQ6wVeinz5CFPrUWmxl0CdipGqx3quCCYlBav31uRDJm2VwwqZNTxeEHWvZN/Y9
tXnkcVSVJkoaJtbGHUvu61g/TAs8ckxlEA9lnaG3SEhNNU5ZSm0aprZJTeRvs81e
iFxzlOKdgZA11GNx8rnj04Ii5MJapgQhDlz5CbOCxvOm6SkZZul+3n6VHxlCJzH2
pQD/GFcynPAbZYitOAyTWAU9K+fZ98dbCkDKJRLMbT8uiZwoc/m59m9ZniDVEKAn
IjUMFrIllpq9F8AZefEFqRV6l9vjM2vam5OS2HC4R+v753JEVKuvTjbJkR7mB9iv
zEGPBciVOjZAWuWmeQVartnDwx4ic2SECxv25w32wcPcrXUs9w0Z/hpIUfG7EAUl
z4dbGlKlXWybq8JLC6ojmL+cgKFAYs+4yfZvE73H9XHlhh1HrGbMc6sd342NCFJN
WjYRbkxGym0=
=LZpe
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hubert Chan]

On Tue, 28 Jun 2005 14:32:56 -0500, David Masover <ninja@slaphack.com> said:

> Hubert Chan wrote:
>> On Tue, 28 Jun 2005 02:01:12 -0400, Kyle Moffett
>> <mrmacman_g4@mac.com> said:

>>> I don't deny them the right to add other interfaces later, but such
>>> should be a secondary or tertiary patch, after the rest of the stuff
>>> is in.  In any case, if we were to provide an interface by which one
>>> could $EDITOR the POSIX ACLs, it should be done in the VFS where all
>>> filesystems can share it.

>> I don't know if VFS is the right place for it, but I agree that it
>> would be good to make it accessible to all filesystems.

Sorry, "accessible to all filesystems" is incorrect terminology (it's
backwards).  What I meant was something more like "able to access
extended data from all filesystems".  But I think everyone understands
what I meant.

> You put it in /meta, which is available to all filesystems.

>> Looking forward to the flamewar that happens when Namesys decides
>> that file-as-dir is ready to be turned on again. ;-)

> Namesys still hasn't commented directly on the /meta proposal, have
> they?  That would avoid the flamewar altogether.

Well, we could still have a flamewar about whether metafs is useful,
what type of functionality it should provide, etc.  I'm sure we could
find something to argue about. ;-)

>From a purely philosophical/aesthetic/theoretical point of view, I like
file-as-dir much better than anything else, and would be ecstatic it if
Namesys was able to make it work.  From a practical/technical point of
view, metafs may be an acceptable compromise.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Kyle Moffett]

On Jun 28, 2005, at 15:32:56, David Masover wrote:
> Hubert Chan wrote:
>> On Tue, 28 Jun 2005 02:01:12 -0400, Kyle Moffett  
>> <mrmacman_g4@mac.com> said:
>>> I don't disagree with the thumbnail/icon/description, but things  
>>> like
>>> POSIX acls and extended attributes have _existing_ interfaces which
>>> should be used.
>
> Any existing interface should be supported, but Reiser4 seems to  
> want to
> replace all existing interfaces except a direct open() and a proposed
> new system call which opens lots of small files at once, efficiently.
>
> The two approaches are not mutually exclusive, though.

I don't think most kernel developers are likely to favor an approach
which smashes a bunch of unrelated data together.  Linux syscall
overhead is incredibly low, so I don't see it being much of a problem.
On the other hand, a simple syscall which smashes together a bunch of
related data would not be that big of an issue either.  Let's wait for
Namesys to tinker and come back with a new patch before commenting
further.

Cheers,
Kyle Moffett

--
Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best  
answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't.  
That's why
I draw cartoons. It's my life."
-- Charles Shultz

Re: reiser4 plugins

[Kyle Moffett]

On Jun 28, 2005, at 13:51:04, Hubert Chan wrote:
> On Tue, 28 Jun 2005 02:01:12 -0400, Kyle Moffett  
> <mrmacman_g4@mac.com> said:
>> I don't disagree with the thumbnail/icon/description, but things like
>> POSIX acls and extended attributes have _existing_ interfaces which
>> should be used.
>
> OK.  I agree with that.  Of course, Reiser4 can always present both
> interfaces, just like it presented two interfaces to the stat data --
> the regular interface and the metas (now '...') interface -- before
> file-as-dir got disabled by default.

Yeah, but let's get the normal interface working first and discuss what
form the alternate one should take

>> I don't deny them the right to add other interfaces later, but such
>> should be a secondary or tertiary patch, after the rest of the stuff
>> is in.  In any case, if we were to provide an interface by which one
>> could $EDITOR the POSIX ACLs, it should be done in the VFS where all
>> filesystems can share it.
>
> I don't know if VFS is the right place for it, but I agree that it  
> would
> be good to make it accessible to all filesystems.

That's somewhat of a contradiction in terms.  The whole point of the VFS
is to hold all of the things that multiple filesystems want to  
share :-D.

>> The key difference here is that Mac OS X does all of the bundle mess
>> in userspace where it belongs. :-D (I know, I use it daily)
>
> Yes.  It's handled by NSWorkspace which is approximately equivalent to
> this sort of thing being handled by GnomeVFS or the KDE  
> equivalent.  Of
> course the problem with handling it in userspace is that behaviour  
> isn't
> uniform -- applications that don't use NSWorkspace (e.g. some
> command-line utils, programs ported over from UNIX, etc.) won't  
> have the
> same behaviour.  Whether or not that is an actual problem seems to be
> debatable.  (I don't use MacOS X, but I've done some programming in
> GNUstep.)

Personally, I think that the multiple views is a good thing.  I like
being able to "cd /Applications/Games/UnrealTournament2004.app/System"
and mess with my game files, while double-clicking it in the Finder
just starts it so I can get on with owning my friends :-D.

> Another problem is that it only works with bundle files.  e.g. I can't
> add an icon to a regular txt file.  Tiger now supports xattrs,  
> which you
> could use for that functionality, but then we run into the problem  
> again
> of not being able to edit it with regular applications.

Maybe we just need better regular applications?  I think that for the
icon case, for the Samba/streams case, and for many others I'm probably
forgetting, we should try to come up with a new "data-stream" VFS API,
so that the icon data and other larger quantities can be stored in a
filesystem without much effort.  Such a layer could even be bridged
onto existing filesystems via a VFS-wrapper bind-mount:

# mount -t reiser4 /dev/hda1 /mnt1

# mount -t ext3 /dev/hda2 /mnt2

# cat $(metapath /mnt1/foo)/streams/description
Some random file

# cat $(metapath /mnt2/foo)/streams/description
cat: Unsupported operation

# mount -t none -o bind,streamify /mnt2 /mnt3

# cat $(metapath /mnt3/foo)/streams/description
Another random file

Such a wrapper interface might use the directory '...' to store files on
the underlying filesystem, but I don't think that the meta interface
itself should use those directories.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$  
r  !y?(-)
------END GEEK CODE BLOCK------

Re: reiser4 plugins

[Hubert Chan]

On Tue, 28 Jun 2005 15:59:03 -0400, Kyle Moffett <mrmacman_g4@mac.com> said:

> On Jun 28, 2005, at 13:51:04, Hubert Chan wrote:

>>  I don't know if VFS is the right place for it, but I agree that it
>> would be good to make it accessible to all filesystems.

> That's somewhat of a contradiction in terms.  The whole point of the
> VFS is to hold all of the things that multiple filesystems want to
> share :-D.

VFS provides a common interface to the filesystem.  I don't think metafs
needs any VFS changes.  It may be able to get by without making changes
to the VFS, and if so, it shouldn't touch the VFS.  It should just be
its own separate filesystem.

I imagine most of it could be implemented by a FUSE filesystem.

> Personally, I think that the multiple views is a good thing.  I like
> being able to "cd /Applications/Games/UnrealTournament2004.app/System"
> and mess with my game files, while double-clicking it in the Finder
> just starts it so I can get on with owning my friends :-D.

Of course.  With file-as-dir, you can "cd /usr/games/tetris/..." and
mess with the game files, or you can run "/usr/games/tetris" and get on
with ... stacking blocks.  The advantage of doing it in the kernel is
that you don't need extra support from the applications (or GnomeVFS or
KDE).  So typing "/usr/games/tetris" from the command prompt does the
same thing as double-clicking it in the file manager.  Of course, this
change does require file managers to understand default actions when
it's ambiguous what to do on a double-click -- but MacOS X has that too,
in the form of the "Open as Folder" option (or whatever it's called).

>> Another problem is that it only works with bundle files.  e.g. I
>> can't add an icon to a regular txt file.  Tiger now supports xattrs,
>> which you could use for that functionality, but then we run into the
>> problem again of not being able to edit it with regular applications.

> Maybe we just need better regular applications?

You mean patch them all so that they understand and can edit
xattr/substreams/etc.?  The file-as-dir interface is meant to avoid
having to do that.  metafs also avoids having to patch all the
applications by exposing them as regular files.

The example you give below isn't a new VFS API.  It's metafs exposing
xattrs/substreams/etc. through the regular file interface.  (Perhaps
we're just using different terminology.)

> I think that for the icon case, for the Samba/streams case, and for
> many others I'm probably forgetting, we should try to come up with a
> new "data-stream" VFS API, so that the icon data and other larger
> quantities can be stored in a filesystem without much effort.  Such a
> layer could even be bridged onto existing filesystems via a
> VFS-wrapper bind-mount:

> # mount -t reiser4 /dev/hda1 /mnt1

> # mount -t ext3 /dev/hda2 /mnt2

> # cat $(metapath /mnt1/foo)/streams/description
> Some random file

> # cat $(metapath /mnt2/foo)/streams/description
> cat: Unsupported operation

> # mount -t none -o bind,streamify /mnt2 /mnt3

> # cat $(metapath /mnt3/foo)/streams/description
> Another random file

> Such a wrapper interface might use the directory '...' to store files
> on the underlying filesystem, but I don't think that the meta
> interface itself should use those directories.

Agreed.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Horst von Brand]

Hubert Chan <hubert@uhoreg.ca> wrote:

[...]

> Of course.  With file-as-dir, you can "cd /usr/games/tetris/..." and
> mess with the game files, or you can run "/usr/games/tetris" and get on
> with ... stacking blocks.

And doing "tar cf /dev/tape /usr/games/tetris" gives you a nice tangle of
undecipherable junk.

>                            The advantage of doing it in the kernel is
> that you don't need extra support from the applications (or GnomeVFS or
> KDE).

I don't see any advantage here... it has to be implemented, and doing it
in-kernel for one filesystem is /much/ harder than doing it in userland,
where it'll probably work whatever the underlying filesystem is.

Besides, why should the Tetris scores (or icon, or whatever) reside inside
the executable, when said executable is perhaps shared by a bunch of
machines on the network, each of which would like to keep its own? Or
perhaps the sysadmin is terminally paranoid, and mounts /usr read-only
(Hey, the FHS people defined what goes in there with /exactly/ that in mind
too!). 

>        So typing "/usr/games/tetris" from the command prompt does the
> same thing as double-clicking it in the file manager.

Right. And what should happen if I run the (logically equivalent) directory
/home? Or /etc? What if I want to shove a directory into the tetris
executable? Symlinks? Hard links to files inside? Outside? Symlinks from
the outside in? Hardlinks? And if you now move that to another filesystem,
or ship it to another machine? No, the answer "All that will be forbidden"
is /not/ allowed, you want this stuff working "normally". "Unpack" the
contents into real files and directories how? "Pack" a directory into one
of those structured objects? What would be the /practical/ difference
between the packed and unpacked forms? (If none, why do it in the first
place? If the packed version has significant restrictions, why use it? If
the packed version does have significant extra capabilities, why not just
give those to regular objects?)

Yet again, what somebody (I forget who) called the "Zero - one - infinity
principle" (I think it was in the area of programming languages; which
arguably are very complex user interfaces, just like what an operating
system provides): It only makes sense giving zero, exactly one, or how many
you wish for of some item/nesting. Here: Either files got /no/ strange
things inside, or /exactly one/ (and then it becomes questionable, as there
is no space for regular data anymore...), or whatever complex
(sub)structure you want. But in the last case, there is /no/ difference
between a file and a directory... and the whole setup is just mess for mess
sake, nothing else.

>                                                        Of course, this
> change does require file managers to understand default actions when
> it's ambiguous what to do on a double-click -- but MacOS X has that too,
> in the form of the "Open as Folder" option (or whatever it's called).

Right. For the sake of a filesystem among many on a minority operating
system /all/ GUI programs have to be rewritten. And all command-line
stuff. Just because.


Please /do/ think the above through, giving reasonable answers for /all/ of
the operations mentioned. Tell what the advantage of all this would be on a
multiuser operating system, when files are shared via the net, when files
are being handled from read-only media (or filesystems). When different
users want different metadata (I'm interested in the names of members of
the band playing a song, you might want a high-resolution image of the
album cover, the next guy wants the song's lyrics translated into swahili,
all for the MP3s on the shared server or on the CD each of us bought
separately). Consider the scenario where all this works only on /a very
few/ machines (no Sun or *BSD box will have this for a very long time, and
to get a significant fraction of just Linux systems will take some time),
for only a limited selection of tools (existing tools will have to be
rewritten or at least adapted, that doesn't happen overnight). Factor in
space and processing requirements for several, even hundreds, of concurrent
users (or versions of metadata at least). How do you account for that? The
1 byte file with GiB of metadata where everybody and their pet iguana store
their junk is handled how by quota systems? What should happen if I email
such a file with several versions of metadata to someone? Do they get just
my metadata, or everybody's? If I copy it to, say, a CD for safekeeping for
myself? For system backup purposes? What if I copy such a file (with only
my, or everybody's) metadata back from a backup? Or try to merge it with
the version I took with my notebook on a trip, changing my parts while the
otghers change theirs? I'd assume all of those will requiere special tools,
or at least flags selecting the right operation or some other unspecified
magic, and the "simple to use" just went out for lunch.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Douglas McNaught]

Horst von Brand <vonbrand@inf.utfsm.cl> writes:

> Hubert Chan <hubert@uhoreg.ca> wrote:
>
>>                                                        Of course, this
>> change does require file managers to understand default actions when
>> it's ambiguous what to do on a double-click -- but MacOS X has that too,
>> in the form of the "Open as Folder" option (or whatever it's called).
>
> Right. For the sake of a filesystem among many on a minority operating
> system /all/ GUI programs have to be rewritten. And all command-line
> stuff. Just because.

I'll just note that the "applications bundled as directories" stuff on
MacOS/NextStep is done completely in userspace--as far as the kernel
is concerned, "Mail.app" is a regular directory.  The file manager
handles recognition and invocation of application bundles (and there
is an 'open' shell command that does the same thing).

-Doug

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

On Wed, Jun 29, 2005 at 09:50:27AM -0400, Douglas McNaught wrote:
>
>I'll just note that the "applications bundled as directories" stuff on
>MacOS/NextStep is done completely in userspace--as far as the kernel
>is concerned, "Mail.app" is a regular directory.  The file manager
>handles recognition and invocation of application bundles (and there
>is an 'open' shell command that does the same thing).

Note that MacOS has the monopoly on what they ship, Linux has a
motherload of file managers and window systems and all.

What pisses me off is the fact that Gnome and friends implement
their own incompatible-with-others VFS's and automounters and
stuff.

Surely supporting this in the kernel and extending the LSB
to require this is the best step to take without infringing
anyone's freedom as such.

*still pissed off about having to hassle an automatic mount*

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Horst von Brand]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1532 bytes --]

Markus Törnqvist <mjt@nysv.org> wrote:

[...]

> Note that MacOS has the monopoly on what they ship, Linux has a
> motherload of file managers and window systems and all.

Yep. Part of what is nice about it, too ;-)

> What pisses me off is the fact that Gnome and friends implement
> their own incompatible-with-others VFS's and automounters and
> stuff.

Then get them to agree on a common framework! They are trying hard to get
other parts of the GUI work well together, so this isn't far off in
wishfull thinking land.

> Surely supporting this in the kernel and extending the LSB
> to require this is the best step to take without infringing
> anyone's freedom as such.

Right. So then we have Gnome's way of doing things (Gnome isn't just for
Linux!), KDE's way, XFCE's way, ... (ditto). Plus the kernel way. Flambee
with a monthly thread on all reachable fora about "Why on &%@# does the
%&~#@ GUI not use the *#%&@ kernel's way?!".

This is /not/ the way of fxing that particular problem. Shoving random
stuff into the kernel /can't/ force its use. At least not until Linux is
the only Unixy system around, and that is still some way off. And when that
has happened, the kernel's way must be /clearly/ better for /all/ users, or
it won't matter.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Hubert Chan]

On Wed, 29 Jun 2005 13:19:12 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:

> Markus Törnqvist <mjt@nysv.org> wrote: [...]

>> Note that MacOS has the monopoly on what they ship, Linux has a
>> motherload of file managers and window systems and all.

> Yep. Part of what is nice about it, too ;-)

I guess that's why Apple doesn't want to make its X11 support too good.
Otherwise the Mac world would get flooded with X11 programs that don't
work the same way as the rest of the Mac programs.

>> What pisses me off is the fact that Gnome and friends implement their
>> own incompatible-with-others VFS's and automounters and stuff.

> Then get them to agree on a common framework! They are trying hard to
> get other parts of the GUI work well together, so this isn't far off
> in wishfull thinking land.

I honestly hope you're right.  The more cooperation we have between
GNOME and KDE and everyone else, the better.

>> Surely supporting this in the kernel and extending the LSB to require
>> this is the best step to take without infringing anyone's freedom as
>> such.

> Right. So then we have Gnome's way of doing things (Gnome isn't just
> for Linux!), KDE's way, XFCE's way, ... (ditto). Plus the kernel
> way. Flambee with a monthly thread on all reachable fora about "Why on
> &%@# does the %&~#@ GUI not use the *#%&@ kernel's way?!".

GNOME has HAL, their hardware abstraction layer, which uses (I assume)
the kernel's facility to interact with the hardware.  If someone ports
GNOME to Windows, then HAL would use the Windows way to interact with
the hardware.  I can't see why it can't do something similar with the
filesystem.  Because who knows, maybe Gnumeric will need to access
substream data under Windows to open an Excel file properly.  Or
Nautilus may have to read ... OS/2 extended attributes :-/

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 3324 bytes --]

On Wed, Jun 29, 2005 at 01:19:12PM -0400, Horst von Brand wrote:
>> What pisses me off is the fact that Gnome and friends implement
>> their own incompatible-with-others VFS's and automounters and
>> stuff.
>Then get them to agree on a common framework! They are trying hard to get
>other parts of the GUI work well together, so this isn't far off in
>wishfull thinking land.

Looking at the situ with devfs vs udev, which should have been clear
ages ago, I don't think this will easily happen.

What would a gnome developer say if I told him to bugger off,
for the storm front of ivman is approaching?

Having the mount system in userspace makes sense, it's implemented
and done.

Having metadata in some database chunk in userland makes less
sense. For something that handles files and directories already,
it must be easier to patch them to look at the assorted meta
stuff inside the data object/file-as-dir.

>> Surely supporting this in the kernel and extending the LSB
>> to require this is the best step to take without infringing
>> anyone's freedom as such.
>
>Right. So then we have Gnome's way of doing things (Gnome isn't just for
>Linux!), KDE's way, XFCE's way, ... (ditto). Plus the kernel way. Flambee
>with a monthly thread on all reachable fora about "Why on &%@# does the
>%&~#@ GUI not use the *#%&@ kernel's way?!".

Sure it's not "Dictum, factum" but having the kernel team bless
an extended VFS which allows for metadata in any given FS is
a significantly bigger thing than a pissing contest between Gnome,
KDE and XFCE.

Gnome not being only for Linux, well, make the official stance that
the userspace VFS part is deprecated and to be used only on non-conforming
systems.

I haven't really used Gnome in long times, but if I have a picture
and Nautilus shows me a thumbnail of it, doesn't the thumbnail live
in a metadata VFS in userspace?
That's a lot less usable than an in-kernel solution, where
a serialized file can be sent to a conforming system or just
the most important, or wanted, part of the file to a non-conforming one.

That's the one thing all distros and managers and all have in
common, the kernel.

>This is /not/ the way of fxing that particular problem. Shoving random
>stuff into the kernel /can't/ force its use. At least not until Linux is
>the only Unixy system around, and that is still some way off. And when that
>has happened, the kernel's way must be /clearly/ better for /all/ users, or
>it won't matter.

No use in making bigger deals of other Unixes on which Gnome
and friends may run; my opinion and is that Linux should extend 
things further than the tradition so far has been.

Maybe being better than the competitors may include breaking some
rules, changing the playing field, making it more flexible.

And the file system, and the VFS, is one of those things that are 
nice to have in the kernel.

On a digressing side note, I found this classic:
http://kernel.org/pub/linux/utils/kernel/hotplug/udev_vs_devfs

"If the kernel tomorrow switches to randomly assign major and minor numbers
to different devices, it would work just fine (this is exactly
what I am proposing to do in 2.7...)"

Apparently there isn't a 2.7 and no solid-enough testing ground
for this either..

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

On Thu, Jun 30, 2005 at 05:05:06PM +0400, Nikita Danilov wrote:

>You will have to force user level to use common framework
>anyway.

Naturally.

>Otherwise one application will use
>
>~/pictures/sunset.jpg/...meta/mime-type
>
>and another one
>
>~/pictures/sunset.jpg/...meta/XML-schema-FOOBAR/version-0.99/attributes/common/MIME-type/value
>
>etc. Putting mechanism into kernel doesn't make it somehow magically
>interoperable. It is _policy_ that does. And policy belongs to the
>kernel not.

Indeed this may not be something where Infinite Diversity in
Infinite Combinations is a good thing.

The best way to make sure everything works ok, is to draw out the
specs for a policy and ensure it, but maybe it can't be enforced
in the kernel.

Unless the policy defined which directories (disregard the irony,
please :) in the meta structure are available and off limits for
mkdir et al.

Sure this may be circumvented still by setting up an xml schema
somewhere and so on and so forth.

So my best guess probably is to have LSB extended with a damned
good policy about How Things Work(tm) with the new meta system.

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

On Thu, Jun 30, 2005 at 07:18:47PM +0400, Nikita Danilov wrote:

>Sorry, I don't see your point. Again: if you think that user level
>developers are unlikely to agree to the common framework, what
>difference it makes whether this framework is defined at the kernel or
>library boundary? Applications would have to be changed to conform to
>the common API either way.

I see it as a heavier incentive to do it by a framework that's in
the kernel.

>If you can force application developers to conform to the LSB why you
>cannot do the same with the library level interface?

If I want to access metadata with bash, do I patch bash to support
both Gnome's and KDE's solutions? Was there one of XFCE too?
And FooBarXyzzyWM that'll want to do it's own VFS next year?

I'd also guess that the upstream guys would much rather have
patches for their progs that conform to the kernel than some
obscure neighbor userspace system.

Sure looks like having this in the kernel makes it easiest; there's
just one common denominator to patch for.

This doesn't even invalidate the userland VFSs of the other guys,
they're still needed for systems whose kernels don't have a
metadata facility.

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Diego Calleja]

El Thu, 30 Jun 2005 18:37:38 +0300,
mjt@nysv.org (Markus   Törnqvist) escribió:

> If I want to access metadata with bash, do I patch bash to support
> both Gnome's and KDE's solutions? Was there one of XFCE too?
> And FooBarXyzzyWM that'll want to do it's own VFS next year?


The freedesktop people is already designing a common userspace VFS,
take a look at http://freedesktop.org/Software/dvfs . At least it looks like
there'll be a common standard, even if the standard is crappy.

Apparently one of the main reasons for DVFS to exist is "POSIX is not
enought for us". 

This approach  is broken by design, having a second-class VFS separated
from the main VFS (what libc functions see, etc) is wrong, IMO.

The Mockup project (they try to bring a BeOS-like API to linux, I think) is
something similar to DVFS but using FUSE.
Take a look at  http://wiki.mockup.org
(currently down, google cache at
http://66.249.93.104/search?q=cache:OnvclDCpwOwJ:wiki.mockup.org/UserSpaceVFS+mockup+wiki+VFS&hl=es&lr=&client=firefox&strip=1 )

Re: reiser4 plugins

[Horst von Brand]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2511 bytes --]

Markus   Törnqvist <mjt@nysv.org> wrote:
> On Thu, Jun 30, 2005 at 07:18:47PM +0400, Nikita Danilov wrote:
> >Sorry, I don't see your point. Again: if you think that user level
> >developers are unlikely to agree to the common framework, what
> >difference it makes whether this framework is defined at the kernel or
> >library boundary? Applications would have to be changed to conform to
> >the common API either way.

> I see it as a heavier incentive to do it by a framework that's in
> the kernel.

API is API, if in-kernel, in-X-lib, or in-userland-VFS-lib is completely
irrelevant.

> >If you can force application developers to conform to the LSB why you
> >cannot do the same with the library level interface?

> If I want to access metadata with bash, do I patch bash to support
> both Gnome's and KDE's solutions? Was there one of XFCE too?
> And FooBarXyzzyWM that'll want to do it's own VFS next year?

It's your only option... or get them together and define a common
framework.

But then again, what would I want to do with metadata in bash? If needed,
it is probably much easier to write tools to extract whatever is needed, no
reason to futz around with the shell. That simple idea was the single most
important advance Unix introduced: The shell is a /simple/ program, it
doesn't do word processing and coffee brewing for you. That is handled by
other programs, one for each task.

> I'd also guess that the upstream guys would much rather have
> patches for their progs that conform to the kernel than some
> obscure neighbor userspace system.

Or just keep only their own obscure userspace system, no need to have to
mess with our own format and a kernel system.

> Sure looks like having this in the kernel makes it easiest; there's
> just one common denominator to patch for.

Again: API is API. If in kernel or in a standard library makes no
difference. Libary is /much/ easier to develop, and hugely more
flexible. It is for a reason that printf(3) and qsort(3) are /not/
in-kernel.

> This doesn't even invalidate the userland VFSs of the other guys,
> they're still needed for systems whose kernels don't have a
> metadata facility.

So the metadata facility in kernel won't be used, for portability's sake.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Hubert Chan]

On Thu, 30 Jun 2005 15:52:25 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:

>> This doesn't even invalidate the userland VFSs of the other guys,
>> they're still needed for systems whose kernels don't have a metadata
>> facility.

> So the metadata facility in kernel won't be used, for portability's
> sake.

Oh gee.  Every operating system does threads differently.  Mozilla has
an abstraction layer called nspr that allows them to handle threads
portably.  glib/gtk has their own threads abstraction.  On Windows, nspr
will use the Windows method for handling threads.  On Linux, it will use
the Linux way.  On systems that don't support threads, it can usually
emulate it using timers.

It's the exact same thing with the userspace VFS.  If GNOME needs to
handle extended attributes, it can use one mechanism under one operating
system, or emulate it using some ugly hack on operating systems that
don't support extended attributes.

Isn't that the whole point of having a VFS?

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Stefan Smietanowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hubert Chan wrote:
> On Thu, 30 Jun 2005 15:52:25 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:
> 
> 
>>>This doesn't even invalidate the userland VFSs of the other guys,
>>>they're still needed for systems whose kernels don't have a metadata
>>>facility.
> 
> 
>>So the metadata facility in kernel won't be used, for portability's
>>sake.
> 
> 
> Oh gee.  Every operating system does threads differently.  Mozilla has
> an abstraction layer called nspr that allows them to handle threads
> portably.  glib/gtk has their own threads abstraction.  On Windows, nspr
> will use the Windows method for handling threads.  On Linux, it will use
> the Linux way.  On systems that don't support threads, it can usually
> emulate it using timers.
> 
> It's the exact same thing with the userspace VFS.  If GNOME needs to
> handle extended attributes, it can use one mechanism under one operating
> system, or emulate it using some ugly hack on operating systems that
> don't support extended attributes.
> 
> Isn't that the whole point of having a VFS?
> 

So basically if I write a program that works in both Gnome and KDE
I should (according to your description) implement my own VFS that
will use the Gnome or KDE VFS that will then use the OS VFS.

Is it only me finding that a little silly?

I mean, if I am to have the same functionality under neither Gnome
nor VFS and they don't support something I need I _NEED_ a vfs so
that my program is so totally independent on anything at all.

My program calling My VFS which calls KDE/Gnome's VFS which calls the OS
VFS will be slowe than just calling the VFS immidiately - I do hope you
can see that.

// Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)

iD8DBQFC0GUxBrn2kJu9P78RApViAJ4q6BrVh0H19S4pN+Zc0bqh7zk2sgCeLRVK
8b+qlg2BHjwxg8HnVANQ5XA=
=2uNQ
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Jaroslav Soltys]

> So basically if I write a program that works in both Gnome and KDE
> I should (according to your description) implement my own VFS that
> will use the Gnome or KDE VFS that will then use the OS VFS.
> 
> Is it only me finding that a little silly?

Maybe. Advantages of kde/gnome/other userland vfs ? Authentication
when using SMB shares, transparent access to fish/webdav/ftp/... but
only for applications that use these libraries. Maybe patched
automount together with lufs plus some GUI for acquiring user
credentials from kde/gnome user could do the same, but kde/gnome vfs
is portable to other unices.

Oh, by the way... mount something with smbmount and turn of that
computer you mounted the share from. You must switch to root to be
able to do umount -lf. But yes, I agree with you, it is silly and I
would also like to have one good solution than two average ones.

> I mean, if I am to have the same functionality under neither Gnome
> nor VFS and they don't support something I need I _NEED_ a vfs so
> that my program is so totally independent on anything at all.

right, wouldn't it be nice to 'cd /mnt/webdav/' or
'/mnt/ssh/infernal.machine/' in bash ? :)

> My program calling My VFS which calls KDE/Gnome's VFS which calls the OS
> VFS will be slowe than just calling the VFS immidiately - I do hope you
> can see that.

Some of kde/gnome vfs are not even filesystems at all, this is the
main advantage of userland vfs. Imagine that it is possible for user
to add new FS to kde/gnome, is it possible for him to add it to kernel
without root access ? compile new module and load it ? (Un)fortunately
not.

jaro

Re: reiser4 plugins

[Hubert Chan]

On Sun, 10 Jul 2005 02:00:49 +0200, Stefan Smietanowski <stesmi@stesmi.com> said:

> So basically if I write a program that works in both Gnome and KDE I
> should (according to your description) implement my own VFS that will
> use the Gnome or KDE VFS that will then use the OS VFS.

Either that, or use a whole lot of #ifdefs, or hope that GNOME and KDE
agree on a common VFS, or don't use their VFS and just use the basic OS
calls and lose the functionality (and portability) of the VFSes, or pick
one VFS and hope that the other users can adapt.

GNOME and KDE already have their own VFS.  I am not asking for anything
new.  If you don't like the idea of a VFS at that level, take it up with
the GNOME and KDE people.

> Is it only me finding that a little silly?

> I mean, if I am to have the same functionality under neither Gnome nor
> VFS and they don't support something I need I _NEED_ a vfs so that my
> program is so totally independent on anything at all.

> My program calling My VFS which calls KDE/Gnome's VFS which calls the
> OS VFS will be slowe than just calling the VFS immidiately - I do hope
> you can see that.

Of course.  It's probably slower than arranging the bits on the hard
drive directly, and hand-coding everything in assembly.  But there's
always a performance price to pay for maintaining the programmer's
sanity.  There's always a price to pay when writing cross-platform
stuff.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Masover]

Horst von Brand wrote:
> Markus Törnqvist <mjt@nysv.org> wrote:
> 
> [...]
> 
> 
>>Note that MacOS has the monopoly on what they ship, Linux has a
>>motherload of file managers and window systems and all.
> 
> 
> Yep. Part of what is nice about it, too ;-)
> 
> 
>>What pisses me off is the fact that Gnome and friends implement
>>their own incompatible-with-others VFS's and automounters and
>>stuff.
> 
> 
> Then get them to agree on a common framework! They are trying hard to get
> other parts of the GUI work well together, so this isn't far off in
> wishfull thinking land.

Unfortunately, this leaves bash out in the cold, as usual.  Kernel-based 
automounter works with Bash.  Why can't GNOME/KDE use the kernel's one 
as at least one backend, even if they support others?

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On Wed, 29 Jun 2005 16:58:20 +0300, Markus =?UNKNOWN?Q?T=F6rnqvist?= said:
> What pisses me off is the fact that Gnome and friends implement
> their own incompatible-with-others VFS's and automounters and
> stuff.

The fact that things like Gnome, which are basically consumers of their own
dogfood, have incompatible versions says very loudly that there's no consensus
on the semantics....

> Surely supporting this in the kernel and extending the LSB
> to require this is the best step to take without infringing
> anyone's freedom as such.

First we need to decide *if* it's to be supported, then *what* to support....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Weinehall]

On Wed, Jun 29, 2005 at 04:58:20PM +0300, Markus   Törnqvist wrote:
> On Wed, Jun 29, 2005 at 09:50:27AM -0400, Douglas McNaught wrote:
> >
> >I'll just note that the "applications bundled as directories" stuff on
> >MacOS/NextStep is done completely in userspace--as far as the kernel
> >is concerned, "Mail.app" is a regular directory.  The file manager
> >handles recognition and invocation of application bundles (and there
> >is an 'open' shell command that does the same thing).
> 
> Note that MacOS has the monopoly on what they ship, Linux has a
> motherload of file managers and window systems and all.
> 
> What pisses me off is the fact that Gnome and friends implement
> their own incompatible-with-others VFS's and automounters and
> stuff.
> 
> Surely supporting this in the kernel and extending the LSB
> to require this is the best step to take without infringing
> anyone's freedom as such.
> 
> *still pissed off about having to hassle an automatic mount*

GNOME and KDE run on operating systems that run other kernels than
Linux, hence they have to implement their own userland VFS anyway.
Adding this to the Linux kernel won't help them one bit, unless
we can magically convince Sun to add it to Solaris, all different
BSD:s to add it to their kernels, etc.  Not going to happen.
An effort to get GNOME and KDE to unify their VFS:s would be
far more benificial, and would if successful probably lead to
other desktop environments (if there are any other DE:s with their
own VFS:s, dunno about that) following their lead.

FreeDesktop is doing a lot of work to make GNOME, KDE, and other
DE:s interoperate as much as possible.  Support their initiative
instead of trying to get a monstrosity (albeit a very cool one,
conceptually) into the kernel.  Sure, it could be made to work,
but not without dropping our Unixness.  And if we do, we should
start by looking at Plan 9 =)


Regards: David
-- 
 /) David Weinehall <tao@acc.umu.se> /) Northern lights wander      (\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\)  http://www.acc.umu.se/~tao/    (/   Full colour fire           (/

Re: reiser4 plugins

[Chet Hosey]

David Weinehall wrote:

>GNOME and KDE run on operating systems that run other kernels than
>Linux, hence they have to implement their own userland VFS anyway.
>Adding this to the Linux kernel won't help them one bit, unless
>we can magically convince Sun to add it to Solaris, all different
>BSD:s to add it to their kernels, etc.  Not going to happen.
>An effort to get GNOME and KDE to unify their VFS:s would be
>far more benificial, and would if successful probably lead to
>other desktop environments (if there are any other DE:s with their
>own VFS:s, dunno about that) following their lead.
>
>FreeDesktop is doing a lot of work to make GNOME, KDE, and other
>DE:s interoperate as much as possible.  Support their initiative
>instead of trying to get a monstrosity (albeit a very cool one,
>conceptually) into the kernel.  Sure, it could be made to work,
>but not without dropping our Unixness.  And if we do, we should
>start by looking at Plan 9 =)
>
>
>Regards: David
>  
>
The point of such ventures is that by placing features at a lower level
you get to keep the advantages of UNIX in the first place -- namely,
that many small tools can do neat things with most objects. By placing
everything in a largish userspace library instead of at a system level
(kernel, libc, etc.) you're essentially saying that, for instance, vi
would have to be rewritten in order to interact with objects presented
by the VFS. So would bash, fmt, sort, less, aspell, or anything else
that can open a file. You'd end up with a situation in which you see
objects via the VFS browser (file manager) that no longer exist when you
want to drop to a shell to use common UNIX utilities and find that the
object doesn't actually exist to the OS itself.

This sounds like Joel Spolsky's law of leaky abstractions, and the fact
that most operating systems lack a useful facility (which is why GNOME
and KDE roll their own VFS) sounds like a poor excuse for keeping useful
features out of the kernel.

I'm *not* arguing for putting anything specific into the kernel. I *am*
arguing that an inconsistent presentation in which some apps see VFS
objects and others don't makes for a less-than-ideal UI.

-- chet

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]

On Wed, Jun 29, 2005 at 10:56:36PM +0200, David Weinehall wrote:
>
>FreeDesktop is doing a lot of work to make GNOME, KDE, and other
>DE:s interoperate as much as possible.  Support their initiative

What about WindowMaker with bash?

>instead of trying to get a monstrosity (albeit a very cool one,
>conceptually) into the kernel.  Sure, it could be made to work,
>but not without dropping our Unixness.  And if we do, we should
>start by looking at Plan 9 =)

What's wrong with "dropping our Unixness" if it means taking
an extra step toward Plan 9?

Why is this a bad idea?

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Linux and Plan-9ness

[Al Boldi]

Markus   Törnqvist wrote: {
On Wed, Jun 29, 2005 at 10:56:36PM +0200, David Weinehall wrote:
>instead of trying to get a monstrosity (albeit a very cool one,
>conceptually) into the kernel.  Sure, it could be made to work, but not 
>without dropping our Unixness.  And if we do, we should start by 
>looking at Plan 9 =)

What's wrong with "dropping our Unixness" if it means taking an extra step
toward Plan 9?

Why is this a bad idea?
}

Please explain!

Re: Linux and Plan-9ness

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

On Thu, Jun 30, 2005 at 03:45:48PM +0300, Al Boldi wrote:
>Markus   Törnqvist wrote: {
>
>What's wrong with "dropping our Unixness" if it means taking an extra step
>toward Plan 9?
>
>Why is this a bad idea?
>}
>
>Please explain!

You mean you want me to explain this or someone else to explain
why this is a bad idea?-)

My take on this is that Linux should be allowed to move toward
the directions outlined here on the list a million times, and
"dropping our Unixness" may be a bit harsh, it should probably
be more along the lines of "not being so uptightly unixy" :)

-- 
mjt



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Linux and Plan-9ness

[studdugie]

If you dropped the "Unixness" you would have to rename. Maybe Linine,
or  Linsoft, or ....

Think of all the chaos that would cause. You don't want chaos do you?
I mean really, do u want to force all the
\w*(?ilinux)\w*(\.(com|org|net))? sites and publications to rename
too? Unconscionable!

On 6/30/05, Al Boldi <a1426z@gawab.com> wrote:
> Markus   Törnqvist wrote: {
> On Wed, Jun 29, 2005 at 10:56:36PM +0200, David Weinehall wrote:
> >instead of trying to get a monstrosity (albeit a very cool one,
> >conceptually) into the kernel.  Sure, it could be made to work, but not
> >without dropping our Unixness.  And if we do, we should start by
> >looking at Plan 9 =)
> 
> What's wrong with "dropping our Unixness" if it means taking an extra step
> toward Plan 9?
> 
> Why is this a bad idea?
> }
> 
> Please explain!
> 
>

Re: reiser4 plugins

[Eric Van Hensbergen]

On 6/30/05, Markus   Törnqvist <mjt@nysv.org> wrote:
> 
> >instead of trying to get a monstrosity (albeit a very cool one,
> >conceptually) into the kernel.  Sure, it could be made to work,
> >but not without dropping our Unixness.  And if we do, we should
> >start by looking at Plan 9 =)
> 
> What's wrong with "dropping our Unixness" if it means taking
> an extra step toward Plan 9?
> 
> Why is this a bad idea?
> 

It's not.  For those who don't already know about it: check out the
v9fs project (http://v9fs.sf.net) - we're taking steps of moving the
Linux kernel towards Plan 9 while trying to preserve Unix semantics
where they make sense.

     -eric

Re: reiser4 plugins

[David Masover]

David Weinehall wrote:
> On Wed, Jun 29, 2005 at 04:58:20PM +0300, Markus   Törnqvist wrote:
> 
>>On Wed, Jun 29, 2005 at 09:50:27AM -0400, Douglas McNaught wrote:
>>
>>>I'll just note that the "applications bundled as directories" stuff on
>>>MacOS/NextStep is done completely in userspace--as far as the kernel
>>>is concerned, "Mail.app" is a regular directory.  The file manager
>>>handles recognition and invocation of application bundles (and there
>>>is an 'open' shell command that does the same thing).
>>
>>Note that MacOS has the monopoly on what they ship, Linux has a
>>motherload of file managers and window systems and all.
>>
>>What pisses me off is the fact that Gnome and friends implement
>>their own incompatible-with-others VFS's and automounters and
>>stuff.
>>
>>Surely supporting this in the kernel and extending the LSB
>>to require this is the best step to take without infringing
>>anyone's freedom as such.
>>
>>*still pissed off about having to hassle an automatic mount*
> 
> 
> GNOME and KDE run on operating systems that run other kernels than
> Linux, hence they have to implement their own userland VFS anyway.
> Adding this to the Linux kernel won't help them one bit, unless
> we can magically convince Sun to add it to Solaris, all different
> BSD:s to add it to their kernels, etc.  Not going to happen.
> An effort to get GNOME and KDE to unify their VFS:s would be
> far more benificial,

Than what?  Creating a unified VFS which I can access from Bash, and 
which obsoletes both GNOME and KDE's VFSes except in their presentation?

> FreeDesktop is doing a lot of work to make GNOME, KDE, and other
> DE:s interoperate as much as possible.  Support their initiative
> instead of trying to get a monstrosity (albeit a very cool one,
> conceptually) into the kernel.  Sure, it could be made to work,
> but not without dropping our Unixness.

(I'm talking about the metafs (/meta) idea, which isn't nearly as much a 
monstrocity, and doesn't kill our unixness, it enhances it.)

Re: reiser4 plugins

[David Weinehall]

On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote:
> David Weinehall wrote:
>
> >GNOME and KDE run on operating systems that run other kernels than
> >Linux, hence they have to implement their own userland VFS anyway.
> >Adding this to the Linux kernel won't help them one bit, unless
> >we can magically convince Sun to add it to Solaris, all different
> >BSD:s to add it to their kernels, etc.  Not going to happen.
> >An effort to get GNOME and KDE to unify their VFS:s would be
> >far more benificial,
> 
> Than what?  Creating a unified VFS which I can access from Bash, and 
> which obsoletes both GNOME and KDE's VFSes except in their presentation?

On one of the platforms that they support, yes.  But only for kernels
newer than 2.6.yy...  So they'd still have to have their own VFS for
2.4.xx, 2.6.xx (xx < yy), FreeBSD, OpenBSD, Solaris, etc...

> >FreeDesktop is doing a lot of work to make GNOME, KDE, and other
> >DE:s interoperate as much as possible.  Support their initiative
> >instead of trying to get a monstrosity (albeit a very cool one,
> >conceptually) into the kernel.  Sure, it could be made to work,
> >but not without dropping our Unixness.
> 
> (I'm talking about the metafs (/meta) idea, which isn't nearly as much a 
> monstrocity, and doesn't kill our unixness, it enhances it.)

Which would neither need VFS changes nor be dependent on Reiser4 in
any way, so I don't see why this thread lives on.  Just get down to
business and implement this metafs =)


Regards: David Weinehall
-- 
 /) David Weinehall <tao@acc.umu.se> /) Northern lights wander      (\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\)  http://www.acc.umu.se/~tao/    (/   Full colour fire           (/

Re: reiser4 plugins

[David Masover]

David Weinehall wrote:
> On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote:
> 
>>David Weinehall wrote:
>>
>>
>>>GNOME and KDE run on operating systems that run other kernels than
>>>Linux, hence they have to implement their own userland VFS anyway.
>>>Adding this to the Linux kernel won't help them one bit, unless
>>>we can magically convince Sun to add it to Solaris, all different
>>>BSD:s to add it to their kernels, etc.  Not going to happen.
>>>An effort to get GNOME and KDE to unify their VFS:s would be
>>>far more benificial,
>>
>>Than what?  Creating a unified VFS which I can access from Bash, and 
>>which obsoletes both GNOME and KDE's VFSes except in their presentation?
> 
> 
> On one of the platforms that they support, yes.  But only for kernels
> newer than 2.6.yy...  So they'd still have to have their own VFS for
> 2.4.xx, 2.6.xx (xx < yy), FreeBSD, OpenBSD, Solaris, etc...

Right.  But, /proc started somewhere, didn't it?

I have the feeling that other systems will duplicate it if it's good.

Even if they don't, it would be more beneficial to me and probably most 
Linux users to have metafs supported in both GNOME and KDE, even if they 
still need an emulation layer to support other systems.

I agree that the emulation layer should be common.

>>>FreeDesktop is doing a lot of work to make GNOME, KDE, and other
>>>DE:s interoperate as much as possible.  Support their initiative
>>>instead of trying to get a monstrosity (albeit a very cool one,
>>>conceptually) into the kernel.  Sure, it could be made to work,
>>>but not without dropping our Unixness.
>>
>>(I'm talking about the metafs (/meta) idea, which isn't nearly as much a 
>>monstrocity, and doesn't kill our unixness, it enhances it.)
> 
> 
> Which would neither need VFS changes nor be dependent on Reiser4 in
> any way, so I don't see why this thread lives on.  Just get down to
> business and implement this metafs =)

I am trying, at least getting the conceptual glitches ironed out.

But, it does need GNOME/KDE VFS changes in order to be sane -- 
otherwise, GNOME/KDE will duplicate it, creating even more of a mess 
than we already have.

I'm not sure why the thread still lives, but now that I've typed this, I 
may as well send it...

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:
> David Weinehall wrote:
> > On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote:
> >>David Weinehall wrote:

> >>>GNOME and KDE run on operating systems that run other kernels than
> >>>Linux, hence they have to implement their own userland VFS anyway.
> >>>Adding this to the Linux kernel won't help them one bit, unless
> >>>we can magically convince Sun to add it to Solaris, all different
> >>>BSD:s to add it to their kernels, etc.  Not going to happen.
> >>>An effort to get GNOME and KDE to unify their VFS:s would be
> >>>far more benificial,

> >> Than what?  Creating a unified VFS which I can access from Bash,
> >> and which obsoletes both GNOME and KDE's VFSes except in their
> >> presentation?

> > On one of the platforms that they support, yes.  But only for kernels
> > newer than 2.6.yy...  So they'd still have to have their own VFS for
> > 2.4.xx, 2.6.xx (xx < yy), FreeBSD, OpenBSD, Solaris, etc...

> Right.  But, /proc started somewhere, didn't it?

Sun.

> I have the feeling that other systems will duplicate it if it's good.

Linux copied here.

> Even if they don't, it would be more beneficial to me

How, exactly?

Besides, /your/ convenience isn't the only thing that matters...

>                                                       and probably
> most Linux users

"Most Linux users" don't use experimental filesystems at all...

>                  to have metafs supported in both GNOME and KDE, even
> if they still need an emulation layer to support other systems.

So Gnome and KDE get larger (and thus slower) for everybody. Besides, Gnome
and KDE will have to agree on the formats involved first, which is /exactly/
what is supposed to be impossible unless this stuff is implemented in the
kernel...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Hans Reiser]

Horst von Brand wrote:

>
>>Right.  But, /proc started somewhere, didn't it?
>>    
>>
>
>Sun.
>  
>
No, plan 9.

It seems to me that what you are all arguing about is whether it is
better to be at the front of the herd, or in the middle, or in the
back.  I don't think this one can be resolved, except that I would
suggest that Linux, by virtue of the kernel configuration functionality,
has space enough for persons of all three inclinations, if we can just
leave each other be a bit.

Hans

Re: reiser4 plugins

[Christoph Hellwig]

On Sun, Jul 03, 2005 at 08:42:24PM -0700, Hans Reiser wrote:
> >>Right.  But, /proc started somewhere, didn't it?
> >>    
> >>
> >
> >Sun.
> >  
> >
> No, plan 9.

Almost on the right track, it was v8, two steps before plan9.  But that's
just the process-part of procfs, not the big mess we have now - that part
is pretty much linux-only.

Re: reiser4 plugins

[David Masover]

Horst von Brand wrote:
> David Masover <ninja@slaphack.com> wrote:
> 
>>David Weinehall wrote:
>>
>>>On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote:
>>>
>>>>David Weinehall wrote:
> 
> 
>>>>>GNOME and KDE run on operating systems that run other kernels than
>>>>>Linux, hence they have to implement their own userland VFS anyway.
>>>>>Adding this to the Linux kernel won't help them one bit, unless
>>>>>we can magically convince Sun to add it to Solaris, all different
>>>>>BSD:s to add it to their kernels, etc.  Not going to happen.
>>>>>An effort to get GNOME and KDE to unify their VFS:s would be
>>>>>far more benificial,
> 
> 
>>>>Than what?  Creating a unified VFS which I can access from Bash,
>>>>and which obsoletes both GNOME and KDE's VFSes except in their
>>>>presentation?
> 
> 
>>>On one of the platforms that they support, yes.  But only for kernels
>>>newer than 2.6.yy...  So they'd still have to have their own VFS for
>>>2.4.xx, 2.6.xx (xx < yy), FreeBSD, OpenBSD, Solaris, etc...
> 
> 
>>Right.  But, /proc started somewhere, didn't it?
> 
> 
> Sun.
> 
> 
>>I have the feeling that other systems will duplicate it if it's good.
> 
> 
> Linux copied here.

So what?

>>Even if they don't, it would be more beneficial to me
> 
> 
> How, exactly?

Go back and read.

> Besides, /your/ convenience isn't the only thing that matters...

Nor yours.  Just because you can't get your mind around a concept 
doesn't mean that it's a bad concept, and that no one else can use it.

>>                                                      and probably
>>most Linux users
> 
> 
> "Most Linux users" don't use experimental filesystems at all...

Actually, they do -- ext3 was experimental once.  ReiserFS was very 
experimental once.

Please stop bashing it just because it's new/experimental.

>>                 to have metafs supported in both GNOME and KDE, even
>>if they still need an emulation layer to support other systems.
> 
> 
> So Gnome and KDE get larger (and thus slower) for everybody.

Smaller (and thus faster) on supported systems, otherwise exactly the 
same, but maybe a little more modular, which is good.

> Besides, Gnome
> and KDE will have to agree on the formats involved first, which is /exactly/
> what is supposed to be impossible unless this stuff is implemented in the
> kernel...

I never said that, but for one thing, whether they do or not, it's nice 
if my shell and my editor and all the other things that I use don't have 
to agree on anything to manipulate the formats involved.

This is not just about GNOME/KDE.  It is about GNOME/KDE not developing 
an additional layer, that you wouldn't like anyway, that cannot be 
accessed from anything except GNOME/KDE.

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:
> Horst von Brand wrote:
> > David Masover <ninja@slaphack.com> wrote:
> >>David Weinehall wrote:
> >>>On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote:
> >>>>David Weinehall wrote:

[...]

> >>Even if they don't, it would be more beneficial to me

> > How, exactly?

> Go back and read.

I have. And have seen /no/ benefit to you. Except, of course, the benefit
accrued from some magic in Linus' kernel, by which all format differences
go in a puff of smoke if they are implemented inside it, and furthermore
all userland gets rebuilt to use the kernel's way overnight.

Sorry, I'm quite sceptical on this count. Sure, for complete outsiders it
may look that way: Something gets implemented in the kernel, soon many
applications are using it, everything is peachy. Except that before there
have been /long/ (even bitter) discussions on if/how to do it, what the
kernel should do (if anything) and what the application's responsibility
should be. Many times it has ended with Linus slamming on the table and
saying "no way", or "we do it /this/ way". Then, /when/ this is clear, it
gets into the kernel and applications. And you only have ever seen the last
phase... which is smooth and fast.

Here, the "how should it be done, if at all" discussion has barely started.
And AFAICS there are plenty of ways of getting 95% of the advantages
/without/ touching the kernel, so that should be the path to follow. Nobody
has shown any real evidence at all for this not being so.

> > Besides, /your/ convenience isn't the only thing that matters...

> Nor yours.

Sure enough.

>            Just because you can't get your mind around a concept
> doesn't mean that it's a bad concept, and that no one else can use it.

I /do/ get my mind around the concept, it /is/ sometimes useful. It /can/
be done without the kernel, and so /should/ be done that way. That is all.

> >>                                                      and probably
> >>most Linux users

> > "Most Linux users" don't use experimental filesystems at all...

> Actually, they do -- ext3 was experimental once.

Right. And ext2, and ext, and xiafs, and ... So? When they were
experimental, only a handful of utter fools commited their data to them.

>                                                   ReiserFS was very
> experimental once.

See above.

> Please stop bashing it just because it's new/experimental.

Sorry?

> >>                 to have metafs supported in both GNOME and KDE, even
> >>if they still need an emulation layer to support other systems.
> > So Gnome and KDE get larger (and thus slower) for everybody.

> Smaller (and thus faster) on supported systems,

Sorry, but just because some $DISTRO gives ReiserFS4 as an /option/ doesn't
mean they will have everything duplicated as "For ReiserFS4" and "For other
filesystems". My assertion stands, until there are ReiserFS4-only
distributions.

>                                                 otherwise exactly the
> same, but maybe a little more modular, which is good.

Right, having to cope with different ways of representing metadata could
induce better code organization. But to get there isn't painless either...

> > Besides, Gnome
> > and KDE will have to agree on the formats involved first, which is /exactly/
> > what is supposed to be impossible unless this stuff is implemented in the
> > kernel...

> I never said that,

You (and others) have told us time and again that each of them have their
own incompatible ways of handling metadata, and that only if this was
handled in the kernel they would make use of a common way of managing it...

>                    but for one thing, whether they do or not, it's
> nice if my shell and my editor and all the other things that I use
> don't have to agree on anything to manipulate the formats involved.

Please, read what you just said. Everybody (kernel somewhat included) will
have to agree on the format of the metadata.

> This is not just about GNOME/KDE.  It is about GNOME/KDE not developing
> an additional layer, that you wouldn't like anyway,

How do you know what I would or would not like?

>                                                     that cannot be
> accessed from anything except GNOME/KDE.

So Gnome and KDE agree on some secret formats that only they can handle?
Behind their user's backs? As open source? And happily shoot their feet by
not giving users much wanted access to said data in decent ways?

I see it more on the lines of libmetadata.so (or such), which is used by
Gnome, KDE, and whatever else wishes to do so. Even your custom-tailored
shell or cat(1). Or just some convention that metadata on ~/my/funky/file
resides in ~/.metadata/.my/.funky/.file/metadata. Hey, you could (almost)
do that today, by wrappers (perhaps even aliases, or shell functions)
around the relevant commands! No kernel at all involved! Not even a
miserable library!!
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[David Masover]

Horst von Brand wrote:
> David Masover <ninja@slaphack.com> wrote:
> 
>>Horst von Brand wrote:
>>
>>>David Masover <ninja@slaphack.com> wrote:
>>>
>>>>David Weinehall wrote:
>>>>
>>>>>On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote:
>>>>>
>>>>>>David Weinehall wrote:
> 
> 
> [...]
> 
> 
>>>>Even if they don't, it would be more beneficial to me
> 
> 
>>>How, exactly?
> 
> 
>>Go back and read.
> 
> 
> I have. And have seen /no/ benefit to you. Except, of course, the benefit
> accrued from some magic in Linus' kernel, by which all format differences
> go in a puff of smoke if they are implemented inside it, and furthermore
> all userland gets rebuilt to use the kernel's way overnight.

Let's say cryptocompress gets implemented.  Not all of userland
rewritten, not even any of userland rewritten, just a cryptocompress
plugin for the kernel.  And instead of having to learn a new tool, I can
just browse around in /meta.

Yes, all of userland gets rebuilt, and the world is better for it.  But
*incrementally*.  Not overnight.

> Here, the "how should it be done, if at all" discussion has barely started.
> And AFAICS there are plenty of ways of getting 95% of the advantages
> /without/ touching the kernel,

Which advantages can we get without touching the kernel?

And why does it make sense to do them outside the kernel?  "Because we
can" is a little HURD-ish...

> so that should be the path to follow. Nobody
> has shown any real evidence at all for this not being so.

Go back and read again.

>>           Just because you can't get your mind around a concept
>>doesn't mean that it's a bad concept, and that no one else can use it.
> 
> 
> I /do/ get my mind around the concept, it /is/ sometimes useful. It /can/
> be done without the kernel,

No, it can't.  Unless you patch *all* of userland.

> and so /should/ be done that way. That is all.

I'll refer you to my HURD comment:

Just about anything done currently in the kernel can be done in
userland.  Witness HURD.

Doesn't mean it should.  Witness HURD.

>>>>                                                     and probably
>>>>most Linux users
> 
> 
>>>"Most Linux users" don't use experimental filesystems at all...
> 
> 
>>Actually, they do -- ext3 was experimental once.
> 
> 
> Right. And ext2, and ext, and xiafs, and ... So? When they were
> experimental, only a handful of utter fools commited their data to them.

But, they had to start somewhere.  You seem hell-bent on keeping
anything "unproven" out of the kernel and away from users.  How do you
suggest it "prove" itself, then?

>>>>                to have metafs supported in both GNOME and KDE, even
>>>>if they still need an emulation layer to support other systems.
>>>
>>>So Gnome and KDE get larger (and thus slower) for everybody.
> 
> 
>>Smaller (and thus faster) on supported systems,
> 
> 
> Sorry, but just because some $DISTRO gives ReiserFS4 as an /option/ doesn't
> mean they will have everything duplicated as "For ReiserFS4" and "For other
> filesystems". My assertion stands, until there are ReiserFS4-only
> distributions.

Why not?  The "For Reiser4" version is just a thin layer on top of metafs.

And after all, metafs can be implemented for other filesystems.

And, as someone else pointed out, dealing with different systems doing
things in different ways is what portability is all about.

>>>Besides, Gnome
>>>and KDE will have to agree on the formats involved first, which is /exactly/
>>>what is supposed to be impossible unless this stuff is implemented in the
>>>kernel...
> 
> 
>>I never said that,
> 
> 
> You (and others)

Others, then.

> have told us time and again that each of them have their
> own incompatible ways of handling metadata, and that only if this was
> handled in the kernel they would make use of a common way of managing it...

All I ever said was that it would be easier for them to get along, and
certainly *much* easier to use existing tools (the commandline) to poke
inside the metadata, whether it's compatible or not.

Ultimately, maybe it leads to cooperation, maybe not.  But there are
immediate benefits even if it doesn't lead to cooperation.

>>                   but for one thing, whether they do or not, it's
>>nice if my shell and my editor and all the other things that I use
>>don't have to agree on anything to manipulate the formats involved.
> 
> 
> Please, read what you just said. Everybody (kernel somewhat included) will
> have to agree on the format of the metadata.
> 
> 
>>This is not just about GNOME/KDE.  It is about GNOME/KDE not developing
>>an additional layer, that you wouldn't like anyway,
> 
> 
> How do you know what I would or would not like?

Maybe I was generalizing "you".  Others don't like Reiser4 as it is
because it's got "layering violations".  GNOME/KDE developing their own
VFS on top of the kernel's for no good reason is a layering violation.

Am I misunderstanding?

>>                                                    that cannot be
>>accessed from anything except GNOME/KDE.
> 
> 
> So Gnome and KDE agree on some secret formats that only they can handle?
> Behind their user's backs? As open source? And happily shoot their feet by
> not giving users much wanted access to said data in decent ways?

Not "secret", but not easy to get at, unless everything links against
libmetadata.so or reimplements it.

> I see it more on the lines of libmetadata.so (or such), which is used by
> Gnome, KDE, and whatever else wishes to do so. Even your custom-tailored
> shell or cat(1). Or just some convention that metadata on ~/my/funky/file
> resides in ~/.metadata/.my/.funky/.file/metadata. Hey, you could (almost)
> do that today, by wrappers (perhaps even aliases, or shell functions)
> around the relevant commands! No kernel at all involved! Not even a
> miserable library!!

Maybe.  Then again, maybe not.

GNOME, at least, likes XML-based systems.  XML is much, much harder for
me to deal with from the shell, and certainly doesn't lend itself to as
easy scripting.

Now, we could tell the GNOME people to make sure to use plain old text
files / images everywhere, and make sure everything works for the shell
-- and then, on filesystems where small files don't pack so well, you
end up taking 10x more space for your metadata, and it's slower for
GNOME to manage.

Or, we implement /meta.  Now I can get to individual bits of metadata
from the shell, but GNOME can still use its XML format, or a db
database, or some database-like system call like the proposed
sys_reiser4 even.  It doesn't matter so much anymore how they store
metadata, but it becomes easier to access it without having to link
against some magic library.

Re: reiser4 plugins

[Jeremy Maitin-Shepard]

David Masover <ninja@slaphack.com> writes:

[snip]

>> I have. And have seen /no/ benefit to you. Except, of course, the benefit
>> accrued from some magic in Linus' kernel, by which all format differences
>> go in a puff of smoke if they are implemented inside it, and furthermore
>> all userland gets rebuilt to use the kernel's way overnight.

> Let's say cryptocompress gets implemented.  Not all of userland
> rewritten, not even any of userland rewritten, just a cryptocompress
> plugin for the kernel.  And instead of having to learn a new tool, I can
> just browse around in /meta.

What is the relationship between file-as-dir or special meta-data and
transparent encryption+compression?  I do not see why file-as-dir would
require such a special interface.

[snip]

-- 
Jeremy Maitin-Shepard

Re: reiser4 plugins

[David Masover]

Jeremy Maitin-Shepard wrote:
> David Masover <ninja@slaphack.com> writes:
> 
> [snip]
> 
> 
>>>I have. And have seen /no/ benefit to you. Except, of course, the benefit
>>>accrued from some magic in Linus' kernel, by which all format differences
>>>go in a puff of smoke if they are implemented inside it, and furthermore
>>>all userland gets rebuilt to use the kernel's way overnight.
> 
> 
>>Let's say cryptocompress gets implemented.  Not all of userland
>>rewritten, not even any of userland rewritten, just a cryptocompress
>>plugin for the kernel.  And instead of having to learn a new tool, I can
>>just browse around in /meta.
> 
> 
> What is the relationship between file-as-dir or special meta-data and
> transparent encryption+compression?  I do not see why file-as-dir would
> require such a special interface.

I'm ignoring file-as-dir until/unless someone figures out a sane way of
doing it.

Under Reiser4, think of all files/directories as objects.  Meta-data, as
accessed through file-as-dir or through /meta, is just a collection of
methods to that file/directory.  Some methods are just accessor
(setter/getter) methods, like foo.mp3/artist -- they are effectively
attributes, although there may be a little more logic going on there
(foo.mp3/artist probably is connected to foo.mp3's id3 tag).  Some
methods actually do something, like secret.txt/compression.

The reason I'm bringing that up is that some people seem determined to
keep the very concept out of the kernel because it's new, strange, and
doesn't seem to have any real applications.  I want it in because it has
real, fairly immediate applications, and much bigger payoffs down the
road, and I don't think it's that strange -- it's a nice, clean design.

Re: reiser4 plugins

[Jeremy Maitin-Shepard]

David Masover <ninja@slaphack.com> writes:

> Jeremy Maitin-Shepard wrote:
>> David Masover <ninja@slaphack.com> writes:
>> 
>> [snip]
>> 
>> 
>>>> I have. And have seen /no/ benefit to you. Except, of course, the benefit
>>>> accrued from some magic in Linus' kernel, by which all format differences
>>>> go in a puff of smoke if they are implemented inside it, and furthermore
>>>> all userland gets rebuilt to use the kernel's way overnight.
>> 
>> 
>>> Let's say cryptocompress gets implemented.  Not all of userland
>>> rewritten, not even any of userland rewritten, just a cryptocompress
>>> plugin for the kernel.  And instead of having to learn a new tool, I can
>>> just browse around in /meta.
>> 
>> 
>> What is the relationship between file-as-dir or special meta-data and
>> transparent encryption+compression?  I do not see why file-as-dir would
>> require such a special interface.

I mistyped this --- I meant to ask "why transparent
encryption+compression would require such a special interface."

> I'm ignoring file-as-dir until/unless someone figures out a sane way of
> doing it.

[snip]

> Some methods actually do something, like secret.txt/compression.

[snip]

Okay, so you are suggesting that file-as-dir would provide the user
interface for enabling the encryption or compression.  Alternatively,
though, an ioctl could be used to control compression and encryption.

-- 
Jeremy Maitin-Shepard

Re: reiser4 plugins

[Hans Reiser]

Jeremy Maitin-Shepard wrote:

>Okay, so you are suggesting that file-as-dir would provide the user
>interface for enabling the encryption or compression.  Alternatively,
>though, an ioctl could be used to control compression and encryption.
>
>  
>
Why is it that /proc does not use an ioctl?  Use of metafiles could
allow eliminating ioctl(), which most folks I know hate as an
interface.  Wouldn't it be cleaner if we could find out what ioctl()s
are supported by a given file using ls filename/..../ioctl?

Excerpt from the ioctl man page, which lacks a list of what features are
implemented or how to find out.

CONFORMING TO
       No single standard.  Arguments, returns, and semantics of
ioctl(2) vary
       according to the device driver in question  (the  call  is  used 
as  a
       catch-all  for  operations  that  don't cleanly fit the Unix
stream I/O
       model). See ioctl_list(2) for a list of many of the known ioctl 
calls.
       The ioctl function call appeared in Version 7 AT&T Unix.

Re: reiser4 plugins

[Christoph Hellwig]

On Tue, Jul 05, 2005 at 06:43:31PM -0400, Jeremy Maitin-Shepard wrote:
> > Let's say cryptocompress gets implemented.  Not all of userland
> > rewritten, not even any of userland rewritten, just a cryptocompress
> > plugin for the kernel.  And instead of having to learn a new tool, I can
> > just browse around in /meta.
> 
> What is the relationship between file-as-dir or special meta-data and
> transparent encryption+compression?  I do not see why file-as-dir would
> require such a special interface.

It's not related at all.  transparent encryption+compression is a nice
extension.

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

On Fri, Jul 01, 2005 at 05:54:46PM +0200, David Weinehall wrote:
>
>Which would neither need VFS changes nor be dependent on Reiser4 in
>any way, so I don't see why this thread lives on.  Just get down to
>business and implement this metafs =)

I've been gone for a while and suddenly drowning in mail...

Anyway, I don't really like the metafs thing.

To access the data, you still need to refactor userspace,
so that's not a real advantage. Doing lookups from /meta
all the time, instead of in the file-as-dir-whatever...

And the best thing to do would be to bring these "Reiser4-specific"
enhancements to every FS.

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[David Masover]

Markus Törnqvist wrote:
> On Fri, Jul 01, 2005 at 05:54:46PM +0200, David Weinehall wrote:
> 
>>Which would neither need VFS changes nor be dependent on Reiser4 in
>>any way, so I don't see why this thread lives on.  Just get down to
>>business and implement this metafs =)
> 
> 
> I've been gone for a while and suddenly drowning in mail...
> 
> Anyway, I don't really like the metafs thing.
> 
> To access the data, you still need to refactor userspace,
> so that's not a real advantage. Doing lookups from /meta
> all the time, instead of in the file-as-dir-whatever...

I don't really see the disadvantage.

Also, metafs means much less of a fight to get people to adopt the whole 
meta concept, because it can be done in a POSIX-compliant way which 
doesn't break tar.

File-as-dir is nice if you're using meta files, but it causes lots of 
unexpected weirdness.  I don't think metafs costs us much in 
performance, and with one or two shell scripts, it wouldn't cost us that 
much efficiency on the commandline.

But, I also like file-as-dir.  I think it might be time for a vote.

I vote metafs.

Or, maybe Hans needs to tell us which way we're going...

> And the best thing to do would be to bring these "Reiser4-specific"
> enhancements to every FS.

Which has nothing to do with whether it's done in "metafs" or not.

Re: reiser4 plugins

[Miquel van Smoorenburg]

In article <42CD3580.4020008@slaphack.com>,
David Masover  <ninja@slaphack.com> wrote:
>Markus Törnqvist wrote:
>> Anyway, I don't really like the metafs thing.
>> 
>> To access the data, you still need to refactor userspace,
>> so that's not a real advantage. Doing lookups from /meta
>> all the time, instead of in the file-as-dir-whatever...
>
>I don't really see the disadvantage.
>
>Also, metafs means much less of a fight to get people to adopt the whole 
>meta concept, because it can be done in a POSIX-compliant way which 
>doesn't break tar.
>
>File-as-dir is nice if you're using meta files, but it causes lots of 
>unexpected weirdness.  I don't think metafs costs us much in 
>performance, and with one or two shell scripts, it wouldn't cost us that 
>much efficiency on the commandline.

file-as-dir is an innovation. Metafs is an ugly compromise.

Mike.

Re: reiser4 plugins

[Hubert Chan]

On Wed, 29 Jun 2005 01:09:05 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:

> Hubert Chan <hubert@uhoreg.ca> wrote: [...]

>> Of course.  With file-as-dir, you can "cd /usr/games/tetris/..." and
>> mess with the game files, or you can run "/usr/games/tetris" and get
>> on with ... stacking blocks.

> And doing "tar cf /dev/tape /usr/games/tetris" gives you a nice tangle
> of undecipherable junk.

Uh, yeah.  I thought we already discussed this in the last flamewar
about merging Reiser4.  You can patch tar/cp/backup programs if you want
to go the file-as-dir route, or you can try to patch all the other
programs if you want to go the xattr route and you really want to edit
you extended attributes using $EDITOR (which I'll admit not everyone
wants to do).

>> The advantage of doing it in the kernel is that you don't need extra
>> support from the applications (or GnomeVFS or KDE).

> I don't see any advantage here... it has to be implemented, and doing
> it in-kernel for one filesystem is /much/ harder than doing it in
> userland, where it'll probably work whatever the underlying filesystem
> is.

Only filesystems that support the capability, or if you have an
appropriate filename munging system (see below) or some other system to
emulate the capability.

> Besides, why should the Tetris scores (or icon, or whatever) reside
> inside the executable, when said executable is perhaps shared by a
> bunch of machines on the network, each of which would like to keep its
> own? Or perhaps the sysadmin is terminally paranoid, and mounts /usr
> read-only (Hey, the FHS people defined what goes in there with
> /exactly/ that in mind too!).

Scores, no.  But Tetris still may have resource data that logically
belongs to it.  Like icons, internationalization data, UI data
(say a Glade file), etc.  It's all part of the same package; it would be
nice (IMHO) if it was grouped together in the filesystem.

I don't think MacOS X stores scores in the app file, but they have a lot
of other stuff in there -- stuff that I mentioned in the previous
paragraph.

>> So typing "/usr/games/tetris" from the command prompt does the same
>> thing as double-clicking it in the file manager.

> Right. And what should happen if I run the (logically equivalent)
> directory /home? Or /etc?

If there's no binary data attached to /home or /etc (i.e. it only has
directory contents), then nothing happens (other than an error
message).  That's like asking what happens if I do
"cat nonexistent.file".  If there's no data there to work with, then
nothing happens.

> What if I want to shove a directory into the tetris executable?
> Symlinks? Hard links to files inside? Outside?  Symlinks from the
> outside in? Hardlinks?

What problems do you see with those?

> And if you now move that to another filesystem, or ship it to another
> machine?

What happens when you copy a file from NTFS that has multiple streams?
Or a file that has extended attributes onto a filesystem that doesn't
support xattr?

> No, the answer "All that will be forbidden" is /not/ allowed, you want
> this stuff working "normally". "Unpack" the contents into real files
> and directories how?

If you really want to, you can do some filename munging.  The contents
of the /usr/games/tetris/ directory could be put in /usr/games/...tetris,
or something like that.

> "Pack" a directory into one of those structured objects? What would be
> the /practical/ difference between the packed and unpacked forms?

It's really, really ugly. ;-)

It's possible to just put all my files all in a single directory.  But
sorting things into different directories makes things look much nicer.

> (If none, why do it in the first place?  If the packed version has
> significant restrictions, why use it? If the packed version does have
> significant extra capabilities, why not just give those to regular
> objects?)

> Yet again, what somebody (I forget who) called the "Zero - one -
> infinity principle" (I think it was in the area of programming
> languages; which arguably are very complex user interfaces, just like
> what an operating system provides): It only makes sense giving zero,
> exactly one, or how many you wish for of some item/nesting. Here:
> Either files got /no/ strange things inside, or /exactly one/ (and
> then it becomes questionable, as there is no space for regular data
> anymore...), or whatever complex (sub)structure you want. But in the
> last case, there is /no/ difference between a file and a
> directory... and the whole setup is just mess for mess sake, nothing
> else.

Files would have two "forks" (can't think of any better terminology
right now).  A data fork, and a directory fork.

--- Begin useless analogy ---
Sort of like a real folder in meatspace.  I can talk about what I've
written on the folder (because I can write stuff on real physical
folders), or I can talk about what's inside the folder.

Or if I have a document, I can talk about the document itself, or the
random post-its, notes, and paperclips that I have attached to it.
---- End useless analogy ----

Right now, most Linux filesystems have xattrs and acls (stored as
xattrs).  Maybe in the future, the Samba people will get support for
multiple streams.  If I want to copy my NTFS file to ext3 and preserve
the multiple streams, then ext3 will need to support multiple streams
now.  Now we have two strange things inside.  Why not just unify the
interface using something like file-as-dir, or metafs that was proposed
in this thread, or openat, or something like that?

>> Of course, this change does require file managers to understand
>> default actions when it's ambiguous what to do on a double-click --
>> but MacOS X has that too, in the form of the "Open as Folder" option
>> (or whatever it's called).

> Right. For the sake of a filesystem among many on a minority operating
> system /all/ GUI programs have to be rewritten. And all command-line
> stuff. Just because.

Not rewritten; patched.  And not all programs.  Actually, most programs
would work just fine.  The Namesys people can tell you how many programs
didn't work properly when they tried out the metas directory, and how
much patching was required to get them to work properly again.  (Not
counting tar/cp/backup programs.  Everyone already knows about the
problems there.)

> Please /do/ think the above through, giving reasonable answers for
> /all/ of the operations mentioned. Tell what the advantage of all this
> would be on a multiuser operating system, when files are shared via
> the net, when files are being handled from read-only media (or
> filesystems). When different users want different metadata (I'm
> interested in the names of members of the band playing a song, you
> might want a high-resolution image of the album cover, the next guy
> wants the song's lyrics translated into swahili, all for the MP3s on
> the shared server or on the CD each of us bought separately). Consider
> the scenario where all this works only on /a very few/ machines (no
> Sun or *BSD box will have this for a very long time, and to get a
> significant fraction of just Linux systems will take some time), for
> only a limited selection of tools (existing tools will have to be
> rewritten or at least adapted, that doesn't happen overnight).

xattr is being added to most filesystems in Linux.  It doesn't work
everywhere, in all operating systems.  Programs will have to be patched
to work with it.  Does that mean that xattr is a bad idea?

> Factor in space and processing requirements for several, even
> hundreds, of concurrent users (or versions of metadata at least). How
> do you account for that? The 1 byte file with GiB of metadata where
> everybody and their pet iguana store their junk is handled how by
> quota systems?

How do you handle it now when everyone wants different information on a
song?

I don't really see what the problem is here.

Note: I'm not saying that keeping everyone's pet metadata with shared
files is a good idea.  This is your example.  The local library doesn't
allow me to make random annotations to their books, but I can do
whatever I want with my own books.

> What should happen if I email such a file with several versions of
> metadata to someone? Do they get just my metadata, or everybody's?

If I email a picture that I took to a friend, I don't send them the
whole 6 megapixel picture in RAW format.  I convert to JPEG and scale it
down.  If I email a music file that has the lyrics translated into
swahili, I can strip that bit out.  Nobody's forcing you to keep
everything.

> If I copy it to, say, a CD for safekeeping for myself?  For system
> backup purposes?  What if I copy such a file (with only my, or
> everybody's) metadata back from a backup?

You do whatever you want to do with your data.  If you want to keep the
pet iguana's metadata, go ahead.

> Or try to merge it with the version I took with my notebook on a trip,
> changing my parts while the otghers change theirs?  I'd assume all of
> those will requiere special tools, or at least flags selecting the
> right operation or some other unspecified magic, and the "simple to
> use" just went out for lunch.

So what do you do right now when you want to do something like that?
File-as-dir (or the other proposals) won't prevent you from doing that.
It just allows you more options.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Ross Biro]

On 6/29/05, Hubert Chan <hubert@uhoreg.ca> wrote:
> On Wed, 29 Jun 2005 01:09:05 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:
> 
> > Hubert Chan <hubert@uhoreg.ca> wrote: [...]
> > And doing "tar cf /dev/tape /usr/games/tetris" gives you a nice tangle
> > of undecipherable junk.
> 

I'm confused.  Can someone on one of these lists enlighten me?

How is directories as files logically any different than putting all
data into .data files and making all files directories (yes you would
need some sort of special handling for files that were really called
.data).  Then it's just a matter of deciding what happens when you
call open and stat on one of these files?

For backwards compatibility, current existing system calls have to
treat these things as directories.  Perhaps an exception could be made
for exec.

But we could have a whole new set of system calls that treat things as
magic, and if files as directories is as cool as many people think,
apps will start using the new api.  If not, they won't and the new api
can be deprecated.

    Ross

Re: reiser4 plugins

[Hubert Chan]

On Wed, 29 Jun 2005 17:34:41 -0400, Ross Biro <ross.biro@gmail.com> said:

> I'm confused.  Can someone on one of these lists enlighten me?

> How is directories as files logically any different than putting all
> data into .data files and making all files directories (yes you would
> need some sort of special handling for files that were really called
> .data).  Then it's just a matter of deciding what happens when you
> call open and stat on one of these files?

Logically, I don't think there is a difference. A filesystem that
doesn't support file-as-dir could implement the same functionality that
way. [1]  In fact, that's essentially what MacOS X/NeXTSTEP does with its
bundle format -- it's just a regular directory with regular files
inside.  The Reiser4 implementation would probably look very similar to
that structurally, except that since it understands file-as-dir, it
doesn't need to use the .data name, and it should still know where to
find it.  (Of course, I'm not a Reiser4 developer, and I don't speak for
them.)

[1] (Well, except that file-as-dir as some people would like it to
operate, can do a whole lot of other magic.  (e.g. transparent compress,
etc.  In Reiser4, it's also supposed to allow an interface to filesystem
stuff so that you can manipulate parameters such as whether it should
use the cryptocompress plugin for that file.)  But I think this is a
sort of separate issue.)

> For backwards compatibility, current existing system calls have to
> treat these things as directories.  Perhaps an exception could be made
> for exec.

Which makes the system really quite ugly.  I certainly wouldn't enjoy
having to type "$EDITOR foo.txt/.data" instead of "$EDITOR foo.txt".  If
you want to introduce new system calls, you would have to patch all the
programs pretty much overnight in order for the system to be useful.  It
also makes portability a big pain if you want to support systems that
don't offer the same system calls.

You also get a chicken-and-egg problem.  Application developers don't
patch their programs because nobody's using the system.  Nobody's using
the system because no applications support it.

> But we could have a whole new set of system calls that treat things as
> magic, and if files as directories is as cool as many people think,
> apps will start using the new api.  If not, they won't and the new api
> can be deprecated.

File-as-dir doesn't require new system calls (that I know of), which is
the whole point of the idea.  Existing programs can edit the strange new
attributes without being modified.

The main thing blocking file-as-dir is that there are some
locking(IIRC?) issues.  And, of course, some people wouldn't want it to
be merged into the mainline kernel.  (Of course, the latter doesn't
prevent Namesys from maintaining their own patches for people to play
around with.)

I would like to see file-as-dir so that we can try it out to see if it's
as useful as some of us think it is.  It might be less useful than I
expect, or it could exceed my expectations and someone might come up
with some killer use for it, once we can start playing with it.  And I
think it would be nice if it was merged into mainline so that it gets
more exposure, and we have more people trying it out.

People like Horst (and probably others, who are less vocal), I think,
don't think that it's even worth trying it out because they don't see
any major advantages.  Or at least they think that the potential
negatives outweigh the potential positives.  I respect that they have
different opinions, but I of course disagree and attempt to convince
them otherwise.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Masover]

Hubert Chan wrote:
> On Wed, 29 Jun 2005 17:34:41 -0400, Ross Biro <ross.biro@gmail.com> said:
> 
> 
>>I'm confused.  Can someone on one of these lists enlighten me?
> 
> 
>>How is directories as files logically any different than putting all
>>data into .data files and making all files directories (yes you would
>>need some sort of special handling for files that were really called
>>.data).  Then it's just a matter of deciding what happens when you
>>call open and stat on one of these files?
> 
> 
> Logically, I don't think there is a difference. A filesystem that
> doesn't support file-as-dir could implement the same functionality that
> way. [1]  In fact, that's essentially what MacOS X/NeXTSTEP does with its
> bundle format -- it's just a regular directory with regular files
> inside.

I, personally, would hate it if everything in my /bin suddenly became a 
directory, mainly because everything would stop working.  Is that the 
kind of thing you're suggesting?

I'm a little confused about the .data idea, I guess.

>>But we could have a whole new set of system calls that treat things as
>>magic, and if files as directories is as cool as many people think,
>>apps will start using the new api.  If not, they won't and the new api
>>can be deprecated.
> 
> 
> File-as-dir doesn't require new system calls (that I know of), which is
> the whole point of the idea.  Existing programs can edit the strange new
> attributes without being modified.

That is indeed the point, but scroll down.

> The main thing blocking file-as-dir is that there are some
> locking(IIRC?) issues.  And, of course, some people wouldn't want it to
> be merged into the mainline kernel.  (Of course, the latter doesn't
> prevent Namesys from maintaining their own patches for people to play
> around with.)

What's the locking issue?  I think that was more about transactions...

[...]
> People like Horst (and probably others, who are less vocal), I think,
> don't think that it's even worth trying it out because they don't see
> any major advantages.  Or at least they think that the potential
> negatives outweigh the potential positives.  I respect that they have
> different opinions, but I of course disagree and attempt to convince
> them otherwise.

Did the /meta (metafs) idea get killed while I was out?  Using that 
approach, your potential negatives are that apps which crawl the entire 
FS tree, starting at /, with hardcoded apps for /proc and /sys, are now 
broken -- but then, /sys already broke them once, so I don't 
particularly care if we break them again.

Potential positives?  I think even just because we like the idea is 
enough, because it doesn't break anything and doesn't really affect 
anyone who doesn't use it.

Maybe there are coding standards, but I think others are working that 
out now.

Re: reiser4 plugins

[Hans Reiser]

It was always the expectation that users would want to have one
mountpoint with the files having metafiles as files, and another with
the same files but strictly posix, and then their apps can use whichever
they have the power to understand.

Hans

David Masover wrote:

> Hubert Chan wrote:
>
>> On Wed, 29 Jun 2005 17:34:41 -0400, Ross Biro <ross.biro@gmail.com>
>> said:
>>
>>
>>> I'm confused.  Can someone on one of these lists enlighten me?
>>
>>
>>
>>> How is directories as files logically any different than putting all
>>> data into .data files and making all files directories (yes you would
>>> need some sort of special handling for files that were really called
>>> .data).  Then it's just a matter of deciding what happens when you
>>> call open and stat on one of these files?
>>
>>
>>
>> Logically, I don't think there is a difference. A filesystem that
>> doesn't support file-as-dir could implement the same functionality that
>> way. [1]  In fact, that's essentially what MacOS X/NeXTSTEP does with
>> its
>> bundle format -- it's just a regular directory with regular files
>> inside.
>
>
> I, personally, would hate it if everything in my /bin suddenly became
> a directory, mainly because everything would stop working.  Is that
> the kind of thing you're suggesting?
>
> I'm a little confused about the .data idea, I guess.
>
>>> But we could have a whole new set of system calls that treat things as
>>> magic, and if files as directories is as cool as many people think,
>>> apps will start using the new api.  If not, they won't and the new api
>>> can be deprecated.
>>
>>
>>
>> File-as-dir doesn't require new system calls (that I know of), which is
>> the whole point of the idea.  Existing programs can edit the strange new
>> attributes without being modified.
>
>
> That is indeed the point, but scroll down.
>
>> The main thing blocking file-as-dir is that there are some
>> locking(IIRC?) issues.  And, of course, some people wouldn't want it to
>> be merged into the mainline kernel.  (Of course, the latter doesn't
>> prevent Namesys from maintaining their own patches for people to play
>> around with.)
>
>
> What's the locking issue?  I think that was more about transactions...
>
> [...]
>
>> People like Horst (and probably others, who are less vocal), I think,
>> don't think that it's even worth trying it out because they don't see
>> any major advantages.  Or at least they think that the potential
>> negatives outweigh the potential positives.  I respect that they have
>> different opinions, but I of course disagree and attempt to convince
>> them otherwise.
>
>
> Did the /meta (metafs) idea get killed while I was out?  Using that
> approach, your potential negatives are that apps which crawl the
> entire FS tree, starting at /, with hardcoded apps for /proc and /sys,
> are now broken -- but then, /sys already broke them once, so I don't
> particularly care if we break them again.
>
> Potential positives?  I think even just because we like the idea is
> enough, because it doesn't break anything and doesn't really affect
> anyone who doesn't use it.
>
> Maybe there are coding standards, but I think others are working that
> out now.
>
>

Re: reiser4 plugins

[David Masover]

Hans Reiser wrote:
> It was always the expectation that users would want to have one
> mountpoint with the files having metafiles as files, and another with
> the same files but strictly posix, and then their apps can use whichever
> they have the power to understand.

It was never in the early betas I tried :(

I'm proposing (or re-proposing) that the /meta mountpoint be strictly 
for accessing meta-files, with no intention of eventually using this by 
default.  Furthermore, /meta should follow POSIX anyway, mostly -- no 
file-as-dir there, either, although you still wouldn't want to use tar 
on it.

With only a few patches, using /meta could be almost as convenient as 
using file/..metas/foo, and it completely kills the file-as-directory 
flamewar -- everybody's happy.

Or so I thought.  It seems that the people arguing for file-as-directory 
are ignoring /meta, and the people arguing against it are arguing 
against all meta-files, saying that the good things about meta-files 
don't justify the risks of file-as-dir.  Only once did I see someone 
bring up /meta.

I don't like the idea of having /meta have file-as-dir and everything as 
we originally wanted, because then we've duplicated the interface.  To 
read the *contents* of /foo, I can either cat /foo or cat /meta/foo. 
I'd hate to have the POSIX mountpoint still lying around if no one's 
using it, even more than I hate the idea of "foo" being in two places 
for no good reason.

Is there a technical (performance?) reason that my approach is wrong? 
(metas go in /meta, files go in /, and everything feels POSIX-y)

> Hans
> 
> David Masover wrote:
> 
> 
>>Hubert Chan wrote:
>>
>>
>>>On Wed, 29 Jun 2005 17:34:41 -0400, Ross Biro <ross.biro@gmail.com>
>>>said:
>>>
>>>
>>>
>>>>I'm confused.  Can someone on one of these lists enlighten me?
>>>
>>>
>>>
>>>>How is directories as files logically any different than putting all
>>>>data into .data files and making all files directories (yes you would
>>>>need some sort of special handling for files that were really called
>>>>.data).  Then it's just a matter of deciding what happens when you
>>>>call open and stat on one of these files?
>>>
>>>
>>>
>>>Logically, I don't think there is a difference. A filesystem that
>>>doesn't support file-as-dir could implement the same functionality that
>>>way. [1]  In fact, that's essentially what MacOS X/NeXTSTEP does with
>>>its
>>>bundle format -- it's just a regular directory with regular files
>>>inside.
>>
>>
>>I, personally, would hate it if everything in my /bin suddenly became
>>a directory, mainly because everything would stop working.  Is that
>>the kind of thing you're suggesting?
>>
>>I'm a little confused about the .data idea, I guess.
>>
>>
>>>>But we could have a whole new set of system calls that treat things as
>>>>magic, and if files as directories is as cool as many people think,
>>>>apps will start using the new api.  If not, they won't and the new api
>>>>can be deprecated.
>>>
>>>
>>>
>>>File-as-dir doesn't require new system calls (that I know of), which is
>>>the whole point of the idea.  Existing programs can edit the strange new
>>>attributes without being modified.
>>
>>
>>That is indeed the point, but scroll down.
>>
>>
>>>The main thing blocking file-as-dir is that there are some
>>>locking(IIRC?) issues.  And, of course, some people wouldn't want it to
>>>be merged into the mainline kernel.  (Of course, the latter doesn't
>>>prevent Namesys from maintaining their own patches for people to play
>>>around with.)
>>
>>
>>What's the locking issue?  I think that was more about transactions...
>>
>>[...]
>>
>>
>>>People like Horst (and probably others, who are less vocal), I think,
>>>don't think that it's even worth trying it out because they don't see
>>>any major advantages.  Or at least they think that the potential
>>>negatives outweigh the potential positives.  I respect that they have
>>>different opinions, but I of course disagree and attempt to convince
>>>them otherwise.
>>
>>
>>Did the /meta (metafs) idea get killed while I was out?  Using that
>>approach, your potential negatives are that apps which crawl the
>>entire FS tree, starting at /, with hardcoded apps for /proc and /sys,
>>are now broken -- but then, /sys already broke them once, so I don't
>>particularly care if we break them again.
>>
>>Potential positives?  I think even just because we like the idea is
>>enough, because it doesn't break anything and doesn't really affect
>>anyone who doesn't use it.
>>
>>Maybe there are coding standards, but I think others are working that
>>out now.
>>
>>
> 
> 

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

> Hans Reiser wrote:
>
>> It was always the expectation that users would want to have one
>> mountpoint with the files having metafiles as files, and another with
>> the same files but strictly posix, and then their apps can use whichever
>> they have the power to understand.
>
>
> It was never in the early betas I tried :(

Yes, well, there are a lot of things missing in functionality still from
what was, and still is, in the plan.  Inheritance, files not listed by
readdir, etc., lots of things need coding.  We have done the hard stuff
first though, so these other things will mostly not be a lot of code....

Hans

Re: reiser4 plugins

[Hubert Chan]

On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said:

> Hubert Chan wrote:

>> The main thing blocking file-as-dir is that there are some
>> locking(IIRC?) issues.  And, of course, some people wouldn't want it
>> to be merged into the mainline kernel.  (Of course, the latter
>> doesn't prevent Namesys from maintaining their own patches for people
>> to play around with.)

> What's the locking issue?  I think that was more about transactions...

It was whatever was Al Viro's (technical) complaint about file-as-dir.
I don't remember exactly what it was.  The technical people know what it
is (and the Namesys guys are probably working on it), and the exact
issue doesn't concern us non-technical people that much, so I don't feel
like looking it up.  But if you want to, just look for Al Viro's message
in this thread.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Hans Reiser]

Hubert Chan wrote:

>On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said:
>
>  
>
>>Hubert Chan wrote:
>>    
>>
>
>  
>
>>>The main thing blocking file-as-dir is that there are some
>>>locking(IIRC?) issues.  And, of course, some people wouldn't want it
>>>to be merged into the mainline kernel.  (Of course, the latter
>>>doesn't prevent Namesys from maintaining their own patches for people
>>>to play around with.)
>>>      
>>>
>
>  
>
>>What's the locking issue?  I think that was more about transactions...
>>    
>>
>
>It was whatever was Al Viro's (technical) complaint about file-as-dir.
>I don't remember exactly what it was.  The technical people know what it
>is (and the Namesys guys are probably working on it), and the exact
>issue doesn't concern us non-technical people that much, so I don't feel
>like looking it up.  But if you want to, just look for Al Viro's message
>in this thread.
>
>  
>
Cycle detection when hard links to directories are allowed.  There is a
debate over whether cycle detection is feasible that can only be
resolved by working code or a formal proof that it is not
computationally feasible.

Hans

Re: reiser4 plugins

[David Masover]

Hans Reiser wrote:
> Hubert Chan wrote:
> 
> 
>>On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said:
>>
>> 
>>
>>
>>>Hubert Chan wrote:
>>>   
>>>
>>
>> 
>>
>>
>>>>The main thing blocking file-as-dir is that there are some
>>>>locking(IIRC?) issues.  And, of course, some people wouldn't want it
>>>>to be merged into the mainline kernel.  (Of course, the latter
>>>>doesn't prevent Namesys from maintaining their own patches for people
>>>>to play around with.)
>>>>     
>>>>
>>
>> 
>>
>>
>>>What's the locking issue?  I think that was more about transactions...
>>>   
>>>
>>
>>It was whatever was Al Viro's (technical) complaint about file-as-dir.
>>I don't remember exactly what it was.  The technical people know what it
>>is (and the Namesys guys are probably working on it), and the exact
>>issue doesn't concern us non-technical people that much, so I don't feel
>>like looking it up.  But if you want to, just look for Al Viro's message
>>in this thread.
>>
>> 
>>
> 
> Cycle detection when hard links to directories are allowed.  There is a
> debate over whether cycle detection is feasible that can only be
> resolved by working code or a formal proof that it is not
> computationally feasible.

Ah.  But then, one solution was to avoid the issue at all, and have the
directory inside a file act as a mountpoint.  After all, mount --bind
doesn't cause problems...

Hey!  This sounds like metafs (/meta) already!  I wonder if we can do
file-as-dir in /meta, and just not support user-created hardlinks there?
 (other than creating brand-new files, of course...)

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

>Hans Reiser wrote:
>  
>
>>Hubert Chan wrote:
>>
>>
>>    
>>
>>>On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said:
>>>
>>>
>>>
>>>
>>>      
>>>
>>>>Hubert Chan wrote:
>>>>  
>>>>
>>>>        
>>>>
>>>
>>>
>>>      
>>>
>>>>>The main thing blocking file-as-dir is that there are some
>>>>>locking(IIRC?) issues.  And, of course, some people wouldn't want it
>>>>>to be merged into the mainline kernel.  (Of course, the latter
>>>>>doesn't prevent Namesys from maintaining their own patches for people
>>>>>to play around with.)
>>>>>    
>>>>>
>>>>>          
>>>>>
>>>
>>>
>>>      
>>>
>>>>What's the locking issue?  I think that was more about transactions...
>>>>  
>>>>
>>>>        
>>>>
>>>It was whatever was Al Viro's (technical) complaint about file-as-dir.
>>>I don't remember exactly what it was.  The technical people know what it
>>>is (and the Namesys guys are probably working on it), and the exact
>>>issue doesn't concern us non-technical people that much, so I don't feel
>>>like looking it up.  But if you want to, just look for Al Viro's message
>>>in this thread.
>>>
>>>
>>>
>>>      
>>>
>>Cycle detection when hard links to directories are allowed.  There is a
>>debate over whether cycle detection is feasible that can only be
>>resolved by working code or a formal proof that it is not
>>computationally feasible.
>>    
>>
>
>Ah.  But then, one solution was to avoid the issue at all, and have the
>directory inside a file act as a mountpoint.  After all, mount --bind
>doesn't cause problems...
>  
>
Can you explain this idea at greater length?

>Hey!  This sounds like metafs (/meta) already!  I wonder if we can do
>file-as-dir in /meta, and just not support user-created hardlinks there?
> (other than creating brand-new files, of course...)
>
>
>  
>

Re: reiser4 plugins

[David Masover]

Hans Reiser wrote:
> David Masover wrote:
> 
> 
>>Hans Reiser wrote:
>> 
>>
>>
>>>Hubert Chan wrote:
>>>
>>>
>>>   
>>>
>>>
>>>>On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said:
>>>>
>>>>
>>>>
>>>>
>>>>     
>>>>
>>>>
>>>>>Hubert Chan wrote:
>>>>> 
>>>>>
>>>>>       
>>>>>
>>>>
>>>>
>>>>     
>>>>
>>>>
>>>>>>The main thing blocking file-as-dir is that there are some
>>>>>>locking(IIRC?) issues.  And, of course, some people wouldn't want it
>>>>>>to be merged into the mainline kernel.  (Of course, the latter
>>>>>>doesn't prevent Namesys from maintaining their own patches for people
>>>>>>to play around with.)
>>>>>>   
>>>>>>
>>>>>>         
>>>>>>
>>>>
>>>>
>>>>     
>>>>
>>>>
>>>>>What's the locking issue?  I think that was more about transactions...
>>>>> 
>>>>>
>>>>>       
>>>>>
>>>>
>>>>It was whatever was Al Viro's (technical) complaint about file-as-dir.
>>>>I don't remember exactly what it was.  The technical people know what it
>>>>is (and the Namesys guys are probably working on it), and the exact
>>>>issue doesn't concern us non-technical people that much, so I don't feel
>>>>like looking it up.  But if you want to, just look for Al Viro's message
>>>>in this thread.
>>>>
>>>>
>>>>
>>>>     
>>>>
>>>
>>>Cycle detection when hard links to directories are allowed.  There is a
>>>debate over whether cycle detection is feasible that can only be
>>>resolved by working code or a formal proof that it is not
>>>computationally feasible.
>>>   
>>>
>>
>>Ah.  But then, one solution was to avoid the issue at all, and have the
>>directory inside a file act as a mountpoint.  After all, mount --bind
>>doesn't cause problems...
>> 
>>
> 
> Can you explain this idea at greater length?

Just don't allow user-created hardlinks inside either metafs (/meta) or
the magical meta directory inside files.

In order to do that, one way would be to have "file/..." appear as a
mountpoint.  I don't know if this is feasable, performance-wise, but I
think it would work.

>>Hey!  This sounds like metafs (/meta) already!  I wonder if we can do
>>file-as-dir in /meta, and just not support user-created hardlinks there?
>>(other than creating brand-new files, of course...)

This is still my preferred solution, because it's not any harder or less
efficient to develop new apps based on metafs than on file-as-dir, but
it means that old apps will see something *entirely* POSIX-compliant,
and the strangeness will be confined to /meta.

Basically, you have to allow hardlinks in /meta, in case some plugin
wants to do that, but if you have files that are also directories, you
also have to make sure that users can't create hardlinks.  I don't think
this would be particularly hard, though.  Pretend it's a read-only
filesystem unless the user is doing something safe, like creating a
brand-new file, deleting an old one, or modifying the contents of an
existing one, all assuming that the plugin responsible for the
file/directory you're doing this in allows it.

But then, if we're doing metafs, I don't think you need
file-as-directory, and certain existing tools that we'd like to be able
to point at metafs might not like file-as-directory anyway.

Now, can anyone think of a situation where we want user-created
hardlinks inside metadata?  More importantly, what do we do about it?

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

>Now, can anyone think of a situation where we want user-created
>hardlinks inside metadata?  More importantly, what do we do about it?
>  
>
I think the equivalent of symlinks would be good enough to get by on for
now for most linking of metafiles.  Maybe some years from now somebody
can fault me for saying this and write a patch to fix it to be better,
at which point I will be happy to concede the point.

So the basic principal here is, one can have hardlinks to directories
without cycles provided that one does not allow any child of the
directory to have a hardlink.  The question is, how cleanly can that
relaxed restriction be coded?

Hans

Re: reiser4 plugins

[Hans Reiser]

If we also add to this the restriction that once you create the file
part of a file-directory, you can never hardlink to a child of it, it
should then all work, yes?

So, /filename/..../owner should be able to avoid colliding with any
common names by virtue of the '....', and not letting any filedir
(file-directory) have children with links to them should also work.  The
one thing that seems inelegant is that when you create the file part of
a filedir, you must check all its children for hardlinks and fail if
they exist, and you must flag all its directory children so that the
plugins for them will disallow hardlinks to their children from that
point onward.  Still, seems workable....

Thanks David,

Hans

Hans Reiser wrote:

>David Masover wrote:
>
>  
>
>>Now, can anyone think of a situation where we want user-created
>>hardlinks inside metadata?  More importantly, what do we do about it?
>> 
>>
>>    
>>
>I think the equivalent of symlinks would be good enough to get by on for
>now for most linking of metafiles.  Maybe some years from now somebody
>can fault me for saying this and write a patch to fix it to be better,
>at which point I will be happy to concede the point.
>
>So the basic principal here is, one can have hardlinks to directories
>without cycles provided that one does not allow any child of the
>directory to have a hardlink.  The question is, how cleanly can that
>relaxed restriction be coded?
>
>Hans
>
>  
>

Re: reiser4 plugins

[David Masover]

Hans Reiser wrote:
> If we also add to this the restriction that once you create the file
> part of a file-directory, you can never hardlink to a child of it, it
> should then all work, yes?
> 
> So, /filename/..../owner should be able to avoid colliding with any
> common names by virtue of the '....', and not letting any filedir
> (file-directory) have children with links to them should also work.  The
> one thing that seems inelegant is that when you create the file part of
> a filedir, you must check all its children for hardlinks and fail if
> they exist, and you must flag all its directory children so that the
> plugins for them will disallow hardlinks to their children from that
> point onward.  Still, seems workable....

Ok, still haven't heard much discussion of metafs vs file-as-directory, 
but it seems like it'd be easier in metafs.

Basically, we are entirely POSIX compliant outside of metafs, so that 
/filename is a file and may be hardlinked, and /directory/ is a 
directory and may not.  No problems yet.

Inside /meta/inode, we have no problems yet because any hardlinks 
created outside /meta won't even show up as hardlinks here, so we don't 
get hardlinked directories.

Inside /meta/vfs, where we can find the metadata of '/filename' under 
'/meta/vfs/filename/...', we have what looks like a problem -- we may 
have hardlinks created outside metafs, which will show up as two 
directories inside metafs, so those directories are hardlinks.

I don't think that's such a problem, since we won't allow users to 
change anything in /meta/vfs except when it's inside metadata, such as 
'/meta/vfs/some/where/...'.

Thus, it's impossible to create a hardlink to a directory unless we do 
it *outside* metafs, as a hardlink to a file.

Now, the question I asked is, what if we want hardlinks *inside* metafs, 
*inside* the metadata for a given file, and would we ever want such a 
thing?  Because we can avoid the whole problem if we just disallow any 
sys_link calls inside metafs.

Of course, sometimes we want to have a chunk of metadata that appears 
*exactly* as if it were a normal, POSIX-compliant directory tree, such 
as the contents of a tarball or a zipfile.  In that case, we might want 
to have the "zip" plugin allow hardlinks inside 
"/meta/file.zip/.../contents" -- the only restriction is that any 
hardlink made to a file inside 'contents' must be made to another file 
inside 'contents'.

> Hans Reiser wrote:
> 
> 
>>David Masover wrote:
>>
>> 
>>
>>
>>>Now, can anyone think of a situation where we want user-created
>>>hardlinks inside metadata?  More importantly, what do we do about it?
>>>
>>>
>>>   
>>>
>>
>>I think the equivalent of symlinks would be good enough to get by on for
>>now for most linking of metafiles.  Maybe some years from now somebody
>>can fault me for saying this and write a patch to fix it to be better,
>>at which point I will be happy to concede the point.
>>
>>So the basic principal here is, one can have hardlinks to directories
>>without cycles provided that one does not allow any child of the
>>directory to have a hardlink.  The question is, how cleanly can that
>>relaxed restriction be coded?
>>
>>Hans
>>
>> 
>>
> 
> 

Re: reiser4 plugins

[Stefan Smietanowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Ok, still haven't heard much discussion of metafs vs file-as-directory,
> but it seems like it'd be easier in metafs.

Why not implement it inside the directory containg the file ?

Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo

This should be suit both camps I'd think?

I mean, editing something is easy and you don't have to "know" how
to navigate /meta and you don't have the clash of files vs metadata
(is /meta/vfs/home/stesmi/foo a file or an attribute called foo of
the dir stesmi ?).

/home/stesmi/foo <- dir
/home/stesmi/.meta/foo <- "dir" containing all metadata
/home/stesmi/.meta/foo/attrib <- some attribute called attrib
/home/stesmi/foo/bar <- file
/home/stesmi/foo/.meta/bar <- "dir" containg all metadata
/home/stesmi/foo/.meta/bar/attrib <- some attribute called attrib

The file is $dir/$file. The attrib dir is $dir/.meta/$file.
The attribute is $dir/.meta/$file/$attribute.

If $attrib is something user-editable it's easy to
$EDITOR $dir/.meta/$file/$attrib

If this has already been taken up, I must've missed it.

// Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)

iD8DBQFC0fBsBrn2kJu9P78RAlt4AJ4qWik6hA4oXBNZMdZ1TkweYrJHmgCdFAY+
m+Qtg9uqBq9m0qKfRkK6iUI=
=ghWb
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[David Masover]

Stefan Smietanowski wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
>>Ok, still haven't heard much discussion of metafs vs file-as-directory,
>>but it seems like it'd be easier in metafs.
> 
> 
> Why not implement it inside the directory containg the file ?
> 
> Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo
> 
> This should be suit both camps I'd think?

You still need to figure out the parent of "foo", which isn't 
necessarily easy, especially considering that even if we store a link to 
the parent, it will be inside the metadata.  Then you have a 
chicken-and-egg situation.

Both camps seem to want to allow easy access to the metadata of a file, 
whether we're given that file as an inode or as a pathname.  That's why 
I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a 
file by name, and sometimes by inode, but either way, I should be able 
to get its metadata.

> I mean, editing something is easy and you don't have to "know" how
> to navigate /meta

But then you have to "know" how to navigate .meta, and more importantly, 
how to find .meta

> and you don't have the clash of files vs metadata
> (is /meta/vfs/home/stesmi/foo a file or an attribute called foo of
> the dir stesmi ?).

So, how do I get metadata of a directory?  If the metadata for /home/me 
is in /home/.meta/me, and the metadata for /home is in /.meta/home, then 
where is the metadata for / ?

> /home/stesmi/foo <- dir
> /home/stesmi/.meta/foo <- "dir" containing all metadata
> /home/stesmi/.meta/foo/attrib <- some attribute called attrib
> /home/stesmi/foo/bar <- file
> /home/stesmi/foo/.meta/bar <- "dir" containg all metadata
> /home/stesmi/foo/.meta/bar/attrib <- some attribute called attrib

[...]

> If this has already been taken up, I must've missed it.

It looks a lot like how I suggested we resolve the ambiguity within 
/meta/vfs, only I called it '...' instead of '.meta'.

Either way, the major challenge to this method is not so much whether 
it'd work, but how to choose a suitable delimiter?  The delimiter must 
be standard if we're going to have apps develop for it, and it must not 
be used in the name of any existing file or directory.

I had a long essay here on how '.meta' breaks every recursive operation 
on directories (rm -rf, tar, mkisofs), but I deleted it when I 
remembered that I played around with the alpha/beta reiser4 which had a 
'metas' dir that did the same thing, but was hidden from the normal 
directory listing.  'cd metas' worked, 'ls metas' worked, but 'ls' 
showed no directory called 'metas'.

I don't *think* there are any apps that will break if you tell them to 
open a path that doesn't exist in a directory listing.  That is, typing 
'vim /home/metas/acl' should always let me edit the acl on /home, and 
vim should never complain about how /home/metas doesn't show up in 
/home.  A program would have to be pretty retarded to fail on something 
like that.

But, we have to support some pretty retarded programs.  That is why I 
like /meta -- the default view is a completely POSIX-compliant tree that 
works with tar, and even the /meta view is POSIX-compliant, even if it'd 
be a bad idea to tar it.  Then we don't have to worry at all about 
stupid programs.

How much should we be worrying about this particular brand of stupidity? 
  And what's a good delimiter?


-- 
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.323 / Virus Database: 267.8.12/46 - Release Date: 7/11/2005

Re: reiser4 plugins

[Stefan Smietanowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi.

>> Why not implement it inside the directory containg the file ?
>>
>> Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo
>>
>> This should be suit both camps I'd think?
> 
> You still need to figure out the parent of "foo", which isn't
> necessarily easy, especially considering that even if we store a link to
> the parent, it will be inside the metadata.  Then you have a
> chicken-and-egg situation.

Either you're describing another problem than I'm thinking of or
/home/stesmi/.meta/foo 's "file" will always be /home/stesmi/foo .
$dir/.meta/$file and $dir/$file .

Or do you mean something else when you say "parent" ?

Do you mean that if you know $dir/.meta/$foo then you don't know
the meta of the parent?
Won't $dir/../.meta/$end-of-path/.meta solve that?

/home/stesmi/foo <- file
/home/stesmi/.meta/foo <- meta of foo
/home/stesmi/ <- parentdir
/home/.meta/stesmi <- meta of parentdir
/home/stesmi/../.meta/stesmi <- also meta of parentdir

Or is your problem that you don't know "stesmi" ?

Ie you can't figure out what the current dir name is ?

So you can't construct /home/stesmi/../stesmi because you don't
know the name of the dir?

In that case do :
/home/stesmi/.meta/. <- meta of parent. Naturally same as
/home/.meta/stesmi

And if then ".." is mirrored as well is a secondary discussion.
/home/stesmi/.meta/. <- meta of "stesmi"
/home/stesmi/.meta/.. <- meta of "home"
/home/stesmi/.meta/foo <- meta of file "foo"

> Both camps seem to want to allow easy access to the metadata of a file,
> whether we're given that file as an inode or as a pathname.  That's why
> I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a
> file by name, and sometimes by inode, but either way, I should be able
> to get its metadata.
> 
>> I mean, editing something is easy and you don't have to "know" how
>> to navigate /meta
> 
> But then you have to "know" how to navigate .meta, and more importantly,
> how to find .meta

The biggest problem in my eyes is exportability.

>> and you don't have the clash of files vs metadata
>> (is /meta/vfs/home/stesmi/foo a file or an attribute called foo of
>> the dir stesmi ?).
> 
> So, how do I get metadata of a directory?  If the metadata for /home/me
> is in /home/.meta/me, and the metadata for /home is in /.meta/home, then
> where is the metadata for / ?

/.meta/. (look above)

The "." is also inside .meta

>> /home/stesmi/foo <- dir
>> /home/stesmi/.meta/foo <- "dir" containing all metadata
>> /home/stesmi/.meta/foo/attrib <- some attribute called attrib
>> /home/stesmi/foo/bar <- file
>> /home/stesmi/foo/.meta/bar <- "dir" containg all metadata
>> /home/stesmi/foo/.meta/bar/attrib <- some attribute called attrib
> 
> 
> [...]
> 
>> If this has already been taken up, I must've missed it.
> 
> 
> It looks a lot like how I suggested we resolve the ambiguity within
> /meta/vfs, only I called it '...' instead of '.meta'.

Ah. When I read the talk about "..." I mistook it for something else.

I'm sorry.

> Either way, the major challenge to this method is not so much whether
> it'd work, but how to choose a suitable delimiter?  The delimiter must
> be standard if we're going to have apps develop for it, and it must not
> be used in the name of any existing file or directory.

I think "..." and ".meta" both serve as a logical delimiter. However
some programs implement their own "..." which would make it clash with
them. Naturally if some program created a directory called .meta we're
equally screwed.

> I had a long essay here on how '.meta' breaks every recursive operation
> on directories (rm -rf, tar, mkisofs), but I deleted it when I
> remembered that I played around with the alpha/beta reiser4 which had a
> 'metas' dir that did the same thing, but was hidden from the normal
> directory listing.  'cd metas' worked, 'ls metas' worked, but 'ls'
> showed no directory called 'metas'.
> 
> I don't *think* there are any apps that will break if you tell them to
> open a path that doesn't exist in a directory listing.  That is, typing
> 'vim /home/metas/acl' should always let me edit the acl on /home, and
> vim should never complain about how /home/metas doesn't show up in
> /home.  A program would have to be pretty retarded to fail on something
> like that.
> 
> But, we have to support some pretty retarded programs.  That is why I
> like /meta -- the default view is a completely POSIX-compliant tree that
> works with tar, and even the /meta view is POSIX-compliant, even if it'd
> be a bad idea to tar it.  Then we don't have to worry at all about
> stupid programs.
> 
> How much should we be worrying about this particular brand of stupidity?
>  And what's a good delimiter?

Very good points. I don't like it being seperate from the dir though so
I'm more inclined to support having the metas "close" to the file itself
(/home/stesmi/.meta/) instead of seperate from (/meta).

// Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)

iD8DBQFC0tygBrn2kJu9P78RAmBYAJ9YVeAacrREuBFrpGz9Ov5STo6dXwCgxrFi
d2BN/T+//CfIZXPCUN9uhC4=
=6SfT
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

Stefan Smietanowski wrote:

>
> I think "..." and ".meta" both serve as a logical delimiter. However
> some programs implement their own "..." which would make it clash with
> them. Naturally if some program created a directory called .meta we're
> equally screwed.

I chose '....' (four dots) because it clashes with less, not three dots.

Re: reiser4 plugins

[Horst von Brand]

Hans Reiser <reiser@namesys.com> wrote:
> Stefan Smietanowski wrote:
> > I think "..." and ".meta" both serve as a logical delimiter. However
> > some programs implement their own "..." which would make it clash with
> > them. Naturally if some program created a directory called .meta we're
> > equally screwed.

> I chose '....' (four dots) because it clashes with less, not three dots.

Is this some kind of "My dots are more than yours" contest?!

/None/ of them is safe. ".meta", "...", "....", ".this.has.five.dots." are
all perfectly legal file (or directory) names, POSIXly. If any one of them
won't do, none will.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Stefan Traby]

On Mon, Jul 11, 2005 at 10:33:24PM -0400, Horst von Brand wrote:
> Hans Reiser <reiser@namesys.com> wrote:
> > I chose '....' (four dots) because it clashes with less, not three dots.
> 
> Is this some kind of "My dots are more than yours" contest?!
> 
> /None/ of them is safe. ".meta", "...", "....", ".this.has.five.dots." are
> all perfectly legal file (or directory) names, POSIXly. If any one of them
> won't do, none will.

Correct. I suggest "lost+found".
It's also a legal name but was always somewhat special.
[not sure if I should place a smiley here]

-- 

  ciao - 
    Stefan

Re: reiser4 plugins

[Hans Reiser]

Horst von Brand wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>Stefan Smietanowski wrote:
>>    
>>
>>>I think "..." and ".meta" both serve as a logical delimiter. However
>>>some programs implement their own "..." which would make it clash with
>>>them. Naturally if some program created a directory called .meta we're
>>>equally screwed.
>>>      
>>>
>
>  
>
>>I chose '....' (four dots) because it clashes with less, not three dots.
>>    
>>
>
>Is this some kind of "My dots are more than yours" contest?!
>
>/None/ of them is safe. ".meta", "...", "....", ".this.has.five.dots." are
>all perfectly legal file (or directory) names, POSIXly. If any one of them
>won't do, none will.
>  
>
There is a long history of encroaching namespaces by creating new
keywords in CS, and it is a survivable problem.

Clearcase (the version control filesystem) is an excellent example. 
They have special meanings for @ and some other common characters, and
you have to learn how to escape those characters if you use them in
Clearcase.

It works, it makes users happy, in practice it is far less of a problem
than one might think.  I think there were two or three times I had to
remember how to escape things in 2 years of using it.....

Better to spend one's mind looking for bugs instead of this issue.....

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
> Horst von Brand wrote:
> 
> 
>>Hans Reiser <reiser@namesys.com> wrote:
>> 
>>
>>
>>>Stefan Smietanowski wrote:
>>>   
>>>
>>>
>>>>I think "..." and ".meta" both serve as a logical delimiter. However
>>>>some programs implement their own "..." which would make it clash with
>>>>them. Naturally if some program created a directory called .meta we're
>>>>equally screwed.
>>>>     
>>>>
>>
>> 
>>
>>
>>>I chose '....' (four dots) because it clashes with less, not three dots.
>>>   
>>>
>>
>>Is this some kind of "My dots are more than yours" contest?!
>>
>>/None/ of them is safe. ".meta", "...", "....", ".this.has.five.dots." are
>>all perfectly legal file (or directory) names, POSIXly. If any one of them
>>won't do, none will.

And, conversely, if any one of them makes sense, they all do.  Well,
with the exception that some of them have never, ever been used before,
and that's what we should be aiming for.  No conflicts with existing
programs, then new programs can be written to avoid conflicting with us.

> There is a long history of encroaching namespaces by creating new
> keywords in CS, and it is a survivable problem.

Agreed, but not one to be taken too lightly.

> Clearcase (the version control filesystem) is an excellent example. 
> They have special meanings for @ and some other common characters, and
> you have to learn how to escape those characters if you use them in
> Clearcase.

I like using @, though.  Without having to escape it.

That's why we're trying to find something that people won't actually
touch, especially since if we design it right, this will be the last
delimiter introduced at the fs/vfs level.

> It works, it makes users happy, in practice it is far less of a problem
> than one might think.  I think there were two or three times I had to
> remember how to escape things in 2 years of using it.....
> 
> Better to spend one's mind looking for bugs instead of this issue.....

.....if bugs were seen as such a big deal.

I think it's far easier to get into the kernel with something
ludicrously buggy than something which actually changes fundamental
behavior.  That is, you can put in an FS which actually corrupts data
(such as the old NTFS write support), so long as it doesn't break POSIX,
or cause other weird restrictions like "No files named 'metas'"

Now, if we can decide that we don't care about being in the vanilla
kernel, then we can just call it ".metas" or "lost+found" or whatever
and get to work on bug fixes and other much-needed features such as a
repacker.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQtRQvXgHNmZLgCUhAQJXWBAAlZggJQGNi2lEZh0MAqnz+rNVT1JYcY/b
adHgvVOZZNiJw2GmVjGLIiG5cqSqML1//f0+4XOxjYED2rbfOwBJiDqcq/0IsKPp
5JJflJV2uWkEZukJ+oA5mspfSeKof2N5egBoD1Ije079VBKXdjoN5Kprkbe4ZYZ2
+Yu+w3FpdcaGvSqVlTRHPWsS0je4z8ieh0u6ql+HNR4ze/hKwMw4nrb2sARW9NOQ
EZ8Ot5NDhVaxz9Rj72rLVuQrUZEF8b0ihkpmzTauVV/nysEGi33SPqYTF7nYGBnM
5mVY63uXG4zxmThMDpn+J/iofA/hS1fd1bY9tCgF1ZAPu9HrCTnVzKaTYoOq+ciD
sY0m7HEc49JfaiyZ/SJGH02WUH3JqLQAVGQftEkqAQLyYdVRbzdBHUVUR2i8hX2M
ofFLM6DGJgFK784PfO0sjNboQT5ay4B8js8NltdfytsOVwDzjMST7dZAWcnMgTZU
jrMCJuMYeYh8468fy1C9pCMUPahVXvmF4O5Xazt7m9HvV4lxRJIUDZsaR0Volg2K
OAQIO3rtT9heK/+/PaUuXX8RYwLQUlhdUmdHfdtjRiLtKRcbw/fYBOqjkgXZfOfy
RzyBgD463sVCVAE8qMbAVnnHNGKZ25tyRTYITiTZ2kfnIeURJawj2SZVmn5ezGQG
xRnI3+Ir5e0=
=moA6
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

>
> That's why we're trying to find something that people won't actually
> touch, especially since if we design it right, this will be the last
> delimiter introduced at the fs/vfs level.

Uh, no, there needs to be about a dozen or so more.

But not this year.

Re: reiser4 plugins

[David Masover]

Hans Reiser wrote:
> David Masover wrote:
> 
> 
>>That's why we're trying to find something that people won't actually
>>touch, especially since if we design it right, this will be the last
>>delimiter introduced at the fs/vfs level.
> 
> 
> Uh, no, there needs to be about a dozen or so more.

Where?

 From what I (vaguely) remember of the future-vision paper, having the 
meta delimiter lets us do everything else from inside the metas.  We can 
certainly add delimiters to stuff in a meta-dir...


-- 
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.323 / Virus Database: 267.8.12/46 - Release Date: 7/11/2005

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:
> Hans Reiser wrote:
> > Horst von Brand wrote:
> >>Hans Reiser <reiser@namesys.com> wrote:
> >>>Stefan Smietanowski wrote:

[...]

> > Better to spend one's mind looking for bugs instead of this issue.....
> 
> .....if bugs were seen as such a big deal.

> I think it's far easier to get into the kernel with something
> ludicrously buggy than something which actually changes fundamental
> behavior.


Wonder why....

[Fixing bugs in the $FOO driver or the $BAR filesystem is /easy/, fixing
 bugs in "fundamental behaviour changes" is /extremely hard/.]

>            That is, you can put in an FS which actually corrupts data
> (such as the old NTFS write support), so long as it doesn't break POSIX,
> or cause other weird restrictions like "No files named 'metas'"

Because that kind of problems are isolated. If you introduce a change that
affects /all/ filesystems, and that change later on has unfixable bugs, or
fundamental design issues, it is /a lot/ of work.

> Now, if we can decide that we don't care about being in the vanilla
> kernel, then we can just call it ".metas" or "lost+found" or whatever
> and get to work on bug fixes and other much-needed features such as a
> repacker.

Great!
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:

[...]

> Both camps seem to want to allow easy access to the metadata of a
> file, whether we're given that file as an inode or as a pathname.
> That's why I suggested /meta/vfs and /meta/inode -- sometimes I want
> to look up a file by name, and sometimes by inode, but either way, I
> should be able to get its metadata.

You should /never/ give access just by inode, as that makes it very easy to
bypass security given by directory permissions.

> > I mean, editing something is easy and you don't have to "know" how
> > to navigate /meta

> But then you have to "know" how to navigate .meta, and more
> importantly, how to find .meta

And what the heck the format of the metadata is today/for this particular
file.

[...]

> Either way, the major challenge to this method is not so much whether
> it'd work, but how to choose a suitable delimiter?  The delimiter must
> be standard if we're going to have apps develop for it, and it must
> not be used in the name of any existing file or directory.

The only characters that aren't used in filenames today are '/' and '\0'.

> I had a long essay here on how '.meta' breaks every recursive
> operation on directories (rm -rf, tar, mkisofs), but I deleted it when
> I remembered that I played around with the alpha/beta reiser4 which
> had a 'metas' dir that did the same thing, but was hidden from the
> normal directory listing.  'cd metas' worked, 'ls metas' worked, but
> 'ls' showed no directory called 'metas'.

And if I try to create a file or directory called metas?

> I don't *think* there are any apps that will break if you tell them to
> open a path that doesn't exist in a directory listing.  That is,
> typing 'vim /home/metas/acl' should always let me edit the acl on
> /home, and vim should never complain about how /home/metas doesn't
> show up in /home.  A program would have to be pretty retarded to fail
> on something like that.

Think a GUI that reads the directory to find out what is available and
present them as icons for mousing around. Then there is no way to futz
around with your metadata.

Think filename completion, bash style. No, not just bash uses this.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Neil Brown]

On Monday July 11, ninja@slaphack.com wrote:
> Stefan Smietanowski wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> > 
> > 
> >>Ok, still haven't heard much discussion of metafs vs file-as-directory,
> >>but it seems like it'd be easier in metafs.
> > 
> > 
> > Why not implement it inside the directory containg the file ?
> > 
> > Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo
> > 
> > This should be suit both camps I'd think?
> 
> You still need to figure out the parent of "foo", which isn't 
> necessarily easy, especially considering that even if we store a link to 
> the parent, it will be inside the metadata.  Then you have a 
> chicken-and-egg situation.
> 
> Both camps seem to want to allow easy access to the metadata of a file, 
> whether we're given that file as an inode or as a pathname.  That's why 
> I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a 
> file by name, and sometimes by inode, but either way, I should be able 
> to get its metadata.

Well, it's not really 'as an inode or as a pathname'.  It is 'as an
open file descriptor or as a path name' which is an important
difference.

Maybe it is worth repeating Al Viro's suggestion at this point.  I
don't have a reference but the idea was basically that if you open
"/foo" and get filedescriptor N, then
   /proc/self/fds/N-meta
is a directory which contains all the meta stuff for "/foo".
Then it is trivial to get the 'meta' stuff given a filedescriptor and
if you have a pathname, you can always get yourself a filedescriptor.

Then to allow
    cat /home/friend/foo/.meta.../perms
you write a little .so which redefines open, stat, and a few others to
scan for ".meta..." in the pathname and to the necessary magic, and
    export LD_PRELOAD=/that/.so

Finally you write a killer app which fundamentally relies on this
functionality, get everyone addicted to it, and *then* (and only then)
start trying to get this functionality into the kernel.

i.e. prototype new semantics in userspace.

NeilBrown

Re: reiser4 plugins

[Hans Reiser]

Neil Brown wrote:

>
>
>Maybe it is worth repeating Al Viro's suggestion at this point.  I
>don't have a reference but the idea was basically that if you open
>"/foo" and get filedescriptor N, then
>   /proc/self/fds/N-meta
>is a directory which contains all the meta stuff for "/foo".
>Then it is trivial to get the 'meta' stuff given a filedescriptor and
>if you have a pathname, you can always get yourself a filedescriptor.
>  
>
This sound like it might be cute, but filedescriptors are too heavy
weight for stat data accesses in quantity.

In general, the whole file handle paradigm is too heavy for lightweight
files.

Hans

Re: reiser4 plugins

[Neil Brown]

On Tuesday July 12, reiser@namesys.com wrote:
> Neil Brown wrote:
> 
> >
> >
> >Maybe it is worth repeating Al Viro's suggestion at this point.  I
> >don't have a reference but the idea was basically that if you open
> >"/foo" and get filedescriptor N, then
> >   /proc/self/fds/N-meta
> >is a directory which contains all the meta stuff for "/foo".
> >Then it is trivial to get the 'meta' stuff given a filedescriptor and
> >if you have a pathname, you can always get yourself a filedescriptor.
> >  
> >
> This sound like it might be cute, but filedescriptors are too heavy
> weight for stat data accesses in quantity.
> 
> In general, the whole file handle paradigm is too heavy for lightweight
> files.

That may well be true, but is completely orthogonal to filesystem name
semantics. 

If you find file descriptors too slow, come up with an alternate (I
suspect you have in the reiser4 syscall, but I haven't looked at
that yet), implement it in the VFS, and show the world benchmarks of
real-world applications that go faster with this new interface.

I doubt that you would then have a great deal of trouble in getting
the interface accepted (some trouble of course as you will need to
convince a few people, but numbers speak quite loudly).

I suspect that there might need to be a new internal interface into
filesystems, and filesystems which don't provide that will not get the
same speed benefit, but that is perfectly acceptable.

NeilBrown

Re: reiser4 plugins

[Hans Reiser]

Neil Brown wrote:

>On Tuesday July 12, reiser@namesys.com wrote:
>  
>
>>Neil Brown wrote:
>>
>>    
>>
>>>Maybe it is worth repeating Al Viro's suggestion at this point.  I
>>>don't have a reference but the idea was basically that if you open
>>>"/foo" and get filedescriptor N, then
>>>  /proc/self/fds/N-meta
>>>is a directory which contains all the meta stuff for "/foo".
>>>Then it is trivial to get the 'meta' stuff given a filedescriptor and
>>>if you have a pathname, you can always get yourself a filedescriptor.
>>> 
>>>
>>>      
>>>
>>This sound like it might be cute, but filedescriptors are too heavy
>>weight for stat data accesses in quantity.
>>
>>In general, the whole file handle paradigm is too heavy for lightweight
>>files.
>>    
>>
>
>That may well be true, but is completely orthogonal to filesystem name
>semantics. 
>
>If you find file descriptors too slow, come up with an alternate (I
>suspect you have in the reiser4 syscall, but I haven't looked at
>that yet), implement it in the VFS, and show the world benchmarks of
>real-world applications that go faster with this new interface.
>
>I doubt that you would then have a great deal of trouble in getting
>the interface accepted (some trouble of course as you will need to
>convince a few people, but numbers speak quite loudly).
>
>I suspect that there might need to be a new internal interface into
>filesystems, and filesystems which don't provide that will not get the
>same speed benefit, but that is perfectly acceptable.
>
>NeilBrown
>
>
>  
>
We need time, and then we can demonstrate sys_reiser4, it is not ready
for showing yet.....

Hans

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Neil Brown wrote:
> On Monday July 11, ninja@slaphack.com wrote:
> 
>>Stefan Smietanowski wrote:
>>
>>>-----BEGIN PGP SIGNED MESSAGE-----
>>>Hash: SHA1
>>>
>>>
>>>
>>>>Ok, still haven't heard much discussion of metafs vs file-as-directory,
>>>>but it seems like it'd be easier in metafs.
>>>
>>>
>>>Why not implement it inside the directory containg the file ?
>>>
>>>Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo
>>>
>>>This should be suit both camps I'd think?
>>
>>You still need to figure out the parent of "foo", which isn't 
>>necessarily easy, especially considering that even if we store a link to 
>>the parent, it will be inside the metadata.  Then you have a 
>>chicken-and-egg situation.
>>
>>Both camps seem to want to allow easy access to the metadata of a file, 
>>whether we're given that file as an inode or as a pathname.  That's why 
>>I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a 
>>file by name, and sometimes by inode, but either way, I should be able 
>>to get its metadata.
> 
> 
> Well, it's not really 'as an inode or as a pathname'.  It is 'as an
> open file descriptor or as a path name' which is an important
> difference.
> 
> Maybe it is worth repeating Al Viro's suggestion at this point.  I
> don't have a reference but the idea was basically that if you open
> "/foo" and get filedescriptor N, then
>    /proc/self/fds/N-meta

How am I supposed to get there with a shell script?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQtROqXgHNmZLgCUhAQLRcg/+I9PWSmFXRwKtj7pnEeMXOCjiTo6GQE3O
61fjH3f6aL9Ydkip4eXum3S14cioiU9bzC11GA5kRIM+W1DKcYex1dIpivrtF9Ht
Rvozn9x2TP5tacDmSfqRJXvAB+xTRtZOu+M/rDjXdLsriDJDA0AdyDH8Yo/8WMbU
6i1DWzLTO0vHS3kEb/8oqgBj7sQ63sS/4KVszBx6+bN0KOikXbORDu6efKjC9w21
3DZPnBG0B03smhdCygd0j0Zmh0JEaZHfuFgNK1ZmRzipbvvUBDtdKY5MJ6f4pHnA
GBO8ybsXp9qxNQr6DNenF/wbAT6n3dMyE/AWuql+qx3iumSwx/Prh7xDAhBZBMXp
Oin7hOa1i583cdju4ErSBPaciRzumGluY6gbFvVA8Yva+IjPxxjPtfLwalK11cH1
k4oQO5Par1W0TmMOpc0PQ/bEeEHHxUcn1ToeJa4NYJWIiBe+UHMb/AyI4hKjSIkt
Kp0wrCPBjRfuBCHXXL89bWZoSeSFkN8fAyOxBV928naxxr8e+vCPUX1/H3ca7UsB
Nxg0Vzt4V9tz4xCw4QAy810Uya8/HSm3aVhqEzjHKBoKboHrMVDJvxRQxfkqQcnC
4jIFYPBdHgGw7OONyhfbgTPLIm1OCNPpcRkS4aidHqg0DkDU50h6zFQkhG5Xwh5Z
x5REgxbqD+A=
=FGm4
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Neil Brown]

On Tuesday July 12, ninja@slaphack.com wrote:
> > 
> > Maybe it is worth repeating Al Viro's suggestion at this point.  I
> > don't have a reference but the idea was basically that if you open
> > "/foo" and get filedescriptor N, then
> >    /proc/self/fds/N-meta
> 
> How am I supposed to get there with a shell script?


function get_meta() 
{
   var=$1
   file=$2
   meta=$3
   val=`cat /proc/self/fd/3-meta/$meta < $file`
   eval var=\$val
}

then
   get_meta varname /home/foo/bar username

will read the 'username' meta-file of 'home/foo/bar' and place it in
varname.

Is that what you wanted?

NeilBrown

Re: reiser4 plugins

[Hans Reiser]

I got it slightly wrong.

One can have hardlinks to a directory without cycles provided that one
does not have hardlinks from the children of that directory to any file
not a child of that directory.  (Mountpoints currently implement that
restriction.)

Question: can one implement that lesser restriction above with elegant
code?  Is the greater restriction below easier to code?  (If no to the
first and yes to the second is correct, then I can accept the greater
restriction described below.)

One can have hardlinks to directories
without cycles provided that one does not allow any child of the
directory to have a hardlink.

Hans

Re: reiser4 plugins

[Alexander G. M. Smith]

Hans Reiser wrote on Tue, 05 Jul 2005 16:56:02 -0700:
> One can have hardlinks to a directory without cycles provided that one
> does not have hardlinks from the children of that directory to any file
> not a child of that directory.  (Mountpoints currently implement that
> restriction.)
> 
> Question: can one implement that lesser restriction above with elegant
> code?  Is the greater restriction below easier to code?  (If no to the
> first and yes to the second is correct, then I can accept the greater
> restriction described below.)
> 
> One can have hardlinks to directories
> without cycles provided that one does not allow any child of the
> directory to have a hardlink.

That sounds equivalent to no hard links (other than the usual parent
directory one).  If there's any directory with two links to it, then
there will be a cycle somewhere!

The tradeoff is the size of the cycle to the amount of memory and other
resources needed to clean up the cycle when rename/deletion changes the
directory graph structure.  Various artificial limits can be imposed,
such as not creating a link which would result in a cycle of more than
some number of files/directories (adjust so that cleanup fits in the
memory size of the machine).  Or just fail the cleanup if it is too
complex (user deletes individual items in the cycle then tries again).

Perhaps you are thinking about the speed improvement for cycle checking
that comes from marking directories as known to not contain children
pointing outside the graph rooted at that directory.  Then that directory
can be treated as an ordinary directory in the cycle cleanup procedure,
avoiding the need to trace down the links to all its descendants.  That's
essentially what your mount point restriction is.

- Alex

Re: reiser4 plugins

[Hubert Chan]

On Tue, 05 Jul 2005 20:50:08 -0400 EDT, "Alexander G. M. Smith" <agmsmith@rogers.com> said:

> That sounds equivalent to no hard links (other than the usual parent
> directory one).  If there's any directory with two links to it, then
> there will be a cycle somewhere!

What we want is no directed cycles.  That is A is the parent of B is the
parent of C is the parent of A.  We don't care about A is the parent of
B is the parent of C; A is the parent of B is the parent of C.

OK, here's a random idea that just popped into my head, and to which
I've given little thought (read: none whatsoever), and may be the
stupidest idea ever proposed on LKML, but thought I would just toss it
out to see if it could stimulate someone to come up with something
better (read: sane):  Conceptually, foo/.... is just a symlink to
/meta/[filesystem]/[inode of foo].

And a question: is it feasible to store, for each inode, its parent(s),
instead of just the hard link count?

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Hans Reiser]

Hubert Chan wrote:

>On Tue, 05 Jul 2005 20:50:08 -0400 EDT, "Alexander G. M. Smith" <agmsmith@rogers.com> said:
>
>  
>
>>That sounds equivalent to no hard links (other than the usual parent
>>directory one).  If there's any directory with two links to it, then
>>there will be a cycle somewhere!
>>    
>>
>
>What we want is no directed cycles.  That is A is the parent of B is the
>parent of C is the parent of A.  We don't care about A is the parent of
>B is the parent of C; A is the parent of B is the parent of C.
>
>OK, here's a random idea that just popped into my head, and to which
>I've given little thought (read: none whatsoever), and may be the
>stupidest idea ever proposed on LKML, but thought I would just toss it
>out to see if it could stimulate someone to come up with something
>better (read: sane):  Conceptually, foo/.... is just a symlink to
>/meta/[filesystem]/[inode of foo].
>  
>
Except that we want the metafiles to go away when the base file goes away.

>And a question: is it feasible to store, for each inode, its parent(s),
>instead of just the hard link count?
>  
>
Ooh, now that is an interesting old idea I haven't considered in 20
years.... makes fsck more robust too....

Re: reiser4 plugins

[David Masover]

Hans Reiser wrote:
> Hubert Chan wrote:
> 
> 
>>On Tue, 05 Jul 2005 20:50:08 -0400 EDT, "Alexander G. M. Smith" <agmsmith@rogers.com> said:
>>
>> 
>>
>>
>>>That sounds equivalent to no hard links (other than the usual parent
>>>directory one).  If there's any directory with two links to it, then
>>>there will be a cycle somewhere!
>>>   
>>>
>>
>>What we want is no directed cycles.  That is A is the parent of B is the
>>parent of C is the parent of A.  We don't care about A is the parent of
>>B is the parent of C; A is the parent of B is the parent of C.
>>
>>OK, here's a random idea that just popped into my head, and to which
>>I've given little thought (read: none whatsoever), and may be the
>>stupidest idea ever proposed on LKML, but thought I would just toss it
>>out to see if it could stimulate someone to come up with something
>>better (read: sane):  Conceptually, foo/.... is just a symlink to
>>/meta/[filesystem]/[inode of foo].
>> 
>>
> 
> Except that we want the metafiles to go away when the base file goes away.

Only, /meta is a filesystem that already makes stuff go away for us, so 
all we have left is the issue of whether using /meta costs us 
performance, or whether breaking POSIX to add a symlink (such as 
foo/...) really gives us that much more usability.

I don't know the first thing about whether it costs us performance, 
although it seems like it could be negligable considering the existance 
of mount --bind.

I don't think file-as-dir gives us that much more usability, because we 
can always create a simple program or shell script that 'cd's us into 
metadata.  It's still easier than having a simple program that 
manipulates the metadata directly, because this way we can do 'cd' and 
'ls' and so on inside the metadata directory.

And, once we start talking about applications, /meta will be more 
readily supported (as in, some apps will go through a pathname and stop 
when they get to a file, and then there's tar).  On apps which don't 
have direct support for /meta, you'd be navigating to the file in 
question and then manually typing '...' into the dialog, so I don't see 
why typing '...' at the end is better than typing '/meta' or '/meta/vfs' 
at the beginning.

That said, I'm still not entirely sure how we get /meta/vfs to work 
other than adding a '...' sort of delimiter anyway.

>>And a question: is it feasible to store, for each inode, its parent(s),
>>instead of just the hard link count?
>> 
>>
> 
> Ooh, now that is an interesting old idea I haven't considered in 20
> years.... makes fsck more robust too....

Doesn't it make directory operations slower, too?

And, will it require a format change?

Re: reiser4 plugins

[Adrian Ulrich]

> so all we have left is the issue of whether using /meta costs us 
> performance, or whether breaking POSIX to add a symlink (such as 
> foo/...) really gives us that much more usability.

IMHO '/meta' isn't such a good idea, because a chrooted application
won't be able to use it.

Re: reiser4 plugins

[David Masover]

Adrian Ulrich wrote:
>>so all we have left is the issue of whether using /meta costs us 
>>performance, or whether breaking POSIX to add a symlink (such as 
>>foo/...) really gives us that much more usability.
> 
> 
> IMHO '/meta' isn't such a good idea, because a chrooted application
> won't be able to use it.

mount --bind.  Is there a performance hit for having too many of those?

mount --bind /meta/vfs/some/chroot /some/chroot/meta

Also, maybe a separately-mounted /meta, maybe with something like
'-o root='

I can't think of when you'd have a chrooted application that uses /meta, 
but wasn't written with /meta in mind, so as to have these mount 
commands in its init scripts.

Re: reiser4 plugins

[Adrian Ulrich]

> mount --bind /meta/vfs/some/chroot /some/chroot/meta

This maybe funny if you got 1-2 chrooted applications.
But it will be a nightmare if you got 20-30 chrooted applications.



-- 
 We're working on it, slowly but surely...or not-so-surely in the spots
we're not so sure... -- Larry Wall

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

> And, once we start talking about applications, /meta will be more
> readily supported (as in, some apps will go through a pathname and
> stop when they get to a file, and then there's tar).  On apps which
> don't have direct support for /meta, you'd be navigating to the file
> in question and then manually typing '...' into the dialog, so I don't
> see why typing '...' at the end is better than typing '/meta' or
> '/meta/vfs' at the beginning.

Performance.  If you type it at the end, and you already have done the
lookup of the filename, then you can go from the file to one of its
methods, instead of a complete new traversal of another tree under /meta

>
> That said, I'm still not entirely sure how we get /meta/vfs to work
> other than adding a '...' sort of delimiter anyway.
>
>>> And a question: is it feasible to store, for each inode, its parent(s),
>>> instead of just the hard link count?
>>>
>>>
>>
>> Ooh, now that is an interesting old idea I haven't considered in 20
>> years.... makes fsck more robust too....
>
>
> Doesn't it make directory operations slower, too?

Not sure.  It consumes space though.

>
> And, will it require a format change?
>
Yes, but we have plugins now, so.....

Re: reiser4 plugins

[David Masover]

Hans Reiser wrote:
> David Masover wrote:
> 
> 
>>And, once we start talking about applications, /meta will be more
>>readily supported (as in, some apps will go through a pathname and
>>stop when they get to a file, and then there's tar).  On apps which
>>don't have direct support for /meta, you'd be navigating to the file
>>in question and then manually typing '...' into the dialog, so I don't
>>see why typing '...' at the end is better than typing '/meta' or
>>'/meta/vfs' at the beginning.
> 
> 
> Performance.  If you type it at the end, and you already have done the
> lookup of the filename, then you can go from the file to one of its
> methods, instead of a complete new traversal of another tree under /meta

Only, it's a different view of the same tree.  That may not matter 
performance-wise, though.

Also, for performance-critical applications, the /meta tree is pretty 
easy -- it becomes more like /meta/inode/some_inode_number/.

There's also the chroot issue, though.

>>That said, I'm still not entirely sure how we get /meta/vfs to work
>>other than adding a '...' sort of delimiter anyway.
>>
>>
>>>>And a question: is it feasible to store, for each inode, its parent(s),
>>>>instead of just the hard link count?
>>>>
>>>>
>>>
>>>Ooh, now that is an interesting old idea I haven't considered in 20
>>>years.... makes fsck more robust too....
>>
>>
>>Doesn't it make directory operations slower, too?
> 
> 
> Not sure.  It consumes space though.
> 
> 
>>And, will it require a format change?
>>
> 
> Yes, but we have plugins now, so.....

So, will the format change happen at mount time?  Will it need a special 
mount flag?  Will I need to use debugfs or some other offline tool?

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

> So, will the format change happen at mount time?  Will it need a
> special mount flag?  Will I need to use debugfs or some other offline
> tool?


First we make sure we have the right answer.  Have we solved the cycle
problem?  Can we run out of memory as Horst/Nikita suggest?

Re: reiser4 plugins

[Jonathan Briggs]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Tue, 2005-07-05 at 23:44 -0700, Hans Reiser wrote:
> Hubert Chan wrote:
> >And a question: is it feasible to store, for each inode, its parent(s),
> >instead of just the hard link count?
> >  
> >
> Ooh, now that is an interesting old idea I haven't considered in 20
> years.... makes fsck more robust too....

Hey, sounds like the idea I proposed a couple months ago of storing the
path names in each file, instead of in directories.  Only better, since
each path component isn't text but a link instead.

It still has the performance and locking problem of having to update
every child file when moving a directory tree to a new parent.  On the
other hand, maybe the benefit is worth the cost.
-- 
Jonathan Briggs <jbriggs@esoft.com>
eSoft, Inc.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Hubert Chan]

On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs <jbriggs@esoft.com> said:

> On Tue, 2005-07-05 at 23:44 -0700, Hans Reiser wrote:
>> Hubert Chan wrote:
>>> And a question: is it feasible to store, for each
>>> inode, its parent(s), instead of just the hard link count?

>> Ooh, now that is an interesting old idea I haven't considered in 20
>> years.... makes fsck more robust too....

If you can store the parents, then finding cycles (relatively) quickly
is pretty easy: before you try to make A the parent of B, walk up the
parent pointers starting from A.  If you ever reach B, you have a cycle.
If not, you don't have a cycle.  (Hmmm.  Do I need a proof of
correctness for this?  It's just depth-first-search, which everybody
learned in their first algorithms course.)

Running time is (at most) just the sum of all (directed) path lengths
from A to the root.  Assuming people don't have too deep of a hierarchy,
or too much branching with hard links (i.e. nodes with too many
parents), which I think is a reasonable assumption, running time should
be fairly quick.

(People may have millions of files in a single directory, which is why
doing the DFS in the forward direction is a bad idea.  But people
probably don't have millions of hard links to the same file, or
hierarchies that are millions of levels deep.)

And it only happens when a hard link is created -- probably not a very
performance-dependent event (i.e. it's not the scheduling algorithm).  I
would imagine that some creative caching could be done to speed up the
situation where you try to make a whole batch of hard links.  (When you
walk up the parent pointers starting from A, cache the inode numbers
that you encounter.  If you then try to make A the parent of C, check if
C is one of the inode numbers in the cache.  If yes, we have a cycle; if
no, we don't have a cycle.)

Extra disk storage is just about one extra word for most nodes (that
only have one parent -- basically, one extra word per parent), if you
can pack the data efficiently.  (i.e. we don't want to waste a extra
whole block per node.)

Of course, since this requires extra storage on the part of the
filesystem, the (kernel) VFS change would have to be something where the
VFS asks the filesystem whether making A the parent of B will create a
cycle; it's not something the VFS can do transparently for everybody.
Filesystems that don't store this extra stuff just return "yes" if B is
a directory (thus disallowing the link), and "no" if B is a file, and we
get the same situation we have right now.

It would require a disk format change (sort of) in Reiser4, but (as Hans
mentioned) since we have plugins, shouldn't require a reformat (as I
understand it).  Just run a tool that builds the parent-pointer data;
until you run that tool, the filesystem can just disallow hard links.
After you run the tool, updating parent pointers is handled
automatically by the filesystem.

> Hey, sounds like the idea I proposed a couple months ago of storing
> the path names in each file, instead of in directories.  Only better,
> since each path component isn't text but a link instead.

> It still has the performance and locking problem of having to update
> every child file when moving a directory tree to a new parent.  On the
> other hand, maybe the benefit is worth the cost.

Every node should store the inode number(s) for its parent(s).  Not the
path name.  So you don't need to do any updates, since when you move a
tree, inode numbers don't change.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Jonathan Briggs]

[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

On Wed, 2005-07-06 at 15:51 -0400, Hubert Chan wrote:
> On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs <jbriggs@esoft.com> said:
[snip]
> > It still has the performance and locking problem of having to update
> > every child file when moving a directory tree to a new parent.  On the
> > other hand, maybe the benefit is worth the cost.
> 
> Every node should store the inode number(s) for its parent(s).  Not the
> path name.  So you don't need to do any updates, since when you move a
> tree, inode numbers don't change.

You do need the updates if you change what inode is the parent.

/a/b/c, /a/b/d, /a/b/e, /b
mv /a/b /b
Now you have to change the stored grand-parent inodes for /a/b/c, /a/b/d
and /a/b/e so that they reference /b/b instead of /a/b.
-- 
Jonathan Briggs <jbriggs@esoft.com>
eSoft, Inc.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[David Masover]

Jonathan Briggs wrote:
> On Wed, 2005-07-06 at 15:51 -0400, Hubert Chan wrote:
> 
>>On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs <jbriggs@esoft.com> said:
> 
> [snip]
> 
>>>It still has the performance and locking problem of having to update
>>>every child file when moving a directory tree to a new parent.  On the
>>>other hand, maybe the benefit is worth the cost.
>>
>>Every node should store the inode number(s) for its parent(s).  Not the
>>path name.  So you don't need to do any updates, since when you move a
>>tree, inode numbers don't change.
> 
> 
> You do need the updates if you change what inode is the parent.
> 
> /a/b/c, /a/b/d, /a/b/e, /b
> mv /a/b /b
> Now you have to change the stored grand-parent inodes for /a/b/c, /a/b/d
> and /a/b/e so that they reference /b/b instead of /a/b.

no, c, d, and e all would reference /a/b's *inode*, not *pathname*. 
Then /a/b would reference /a.  So with the above mv, you just change the 
reference in /a/b (now /b/b) to point to /b as parent.

Only way you would have to touch c, d, and e is if you did:

mkdir /b/b
mv /a/b/* /b/b/

but, since you're also essentially unlinking them from /a/b and linking 
them into /b, it's not that much more of a performance hit to update one 
pointer in the file.

Re: reiser4 plugins

[Hubert Chan]

On Wed, 06 Jul 2005 14:33:18 -0600, Jonathan Briggs <jbriggs@esoft.com> said:

> On Wed, 2005-07-06 at 15:51 -0400, Hubert Chan wrote:
>> On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs
>> <jbriggs@esoft.com> said:
> [snip]
>>> It still has the performance and locking problem of having to
>>> update every child file when moving a directory tree to a new
>>> parent.  On the other hand, maybe the benefit is worth the cost.

>> Every node should store the inode number(s) for its parent(s).  Not
>> the path name.  So you don't need to do any updates, since when you
>> move a tree, inode numbers don't change.

> You do need the updates if you change what inode is the parent.

> /a/b/c, /a/b/d, /a/b/e, /b
> mv /a/b /b
> Now you have to change the stored grand-parent inodes for /a/b/c,
> /a/b/d and /a/b/e so that they reference /b/b instead of /a/b.

Don't store (great)*grand-parent pointers; only parent pointers.  c
points to b, b points to a, a points to root.  c doesn't need to know
about /a or root directly.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Horst von Brand]

Hubert Chan <hubert@uhoreg.ca> wrote:
> On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs <jbriggs@esoft.com> said:
> > On Tue, 2005-07-05 at 23:44 -0700, Hans Reiser wrote:
> >> Hubert Chan wrote:
> >>> And a question: is it feasible to store, for each
> >>> inode, its parent(s), instead of just the hard link count?

> >> Ooh, now that is an interesting old idea I haven't considered in 20
> >> years.... makes fsck more robust too....

> If you can store the parents, then finding cycles (relatively) quickly
> is pretty easy: before you try to make A the parent of B, walk up the
> parent pointers starting from A.  If you ever reach B, you have a cycle.
> If not, you don't have a cycle.  (Hmmm.  Do I need a proof of
> correctness for this?  It's just depth-first-search, which everybody
> learned in their first algorithms course.)

Correct. And you need space for potentially a huge lot of up pointers for
each file. And then there is the (very minor) problem is that meanwhile
/nothing/ can touch the filesystem to do any change...

How do you delete such an object? You will have to delete from each child
down to the first object that has a pointer to it from the outside, if you
don't want garbage left behind. And that means walking down to /each/
reachable object, then walking up from there to see if all its parents are
in the DAG rooted at what you are going to delete. This means potentially
walking through the whole filesystem (if you want to delete the root, or
something near). You will run out of memory, and again, meanwhile no
changes can be allowed.

It is for a reason that Unix filesystems don't have up pointers for files,
and just leaves (i.e., files) can have multiple hard links.

And this /was/ explained in detail the last flamefest over this exact same
point. I thought the ReiserFS people had learned from that...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Hubert Chan]

On Wed, 06 Jul 2005 16:33:23 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:

> Hubert Chan <hubert@uhoreg.ca> wrote:
>> If you can store the parents, then finding cycles (relatively)
>> quickly is pretty easy: before you try to make A the parent of B,
>> walk up the parent pointers starting from A.  If you ever reach B,
>> you have a cycle.  If not, you don't have a cycle.  (Hmmm.  Do I need
>> a proof of correctness for this?  It's just depth-first-search, which
>> everybody learned in their first algorithms course.)

> Correct. And you need space for potentially a huge lot of up pointers
> for each file.

People (that I know of) don't normally have a huge lot of hardlinks to a
single file.

> And then there is the (very minor) problem is that meanwhile /nothing/
> can touch the filesystem to do any change...

If the DFS is quick, it shouldn't make much difference.

Lock nodes as you walk up the tree, and people can change unlocked nodes
without harming anything.  All you need to do is make sure that no up
pointers get added to nodes that you've already looked at.

People can also still edit files without changing the tree (and hence
not screwing up the DFS).

> How do you delete such an object? You will have to delete from each
> child down to the first object that has a pointer to it from the
> outside, if you don't want garbage left behind. And that means walking
> down to /each/ reachable object, then walking up from there to see if
> all its parents are in the DAG rooted at what you are going to
> delete. This means potentially walking through the whole filesystem
> (if you want to delete the root, or something near). You will run out
> of memory, and again, meanwhile no changes can be allowed.

What does deleting have to do with up pointers?  If you delete a
directory that has refcount greater than 1, you decrease the refcount,
remove the appropriate up pointer, and remove the inode entry from its
parent.  If you delete a directory that has refcount equal to 1, well,
we're in the same situation we're in right now when you delete a
directory; you need to "rm -r", which doesn't lock up the whole
filesystem.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Masover]

Hubert Chan wrote:
> On Wed, 06 Jul 2005 16:33:23 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:
> 
> 
>>Hubert Chan <hubert@uhoreg.ca> wrote:
>>
>>>If you can store the parents, then finding cycles (relatively)
>>>quickly is pretty easy: before you try to make A the parent of B,
>>>walk up the parent pointers starting from A.  If you ever reach B,
>>>you have a cycle.  If not, you don't have a cycle.  (Hmmm.  Do I need
>>>a proof of correctness for this?  It's just depth-first-search, which
>>>everybody learned in their first algorithms course.)
> 
> 
>>Correct. And you need space for potentially a huge lot of up pointers
>>for each file.
> 
> 
> People (that I know of) don't normally have a huge lot of hardlinks to a
> single file.

And speaking of which, the only doomsday scenario (running out of RAM) 
that I can think of with this scheme is if we have a ton of hardlinks to 
the same file and we try to move one of them.  But this scales linearly 
with the number of hardlinks, I think.  Maybe not quite, but certainly 
not exponentially.

The only other doomsday scenario is if we have a ludicrously deep tree.

To make this work in real usage, not DOS testing, we really need both of 
those, and even then I'm not sure it can work.  What's the maximum 
number of hardlinks supported to a single file?  What's the maximum tree 
depth?  Can these be limited to prevent actual DOS attempts?

Now, if we were to ever actually *create* a cycle, then yes, we might 
end up traversing the whole tree to delete it, possibly running out of 
RAM.  But we don't ever actually create cycles except if we were to have 
corruption, which could as easily create cycles in any other FS.

Re: reiser4 plugins

[Jan Harkes]

On Wed, Jul 06, 2005 at 09:33:13PM -0500, David Masover wrote:
> And speaking of which, the only doomsday scenario (running out of RAM) 
> that I can think of with this scheme is if we have a ton of hardlinks to 
> the same file and we try to move one of them.  But this scales linearly 
> with the number of hardlinks, I think.  Maybe not quite, but certainly 
> not exponentially.
> 
> The only other doomsday scenario is if we have a ludicrously deep tree.

rename(a/b, c/b), if a and c are identical.

End result, either you deadlock trying to lock the same object twice, or
you end up removing b since the target of a rename is unlinked.

The VFS uses dentries, there is one per hardlinked object, and they have
a single parent only. So a/b and c/b are represented by the same inode,
but they have completely different dentry-aliases associated with them. 

Similar things for removal, we can safely remove a file, but not a
directory that still has children, if you have 'meta-files' hanging of a
file, you'd have to get rid of the metadata objects first.

If you want to use a file as a directory, it probably will need the same
restrictions as a directory if you expect the dcache and VFS locking to
work correctly. So that means, _no hardlinks to files_ (the file system
could internally implement copy-on-write type links, or use a content
addressable storage to deal with diskspace issues) and the file system
probably has to d_unhash/destroy metadata objects before it can unlink
the file object, etc.

> To make this work in real usage, not DOS testing, we really need both of 
> those, and even then I'm not sure it can work.  What's the maximum 
> number of hardlinks supported to a single file?

I believe nlink is a 16-bit value, so that would be either 32K or 64K
depending on signedness.

> What's the maximum tree depth?

Last time I checked, PATH_MAX is 4096, so that would be about 2048
single character directory names.

> Can these be limited to prevent actual DOS attempts?

Is that the 'nobody needs more than 64KB of memory' kind of DOS?

Jan

Re: reiser4 plugins

[Hans Reiser]

Jonathan Briggs wrote:

>On Tue, 2005-07-05 at 23:44 -0700, Hans Reiser wrote:
>  
>
>>Hubert Chan wrote:
>>    
>>
>>>And a question: is it feasible to store, for each inode, its parent(s),
>>>instead of just the hard link count?
>>> 
>>>
>>>      
>>>
>>Ooh, now that is an interesting old idea I haven't considered in 20
>>years.... makes fsck more robust too....
>>    
>>
>
>Hey, sounds like the idea I proposed a couple months ago of storing the
>path names in each file, instead of in directories.  Only better, since
>each path component isn't text but a link instead.
>
>It still has the performance and locking problem of having to update
>every child file when moving a directory tree to a new parent.  On the
>other hand, maybe the benefit is worth the cost.
>  
>
Oh no, don't store the whole path, store just the parent list.  This
will make fsck more robust in the event that the directory gets
clobbered by hardware error.

I don't think it affects the cost of detecting cycles though, I think it
only makes fsck more robust.

Hans

Re: reiser4 plugins

[Hubert Chan]

On Wed, 06 Jul 2005 23:42:50 -0700, Hans Reiser <reiser@namesys.com> said:

> Oh no, don't store the whole path, store just the parent list.  This
> will make fsck more robust in the event that the directory gets
> clobbered by hardware error.

> I don't think it affects the cost of detecting cycles though, I think
> it only makes fsck more robust.

It doesn't affect the worst-case cost of detecting cycles; by necessity,
any (static [1]) directed cycle detection algorithm must take O(n) time,
n being the size of the tree (nodes + links).  However, under
"reasonable assumptions" (where the reasonableness of those assumptions
may be questioned, but I think they're reasonable), I think that doing a
depth-first-search through the parent links makes the algorithm
not-too-terrible.  Namely, the "reasonable assumptions" are that a node
doesn't have "too many" parents, and the filesystem hierarchy is not
"too deep".

[1] BTW, I had also previously looked at online/dynamic algorithms, for
those who are familiar with that area.  The best known so far is still
O(n) worst case, but much, much smaller in "most cases".

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Masover]

Hubert Chan wrote:
> On Wed, 06 Jul 2005 23:42:50 -0700, Hans Reiser <reiser@namesys.com> said:
> 
> 
>>Oh no, don't store the whole path, store just the parent list.  This
>>will make fsck more robust in the event that the directory gets
>>clobbered by hardware error.
> 
> 
>>I don't think it affects the cost of detecting cycles though, I think
>>it only makes fsck more robust.
> 
> 
> It doesn't affect the worst-case cost of detecting cycles; by necessity,
> any (static [1]) directed cycle detection algorithm must take O(n) time,
> n being the size of the tree (nodes + links).  However, under
> "reasonable assumptions" (where the reasonableness of those assumptions
> may be questioned, but I think they're reasonable), I think that doing a
> depth-first-search through the parent links makes the algorithm
> not-too-terrible.  Namely, the "reasonable assumptions" are that a node
> doesn't have "too many" parents, and the filesystem hierarchy is not
> "too deep".

And, remember, there are other things affected by depth of the tree 
anyway.  For that matter, most of the time, when you push a system past 
"reasonable assumptions", you hit performance issues, if not stability 
ones.  Most everything but Reiser assumes that you won't have "too many" 
files in a directory, or "too many" small files in the FS as a whole.

I think it's fair to assume people will keep things "reasonable", 
especially if we tell them what "reasonable" means.  As in, "This is the 
kind of organization which Reiser4 handles really well, and this is the 
kind of organization which will absolutely kill your performance."

Re: reiser4 plugins

[Neil Brown]

On Tuesday July 5, reiser@namesys.com wrote:
> I got it slightly wrong.
> 
> One can have hardlinks to a directory without cycles provided that one
> does not have hardlinks from the children of that directory to any file
> not a child of that directory.  (Mountpoints currently implement that
> restriction.)
> 
> Question: can one implement that lesser restriction above with elegant
> code?  Is the greater restriction below easier to code?  (If no to the
> first and yes to the second is correct, then I can accept the greater
> restriction described below.)

<technical-content>
I think the "lesser restriction above" can be implemented elegantly,
but it would require major dcache surgery.

Currently all the dentries of names in a directory are linked to the
dentry of the directory.  As you would have to let a directory have
multiple dentries, it would be best to change that linkage so that the
dentries of names in a directory were linked to the "inode" of that
directory (of which there is still only one).
Thus instead of just having a dentry tree with inodes attached at each
point, you would have a dentry/inode tree with inodes a more integral
part of the tree. (i.e. the path from the root down would be dentry ->
inode -> dentry ->inode -> dentry etc).
This would have major implications for the current code.

The "greater restriction below" should be easy to code providing you
were willing to have two sorts of directories: those which could be
linked (ie. they sometimes look like files) and those which cannot
(they never look like files).  Then for each dentry, you remember the
closest parent which is a can-be-linked directory an make sure a
hard-link will never want to change the can-be-linked directory for
the target.

If you want to be more general, and have only one sort of directory
that just behaves differently in different situations, then it would
be much harder.
You have to make sure both 
 a/ that you never hard link a file that is under a hard-linked
    directory to somewhere outside of that hard-linked directory and
 b/ that you never hard link a directory that contains a file which is
    hard-linked to somewhere outside that directory.

The first is probably quite manageable.  The second is essentially the
cycle-detection problem.

</technical-content>

<humour>
SUN used to advertise:
  "The network is the computer"
However I think we have all come to realise that the network is the
network, and the computer is the computer.

Now Hans wants to tells that
   "The directory is the file"

but I don't find it any more convincing than SUN's message...

</humour>

<opinion>
If you really want to change traditional Unix semantics, I would
suggest you get rid of hard-links.  They really are more trouble than
they are worth, and discarding them makes this whole issue moot.
</opinion>

NeilBrown

Re: reiser4 plugins

[Nate Diller]

Neil Brown wrote:

>On Tuesday July 5, reiser@namesys.com wrote:
>  
>
>>I got it slightly wrong.
>>
>>One can have hardlinks to a directory without cycles provided that one
>>does not have hardlinks from the children of that directory to any file
>>not a child of that directory.  (Mountpoints currently implement that
>>restriction.)
>>
>>Question: can one implement that lesser restriction above with elegant
>>code?  Is the greater restriction below easier to code?  (If no to the
>>first and yes to the second is correct, then I can accept the greater
>>restriction described below.)
>>    
>>
>
><technical-content>
>I think the "lesser restriction above" can be implemented elegantly,
>but it would require major dcache surgery.
>
>Currently all the dentries of names in a directory are linked to the
>dentry of the directory.  As you would have to let a directory have
>multiple dentries, it would be best to change that linkage so that the
>dentries of names in a directory were linked to the "inode" of that
>directory (of which there is still only one).
>Thus instead of just having a dentry tree with inodes attached at each
>point, you would have a dentry/inode tree with inodes a more integral
>part of the tree. (i.e. the path from the root down would be dentry ->
>inode -> dentry ->inode -> dentry etc).
>This would have major implications for the current code.
>
>The "greater restriction below" should be easy to code providing you
>were willing to have two sorts of directories: those which could be
>linked (ie. they sometimes look like files) and those which cannot
>(they never look like files).  Then for each dentry, you remember the
>closest parent which is a can-be-linked directory an make sure a
>hard-link will never want to change the can-be-linked directory for
>the target.
>
>If you want to be more general, and have only one sort of directory
>that just behaves differently in different situations, then it would
>be much harder.
>You have to make sure both 
> a/ that you never hard link a file that is under a hard-linked
>    directory to somewhere outside of that hard-linked directory and
> b/ that you never hard link a directory that contains a file which is
>    hard-linked to somewhere outside that directory.
>
>The first is probably quite manageable.  The second is essentially the
>cycle-detection problem.
>
></technical-content>
>
><humour>
>SUN used to advertise:
>  "The network is the computer"
>However I think we have all come to realise that the network is the
>network, and the computer is the computer.
>
>Now Hans wants to tells that
>   "The directory is the file"
>
>but I don't find it any more convincing than SUN's message...
>
></humour>
>
><opinion>
>If you really want to change traditional Unix semantics, I would
>suggest you get rid of hard-links.  They really are more trouble than
>they are worth, and discarding them makes this whole issue moot.
></opinion>
>  
>
have you read Giampaolo's BeFS design book?  he talks (ch 11) about this 
big argument they had at Be about how their API would let programs store 
references to files.  the macintosh people wanted to have the program 
store an inode number, and the posix guys wanted it to store a path.  in 
my mind, the only reason posix is right on this is because it allows 
hard links AND symlinks as a way of giving a choice in the matter, since 
hard links increment the reference count and softlinks don't.  it seems 
much *more* important to be able to have this choice for file attributes 
than for the traditional posix namespace.

as an example, if a program were to store some things it needs access to 
in its executable's attributes, it should have the option of keeping a 
hard reference to something, so that it can't be deleted out from 
underneath.  this enables sane sharing of resources without ownership 
tracking problems (see windows DLL hell for details).  the attribute 
space should be indistinguishable from the rest of the namespace, and 
should be able to link (soft or hard) anywhere in the FS.  anything less 
is too much work for too little reward.

NATE

>NeilBrown
>
>
>  
>

Re: reiser4 plugins

[Hans Reiser]

Nate Diller wrote:

>
>
> as an example, if a program were to store some things it needs access
> to in its executable's attributes, it should have the option of
> keeping a hard reference to something, so that it can't be deleted out
> from underneath.  this enables sane sharing of resources without
> ownership tracking problems (see windows DLL hell for details).  the
> attribute space should be indistinguishable from the rest of the
> namespace, and should be able to link (soft or hard) anywhere in the
> FS.  anything less is too much work for too little reward.
>
You already have a problem with hardlinks not crossing mount points, but
I understand your point.  If we can write code for solving the cycle
problem cleanly, it would be best.

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:

[...]

> Just don't allow user-created hardlinks inside either metafs (/meta) or
> the magical meta directory inside files.

And what is it useful for, after its advantage was that it was /exactly/
like regular files &c, and now it is severely crippled?
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[David Masover]

Horst von Brand wrote:
> David Masover <ninja@slaphack.com> wrote:
> 
> [...]
> 
> 
>>Just don't allow user-created hardlinks inside either metafs (/meta) or
>>the magical meta directory inside files.
> 
> 
> And what is it useful for, after its advantage was that it was /exactly/
> like regular files &c, and now it is severely crippled?

The advantage was never that meta files look exactly like regular files, 
but that they look enough like regular files that you can use existing 
tools on them.  Of course, there are some times when you want meta files 
to look exactly like regular files, as in (I hate to bring it up again, 
but...) zip files.

So, a zip file (/meta/vfs/some/zip/file/.../contents/) would allow 
hardlinks between files inside the zipfile, but not outside of it.  This 
would be like creating a separate mountpoint for the zipfile's contents, 
and doing "mount --bind" when a hardlink to the zipfile is created.

I'm not sure if we actually have to pretend it's a mountpoint, or if we 
can just take the checking that mountpoints already do and generalize it.


Can you think of a way in which hardlinks are useful in /meta, in a 
*general* way, instead of within a specific directory controlled by a 
specific plugin?

Re: reiser4 plugins

[Hans Reiser]

Ross Biro wrote:

>On 6/29/05, Hubert Chan <hubert@uhoreg.ca> wrote:
>  
>
>>On Wed, 29 Jun 2005 01:09:05 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:
>>
>>    
>>
>>>Hubert Chan <hubert@uhoreg.ca> wrote: [...]
>>>And doing "tar cf /dev/tape /usr/games/tetris" gives you a nice tangle
>>>of undecipherable junk.
>>>      
>>>
>
>I'm confused.  Can someone on one of these lists enlighten me?
>
>How is directories as files logically any different than putting all
>data into .data files and making all files directories (yes you would
>need some sort of special handling for files that were really called
>.data). 
>
Add to this that you make .data the default if the file within the
directory is not specified, and define a stanadard set of names for
metafiles, and you've got the essential idea, and any differences are
details.

> Then it's just a matter of deciding what happens when you
>call open and stat on one of these files?
>
>For backwards compatibility, current existing system calls have to
>treat these things as directories.  Perhaps an exception could be made
>for exec.
>
>But we could have a whole new set of system calls that treat things as
>magic, and if files as directories is as cool as many people think,
>apps will start using the new api.  If not, they won't and the new api
>can be deprecated.
>
>    Ross
>
>
>  
>

Re: reiser4 plugins

[Hubert Chan]

On Wed, 29 Jun 2005 20:04:58 -0700, Hans Reiser <reiser@namesys.com> said:

> Ross Biro wrote:
>> How is directories as files logically any different than putting all
>> data into .data files and making all files directories (yes you would
>> need some sort of special handling for files that were really called
>> .data).

> Add to this that you make .data the default if the file within the
> directory is not specified,

It's sort of like the way web servers handle index.html, for those who
think it's a stupid idea.  (Of course, some people may still think it's
a stupid idea... ;-) )

> and define a stanadard set of names for metafiles, and you've got the
> essential idea, and any differences are details.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Hans Reiser]

Hubert Chan wrote:

>On Wed, 29 Jun 2005 20:04:58 -0700, Hans Reiser <reiser@namesys.com> said:
>
>  
>
>>Ross Biro wrote:
>>    
>>
>>>How is directories as files logically any different than putting all
>>>data into .data files and making all files directories (yes you would
>>>need some sort of special handling for files that were really called
>>>.data).
>>>      
>>>
>
>  
>
>>Add to this that you make .data the default if the file within the
>>directory is not specified,
>>    
>>
>
>It's sort of like the way web servers handle index.html, for those who
>think it's a stupid idea.  (Of course, some people may still think it's
>a stupid idea... ;-) )
>
>  
>
Good analogy.

>>and define a stanadard set of names for metafiles, and you've got the
>>essential idea, and any differences are details.
>>    
>>
>
>  
>

Re: reiser4 plugins

[David Weinehall]

On Thu, Jun 30, 2005 at 12:33:10AM -0400, Hubert Chan wrote:
> On Wed, 29 Jun 2005 20:04:58 -0700, Hans Reiser <reiser@namesys.com> said:
> 
> > Ross Biro wrote:
> >> How is directories as files logically any different than putting all
> >> data into .data files and making all files directories (yes you would
> >> need some sort of special handling for files that were really called
> >> .data).
> 
> > Add to this that you make .data the default if the file within the
> > directory is not specified,
> 
> It's sort of like the way web servers handle index.html, for those who
> think it's a stupid idea.  (Of course, some people may still think it's
> a stupid idea... ;-) )

And guess what?  That's handled on the web server level (userland),
not by the file system.  So different web servers can handle it
differently (think index.html.sv, index.html.zh, index.php, etc).

Amazing isn't it?

[snip]


Regards: David Weinehall
-- 
 /) David Weinehall <tao@acc.umu.se> /) Northern lights wander      (\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\)  http://www.acc.umu.se/~tao/    (/   Full colour fire           (/

Re: reiser4 plugins

[Hubert Chan]

On Thu, 30 Jun 2005 08:29:56 +0200, David Weinehall <tao@acc.umu.se> said:

> On Thu, Jun 30, 2005 at 12:33:10AM -0400, Hubert Chan wrote:
>> It's sort of like the way web servers handle index.html, for those
>> who think it's a stupid idea.  (Of course, some people may still
>> think it's a stupid idea... ;-) )

> And guess what?  That's handled on the web server level (userland),
> not by the file system.  So different web servers can handle it
> differently (think index.html.sv, index.html.zh, index.php, etc).

>From the web *browser*'s point of view, it is handled by the
"filesystem" (which is provided by the various servers).  The browser
doesn't care how or where the data is stored.  It just requests a file,
and gets some data back.  So the browser doesn't have to check for
http://www.example.com/, get a failure (trying to read a directory),
check for http://www.example.com/index.html, etc.  In this way, the web
server controls (which I think takes the place of the filesystem in this
case) what gets shown (index.html.sv, etc.), instead of the leaving it
up to the browser.

In the same way, you don't want every single user program to have to
guess whether it should look at .data, or .contents, or whatever.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Weinehall]

On Thu, Jun 30, 2005 at 11:57:27AM -0400, Hubert Chan wrote:
> On Thu, 30 Jun 2005 08:29:56 +0200, David Weinehall <tao@acc.umu.se> said:
> 
> > On Thu, Jun 30, 2005 at 12:33:10AM -0400, Hubert Chan wrote:
> >> It's sort of like the way web servers handle index.html, for those
> >> who think it's a stupid idea.  (Of course, some people may still
> >> think it's a stupid idea... ;-) )
> 
> > And guess what?  That's handled on the web server level (userland),
> > not by the file system.  So different web servers can handle it
> > differently (think index.html.sv, index.html.zh, index.php, etc).
> 
> >From the web *browser*'s point of view, it is handled by the
> "filesystem" (which is provided by the various servers).  The browser
> doesn't care how or where the data is stored.  It just requests a file,
> and gets some data back.  So the browser doesn't have to check for
> http://www.example.com/, get a failure (trying to read a directory),
> check for http://www.example.com/index.html, etc.  In this way, the web
> server controls (which I think takes the place of the filesystem in this
> case) what gets shown (index.html.sv, etc.), instead of the leaving it
> up to the browser.
> 
> In the same way, you don't want every single user program to have to
> guess whether it should look at .data, or .contents, or whatever.

For the analogy to be complete:

User has a file browser (say Nautilus)
The file browser sees the userland VFS (say a unified VFS between GNOME
and KDE)
The VFS sees the real file system

This way the userland VFS can be ported to almost any platform.

When toying around on the desktop, an abstraction of what a file
is works fine with me.  When doing serious work (programming,
tar:ing up stuff, etc) I want to be bloody sure that I see the
files in the same way always.  I don't want surprises such as
files suddenly behaving as directories or vice versa.


Regards: David
-- 
 /) David Weinehall <tao@acc.umu.se> /) Northern lights wander      (\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\)  http://www.acc.umu.se/~tao/    (/   Full colour fire           (/

Re: reiser4 plugins

[Horst von Brand]

David Weinehall <tao@acc.umu.se> wrote:
> On Thu, Jun 30, 2005 at 11:57:27AM -0400, Hubert Chan wrote:
> > On Thu, 30 Jun 2005 08:29:56 +0200, David Weinehall <tao@acc.umu.se> said:

[...]

> > >From the web *browser*'s point of view, it is handled by the
> > "filesystem" (which is provided by the various servers).  The browser
> > doesn't care how or where the data is stored.  It just requests a file,
> > and gets some data back.  So the browser doesn't have to check for
> > http://www.example.com/, get a failure (trying to read a directory),
> > check for http://www.example.com/index.html, etc.  In this way, the web
> > server controls (which I think takes the place of the filesystem in this
> > case) what gets shown (index.html.sv, etc.), instead of the leaving it
> > up to the browser.

> > In the same way, you don't want every single user program to have to
> > guess whether it should look at .data, or .contents, or whatever.
> 
> For the analogy to be complete:

> User has a file browser (say Nautilus)
> The file browser sees the userland VFS (say a unified VFS between GNOME
> and KDE)
> The VFS sees the real file system

> This way the userland VFS can be ported to almost any platform.

That is /very/ nice to have.

> When toying around on the desktop, an abstraction of what a file
> is works fine with me.  When doing serious work (programming,
> tar:ing up stuff, etc) I want to be bloody sure that I see the
> files in the same way always.  I don't want surprises such as
> files suddenly behaving as directories or vice versa.

Let me moderate that a bit: I'd be happy to have (some) files behaving
strangely, /if in exchange I get something very worthwhile/. I.e., Unix
filesystem sockets, named pipes, virtual filesystems all behave in weird
ways, but this downside is more than compensated by huge advantages (even
being able to do things that would otherwise be impossible). But all I see
is that file-is-directory advocates explain over and over how they are
bending over backards to get the whole mess working exactly like true
directories (nothing more, in the end quite a bit less). The uses I've seen
discussed don't really need files-as-directories (you get most of the
advantages by just tar(1)-ing up the contents, or packing them in some
other way; OpenOffice /has/ structured files, XML inside zipped files, Java
also uses zip files for its structuring needs, etc), or are ideas that
might be nice to have on exclusively one-user, isolated machines, like
"keep /my/ annotations/icon/application name/whatever with the file's
data", but that break down in multiuser (even serially, one user after the
other) systems or when files are shared (via network, or simply by sending
by mail or by copying from one machine to the other). In my rather ample
experience, that situation is rare today, rapidly dwindling in the arena
where Linux is mostly used. So this particular case of strange semantics
for files has no upsides I can see, only downsides. The downsides have been
discussed to death, and AFAICS there are fundamental problems with the idea
that can't be fixed or sanely worked around. So why bother?
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Kevin Bowen]

> might be nice to have on exclusively one-user, isolated machines, like
> "keep /my/ annotations/icon/application name/whatever with the file's
> data", but that break down in multiuser (even serially, one user after the

If this is really the core of your (and the rest of the reiser-
obstructionist crowd's) objection to the file-as-directory concept, then
you just haven't thought it through thoroughly enough. Ignore for the
moment the case of system-wide or network-wide shared data, and think
about it just limited to a user's home directory, and the storage and
organization of actual *data* (as opposed to system files). The desire
amongst users for ubiquitous metadata is very real - the current wave of
"desktop search" products and technologies demonstrates this - but
search is really only the lowest-hanging fruit of this new way of
looking at data. Application-layer solutions like Beagle, Google Desktop
Search et al allow for querying on metadata, but actually *acting* on
the results of those queries requires that they be exposed via first
class primitives which can be manipulated with arbitrary tools, not via
some proprietary userland api which only one tool ever actually
implements. 

As to the case of system-wide shared files, there is already a mechanism
to prevent users from inappropriately annotating files that don't belong
to them: file permissions. If you're sysadmining a multiuser reiser4
box, and your users are able to modify the metadata of files they don't
own, then you go to sysadmin purgatory. 

> other way; OpenOffice /has/ structured files, XML inside zipped files,
Java
> also uses zip files for its structuring needs, etc), or are ideas that

And as a Java developer, I can tell you that the wide consensus is that
this solution is half-assed and insufficient for both developers and
users needs. In fact, I believe there is currently a JSR in progress to
develop a more sophisticated Java packaging model.

-- 
kevin@ucsd.edu

Re: reiser4 plugins

[Horst von Brand]

Kevin Bowen <kevin@ucsd.edu> wrote:
> > might be nice to have on exclusively one-user, isolated machines, like
> > "keep /my/ annotations/icon/application name/whatever with the file's
> > data", but that break down in multiuser (even serially, one user after the

> If this is really the core of your (and the rest of the reiser-
> obstructionist crowd's) objection to the file-as-directory concept, then
> you just haven't thought it through thoroughly enough. Ignore for the
> moment the case of system-wide or network-wide shared data,

I.e., something like 90% of the use of Linux here. OK.

>                                                             and think
> about it just limited to a user's home directory, and the storage and
> organization of actual *data* (as opposed to system files).

Who is to say what is "data" and what is "system files"? And if you are
limiting yourself to /user/ data, why not have the /user/ decide how they
want to organize it, and give them the tools?

>                                                             The desire
> amongst users for ubiquitous metadata is very real - the current wave of
> "desktop search" products and technologies demonstrates this - 

Just like each previous claim "this /must/ be the next cool technology!",
also called later the "dotcom crash"...

>                                                                but
> search is really only the lowest-hanging fruit of this new way of
> looking at data. Application-layer solutions like Beagle,

That works without screwing up the whole system, AFAIU.

>                                                           Google Desktop
> Search et al allow for querying on metadata, but actually *acting* on
> the results of those queries requires that they be exposed via first
> class primitives which can be manipulated with arbitrary tools, not via
> some proprietary userland api which only one tool ever actually
> implements. 

Could you please explain in plain english? The only part I get out is
"propietary API", and I don't see anybody advocating such here.

> As to the case of system-wide shared files, there is already a mechanism
> to prevent users from inappropriately annotating files that don't belong
> to them: file permissions.

And who says that a normal user isn't allowed to annotate each and every
file with its purpose or something else? I can very well imagine a system
in which users (say students in a Linux class) want to do so... on a shared
machine. Or users having a shared MP3 or photograph or ... collection, with
individual notes on each. Or even developers wanting to annotate source
code files with their comments, but leave them read-only (or have them
under SCM).

>                            If you're sysadmining a multiuser reiser4
> box, and your users are able to modify the metadata of files they don't
> own, then you go to sysadmin purgatory. 

Bingo! And thus goes much of the supposed advantage of this nonsense.

[I see that /opponents/ are accused here of lack of imagination, while I
 see that the /proponents/ lack imagination... or perhaps just real-world
 experience.]

> > other way; OpenOffice /has/ structured files, XML inside zipped files,
> > Java also uses zip files for its structuring needs, etc), or are ideas
> > that

> And as a Java developer, I can tell you that the wide consensus is that
> this solution is half-assed and insufficient for both developers and
> users needs. In fact, I believe there is currently a JSR in progress to
> develop a more sophisticated Java packaging model.

Presumably based on ReiserFS 4, which then has to be ported to whatever
platform you want to run Java on ASAP? Great for you! Wait a bit, and
you'll get what you want then, even across the board!
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[David Masover]

Horst von Brand wrote:
> Kevin Bowen <kevin@ucsd.edu> wrote:

[...]

>>                                                            The desire
>>amongst users for ubiquitous metadata is very real - the current wave of
>>"desktop search" products and technologies demonstrates this - 
> 
> 
> Just like each previous claim "this /must/ be the next cool technology!",
> also called later the "dotcom crash"...

Have you used this technology?

For that matter, some of these claims have been true.  The Internet, 
despite the dotcom crash, is now a requirement -- few people would use 
an OS with no networking support on a desktop PC.  Same goes for GUIs. 
Both of these technologies had to have at least some amount of kernel 
support, both were once weird and experimental, and both are now in 
demand by users and likely to be for some time.

If we're right, and searching is the next best thing, then it would be a 
good idea to get started now.  If we're wrong, what's the harm?

(Assuming /meta, of course.  I accept that the issues with 
file-as-directory may never be solved to the point where we can do it at 
the kernel's FS/VFS level.)

>>                                                          Google Desktop
>>Search et al allow for querying on metadata, but actually *acting* on
>>the results of those queries requires that they be exposed via first
>>class primitives which can be manipulated with arbitrary tools, not via
>>some proprietary userland api which only one tool ever actually
>>implements. 
> 
> 
> Could you please explain in plain english? The only part I get out is
> "propietary API", and I don't see anybody advocating such here.

I don't understand it much, but I think the point being made is that 
while desktop searches may be able to search metadata (not sure how they 
do this -- their own plugins or just full-text searching?), they cannot 
write to said metadata.  To change metadata on a file, one currently 
needs to use a proprietary userland api which is only implemented by one 
tool.

So, for instance, if I want to grab all mp3s with Artist "Paul 
Oakenfold" and change the genre to "techno" (can you do that?), I can 
use Beagle's search tool to find all mp3s by Oakenfold, but to change 
the genre, I have to use some separate tool to manipulate id3 tags, 
which may or may not exist in an easily scriptable form.

I know there's a decent tool for id3 tags, but there's so many kinds of 
metadata and so many tools out there that the user can't possibly 
memorize them all.  This is why I use the zipfile example, despite its 
flaws.

Now, the "arbitrary tools" bit is about how one could do this kind of 
thing with file-as-dir or /meta.  With file-as-dir, it might be as 
simple as:

$ for i in `magical-find-tool artist='Paul Oakenfold'`; do echo 'techno' 
 > "$i/.../genre"; done

Or even

$ echo 'techno' > /.../search/artist='Paul Oakenfold'/*/.../genre

It doesn't get too much more complicated for /meta, which has a nice 
side effect of not killing things like tar and generally not messing 
with POSIX.

One could also use vim, emacs, or whatever else to edit this metadata, 
instead of being limited to one program.  This is especially nice 
considering you might have to look up which one program to use.

>>As to the case of system-wide shared files, there is already a mechanism
>>to prevent users from inappropriately annotating files that don't belong
>>to them: file permissions.
> 
> 
> And who says that a normal user isn't allowed to annotate each and every
> file with its purpose or something else? I can very well imagine a system
> in which users (say students in a Linux class) want to do so... on a shared
> machine. Or users having a shared MP3 or photograph or ... collection, with
> individual notes on each. Or even developers wanting to annotate source
> code files with their comments, but leave them read-only (or have them
> under SCM).

File-as-dir is actually pretty nice for this, I think.  Since the 
annotations appear as files, if the app supports it, the users can each 
store individual annotations, world-readable, but only user-writable.

With something like Gnome-VFS, I doubt there's even a mechanism whereby 
such annotations can be shared to begin with.

But this is just a thought experiment atm, don't pick it apart too 
brutally...

Re: reiser4 plugins

[Kevin Bowen]

> > Could you please explain in plain english? The only part I get out is
> > "propietary API", and I don't see anybody advocating such here.

Proprietary was a poor choice of words on my part.

> I don't understand it much, but I think the point being made is that 
...
> So, for instance, if I want to grab all mp3s with Artist "Paul 
> Oakenfold" and change the genre to "techno" (can you do that?), I can 
> use Beagle's search tool to find all mp3s by Oakenfold, but to change 
> the genre, I have to use some separate tool to manipulate id3 tags, 

Yes, this is basically what I was getting at, although I was thinking
more in terms of the reiser5/6/whatever set theoretic semantics, which,
from my point of view at least, reiser4 is simply the first step towards
building the enabling infrastructure of. But the fact that reiser4
semantics + trivial shell scripting enables, as illustrated by David,
the rough equivalent of set-theoretic semantics, demonstrates how
reiser4 is in fact a step in this direction.

> > moment the case of system-wide or network-wide shared data,
> I.e., something like 90% of the use of Linux here. OK.

90% of *what* exactly? 90% of what machines deal with, or 90% of what
humans interact with?

> > users needs. In fact, I believe there is currently a JSR in 
> > progress to develop a more sophisticated Java packaging model.
>
> Presumably based on ReiserFS 4, which then has to be ported to 
> whatever platform you want to run Java on ASAP? Great for you! Wait a
> bit, and you'll get what you want then, even across the board!
 
No, obviously that's not what I was saying. But the need for these kinds
of domain-specific packaging/metadata formats, each requiring their own
tools to work with, would be alleviated if there were simply a more
powerful filesystem semantic. Clearly forcing reiser on the world is a
non-starter, but try extending your imagination a little bit to a future
in which there's a 'new POSIX' specifying a set-theoretic filesystem
model. So-called 'database-filesystems' ARE coming, whether from
Microsoft, Apple, or us. Who gets there first will determine who gets to
make the rules.


-- 
kevin@ucsd.edu

Re: reiser4 plugins

[Horst von Brand]

Kevin Bowen <kevin@ucsd.edu> wrote:

[...]

> > So, for instance, if I want to grab all mp3s with Artist "Paul 
> > Oakenfold" and change the genre to "techno" (can you do that?), I can 
> > use Beagle's search tool to find all mp3s by Oakenfold, but to change 
> > the genre, I have to use some separate tool to manipulate id3 tags, 

> Yes, this is basically what I was getting at, although I was thinking
> more in terms of the reiser5/6/whatever set theoretic semantics, which,
> from my point of view at least, reiser4 is simply the first step towards
> building the enabling infrastructure of. But the fact that reiser4
> semantics + trivial shell scripting enables, as illustrated by David,
> the rough equivalent of set-theoretic semantics, demonstrates how
> reiser4 is in fact a step in this direction.

I don't see any "trivial shell scripting", just need for a plethora of
magic extract-this-or-that-metadata programs. Which can very well work
exactly the same on independent files, no need to shove junk /into/ the
indexed files.

> > > moment the case of system-wide or network-wide shared data,
> > I.e., something like 90% of the use of Linux here. OK.

> 90% of *what* exactly? 90% of what machines deal with, or 90% of what
> humans interact with?

Both. Most use here is in computer labs, where most is shared via NFS
(readonly), plus user data mounted read-write.

> > > users needs. In fact, I believe there is currently a JSR in 
> > > progress to develop a more sophisticated Java packaging model.

> > Presumably based on ReiserFS 4, which then has to be ported to 
> > whatever platform you want to run Java on ASAP? Great for you! Wait a
> > bit, and you'll get what you want then, even across the board!

> No, obviously that's not what I was saying. But the need for these kinds
> of domain-specific packaging/metadata formats, each requiring their own
> tools to work with, would be alleviated if there were simply a more
> powerful filesystem semantic.

*Please explain HOW*!! The domain-specific formats /will/ stay (if nothing
else, because the /domains/ have /specific/ needs), the special tools to
work with them /will/ have to be written exactly the same (because of the
above). All for use on /one/ non-standard filesystem. Plus exactly the same
stuff to work on sane filesystems. The kernel is *not* a magic piece of
software that solves the problem of world hunger if you just can figure out
what strange extension to force onto Linus' kernel version.

>                               Clearly forcing reiser on the world is a
> non-starter, but try extending your imagination a little bit to a future
> in which there's a 'new POSIX' specifying a set-theoretic filesystem
> model.

Sorry, I just don't see any need of shoving Oracle into the kernel. It
leads a quite confortable life in userland.

>        So-called 'database-filesystems' ARE coming, whether from
> Microsoft, Apple, or us.

IBM did it, long ago (AS/400 and OS/400 ring a bell?). And is now slowly
moving away from it (and other structured filesystems), AFAICS, towards
(guess what!) POSIX and Linux...

Again, Linux is what it is today in large part because "We have to get
$FEATURE, because if not, $COMPETITOR will win on us!" arguments have no
traction.

>                          Who gets there first will determine who gets to
> make the rules.

So what? Let /them/ make the mistakes and pay for them, and learn how to do
it /right/, even if later!
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[David Masover]

Horst von Brand wrote:
> Kevin Bowen <kevin@ucsd.edu> wrote:
> 
> [...]
> 
> 
>>>So, for instance, if I want to grab all mp3s with Artist "Paul 
>>>Oakenfold" and change the genre to "techno" (can you do that?), I can 
>>>use Beagle's search tool to find all mp3s by Oakenfold, but to change 
>>>the genre, I have to use some separate tool to manipulate id3 tags, 
> 
> 
>>Yes, this is basically what I was getting at, although I was thinking
>>more in terms of the reiser5/6/whatever set theoretic semantics, which,
>>from my point of view at least, reiser4 is simply the first step towards
>>building the enabling infrastructure of. But the fact that reiser4
>>semantics + trivial shell scripting enables, as illustrated by David,
>>the rough equivalent of set-theoretic semantics, demonstrates how
>>reiser4 is in fact a step in this direction.
> 
> 
> I don't see any "trivial shell scripting", just need for a plethora of
> magic extract-this-or-that-metadata programs. Which can very well work
> exactly the same on independent files, no need to shove junk /into/ the
> indexed files.
> 
> 
>>>>moment the case of system-wide or network-wide shared data,
>>>
>>>I.e., something like 90% of the use of Linux here. OK.
> 
> 
>>>>users needs. In fact, I believe there is currently a JSR in 
>>>>progress to develop a more sophisticated Java packaging model.
> 
> 
>>>Presumably based on ReiserFS 4, which then has to be ported to 
>>>whatever platform you want to run Java on ASAP? Great for you! Wait a
>>>bit, and you'll get what you want then, even across the board!
> 
> 
>>No, obviously that's not what I was saying. But the need for these kinds
>>of domain-specific packaging/metadata formats, each requiring their own
>>tools to work with, would be alleviated if there were simply a more
>>powerful filesystem semantic.
> 
> 
> *Please explain HOW*!! The domain-specific formats /will/ stay (if nothing
> else, because the /domains/ have /specific/ needs), the special tools to
> work with them /will/ have to be written exactly the same (because of the
> above). All for use on /one/ non-standard filesystem.

One filesystem that exists right now.

/proc, as people have made clear, was originally implemented on *one* 
"non-standard" Unix.  Others then copied it.  I see no reason why if 
/meta is a good idea, and programs start using it, that other FSes won't 
implement it.

> Plus exactly the same
> stuff to work on sane filesystems.

"sane" -- I assume you mean "traditional".

Well, yeah.  So?

How big a program do you need to create a new interface to an existing 
system?

Consider:  BitTorrent has at least three interfaces that I can remember 
at the moment.  btdownloadgui.py, btdownloadcurses.py, 
btdownloadheadless.py.

Are you saying that btdownloadgui.py was so much work if you start from 
btdownloadcurses.py that there should be no GTK library, because it 
takes so much work to convert existing curses apps to GTK?

Remember, even if I have GTK installed, I can still get to my curses 
apps, and they may not even know that GTK exists.  Even if I have /meta 
enabled, I can still use my POSIX apps, and they may not even know that 
/meta exists.

>>       So-called 'database-filesystems' ARE coming, whether from
>>Microsoft, Apple, or us.
> 
> 
> IBM did it, long ago (AS/400 and OS/400 ring a bell?). And is now slowly
> moving away from it (and other structured filesystems), AFAICS, towards
> (guess what!) POSIX and Linux...

Linux over POSIX, I think.  Because Linux is big and open-source, not
beacuse POSIX is so great.

> Again, Linux is what it is today in large part because "We have to get
> $FEATURE, because if not, $COMPETITOR will win on us!" arguments have no
> traction.

I agree with that, at least.  But Linux is also what it is today in 
large part because "I don't need $FEATURE, so it should be kept out" 
isn't a particularly good argument either.

Re: reiser4 plugins

[Chet Hosey]

Horst von Brand wrote:

>Kevin Bowen <kevin@ucsd.edu> wrote:
>  
>
>>                                                            and think
>>about it just limited to a user's home directory, and the storage and
>>organization of actual *data* (as opposed to system files).
>>    
>>
>
>Who is to say what is "data" and what is "system files"? And if you are
>limiting yourself to /user/ data, why not have the /user/ decide how they
>want to organize it, and give them the tools?
>  
>
That's exactly what the idea of exporting the metadata as files within a
file-as-dir is intended to do: let the user decide how to manage
metadata. They have the tools -- anything which can interact with a
file. This includes users who never start X.

>  
>
>>                                                            The desire
>>amongst users for ubiquitous metadata is very real - the current wave of
>>"desktop search" products and technologies demonstrates this - 
>>    
>>
>
>Just like each previous claim "this /must/ be the next cool technology!",
>also called later the "dotcom crash"...
>  
>
How does that analogy hold? An example of a technology which is actually
in demand is similar to the tech bubble exactly how?

>>                                                               but
>>search is really only the lowest-hanging fruit of this new way of
>>looking at data. Application-layer solutions like Beagle,
>>    
>>
>
>That works without screwing up the whole system, AFAIU.
>
>  
>
If you define "works" as "implements a proprietary layer which can't be
used naturally by the same tools UNIX users have used for years to
manage data".

>And who says that a normal user isn't allowed to annotate each and every
>file with its purpose or something else? I can very well imagine a system
>in which users (say students in a Linux class) want to do so... on a shared
>machine. Or users having a shared MP3 or photograph or ... collection, with
>individual notes on each. Or even developers wanting to annotate source
>code files with their comments, but leave them read-only (or have them
>under SCM).
>
>  
>
This same argument could be used to attack the idea of group permissions
-- that groups of users might have conflicting goals. Implementing
metadata in userspace via bundled files has the same drawback.

>>                           If you're sysadmining a multiuser reiser4
>>box, and your users are able to modify the metadata of files they don't
>>own, then you go to sysadmin purgatory. 
>>    
>>
>
>Bingo! And thus goes much of the supposed advantage of this nonsense.
>
>  
>
You seem to be implying that on a Reiser4 filesystem used by multiple
users that people other than file owners will be about to apply
arbitrary metadata to any object. It seems to me that adding metadata to
a file-as-dir will require the same permissions as writing to the file
itself, giving Reiser4 *zero* disadvantage against other multiuser
systems. What gives you the impression that adding metadata to a
file-as-dir is any harder to secure against modification from other
users than the files themselves are in the first place?

>[I see that /opponents/ are accused here of lack of imagination, while I
> see that the /proponents/ lack imagination... or perhaps just real-world
> experience.]
>
>  
>
Huh? The idea of arbitrary metadata isn't just with annotating MP3
files. One seemingly important consideration would be the flexible
implementation of add-on mechanisms, like security data, that could be
implemented with zero changes to the filesystem. And you get to use
existing tools to deal with new and arbitrary interfaces instead of
having to deal with GUI garbage just to interact with metadata.

>>>other way; OpenOffice /has/ structured files, XML inside zipped files,
>>>Java also uses zip files for its structuring needs, etc), or are ideas
>>>that
>>>      
>>>
>
>  
>
>>And as a Java developer, I can tell you that the wide consensus is that
>>this solution is half-assed and insufficient for both developers and
>>users needs. In fact, I believe there is currently a JSR in progress to
>>develop a more sophisticated Java packaging model.
>>    
>>
>
>Presumably based on ReiserFS 4, which then has to be ported to whatever
>platform you want to run Java on ASAP? Great for you! Wait a bit, and
>you'll get what you want then, even across the board!
>  
>
Are unable to differentiate between a statement that a given solution is
non-ideal and a suggestion that everything be done in a way that
requires Reiser4? Someone mentions that the current solution isn't quite
meeting all of their needs, and you respond with rhetoric accusing them
of wanting everything to depend on Reiser4 and bloat the kernel. How is
this productive?

Re: reiser4 plugins

[Hubert Chan]

On Fri, 01 Jul 2005 03:41:00 -0400, Chet Hosey <chosey@nauticom.net> said:

> Horst von Brand wrote:
>> And who says that a normal user isn't allowed to annotate each and
>> every file with its purpose or something else?

Explain how you currently allow users to annotate arbitrary files.

>> I can very well imagine a system in which users (say students in a
>> Linux class) want to do so... on a shared machine. Or users having a
>> shared MP3 or photograph or ... collection, with individual notes on
>> each. Or even developers wanting to annotate source code files with
>> their comments, but leave them read-only (or have them under SCM).

> This same argument could be used to attack the idea of group
> permissions -- that groups of users might have conflicting
> goals. Implementing metadata in userspace via bundled files has the
> same drawback.

The situation is even better with file-as-dir.  If the administrator
wants to allow users to edit the description metadata for the file foo,
the administrator can set the appropriate permissions for
foo/.../description, and keep foo read-only.

>>Kevin Bowen <kevin@ucsd.edu> wrote:
>>> If you're sysadmining a multiuser reiser4 box, and your users are
>>> able to modify the metadata of files they don't own, then you go to
>>> sysadmin purgatory.

Actually, you could use something like unionfs to allow users to keep
their own annotations without affecting everyone else's.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Horst von Brand]

Hubert Chan <hubert@uhoreg.ca> wrote:
> On Fri, 01 Jul 2005 03:41:00 -0400, Chet Hosey <chosey@nauticom.net> said:
> > Horst von Brand wrote:
> >> And who says that a normal user isn't allowed to annotate each and
> >> every file with its purpose or something else?

> Explain how you currently allow users to annotate arbitrary files.

By keeping annotations /outside/ the files.

[...]

> The situation is even better with file-as-dir.  If the administrator
> wants to allow users to edit the description metadata for the file foo,
> the administrator can set the appropriate permissions for
> foo/.../description, and keep foo read-only.

So now root is responsible in exquisite detail for random other users being
able to keep info about my files?

[...]

> Actually, you could use something like unionfs to allow users to keep
> their own annotations without affecting everyone else's.

Again, root has to mount that stuff for each and every user?
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[David Masover]

Horst von Brand wrote:
> Hubert Chan <hubert@uhoreg.ca> wrote:
> 
>>On Fri, 01 Jul 2005 03:41:00 -0400, Chet Hosey <chosey@nauticom.net> said:
>>
>>>Horst von Brand wrote:
>>>
>>>>And who says that a normal user isn't allowed to annotate each and
>>>>every file with its purpose or something else?
> 
> 
>>Explain how you currently allow users to annotate arbitrary files.
> 
> 
> By keeping annotations /outside/ the files.
> 
> [...]
> 
> 
>>The situation is even better with file-as-dir.  If the administrator
>>wants to allow users to edit the description metadata for the file foo,
>>the administrator can set the appropriate permissions for
>>foo/.../description, and keep foo read-only.
> 
> 
> So now root is responsible in exquisite detail for random other users being
> able to keep info about my files?

If it's the general info that's associated with the file, and may even 
be stored inside the file, then yes, that's fair.

Although I could certainly imagine foo/.../descriptions being a 
directory that's world-writable, allowing each user to maintain their 
own file inside of it.  You can even set these per-user descriptions to 
be stored somewhere else, like the user's home directory, and that could 
work for CDs.

>>Actually, you could use something like unionfs to allow users to keep
>>their own annotations without affecting everyone else's.
> 
> 
> Again, root has to mount that stuff for each and every user?

Why is that a problem?  Put it in a script.  Mount each user's unionfs 
at boot.

And it's "something like unionfs" -- maybe it's a feature of metafs or 
reiserfs that we haven't thought of yet.  It certainly can't be unionfs 
as it stands, as unionfs doesn't work on top of any reiser.

Re: reiser4 plugins

[Hubert Chan]

On Tue, 05 Jul 2005 22:51:07 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:

> Hubert Chan <hubert@uhoreg.ca> wrote:
>> On Fri, 01 Jul 2005 03:41:00 -0400, Chet Hosey <chosey@nauticom.net>
>> said:
>> > Horst von Brand wrote:
>> >> And who says that a normal user isn't allowed to annotate each and
>> >> every file with its purpose or something else?

>> Explain how you currently allow users to annotate arbitrary files.

> By keeping annotations /outside/ the files.

So if I want to share annotations, I have to look in 20 different
places?

> [...]

>> The situation is even better with file-as-dir.  If the administrator
>> wants to allow users to edit the description metadata for the file
>> foo, the administrator can set the appropriate permissions for
>> foo/.../description, and keep foo read-only.

> So now root is responsible in exquisite detail for random other users
> being able to keep info about my files?

I can grant people permissions to write random info into my own files.
Or they can use unionfs if I don't grant them permissions.

Remember: the above argument was citing an advantage of file-as-dir over
packed files (storing metadata as a tar or zip file, similar to what
OpenOffice.org does, or even something like exif or id3).  In a packed
file, I can't allow people to edit the description attribute without
allowing them to edit the entire file.  With file-as-dir, I get much
finer grained permissions.

> [...]

>> Actually, you could use something like unionfs to allow users to keep
>> their own annotations without affecting everyone else's.

> Again, root has to mount that stuff for each and every user?

suid program that allows union mount into a directory within my own tree
(or just into any directory that I have write permissions should be
sufficiently secure).

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 2764 bytes --]

On Thu, 2005-06-30 at 14:47 -0400, Horst von Brand wrote:
[snip]
> Let me moderate that a bit: I'd be happy to have (some) files behaving
> strangely, /if in exchange I get something very worthwhile/. I.e., Unix
> filesystem sockets, named pipes, virtual filesystems all behave in weird
> ways, but this downside is more than compensated by huge advantages (even
> being able to do things that would otherwise be impossible). But all I see
> is that file-is-directory advocates explain over and over how they are
> bending over backards to get the whole mess working exactly like true
> directories (nothing more, in the end quite a bit less). The uses I've seen
> discussed don't really need files-as-directories (you get most of the
> advantages by just tar(1)-ing up the contents, or packing them in some
> other way; OpenOffice /has/ structured files, XML inside zipped files, Java
> also uses zip files for its structuring needs, etc), or are ideas that
> might be nice to have on exclusively one-user, isolated machines, like
> "keep /my/ annotations/icon/application name/whatever with the file's
> data", but that break down in multiuser (even serially, one user after the
> other) systems or when files are shared (via network, or simply by sending
> by mail or by copying from one machine to the other). In my rather ample
> experience, that situation is rare today, rapidly dwindling in the arena
> where Linux is mostly used. So this particular case of strange semantics
> for files has no upsides I can see, only downsides. The downsides have been
> discussed to death, and AFAICS there are fundamental problems with the idea
> that can't be fixed or sanely worked around. So why bother?

Structured files are fine when they're small but quickly become
disgusting as they get bigger.  Either you slowly rewrite the whole
thing or you "fast save" by writing new sections to the end of it that
replace existing sections.  That's how you end up with documents that
contain "deleted" information that was supposed to be confidential.

Having the filesystem track each part for you and then creating a
structured file when needed (and without needing to remember to run a
special tool) is a far better idea.  (reiser4 doesn't do this but its a
possible idea)

Another advantage to file-as-directory is being able to access all the
file's meta-data through a single channel: file names and text data.
It removes the need for chmod, chown, touch, getxattr, chacl and all the
rest of the special tools needed to work with Unix files.  It also makes
it far easier to add new meta-data in the future, because no new tools
have to be written to use it.

So that's two reasons to bother.
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[David Masover]

Zan Lynx wrote:
> On Thu, 2005-06-30 at 14:47 -0400, Horst von Brand wrote:
> [snip]
[snip-some-more]
> Structured files are fine when they're small but quickly become
> disgusting as they get bigger.  Either you slowly rewrite the whole
> thing or you "fast save" by writing new sections to the end of it that
> replace existing sections.  That's how you end up with documents that
> contain "deleted" information that was supposed to be confidential.

Doesn't OpenOffice already do something similar with zipfiles?  I 
remember noticing huge differences in save times after adding large 
images to a presentation versus just changing some text.  So, it's 
obviously not repacking each image every time I save.  But, as the 
internal text is all XML, I doubt it's leaving "deleted" sections lying 
around.

> Having the filesystem track each part for you and then creating a
> structured file when needed (and without needing to remember to run a
> special tool) is a far better idea.  (reiser4 doesn't do this but its a
> possible idea)

Been discussed to death, and chances are, Reiser4 will do this with 
plugins.  I imagine it being implemented in a more general way, though. 
  For example, a meta-file which, when read, produces a 
zipfile/tarball/dump of the directory it belongs to.  Then, to send such 
a document via email, you just navigate to the directory where it lives 
while looking for files to attach, only from /meta/vfs as a base, then 
attach the meta-file.  The zipfile gets built automatically.

To simplify the process, all you *really* need is a button on all FS 
navigation windows (Nautilus, Open/Save dialogs, etc) to switch back and 
forth between the meta view and the normal view.  That, or a separate pane.

This way, instead of patching the mail client to support all possible 
transformations you'd want to do before sending, or patching all 
applications to use zip (and any other transformation a user might want 
to do), or forcing the user to run another app, you automatically get 
*all* available transformations, in a way that could potentially look as 
smooth as adding zip support directly to the app.

> Another advantage to file-as-directory is being able to access all the
> file's meta-data through a single channel: file names and text data.
> It removes the need for chmod, chown, touch, getxattr, chacl and all the
> rest of the special tools needed to work with Unix files.  It also makes
> it far easier to add new meta-data in the future, because no new tools
> have to be written to use it.

All good points except the last one.  New tools vs new plugins?  People 
already know how to write the tools...

> So that's two reasons to bother.

That, and the possible paradigm shift.  Navigating to the /meta dir for 
a file could be the commandline equivalent of right-clicking on the file.

Of course, to some people, paradigm shifts are a reason *not* to bother...

Re: reiser4 plugins

[Hubert Chan]

On Thu, 30 Jun 2005 19:10:57 +0200, David Weinehall <tao@acc.umu.se> said:

> For the analogy to be complete:

> User has a file browser (say Nautilus)
> The file browser sees the userland VFS (say a unified VFS between
> GNOME and KDE)
> The VFS sees the real file system

I would say that this only works if everyone is forced to use the same
VFS.  In the web example, everyone is forced to use the same API (HTTP),
and so everyone gets the same view.  For the real filesystem, not
everyone is forced to use the hypothetical unified GNOME/KDE VFS.  So if
I try to edit with gedit or kate, I get something different than if I
try to edit with vi or emacs.

As I see it:
web browser <-> Nautilus/user apps
web server <-> filesystem
web server's filesystem/database/etc. <-> physical disk storage

Of course, we all know that analogies are not perfect.  The layering in
both sides isn't exactly the same.  And other people could assign
different equivalences (e.g. web browser <-> Nautilus/user apps; web
server <-> VFS; web server's filesystem <-> filesystem).

Anyways, the analogy wasn't supposed to be about where to handle the
magic extra functionality, whether userspace or kernel space.  The
analogy was for people who might think that it's stupid to return
foo/.data when the user tries to open the directory foo; it was meant to
illustrate that that idea isn't completely braindead.

> This way the userland VFS can be ported to almost any platform.

I think GNOME and KDE will always need a VFS if it wants to be
cross-platform.  But I think that if the kernel provides extra
functionality, GNOME may be better off.  For example, glib provides its
own threading abstraction.  But on systems that use pthreads, glib's
threading library uses it for its implementation.  And I think it can
even be used on systems that don't offer threading, by doing its own
emulation of threading.

> When toying around on the desktop, an abstraction of what a file is
> works fine with me.  When doing serious work (programming, tar:ing up
> stuff, etc) I want to be bloody sure that I see the files in the same
> way always.  I don't want surprises such as files suddenly behaving as
> directories or vice versa.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Masover]

Hubert Chan wrote:
> On Thu, 30 Jun 2005 08:29:56 +0200, David Weinehall <tao@acc.umu.se> said:
> 
> 
>>On Thu, Jun 30, 2005 at 12:33:10AM -0400, Hubert Chan wrote:
>>
>>>It's sort of like the way web servers handle index.html, for those
>>>who think it's a stupid idea.  (Of course, some people may still
>>>think it's a stupid idea... ;-) )
> 
> 
>>And guess what?  That's handled on the web server level (userland),
>>not by the file system.  So different web servers can handle it
>>differently (think index.html.sv, index.html.zh, index.php, etc).
> 
> 
> From the web *browser*'s point of view, it is handled by the
> "filesystem" (which is provided by the various servers).  The browser
> doesn't care how or where the data is stored.  It just requests a file,
> and gets some data back.  So the browser doesn't have to check for
> http://www.example.com/, get a failure (trying to read a directory),
> check for http://www.example.com/index.html, etc.  In this way, the web
> server controls (which I think takes the place of the filesystem in this
> case) what gets shown (index.html.sv, etc.), instead of the leaving it
> up to the browser.

Somewhat flawed analogy, though.  After the protocol definition, the 
browser proper will take any URL that the protocol handler likes, which 
is why file:// works.  After the domain, the http/https handler will 
take any URL at all, except for maybe some character set issues.  So 
assuming the server is compatible with itself, we don't have to worry 
about whether the browser supports going to a directory and having it 
behave as a file (index.html behavior), or going to a file and having it 
behave as a directory (as some scripts do -- I've seen urls that look 
like http://example.com/foo.cgi/bar.html)

Among protocols that behave more like filesystems, such as FTP, you 
can't really pull the same tricks.  When an FTP client asks for a 
directory, it's probably asking for a directory listing, and would be 
quite surprised to find a file there -- or the user would when binary 
data floods their terminal.

I *think* this is how FTP works, but I haven't used it in years, except 
through a web browser.  I still get the feeling that even a web browser 
expects an FTP file to behave as a file and an FTP directory to behave 
as a directory.

But I'm also pretty sure that FTP would be much more receptive to 
file-as-directory than your average sysadmin would.  For one, breaking 
tar is unforgivable, and the only ways I can think of fixing that issue 
are shaky at best when you consider how many apps might do things 
oh-so-slightly different than tar.

Re: reiser4 plugins

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

hoi :)

On Wed, Jun 29, 2005 at 08:04:58PM -0700, Hans Reiser wrote:
> >How is directories as files logically any different than putting all
> >data into .data files and making all files directories (yes you would
> >need some sort of special handling for files that were really called
> >.data). 
> >
> Add to this that you make .data the default if the file within the
> directory is not specified, and define a stanadard set of names for
> metafiles, and you've got the essential idea, and any differences are
> details.

sure, that would work more or less, but what would it give you?
You haven't introduced anything new that userspace couldn't do before.
Just write a library which mangles pathnames and treats files and
directories the same. You don't need the kernel to do that.

Filesystems are there to store files.
Everything else can be done in userspace.

-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Jonathan Briggs]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

On Tue, 2005-07-05 at 17:46 +0200, Martin Waitz wrote:
[snip]
> Filesystems are there to store files.
> Everything else can be done in userspace.

You could do filesystems in userspace too and just use the kernel's
block layer.

In fact you can reduce the OS kernel to just interrupts, memory
management, context switches and message passing.

Everything else can be done in userspace, but that doesn't always make
it a good idea.
-- 
Jonathan Briggs <jbriggs@esoft.com>
eSoft, Inc.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

hoi :)

On Tue, Jul 05, 2005 at 04:32:00PM -0600, Jonathan Briggs wrote:
> You could do filesystems in userspace too and just use the kernel's
> block layer.

but you can't do that in an library, you have to use a filesystem
server in order to get access control.
But you can build a library that handles uniform access to
files and directories.

Don't get me wrong, I'm all for a uniform interface for files and
metadata, but I don't think that it has to be in the kernel.
Gnome and KDE already have their own userspace VFS, something
like that should be used.

One has to distinguish between the low-level filesystem and
the storage system which is presented to the user.

-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Hans Reiser]

Martin Waitz wrote:

>hoi :)
>
>On Tue, Jul 05, 2005 at 04:32:00PM -0600, Jonathan Briggs wrote:
>  
>
>>You could do filesystems in userspace too and just use the kernel's
>>block layer.
>>    
>>
>
>but you can't do that in an library, you have to use a filesystem
>server in order to get access control.
>But you can build a library that handles uniform access to
>files and directories.
>
>Don't get me wrong, I'm all for a uniform interface for files and
>metadata, but I don't think that it has to be in the kernel.
>Gnome and KDE already have their own userspace VFS, something
>like that should be used.
>
>One has to distinguish between the low-level filesystem and
>the storage system which is presented to the user.
>
>  
>
Yes, but to do what you advocate properly, the existing semantics
currently in the kernel should be moved out of it into user space.

I think the exokernel approach by Frans is a very interesting approach. 
I wish I had the experience with it necessary to know if it was
effective.  I do NOT take the position that name resolution should be in
the kernel.  I DO take the position that it should be either in the
kernel or out of the kernel, and should constitute one cohesive and
coherent body of code.  If someone talks Linus into trying the exokernel
approach, I will be happy to educate myself to where I have an opinion
on whether that works.  It is easy to see powerful advantages to the
exokernel approach: I wish I understood the security model for it, and I
wish I was sure that name resolution would not require too many context
switches as one fetches each storage component required by a name
resolution.

Masover's words about HURD earlier in this thread were very well said. 
Context switching is expensive, and I find it easier to be sure my code
will both perform well and be cohesive if it is all done in the kernel
by one body of code.

Re: reiser4 plugins

[Horst von Brand]

Hans Reiser <reiser@namesys.com> wrote:

[...]

> I think the exokernel approach by Frans is a very interesting approach. 
> I wish I had the experience with it necessary to know if it was
> effective.  I do NOT take the position that name resolution should be in
> the kernel.  I DO take the position that it should be either in the
> kernel or out of the kernel, and should constitute one cohesive and
> coherent body of code.

Right.

>                         If someone talks Linus into trying the exokernel
> approach,

Are you nuts?! Such radical experiments do /not/ belong in the kernel on
which millions of machines depend!

Go and fork off a branch to play around with this, and if it does show real
promise, you can then come back and try to integrate this into the official
kernel.

>           I will be happy to educate myself to where I have an opinion
> on whether that works.  It is easy to see powerful advantages to the
> exokernel approach: I wish I understood the security model for it, and I
> wish I was sure that name resolution would not require too many context
> switches as one fetches each storage component required by a name
> resolution.

Exactly the kinds of questions that have to get solid answers before any
experimental patches can get off the ground.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[Hans Reiser]

Horst von Brand wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>
>[...]
>
>  
>
>>I think the exokernel approach by Frans is a very interesting approach. 
>>I wish I had the experience with it necessary to know if it was
>>effective.  I do NOT take the position that name resolution should be in
>>the kernel.  I DO take the position that it should be either in the
>>kernel or out of the kernel, and should constitute one cohesive and
>>coherent body of code.
>>    
>>
>
>Right.
>
>  
>
>>                        If someone talks Linus into trying the exokernel
>>approach,
>>    
>>
>
>Are you nuts?! Such radical experiments do /not/ belong in the kernel on
>which millions of machines depend!
>  
>
Your response is a flame, and I will not respond to it.

Re: reiser4 plugins

[David Masover]

Hubert Chan wrote:
> On Tue, 28 Jun 2005 15:59:03 -0400, Kyle Moffett <mrmacman_g4@mac.com> said:
> 
> 
>>On Jun 28, 2005, at 13:51:04, Hubert Chan wrote:
> 
> 
>>> I don't know if VFS is the right place for it, but I agree that it
>>>would be good to make it accessible to all filesystems.
> 
> 
>>That's somewhat of a contradiction in terms.  The whole point of the
>>VFS is to hold all of the things that multiple filesystems want to
>>share :-D.
> 
> 
> VFS provides a common interface to the filesystem.  I don't think metafs
> needs any VFS changes.  It may be able to get by without making changes
> to the VFS, and if so, it shouldn't touch the VFS.  It should just be
> its own separate filesystem.
> 
> I imagine most of it could be implemented by a FUSE filesystem.

"could", yes.  "should", no.  I'll refer you to my HURD comment.

That's not to say that none of this should be userspace, just that some 
of it most certainly *never* needs to touch userspace, such as 
cryptocompress.

I'm not guessing that you wanted to make it FUSE, I just want to be 
pre-emptive here.  FUSE will NOT work well for this.

>>Maybe we just need better regular applications?
> 
> 
> You mean patch them all so that they understand and can edit
> xattr/substreams/etc.?  The file-as-dir interface is meant to avoid
> having to do that.  metafs also avoids having to patch all the
> applications by exposing them as regular files.

Metafs also avoids having to patch tar.  It's assumed that legacy backup 
systems can always avoid metafs and still catch almost everything 
important, and certainly everything they already do catch.  With a 
hybrid or an entirely new backup system, we could catch everything, 
including any new ACL-like animals that people invent.

Re: reiser4 plugins

[Jeremy Maitin-Shepard]

David Masover <ninja@slaphack.com> writes:

[snip]

> That's not to say that none of this should be userspace, just that some 
> of it most certainly *never* needs to touch userspace, such as 
> cryptocompress.

It seems that transparent encryption and compression really has little
to do file-as-directory and metafs.  There needs to be a mechanism for
specifying (per-process or per-user) keys to the kernel, but other than
that, the file or directory on which this transparent
encryption/compression has been enabled can just be accessed normally.
An attempt to access a file or directory for which a key is not
available would give EPERM (or possibly the encrypted data, but this
would be problematic for directories; it also might be the case that
only encryption of file contents and file names would be supported).

> I'm not guessing that you wanted to make it FUSE, I just want to be 
> pre-emptive here.  FUSE will NOT work well for this.

In suggesting the file-path-based search facility (which, by the way, is
an interface that does not allow for searching with terms that include
'/'), you surely cannot be suggesting that search-by-artist or IDv3 tag
support be added to the kernel.

[snip]

-- 
Jeremy Maitin-Shepard

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kyle Moffett wrote:
> On Jun 27, 2005, at 22:21:35, Hubert Chan wrote:
> 
>> On Mon, 27 Jun 2005 18:27:26 -0500, David Masover 
>> <ninja@slaphack.com> said:
>>
>>> Kyle Moffett wrote:
>>>
>>>> I think the '...' is just a bad idea in general, because it breaks
>>>> tar and such.  An interface like this might be simpler to design and
>>>> use:
>>>>
>>>> # mkdir /attr
>>>> # mount -t attrfs attrfs /attr
>>>>
>>>> /attr/fd/$FD_NUM == '...' directory for a filedescriptor
>>>> /attr/fs/$DEV_NUM/$INODE_NUM == '...' directory for an inode
>>
>>
>> The most obvious (at least obvious for me) complaint about this is  that
>> the attributes are separate from the file.  (In other words, it's a  bit
>> ugly. ;-) ) So if you want to backup a file, along with all its
>> (extended) attributes (or substreams, or ... etc. ...), you need to
>> backup the file, and find the appropriate /attr/fs/$DEV_NUM/$INODE_NUM
>> and back that up.  If I want to edit an attribute, I need to find
>> $DEV_NUM and $INODE_NUM (which I have no idea how to do), rather than
>> just "edit foo/.../extended_attribute".  (Or using your "getattrpath"
>> command, it would be a two-step process -- run getattrpath, then run
>> editor -- rather than a one-step process.  Of course, this is only
>> mildly annoying.)
> 
> 
> These are not really "attributes" so much as they are "metadata", for
> example, a "contents" subdirectory, if one existed, would be based on
> the original file, and therefore non-unique, but would be looked up
> based on information about the original file.

If you say so.  I like the name "meta" because it looks cooler and is
easier to type, but I could live with "attr".

>> It also exposes a difference between attributes and regular files.
>> e.g. can I add attributes to an attribute?  (say, I have a thumbnail
>> attribute for the file ~/foo, and I want to add a mime-type  attribute to
>> that thumbnail attribute.)  If you want to allow it, you have to be
>> careful not to run into a big loop generating an infinite number of
>> inodes in the attrfs. (e.g.
>> /attr/fs/$(getattrpath /attr/fs/$(getattrpath ~/foo)/thumbnail)/ mimetype
>> -- attrfs shouldn't generate the inode for that until
>> /attr/fs/$(getattrpath~/foo)/thumbnail is accessed, and maybe not even
>> then...)
> 
> 
> Agreed.  I discuss this more below
> 
>> That said, I prefer that interface over xattr or openat.
> 
> 
> Absolutely.  On the other hand, that's not to say it can't be improved.
> 
>>
>>>> It would be usable from a shell with a simple "getattrpath" command
>>>> which returns "fs/$DEV_NUM/$INODE_NUM" by stat-ing any given path.
>>
>>
>>> Still pretty annoying, but maybe a good idea, especially if the shell
>>> gets extended to support it.  Not sure I like using inode numbers,
>>> though -- I like the idea of being able to symlink to stuff inside  the
>>> meta-file dir.
>>>
>>
>> The advantage of using inodes rather than pathnames is that it is  robust
>> with respect to file renaming/moving.  It also allows things like
>> adding attributes to symlinks (since the inode number for a symlink is
>> different from the inode number for the file that it points to).
>>
>> You can still symlink.  It just takes a little more effort to  figure out
>> what $DEV_NUM/$INODE_NUM to use.
> 
> 
> Also, unlike a symlink, if the path doesn't change and the file does, it
> will break, also, if the file is removed and another created  elsewhere, it
> will be redirected improperly.  Perhaps a new symlink syntax is  needed to
> allow attribute specification (Ick, more changes :-\).

There are symlinks, and there are hardlinks.  When I want a link to a
file which doesn't change even if the file is moved, I use a hardlink.
When I want a link to a particular location, even if the file there is
deleted and then replaced, or if the file is not likely to move anytime
soon, I use a symlink.

Besides, can't we have it automagically figure out what the fs is once
we give it the device number?

I propose we should be able to have:

	/meta/$DEV_NUM/$INODE_NUM
	/meta/vfs/path/to/file

That way, I can symlink to whichever is appropriate.

>>> Hans, thoughts?  Seems to be namespace fragmentation, but seems
>>> usable, less breakage, and so on.  And should it be /attr or /meta?
>>
>>
>> For the mount point, it doesn't matter; it's up to the user.  It's the
>> attrfs or metafs or ???fs that matters (but which will greatly  influence
>> whether people user /attr or /meta).
> 
> 
> "meta" seems the more descriptive name.  There should also probably be a
> somewhat standardized location for this, such that programs can  locate it
> without much trouble.  This mechanism would be usable from all FSs, and
> could be built into the VFS.

Maybe.  I think it should be its own FS, mountable and unmountable, just
like /proc or /sys.  It's a standardized location anyway, but easier to
admin that way.

But we definitely want to support multiple FSes.  Even more reason to
make it a separate system, accessible as its own mountable FS.

> Also, it would allow one to access the  meta
> data of meta data (if supported by the filesystem, and possibly only
> through the file descriptor lookup, due to numbering limitations.)

Or just use:

/meta/vfs/meta/$DEV_NUM/$INODE_NUM
/meta/vfs/meta/vfs/path/to/file/some/meta/file

Loop as many times as the FS supports -- if the FS doesn't like a
particular amount of looping, the file will simply not be there.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsGuHngHNmZLgCUhAQLFDA//etF3JuQVLCz4XV60yDWZ3rEtf9mHew7F
Ki0ON1q2P94TppOhjhUWPUQ8v53K5AIRkXyr0Uc1AVxqzj8ff5IHxYRXOJqB1OAC
3sJpRawIW+4g7Odwej8320gVrSPGJ4xMOm5HN2QFWhjlqPdAcMIydoZnM5WFAV/N
0/iaHA54wIrxGepCp0KPmyW3PJTRQj3PiBInvEER6dSnPpjKScEbSGLgVXHaHVpA
X3ujOvNjFzmPrbmytQuJY3lB3dFSOXVarv0RKOaCqbDUW1KoWLiXnXD1t+9qLqRy
w3LlRFMTooh5jsmWsorEVDqIS2ZsPL5fnYOkx5pTgUU2/pVLyedJWnB3zbzOqZ4T
kypdr9yZZqoVakYNTmHqEGQp4z5BuKzZgoAjAJXJrST1+IhdDfjAWQ61NrSsFP3C
XIVfsuY2WwUiy3wUrATUgIL8cpeKz9WRzJOsnD7b5e8YB8ceR8uElSD2VB5RKF+q
v7E1chqofOWgIgclSPMq74jCa3iD6PF5W8ohS5J0P4TafyTr8KGB17Is81O55d5D
3aoioC6JzKW9iQ4SM+CxeS5gTNqI/O9cA/NU+aZzrz9P6xQSB5bOoMMfm1N3pzLD
obhZaGyVZtpn5mc+AoVi9JLKs105JFZzHgdZYiwsJdIR07oMZYfOU34As0dgp78+
O6Em4T/9YAQ=
=hvkI
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hubert Chan wrote:
> On Mon, 27 Jun 2005 18:27:26 -0500, David Masover <ninja@slaphack.com> said:
> 
> 
>>Kyle Moffett wrote:
>>
>>>I think the '...' is just a bad idea in general, because it breaks
>>>tar and such.  An interface like this might be simpler to design and
>>>use:
>>>
>>># mkdir /attr
>>># mount -t attrfs attrfs /attr
>>>
>>>/attr/fd/$FD_NUM == '...' directory for a filedescriptor
>>>/attr/fs/$DEV_NUM/$INODE_NUM == '...' directory for an inode
> 
> 
> The most obvious (at least obvious for me) complaint about this is that
> the attributes are separate from the file.  (In other words, it's a bit
> ugly. ;-) ) So if you want to backup a file, along with all its
> (extended) attributes (or substreams, or ... etc. ...), you need to
> backup the file, and find the appropriate /attr/fs/$DEV_NUM/$INODE_NUM
> and back that up.  If I want to edit an attribute, I need to find
> $DEV_NUM and $INODE_NUM (which I have no idea how to do), rather than
> just "edit foo/.../extended_attribute".  (Or using your "getattrpath"
> command, it would be a two-step process -- run getattrpath, then run
> editor -- rather than a one-step process.  Of course, this is only
> mildly annoying.)

Especially considering you could add a tar/rsync flag to handle it for
you.  And most likely, each meta dir would have its own serialization
method (read: file) so that you didn't find yourself trying to back up
huge infinitely recursive structures, or accidentally triggering a huge
tar operation, or something...

> It also exposes a difference between attributes and regular files.
> e.g. can I add attributes to an attribute?

Under my scheme, yes, and it works the same way.

> (say, I have a thumbnail
> attribute for the file ~/foo, and I want to add a mime-type attribute to
> that thumbnail attribute.)  If you want to allow it, you have to be
> careful not to run into a big loop generating an infinite number of
> inodes in the attrfs.

I'm not sure if inodes are a problem, but this kind of stuff would be up
to the fs, not the attrfs or metafs.

> That said, I prefer that interface over xattr or openat.

Absolutely.  But I think both should exist, so that programs currently
using xattr will still work.  Maybe you'd have:

/meta/vfs/home/foo/attrs

or something...

Come to think of it, that changes the proposal a bit.  You need a
different way to access the meta-dir for files than for directories, if
we're going to support /meta/vfs.  No biggie:

/meta/vfs/file/home/bob/sue.mpg/acl
/meta/vfs/dir/home/bob/acl

>>>It would be usable from a shell with a simple "getattrpath" command
>>>which returns "fs/$DEV_NUM/$INODE_NUM" by stat-ing any given path.
> 
> 
>>Still pretty annoying, but maybe a good idea, especially if the shell
>>gets extended to support it.  Not sure I like using inode numbers,
>>though -- I like the idea of being able to symlink to stuff inside the
>>meta-file dir.
> 
> 
> The advantage of using inodes rather than pathnames is that it is robust
> with respect to file renaming/moving.  It also allows things like
> adding attributes to symlinks (since the inode number for a symlink is
> different from the inode number for the file that it points to).

I think we need both interfaces, especially because the /meta/vfs
interface makes looping easy.

> You can still symlink.  It just takes a little more effort to figure out
> what $DEV_NUM/$INODE_NUM to use.

Doesn't have the same effect.

>>Actually, I like this.  Give me some time to let it percolate.
> 
> 
>>Hans, thoughts?  Seems to be namespace fragmentation, but seems
>>usable, less breakage, and so on.  And should it be /attr or /meta?
> 
> 
> For the mount point, it doesn't matter; it's up to the user.

No, it's not.  It's up to the user and the application developers.
Sure, I could mount /proc somewhere else, but that would confuse some
things.  Mounting /meta somewhere else would confuse even more things.

> It's the
> attrfs or metafs or ???fs that matters (but which will greatly influence
> whether people user /attr or /meta).

True, but I think they should still hand down a decree of where it is to
be mounted by default.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsGxn3gHNmZLgCUhAQLWkg/+MIzXbygGoHubwLcw5SI1eeanVssukzF7
9kl+3qYcpbC6l/4fFkezBA8YCb8yy/VCfadAJ4Vgsr7T0rvGq1fvKIER6Q4LfU22
YCkHfU6dCnTmTjqHTFyYqZX95J6JXKUT/HCXn5AaYC5DIOYMp1gFHgj6P1wKOb/o
KRvuOSIO0ta8fpPg6tBe6ve92sp99zqbacUplN+38+eBEIZUI7s7Dkg1lWX7ScyZ
LVtHDEDJUuP0/Frq3xZHFKdgsVhoDtCm5Cl6EBX0OSubExPa/7ID2ANA+W01lqbm
9KMtXAcp/K8uOVkM8YDpFvvP2Ru/gp+iKy993lpGY9bEhIZ6BVz4zpVMvloHQe3a
GCdsvub1yP7qQa/7LpkN6Shbw8W25iXcG0QUXa6XuiaJZIsVvRThrWOPXstx9GGQ
KDZtMGIzH5vJcXQzIKzHMrfJBIKMmu3c/Et1EmdJy1t1PkH95+gGCrYc/nfBizfk
NMYIp8S+RdGXwIcOWcwu4WspTJEZg1w7spXuydBZHd55YCq5JJE7lLNoK8Y3ZXks
xC7b6Eo/0FgsxUmoQ9QYl9kzrxAejBGWmmTfsQGjFhu0Ep7jVCZHRRSq8r2RwRj/
AYY2JeFu7yRHsKy29BYipo0etR38Z5+Z0ZOn3iZZHHVz7D6u12U+69LXUEeyp46H
mYzXXJOmIVA=
=h/Gv
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hubert Chan]

On Tue, 28 Jun 2005 15:22:55 -0500, David Masover <ninja@slaphack.com> said:

> Come to think of it, that changes the proposal a bit.  You need a
> different way to access the meta-dir for files than for directories,
> if we're going to support /meta/vfs.  No biggie:

> /meta/vfs/file/home/bob/sue.mpg/acl
> /meta/vfs/dir/home/bob/acl

What if /home has an extended attribute named bob?  Then is
/meta/vfs/dir/home/bob a file or a directory?

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Masover]

Hubert Chan wrote:
> On Tue, 28 Jun 2005 15:22:55 -0500, David Masover <ninja@slaphack.com> said:
> 
> 
>>Come to think of it, that changes the proposal a bit.  You need a
>>different way to access the meta-dir for files than for directories,
>>if we're going to support /meta/vfs.  No biggie:
> 
> 
>>/meta/vfs/file/home/bob/sue.mpg/acl
>>/meta/vfs/dir/home/bob/acl
> 
> 
> What if /home has an extended attribute named bob?  Then is
> /meta/vfs/dir/home/bob a file or a directory?

Ah, you caught it.  I knew it didn't feel right.  So /meta/vfs/file and 
/meta/vfs/dir are worthless.

Ok, there has to be a delimiter of some kind.  That, or there has to be 
a way we can pick up front how deep we're going.  Or, someone else has 
to come up with a better idea.  Or, start out with the assumption that 
we're talking about metadata, instead of the other way around.

I don't like delimiters because the way we usually deal with these is 
escaping them when we need to.  For instance, if I want to actually pass 
a literal " character to some command, I can wrap it in ' characters or 
add a \ to the front of it.

But we're expecting this to be simpler than that.  A program trying to 
access the metas for some file shouldn't have to worry about escaping.

Last time we tried to come up with a delimiter, we ended up with ... and 
..metas as possible candidates, and ..metas was in the source.  As 
unlikely as it is to hit either of those, I still don't like them.

Programs would mostly use the inodes anyway, and users would mostly use 
the /meta/vfs interface, but it's still not pretty.

But, the only alternative I can think of now is /meta/vfs/2/home/bob is 
a file and /meta/vfs/3/home/bob is a directory.  That's not something we 
want our users to have to do, especially considering we want to be able 
to have metadata of metadata.

That, or we can try something like:

/meta/vfs/#/home/#/bob	is a directory
/home/vfs/#/home/bob	is a file

where we add a delimiter to the /meta dir which identifies where each 
directory begins, not each set of metadata.  To clarify a bit:

$ ls /meta/vfs/
mime
permissions
acls
#
[snip]

$ ls /meta/vfs/#/
proc
sys
tmp
etc
home
[snip]

$ ls /meta/vfs/#/home
mime
permissions
acls
#
[snip]

$ ls /meta/vfs/#/home/#/
bob
billy
hank
sue
[snip]

This is pretty annoying for the user, though, because it's more verbose. 
  But, at least this way, we can guarentee that no one will kill our 
delimiter, because it exists in a new namespace we're creating anyway. 
That is, if we have to, the old xattrs stuff can go in its own folder, 
and so even if some app currently depends on a '#' app, it won't kill 
our delimiter.

Now, if only there was an easier delimiter to type...


Ok, since file-as-dir was going to do this anyway, I think the way to go 
is probably the original '...' delimiter.  It's safer, because it stays 
in /meta, but it's dangerous, because someone might actually try to 
create a '...' file on an existing system.  But, I think we can live 
with introducing a delimiter, more than we can live with uglifying the 
/meta/vfs interface, especially because "touch ..." wouldn't make the 
system blow up, you just couldn't easily get metadata for "..."

Re: reiser4 plugins

[Lee Revell]

On Mon, 2005-06-27 at 18:27 -0500, David Masover wrote:
> Right on all points.  Just remember that some change is good.  Why do
> we
> have ALSA now?  Everything a user can do with ALSA, they can do with
> OSS, AFAIK.
> 

Wrong, you have it backwards.  The ALSA API is a superset of the OSS
API.  Otherwise, what would the point have been?

Lee

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:
> Kyle Moffett wrote:
> > On Jun 26, 2005, at 22:37:48, David Masover wrote:

[...]

> That just means the zip plugin has to be more carefully written than I
> thought.  It would have to be sandboxed and limited to prevent buffer
> overruns and zip bombs.

[...]

> Remember that zip, at least, would not be in the kernel.  I think what
> is going in the kernel is gzip and lzo, and it's being done so
> transparently that you never actually see the compressed version.

Doesn't matter if it is zip of some other compression, the problem is
exactly the same.

> > Can you illustrate for me with precise, clear, and unambiguous arguments

> That will take some time.  And some thinking.

See? Exactly what has been demanded by all the "unfair", "ReiserFS-racist",
"shove-any-junk-but-do-not-accept-perfect-filesystems-into-the-kernel" etc
people here on LKML from the very start.

> > how this can avoid all possible pitfalls,

> Especially if you want perfection.

Perfection would be nice, but (IMHO) not required.

[...]

> Now, the cryptocompress as it currently stands does not involve ... and
> does not introduce any new security holes in the way that you are
> describing.  There might be some issues with key management (someone
> hinted vaguely at that), but nothing insurmountable.

OK, I see a week of flamefest going by until you admit it has the same
problemas as compression (Hint: Most encryption compresses first, in order
to give would-be cryptoanalysts less of a toehold via non-uniform
distributions, and having less work to do), and then a whole lot of its
/own/ problems (that somebody can peek at a cached uncompressed copy of my
files is not so bad, if they can peek at my decrypted files I'd be rather
less pleased... and here you have to include a malicious root (Yes, I'm
paranoid. Doesn't mean root is not out to get me.).). Perhaps this can't be
all solved, but the exact boundaries of what /is/ provided, and what
/isn't/ must be made clear.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
> David Masover <ninja@slaphack.com> wrote:
> 
>>Kyle Moffett wrote:
>>
>>>On Jun 26, 2005, at 22:37:48, David Masover wrote:
> 
> 
> [...]
> 
> 
>>That just means the zip plugin has to be more carefully written than I
>>thought.  It would have to be sandboxed and limited to prevent buffer
>>overruns and zip bombs.
> 
> 
> [...]
> 
> 
>>Remember that zip, at least, would not be in the kernel.  I think what
>>is going in the kernel is gzip and lzo, and it's being done so
>>transparently that you never actually see the compressed version.
> 
> 
> Doesn't matter if it is zip of some other compression, the problem is
> exactly the same.

zlib is in the kernel.  Isn't lzo also in the kernel?

The problem is still the same, but not as bad.

And it's completely irrelevant to the current cryptocompress.  Saying
that someone could create a malicious transparently-compressed file that
blows up when decompressed (inflated?) is like saying that my /dev/hda5
is no longer trusted.  Because that's the only layer at which a userland
program can see the compressed version of a transparently-compressed file.

Now, if you're talking about going the other way, I think the problems
were less about buffer overflows and more about bombs where 1k
compressed data = 1 gig uncompressed data.  Going the other way is kind
of pointless if you're an attacker.

Unless there are buffer overflows in the kernel's zlib.  I hope not!
After all, I don't necessarily trust all the CDs that I try to mount!

>>>Can you illustrate for me with precise, clear, and unambiguous arguments
> 
> 
>>That will take some time.  And some thinking.
> 
> 
> See? Exactly what has been demanded by all the "unfair", "ReiserFS-racist",
> "shove-any-junk-but-do-not-accept-perfect-filesystems-into-the-kernel" etc
> people here on LKML from the very start.

Back at ya.

Many of the people I can't abide here are people who don't seem to
actually understand anywhere near as much of Reiser4 as I do.  And I've
never read much of its source code.

It would be nice if this was approached more along the lines of helping
get good stuff into the kernel than keeping anything bad out.  But
that's too much to ask, no sarcasm intended.  Didn't Linus say that his
job is more to keep stuff out anyway?


> [...]
> 
> 
>>Now, the cryptocompress as it currently stands does not involve ... and
>>does not introduce any new security holes in the way that you are
>>describing.  There might be some issues with key management (someone
>>hinted vaguely at that), but nothing insurmountable.
> 
> 
> OK, I see a week of flamefest going by until you admit it has the same
> problemas as compression

Did I admit that compression has problems?  I admitted that zip has
problems, not the planned cryptocompress plugin.

> and then a whole lot of its
> /own/ problems (that somebody can peek at a cached uncompressed copy of my
> files is not so bad, if they can peek at my decrypted files I'd be rather
> less pleased... and here you have to include a malicious root (Yes, I'm
> paranoid. Doesn't mean root is not out to get me.).). Perhaps this can't be
> all solved, but the exact boundaries of what /is/ provided, and what
> /isn't/ must be made clear.

Can't trust the kernel at all if root is malicious.  For that matter,
can't trust anything if root is malicious.

Or is there a system which blocks such things as keyloggers, kernel
modifications, eavesdropping on random ptys, or simply brute forcing
/dev/mem?  I don't see any existing solutions other than heavily patched
everything (text editors included) for even blocking simple temporary
file hijacking.

Now, if root is not malicious, we can just go back to UNIX permissions
to keep your files safe -- only now your stuff is safe if the FBI steals
your disk, and I assume your root can be trusted to power off the system
before they can touch it in that event?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+aM3gHNmZLgCUhAQLEGw//YgLNHfZNRBBD9wFd6Si0xnl+75eAwqOm
7WMDPdtXeZORhUnmNnS+EO71nMupUQmOaMI/AGbAJgTRHJKAiVKz1rt25TtMfkMg
tOeZ4PGOmI2eMe2Ltw8ocR6YDYLc/2VTLCE51pCvVswHPbcY2W+j1JHoIwo/y/3f
/WnO8QYNdGnvlYJB+smNEpO8ggwPItK5Ge2PoK1+3A+e+boX2xZyk3RzZ5Oth3Se
H8oW9wfNXoXp50BjVRXcCcOSbHiFFYWMRUD/i3izFwB3JNS523rMLjyJH/5zeciL
lW+b9dCjHqt0ULtkuw0gVbEQh4LTPBKM8WIKDRNkuFl9kz8FQk0BuQrpvmr8JZ3z
4S4etxhdnmuxMkXznJ0ioUTa+p7hXjRxpN1wZLXQi9xJfnOEkUHazI8GZui7qrlQ
UlRyxyZTezEfROk+Ova0DWfAJEV4qY2SktxXZeMr7Z2a8WdkdozfPOFHt1GG38c4
xc/L5z2maXAG0fQbZPZtrYi65ES5MQ472T6KNnwNb3nFJr1CRO0l4PpZa6kNLQ1G
XDKKpsPUR9A1/oOh81Ep1YN+RfJKWJCU3O59z1+ro3eKa1ckWEjX2TEwKIjeUaGd
B8n5FenOsSslHw1of1aVCRyNVUMFH2wTRf0CcIVBIMxZJ/RhJ6m2tqbOIsIK4V+H
rMEvogPEbtM=
=1+LH
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 898 bytes --]

On Sun, 26 Jun 2005 21:37:48 CDT, David Masover said:

> Assume we can do on-disk caching, similar to fscache/cachefs for nfs.
> Now, benchmark:
> 
> $ unzip linux-2.6.12.zip && make -C linux-2.6.12
> 
> versus the hypothetical
> 
> $ make -C linux-2.6.12.zip/.../contents
> 
> This is an automatic performance gain, in theory, because the second
> command is identical to unzipping just the parts you need into
> linux-2.6.12, then running "make".

Nope, they're not identical.  The first specifically unzips it into the file
system, leaving the zip file intact.  The second, you're having to take all
those .o files and other stuff that the 'make' generates and put them back
into the .zip file *on the fly* - when the 'make' is half done, the .zip should
reflect a directory tree that has had half the make execute....

(Think - after that hyptothetical 'make' completes, where is 'vmlinux'? ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Sun, 26 Jun 2005 21:37:48 CDT, David Masover said:
> 
> 
>>Assume we can do on-disk caching, similar to fscache/cachefs for nfs.
>>Now, benchmark:
>>
>>$ unzip linux-2.6.12.zip && make -C linux-2.6.12
>>
>>versus the hypothetical
>>
>>$ make -C linux-2.6.12.zip/.../contents
>>
>>This is an automatic performance gain, in theory, because the second
>>command is identical to unzipping just the parts you need into
>>linux-2.6.12, then running "make".
> 
> 
> Nope, they're not identical.  The first specifically unzips it into the file
> system, leaving the zip file intact.  The second, you're having to take all
> those .o files and other stuff that the 'make' generates and put them back
> into the .zip file *on the fly* - when the 'make' is half done, the .zip should
> reflect a directory tree that has had half the make execute....

This assumes we limit ourselves that way.  I doubt something I think of
in five minutes is the final API.

> (Think - after that hyptothetical 'make' completes, where is 'vmlinux'? ;)

In the cache where the stuff was unpacked to.

Anyone who wants to discuss things that will actually be done soon, or
anything relevant to merging reiser4 in the near future, can stop
reading now.

*If* we decide that this must go both ways, *then* we either turn off
write support inside the zipfile and do "make" with a symlink farm (cp
- -as isn't hard), or (better) we can set things up so that only on access
(most likely a read) of the original zipfile do we re-add all the changes.

I think copy-on-write files are planned sometime, too.  So one could
imagine doing a COW of the original zipfile, so that you have:

linux-2.6.12.zip	# original, for redistributing and patching
linux-2.6.12-mine.zip	# your own patchset, ready for redistributing

And, of course, there's always an option to treat the cache as its own
COW of the original zipfile.

This system (which I did discuss on "Silent Semantic") is also useful
for various other ways one would want to package something.  Imagine
someone's made a bootable iso which uses zisofs for transparent
compression.  You can set things up so that the iso->files plugin works
as zip does above, grabbing the files you need on the fly and putting
them in an on-disk cache, and files->iso takes only the files which have
changed and runs them through mkzftree, then takes the whole tree and
runs it through mkisofs.

This actually isn't a performance win at all, if you know what you're
doing.  You could always set up a script (as I did when playing with
this) which uses mkzftree's --crib-path option.

But, transparency is nice.  This way, all that's needed to build a new
image is to chroot into some_file.iso/.../contents and change whatever
you wanted, even use a package manager from the image, and when you're
done, "cdrecord some_file.iso" will build the image and burn it.

It also doesn't have to be lazy.  Files could be compressed as you go,
either in a blocking way or at a low priority, or they could be put off
until you actually want to burn the image.

I can imagine all kinds of packaging getting easier this way.  A
universal interface to peek inside of some sort of package, image, or
software distribution and change its contents, then build a modified
image.  I guess that makes it a sort of "make" plugin.




OK, here's an example where the "zip"-like behavior makes sense:

Suppose you buy a computer with a desktop Linux pre-installed.  But, on
your first boot, the system is entirely composed of one giant
copy-on-write tree, which refers to all the packages needed to build the
entire system.  All the packages in the standard distribution are
pre-loaded onto the hard drive as (say) .deb files.

In this situation, every time the user runs a program for the first
time, it unpacks from a (presumably compressed) package.  When a package
is entirely unpacked, it may be removed.  Since most users won't use a
majority of the software, there's a huge amount of disk space saved.
Yet when a user does want to use some random piece of software, they
simply run it, and it behaves as if it was already installed.

Download a package, and it can be run almost immediately after
"installation".  A bit of a slow startup, but maybe not as much as
trying to unpack absolutely everything in the package all at once and
get it set up.  For example, on Debian, the Foomatic PPDs all come in
one package, making it easy to find the right one from the CUPS web
interface -- but you only need a maximum of one PPD file per printer you
want to use, and there are hundreds of PPD files in the foomatic-ppds
package.  There are plenty of other programs (Mozilla, OpenOffice) which
install lots of components which you may never use, and certainly won't
use immediately on your very first run.

This way you save space and installation time.

I'm sure this was also discussed in "Silent Semantic", and I think it is
mostly not my idea.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+PQXgHNmZLgCUhAQLwoQ//f2vNSd4ZGBPjoaZBq8VlZ5l+wWCzinCF
h21YI5f7l3d1uGO/oo/E4AESrAFN0A410fNNfAVEtNPJy64SlrrLY/PMBuOB5w7k
83ZNLSqExdblA64FkPkTB6/l+g1fmhey2xAIRvTUrSyAwnbs6fYD6sNq2oVV/czR
G4t01iWyEGNyt3Ik41Pl/UqVCepHWcNNRUJL5eQYEakbp1hSwTU3NVkzwHeq/MJ/
OPP81Pq5NMfUUkH5h+G3/zghM44czSTPezC2Be/VS7vx0BvLWOSmS2qaPCogr35o
QJwksZqaUirHtVotG353vShTyOSXlkEfhelyEKrMp7V+dK9ChP4hJeNTrS4eAxsF
Wg25ZhFWN2D3ALdYdiG4dG2qFqSgR/mM2wwxzgumXI1YY1g0ppHDBtgRi8feXfjn
COB+G0w1cM2ofGz2v2eCZIps2cS/5bWG6eyjdFN/z+KnlO6vuDiQ/z9y1KwUKURs
u8+VLr9aWDNKuFisDbRoIc2Emq9sHUFNVG9WxrTY2Yd7RXWQfv1+gzvJt8FOwKVW
49L1jU2YLR31Kj6cc7HnHowzzVBTuZxJV9nnZkvpVQNVa/NFM2n44Og+Uvm06ikG
1JK2oLNgQGwRWW+UXJIBDqABYqh5COx4PMeRI3/cRPOxBLUdk34Hp+35PnvNSx3D
pffWHDlDaew=
=MtUX
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

On Mon, 27 Jun 2005 00:31:46 CDT, David Masover said:

> *If* we decide that this must go both ways, *then* we either turn off
> write support inside the zipfile

Oh, *that* will do wonders for command symmetry.  And you just shot down
the whole 'mv foo bar' being equivalent to 'zip bar foo' concept. ;)

>                                  and do "make" with a symlink farm (cp
> - -as isn't hard), or (better) we can set things up so that only on access
> (most likely a read) of the original zipfile do we re-add all the changes.

Those chuckleheads who have filled up a disk by saying 'tar cvf foo.tar .' just
got a whole new way to fill the disk... ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Mon, 27 Jun 2005 00:31:46 CDT, David Masover said:
> 
> 
>>*If* we decide that this must go both ways, *then* we either turn off
>>write support inside the zipfile
> 
> 
> Oh, *that* will do wonders for command symmetry.  And you just shot down
> the whole 'mv foo bar' being equivalent to 'zip bar foo' concept. ;)

In one of three possible settings for the imaginary zipfile plugin, yes.
 But if we're talking about a kernel source tree, how many of us
actually build zipfiles/tarballs of their kernel source trees, rather
than unpack existing ones?

>>                                 and do "make" with a symlink farm (cp
>>- -as isn't hard), or (better) we can set things up so that only on access
>>(most likely a read) of the original zipfile do we re-add all the changes.
> 
> 
> Those chuckleheads who have filled up a disk by saying 'tar cvf foo.tar .' just
> got a whole new way to fill the disk... ;)

Just a new interface :P

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+VYngHNmZLgCUhAQK/bQ/+MbWZrE+sg3te+YWCajaZhsZLyM4CZrGe
BrvUb3dLSKLLg+2CzsKXQp9u3bavJg7OEEmnzgY4dK3FaOHZSATzbwmRgi1hYtfe
TfVWKQb4z93zzS6MXafRkgL98qEIUV4n0aFKIEzDgeXAwMYx+Gv5Q7YMky7DEBNq
HV5ZiKxhsqCEJdsUKo80G2R12XBm9reunJM/xRt+iA7WfsevBi+/Fl7qbAIDG+sd
t83kZHiHQROTEmyyMj4zsJ0j7gRkJsXWKng3HEMl08AkRBIZxQRx0xmzWyZEs+wW
AaPPFvsFo04tfwplasvwZ7EcLoCgFP2rfnjx190wov6ZvEWwIbKepsyLqusEuRcR
hY0ohrWzYmQkHDXxs4FW+/QMsDF0L7YojHJgHOI1xSeCMOEgGPmHG9Jmf9FeK97Q
4xBXLDGXjlcDBZvDZciLzJCloH3pb+3TrzSkbIkTGnGbgtsS46bWRUAFOY1d2ynM
C1FTebc5LPGoDetsu9/cLLuJrCJbbmpN/AWR33WUOnu9/O+H0lWNhiYyOg0x2G/z
p6toqr95RPKO6tC5OliU7GfEpqjDhr+RQkjSUbrHqoNjZp3P07nCwGCfwMTCdsBx
AkpQyuyfn/RfZvaFOnpUSOTKCWfZKYB2/J8ij+FH8tcPpYl5uzb6H6mHNyEPlvAa
DOlFJixKQ7o=
=6LFS
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

On Mon, 27 Jun 2005 00:57:54 CDT, David Masover said:

> In one of three possible settings for the imaginary zipfile plugin, yes.
>  But if we're talking about a kernel source tree, how many of us
> actually build zipfiles/tarballs of their kernel source trees, rather
> than unpack existing ones?

I dunno.  I'll often build a tarball of "-mm plus local patches" known to
be working at the moment, precisely so I can just untar that as a known good
base for the next kernel-hackfest, rather than untar Linus's tree, apply all
of the -mm patch, then all my local patches again...

And even if I'm not *that* ambitious, I'll at least tar up a clean -mm tree
to use as a base. :)

And even if I didn't do that, you *do* have to do something when the disk
gets backed up.  You *do* intend for sensible things to happen then, right? ;)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Mon, 27 Jun 2005 00:57:54 CDT, David Masover said:
> 
> 
>>In one of three possible settings for the imaginary zipfile plugin, yes.
>> But if we're talking about a kernel source tree, how many of us
>>actually build zipfiles/tarballs of their kernel source trees, rather
>>than unpack existing ones?
> 
> 
> I dunno.  I'll often build a tarball of "-mm plus local patches" known to
> be working at the moment, precisely so I can just untar that as a known good
> base for the next kernel-hackfest, rather than untar Linus's tree, apply all
> of the -mm patch, then all my local patches again...

What you really want is a copy-on-write tree, or a simpe cp.  Or are you
that short on space that you need compression?

Speaking of which, I know copy-on-write files are planned.  What about
whole trees?

But, short of that, you can always do a poor man's copy-on-write with
"cp -al" or "cp -as", assuming you remember to copy when you write...

> And even if I'm not *that* ambitious, I'll at least tar up a clean -mm tree
> to use as a base. :)

I just keep a -mm patch around and the original vanilla tarball, but to
each his own.

> And even if I didn't do that, you *do* have to do something when the disk
> gets backed up.  You *do* intend for sensible things to happen then, right? ;)

I back up with rsync, actually.

Speaking of backup, that's another nice place for a plugin.  Imagine a
dump that didn't have to be of the entire FS, but rather an arbitrary
tree...  That might be a nice new archive format.  I know Apple already
uses something like this for their dmg packages.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+cTXgHNmZLgCUhAQIvvxAAk70+cz4yOZaMX6TDIuUWNsPjqM890FMa
jpNE5I2K3ZV91yJFAMdSZW1fQSQakorJyt0DV6wnxu3EbIZV8ATeIkNgJsqcDxAD
O6dZoIQnFZND90Fh3HdJb46DRZyml4NbeEKhQRzIAnuANIAa4+6it2aERcpbDNxE
ijOiVVpjTkUEutI2uuCJTYSXOGDa+rI0Jmth/9VnPAb5r+wqUb49wUziaSslnZs5
Jt6UlIcXU/OymzhWe+9JM0gydP+V2nP4N14oYoiXaqINPCA9OHV7BudClkKDeCyv
bYXyRfjmuiDNoOel6ZfURqUFR2GK/dPqI1PBV6Vc4UMsUqIKkGLAuQ5rU0csb39f
eS+6Dp8DJ7tOvQp73x1KTMJWP0lha1VRAj8s3SwdCp0Xar8YHymfCDjHvx/iJe50
rY5aZfWXGCzVmbKVETE8ACWeF5bgpnzwMDwU2RlaWhV/1yIZSOttuJFWMHNG4Rns
ajUrRsV9cmmngIJBMcM6XaZD9eRsJ9gb+E9POpsFbo6OwhS0IpRlzu/AGGoEiHSW
ZGNBoSDuLDluHP1jiXrnM7ONcFPEe2opUD7ltgSkYPKpya9C7lEfsEBzXh5Gqc5X
Iw8P0Md7U6rjCJYGb0pwnJQa0MYMimXq3hOTVsLdoj4A1Nn0EVntHzpB/QTe4UX2
NjMlwpPIXkU=
=fqm7
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

On Mon, 27 Jun 2005 01:27:25 CDT, David Masover said:

> I back up with rsync, actually.

Doesn't matter what it is.  You still need to define sane semantics for
it.. ;)

> Speaking of backup, that's another nice place for a plugin.  Imagine a
> dump that didn't have to be of the entire FS, but rather an arbitrary
> tree...  That might be a nice new archive format.  I know Apple already
> uses something like this for their dmg packages.

Hmm.. you mean like 'tar' or 'cpio' or 'pax' or 'rsync'? :) 

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Mon, 27 Jun 2005 01:27:25 CDT, David Masover said:

[...]

>>Speaking of backup, that's another nice place for a plugin.  Imagine a
>>dump that didn't have to be of the entire FS, but rather an arbitrary
>>tree...  That might be a nice new archive format.  I know Apple already
>>uses something like this for their dmg packages.
> 
> 
> Hmm.. you mean like 'tar' or 'cpio' or 'pax' or 'rsync'? :) 

No, a dmg is an OS X program installer.  It appears to be a disk image
of sorts.  So this is the backup idea in reverse.

They even "optimize" (repack? defrag?) the hard drive after each update.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+kIXgHNmZLgCUhAQKMbg/9GMCtXtlC76wYEvzALVQROXh5xTKHJg8q
+sngIMVKfCPOwGVe62tUfrzGaQ0/t9FJ/p40yAzRrWHV7msJWwQZYjf918AA1Vmg
IMmsbzFPH+3oRROSfwWcPB/MPG5n1YVfsaq09BUSFM7pasubGgEiFz3TA7gwRTh1
sOw/3J1mlhcxUUbQJrLPAe6e4u59h6MwUZVSdnF2D0Gnnxgwvl8ZemLVSqDaaMGs
F7fHJWUd7pZO+d4h2c9AnFLQQL5TvJHDOWuHcGVHZbt/Vaz7F79TWWyC74MeDUA6
ErSGbrZ4fLnocP8zqmDkWJ9GnqE/w9HAhDC2vPtRqCl4z8Mbc/0e5FOxi3M+7WVH
TXXBTKnBQG97DWEwneVRFndmhzP/rccnCYIlWhTugiATkStNkOKabTHVp2tpC7lB
VP+NwBsMFoaS3vvKcYMF9709OglLRdGmHoju87UIvyLLUZ1fYvZ2+7WlEez0cqSh
I0GTCq8gshagIAKhjR/vhVT0i20xyrS/oJEIYdtXz470E2S9tf1NvIwlokO153wL
6PaAN2okmC+YLG+lMeHDZs2lKDfmPzOtbzxR8XyEQHLxsbdWJBnm9B6Epa02iiE1
wZ8feMMsAOjzb/8iiZFqMRROe+j9m+beDLMdc3fYVrNQM72EyPAMw1zUr1GgXeVA
USJDqNWK5S4=
=BpTW
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[David Weinehall]

On Mon, Jun 27, 2005 at 02:00:49AM -0500, David Masover wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Valdis.Kletnieks@vt.edu wrote:
> > On Mon, 27 Jun 2005 01:27:25 CDT, David Masover said:
> 
> [...]
> 
> >>Speaking of backup, that's another nice place for a plugin.  Imagine a
> >>dump that didn't have to be of the entire FS, but rather an arbitrary
> >>tree...  That might be a nice new archive format.  I know Apple already
> >>uses something like this for their dmg packages.
> > 
> > 
> > Hmm.. you mean like 'tar' or 'cpio' or 'pax' or 'rsync'? :) 
> 
> No, a dmg is an OS X program installer.  It appears to be a disk image
> of sorts.  So this is the backup idea in reverse.

Yeah, disk images are really a new invention...  It's not like creating
an arbitrarily large solid file and then doing mkfs on it would
accomplish the same thing =)

> They even "optimize" (repack? defrag?) the hard drive after each update.

That's what they call the updating of the prelinking information,
AFAIK.


Regards: David Weinehall
-- 
 /) David Weinehall <tao@acc.umu.se> /) Northern lights wander      (\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\)  http://www.acc.umu.se/~tao/    (/   Full colour fire           (/

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Weinehall wrote:
> On Mon, Jun 27, 2005 at 02:00:49AM -0500, David Masover wrote:
> 
>>-----BEGIN PGP SIGNED MESSAGE-----
>>Hash: SHA1
>>
>>Valdis.Kletnieks@vt.edu wrote:
>>
>>>On Mon, 27 Jun 2005 01:27:25 CDT, David Masover said:
>>
>>[...]
>>
>>
>>>>Speaking of backup, that's another nice place for a plugin.  Imagine a
>>>>dump that didn't have to be of the entire FS, but rather an arbitrary
>>>>tree...  That might be a nice new archive format.  I know Apple already
>>>>uses something like this for their dmg packages.
>>>
>>>
>>>Hmm.. you mean like 'tar' or 'cpio' or 'pax' or 'rsync'? :) 
>>
>>No, a dmg is an OS X program installer.  It appears to be a disk image
>>of sorts.  So this is the backup idea in reverse.
> 
> 
> Yeah, disk images are really a new invention...  It's not like creating
> an arbitrarily large solid file and then doing mkfs on it would
> accomplish the same thing =)

Scroll up a little...

"Imagine a dump that didn't have to be of the entire FS, but rather an
arbitrary tree..."

Such a dump might produce the same thing as mkfs would, but it would do
it faster, maybe even copy-on-write.

The OS X example is to point out that using some chunk of an FS as an
archive format isn't necessarily a bad idea.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsAWa3gHNmZLgCUhAQJnRw//Sh0L38YZ/1fuxjo1TXQtfpMs2YpOIwSm
brJtx53rJD8nvAnOP1rwtWc5RiIl+YDzlWIO31L+U45Ap4B9+M4f7elqOlXXyrhw
HC/PIu3kai+MZYV05As8IaQNlU7fDPygnYVfIdFzz1NfwwiRo7TPXjjoPlkwaN8w
SluYj9CtwTUarNc7Nqct4gmmoxZ20YuyTmpQLyYkU4UMFmGcUfokhg03WUaFPGZr
bJL5TjW4YePyFtPcU53JcztjLD2z5pQrj1QdzK5hE2FM0UAM+0mWVEalkR19bNk3
z/lLEkfWBHP3gib9mdC5RsT/aik8nwRdr6X80WtPEMPwqAgSN+7x37u8Jp3paqLc
iQu54zVb7h9GJQfoi7sRCfO3GXxRKm3HDB8RnLZD39RKq3duA9M5hN+X9hMYVjd7
KE8kH387QvhLH7TwlfCyIQ2yY6/9abw4Jw9kdoBcX1pq5HpSf1s9TreA/w+89gan
W5NoIMo9VPR6Lhx9h6f7O07kxqVNdpjgw6w1LAGmgXKtlT/ojZ3lunXXPsjKknWN
M11XYiG6wBRNWJKsWsbh19O9FdcjpNknCxvrSLdiQDD44BTqc7l+Kt6dhACiyigs
BBMG3ceowL+vh52spCkakksqCIU4fb6+ndv2g43Q0d2Sk9tpCFUakzEugGfXrHYR
0qNwqhqbe2o=
=lP7Z
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On Mon, 27 Jun 2005 02:00:49 CDT, David Masover said:

> >>Speaking of backup, that's another nice place for a plugin.  Imagine a
> >>dump that didn't have to be of the entire FS, but rather an arbitrary
> >>tree...  That might be a nice new archive format.  I know Apple already
> >>uses something like this for their dmg packages.
> > Hmm.. you mean like 'tar' or 'cpio' or 'pax' or 'rsync'? :) 
> No, a dmg is an OS X program installer.  It appears to be a disk image
> of sorts.  So this is the backup idea in reverse.

I was addressing the ability to deal with an arbitrary tree.  By that definition,
a dmg, being a disk image and not a tree image, is *not* what you want....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Mon, 27 Jun 2005 02:00:49 CDT, David Masover said:
> 
> 
>>>>Speaking of backup, that's another nice place for a plugin.  Imagine a
>>>>dump that didn't have to be of the entire FS, but rather an arbitrary
>>>>tree...  That might be a nice new archive format.  I know Apple already
>>>>uses something like this for their dmg packages.
>>>
>>>Hmm.. you mean like 'tar' or 'cpio' or 'pax' or 'rsync'? :) 
>>
>>No, a dmg is an OS X program installer.  It appears to be a disk image
>>of sorts.  So this is the backup idea in reverse.
> 
> 
> I was addressing the ability to deal with an arbitrary tree.  By that definition,
> a dmg, being a disk image and not a tree image, is *not* what you want....

Not exactly what I want, no.

I was just trying to avoid the "people will never adopt a new archive
format" argument by pointing out that a similar archive format was
recently created and adopted.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsBEingHNmZLgCUhAQLTCw//XdrVgjaxvz9jzbiiSXVY60ItW5aechNV
JdvHBf1edVOCHs6OLil/cioK4Pjsfw3pKU+aBLN+eroS0RDomQkrqcy2zmFQ5JW6
oFJQEnb/uA95AhFtdCrbOF3rKABfJNo0TjwsRBuKyiabpYTGjfTJ0gDaQGAtBoOb
JmycKjZlJxhQgef9Q3LhG6UQaszCQrKX++pakKYaqOlughIpZ4O2AJTQWEq5ujdu
N9GNtO2DGd2sumWdxNpb0KSISZIs6PmPVthPsHwOvD0E6533q9a2AJH49j8AcuOz
1FCU7oFtpm994lwvc4G7eubRMbJ5Xgyy+suqfhTbumOgKzzBw8cKgpxkXFlcyDte
4WyNUlyIqAn0n9GAEVHWSDZ+vqN3E1u+bgLWJq0PEJJSjv9dJzhtH1WPu1bDA2v2
s1qNQDQromF+VfE1QF/fhy1Ttpqh7xAjBmxdxi+g3OU6Vlij7/j0fpMurIELknI0
MRFmw63TzMnFB07Vwi00L7ZR8GKSWDT0/smYF8R1V7stRqrAxHKDlT6E526ESMv+
ALa2pa1lGKWyCSgBMwPC0Cm9FBcOfaqxi60/5KE+bQOQc3Yc+20HCH5BIIQcNgWq
axOUO4txg9uz0GxfuvHWaa2tKE1vpbNO0Oy92QqdpCRmssyTNxs7EyhmdUB92Xar
SZVTSjpEmv0=
=WLzu
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

On Mon, 27 Jun 2005 13:25:14 CDT, David Masover said:

> I was just trying to avoid the "people will never adopt a new archive
> format" argument by pointing out that a similar archive format was
> recently created and adopted.

Out of curiosity, adopted by popular acclaim, or because an 800 pound gorilla
said "This is the format we're shipping <XYZ> in, learn to deal with it"?
(I've seen both happen multiple times in the last quarter century, on many
different operating systems....)

(For that matter, all of my production boxes are backed up by either Tivoli or
Legato, and I haven't a *clue* what format those tapes are in.  As a practical
matter, it doesn't really matter - after the first quarter petabyte or so of
backed up data, you're not going to do a restore without the software's help
anyhow.. ;)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

On Sun, 26 Jun 2005 21:37:48 CDT, David Masover said:

> > Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives.
> 
> So what?  I don't intend to convince anyone based on how much
> slower/faster their kernel compiles.  It's meant to illustrate the
> principle of the thing.

No, you seemed convinced that you'd have a big win based on the fact that
big chunks don't get unpacked - when in fact it's not as much of a win as
you might think.

And at least in the real world, performance *does* matter - if doing it the
traditional way is 3 times faster, nobody's going to be interested.

> Besides, your point was that you could not run make inside of a kernel
> tarball/zipfile.  Nobody ever suggested that you would actually want to.

"Here's a new facility.  Don't bother trying to actually use it".

Is that the message you're trying to send?

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Valdis.Kletnieks@vt.edu wrote:
> On Sun, 26 Jun 2005 21:37:48 CDT, David Masover said:
> 
> 
>>>Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives.
>>
>>So what?  I don't intend to convince anyone based on how much
>>slower/faster their kernel compiles.  It's meant to illustrate the
>>principle of the thing.
> 
> 
> No, you seemed convinced that you'd have a big win based on the fact that
> big chunks don't get unpacked - when in fact it's not as much of a win as
> you might think.

Not in this case.

> And at least in the real world, performance *does* matter - if doing it the
> traditional way is 3 times faster, nobody's going to be interested.

Not true.  I've said it before, and I'll say it again:

http://www.apple.com/macosx/features/spotlight/

Users *are* willing to trade speed for features, under the right
circumstances.  Tiger *is* noticeably slower.  People *do* make
webservers out of it anyway.

Or, for an extreme example:

http://www.doom3.com/

Doom 3 can be made to run on much older hardware (the Voodoo3, I think),
by disabling things like the dynamic lighting.  The old way of doing
lighting is probably more than 3 times faster, I know it's more than
twice as fast.  The new way looks much, much better, and many people
upgraded their hardware -- at least a couple hundred dollars upgrade,
sometimes an entirely new box -- in order to play this game.

For that matter, the old way of doing this kind of discussion is email,
the new way seems to be a PHP Wiki, which has got to be more of a load
on the server, but people use it anyway.

In the real world, killer features trump performance.

All of this is irrelevent.  The performance isn't that bad.  It may even
be better, even if it's not *much* better.

>>Besides, your point was that you could not run make inside of a kernel
>>tarball/zipfile.  Nobody ever suggested that you would actually want to.
> 
> 
> "Here's a new facility.  Don't bother trying to actually use it".

Not the facility.  The specific example.

> Is that the message you're trying to send?

I'm lacking a little imagination right now.  It's Sunday night, after all.

Actually, I think it might be a big win for Debian-like package
management systems.  That's also somewhere in the "Silent Semantic"
archives.

Here's another, somewhat better example:  modifying a zisofs CD image.
I'm going to go reply to your other mail now; I think the details belong
there.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+F3XgHNmZLgCUhAQJWDA/+NmbL7kCANpEEeV+4zKx3MUmABuoRRo6Y
hbRPDlBk9PCPM8GWQx+XwRh4XXb17FQklP7oIJHnnGtQelwDtcvbX8bSsDUTPd3s
Gt8a11GMAJkGcPAHh2oBtlGjslJj7O/eJRlj3eeYv5LIRxq/KEfvBt0wvx2W73uN
d8Mm65baeq7ipWDqKf0yQTcJT8M8kQQczO5gb9E5Q68fL+8rO7diQEOCm2A+Sp2G
rlKpvcQuhQ/P7IBOnX8ABUG4UphQfMqtICIwMmkLimLfjO+ID9pnO516K7O4erMI
Q2deFh6+KQ62Ngmokkh6GOu7VEmUD4XZubJpJknGWWBaKdCz12LsvHt2DeucKD5e
5mWi4oVwSInlxIRboso0CmRkoJZdEmqBJWFPtjXEtGGdufqJFySIHxPXpsqwRsa4
SuIIa2SK5vDXoc9URqKtPZtGkrGjG4VL1fxLNp9hD91hp+T5bLhsO+/8v2ZqfyRC
UvTKb9u3mqdu//UxJhhj0LTTjNv3eNc+2UQFhAg7uSpa7FZjV+nEvD5ZwlpimQL1
InDt+nd5wQWWQ+SfjbZNvdvE/XGfmMArT/Ik7tYv9LkSLp0ohwOVQw3fThPGY+BD
1SW0E5FLruAMh/cRl0mwIhDQ9CZO/jrKjRbueyC7CYFMjmyBLUxbt0vUIJqsB8rJ
P58IcgUaWFs=
=R57s
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

Valdis.Kletnieks@vt.edu wrote:

>>tarball/zipfile.  Nobody ever suggested that you would actually want to.
>>    
>>
> Besides, your point was that you could not run make inside of a kernel
>
Umm, try it when we have it working, on a 1-4GB RAM machine it might not
be so bad.....  We have the compression (albeit still with bugs) but not
the tar plugin.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
> Valdis.Kletnieks@vt.edu wrote:
> 
> 
>>>tarball/zipfile.  Nobody ever suggested that you would actually want to.
>>>   
>>>
>>
>>Besides, your point was that you could not run make inside of a kernel
>>
> 
> Umm, try it when we have it working, on a 1-4GB RAM machine it might not
> be so bad.....  We have the compression (albeit still with bugs) but not
> the tar plugin.

We *desperately* need either more plugin-like "plugins" or a FUSE-like
way of writing plugins.  I mean, tar is small enough, but I don't want
you chipping away at my (soon-to-be) 2 gigs of desktop RAM -- I need
that for Doom 3!

Or, do you mean using the RAM for keeping the working copy of the
zipfile?  And here I was trying to establish that it was better to cache
this.  If the cache appears as a separate Reiser tree, it's not much
different than keeping it in RAM anyway, but it can be flushed if I need
the space for... work... ;)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+WdngHNmZLgCUhAQLYrw//V8z1ywRTyXk7zaHU+COPO2PCV8kKXtWJ
7BkDXm65cVCUVNN6hgiJxMGEx7fe8ZWNJRHhvdIWGt5G68qI6G0F/tYQajtT59mk
E7AF2WWJ2ip0E0WKzBVVYKYnaeB+w9rF/8RJ/0Zj7C3zQ+fzvmVCGw6y9bLWYQle
VCUYjSMKaQz1cU4RqtDd+pIgr9w5/kxfhd30Aj9nK2qaQZLSxeFTv7p9c2vW5ZwM
YeQG8K4EZ0gO1GH0e85BAa9McLnDurosxeeLumaPqbIbFzc+1YXy9+5PsXjPKGDN
IjKVm/R0HpPP9KpCLdpRvp3GyWz1EuYSp9zpUEh5mDPH/gOSgRCva3Vx/2TW+YTk
3U73tHbTI5b9Mf/dlK3GPAoJ1krRm83sNMMAyGaTv+n6kRVWaSWIelZrVEJ5/jHO
pgY+5mDJwROlXdDqO1CV9PJHx4DivyODOrKzFnIykfQ2MO1vs1+Fy94LyYGkrPGd
Nq6y6XGxjx5MZ1fX3vJfovX0z+eR9s3R1uIgCMXlVoELEh9XKoFGGWIhrb7MhUdK
Q+n2ljixMZhIakeEUflhycZw0CY8jlgTK+vULUS7mDhOExVxombdAalVhhQ3cK0+
ZasRPyS3A3mCE7U6AKt8oc1AjRpdNIR1YKmfh+C44Yy+2e2wIWAoIb4mg0uY6jEa
GJl0uZIj4rY=
=uAm0
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Sun, 26 Jun 2005 17:35:48 CDT, David Masover said:
> 
> 
>>>Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you.
>>
>>In that example (shouldn't have used the name "crypto", but oh well), it
>>should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is
>>the gzip'ed file and foo is the transparently compressed/decompressed
>>file.  Basically, these are equivalent:
>>
>>$ zcat crypto/raw/foo.gz
>>$ cat crypto/inflated/foo
> 
> 
> I'm *quite* aware of what your preconceived notions think it *should* be.

So, by "what would ... give you", you meant what the benefits are?  I
was just clarifying how they work, I *thought* that's what you meant...

> Maybe the two examples I asked for have *real-world* meanings that you should
> be allowing for.  Like, for instance, on a mail server, where the A/V software
> may need to unzip a file 5 or 6 times to find out if there's malicious content.

Why would it want to unzip the *same* file that many times?

If you're talking about nested zipfiles, we can do nested plugins just
as easily.

> Or seeing if it's a ".zip bomb", where a small .zip will decompress to gigabytes.

> Or I'm testing a new compression algorithm, to see if multiple compressions help
> (yes, I know that it *shouldn't* help - but I've seen real-world cases where the
> algorithm could only look at a 4K or 8K window at a time, and if you hit a *very*
> long run of duplicate 4K segments, a second compression would compress all the
> identical or near-identical "this is a 4K chunk" tokens...)

Tune the plugin, or use zip directly.  I'm not proposing necessarily to
embed zip in the kernel and drop it from userland, just to have a kernel
interface so that apps/people don't have to all know about the zip
program and how to use it.

Besides, in the zip example, I made very sure that unless you point a
program specifically at where the zip plugin unpacks stuff, you can
easily get the zipfile.

Of course, that could be turned on its head if it was more low-level
transparent compression, the kind where you just compress everything in
/etc because it's all tiny text files.  In that case, you would want a
setup where you have to work to get the raw data.  But in that case,
it's a different kind of plugin anyway -- a plugin for storing data,
rather than just accessing it.

Stop worrying about automagic stuff, though.  It won't trigger unless
you go looking for it.  It will just be easier to find if you do.

>>I guess I need a new name for this approach.  That's three possible ways
>>of doing this?
> 
> 
> I *said* you need to think this through in detail, didn't I? ;)

I said we already did, which is true.  I don't think I'm making these
up, I just don't remember where they were in Silent Semantic.


(oops, I fired the last one off before I was done...)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr9pUXgHNmZLgCUhAQKQ8w/+OQXWdt1pEF6t/0D+x3mPYWq2lhlUbchZ
BYywcGN6Z/yFZQrZovJABZVnB+CXlUlx8DGqQN+Lj9+8HLko/P75ErTWHfuiscpS
wRJIN5dVeZ4f1RImEa8PjQf3c+n+B/ft9hq28yaR58C9vBQwAWOPVL0/4n4unNAV
49odg/IKJM9sdSF+6sVwgWNfuacRcZ5/jXtkDFXIIyJzKl6r45r3HmTkbgRdxqpU
p1aOIsXa7fciN+UK+eiw5jruzJsKHaJtBdISsMWbPdVBFjsDVZmhsdOMv2RZMPo6
2V78OQ4g7pJHSMZsaWQ/vvD3PuHxZm9qglJcdGnL6jNk8OXkKzxXaGDn/SHjYFTu
c8keR1KYu+U5r5RmTiihavFPpN3zefS5W5o5IyLvEZkApRCngDFitq4t2tRfDbfT
zV1JZWz7/bqtoY0zdaZ3gFwxXxh8FMw/LCnsjKkBQ7etlvnvcW2IPssd9rTIV0+4
9CmA2EaYIiXAwGoFzLbbPoKf0a+6tB38oHanBrRBZuTzu9mKHZvyefQa0j2+1KOp
iiCmrK49/APXfp4IUu284r2dPUqAu33LtomxwFbNLaPDZFWgQ9qPjafJBO3L8Y/9
HlFe46Jo5tL7YAEq/UyKpDYWxVcUiDZ40z2w/WKq+v9JjmUZMm+jonXHa9Nq0cQn
YlE9bxtEac8=
=5eSh
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hubert Chan]

On Sun, 26 Jun 2005 20:40:29 -0400, Valdis.Kletnieks@vt.edu said:

> On Sun, 26 Jun 2005 17:35:48 CDT, David Masover said:

>> The kernel does not decide that.  You do.  And it doesn't
>> automatically decide that every time you create a file.  You have to
>> use some interface to trigger the plugins.

> Oh, I'm waiting for the fun the first time somebody deploys a plugin
> that has similar semantics to 'chmod g+s dirname/' ;)

Reiser4 plugins have to be compiled into the kernel.  (They're not
plugins in the sense that most people use that word.)  And any admin who
would compile that kind of plugin into the kernel needs to have his head
examined.  Not to mention that plugins must first go through Hans and/or
Linus before they can get included into the kernel.

The kernel defines the set of plugins available to the user.  The user
selects (to a certain degree) which plugins to use.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 3125 bytes --]

On Sun, 26 Jun 2005 23:10:43 EDT, Hubert Chan said:
> On Sun, 26 Jun 2005 20:40:29 -0400, Valdis.Kletnieks@vt.edu said:

> > Oh, I'm waiting for the fun the first time somebody deploys a plugin
> > that has similar semantics to 'chmod g+s dirname/' ;)

(You *did* notice it was set-GID of a *directory* not an executable file,
right?)

> Reiser4 plugins have to be compiled into the kernel.  (They're not
> plugins in the sense that most people use that word.)  And any admin who
> would compile that kind of plugin into the kernel needs to have his head

Oh?  You saying that it *wont* be permitted for a user to say:

mkdir $HOME/zipped
chattr "files under here are ZIP files" $HOME/zipped

and instead you have to do that chattr by hand for *every* *single* zip file?

Or "files on this filesystem are encrypted by default"?

I suspect that this sort of thing is going to be one of the *first* things
that will get created, and any admin who tries to sell this idea to the users
*without* that sort of functionality will be handed their head.

Or, if "that type of plugin.. needs to have their head examimed", I suggest
that you go to your kernel source tree, find fs/ext3/ialloc.c, and this code
in ext3_new_inode():

        if (test_opt (sb, GRPID))
                inode->i_gid = dir->i_gid;
        else if (dir->i_mode & S_ISGID) {
                inode->i_gid = dir->i_gid;
                if (S_ISDIR(mode))
                        mode |= S_ISGID;
        } else
                inode->i_gid = current->fsgid;

and #ifdef out all but the last line, and see if anything breaks. ;)

> examined.  Not to mention that plugins must first go through Hans and/or
> Linus before they can get included into the kernel.
> 
> The kernel defines the set of plugins available to the user.  The user
> selects (to a certain degree) which plugins to use.

The point you missed was that plugins *will* have interactions, and as
the guys who are working on a stacker for LSM modules have found out the
hard way, trying to deal with the composition of functions is fiendishly
difficult.

And notice that it doesn't *have* to be quite so obvious - how about if a
user creates a directory $HOME/zipped/ and flags it as "anything under here
is a zipped file".

Now throw in multiple users and CPU limits.  User A enters that directory and
references everything, causing the buffer cache to get filled up.  While there,
A makes changes, so the pages are dirty - "for i in */*; do echo " " >> $i; done"
would do the job...  User B now does something that causes a writeback of one
of those buffer cache pages.

A) What process currently gets ticked for the CPU and I/O for the writeback?

B) In your model, who will get ticked for the resources?

C) Will the users riot? (Note that you can't win here - currently, the "price"
of writing back A's and B's pages are about equal.  However, if A gets dinked
for an expensive writeback due to B's process, A will get miffed.  If B gets
charge for an expensive writeback of A's, B will get miffed. If you say "screw it"
and bill it to a kernel thread, the bean counters will get miffed... ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Sun, 26 Jun 2005 23:10:43 EDT, Hubert Chan said:
> 
>>On Sun, 26 Jun 2005 20:40:29 -0400, Valdis.Kletnieks@vt.edu said:
> 
> 
>>Reiser4 plugins have to be compiled into the kernel.  (They're not
>>plugins in the sense that most people use that word.)  And any admin who
>>would compile that kind of plugin into the kernel needs to have his head
> 
> 
> Oh?  You saying that it *wont* be permitted for a user to say:
> 
> mkdir $HOME/zipped
> chattr "files under here are ZIP files" $HOME/zipped
> 
> and instead you have to do that chattr by hand for *every* *single* zip file?

chown works this way.  At least for username, there's no default user
possible per directory.  But scroll down a bit...

> Or "files on this filesystem are encrypted by default"?
> 
> I suspect that this sort of thing is going to be one of the *first* things
> that will get created, and any admin who tries to sell this idea to the users
> *without* that sort of functionality will be handed their head.

I think it may be possible to set defaults to a certain extent, but I
think that it's still actually a per-file setting whether the file is
encrypted.  Obviously, the keys have to be shared somehow, so there's
probably more grouping than just defaults...

There has been some mention of inheritance, but I've forgotten how
that's supposed to work.  If there's some sort of inheritance where
children inherit properties of their parent directory, and also inherit
changes to those properties, than Hans probably wants that to be the
prefered way of doing things?

I don't think there's that kind of inheritance, though.

> And notice that it doesn't *have* to be quite so obvious - how about if a
> user creates a directory $HOME/zipped/ and flags it as "anything under here
> is a zipped file".
> 
> Now throw in multiple users and CPU limits.  User A enters that directory and
> references everything, causing the buffer cache to get filled up.  While there,
> A makes changes, so the pages are dirty - "for i in */*; do echo " " >> $i; done"
> would do the job...  User B now does something that causes a writeback of one
> of those buffer cache pages.
> 
> A) What process currently gets ticked for the CPU and I/O for the writeback?
> 
> B) In your model, who will get ticked for the resources?
> 
> C) Will the users riot? (Note that you can't win here - currently, the "price"
> of writing back A's and B's pages are about equal.  However, if A gets dinked
> for an expensive writeback due to B's process, A will get miffed.  If B gets
> charge for an expensive writeback of A's, B will get miffed. If you say "screw it"
> and bill it to a kernel thread, the bean counters will get miffed... ;)

If I understand this correctly, this is somewhat like if user A creates
a 50 meg file on a system with 100 megs free RAM, and user B runs
"sync".  Also similar to if B were to suddenly fill up 75 megs of RAM,
forcing A's file to be flushed -- last I checked, in Reiser4, only a
sync or memory pressure causes writes to flush.

Right?  This is tempting to comment on, but I want to make sure I grok
it first...

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+UiHgHNmZLgCUhAQK+KBAAm6BSQuoFNeY5+cPQQaVK2BACqwJZabMH
Ze440Hjf9Pgn6Is/xWbCKKv6Mx4Vfx5P4+E4dkZJOyFBaVym6v5wy7ovPhFZVD8f
oW8IOUnCngnQ/Ea7fhxwr+hst2L4jEbMF0deG3zg328zYKwHoY0NA7hQZg2xLhF6
+J5jQWNR+CWyhFCBMD/NG+XwtSd4pxzOjb42e7zlEuKoCgGiTB92qPGDcOYMw/Te
3ez1Z+iJ6gIMUgwrzO4J6TzsgR9d9W0rJKMpm6ulto0AQtg1Joln3Vj5pxBX6ULe
5hkV6zeNOW8Uisz6+tdUX6PC+hjfSPvJhzkPMccTm8cJGyiF+j+PI5nUj37hyAz6
iL8kBPMrsulrGphuJzeb81yLzmgknX4+tc6WrVqMPcCpP4iIYOi0RMpWxSxQuRH9
ooSUnN/JRuhHaz4JeJ6VOvkwwl4nw1dcChBnBiws4IlFQS+mgKTAzZHhHm/+/E6q
nHY/uiFo1Tr95wyxrWyNcdGA/axriSIAaCXc1bEpQpzpOOcXr5d437TwEhBTIeXr
UlXwM6CdSrA3XGy7ksHovRghdm/7MOmNxL4SagVvY98Dlc+UlWi755rpin7JRget
9tFGtH0IESmPwSXfsBJYNIZCk4bG8X9d2W1/6e96yVtvKdn16MgX6n3Hdt3aVgzv
Ec9kEA42Ixs=
=qALY
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 2675 bytes --]

On Mon, 27 Jun 2005 00:54:17 CDT, David Masover said:

> There has been some mention of inheritance, but I've forgotten how
> that's supposed to work.  If there's some sort of inheritance where
> children inherit properties of their parent directory, and also inherit
> changes to those properties, than Hans probably wants that to be the
> prefered way of doing things?

Well, the 'chmod g+s dirname/' example *is* just "children inherit the
group of the directory", and somebody didn't like that.. ;)

> > Now throw in multiple users and CPU limits.  User A enters that directory and
> > references everything, causing the buffer cache to get filled up.  While there,
> > A makes changes, so the pages are dirty - "for i in */*; do echo " " >> $i; done"
> > would do the job...  User B now does something that causes a writeback of one
> > of those buffer cache pages.
> > 
> > A) What process currently gets ticked for the CPU and I/O for the writeback?
> > 
> > B) In your model, who will get ticked for the resources?
> > 
> > C) Will the users riot? (Note that you can't win here - currently, the "price"
> > of writing back A's and B's pages are about equal.  However, if A gets dinked
> > for an expensive writeback due to B's process, A will get miffed.  If B gets
> > charge for an expensive writeback of A's, B will get miffed. If you say "screw it"
> > and bill it to a kernel thread, the bean counters will get miffed... ;)
> 
> If I understand this correctly, this is somewhat like if user A creates
> a 50 meg file on a system with 100 megs free RAM, and user B runs
> "sync".  Also similar to if B were to suddenly fill up 75 megs of RAM,
> forcing A's file to be flushed -- last I checked, in Reiser4, only a
> sync or memory pressure causes writes to flush.

Exactly the same sort of thing - traditionally it's been more or less ignored
in the system accounting, because A would usually average out to causing as
many I/Os as B did, and they were roughly equal in cost so it was a wash.
However, if one user has a much higher per-page cost than the other, the
imbalance can start to matter *very* quickly....

> Right?  This is tempting to comment on, but I want to make sure I grok
> it first...

For more fun, consider how you can write 1 megabyte of data to a file,
lseek to the beginning and start writing again - and you go over quota
on the *second* write even though you're over-writing already existing
data.  Can happen if you're compressing, and the second write doesn't
compress as well as the first. (To be fair, we already have similar
issues with sparse files - but at least 'tar --sparse' has an easy way
to deal with it compared to this. ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Mon, 27 Jun 2005 00:54:17 CDT, David Masover said:
> 
> 
>>There has been some mention of inheritance, but I've forgotten how
>>that's supposed to work.  If there's some sort of inheritance where
>>children inherit properties of their parent directory, and also inherit
>>changes to those properties, than Hans probably wants that to be the
>>prefered way of doing things?
> 
> 
> Well, the 'chmod g+s dirname/' example *is* just "children inherit the
> group of the directory", and somebody didn't like that.. ;)

Respectfully ignoring this until he can comment.  I suspect he made a
mistake, but I'm trying to avoid putting words in his mouth...

>>>Now throw in multiple users and CPU limits.  User A enters that directory and
>>>references everything, causing the buffer cache to get filled up.  While there,
>>>A makes changes, so the pages are dirty - "for i in */*; do echo " " >> $i; done"
>>>would do the job...  User B now does something that causes a writeback of one
>>>of those buffer cache pages.
>>>
>>>A) What process currently gets ticked for the CPU and I/O for the writeback?
>>>
>>>B) In your model, who will get ticked for the resources?
>>>
>>>C) Will the users riot? (Note that you can't win here - currently, the "price"
>>>of writing back A's and B's pages are about equal.  However, if A gets dinked
>>>for an expensive writeback due to B's process, A will get miffed.  If B gets
>>>charge for an expensive writeback of A's, B will get miffed. If you say "screw it"
>>>and bill it to a kernel thread, the bean counters will get miffed... ;)
>>
>>If I understand this correctly, this is somewhat like if user A creates
>>a 50 meg file on a system with 100 megs free RAM, and user B runs
>>"sync".  Also similar to if B were to suddenly fill up 75 megs of RAM,
>>forcing A's file to be flushed -- last I checked, in Reiser4, only a
>>sync or memory pressure causes writes to flush.
> 
> 
> Exactly the same sort of thing - traditionally it's been more or less ignored
> in the system accounting, because A would usually average out to causing as
> many I/Os as B did, and they were roughly equal in cost so it was a wash.

Even if A is doing A/V work and B is programming?

> However, if one user has a much higher per-page cost than the other, the
> imbalance can start to matter *very* quickly....

I see.  My instinct is to charge A if B just caused a sync, but change B
if B was automagically forcing A to do something...

But I still don't understand the layers involved well enough.

>>Right?  This is tempting to comment on, but I want to make sure I grok
>>it first...
> 
> 
> For more fun, consider how you can write 1 megabyte of data to a file,
> lseek to the beginning and start writing again - and you go over quota
> on the *second* write even though you're over-writing already existing
> data.  Can happen if you're compressing, and the second write doesn't
> compress as well as the first. (To be fair, we already have similar
> issues with sparse files - but at least 'tar --sparse' has an easy way
> to deal with it compared to this. ;)

To be even more fair, but possibly launch another flame, Reiser4
reserves 5% of disk space by default to avoid this kind of thing.  But
that's talking about total disk, not quotas.

Also doesn't answer my question, but it seems I'm starting to get it.

How do we get over quota errors, btw?  Can we get them from write()
calls?  If so, I don't see a Problem(TM), just an annoyance.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+lwngHNmZLgCUhAQKYUw/9HmIZRqL7+1qcji/y4EhgUMftbdaKHUaJ
D+gyssdIgkfNwAR5vEvE6Lb0Fdj4zWU12nnx/ppL0nU1n6pWOBqyOMMtbHs4dbwe
iTJmoe1OCBM5uOS8bSdYUfVFUWNRF5+MPc8cWcpBa1LPCBmpzUnr5hS8iuNknWl2
7r6Hs64GSJ1JI7djE0P8fZVTou95azc/1pRU4kRtlsavXMmmAL3sefA9HS4b85VB
haOuNPmDIhJhPXhAMY6ZYKq3xnVyGWuiU6Z1Clv8JyP0Y7jaqGjF+V16zVSKvJH8
d0BrN05zGL1CIYDQrLtC0CQHuuLek34NDYimgVSLhkSB73GJ/MHxfO++68Pnhxhz
NOF86tPhoDQHhdEkUw0Kq5ZeI43ETejbMRIMoPAhuWo1vCQNFsfsSLljWLatBGNr
Lzgb0Q74Iqnrw9fPwrQttA/aPdlnVmOGv1qVYz/LQ8C+UHPVLXkjm5NVzHFDesPS
XXRZxSxPcOcMK9zPxLNer3zPlGPNOjQbylfL6+RwcJDmHq3IrDJDLY4TB4cZzJpc
gJyXrpSjprYCbSi+cc1d1R4cLcbdMHjR/rEszibLyeMuINYyx6vJklKy9xvpmyEB
woGrADFDnmKyv4WOzswSgIM6s8U2dLY4GFM/GW7KLx+rMF98UDhGyLjvSyvIpigs
BJz4T03qCtA=
=GoJz
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]

On Mon, 27 Jun 2005 02:07:46 CDT, David Masover said:
> > Exactly the same sort of thing - traditionally it's been more or less ignored
> > in the system accounting, because A would usually average out to causing as
> > many I/Os as B did, and they were roughly equal in cost so it was a wash.
> 
> Even if A is doing A/V work and B is programming?

I said "traditionally" - it's been a "oh well, we can't do much about it"
problem for a *long* time (for instance, time spent in an interrupt handler
has usually been charged off against whoever's timeslide the interrupt handler
took a chunk out of).  It's only been tolerated so far because (a) the costs
for both users are about equal and (b) you rarely have a heavy I/O DB and a
number cruncher on the same box, or a user doing A/V work and a user doing
programming - if it's not a single-use machine, there's *multiple* number
crunchers, DBs, or programmers, and they tend to balance out.

Said tendency can dissapear quite easily here....

> How do we get over quota errors, btw?  Can we get them from write()
> calls?  If so, I don't see a Problem(TM), just an annoyance.

One gotcha here is that it means that you can't do delayed allocation on
writes - you *have* to allocate disk space at each write and then update
the quotas. (And yes, I know that 'man 2 close' says that bad stuff can
happen to your data even after your program exits - that doesn't mean we
should go out of our way to make things worse.. ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[Artem B. Bityuckiy]

Valdis.Kletnieks@vt.edu wrote:
> For more fun, consider how you can write 1 megabyte of data to a file,
> lseek to the beginning and start writing again - and you go over quota
> on the *second* write even though you're over-writing already existing
> data.  Can happen if you're compressing, and the second write doesn't
> compress as well as the first. (To be fair, we already have similar
> issues with sparse files - but at least 'tar --sparse' has an easy way
> to deal with it compared to this. ;)
Sorry, may be I'm out of the context, but here is my view.

In case of compression in the kernel space you may take into account the 
size of file in the _uncompressed_ form and how much it takes in the 
compressed form - doesn't matter. So, no problems with rewriting.

Moreover, are you sure that the current quota model is enough for FSes 
with on-fligh compression? The model should probably be extended/changed.

Problems with quota and accounting is not the reason to forbid the 
on-flight compression. And they don't have to co-exist well.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: reiser4 plugins

[Hubert Chan]

On Mon, 27 Jun 2005 00:59:39 -0400, Valdis.Kletnieks@vt.edu said:

> On Sun, 26 Jun 2005 23:10:43 EDT, Hubert Chan said:
>> On Sun, 26 Jun 2005 20:40:29 -0400, Valdis.Kletnieks@vt.edu said:

>> Oh, I'm waiting for the fun the first time somebody deploys a
>> plugin that has similar semantics to 'chmod g+s dirname/' ;)

> (You *did* notice it was set-GID of a *directory* not an executable
> file, right?)

>> Reiser4 plugins have to be compiled into the kernel.  (They're not
>> plugins in the sense that most people use that word.)  And any admin
>> who would compile that kind of plugin into the kernel needs to have
>> his head

> Oh?  You saying that it *wont* be permitted for a user to say:

[...]

Sorry.  I completely misunderstood what you were getting at.  I thought
that you were implying that with the right kind of plugin, users would
be able to get around security measures.  Of course, this is probably
true, which means that plugins need to be coded carefully (as with
everything else in the kernel).  My point that I was trying to make was
that users can't install arbitrary plugins, and the set of plugins
available is controlled by the administrator.

Since that wasn't what you were talking about, I retract my previous
email.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[Hans Reiser]

Valdis.Kletnieks@vt.edu wrote:

> (Hint - work out how long a kernel 'make' would take
>if you were doing it inside a .tar.bz2).
>  
>
After the first time, not very long, if you had enough ram....  the
plugin would keep the data uncompressed until it flushed it to disk.

Performance might even improve since less would be written to disk.

Re: reiser4 plugins

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

On Sun, 26 Jun 2005 15:54:25 PDT, Hans Reiser said:
> Valdis.Kletnieks@vt.edu wrote:
> 
> > (Hint - work out how long a kernel 'make' would take
> >if you were doing it inside a .tar.bz2).
> >  
> >
> After the first time, not very long, if you had enough ram....  the
> plugin would keep the data uncompressed until it flushed it to disk.

You're not allowed to use current existing stuff like the disk buffer cache
to weasel your way out on this one.  "if you had enough ram" has been true
for decades.  The trouble is that quite often you *don't* have enough ram....
 
> Performance might even improve since less would be written to disk.

I've worked with filesystems where performance improves due to compression
(AIX's JFS).  It's a lot harder to provide an improvement if you're writing
37 more bytes in between bytes 399457 and 399458.... (I suppose by aligning
byte 399458 so it actually is on the start of a 4K block you can do that, but
then you're losing the advantages of the compression.. ;)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: reiser4 plugins

[Thomas Rösner]

Hi,

I hereby throw myself into the neverending flames called "reiser4
plugins" - may it please the gods.

David Masover (who really could use pgp mime attatchments):
> So, the API becomes something like:
> 
> cat crypto/inflated/foo         # transparently decompressed
> cat crypto/raw/foo.gz           # raw, gzip-compressed

Wee, I've seen this before. Wait, its AVFS! But didn't AFVS work with
any underlying fs? Even nfs?

cd foo.zip#/bar/
cat foo.tgz#/bar/plonk.txt

There is no reason why there shouldn't be a gpg AVFS-plugin, too. OK,
AVFS worked via coda's redir.o and IMHO we need smth like this in 2.6.
But, apart from more efficient compression on disk (if that really
ever works) you didn't show any convincing use cases.

Or am I fundamentally wrong?

regards,
  Thomas

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lincoln Dale wrote:
> Hans Reiser wrote:
> 
>> There has been no response to the technical email asking for what
>> exactly it is that is duplicative, and what exactly it is that ought to
>> be changed in how the code works, as opposed to what file the code is
>> placed in, or who is considered its maintainer.    There has been no
>> response to the questions about whether the difference between class and
>> instance makes our layer non-duplicative.
>>
>> Perhaps no response was possible?
>>  
>>
> Hans,
> 
> the l-k community have asked YOU may times.  any lack of response isn't
> because of the kernel cabal .. its because YOU are refusing to entertain
> any notion that what Reiser4 has implemented is unpalatable to the
> kernel community.
> 
> you can threaten all you want to take your code and move it to BSD.  or
> fork the kernel.  whatever.
> but if you want to get your work into the mainline kernel, you need to
> provide answers to the question that keeps being asked of you - and
> which you patently keep ignoring time & time again.
> 
> in short,  as per Message-ID: <42BBC710.8010906@cisco.com>:
>    posting to l-k on "why Reiser4 cannot use VFS infrastructure for
>    [crypto,compression,blahblah] plugins" - ideally, for each plugin.
> 
> or again, in Message-ID: <42BBB1FA.7070400@cisco.com>:
>    [..] but instead just understand that this is the framework that you
> have to
>    work in to get it into the mainline kernel.
>    if you don't want to go down that path, you're free to do so.
>    its open source, you can keep your own linux-kernel fork.
>    but if you want to get your code into mainline, i don't think its
>    so much a case of l-k folks telling you how to make your code
>    work under VFS.  the onus is on you to say WHY your code
>    and plugins could never be put into VFS.
> 
> or further back in Message-ID: <42BB7083.2070107@cisco.com>
>    you know that VFS is the mechanism in Linux.  you know
>    (i hope..) how it works.  it isn't so hard to see how many
>    of the Reiser4 "plug-ins" could be tied into VFS calls.
>    OR, if they cannot TODAY, propose how VFS _COULD_ be
>    made to do this.
> 
> 
> NB. it doesn't matter what David thinks.  this is linux-kernel, not
> linux-users.

Gee, thanks.  Great attitude.  I'm sure your users love you.
By the way, Groklaw is run by a user.

Ok, I'll bite.  Hans put it best a moment ago:
"Because our code is 90% library routines (aka plugins), eliminating the
plugin layer is like eliminating main() in a user space program."

We want to get a feeling of exactly which plugins you are talking about.
 Because if we pick the wrong ones and spend a few weeks implementing
them, and come back and you think we're still too "layered" or maybe
we've "bloated" VFS too much, and maybe the cycle repeats for another
two years...

What Hans seems to be worried about is having food and rent for two
years, and the fact that in those two years, the XFS guys or the ext3
guys may duplicate enough of the "plugin" architecture that he gets no
credit anymore, AND has to rewrite his stuff to work with theirs.

I have to be neutral on that, as I realize that many of the linux-kernel
people are sick of hearing it and honestly don't care.  And to tell the
truth, I'm a little sick of hearing it, and more than a little sick of
the personal attacks from both sides.

No, as a user, I just want a working plugin architecture to play with
(I'm not *just* a user), and a working Reiser4 'cause it's fast, and I
am eager to see new improvements coming out of Namesys, instead of two
years spent trying to keep up with the vanilla kernel *and* adapt to
some unspecified, possibly unneccessary, decree of a benevolent dictator.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr46kHgHNmZLgCUhAQKA+g//U9RiQpbaRQTBcYT4WCh+SPKJGheZCQ/S
1Xz1gI/M0phGVjfuLenpbz7CpIdfuZE670xccMXaqlYauW3D8NAkZopwLQQtuX9v
MlctbWDX86Xbq0Ng3Zi6UPgKrY3kuWLP+NIjyq0geu5rnXFCgZLn2qOpZzWn1Uyf
O0PNwKloBFoGeFcCs2F3HmzQ4sw2pVL645UxyatCmB/omNZIFTChkyMR4A7Ybvfc
nUDtH9AMqJ/EROb3LkCQIK79I4yoDJraD64glkp/iIuADUCioVlAbyzTHuGIbFYY
U3QqdOFSoCXJHC8PVSXCN1LBv4YWS2EWsYYiPiKisCHtRQi5jpZEgFdcAGbmiNaA
PNIP6zfcAC8bxJ9aeH9LK+QbfzBU9LFjDIfn4TgrZdkDp+q6rHaS4EO3KWaVubn/
YDbDRd+QCDLgnzjNvQZdTXHrtotFXk+xWkKfN5e4fP5Z56EWXY1SUkv+oRC3vdJI
yE0D+a8qg0XJKuFsEzhOa4Pxhu27eRCVPQ4b+s3ivmJmep3Og6v4MaG/SLTeCGMX
ESHRpYUZfG81mwpI5GmkIm8E6dnZp6YmaQx9ZI9+J1B6mJ/1payL78TBIckq79Tx
mKudpapkH3cZ93Br54f5C+pY/XjaHhepYZbkytl0BNb75R4T3r93s81M2q5N3ghN
3+ruvJY2Kto=
=Skae
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Lincoln Dale]

David Masover wrote:

>Ok, I'll bite.  Hans put it best a moment ago:
>  
>
[..]

you seem to have some misconceived notion that this is somehow a 
"ReiserFS versus XFS" or "ReiserFS versus ext3" case.
l-k could do without those conspiracy theories.  lets just stick to the 
facts.

>No, as a user, I just want a working plugin architecture to play with
>(I'm not *just* a user), and a working Reiser4 'cause it's fast, and I
>am eager to see new improvements coming out of Namesys, instead of two
>years spent trying to keep up with the vanilla kernel *and* adapt to
>some unspecified, possibly unneccessary, decree of a benevolent dictator.
>  
>
one of the nice things about linux is the freedom open-source gives you.
there is no reason why Hans cannot keep whatever proprietary mechanisms 
to do whatever fancy plugins he wants and you as a user can use until 
the cows come home.

but just don't have the false expectation that just because something is 
'kool' that its going to get into the kernel without rigorous peer 
review & approval


cheers,

lincoln.

>  
>

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lincoln Dale wrote:
> David Masover wrote:
> 
>> Ok, I'll bite.  Hans put it best a moment ago:
>>  
>>
> [..]
> 
> you seem to have some misconceived notion that this is somehow a
> "ReiserFS versus XFS" or "ReiserFS versus ext3" case.
> l-k could do without those conspiracy theories.  lets just stick to the
> facts.

erm, let me take a peek at what you just chopped out.

Yeah.  Um.  I mention XFS and ext3 *once* in the whole message.  As an
example of filesystems in general, because the only other two I'd
consider (or ever use) for a desktop FS at this point is XFS or ext3.
And it was a hypothetical opinion (I don't really know if / how much
Hans feels that way), and I personally find it far easier to believe
that there are stubborn people or that there isn't enough communication
of facts than that there is actually some anti-reiser conspiracy.

So, my mistake for trying to speak for other people.

>> No, as a user, I just want a working plugin architecture to play with
>> (I'm not *just* a user), and a working Reiser4 'cause it's fast, and I
>> am eager to see new improvements coming out of Namesys, instead of two
>> years spent trying to keep up with the vanilla kernel *and* adapt to
>> some unspecified, possibly unneccessary, decree of a benevolent dictator.

[...]
> but just don't have the false expectation that just because something is
> 'kool' that its going to get into the kernel without rigorous peer
> review & approval

I don't, actually.

I did expect the peer review to be a little more efficient, less
confrontational, and (maybe) more fair.  But I'm actually a bit content
right now.  After all, I *am* running reiser4 on two of the three
machines I maintain at home.

And, if you scroll up a bit, at this point I just want this resolved.  I
don't particularly care if Hans decides to just keep maintaining a
patchset, although I would be happier if it got into the mainstream
kernel -- I wouldn't have to wait as long to install each new 2.6.11 or
2.6.12 + nvidia module.

But this is a lot of time and energy from a lot of people on both sides
who I would rather see working on something else.

I appologize for the tone of that, too.  I know you don't work for me...

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr5gq3gHNmZLgCUhAQLgixAAgC1WTVa8+6wP3CbCJafz4V5+f/hcJkuG
xtXeh/CPvN4FzRRu+UjDVgji6yrLQ9AxFOa9kg9iJzZLIDDUNKu6UvFe+j22Mmpv
F/24aLD8NAtF4JNGOJv6xFZtwk03N8Q92+CU0b5jPEViom2h55OkKfSIzoGz47Ee
45XGDx0v2LCHVG+HhVuG3EVQNjI4oBiwQteErHjmoNcvh7npkbdYGvEHRULgX3rO
eCals0WPCQ+A10xDoTll6NvqEU59aHeheDw+FBkCZw4GhGaSCdZn0q8EHqqTdufL
iU5z/Q6J98KvjeMdhlCW8QRWA+hSIwQJcn+09IzI2lT4QnpPDRTeX3NxsdCrVbLu
fg37+d46cfWNrXpIrm3SoaTMl5GGvGGTekD46deTtotbJ40fSXGv3FbB6KCFK04s
U6kjfnqO8fFG/iKWExCqts6HUPiboI/zpz8w/oL7XudJO/jxKavKQGk+POCFquyT
U2e7KvZig5Ct0aunlIec7NrrRbutfYU6TFlYYRlV6XmIDP29ZDtqN9DsIBEvSeP5
7RFs4r0nTaIj17mpWmX5XCaLGNSUqzsQ1bzIoAl/D8NzbRTD9dsukvZyz5lW7tiZ
zsWe4mHtJrZ1/mQf/oCj36FXID+bd3xN0WGIjli5gzMjCK8uCkS09MQ+JeLGQHES
M/wMDcMcrwE=
=3y4o
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hubert Chan]

On Sat, 25 Jun 2005 16:31:37 -0400, Valdis.Kletnieks@vt.edu said:

[...]

> Meanwhile, PGP was designed to be used in an environment where you
> could do this: "Today's secret plans are AES256 encrypted.  The key is
> the next key in your one-time-pad book, XOR'ed with your 128-bit
> secret key - do it in your head".  (And yes, you can easily memorize a
> 16-digit hex number and learn to do an XOR with another 16-digit
> number, if failing to do so means you could end up dead).

Huh?  I have no idea what this has to do with PGP vs. encryption being
handled by the filesystem/loopback/dm-crypt.  What's the difference
between that scenario and "Today's secret plans are stored in the
loopback file foo.iso on an AES256 encrypted ISO.  The key is ... "?

Or "Today's secret plans are stored AES256 encrypted in foo.txt.  To
access it, you'll need to do {magic command to enter the key into the
filesystem} [1].  The key is ..."?

[1] I have no idea what kind of interface the crypto plugin in Reiser4
will have.  I'm assuming that there will be some commands for adding and
removing keys from the plugin.  If such commands don't exist, then we
have a seriously broken system.

> This is inconvenient for the user, but intractable for an attacker to
> create a scenario where they can just 'vi /each/decrypted/file' ;)

If you can memorize a 16-digit hex number and do XOR in your head, you
can learn to unmount the loopback/remove the key from the filesystem
too.

Which is definitely easier than remembering to create a temporary RAMFS,
mount it (with the right permissions!), decrypt the secret plans to that
FS, edit the plans, re-encrypt the plans, blow away the RAMFS...

>> won't do it.  If security is transparent, just enter a password and
>> go, then more people would do it.

> "Just enter a password and go, then more people would do it".

> Two words: "phishing e-mail".

Social engineering works in meatspace too.  Most people have locks on
their doors, even though they're easy to get around if you can pretend
to be from the electrical/gas/water company.  But having locks on your
doors is better than not having locks, because it prevents other types
of attacks.

I don't care if phishing gets around encryption.  It would work the same
if people didn't have their stuff encrypted.  But users with encrypted
stuff are safer from other types of attacks, and no less safe from
phishing.

I'd rather have people encrypting all their stuff and still be
vulnerable to phishing but secure from someone stealing their computer
and fetching all their personal data, than having people not encrypt
their stuff and have all their personal data harvested when they lose
their computers and still be vulnerable to phishing.

P.S.  Is the custom on the linux-kernel list really to Cc: everyone and
their dog?  I'm seeing a lot of long Cc: lists here...

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hubert Chan wrote:
> On Sat, 25 Jun 2005 16:31:37 -0400, Valdis.Kletnieks@vt.edu said:
> 
> [...]
> 
> 
>>Meanwhile, PGP was designed to be used in an environment where you
>>could do this: "Today's secret plans are AES256 encrypted.  The key is
>>the next key in your one-time-pad book, XOR'ed with your 128-bit
>>secret key - do it in your head".  (And yes, you can easily memorize a
>>16-digit hex number and learn to do an XOR with another 16-digit
>>number, if failing to do so means you could end up dead).

[...]

> [1] I have no idea what kind of interface the crypto plugin in Reiser4
> will have.  I'm assuming that there will be some commands for adding and
> removing keys from the plugin.  If such commands don't exist, then we
> have a seriously broken system.

If we do meta-files as was originally intended, the command will be a
shell script could look something like this:

#!/bin/bash
read -sp 'passphrase: ' key
echo "$key" > "$0"/.../plugins/cryptocompress/key

The syntax of that echo command may change, but this is what we like
about metas -- less namespace fragmentation, less random interfaces.

>>Two words: "phishing e-mail".

[...]

> I'd rather have people encrypting all their stuff and still be
> vulnerable to phishing but secure from someone stealing their computer
> and fetching all their personal data, than having people not encrypt
> their stuff and have all their personal data harvested when they lose
> their computers and still be vulnerable to phishing.

Thank you, I was just about to say that.

There's a quote about this.  Someone once asked Mohammed, "Should we tie
up the horses or trust in Allah?"  Mohammed said, "Trust in Allah.
...But, tie up the horses."

> P.S.  Is the custom on the linux-kernel list really to Cc: everyone and
> their dog?  I'm seeing a lot of long Cc: lists here...

It seems to be the custom of any list to just hit "reply-to-all".  That
way, even if someone posted a reply after reading the archives or from a
forwarded mail, or even if they unsubscribe from the list, or even if
someone simply opened up their client and started a thread by mailing
linux-kernel@vger.kernel.org directly, or if someone just randomly adds
someone who might be interested to the CC list directly -- no matter
what, someone who's posted on a topic will see replies to that post
until the topic dies.

Of course, this isn't true of all lists.  Some lists munge headers,
meaning that either "reply" or "reply-to-all" goes to the same place,
meaning no automatic CC's.  I don't like that, because then it's harder
to take something off-list, and easier to accidently send something
supposedly private to the entire list.

If you take something off-list when you don't mean to, you can just
re-send it, but if you put something on-list when it was supposed to be
private, there isn't much you can do...

Of course, there's the annoying side-effect that if you are subscribed
to the list, you'll get lots of dupes, but so what?  Bandwidth is cheap.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr4XAngHNmZLgCUhAQK+JRAAgg4YlZ9cb0S5hp9JzGdZHm0FeJDoKIok
voT5LvqgooQpJZk3mwyagYqvP5uvY2UgFA74seMqzTHRnynCDp4orCPGgADofaFU
KfjGcVUS6SuY3VgeF7WBEjFj1NHSwDp3h0LfeRocEglbFxoPGJvw3gWSnX1QDZ68
nmuSRGVSB7nb1Os6c2zsM0uXvJA43HNJ+W/UrEPOFjiRI1bOOioxzphgVwCAQH3j
8Vb2/HWmPLTCqXoOYESZ5OBQOR6iZViegh/x/Rn+O99UfiENdacoX5xwGM68SAXM
CR3JjRA3JEA1iScz9I2byD2dZyb2596Q09Xh/5/5PQxK8zGm0FtWWEbOvDeF7Re7
cQiXkZB9uLQDJel+jlwatKGCPRVlx9wDJ8puIMf47QOsWhx0TY8lAxCebu4zorjz
K2vQiF/ZMOYXFB5nBCvgHL7XG24FRpuaU0wWo7+0cY2o4WBhvfBO5o+93Klwa7Na
ltPKRFPv6B6KBDD71quSwZ9M1YkjfR0vaPZWV9T5TFBklfRv26N1DhNmW1o2KjRI
wX3yrsIbAAG9dK3Vs71oxWIw0hqmfpo4UZTZGRoi8GL+drHBNvHxzNtaeSBF4AeO
fxYnQHXO1+Us3zbb/oBR4Hm3ugvsRYXMyArm1hHHFlmcXdggjz+K36IiCK7wKpfp
2gVec7JaEkY=
=3Mkk
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
> On Sat, 25 Jun 2005 13:33:27 CDT, David Masover said:
> 
>>>Now *think* for a moment - how does a hypothetical Reiser4 using ext3 format
>>>gain any speed advantage with small files, when the speed advantage is based
>>>on using a format other than ext3?
> 
> 
>>happen in RAM.  If you do a ton of work with a dataset that stays in
>>RAM, Reiser probably performs as well or better than a ramdisk, because
>>changes that get overwritten while still in RAM never actually touch the
>>disk.
> 
> 
> At that point, since the actual buffer management is being done at the VFS
> level (see fs/buffer.c and friends) what you're really comparing is the speed
> of journalling metadata - at which point you need to be *very* careful to

No, just metadata.

> specify exactly what configuration you're talking about.  If you don't believe
> me, investigate why mounting a filesystem with 'noatime,nodiratime' can make a
> dramatic difference totally independent of the underlying filesystem, but the
> actual amount of gain is dependent on format (hint - how far do the heads have
> to move to record 3 atime updates against 3 random inodes on an ext2, an ext3,
> and a VFAT filesystem, assuming no other disk activity), and why
> ext3 has 3 different modes data=ordered/writeback/journal.

I'm not saying format doesn't matter for _all_ optimizations, but there
are some common ones for which it does matter.

>>       Reiser also doesn't fragment as quickly as ext3, and I don't
>>think that has anything to do with its format.
> 
> 
> Care to explain why it's not format-dependent? 

Reiser4 and XFS both do lazy allocation.  They both avoid allocating
blocks until they run out of buffer RAM.  When they have to, they
allocate and flush everything.

This may have changed a bit -- Hans was talking about a more stack-like
system, to avoid the system locking up for tens of seconds at a time
while ALL ram is flushed -- but the principle is the same.

That principle is that if you have a large chunk of data to allocate all
at once, you are more likely to know how it should be arranged on disk.
 If you allocate every write(2) or every 5 seconds, how do you know if
they are writing 2K or 2 gigs?  You might try to pack it into 1 meg of
space, then 5 megs, and so on, but you end up with quite a fragmented file.

With allocate-on-flush, if you've got more like 100 megs to allocate at
once, you can find space for that 100 megs.  In fact, with the Reiser
format, you can pack lots of tiny (much less than a block) files
together to save space and speed things up.

But the lack of fragmentation is not format-dependent.

>>The FS that gets merged ahead of time without plugins would no longer be
>>Reiser4.  Go read the whitepaper, or tell me why I'm wrong, but even
>>symlinks are implemented as plugins.
> 
> 
> Which is another way of saying Reiser4 can't be merged in its present form.

Actually, I'll have to go back on this a bit.  There are different kinds
of plugins, and I'm not sure exactly which people want in the VFS.  This
may be because nobody's said that, though.

Still, while plugins may not depend on Reiser, Reiser depends on plugins.

>>Actually, plugins are just as easy or easier than crypto-loop or
>>dm-crypt.  And why shouldn't my crypto be easy?  Most users are insecure
>>in all kinds of ways because of that attitude -- security is HARD, so I
> 
> There's a vast distinction between "easy for implementors" and "easy for
> users".  Jaari Russo's loop-aes stuff does a wonderful job of being "easy for
> users" - just say "mount", answer the passphrase, and you're good to go.  The

Not as easy as a plugin, though.  Per-file granularity is nice.
Especially on a USB key, where you're changing the files all the time.
Sometimes you want just a few (encrypted) SSH/GPG keys and a bunch of
(unencrypted) mpegs, and sometimes you want just a few (unencrypted)
HTML files and a bunch of (encrypted) top-secret blueprints in extremely
high-res JPEG/PNG format.

A loop file on a keychain is only slightly better than partitions.

[...]
> Meanwhile, PGP was designed to be used in an environment where you could do
> this:  "Today's secret plans are AES256 encrypted.  The key is the next key in
> your one-time-pad book, XOR'ed with your 128-bit secret key - do it in your head".
> (And yes, you can easily memorize a 16-digit hex number and learn to do an XOR
> with another 16-digit number, if failing to do so means you could end up dead).
> 
> This is inconvenient for the user, but intractable for an attacker to create a
> scenario where they can just 'vi /each/decrypted/file' ;)

Just as intractable the plugin way.  Not to mention, setting up a
ramdisk properly and decrypting to said ramdisk is a step you might
forget, which might put your files on the disk unencrypted.  Much more
unforgivable than the heinous crime of making top-secret government work
easier!

>>won't do it.  If security is transparent, just enter a password and go,
>>then more people would do it.
> 
> 
> "Just enter a password and go, then more people would do it".
> 
> Two words: "phishing e-mail".

If someone's going to learn about crypto, they are going to learn about
how to avoid phishing scams.  Both can be easy.

After all, private/public keys can be made user-friendly and secure.  At
the same time!

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr4dmngHNmZLgCUhAQLJVA//WE20P/91cVuDclUT0W6bOX8bJvycOilB
v55mUI9eoTdWink80RNo8QZ4aM1sidOrbtKglNmWdku1Fr80JW8HEPeQWRWR8PUL
midO7ulTfRldBdM0mU+Ojl+Dj8vln6Qr+80Rveo5XDUlcWVwKHH7d87ONlFRlwIO
NZMQXbB90huexJVoiITIGkFcDeoSdth/Em3cKjGhoEpreGkw6DuQzvS3aTOV+Fw4
c8neGsw2Tnx05+LuwE1VKxL7aV6U/Z0JDXwMqJtvINvkxnu6mKR++hRkWKW374LY
71idZzBu92oTNlEl02+y/EUGGU7+TMtduTo98ww4eN9r8bpB/WbMuOm+w05EKHKg
H1OT1CX+G4sP/0GZaslPzh8z7cGI6318FN3Twk922J8t3dWcmNer4ULt6rp+c/38
stjbammdaBTWNTYg/BjZ+oedhMa4BHVTzwU2R0HpEzI3EIj/5rjek/D1cWzuBu/C
NQ4QfVKJPHLgjKK4DV3iLQTkNpXC2skcluJ4FC1c+VYZTx8wvIrVUmJfPJajLAVg
en42jbCYzdldtzR/yYal3b7KCeJoi1x1PUvStgL6iWSe7MjEJoF4jV/TcmE2gLf3
RzQUQ0UuMMisdQ2qbKAA3BrbZV239P/HUtyZdnIj8yWX4hF5Hhu7G1Ncz545VCdk
+e0D/XVpgbc=
=EBr/
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

Think of reiser4 as being designed to be 90% library routines, and 10%
program.  Now, can these libraries be used by other filesystems?  Why
yes, some can.  How many of them should be used by other filesystems? 

Reality:  few people are going to rewrite their existing filesytems to
mostly use our code.  However, people writing new filesystems, if we
have done our job right, will decide that our code is the easiest code
base to use for implementing whatever differentiates them while avoiding
reinventing what they don't seek to be better at.  Detail: Regarding our
allocation and balance and compress and encrypt at flush time code,
unless your filesystem is also based on balanced trees it might be the
least reusable code in the filesystem.  It is the hardest code in the
filesystem, because the algorithms we implemented were simply hard. 
Number one task for me, after we go in and I can stop dealing with
prerequisites to inclusion, is to review the vm interaction from
beginning to end one more time (and kill the emergency flush code).

Good part: if you want to implement new filesystem semantics, our
storage layer is more suitable for your innovating on top of than any
other.  Less work to code it, more functionality and flexibility,
plugins are great for hacking on top of.  The storage layer is very very
high performance, and if semantics are your focus, using our storage
layer is likely to be a good choice because it is well abstracted and
works without understanding what it works on.  For example, you can
invent new items for the tree to balance (e.g. new directory entry
formats), and all you have to do is write an item handler for the item
and you don't have to touch the balancing code.

In general, if a new filesystem can do some narrow aspect better than
our existing library routine, it should do its aspect using its
innovative new code, and where it is not trying to be better than us, it
can reuse our code more easily than any alternative.   Many people who
would write a new filesystem from scratch, can, if they use our code
base, accomplish their objectives with just writing some new plugins and
contributing them to the collection.  Others can define a new filesystem
to consist of a particular configuration of plugins and glue that are
what you get when you mount fubarfs.  We would be happy to accomodate
that by creating subdirectories for different flavorings of reiser4 to
live in.

Because our code is 90% library routines (aka plugins), eliminating the
plugin layer is like eliminating main() in a user space program.

Hans

David Masover wrote:

> Actually, I'll have to go back on this a bit.  There are different kinds
> of plugins, and I'm not sure exactly which people want in the VFS.  This
> may be because nobody's said that, though.
>
> Still, while plugins may not depend on Reiser, Reiser depends on plugins.
>
>

Re: reiser4 plugins

[Horst von Brand]

David Masover <ninja@slaphack.com> wrote:
> Horst von Brand wrote:
> > David Masover <ninja@slaphack.com> wrote:
> >>Horst von Brand wrote:
> >>>David Masover <ninja@slaphack.com> wrote:
> >>>>Hans Reiser wrote:
> >>>>>Jeff Garzik wrote:

[...]

> >>"Ain't broke" is the battle cry of stagnation.

> > I see it as the battle cry of those that are looking for /real/
> > problems to solve.

> I'll refer you to my other rant about stagnation and oil.

Read it. Makes no sense, wind and solar power have their /own/ problems,
environmentally speaking. Besides, as things stand today, they are *extremely*
expnsive and hard to use, so next to useless (for now).

> And, listen to yourself.  "Good riddance, no, I don't use DOS" but
> "Ain't broke is the battle cry of those looking for real problems to solve."

> You can solve real problems in DOS,

Sure. But not all (or even most) problems I'd like to solve with a computer
as of today. So...

>                                     it's usable, it ain't broke, there
> are even some decent games (doom) and windowing systems (win3.1 and
> others) for it.

Sure. But that's not enough.

> But Linux is better.  DOS ain't broke,

For me, it is.

>                                        but Linux is better.  So maybe
> VFS ain't broke,

I think it is doing fine.

>                  but plugins would be better.

Maybe. It's up to /you/ to convince the head kernel hackers (and me on the
way).

>                                                I guess we'll only know
> if we let Reiser4 merge...

No. Just like devfs was "the obvious right way", it /had/ to be merged
ASAP, "wrinkles will be ironed out later". Turned out it wasn't wrinkles,
but fundamental brokenness, and it was soon abandoned by the people who
promised to maintain it forever. And now we have the flamewar about its
removal...

> >>But, there are some things Reiser does better and faster than ext3,

> > Yes, I've heard it is supposed to be faster on huge directories, and
> > doesn't run out of inodes. And it is more efficient spacewise on small
> > files. But then again, space is extremely cheap today...

> Speed isn't.  CPU is, but not disk speed.  And packing stuff more
> efficiently, without actually compressing it, will give you some of that
> speed.

For my current uses, ext3 is plenty fast enough. No pressing need to change.

> Also, space is not so cheap that I won't take 25% more.

It is cheap enough that I can't realistically fill the disks I have with
/usefull/ stuff. So...

> > And again, on a list around here I've seen several cries for help with
> > completely hosed filesystems, all ReiserFS. No solution has ever come
> > forth.

> I haven't been counting, but I've seen a number of solutions fly around
> reiserfs-list for people with reiser4 problems.

It was ReiserFS 3. Maybe the problems are fixed now, but as they say about
burned children...
[...]

> A lot of what people like about ext3 is its stability and fairly
> universally accepted format.  A lot of what people like about XFS is its
> stability and speed, mainly with large files.  A lot of what people like
> about Reiser4 (as it is today) is its speed, with large and especially
> with small files.

Right. And mushing it all together is way more likely to combine all /bad/
features than to retain some of the /good/ ones.

> Those are broad and somewhat uneducated statements, but I doubt most
> people care what FS they are using if the stability and performance is
> what they want.  In that case, why have so much duplicated effort
> between different filesystems -- even between ISO and UDF and Reiser and
> XFS -- when most of what's really different is the on-disk format and
> the optimization?

Because they are different on-disk and are optimized for different uses,
perhaps? I can't use ext3 for reading my CDs, I need NTFS to access the
Windows partition here, ...

> So, in this hypothetical situation where ext3 is a reiser4 plugin,
> suddenly all the ext3 developers are trying to improve the speed and
> reliability of reiser4, which benefits both ext3 and reiser4, instead of
> just ext3.

I guess that it won't ever turn out to be that simple. ext3 developers will
have to consider not screwing up XFS, etc. And I don't see any real
difference there from where we stand today... just a bigger mess: VFS with
ReiserFS with plugins for ext3, instead of VFS with ext3. No gain, much
pain.

[...]

> > And that would surely break Windows compatibility (because you have to keep
> > the data on what to encrypt one the filesystem itself). Besides, having

> Aside from what someone else already said about this, why not just have
> support for accessing, say, a .gpg file as transparently decrypted?

That should, if anything, be a /user/ decision, not a /kernel/ one. Even
sometimes decrypt, sometimes don't.

>                                                                      You
> don't even need file-as-directory, just create a file called foo which
> is really the decrypted version of foo.gpg.  No need to change the
> format, just the filesystem.

No need to change the filesystem, learn to use the tools at hand.

> > pgp, or gpg, or crypt, or my own whacky encryption proggie do the job in
> > /userland/, and not shoving into the kernel and so down the next user's
> > throat, is in the end a largeish part of what Unix is all about for me.

> Do you use ipv6?

Sometimes.

>                   I don't.  And it's not shoved down my throat, either,
> although it's in the kernel.  I simply disable it, and you would (I'm
> sure) disable the crypto stuff.

An Internet standard is a quite different kettle of fish than the pet
experimental filesystem for a minority operating system...

> Plus, as someone else said, it's much easier to do
> $ vim /some/encrypted/file
> than
> $ gpg --decrypt /some/encrypted/file > /some/decrypted/file
> $ vim /some/decrypted/file
> $ gpg --encrypt /some/decrypted/file > /some/encrypted/file
> $ shred /some/decrypted/file

So what? Write your script to do it. Or use emacs, I'm sure it either has
now or will soon have a plugin for it... much easier to develop, much more
flexible to use, ...

> Not to mention, shred doesn't work on modern filesystems, so you need to
> either patch vim (and any other program you might want to use), create
> an actual ramdisk, or deactivate swap and mount a tmpfs.

Right. And all this complex futzing around (and making sure the unencrypted
data doesn't remain if the power is cut in the right moment, and...)

> On doing stuff in userland:  I am all for moving tons of stuff to
> userland, but not until someone does a microkernel right, if it's even
> possible.  And even then, I'd probably still use Linux and argue for
> some stuff in the kernel, because Linux has developers.  Developers.
> Developers.  Developers.

Sorry, I don't buy this. Linux (the kernel) has tons of developers. But
that doesn't mean everybody is interested in everything. Just like there
are probably much more userland developers (for one, it is /much/ easier to
work with...), and there not everybody is interested in everything. Putting
stuff in the kernel that doesn't belong there (== can't be done elsewhere)
is useless bloat.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
> David Masover <ninja@slaphack.com> wrote:
> 
>>Horst von Brand wrote:
>>
>>>David Masover <ninja@slaphack.com> wrote:
>>>
>>>>Horst von Brand wrote:
>>>>
>>>>>David Masover <ninja@slaphack.com> wrote:
>>>>>
>>>>>>Hans Reiser wrote:
>>>>>>
>>>>>>>Jeff Garzik wrote:
> 
> 
> [...]
> 
> 
>>>>"Ain't broke" is the battle cry of stagnation.
> 
> 
>>>I see it as the battle cry of those that are looking for /real/
>>>problems to solve.
> 
> 
>>I'll refer you to my other rant about stagnation and oil.
> 
> 
> Read it. Makes no sense, wind and solar power have their /own/ problems,
> environmentally speaking. Besides, as things stand today, they are *extremely*
> expnsive and hard to use, so next to useless (for now).

Have you bothered to research them?

The main problem is in the construction.  After that...  hard to use?
Not really.  Expensive?  They pay for themselves, eventually.

Plus, they get better as money goes into them.  Oil *can't* get much
better, but all the money goes there, instead of into alternatives.
Thank you, Republicat administration...

> Maybe. It's up to /you/ to convince the head kernel hackers (and me on the
> way).
> 
> 
>>                                               I guess we'll only know
>>if we let Reiser4 merge...
> 
> 
> No. Just like devfs was "the obvious right way", it /had/ to be merged
> ASAP, "wrinkles will be ironed out later". Turned out it wasn't wrinkles,
> but fundamental brokenness, and it was soon abandoned by the people who
> promised to maintain it forever. And now we have the flamewar about its
> removal...

Should it have been kept out in the beginning, before we knew we needed
udev?  Would we have udev at all, had devfs not been merged?

>>>>But, there are some things Reiser does better and faster than ext3,
> 
> 
>>>Yes, I've heard it is supposed to be faster on huge directories, and
>>>doesn't run out of inodes. And it is more efficient spacewise on small
>>>files. But then again, space is extremely cheap today...
> 
> 
>>Speed isn't.  CPU is, but not disk speed.  And packing stuff more
>>efficiently, without actually compressing it, will give you some of that
>>speed.
> 
> 
> For my current uses, ext3 is plenty fast enough. No pressing need to change.

That is the problem I attempted to illustrate with the oil rant.  No
pressing need to change doesn't mean that something astronomically
better shouldn't be adopted.

Amish people live just fine.  There's no pressing need for them to
change.  But I bet that many of them would be happier if they had a car.


>>>And again, on a list around here I've seen several cries for help with
>>>completely hosed filesystems, all ReiserFS. No solution has ever come
>>>forth.
> 
> 
>>I haven't been counting, but I've seen a number of solutions fly around
>>reiserfs-list for people with reiser4 problems.
> 
> 
> It was ReiserFS 3. Maybe the problems are fixed now, but as they say about
> burned children...

speaking for yourself?

>>A lot of what people like about ext3 is its stability and fairly
>>universally accepted format.  A lot of what people like about XFS is its
>>stability and speed, mainly with large files.  A lot of what people like
>>about Reiser4 (as it is today) is its speed, with large and especially
>>with small files.
> 
> 
> Right. And mushing it all together is way more likely to combine all /bad/
> features than to retain some of the /good/ ones.

Actually, it wasn't "mushing it all together".  It ended up throwing out
a fair bit of it in the example I was talking about.  But I really
shouldn't be arguing that.  It's not what people care about.

>>Those are broad and somewhat uneducated statements, but I doubt most
>>people care what FS they are using if the stability and performance is
>>what they want.  In that case, why have so much duplicated effort
>>between different filesystems -- even between ISO and UDF and Reiser and
>>XFS -- when most of what's really different is the on-disk format and
>>the optimization?
> 
> 
> Because they are different on-disk and are optimized for different uses,
> perhaps? I can't use ext3 for reading my CDs, I need NTFS to access the
> Windows partition here, ...

As things stand.

And it would be pointless to change some of those, like CDs.  But then,
transparent decompression as a plugin might be better.

>>So, in this hypothetical situation where ext3 is a reiser4 plugin,
>>suddenly all the ext3 developers are trying to improve the speed and
>>reliability of reiser4, which benefits both ext3 and reiser4, instead of
>>just ext3.
> 
> 
> I guess that it won't ever turn out to be that simple. ext3 developers will
> have to consider not screwing up XFS, etc. And I don't see any real
> difference there from where we stand today... just a bigger mess: VFS with
> ReiserFS with plugins for ext3, instead of VFS with ext3. No gain, much
> pain.

That you don't see the gain doesn't mean it doesn't exist.

I don't deny the pain, though I don't think it will be as bad as you
think.  For one thing, storage plugins are fairly self-contained, or
should be.

>>>And that would surely break Windows compatibility (because you have to keep
>>>the data on what to encrypt one the filesystem itself). Besides, having
> 
> 
>>Aside from what someone else already said about this, why not just have
>>support for accessing, say, a .gpg file as transparently decrypted?
> 
> 
> That should, if anything, be a /user/ decision, not a /kernel/ one. Even
> sometimes decrypt, sometimes don't.

Sometimes decrypt, sometimes don't, would certainly be allowed.

What's not allowed now is an easy way to vim an encrypted file without
patching/wrapping vim or dealing with ramdisks.  And that's assuming vim
is the only program I want to use.

Transparency is a good thing.

>>                                                                     You
>>don't even need file-as-directory, just create a file called foo which
>>is really the decrypted version of foo.gpg.  No need to change the
>>format, just the filesystem.
> 
> 
> No need to change the filesystem, learn to use the tools at hand.

Read above.

>>>pgp, or gpg, or crypt, or my own whacky encryption proggie do the job in
>>>/userland/, and not shoving into the kernel and so down the next user's
>>>throat, is in the end a largeish part of what Unix is all about for me.
> 
> 
>>Do you use ipv6?
> 
> 
> Sometimes.
> 
> 
>>                  I don't.  And it's not shoved down my throat, either,
>>although it's in the kernel.  I simply disable it, and you would (I'm
>>sure) disable the crypto stuff.
> 
> 
> An Internet standard is a quite different kettle of fish than the pet
> experimental filesystem for a minority operating system...

"standard"?  How many Linux people use ipv6?  How many use reiser4?

And what's your point, anyway?  Disable it if you don't like it.

>>Plus, as someone else said, it's much easier to do
>>$ vim /some/encrypted/file
>>than
>>$ gpg --decrypt /some/encrypted/file > /some/decrypted/file
>>$ vim /some/decrypted/file
>>$ gpg --encrypt /some/decrypted/file > /some/encrypted/file
>>$ shred /some/decrypted/file
> 
> 
> So what? Write your script to do it. Or use emacs, I'm sure it either has
> now or will soon have a plugin for it... much easier to develop, much more
> flexible to use, ...

more flexible?

You don't seem to understand plugins.  That makes sense; you don't want to.

Plugins are far more flexible than trying to patch every single program
I want to deal with.  And the script described doesn't work because
shred doesn't work, meaning you have to deal with ramdisks, which is
even worse and less likely to work out-of-the-box.

>>Not to mention, shred doesn't work on modern filesystems, so you need to
>>either patch vim (and any other program you might want to use), create
>>an actual ramdisk, or deactivate swap and mount a tmpfs.
> 
> 
> Right. And all this complex futzing around (and making sure the unencrypted
> data doesn't remain if the power is cut in the right moment, and...)

Unencrypted data stays in RAM...

with plugins.

>>On doing stuff in userland:  I am all for moving tons of stuff to
>>userland, but not until someone does a microkernel right, if it's even
>>possible.  And even then, I'd probably still use Linux and argue for
>>some stuff in the kernel, because Linux has developers.  Developers.
>>Developers.  Developers.
> 
> 
> Sorry, I don't buy this. Linux (the kernel) has tons of developers. But
> that doesn't mean everybody is interested in everything. Just like there
> are probably much more userland developers (for one, it is /much/ easier to
> work with...), and there not everybody is interested in everything. Putting
> stuff in the kernel that doesn't belong there (== can't be done elsewhere)
> is useless bloat.

90% of what's currently in the kernel can be done elsewhere.  Witness HURD.

Doesn't mean it should.  Witness HURD.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr4043gHNmZLgCUhAQI4/A/9E6kVoYyVgQWFrrw7SO6e6vVZzQufsTFe
h2prUuaITxSDYc6oO6gR4UorkQAGtHjFFKCzdyH+/Eo2cmYAGtom14y8aNjWqQOj
iz4hqyH4RZJ3eSP0/YsMdopoDpkdtrlD6QCWL49C7A1h1Mrq0gL+SrRNzpLYc+QW
XfyqQppMgDRIQfCHKG+ZgS9cloW1HnYHppxYakvhkS4ZiFktEi1UHXj+2GQ22Mm5
EN56EHRthS8v2k7vOdsrkQE02lrgDtWWOt8bkrkHFCFB6lU9oXbqlKHWRrU/dd27
zYgvajgVZWjGT3iAXgyhcojCuIEpyOB2+Cegdga2D/H+kxh8tk1J1f1MJh9oXosb
4ScrlUIq+ghVx2+/YMvTmA8geKMqqi9OBvB1npfFXklXJHa/QJzTTo3/MJGxJ6Ko
fKyG0sWWNwbMdCmLoCQp1T+WycxMeOILw8jheVfQbF+JAXm/ialDXJ/rH5RYHeG4
vhlZHxKqihgiuQ0ZGvj8niR5vlei9tKZ8fBFiVgDxQhoUhrutiVQKIzxT6YXMQqM
PpXefWdDNi4vnNFn6eDVcXTs14kvBod8SM/5WxKUnnlCJNvjFUwo5x4lQ9iXnlYT
LHbJ06lGbLU6y8RfaY2+eBYziyMSStql77mwW9y7WssXI0cV7VzgEYSTd+S1eBN5
Yr+8chQqWOI=
=/wpm
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hubert Chan]

On Sun, 26 Jun 2005 00:01:39 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said:

> David Masover <ninja@slaphack.com> wrote:

[...]

>> Also, space is not so cheap that I won't take 25% more.

> It is cheap enough that I can't realistically fill the disks I have
> with /usefull/ stuff. So...

And since when did you become the spokesperson for all Linux users?

Just because *you* can't think of a use for 25% more space doesn't mean
that the rest of us don't want it.  Do you really think that there isn't
a significant number of people who would love to have more space?

[...]

> So what? Write your script to do it. Or use emacs, I'm sure it either
> has now or will soon have a plugin for it... much easier to develop,
> much more flexible to use, ...

Oh yes.  Please.  Let's make everyone use emacs.  Even if they want to
edit an OpenOffice document or a picture.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

cryptocompress [was Re: reiser4 plugins]

[Pavel Machek]

Hi!

> >                    and vfat people may decide they want cryptocompress,
> 
> I'm sure they don't, because it is mostly for Windows and assorted devices
> (pendrives, digital cameras, ...) compatibility.

Actually cryptocompress for vfat *would* be nice; pen drives are
small, slow, and likely to be lost/stolen. That makes them very nice
candidates for cryptocompress ;-).
								Pavel
-- 
teflon -- maybe it is a trademark, but it should not be.

Re: reiser4 plugins

[Alan Cox]

On Iau, 2005-06-23 at 06:58, Hans Reiser wrote:
> Jeff Garzik wrote:
> 
> > We have to maintain said ugly code for decades.
> 
> No you don't, I do.

Really. I must be mis-remembering some of the comments made about
reiserfs 3 during the 2.5 to 2.6 period. 

> I am entitled to get some advantage from being the first on the block

You were not first on the block. Linus was, thats why it's Linux (well
not directly his fault about the name) and why he sets policy. I've
never heard Linus espousing such an idea so I wonder why you think that
way. Rather I've heard Linus tell groups of people with differing ideas
about interfaces to go figure it out first and build the right interface
then come back with a consensus (for example with SELinux)

> It is odd how Hellwig no longer describes himself as XFS associated

Why would that be "odd". What are you trying to imply and why not just
say it if you actually have something to say.

Alan

Re: reiser4 plugins (back to flames, oh well)

[Hans Reiser]

Alan Cox wrote:

>>I am entitled to get some advantage from being the first on the block
>>    
>>
>
>You were not first on the block. Linus was, 
>
>thats why it's Linux (well
>not directly his fault about the name) and why he sets policy. I've
>never heard Linus espousing such an idea so I wonder why you think that
>way. Rather I've heard Linus tell groups of people with differing ideas
>about interfaces to go figure it out first and build the right interface
>then come back with a consensus (for example with SELinux)
>  
>
We aren't people with differing ideas about interfaces, we are one
filesystem team with functionality that no other filesystem has yet
expressed a concrete desire to use, and some guy who complains about it
after it has been the basis for coding for 4 years without expressing
any desire to use it himself, merely expressing a desire that we not use
it because he never paid any attention to our website or mailing list
for the last 4 years as we developed it.  

Oh, and he says it is duplicative without explaining where in VFS
plugins and pluginids are currently defined. 

Oh, and rather than saying "clever guys, thanks for coming up with this,
I am going to go write a patch that makes your ideas available to other
filesystems to do this", he says, stay out of the tree because your code
is so crappy that it has to be used by everyone else before you can go in.

There is absolutely nothing preventing him from generalizing it to other
filesystems after we go into the tree.  If you take the existing code as
a base, the generic part of the work required is I hope small.   The fs
specific part is more though, as each filesystem has to implement
methods for storing and retrieving plugins, and then a set of plugins
(and their methods).

It is all pure turf war.  Some guy suddenly realized that VFS was not
doing something, and got upset that someone else was writing code that
did the job.  Since it was someone else's code, and ready to go, and he
had nothing himself written yet, it had to be crap and kept out.

In fairness, it might be possible to eliminate a function indirection or
two if the effort was made.   There is no need to do it this week as
merging our code will not make it the least bit more difficult to change
things later, but hey, if someone wants it, happy to discuss the right
way to do it.

This is why Linux doesn't have the contributors its market share would
suggest it should expect.  Contributing to Linux is such a misery.

If you guys would just say "plugins, interesting, can we make it more
general so other filesystems can use it?"  I would be happy to have Zam
do it for you.

Re: reiser4 plugins

[Christoph Hellwig]

On Tue, Jun 21, 2005 at 11:25:24PM -0500, David Masover wrote:
> > You're basically implementing another VFS layer inside of reiser4, which
> > is a big layering violation.
> 
> There's been sloppy code in the kernel before.  I remember one bit in
> particular which was commented "Fuck me gently with a chainsaw."  If I
> remember correctly, this had all of the PCI ids and the names and
> manufacturers of the corresponding devices -- in a data structure -- in
> C source code.  It was something like one massive array definition, or
> maybe it was a structure.  I don't remember exactly, but...

Every device driver has a big array of corresponing device ids as an
array in C code - oh my god we're doomed  .. not.


> I agree there, too.  In fact, some people have suggested that all
> "legacy" (read: non-reiser) filesystems should be implemented as Reiser4
> plugins, effectively killing VFS.*
> 
> So, Reiser4 may eventually take over VFS and be the only Linux
> filesystem, but if so, it will have to do it much more slowly.  Take the
> good ideas -- things like plugins -- and make them at least look like
> incremental updates to the current VFS, and make them available to all
> filesystems.

And why exactly would we replace a stable, working abstraction with an unpoven
mess?

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Christoph Hellwig wrote:
> On Tue, Jun 21, 2005 at 11:25:24PM -0500, David Masover wrote:
>
>>>You're basically implementing another VFS layer inside of reiser4, which
>>>is a big layering violation.
>>
>>There's been sloppy code in the kernel before.  I remember one bit in
>>particular which was commented "Fuck me gently with a chainsaw."  If I
>>remember correctly, this had all of the PCI ids and the names and
>>manufacturers of the corresponding devices -- in a data structure -- in
>>C source code.  It was something like one massive array definition, or
>>maybe it was a structure.  I don't remember exactly, but...
>
>
> Every device driver has a big array of corresponing device ids as an
> array in C code - oh my god we're doomed  .. not.

I could throw the same sarcasm back at you.  We must be doomed because
Reiser does some stuff that VFS already does!  Or am I misunderstanding
the complaint?

>>I agree there, too.  In fact, some people have suggested that all
>>"legacy" (read: non-reiser) filesystems should be implemented as Reiser4
>>plugins, effectively killing VFS.*
>>
>>So, Reiser4 may eventually take over VFS and be the only Linux
>>filesystem, but if so, it will have to do it much more slowly.  Take the
>>good ideas -- things like plugins -- and make them at least look like
>>incremental updates to the current VFS, and make them available to all
>>filesystems.
>
>
> And why exactly would we replace a stable, working abstraction with an
unpoven
> mess?

How does it get proven if you won't give it a chance as a *separate*
unproven mess, with a big fat EXPERIMENTAL flag, for users to play with?

I know, it exists as a separate patch.  But it works now, and I think
the best way to "prove" it would be to package it with the kernel.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrkXY3gHNmZLgCUhAQLQUw//ZFN1KS+2wS/yDMa+/oWXVemZ690sMCLx
ZlKGg82bnv2XxqMXQwuPG9V02oN/D+1bkPmZzr8rD/tm5WshxpAHroIhnp3ZVpRi
lbMwULFQ8Z8fcsY3+YUag4XAUYGK+tmIeZc47FJGL0avsRa3RFJsFm+Kb6E/fi2f
H4wda43rt2CJYD5GqCtMsqyxzHzPclKHq25betIcPWBOqvE5NzQbc2tFTo0n3KMb
vmyZc4B34kiKhrrW/7pZCxDpiGjoxr87F19Tk8IIltM9kAuSVLXgtY/T+2DA2vJE
2N/Offr1rZh9zSq8PGkGoI+K41AaY3CkeYGjUF2eiZd4qwE624/1jUSEg685Puse
091EuIMzdndJYM0H+OsaFtvH9Rc67Hv6yR7aucNF5j8sIam37y7Fl+MToRgJK1+E
7YSpm/Ld61RaPqbJ4mqv4f0fHLTa2SpbFI8vmA1ARuiA+/YtUz9jBjLrPtMo4ppj
VvNTVMmftUgRr1NlQ+MKJO4Kxt4kKQnt1OtUe2y4bjCqO7ldUvPWLKGhsY0EsS0k
9yymlBbhsjTFrY9CsyrThshyHe9ikBVSLY7i16W+KhjLF/FKaq9k93nHd4B5Shni
Km9zyd0DlCUr3Y20SpBDITCWtM0CL0YQzeEW0JJTxVpHIDjh6s65XcBfrlWwEUiw
j/GJZA5h+bw=
=fBov
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Christoph Hellwig]

On Wed, Jun 22, 2005 at 02:46:44AM -0500, David Masover wrote:
> >>There's been sloppy code in the kernel before.  I remember one bit in
> >>particular which was commented "Fuck me gently with a chainsaw."  If I
> >>remember correctly, this had all of the PCI ids and the names and
> >>manufacturers of the corresponding devices -- in a data structure -- in
> >>C source code.  It was something like one massive array definition, or
> >>maybe it was a structure.  I don't remember exactly, but...
> >
> >
> > Every device driver has a big array of corresponing device ids as an
> > array in C code - oh my god we're doomed  .. not.
> 
> I could throw the same sarcasm back at you.  We must be doomed because
> Reiser does some stuff that VFS already does!  Or am I misunderstanding
> the complaint?

I rather wanted to say I absolutely don't see any correlation of your
PCI driver example to what we're discussing here.  PCI driver hardcode
ID tables because they are supposed to do that.  And if a PCI driver works
around hardware bugs for a specific subset of hardware it needs to use
an ID table for that one aswell.  And adding a strong comment about broken
hardware is considered to be just fine in Linux kernel land aswell.

Now to your reply.  We're not doomed if a driver re-implements "something"
we already have in common code.  We would be doomed if every driver
reimplements lots of things, that's why we push hard to avoid drivers
doing that.  It gets even more important if that "something" duplicated is
not some simple piece code but complex abstractions.

> How does it get proven if you won't give it a chance as a *separate*
> unproven mess, with a big fat EXPERIMENTAL flag, for users to play with?
> 
> I know, it exists as a separate patch.  But it works now, and I think
> the best way to "prove" it would be to package it with the kernel.

You're free to test it as much outside the kernel tree.  This might make
it more proven one day, but not automatically less of a mess.  The real
point here is that we already have a useful abstraction in that area, and
we spent a lot of work to make it proven and not a mess (or just a little
bit of a mess..), and thus we'd rather see people extending it reasonably
for their needs instead of duplicating lots of infastructure in less than
ideal ways.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christoph Hellwig wrote:
> On Wed, Jun 22, 2005 at 02:46:44AM -0500, David Masover wrote:
> 
>>>>There's been sloppy code in the kernel before.  I remember one bit in
>>>>particular which was commented "Fuck me gently with a chainsaw."  If I
>>>>remember correctly, this had all of the PCI ids and the names and
>>>>manufacturers of the corresponding devices -- in a data structure -- in
>>>>C source code.  It was something like one massive array definition, or
>>>>maybe it was a structure.  I don't remember exactly, but...
>>>
>>>
>>>Every device driver has a big array of corresponing device ids as an
>>>array in C code - oh my god we're doomed  .. not.
>>
>>I could throw the same sarcasm back at you.  We must be doomed because
>>Reiser does some stuff that VFS already does!  Or am I misunderstanding
>>the complaint?
> 
> 
> I rather wanted to say I absolutely don't see any correlation of your
> PCI driver example to what we're discussing here.  PCI driver hardcode
> ID tables because they are supposed to do that.  And if a PCI driver works
> around hardware bugs for a specific subset of hardware it needs to use
> an ID table for that one aswell.  And adding a strong comment about broken
> hardware is considered to be just fine in Linux kernel land aswell.

I seem to remember the comment was more about hardcoded ID tables.

And this was the generic ID code database, which is now maintained out
of the kernel:

/usr/share/misc/pci.ids
	A list of all known PCI ID's (vendors, devices, classes and sub-classes).

That is what I was referring to, that used to be in the kernel.

What this has to do with is whether you believe that it's better to keep
code out until it's perfectly clean, or to let in code that has some
things about it that you don't like, but works, is useful, and the
maintainers seem very willing to fix it later on.

I believe it's better to let stuff in so long as it works and doesn't
break anything, so long as it is being improved to the *real* standards
along the way.  But, this opinion obviously isn't shared by most people
here, so I'm going to stop arguing it now.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr7kMHgHNmZLgCUhAQIHdg/+M0ExRndz9Wym0rTcnSjlg4dc3cM5ZW7N
LZ1lcLM+4Vtwsj16dc9ezNj7VLAx7Vmj/3afW3TxmjQKn50J53ZlTfd6HgpBvkAi
i3V7syjYJg/WaqOlEWCDwO3i/HzYdd9QAgJBbL30g56/ZtJj6SNFlKvb6UizYjIf
dHgY/ZG3BuUKLBTQFUaFuBmkb/eHlhZAq7qbwn6om3bR9UmjDr41l6nP/Ry9k438
gKvwNUQZX9EkQK/F34uDM8S1bbqt8YBcULbUBYp6A12kTL8dLS3Ax8TMbHK0Zxk1
OAMpel7e9kVUbC+NAGv2PRgJQEL4jDzwWS8kP12oBbeacxRnwj1WAKixBM+3v+uq
ThGwcN8CXCfPW8DBokzqYw32vVA+PVpz3tCmnVyobPPPQZdZ5wlKTgZ+Yg4NCp+M
WstSf/LE6OjVIH7xjVLBeZGuynmXHshuEubwRwaiHJKp9kUli+WEpJconvR0W3ph
4WQ6/7px64XxOPnxfR7jjSCNVKjzEZjeXOf1LbJF0a8yh6eVd5g1lsxYS3Ht04w3
yEup3PKWkhvIlGZw8w7IxR4fDI80/t4F9EugILJMLDwMwvJ1k5rRWrEpzF1/I6OM
wdq99cj380B600peLWZ1PIQpqb0iDj+FBqio/MIQSz4Pqlg7qaME21kI8PHxIPib
kJUIRwlarKA=
=TNGY
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Christoph Hellwig]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1047 bytes --]

On Sun, Jun 26, 2005 at 12:21:52PM -0500, David Masover wrote:
> I seem to remember the comment was more about hardcoded ID tables.
> 
> And this was the generic ID code database, which is now maintained out
> of the kernel:
> 
> /usr/share/misc/pci.ids
> 	A list of all known PCI ID's (vendors, devices, classes and sub-classes).
> 
> That is what I was referring to, that used to be in the kernel.

And you once again showed that you don't understand what you're talking
about.  Said database is a pci id to name mapping, which is completely
irrelevant for any driver.  For things like your example there's very
little thing you can do but hardcoding a set of pci ids in one way or
another.

> What this has to do with is whether you believe that it's better to keep
> code out until it's perfectly clean, or to let in code that has some

I doesn't need to be perfect, we just need it in a reasonable state and
have a buying that it's going to continue to evolve in the rigþt direction.

And we're are very far from both of them in this ccase.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christoph Hellwig wrote:
> On Sun, Jun 26, 2005 at 12:21:52PM -0500, David Masover wrote:
> 
>>I seem to remember the comment was more about hardcoded ID tables.
>>
>>And this was the generic ID code database, which is now maintained out
>>of the kernel:
>>
>>/usr/share/misc/pci.ids
>>	A list of all known PCI ID's (vendors, devices, classes and sub-classes).
>>
>>That is what I was referring to, that used to be in the kernel.
> 
> 
> And you once again showed that you don't understand what you're talking
> about.  Said database is a pci id to name mapping, which is completely
> irrelevant for any driver.  For things like your example there's very
> little thing you can do but hardcoding a set of pci ids in one way or
> another.

I hope I admitted ahead of time that I don't understand what I'm talking
about.  For the record -- in this particular example, I don't.

But, the distinction is irrelevant.  PCI id to name mapping used to be
(I think?) in the kernel, it was ugly.  It may not have been where the
chainsaw comment was, though.

>>What this has to do with is whether you believe that it's better to keep
>>code out until it's perfectly clean, or to let in code that has some
> 
> 
> I doesn't need to be perfect, we just need it in a reasonable state and
> have a buying that it's going to continue to evolve in the rigýt direction.
> 
> And we're are very far from both of them in this ccase.

I believe that we are closer to both of those than much of what's
already in the kernel.  Sorry I picked such a bad example, though.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr7rLngHNmZLgCUhAQIl7A//QTNclAuPe0Q+qzZjT7uMXeWXVgxlKGbR
cif4g55qtafpCA0m4/SkLYUmXpAL9Il384tCSK3vvnK7w8bQjGMCGk+xBeWQrDOC
qvuyu+GOaZh+jCeo25IMm5rQRyxrsFb9d+0r+mclN2MDBmzn3l/lMiAIFq6NnlxZ
gP0dOrCGpG1LXohfRhxLTphcmG1UMX/q7WbSSOCNAIMOxoqH2ez0ahdkJ44K+L45
hyMDtDugKCMeJw5r0No8RFl37h6bES/Q3DRv6Q1OQjbk4NYUKo9xCt69ypIvYRLJ
SSNE63CRIjOiOn2sgii1q5kwNj+nShnMrl7bBTjtslMLoariWRPJwswwxMPJh6Nv
xlovTxJKQnmg++KkJSZ6eDc7oCQapcGjeVxRCryHTIphuJ0OgRpw7xM4fsmOSj5R
ruE1XrJZADage92NaozNvCASDh9wLdnkJG9cepJVMsZTb/emDQ54UbOv3yqwtHEm
IbKnDNdSUVs0sgnLErWjRiQjfTh2xn0jof60mquVLufcJ0Ev9n7vWDTUBgKsFLVh
xU4n9y+GWDqT31nql+si+pEJlBeYQCt4Sz9aci7Sni+sVDv929HnRbf4T9GTi99J
5RumBvrBRdfbo/SJos4ttxEQDxUFbIc92UJtYESyLzwH1SIl8lihqfN+fP/oPDdf
unJVGXcgnMQ=
=BJ6I
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Stefan Smietanowski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi David

> And here is the crucial point.  Reiser4 is usable and useful NOW, not
> after it has undergone massive surgery, and Namesys is bankrupt, and
> users have given up and moved on to XFS.  But the massive surgery should
> happen eventually, partly to make all filesystems better (see below),
> and partly to make the transition easier and more palatable for those
> who don't work for Namesys.

Sometimes someone comes with "I have this code NOW and if we just merge
it I'll fix it up properly". It's rejected, stating "come back when
you've fixed it up".

We don't even have that here. We have "I have this code NOW and when
it's merged I have no intention of fixing it up properly".

In my eyes I'd rather take the first one than the second, there at least
there's the INTENTION of fixing it up.

Oh and btw, I'm not pro- or against- reiserfs in any way. I haven't
tried reiserfs4 but I hear it's good so don't take this is a statement
of my like or dislike of it, I'm just stating something worth thinking
about.

// Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)

iD8DBQFCuTXZBrn2kJu9P78RAi0JAJwIp6uyAd3AL4mgSBMAH59h7CrklQCfRW+V
8ZlZqxHwnpqpHa8OURtUEuw=
=OD/E
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Rik Van Riel]

On Tue, 21 Jun 2005, David Masover wrote:

> The point is, this was in the kernel for quite awhile, and it was so
> ugly that someone would rather be fucked with a chainsaw.  If something
> that bad can make it in the kernel and stay for awhile because it
> worked, and no one wanted to replace it

I would like to think we could learn from the mistakes
made in the past, instead of repeating them.

Ugly code often is so ugly people don't *want* to fix
it, so merging ugly code is often a big mistake.

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics

Re: reiser4 plugins

[Alexander Zarochentsev]

On Wednesday 22 June 2005 05:14, Jeff Garzik wrote:
> Hans Reiser wrote:
> > Christoph,
> >
> > Reiser4 users love the plugin concept, and all audiences which have
> > listened to a presentation on plugins have been quite positive about
> > it.  Many users think it is the best thing about reiser4.  Can you
> > articulate why you are opposed to plugins in more detail?  Perhaps you
> > are simply not as familiar with it as the audiences I have presented
> > to.  Perhaps persons on our mailing list can comment.....
> >
> > In particular, what is wrong with having a plugin id associated with
> > every file, storing the pluginid on disk in permanent storage in the
> > stat data, and having that plugin id define the set of methods that
> > implement the vfs operations associated with a particular file, rather
> > than defining VFS methods only at filesystem granularity?
>
> You're basically implementing another VFS layer inside of reiser4, which
> is a big layering violation.

Reiser4 suggests a bit more file types than VFS is aware of. I don't even 
think VFS should be.  It is enough that VFS allows each inode/file/dentry  to 
behave own way.  IIRC VFS object's ->foo_ops is for that.   

Reiser plugins are for the same.  Would you agree with reiser4 plugin design 
if the plugins will not dispatch VFS object methods calls by themselves but 
set ->foo_ops fileds instead?  I guess you don't like to have the two 
dispatching systems at the same level.

> This sort of feature should -not- be done at the low-level filesystem
> level.
>
> What happens if people decide plugins are a good idea, and they want
> them in ext3?  We need massive surgery to extract the guts from reiser4.if 
>
> 	Jeff

Thanks,
Alex

Re: reiser4 plugins

[Christoph Hellwig]

On Wed, Jun 22, 2005 at 06:24:32PM +0400, Alexander Zarochentsev wrote:
> Reiser plugins are for the same.  Would you agree with reiser4 plugin design 
> if the plugins will not dispatch VFS object methods calls by themselves but 
> set ->foo_ops fileds instead?  I guess you don't like to have the two 
> dispatching systems at the same level.

That is exactly the point I want to make.  I haven't looked at the design
in detail for a long time, but schemes to allow different object to have
different operation vectors is a good idea.  We already have that to
varying degrees in all filesystems, and making that more formal is a good
thing.  

Re: reiser4 plugins

[Hans Reiser]

Correct me if I am wrong:

What exists currently in VFS are vector instances, not classes. Plugins,
selected by pluginids, are vector classes, with each pluginid selecting
a vector class. You propose to have the vector class layer (aka plugin
layer) in reiser4 export the vector instance to VFS for VFS to handle
for each object, rather than having VFS select reiser4 and reiser4
selecting a vector class (aka plugin) which selects a method.

If we agree on the above, then I will comment further.

Christoph Hellwig wrote:

>On Wed, Jun 22, 2005 at 06:24:32PM +0400, Alexander Zarochentsev wrote:
>  
>
>>Reiser plugins are for the same.  Would you agree with reiser4 plugin design 
>>if the plugins will not dispatch VFS object methods calls by themselves but 
>>set ->foo_ops fileds instead?  I guess you don't like to have the two 
>>dispatching systems at the same level.
>>    
>>
>
>That is exactly the point I want to make.  I haven't looked at the design
>in detail for a long time, but schemes to allow different object to have
>different operation vectors is a good idea.  We already have that to
>varying degrees in all filesystems, and making that more formal is a good
>thing.  
>
>
>  
>

Re: reiser4 plugins

[Christoph Hellwig]

On Wed, Jun 22, 2005 at 08:39:01PM -0700, Hans Reiser wrote:
> Correct me if I am wrong:
> 
> What exists currently in VFS are vector instances, not classes. Plugins,
> selected by pluginids, are vector classes, with each pluginid selecting
> a vector class. You propose to have the vector class layer (aka plugin
> layer) in reiser4 export the vector instance to VFS for VFS to handle
> for each object, rather than having VFS select reiser4 and reiser4
> selecting a vector class (aka plugin) which selects a method.
> 
> If we agree on the above, then I will comment further.

I'm a bit confused about what you're saying here.  What does the 'vector'
in all these expressions mean?

In OO terminology our *_operation structures are interfaces.  Each particular
instance of such a struct that is filled with function pointers is a class.
Each instance of an inode/file/dentry/superblock/... that uses these operation
vectors is an object.

What I propose (or rather demand ;-)) is that you don't duplicate this
infrastructure, and add another dispath layer below these objects but instead
re-use it for what it is done and only handle things specific to reiser4
in your own code. 

Re: reiser4 plugins

[Artem B. Bityuckiy]

Christoph Hellwig wrote:
> I'm a bit confused about what you're saying here.  What does the 'vector'
> in all these expressions mean?
> 
> In OO terminology our *_operation structures are interfaces.  Each particular
> instance of such a struct that is filled with function pointers is a class.
> Each instance of an inode/file/dentry/superblock/... that uses these operation
> vectors is an object.
> 
> What I propose (or rather demand ;-)) is that you don't duplicate this
> infrastructure, and add another dispath layer below these objects but instead
> re-use it for what it is done and only handle things specific to reiser4
> in your own code. 

Just out of curiosity, could you please specify few exact examples with 
specific file/function names which duplicate the existing 
infrastructure. What do they duplicate and why? How should these 
functions be implemented on VFS? Ho should the the other FSes 
implement/ignore them? Why are you shure they will fit VFS well? etc.

Thanks.

-- 
Best regards, Artem B. Bityuckiy
Oktet Labs (St. Petersburg), Software Engineer.
+78124286709 (office) +79112449030 (mobile)
E-mail: dedekind@oktetlabs.ru, web: http://www.oktetlabs.ru

Re: reiser4 plugins

[Christoph Hellwig]

On Sun, Jun 26, 2005 at 09:07:00PM +0400, Artem B. Bityuckiy wrote:
> Just out of curiosity, could you please specify few exact examples with 
> specific file/function names which duplicate the existing 
> infrastructure. What do they duplicate and why? How should these 
> functions be implemented on VFS? Ho should the the other FSes 
> implement/ignore them? Why are you shure they will fit VFS well? etc.

Right now every file/inode/etc method in reiser4 is just a trivial
wrapper around a plugin call.  It should instead just set the method
table directly in the plugin initialization.  Example:

static int
reiser4_permission(struct inode *inode /* object */ ,
                   int mask,    /* mode bits to check permissions
                                 * for */
                   struct nameidata *nameidata)
{
	/* reiser4_context creation/destruction removed from here,
	   because permission checks currently don't require this.

           Permission plugin have to create context itself if necessary. */
	assert("nikita-1687", inode != NULL);

	return perm_chk(inode, mask, inode, mask);
}

besides a useless assert we just call into perm_chk, which is a macro
obsfucation to call generic_permission which would be called if
->permission was zero.  A hypothetical reiser4 "plugin" that would need
redefine ->permission would just override it in a set inode_operations.

Re: reiser4 plugins

[randy_dunlap]

On Sun, 26 Jun 2005 17:46:06 +0100 Christoph Hellwig wrote:

| On Wed, Jun 22, 2005 at 08:39:01PM -0700, Hans Reiser wrote:
| > Correct me if I am wrong:
| > 
| > What exists currently in VFS are vector instances, not classes. Plugins,
| > selected by pluginids, are vector classes, with each pluginid selecting
| > a vector class. You propose to have the vector class layer (aka plugin
| > layer) in reiser4 export the vector instance to VFS for VFS to handle
| > for each object, rather than having VFS select reiser4 and reiser4
| > selecting a vector class (aka plugin) which selects a method.
| > 
| > If we agree on the above, then I will comment further.
| 
| I'm a bit confused about what you're saying here.  What does the 'vector'
| in all these expressions mean?
| 
| In OO terminology our *_operation structures are interfaces.  Each particular
| instance of such a struct that is filled with function pointers is a class.
| Each instance of an inode/file/dentry/superblock/... that uses these operation
| vectors is an object.
| 
| What I propose (or rather demand ;-)) is that you don't duplicate this
| infrastructure, and add another dispath layer below these objects but instead
| re-use it for what it is done and only handle things specific to reiser4
| in your own code. 

I think that what Hans is trying to say is that the functionality
that is handled by plugin_operations are a different dimension or
vector from the inode/file/dentry etc. operations.
Whether he is right or not is part of the discussion.
and "plugin" is a bad name :(

Is part of the problem that plugins are too extensible?
I.e., the plugin_operations methods can do almost anything?

I would want to see a list of methods that should be supported and then
a reasonable infrastructure for supporting those.
Pick one (like metadata) and drill down on how to support that.

---
~Randy

Re: reiser4 plugins

[Hans Reiser]

Christoph Hellwig wrote:

>On Wed, Jun 22, 2005 at 08:39:01PM -0700, Hans Reiser wrote:
>  
>
>>Correct me if I am wrong:
>>
>>What exists currently in VFS are vector instances, not classes. Plugins,
>>selected by pluginids, are vector classes, with each pluginid selecting
>>a vector class. You propose to have the vector class layer (aka plugin
>>layer) in reiser4 export the vector instance to VFS for VFS to handle
>>for each object, rather than having VFS select reiser4 and reiser4
>>selecting a vector class (aka plugin) which selects a method.
>>
>>If we agree on the above, then I will comment further.
>>    
>>
>
>I'm a bit confused about what you're saying here.  What does the 'vector'
>in all these expressions mean?
>  
>
Was your word, I thought, for the *_operation structures.

>In OO terminology our *_operation structures are interfaces.  Each particular
>instance of such a struct that is filled with function pointers is a class.
>Each instance of an inode/file/dentry/superblock/... that uses these operation
>vectors is an object.
>
>What I propose (or rather demand ;-)) is that you don't duplicate this
>infrastructure, and add another dispath layer below these objects but instead
>re-use it for what it is done and only handle things specific to reiser4
>in your own code. 
>
Well, that is very different from saying that we get rid of the plugin
layer.

So, rather than having a hierarchy, in which we first select filesystem,
and then select plugin, VFS should treat each plugin as though it was a
filesystem, if I understand you. Plugins and pluginids continue to exist
with the exact same functionality, we just eliminate a function
dereference for the *_operation structures. Let's see how it codes up in
its details.

Zam, can you work on this?

Hans

Re: reiser4 plugins

[Christoph Hellwig]

On Sun, Jun 26, 2005 at 04:42:35PM -0700, Hans Reiser wrote:
> >I'm a bit confused about what you're saying here.  What does the 'vector'
> >in all these expressions mean?
> >  
> >
> Was your word, I thought, for the *_operation structures.

We tend to use the term operation vectors, yes.  But that's a different
terminology that doesn't mix up very well with the OO terminology.

> So, rather than having a hierarchy, in which we first select filesystem,
> and then select plugin, VFS should treat each plugin as though it was a
> filesystem, if I understand you.

Kinda.  The VFS doesn't have knowledge about what constitutes a file
system driver, we register filesystems only to know what routine to
call in on a mount with a particular filesystem type - the structure that
does describe a filesystem (struct file_system_type) thus is very small.
Starting from there the filesystem has always been able to use different
methods for different objects (that's something very different from the
vnode-based VFSes in the other UNIX variants).  

> Plugins and pluginids continue to exist
> with the exact same functionality, we just eliminate a function
> dereference for the *_operation structures. Let's see how it codes up in
> its details.

For the file and dir plugin that's pretty much that case.  What's more
problematic is your security plugins.  With LSM we alredy have an
infrastructure to provide pluggable access control.  If there actually
was an implementation of "security" (it's actually access control) for it
besides the default unix permission one it should move to an LSM.  You
could even make your lsm part of the reiser4 module and poke into reiser4
directly from a technical point of view.

Re: reiser4 plugins (the technical email in this flame fest)

[Hans Reiser]

Since this did not get an answer, and since it is the technical email in
the flamefest, I thought I would resend it appropriately labeled.

Correct me if I am wrong:

What exists currently in VFS are vector instances, not classes. Plugins,
selected by pluginids, are vector classes, with each pluginid selecting
a vector class. You propose to have the vector class layer (aka plugin
layer) in reiser4 export the vector instance to VFS for VFS to handle
for each object, rather than having VFS select reiser4 and reiser4
selecting a vector class (aka plugin) which selects a method.

If we agree on the above, then I will comment further.

Christoph Hellwig wrote:

>On Wed, Jun 22, 2005 at 06:24:32PM +0400, Alexander Zarochentsev wrote:
>  
>
>>Reiser plugins are for the same.  Would you agree with reiser4 plugin design 
>>if the plugins will not dispatch VFS object methods calls by themselves but 
>>set ->foo_ops fileds instead?  I guess you don't like to have the two 
>>dispatching systems at the same level.
>>    
>>
>
>That is exactly the point I want to make.  I haven't looked at the design
>in detail for a long time, but schemes to allow different object to have
>different operation vectors is a good idea.  We already have that to
>varying degrees in all filesystems, and making that more formal is a good
>thing.  
>
>
>  
>

Re: reiser4 plugins

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
> What is wrong with having an encryption plugin implemented in this
>  manner?  What is wrong with being able to have some files implemented
>  using a compression plugin, and others in the same filesystem not.
> 
>  What is wrong with having one file in the FS use a write only plugin, in
>  which the encrypion key is changed with every append in a forward but
>  not backward computable manner, and in order to read a file you must
>  either have a key that is stored on another computer or be reading what
>  was written after the moment of cracking root?
> 
>  What is wrong with having a set of critical data files use a CRC
>  checking file plugin?

I think the concern here is that this is implemented at the wrong level.

In Linux, a filesystem is some dumb thing which implements
address_space_operations, filesystem_operations, etc.

Advanced features such as those which you describe are implemented on top
of the filesystem, not within it.  reiser4 turns it all upside down.

Now, some of the features which you envision are not amenable to
above-the-fs implementations.  But some will be, and that's where we should
implement those.

Re: reiser4 plugins

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 5391 bytes --]

Am Dienstag, den 21.06.2005, 18:18 -0700 schrieb Andrew Morton:

> > What is wrong with having an encryption plugin implemented in this
> >  manner?  What is wrong with being able to have some files implemented
> >  using a compression plugin, and others in the same filesystem not.
> > 
> >  What is wrong with having one file in the FS use a write only plugin, in
> >  which the encrypion key is changed with every append in a forward but
> >  not backward computable manner, and in order to read a file you must
> >  either have a key that is stored on another computer or be reading what
> >  was written after the moment of cracking root?
> > 
> >  What is wrong with having a set of critical data files use a CRC
> >  checking file plugin?
> 
> I think the concern here is that this is implemented at the wrong level.
> 
> In Linux, a filesystem is some dumb thing which implements
> address_space_operations, filesystem_operations, etc.
> 
> Advanced features such as those which you describe are implemented on top
> of the filesystem, not within it.  reiser4 turns it all upside down.
> 
> Now, some of the features which you envision are not amenable to
> above-the-fs implementations.  But some will be, and that's where we should
> implement those.

Yes, but that would be difficult, probably worse. The name "plugins" is
perhaps a bit misleading. These plugins are most of the time some sort
client to the reiser4 on-disk database structure. The core code is does
on-disk tree handling, journalling and these things. The plugins in turn
glue this core database system to the rest of the system in order to
make a filesystem of it. The "file plugin" tells the database how to
store files.

A compression plugins is more tricky. Files should be randomly
accessible and if you write in the middle of the file the compressed
block might change in size. For reiser4 this is not a problem since it
just tells the underlying database "give me some room for 1234 bytes and
insert it in the tree instead of the other block". The reiser4 core has
totally different semantics than the VFS layer and I don't see how these
things could be handled elsewhere in this simple way.

The reiser4 core is more some sort of library that abstracts a block
device and provides some sort of database API (which is highly optimized
for filesystem purposes). The actual filesystem is then another layer on
top of this. You could actually implement lots of different filesystems
on top of that database core.

The actual layer is a bit different though. The database core itself
registers with the Linux VFS and then passes the calls down to one of
the plugins which then calls back into the reiser4 core to do the actual
database modification. I think this was the point that Christoph was
criticizing the most.

Currently it looks like this:

             ,--------------.       ,--------------.
VFS -------> |              | ----> |              |
             | /fs/reiser4/ |       | .../plugins/ |
blockdev <-- |              | <---> |              |
             `--------------'       `--------------'

So the reiser4 code is introducing another abstraction of the Linux VFS
layer instead of letting the plugins define the Linux VFS ops directly.
Which would look like this:

                                    ,--------------.
VFS ------------------------------> |              |
             ,--------------,       | .../plugins/ |
blockdev <-- | /fs/reiser4/ | <---> |              |
             `--------------'       `--------------'

Which probably would be okay for most of the time except for some
details (which could probably be solved otherwise).

Actually the flow is not always this simple, usually the path goes back
and forth multiple time between the core and the plugins.

One of the features Hans is using is that there can be different kinds
of files. The on-disk structure tells the core which of the plugins is
responsible for the "database object" found on the disk. It could be a
directory or a "stat data" (inode) or a file. The file itself could be
handled by different plugins like one that stores the data directly or
one that compresses it.

reiser4 is different than other filesystems in that it uses a lot more
abstraction levels. The database aspect and the semantic aspect of a
traditional filesystems are strongly separated.

To understand the code probably means a lot of work because it is a bit
different. Some of the layering concerns may be right, other probably
not.

The plugins that add additional VFS semantics (that are currently
disable) should most definitely not be implemented only inside the
filesystem. I think Hans did this because it would have been a lot more
work doing this at the proper layer (which means talking to people and a
lot of politics...).

The last time I looked at the code is a while ago, so if I'm wrong on
something, please don't shoot me. The only thing I can say is that
reiser4 has very stable for me (though I've gone back to reiser3 for
most of my filesystems because I wanted acl/xattr).

So here's my advice: Instead of insulting each other or doing pure
marketing talk, please try to address each detail and explain why
something has been done and if it's good or bad and if it should be
changed and how.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Artem B. Bityuckiy]

Christophe Saout wrote:
> The plugins that add additional VFS semantics (that are currently
> disable) should most definitely not be implemented only inside the
> filesystem. I think Hans did this because it would have been a lot more
> work doing this at the proper layer (which means talking to people and a
> lot of politics...).
I would even do s/should most definitely not/must not/

More filesystems in future may want to use these semantics. There are 
cases when one can't use Reiser4 to implement them, but instead, need to 
implement another file system with the same semantics.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 892 bytes --]

On Wed, Jun 22, 2005 at 07:10:58PM +0400, Artem B. Bityuckiy wrote:
>
>More filesystems in future may want to use these semantics. There are 
>cases when one can't use Reiser4 to implement them, but instead, need to 
>implement another file system with the same semantics.

So merge it as it is and move the stuff to the VFS as needed or
deemed necessary. And enable the pseudo interface, or at least
set it in menuconfig and enable by default, it needs testing too.

Then someone says "we can't implement this in the fs" and someone
else says "we can't implement this in the vfs" and someone else
says "this is a good thing which we want but you won't let us"
and we stagnate again...

Isn't this bickering getting a bit old?
After all, it seems the code is merged in -mm, the big issues are fixed
and all should be ready for more real-life testing and such?

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Artem B. Bityuckiy]

Markus TЖrnqvist wrote:
> So merge it as it is and move the stuff to the VFS as needed or
> deemed necessary. And enable the pseudo interface, or at least
> set it in menuconfig and enable by default, it needs testing too.
Reiser4 has a number of great (IMO) things like file as directory, 
atomic operations, different kinds of stat data, fibretions, etc, etc. 
Some thing is not yet ready - doesn't matter. Some of this is of general 
interest, some is Reiser4-dedicated.

New interfaces are needed to allow users to utilize that all. My point 
is that the things that are of general interest must not be 
Reiser4-only. For example, I should have a possibility to implement 
files-like-dir in _another_ FS using the same interfaces that Reiser4 
uses. That's all I wanted to say.

The other question that it may be difficult to foresee everything and it 
may make sense to move some things upper in future.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: reiser4 plugins

[M.]

Hi,

Is it not simpler to ask the reiserfs guys for a detailed explanation
of why and where this plugins' layer differs from using VFS for
plugins and let others comment on that ?
If something cant be done using VFS this layer is needed by reiser4
and has to be merged.

Michele

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

On Wed, Jun 22, 2005 at 06:46:50PM +0200, M. wrote:
>
>Is it not simpler to ask the reiserfs guys for a detailed explanation
>of why and where this plugins' layer differs from using VFS for
>plugins and let others comment on that ?

I hope this is not FUD or something like that, but it seems to me
the VFS guys are not too willing to implement Reiser4-HiFi stuff anywhere
else, like the VFS, and don't want Reiser4 in, because it duplicates.

Think of the chicken and the egg.

I may be wrong, though.

>If something cant be done using VFS this layer is needed by reiser4
>and has to be merged.

But it's still possible to let Reiser4 in and take them, say, a feature
at a time to the VFS.

Unless it's going to be a political wankfest/pissing contest about
if things like this should even be implemented, anywhere, ever :P

-- 
mjt



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Horst von Brand]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2883 bytes --]

Artem B. Bityuckiy <dedekind@yandex.ru> wrote:
> Markus TЖrnqvist wrote:
> > So merge it as it is 

Fix it first. The "merge as it stands" just gives rise to stuff that is
/never/ fixed properly.

> >                      and move the stuff to the VFS as needed or
> > deemed necessary. And enable the pseudo interface, or at least
> > set it in menuconfig and enable by default, it needs testing too.

Then test it out of the standard tree...

> Reiser4 has a number of great (IMO) things like file as directory,

Urgh.

> atomic operations,

What is atomic that isn't in the standard filesystems? How do you guarantee
it doesn't stop the system dead in its tracks waiting for some very long
transaction to finish?

>                    different kinds of stat data,

Usefulness? Sounds like kernel bloat leading to userspace bloat and
applications/users wondering what the heck goes on when they don't grok the
particular stat format.

>                                                  fibretions, etc,

???

> etc. Some thing is not yet ready - doesn't matter. Some of this is of
> general interest, some is Reiser4-dedicated.

I don't see anything that would interest me at least, so you can safely
scratch the "general interest" part.

> New interfaces are needed to allow users to utilize that all.

That is a quite strong argument /against/ it all in my book. It means bloat
in /every/ filesystem, and they have shown to be able to do without for
some 30 years now. I'd need /very/ strong reasons for adding something.

>                                                               My point
> is that the things that are of general interest must not be
> Reiser4-only.

Reiser4-only stuff is of very limited use, if it isn't just internal
stuff. And that doesn't need any changes.

>               For example, I should have a possibility to implement
> files-like-dir in _another_ FS using the same interfaces that Reiser4
> uses. That's all I wanted to say.

It has been argued over and over that that particular feature /can't/ be
implemented sanely anyway, so it has to stay out. Besides not making any
sense. "You've got files and directories" is a bit asymetrical and so not
quite nice; "all you have is directories" is symmetrical, estetic, and
completely useless; "some files are directories, some aren't; files in
file-directories are different than regular files in directory-directories"
is just a mindless jumble.

> The other question that it may be difficult to foresee everything and
> it may make sense to move some things upper in future.

Another good reason to keep it out ;-)
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
> Artem B. Bityuckiy <dedekind@yandex.ru> wrote:
> 
>>Markus TЖrnqvist wrote:
[...]
>>>                     and move the stuff to the VFS as needed or
>>>deemed necessary. And enable the pseudo interface, or at least
>>>set it in menuconfig and enable by default, it needs testing too.
> 
> 
> Then test it out of the standard tree...

Standard tree is actually a good place to test things.  That's why there
are so many things there that say "EXPERIMENTAL", and so many of those
don't work.  Some even refuse to compile.

>>atomic operations,
> 
> 
> What is atomic that isn't in the standard filesystems? How do you guarantee

We've been discussing this for quite awhile.  In standard filesystems,
the only things that are atomic are metadata.  If I rename a file and
crash, I'm guarenteed to either have renamed the file or not, and not be
caught halfway.

This does not apply to files.  In fact, the only way to perform an
atomic write on a file, using the filesystem's atomicity, is to write a
new file, nuke the old one, and rename the new one on top of the old one.

What we want is to have programs that can write small changes to one
file or to many files, lump all those changes into a transaction, and
have the transaction either succeed or fail.

> it doesn't stop the system dead in its tracks waiting for some very long
> transaction to finish?

We've also discussed this.  For one thing, if we can have transactions
in databases which don't stop the database dead in its tracks, why can't
we do it with filesystems?

But anyway, if you really want to know, ask someone else or read the
archives.  I wasn't really paying attention except to remember that this
issue was resolved.

>>                   different kinds of stat data,
> 
> 
> Usefulness? Sounds like kernel bloat leading to userspace bloat and
> applications/users wondering what the heck goes on when they don't grok the
> particular stat format.

So you allow multiple stat formats.  Bloat is not as big an issue here
as the bloat of existing systems which run on top of the FS and don't
cooperate.  Gnome and KDE each have their own VFS, for instance.

>>                                                 fibretions, etc,
> 
> 
> ???

Low-level tweaking.  I think the word is from some sort of calculus.

>>etc. Some thing is not yet ready - doesn't matter. Some of this is of
>>general interest, some is Reiser4-dedicated.
> 
> 
> I don't see anything that would interest me at least, so you can safely
> scratch the "general interest" part.

You're the sole general public?

>>New interfaces are needed to allow users to utilize that all.
> 
> 
> That is a quite strong argument /against/ it all in my book. It means bloat
> in /every/ filesystem, and they have shown to be able to do without for
> some 30 years now. I'd need /very/ strong reasons for adding something.

Spotlight on the Mac.  Users love it.  We can do it.  But not without
changing something in the filesystem.

Actually, I think we came up with several ways to do this, all of which
required Reiser4 interfaces.

It's "bloat" until you need it.

>>                                                              My point
>>is that the things that are of general interest must not be
>>Reiser4-only.
> 
> 
> Reiser4-only stuff is of very limited use, if it isn't just internal
> stuff. And that doesn't need any changes.

Only it does.  Reiser4 can't get into the kernel because it duplicates
the VFS in order to extend it.  It couldn't get in before because it
extended the VFS directly.

Maybe you just don't want this system to EVER get in, no matter what
they do to it?

>>              For example, I should have a possibility to implement
>>files-like-dir in _another_ FS using the same interfaces that Reiser4
>>uses. That's all I wanted to say.
> 
> 
> It has been argued over and over that that particular feature /can't/ be
> implemented sanely anyway, so it has to stay out. Besides not making any

It's been dropped for the moment, but it's been argued just as many
times that it can be done sanely.

> sense. "You've got files and directories" is a bit asymetrical and so not
> quite nice; "all you have is directories" is symmetrical, estetic, and
> completely useless; "some files are directories, some aren't; files in
> file-directories are different than regular files in directory-directories"
> is just a mindless jumble.

Or you can just say "There are no files.  There are no directories.
There are only objects, which contain a chunk of linear data and other
objects."

If you actually go read the whitepaper, you will discover that this is
actually a cleaner, more esthetic, and more useful model than the
current one.  It's just a little harder to do.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrndSHgHNmZLgCUhAQKOCxAAlhBiu234fmMaUv7kYp0Xt1V2yjP67c8U
ggUbB6sSac1CDo3XQm3cWcR9jqVTTxDCZx012XVEM2MQ7n69YF6+xTxFK/wS69rO
cFh7ChBDe71YOB3aBQ1ojG5D/cM8G4VPVF3pt7ywqoEK67VNwynLvdjjWYYD5w9y
mZd+eRo6kpCmjakNwOpU4LOA8u3C+OK3o2bduzFSMquV9dckV4vt8rxTi1R1l12s
aTBuGc1yvye19vo6DQv28hVcqbv7GmyA5+oBg+TdsFF6Oy/+oR5h4bsfDlFR6Ycs
u23K0yS1b+Iivszqb/wM4CewmkiTOKlxjlR4t93isfsRM1alVeFTfsVP3OYBVRwO
42upk/yzVvliP32n5smZVssf99UK0LuXpb+NTIXABm7WDOtcGV1gC1zz93g0NcBw
Huh82+OPH1l8JXn+4iuZsHBx2VtG5Q7ZENGqXRcEZMb+bYECzFhPNVYBWlPBK+LC
rk/byrkO6zKBkQ5mh7BeK/JrcKVR/IEH+OjpSPmwuam5oPSaoHt9ep5zV+oFYCU6
KIRh8/p7br9i6A2WKUMs+v8j+LcA+Lg+8fbW7RC5LXjG6+QpwvBTuZGU5ykRbj2i
KmrQPCg3x3ZXDGkMLEUySPFL0P3JHpLxsf6tfVEZtsyXCHn1MOeo5sS13NLU719P
6wYcB+VAbSE=
=0NX7
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Nikita Danilov]

David Masover writes:

[...]

 > 
 > What we want is to have programs that can write small changes to one
 > file or to many files, lump all those changes into a transaction, and
 > have the transaction either succeed or fail.

No existing file system guarantees such behavior. Even atomicity of
single system call is not guaranteed.

 > 
 > > it doesn't stop the system dead in its tracks waiting for some very long
 > > transaction to finish?
 > 
 > We've also discussed this.  For one thing, if we can have transactions
 > in databases which don't stop the database dead in its tracks, why can't
 > we do it with filesystems?

Because to have such transactions databases pay huge price in both
resource consumption and available concurrency (isolation, commit-time
locks, etc.), and yet mechanism they use to deal with stuck transactions
(which is simply to abort it) is not very suitable for the file system.

 > 
 > But anyway, if you really want to know, ask someone else or read the
 > archives.  I wasn't really paying attention except to remember that this
 > issue was resolved.

That would be real breakthrough.

[...]

 > 
 > >>                                                 fibretions, etc,
 > > 
 > > 
 > > ???
 > 
 > Low-level tweaking.  I think the word is from some sort of calculus.

Fibration. http://marc.theaimsgroup.com/?l=linux-kernel&m=108032604606183&w=2

Nikita.

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Nikita Danilov wrote:
> David Masover writes:
> 
> [...]
> 
>  > 
>  > What we want is to have programs that can write small changes to one
>  > file or to many files, lump all those changes into a transaction, and
>  > have the transaction either succeed or fail.
> 
> No existing file system guarantees such behavior. Even atomicity of
> single system call is not guaranteed.

No _existing_ filesystem.  But I seem to recall that this was one of the
design decisions of Reiser4, and that the system call itself was pushed
off to 4.1?

Maybe I'm just wrong about how big a transaction can be.  Maybe it was
limited to a single file.  I don't think so, though.  From the
whitepaper:  "Stuffing a transaction into a single file just because you
need the transaction to be atomic is hardly what one would call flexible
semantics."

I also seem to recall that the rolling back of the transaction, should
it fail, was supposed to be handled by the application.  This doesn't
quite click with the whitepaper, but it could work.

More whitepaper goodness:

"A new system call sys_reiser4() will be implemented to support
applications that don't have to be fooled into thinking that they are
using POSIX. Through this entry point a richer set of semantics will
access the same files that are also accessible using POSIX calls.
Reiser4() will not implement more than hierarchical names. A full set
theoretic naming system as described on our future vision page will not
be implemented before Reiser6() is implemented (Reiser5 is our
distributed filesystem, Reiser6 is our enhanced semantics, whether we
implement Reiser5 or Reiser6 first depends on which sponsors we find ;-)
). Reiser4() will implement all features necessary to access ACLs as
files/directories rather than as something neither file nor directory.
These include opening and closing transactions, performing a sequence of
I/Os in one system call, and accessing files without use of file
descriptors (necessary for efficient small I/O). Reiser4 will use a
syntax suitable for evolving into Reiser5() syntax with its set
theoretic naming."

So, some sort of transaction is planned.

But, as I said, I wasn't paying enough attention.  Maybe there is a
technical reason why this can't be done in Linux?

>  > > it doesn't stop the system dead in its tracks waiting for some very long
>  > > transaction to finish?
>  > 
>  > We've also discussed this.  For one thing, if we can have transactions
>  > in databases which don't stop the database dead in its tracks, why can't
>  > we do it with filesystems?
> 
> Because to have such transactions databases pay huge price in both
> resource consumption and available concurrency (isolation, commit-time
> locks, etc.), and yet mechanism they use to deal with stuck transactions
> (which is simply to abort it) is not very suitable for the file system.

Oh, really?  If we've got application support through sys_reiser4?  The
application should be ready to deal with a transaction abort.

I'm still not convinced of any of that paragraph.  I don't know enough
to argue the point, but it intuitively feels wrong.  After all, if the
metadata is atomic, and we are allowed to make our own system calls, why
can't we make the data atomic?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrrGOngHNmZLgCUhAQJZhw//dmJ6S2GlGT6J5YI9DTCyoTDIPUYNb8o0
M1me6KDTElzzQ3yUak/eUd0sbGBAQcf0vn/iVscfq2DoAwnUWxHjht+PaOA98axR
0pnofqE291QLTQJ36epW0kKqFjavVVsrpD80llcaCFz9Rq48W40DoI5CWuX1RQqK
pCnr9vYe8cAsRY+PzV9/KUaSQ+eZJ9daLsAmMwA3Gcxo4XYqILlZm90X3QQTdc8W
gnKSabG3zIjEozfgG/nvtV/09mktHINGq3ud8W1XubBOXs4z+ECsLyvi7QNW83Bq
b/wTDUX3PkrjDHnfcmFkFZJqRrCBD9Ko36f9NThxuaba5eV7kb6h+qx+kS5ZM6Lm
bh90TjJrIpJ4aQr2qrPRAE85GSnvSlyi3E01gk/+UnkBFMoTqTvw2dPb0GhvMINM
EhSUhEyeaopWXIdv3IszOOpbHJLwczixLDBtZ8OFDS26bnJGj7YlnTjdf+TZ9CGf
ZXn7GaG16CiSTOt0YkKk2UGZz+AOubPAUHc6v8Wg587qXWKD3cXVQeZVqYwJG/8B
G5qq51LB6jypjAoP4uSeuTs4DfANGi2H2mHjZVAyaGcwzhxf3ffGRFkfjElAj8RA
RdmB6bq10nt32mL7YP8+n7xa38iCP+ks9wsoOY2KBBDlOpHu07xb/c2DS9yfTCpj
FvMShQw6EBI=
=S7ds
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

David Masover wrote:

> Nikita Danilov wrote:
>
> >David Masover writes:
>
> >[...]
>
> > >
> > > What we want is to have programs that can write small changes to one
> > > file or to many files, lump all those changes into a transaction, and
> > > have the transaction either succeed or fail.
>
> >No existing file system guarantees such behavior. Even atomicity of
> >single system call is not guaranteed.
>
>
> No _existing_ filesystem.  But I seem to recall that this was one of the
> design decisions of Reiser4, and that the system call itself was pushed
> off to 4.1?

Thats right.

>
> Maybe I'm just wrong about how big a transaction can be.

No, you are not.

Isolation is not easy for an fs, but fusing multiple atoms from multiple
file modifications into one "guaranteed to commit or fail all together"
atom is easy.

Re: reiser4 plugins

[Nikita Danilov]

David Masover writes:
 > -----BEGIN PGP SIGNED MESSAGE-----
 > Hash: SHA1
 > 
 > Nikita Danilov wrote:
 > > David Masover writes:
 > > 
 > > [...]
 > > 
 > >  > 
 > >  > What we want is to have programs that can write small changes to one
 > >  > file or to many files, lump all those changes into a transaction, and
 > >  > have the transaction either succeed or fail.
 > > 
 > > No existing file system guarantees such behavior. Even atomicity of
 > > single system call is not guaranteed.
 > 
 > No _existing_ filesystem.  But I seem to recall that this was one of the
 > design decisions of Reiser4, and that the system call itself was pushed
 > off to 4.1?

First off, my comment was in response to the message

http://marc.theaimsgroup.com/?l=linux-kernel&m=111945793205480&w=2

claiming that reiser4 was "atomic". As it stands, currently reiser4
cannot guarantee that single write(2) call is atomic. Think of system
with X bytes of physical memory and 9*X bytes of swap. User does

        buf = malloc(10 * X);
		memset(buf, 42, 10 * X);
        write(reiser4_fd, buf, 10 * X);

As far as I know, current reiser4 code cannot guarantee that such write
is always handled as a single transaction. It is _possible_ to implement
such a guarantee, I believe, but this requires a lot of work.

 > 
 > Maybe I'm just wrong about how big a transaction can be.  Maybe it was
 > limited to a single file.  I don't think so, though.  From the
 > whitepaper:  "Stuffing a transaction into a single file just because you
 > need the transaction to be atomic is hardly what one would call flexible
 > semantics."

I don't quite understand what do you mean. Whitepaper describes how
things work according to some grand visionary scheme. It doesn't go to
the level of technical details, it doesn't discuss possible
obstacles. There is a huge gap between saying "it's nice to have
user-visible transactions" and having ones. So huge, in my opinion, that
compared with it, difference in reiser4 and, say, ext3 as possible
starting points is immaterial.

 > 
 > I also seem to recall that the rolling back of the transaction, should
 > it fail, was supposed to be handled by the application.  This doesn't
 > quite click with the whitepaper, but it could work.

What if user level application err... crashed (does this happen?) or
stuck in the infinite loop (hmm... user level programmers never do this,
I hope)?

[...]

 > > 
 > > Because to have such transactions databases pay huge price in both
 > > resource consumption and available concurrency (isolation, commit-time
 > > locks, etc.), and yet mechanism they use to deal with stuck transactions
 > > (which is simply to abort it) is not very suitable for the file system.
 > 
 > Oh, really?  If we've got application support through sys_reiser4?  The
 > application should be ready to deal with a transaction abort.

Transactional systems have to deal with so-called external
actions. E.g., in the ATM external action is to hand cash out to the
person doing withdraw. After external action was done as part of
transaction, that transaction can no longer be aborted (because one
cannot reverse external action). Typical example of file system
transactions is mail server doing something roughly following:

        (0) begin work;
        (1) take message from the queue;
        (2) send message to the recipient;
        (3) wait for confirmation;
        (4) remove message;
        (5) commit work;

Here (2), and (3) are external actions, they cannot be aborted or
reversed. This means that should transaction abort at the step (4)
system would be left in the inconsistent state.

 > 
 > I'm still not convinced of any of that paragraph.  I don't know enough
 > to argue the point, but it intuitively feels wrong.  After all, if the
 > metadata is atomic, and we are allowed to make our own system calls, why
 > can't we make the data atomic?

Because meta-data are under tight kernel control. There is a difference
between implementing transaction engine for use by a small body of
trusted code (kernel) and a general purpose transaction manager.

Nikita.

 > 
 > -----BEGIN PGP SIGNATURE-----
 > Version: GnuPG v1.4.1 (GNU/Linux)
 > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
 > 
 > iQIVAwUBQrrGOngHNmZLgCUhAQJZhw//dmJ6S2GlGT6J5YI9DTCyoTDIPUYNb8o0
 > M1me6KDTElzzQ3yUak/eUd0sbGBAQcf0vn/iVscfq2DoAwnUWxHjht+PaOA98axR
 > 0pnofqE291QLTQJ36epW0kKqFjavVVsrpD80llcaCFz9Rq48W40DoI5CWuX1RQqK
 > pCnr9vYe8cAsRY+PzV9/KUaSQ+eZJ9daLsAmMwA3Gcxo4XYqILlZm90X3QQTdc8W
 > gnKSabG3zIjEozfgG/nvtV/09mktHINGq3ud8W1XubBOXs4z+ECsLyvi7QNW83Bq
 > b/wTDUX3PkrjDHnfcmFkFZJqRrCBD9Ko36f9NThxuaba5eV7kb6h+qx+kS5ZM6Lm
 > bh90TjJrIpJ4aQr2qrPRAE85GSnvSlyi3E01gk/+UnkBFMoTqTvw2dPb0GhvMINM
 > EhSUhEyeaopWXIdv3IszOOpbHJLwczixLDBtZ8OFDS26bnJGj7YlnTjdf+TZ9CGf
 > ZXn7GaG16CiSTOt0YkKk2UGZz+AOubPAUHc6v8Wg587qXWKD3cXVQeZVqYwJG/8B
 > G5qq51LB6jypjAoP4uSeuTs4DfANGi2H2mHjZVAyaGcwzhxf3ffGRFkfjElAj8RA
 > RdmB6bq10nt32mL7YP8+n7xa38iCP+ks9wsoOY2KBBDlOpHu07xb/c2DS9yfTCpj
 > FvMShQw6EBI=
 > =S7ds
 > -----END PGP SIGNATURE-----

Re: reiser4 plugins

[Hans Reiser]

Nikita, I respectfully disagree with what you say about the state of our
atomicity code.   It is not so far away as you describe, and probably 6
man weeks work could polish it off.  You don't see the value in what I
define as useful, namely atomicity without isolation.  Since you don't
see that, it is harder for you to see that something is close to working
and just needs 6 weeks of someone who groks what I am asking for.

Hans

Re: reiser4 plugins

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
> Nikita, I respectfully disagree with what you say about the state of our
> atomicity code.   It is not so far away as you describe, and probably 6
> man weeks work could polish it off.  You don't see the value in what I
> define as useful, namely atomicity without isolation.  Since you don't
> see that, it is harder for you to see that something is close to working
> and just needs 6 weeks of someone who groks what I am asking for.

How about a poor-man's isolation -- any process other than that
responsible for the transaction sees a consistent state, never a
transaction-in-progress.  I'm sure there's a name for that.

So, until the transaction commits, all apps can still see the original
state, except the one actually writing the transaction.  After it's
done, all apps see the new consistent state.

Isolation can then be done easily in userspace, via voluntary locking.
If the isolation needs to be global, the user can take a performance hit
and do mandatory locking.  Either way, locking and unlocking could be
seen as part of the transaction.

Details follow, most of them pure speculation on my part...

I'm not sure how this ties into the bigger kind of transaction, as
you've decided to let userspace handle the rollback in the large,
multi-file transactions, but for a single file, it seems obvious (famous
last words).  You've got the wandering stuff, meaning that the new file
(or changes to a file) are written to a new space on disk, after which
you update the metadata (blocklist?) -- presumably atomically.

So, it already seems inherent in the system, but it should be defined.
For some transactions, it doesn't make sense to do this, especially huge
ones (where storing changes separately may well fill up the disk) which
make more sense to have an application handle the rollback.

Here's a bad example of such a situation:  We're doing a brand-new
install, and we start by unpacking a tarball from the installation
media.  After the tarball is unpacked, we start installing packages.
But, if the installation fails, maybe we want to roll back to the
initial tarball, change some installation parameters, and try again.  In
such a situation, it makes more sense to just unpack the tarball again
than to try to roll back each change individually.

This is a bad example because you shouldn't have to start over from the
beginning, and even if you did, exit codes should be enough for this
kind of installer.

Anyway, I'm not sure quite how this works when dealing with more than
one file, but we already have atomic metadata operations, and here we
have an example of a somewhat atomic file operation, so I'm sure the two
can be stringed together... somehow...  Conceptually, it seems you'd
just be building a COW subtree, and the actual "atomic" part consists of
simply changing the pointer from the original tree to the modified one.
 The cleanup of the original tree need not be atomic.

We do have COW support, don't we?  It was discussed, and we should.

Main point is, I've just described two kinds of transactions:  entirely
FS-managed and userland-rolled-back.  I think they are both useful.
Also, neither looks incredibly hard to implement.  I'm sure I'm wrong
about that, though.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQru0fHgHNmZLgCUhAQLX9hAAjSO4WkIaqyzeTYyz3l9PW5mWvvUKhxrv
eK3glrQvrEf2HHy8gBfdcqv5RV8BAHfYIh6V/4S9VqmMzpBcUfBfEGRCJg0vcTsj
oVSpzV0/ASdxitMjsHqIz3q3kNRhRuhZoD/ppsVVlH4zZtefioYecyD2asEzTz6w
vAMN08chHR/vQiSDMBPDwkJ5Ov1tyPqdpSA4vBuRreWhDpD32ChBd3epEVXjt090
P0HxddqGR6FXCo9m4z2nTgSFHhAPkrOJE5bNS72hpk94nTfLRMI1DDTKw7J4AO10
U2mdjaXD5AUyahwE3diMcsxrH+94orE5bmK4yvgLUquBZ+OeRFoH0u8FG7sD3coP
2YkJewbhSrCQD69hrrR3pluwds3aoe+awj3x2WOk+6NV9xajw0czPs1P3cEw2Dwt
Fl2+92M6545UUbysL51Dvk+DWp/PRniAL1nBoZ1q9nBlhYIiL7ccr48sXP2PxoZw
nSQ1BDASCZalS6AFlMhI6A6U/OxIMzEZQr/cq1J9/dwRdrST2VLqiQdaCmsfgmL5
EVxA1C0tFjOPWwAlzVHB7MisxDp491w6WrxzV3r8x9vCwJZ747+2RNF3z2ah0A56
bVV6ZTjOK/pAsC0p+cfZS3NOTtvmP8x9QXQFll9WZvVEgFMYaOYkPlL6qwtpdNNR
r0yOvDy1TLI=
=FeCh
-----END PGP SIGNATURE-----

Re: reiser4 plugins

[Bernd Eckenfels]

In article <42BBB47C.9010002@slaphack.com> you wrote:
> How about a poor-man's isolation -- any process other than that
> responsible for the transaction sees a consistent state, never a
> transaction-in-progress.  I'm sure there's a name for that.

It is Isolation Level Serializeable. It is the less performant isolation
level and still can generate deadlocks if you have two process doing
transactions.

A more simpler solution would be that process without transactions never see
infligt tx and a second process simple returnes a retry error if touching a
locked ressource.

Bernd

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser <reiser@namesys.com> writes:

> Nikita, I respectfully disagree with what you say about the state of our
> atomicity code.   It is not so far away as you describe, and probably 6
> man weeks work could polish it off.  You don't see the value in what I
> define as useful, namely atomicity without isolation.  Since you don't
> see that, it is harder for you to see that something is close to working
> and just needs 6 weeks of someone who groks what I am asking for.

No, I see the value of "atomicity", and think it is very useful. What I
don't see the value of is the making of premature claims.

_When_ reiser4 has atomic write(2), you have full right to call it
atomic, not before.

_When_ reiser4 is tested through objective benchmark-set exercising
various workloads, you can refer to these benchmarks as the proof of
reiser4 technical superiority, not before.

On a more personal note, I invested large amount of my time and effort
into developing reiser4, and I feel attached to it and to the great
ideas embodied in it. For reiser4 to rot on the forgotten shelf in
obscurity is the thing I want least. I want it to be included into
mainline kernel, but for this to happen, you have to take more realistic
stance towards err... reality.

>
> Hans

Nikita.

Re: reiser4 plugins

[Roland Dreier]

    David> Spotlight on the Mac.  Users love it.  We can do it.  But
    David> not without changing something in the filesystem.

    David> Actually, I think we came up with several ways to do this,
    David> all of which required Reiser4 interfaces.

It seems the existing Beagle project is a counterexample to this.  In
fact, to make things even more perfect, Beagle doesn't work on top of
Reiser4.

 - R.

Re: reiser4 plugins

[Artem B. Bityuckiy]

Horst von Brand wrote:
> I don't see anything that would interest me at least, so you can safely
> scratch the "general interest" part.
For me that's all very interesting. I believe people must take a chance 
Reiser4 to evolve as a part of the kernel, albeit there are many issues.
Rape me, but I even think it is a kind of privilege for Linux to have 
Reiser4 in it (it could have gone to another OS).

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: reiser4 plugins

[Markus Törnqvist]

[-- Attachment #1: Type: text/plain, Size: 3496 bytes --]

On Wed, Jun 22, 2005 at 01:33:14PM -0400, Horst von Brand wrote:
>> > So merge it as it is 
>Fix it first. The "merge as it stands" just gives rise to stuff that is
>/never/ fixed properly.

It's in -mm already and it was said the big grudges are fixed.

What do you still want?

>> >                      and move the stuff to the VFS as needed or
>> > deemed necessary. And enable the pseudo interface, or at least
>> > set it in menuconfig and enable by default, it needs testing too.
>
>Then test it out of the standard tree...

Huh? You mean -mm? That's what people do, backport to mainline.

Or did I miss something?

>> Reiser4 has a number of great (IMO) things like file as directory,
>Urgh.

Your opinion :)

>>                    different kinds of stat data,
>Usefulness? Sounds like kernel bloat leading to userspace bloat and
>applications/users wondering what the heck goes on when they don't grok the
>particular stat format.

Oh no! NOOOOOOOO! New features come at a cost?!

Shit, I better unpack my C64.

>> etc. Some thing is not yet ready - doesn't matter. Some of this is of
>> general interest, some is Reiser4-dedicated.
>
>I don't see anything that would interest me at least, so you can safely
>scratch the "general interest" part.

These are things people will start using when available.

They're given a framework which enables them to do things other
people can't imagine.

Too bad you're convinced everything everyone does is on the level of "Urgh"

>> New interfaces are needed to allow users to utilize that all.
>
>That is a quite strong argument /against/ it all in my book. It means bloat
>in /every/ filesystem, and they have shown to be able to do without for
>some 30 years now. I'd need /very/ strong reasons for adding something.

But the competitor's filesystems are evolving past that 30-year-old
paradigm, why shouldn't Linux's?

And if Reiser4 is merged and its VFS-doubling things merged into the VFS,
how much will it bloat Ext2 if Ext2 doesn't go near files-as-dirs
or extended stat data?

>It has been argued over and over that that particular feature /can't/ be
>implemented sanely anyway, so it has to stay out. Besides not making any

Arguments left and right for and against this.

I'd rather see someone write the code that does this, or, even better,
you can write the code that proves this definitively wrong and someone
else will get it to work.

(Ok, I hate the "DIY" counter-argument as much as the next man, but
this seemed almost an appropriate place to use it)

>sense. "You've got files and directories" is a bit asymetrical and so not
>quite nice; "all you have is directories" is symmetrical, estetic, and
>completely useless; "some files are directories, some aren't; files in
>file-directories are different than regular files in directory-directories"
>is just a mindless jumble.

"You've got data objects"

All we need is the extra bit that's MAY_LOOKUP so we don't need +x
to access metadata. But apparently this is too much to ask for.

Why? /Probably/ standards compliance. 
So why be compliant, if it's backwards compatible? SILENCE!
No really, why not extend something without breaking? ...we don't wanna...

>> The other question that it may be difficult to foresee everything and
>> it may make sense to move some things upper in future.
>Another good reason to keep it out ;-)

Sure, extinguish innovation. Got it.

-- 
mjt


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Hans Reiser]

This is a very nice description, thank you Christophe.  My comments are
below.

Christophe Saout wrote:

>Am Dienstag, den 21.06.2005, 18:18 -0700 schrieb Andrew Morton:
>
>  
>
>>>What is wrong with having an encryption plugin implemented in this
>>> manner?  What is wrong with being able to have some files implemented
>>> using a compression plugin, and others in the same filesystem not.
>>>
>>> What is wrong with having one file in the FS use a write only plugin, in
>>> which the encrypion key is changed with every append in a forward but
>>> not backward computable manner, and in order to read a file you must
>>> either have a key that is stored on another computer or be reading what
>>> was written after the moment of cracking root?
>>>
>>> What is wrong with having a set of critical data files use a CRC
>>> checking file plugin?
>>>      
>>>
>>I think the concern here is that this is implemented at the wrong level.
>>
>>In Linux, a filesystem is some dumb thing which implements
>>address_space_operations, filesystem_operations, etc.
>>
>>Advanced features such as those which you describe are implemented on top
>>of the filesystem, not within it.  reiser4 turns it all upside down.
>>
>>Now, some of the features which you envision are not amenable to
>>above-the-fs implementations.  But some will be, and that's where we should
>>implement those.
>>    
>>
>
>Yes, but that would be difficult, probably worse. The name "plugins" is
>perhaps a bit misleading. These plugins are most of the time some sort
>client to the reiser4 on-disk database structure. The core code is does
>on-disk tree handling, journalling and these things. The plugins in turn
>glue this core database system to the rest of the system in order to
>make a filesystem of it. The "file plugin" tells the database how to
>store files.
>
>A compression plugins is more tricky. Files should be randomly
>accessible and if you write in the middle of the file the compressed
>block might change in size. For reiser4 this is not a problem since it
>just tells the underlying database "give me some room for 1234 bytes and
>insert it in the tree instead of the other block". The reiser4 core has
>totally different semantics than the VFS layer and I don't see how these
>things could be handled elsewhere in this simple way.
>
>The reiser4 core is more some sort of library that abstracts a block
>device and provides some sort of database API (which is highly optimized
>for filesystem purposes). The actual filesystem is then another layer on
>top of this. You could actually implement lots of different filesystems
>on top of that database core.
>
>The actual layer is a bit different though. The database core itself
>registers with the Linux VFS and then passes the calls down to one of
>the plugins which then calls back into the reiser4 core to do the actual
>database modification. I think this was the point that Christoph was
>criticizing the most.
>
>Currently it looks like this:
>
>             ,--------------.       ,--------------.
>VFS -------> |              | ----> |              |
>             | /fs/reiser4/ |       | .../plugins/ |
>blockdev <-- |              | <---> |              |
>             `--------------'       `--------------'
>
>So the reiser4 code is introducing another abstraction of the Linux VFS
>layer instead of letting the plugins define the Linux VFS ops directly.
>Which would look like this:
>
>                                    ,--------------.
>VFS ------------------------------> |              |
>             ,--------------,       | .../plugins/ |
>blockdev <-- | /fs/reiser4/ | <---> |              |
>             `--------------'       `--------------'
>  
>
Note that the proposed change (as I understand it) creates a need to
load plugin definitions (classes) into VFS vectors (instances), which
requires additional code and operations. 

Plugins need to be FS specific by their nature (unless someone wants the
nightmare of allocating pluginids to each of the different filesystems
and dealing with the inevitable collisions and violations).

Note that with the proposed change there are now two places the contents
of the plugin definition must reside:  once in each VFS instance of that
plugin, and once in the plugin definition.  As Codd taught us, putting
data in two places that must be kept synchronous is strongly
undesirable, and it always causes more bugs over time than people think
it will.

By loading the instance from the plugin layer into VFS, we are creating
a need to instantiate something that could instead be shared among all
the instances.  It seems an additional step that is unneeded.

The advantage though is that it avoids a function dereference.

Perhaps I miss something here?

>Which probably would be okay for most of the time except for some
>details (which could probably be solved otherwise).
>
>Actually the flow is not always this simple, usually the path goes back
>and forth multiple time between the core and the plugins.
>
>One of the features Hans is using is that there can be different kinds
>of files. The on-disk structure tells the core which of the plugins is
>responsible for the "database object" found on the disk. It could be a
>directory or a "stat data" (inode) or a file. The file itself could be
>handled by different plugins like one that stores the data directly or
>one that compresses it.
>
>reiser4 is different than other filesystems in that it uses a lot more
>abstraction levels. The database aspect and the semantic aspect of a
>traditional filesystems are strongly separated.
>
>To understand the code probably means a lot of work because it is a bit
>different. Some of the layering concerns may be right, other probably
>not.
>
>The plugins that add additional VFS semantics (that are currently
>disable) should most definitely not be implemented only inside the
>filesystem. I think Hans did this because it would have been a lot more
>work doing this at the proper layer (which means talking to people and a
>lot of politics...).
>
>The last time I looked at the code is a while ago, so if I'm wrong on
>something, please don't shoot me. The only thing I can say is that
>reiser4 has very stable for me (though I've gone back to reiser3 for
>most of my filesystems because I wanted acl/xattr).
>
>So here's my advice: Instead of insulting each other or doing pure
>marketing talk, please try to address each detail and explain why
>something has been done and if it's good or bad and if it should be
>changed and how.
>
>  
>

Re: reiser4 plugins

[Alexander Zarochentsev]

On Wednesday 22 June 2005 05:18, Andrew Morton wrote:
> Hans Reiser <reiser@namesys.com> wrote:
> > What is wrong with having an encryption plugin implemented in this
> >  manner?  What is wrong with being able to have some files implemented
> >  using a compression plugin, and others in the same filesystem not.
> >
> >  What is wrong with having one file in the FS use a write only plugin, in
> >  which the encrypion key is changed with every append in a forward but
> >  not backward computable manner, and in order to read a file you must
> >  either have a key that is stored on another computer or be reading what
> >  was written after the moment of cracking root?
> >
> >  What is wrong with having a set of critical data files use a CRC
> >  checking file plugin?
>
> I think the concern here is that this is implemented at the wrong level.

> In Linux, a filesystem is some dumb thing which implements
> address_space_operations, filesystem_operations, etc.
>

It is not so already.  XFS, FUSE, network fs are not of that type.

Thanks,
Alex.

Re: reiser4 plugins

[Christoph Hellwig]

On Tue, Jun 21, 2005 at 06:07:58PM -0700, Hans Reiser wrote:
> Christoph,
> 
> Reiser4 users love the plugin concept, and all audiences which have
> listened to a presentation on plugins have been quite positive about
> it.  Many users think it is the best thing about reiser4.  Can you
> articulate why you are opposed to plugins in more detail?  Perhaps you
> are simply not as familiar with it as the audiences I have presented
> to.  Perhaps persons on our mailing list can comment.....

You're duplicating the VFS infrastructure with your pluging system.

> In particular, what is wrong with having a plugin id associated with
> every file, storing the pluginid on disk in permanent storage in the
> stat data, and having that plugin id define the set of methods that
> implement the vfs operations associated with a particular file, rather
> than defining VFS methods only at filesystem granularity?

Hans, this only shows you didn't listen to be the last few times I
explained it to you.  There's nothing wrong with defining plug in IDs
associated with every file, and the linux file_operations, inode_operations,
etc.. are _not_ filesystem granularity but inode granularity (except in
reiser4 where you throw everything together just to add your own
de-multiplexer below).  So the only thing your plugin might need to is to
define it's own file or inode operations (in fact it might need a few
more things speaking from experience with kinda similar things, but it
certainly doesn't need to duplicate what's there)

> What is wrong with having an encryption plugin implemented in this
> manner?  What is wrong with being able to have some files implemented
> using a compression plugin, and others in the same filesystem not.

There's nothing wrong with that, although such things might be better
as a stackable filesystem.  Maybe they're not, and we'll find out once
people are prototyping these things and playing with them. But you don't
need your additional layer of abstraction for those anyway

> What is wrong with having one file in the FS use a write only plugin, in
> which the encrypion key is changed with every append in a forward but
> not backward computable manner, and in order to read a file you must
> either have a key that is stored on another computer or be reading what
> was written after the moment of cracking root?

Because root can read kernel memory this is completely useless :)
But if you want it as your private patch no one forbits you to do it,
just don't expect such security by obscurity to go into mainline.

> What we have hurts no one but us.  I have never seen an audience for one
> of my talks that thought it hurt us..... most audiences think it is
> great.  

most of your audience doesn't understand the fine points of filesystem
implementation.  they want your feature but don't care how it's implemented.
here it's the other way around - we don't care too much about what not so
important features we have but more about how sanely they're implemented.

> Let us tinker with our FS, and you tinker with yours, and so long as
> what we do does not affect your FS, let the users choose.

Now we're getting personal again, right?  My filesystems according to
MAINTAINERS are freevxfs and sysfs, but if you look at commit logs I've
done work on pretty much any filesystem in the tree, and often doing tree-wide
changes.  That's why I care about every new filesystem in the tree and not
a single one.

Re: reiser4 plugins

[Hans Reiser]

Christoph Hellwig wrote:

>
>
>  
>
>>What is wrong with having one file in the FS use a write only plugin, in
>>which the encrypion key is changed with every append in a forward but
>>not backward computable manner, and in order to read a file you must
>>either have a key that is stored on another computer or be reading what
>>was written after the moment of cracking root?
>>    
>>
>
>Because root can read kernel memory this is completely useless :)
>  
>
You missed the point of it rather nicely.  If root can read kernel
memory, that only gets it the appends made after the point in time of
cracking root.


It is not my idea, and it is not yet present in our code, let me not
seem to take credit for it though I think it a good idea.

Re: -mm -> 2.6.13 merge status

[Andrew Morton]

Jeff Garzik <jgarzik@pobox.com> wrote:
>
> > sparsemem
> > 
> >     OK by me for a merge.  Need to poke arch maintainers first, check that
> >     they've looked at it sufficiently closely.
> 
> seems sane, though there are some whitespace niggles that should be 
> cleaned up
> 

There are?  I thought I fixed most of them.

*general sigh*.  I wish people would absorb CodingStyle.  It's not hard,
and fixing the style post-facto creates a real mess.  I now have a great
string of kexec patches followed by a "kexec-code-cleanup.patch" which
totally buggers up the patch sequencing and really needs to be split into
18 parts and sprinkled back over the entire series.

> > rapidio-*
> > 
> >     Will merge.
> 
> send through netdev, as is proper
> 

OK.  But then the master version vanishes into the jgarzik git forest and I
won't know how to get it ;)

> > connector.patch
> > 
> >     Nice idea IMO, but there are still questions around the
> >     implementation.  More dialogue needed ;)
> > 
> > connector-add-a-fork-connector.patch
> > 
> >     OK, but needs connector.
> 
> I don't like connector
> 

How come?

> 
> > pcmcia-*.patch
> > 
> >     Makes the pcmcia layer generate hotplug events and deprecates cardmgr.
> >     Will merge.
> 
> Testing?  The goal behind the patch is certainly good, but I worry about 
> exposure.
> 

Yes, there will be a few problems I guess.  But people are testing it - we
know, because we've had lots of bug reports which were actually due to
greg-pci breakage...

> 
> > cachefs
> > 
> >     This is a ton of code which knows rather a lot about pagecache
> >     internals.  It allows the AFS client to cache file contents on a local
> >     blockdev.
> > 
> >     I don't think it's a justified addition for only AFS and I'd prefer to
> >     see it proven for NFS as well.
> > 
> >     Issues around add-page-becoming-writable-notification.patch need to
> >     be resolved.
> > 
> > cachefs-for-nfs
> > 
> >     A recent addition.  Needs review from NFS developers and considerably
> >     more testing.
> > 
> >     These things aren't looking likely for 2.6.13.
> 
> If I could vote more than once, I would!  I really like cachefs, and 
> have been pushing for its inclusion for a while.
> 

You've been using it?

> > kexec and kdump
> > 
> >     I guess we should merge these.
> > 
> >     I'm still concerned that the various device shutdown problems will
> >     mean that the success rate for crashing kernels is not high enough for
> >     kdump to be considered a success.  In which case in six months time we'll
> >     hear rumours about vendors shipping wholly different crashdump
> >     implementations, which would be quite bad.
> > 
> >     But I think this has gone as far as it can go in -mm, so it's a bit of
> >     a punt.
> 
> I'm not particularly pleased with these,

How come?

> and indeed vendors ARE shipping 
> other crashdump methods.

Which ones?

> 
> > reiser4
> > 
> >     Merge it, I guess.
> > 
> >     The patches still contain all the reiser4-specific namespace
> >     enhancements, only it is disabled, so it is effectively dead code.  Maybe
> >     we should ask that it actually be removed?
> 
> The plugin stuff is crap.  This is not a filesystem but a filesystem + 
> new layer.  IMO considered in that light, it duplicates functionality 
> elsewhere.
> 

hm.

Re: -mm -> 2.6.13 merge status

[Gerrit Huizenga]

On Tue, 21 Jun 2005 13:22:04 PDT, Andrew Morton wrote:
> Jeff Garzik <jgarzik@pobox.com> wrote:
> > > kexec and kdump
> > > 
> > >     I guess we should merge these.
> > > 
> > >     I'm still concerned that the various device shutdown problems will
> > >     mean that the success rate for crashing kernels is not high enough for
> > >     kdump to be considered a success.  In which case in six months time we'll
> > >     hear rumours about vendors shipping wholly different crashdump
> > >     implementations, which would be quite bad.
> > > 
> > >     But I think this has gone as far as it can go in -mm, so it's a bit of
> > >     a punt.
> > 
> > I'm not particularly pleased with these,
> 
> How come?
> 
> > and indeed vendors ARE shipping 
> > other crashdump methods.
> 
> Which ones?

And which ones that __WORK__?  We have a few customers out there from
both distros and all the crash dump methods that I've seen fail either
ALWAYS or ALMOST ALWAYS on customer sites.  And yes, we hear about them
and I believe that our partners understand the pain that this causes
us and our customers.

Kexec/kdump has a chance of working reliably.  The others are complete
crap.

gerrit

Re: -mm -> 2.6.13 merge status

[Andrew Morton]

Gerrit Huizenga <gh@us.ibm.com> wrote:
>
> Kexec/kdump has a chance of working reliably.

IOW: Kexec/kdump has a chance of not working reliably.

Worried.

Re: -mm -> 2.6.13 merge status

[Gerrit Huizenga]

On Tue, 21 Jun 2005 14:04:41 PDT, Andrew Morton wrote:
> Gerrit Huizenga <gh@us.ibm.com> wrote:
> >
> > Kexec/kdump has a chance of working reliably.
> 
> IOW: Kexec/kdump has a chance of not working reliably.
> 
> Worried.

No worries.  Machine locks up hard, hardware failures, etc., there
is a possibility that nothing but a hard reset can unlock a machine.
But that is rare and outside the scope of the simple locking problems
that today prevent crash dumps.  There are still some rough edges in
PCI shutdown code, reinitialization, etc. that will need to be shaken
out over time with more experience, but those at least can be addressed
in the fundamental architecture of kexec/kdump.

About the only possible solution that *might* be fail proof (and even
that case I doubt) would be an external dump program under control
of the firmware (assuming the firmware can still run) which does a
copy of memory off to some device without any assistance from the
kernel.

Kexec/kdump needs much wider exposure at this point and there will
a few bumps along the way.  They should be isolated to cases where
the machine is already crashing and the only thing that doesn't work
is the crash dump/restart.  And those we will fix like any other bugs.

gerrit

Re: -mm -> 2.6.13 merge status

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

On Tue, 2005-06-21 at 14:04 -0700, Andrew Morton wrote:
> Gerrit Huizenga <gh@us.ibm.com> wrote:
> >
> > Kexec/kdump has a chance of working reliably.
> 
> IOW: Kexec/kdump has a chance of not working reliably.
> 
> Worried.

it's about rates... you can hose your system so bad that nothing can fix
it.

the "distro" stuff probably succeeds 10% of the time in non-artificial
cases, the kexec one has the potential at least to go 90% (so yes I
admit that I pull these numbers out of my backside but I'm with Gerrit
on this one). The stuff the distros ship is also not so nice in that you
can't dump to LVM or SAN or ... etc; kexec gets all that right. It's
also far more an all-or-nothing thing; if you make the transition you
know you're going to get a good dump; most of the other dumps die
halfway in the process at some point.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: -mm -> 2.6.13 merge status

[Martin J. Bligh]

--Andrew Morton <akpm@osdl.org> wrote (on Tuesday, June 21, 2005 14:04:41 -0700):

> Gerrit Huizenga <gh@us.ibm.com> wrote:
>> 
>> Kexec/kdump has a chance of working reliably.
> 
> IOW: Kexec/kdump has a chance of not working reliably.
> 
> Worried.

Personally I'm more concerned about the design issues - I can't see how
any of the other options are sustainable / workable. Things that require
maintaining their own driver base are just insane. Things that dump from
the panicing kernel are just broken. People want to be able to dump to 
disk / network / flash-ram card / god-knows-what, so we need something
that's flexible.

I don't think kdump is perfect and bug-free yet, but at least it has a 
design that looks like it'll be workable and sustainable through the future. 
Plus it's a small patch on top of kexec, which is useful in it's own right
(for fast reboot, etc) so we get to reuse a lot of code.

We could go into how crashdump itself is important (eg. first time failure 
capture is critical for customers, less downtime, I can ship you better 
data on bugs I find in test, etc, etc) but I kind of assumed most people
were convinced of that by now. Even Linus seemed to think kdump was the
sensible way forward (at KS last year), and he seems to be one of the 
most ardent sceptics of crashdump I've ever met ;-)

M.

Re: -mm -> 2.6.13 merge status (kexec/kdump)

[Vara Prasad]

Andrew Morton wrote:

>Gerrit Huizenga <gh@us.ibm.com> wrote:
>  
>
>>Kexec/kdump has a chance of working reliably.
>>    
>>
>
>IOW: Kexec/kdump has a chance of not working reliably.
>
>Worried.
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>
I understand your concerns based on the device shutdown issues but these 
are not fundamental design problems that we can not solve. We should be 
able to fix them either through generic solutions to a class of devices 
or one of kind for special devices. As you know we are engaging in those 
discussions and providing fixes.

I think all the alternatives out there are less reliable than Kdump 
based on the design. Vendors are currently shipping other solutions 
since they didn't have any better alternatives until now. The existing 
solutions in the two major distro's doesn't work lot of times. I don't 
know what percentage of times they work as i only get involved when they 
don't work, but i can certainly tell you they don't work many a times. 
It is very embarrassing to tell the customer sorry we couldn't get dump 
can you try reproducing the problem again.  At least two major distros 
expressed interest in replacing their current solutions with kdump once 
it matures. As you are well aware we are doing testing with as many 
configurations as we can to iron out the bugs. Hope this addresses some 
of your concerns.

Re: -mm -> 2.6.13 merge status (kexec/kdump)

[Andrew Morton]

Vara Prasad <prasadav@us.ibm.com> wrote:
>
> I think all the alternatives out there are less reliable than Kdump 
>  based on the design. Vendors are currently shipping other solutions 
>  since they didn't have any better alternatives until now. The existing 
>  solutions in the two major distro's doesn't work lot of times. I don't 
>  know what percentage of times they work as i only get involved when they 
>  don't work, but i can certainly tell you they don't work many a times. 
>  It is very embarrassing to tell the customer sorry we couldn't get dump 
>  can you try reproducing the problem again.  At least two major distros 
>  expressed interest in replacing their current solutions with kdump once 
>  it matures. As you are well aware we are doing testing with as many 
>  configurations as we can to iron out the bugs. Hope this addresses some 
>  of your concerns.

Yes, thanks.

And the meta-goodness here is that at least we have a *design* which is
acceptable from this-is-sane standpoint.  So at least everyone will be
pulling in the same direction.

So as I said, it's a bit of a bet at this point in time, but we've gone as
far as we can get with it out-of-tree, so let's merge it and hope that it
matures into an acceptably useful dumper.

Re: -mm -> 2.6.13 merge status

[Carsten Otte]

On 6/21/05, Andrew Morton <akpm@osdl.org> wrote:
> > and indeed vendors ARE shipping
> > other crashdump methods.
> 
> Which ones?
For 390, we ship standalone bootable crashdump tools with both sles
and rhel. As for kexec, I'd like to see a kexec based 390 bootloader
in the future which would be more flexible then our current one. So
I'd like to vote for merging kexec/kdump.

Re: -mm -> 2.6.13 merge status

[Chris Wright]

* Carsten Otte (cotte.de@gmail.com) wrote:
> On 6/21/05, Andrew Morton <akpm@osdl.org> wrote:
> > > and indeed vendors ARE shipping
> > > other crashdump methods.
> > 
> > Which ones?
> For 390, we ship standalone bootable crashdump tools with both sles
> and rhel. As for kexec, I'd like to see a kexec based 390 bootloader
> in the future which would be more flexible then our current one. So
> I'd like to vote for merging kexec/kdump.

Xen is making similar noises w.r.t. using kexec for flexible bootloader.

thanks,
-chris

Re: -mm -> 2.6.13 merge status

[Carsten Otte]

On 6/23/05, Chris Wright <chrisw@osdl.org> wrote:
> * Carsten Otte (cotte.de@gmail.com) wrote:
> > For 390, we ship standalone bootable crashdump tools with both sles
> > and rhel. As for kexec, I'd like to see a kexec based 390 bootloader
> > in the future which would be more flexible then our current one. So
> > I'd like to vote for merging kexec/kdump.
> 
> Xen is making similar noises w.r.t. using kexec for flexible bootloader.

Oh cool, then we should look at what they're doing instead of reinventing
the wheel. Any pointer we can follow, or person we would contact?

Re: -mm -> 2.6.13 merge status

[Eric W. Biederman]

Andrew Morton <akpm@osdl.org> writes:

> Jeff Garzik <jgarzik@pobox.com> wrote:
> >
> > > sparsemem
> > > 
> > >     OK by me for a merge.  Need to poke arch maintainers first, check that
> > >     they've looked at it sufficiently closely.
> > 
> > seems sane, though there are some whitespace niggles that should be 
> > cleaned up
> > 
> 
> There are?  I thought I fixed most of them.
> 
> *general sigh*.  I wish people would absorb CodingStyle.  It's not hard,
> and fixing the style post-facto creates a real mess.  I now have a great
> string of kexec patches followed by a "kexec-code-cleanup.patch" which
> totally buggers up the patch sequencing and really needs to be split into
> 18 parts and sprinkled back over the entire series.

It looks like I have another hole in my schedule where I can put some
work into kexec so I will see what I can do.

If you want people to absorb CodingStyle it needs to be more explicit.
Of the things that patch fixes you almost have to read between
the lines of CodingStyle to see.  If there is anything backing
it up at all.  Until the problems were pointed out to me I simply
could not see them, and reading CodingStyle was not enlightening.
I point this out not to complain but more so people know which
part of the process needs fixing.

> > > kexec and kdump
> > > 
> > >     I guess we should merge these.
> > > 
> > >     I'm still concerned that the various device shutdown problems will
> > >     mean that the success rate for crashing kernels is not high enough for
> > > kdump to be considered a success.  In which case in six months time we'll
> 
> > >     hear rumours about vendors shipping wholly different crashdump
> > >     implementations, which would be quite bad.
> > > 
> > >     But I think this has gone as far as it can go in -mm, so it's a bit of
> > >     a punt.
> > 
> > I'm not particularly pleased with these,
> 
> How come?

Please tell.

With respect to users of crashdumps there are at least two groups
converging on kexec based crashdumps.  Is there active development
on any of the rest of the tools.

On to the practical response.  The kexec has effectively zero
impact on the kernel, except when it is invoked, and the
code is small.  Kexec is also useful for a lot more than 
just crash dumps.  It happens that crashdumps seem to be the only
case where the other alternatives are noticeably less sane.

There is also another important piece about kexec based crashdumps
that is not usually envisioned.  The kexec based solution is much more
flexible.  For example on a cluster the worst case scenario for
a network based crashdump is all 1000+ nodes will output a crashdump
simultaneously.  Poor crashdump server.  Where with the kexec based
approach it is possible to have the machines send an SNMP alert
and then you can come in and have the machine dump only when you are
ready.  Or you might even start a gdb-stubs server and interact
with a live core dump. :)  And all of this happens to fall out
naturally from using a kernel after the crash.

There are a few associated bug fixes that are problematic but I think
they are fixable.   For the crashdump case the work really is going
into getting hardening linux so it can run on imperfectly behaving hardware.
I.e.  things that are bugs in any event but that using the kernel to
take a crashdump exacerbates.

Andrew the good news is unless something comes up I will have time
starting about Monday to help with the 2.6.13 merge.  It looks like
the first thing I should do is split up the indent patch so it pairs
well with the rest.  And then I have a few pending patches for the user
space and I think I can fix a number of the device_shutdown problems,
for at least the normal kexec path.

Eric

Re: -mm -> 2.6.13 merge status

[Andrew Morton]

ebiederm@xmission.com (Eric W. Biederman) wrote:
>
> Andrew the good news is unless something comes up I will have time
> starting about Monday to help with the 2.6.13 merge.  It looks like
> the first thing I should do is split up the indent patch so it pairs
> well with the rest.  And then I have a few pending patches for the user
> space and I think I can fix a number of the device_shutdown problems,
> for at least the normal kexec path.

I'd be inclined to leave it as-is, really.  I'm not sure that it's worth
the effort+risk of significantly refactoring the patches.

I've cut it down from 58 patches to 45 and will take another pass at it in
the next day or two, but it's looking like 40-odd patches is the right
number.

Re: -mm -> 2.6.13 merge status

[Eric W. Biederman]

Andrew Morton <akpm@osdl.org> writes:

> ebiederm@xmission.com (Eric W. Biederman) wrote:
> >
> > Andrew the good news is unless something comes up I will have time
> > starting about Monday to help with the 2.6.13 merge.  It looks like
> > the first thing I should do is split up the indent patch so it pairs
> > well with the rest.  And then I have a few pending patches for the user
> > space and I think I can fix a number of the device_shutdown problems,
> > for at least the normal kexec path.
> 
> I'd be inclined to leave it as-is, really.  I'm not sure that it's worth
> the effort+risk of significantly refactoring the patches.
> 
> I've cut it down from 58 patches to 45 and will take another pass at it in
> the next day or two, but it's looking like 40-odd patches is the right
> number.

Ok.  Then I won't put a priority on that piece.

I will still look at cleaning up the sematics of the reboot
path so fewer things break.

Eric

Re: -mm -> 2.6.13 merge status

[Paul Jackson]

> I wish people would absorb CodingStyle.

Some people just can't see it, Andrew.  Just like some people
are tone deaf, some people don't notice minor variations in
code spacing and layout, unless pointed out in tedious detail.

Not that I disagree with you ... ;).

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: -mm -> 2.6.13 merge status

[randy_dunlap]

On Thu, 23 Jun 2005 21:06:38 -0700 Paul Jackson wrote:

| > I wish people would absorb CodingStyle.
| 
| Some people just can't see it, Andrew.  Just like some people
| are tone deaf, some people don't notice minor variations in
| code spacing and layout, unless pointed out in tedious detail.
| 
| Not that I disagree with you ... ;).

I also agree.

The problem (for me at least) is that bad coding style needs
to be fixed before I can do a functional code review, so it
slows down the review cycle quite a bit.  Further, it's mostly
well-known what the requirements are, so there aren't very
many good excuses not to follow CodingStyle...

---
~Randy

Re: -mm -> 2.6.13 merge status

[Lee Revell]

On Mon, 2005-06-20 at 23:54 -0700, Andrew Morton wrote:
> CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ
> Kconfigurable.
> 
>     Will merge (will switch default to 1000 Hz later if that seems
> necessary) 

Are you serious?  You're changing the *default* HZ in a stable kernel
series?!?

This is a big regression, it degrades the resolution of system calls.

Lee

Re: -mm -> 2.6.13 merge status

[Nish Aravamudan]

On 6/21/05, Lee Revell <rlrevell@joe-job.com> wrote:
> On Mon, 2005-06-20 at 23:54 -0700, Andrew Morton wrote:
> > CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ
> > Kconfigurable.
> >
> >     Will merge (will switch default to 1000 Hz later if that seems
> > necessary)
> 
> Are you serious?  You're changing the *default* HZ in a stable kernel
> series?!?
> 
> This is a big regression, it degrades the resolution of system calls.

Not that my opinion should sway anybody else, but I really would
prefer more of the in-kernel sleep callers were converted to use
human-time units (and thus were verified to be correct) so that
switching HZ will result in the *same* latencies. How much of moving
to lower HZ values is due to the fact that everything is request 10ms
for 1 jiffy of sleep instead of 1 ms? It's hard to tell if the gain is
there or from the lower frequency of interrupts.

I've sent out a lot of patches in this direction (using msleep() and
msleep_interruptible() and my soft-timer rework on top of John
Stultz's timeofday rework converts the entire soft-timer subsystem to
use human-time instead of jiffies as it's unit of expiration), but
there is still *a lot* of work left to do :( I will keep sending
patches, but am being pulled in other directions currently.

Just my $.02.

Thanks,
Nish

Re: -mm -> 2.6.13 merge status

[Nish Aravamudan]

On 6/21/05, Nish Aravamudan <nish.aravamudan@gmail.com> wrote:
> On 6/21/05, Lee Revell <rlrevell@joe-job.com> wrote:
> > On Mon, 2005-06-20 at 23:54 -0700, Andrew Morton wrote:
> > > CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ
> > > Kconfigurable.
> > >
> > >     Will merge (will switch default to 1000 Hz later if that seems
> > > necessary)
> >
> > Are you serious?  You're changing the *default* HZ in a stable kernel
> > series?!?
> >
> > This is a big regression, it degrades the resolution of system calls.
> 
> Not that my opinion should sway anybody else, but I really would
> prefer more of the in-kernel sleep callers were converted to use
> human-time units (and thus were verified to be correct) so that
> switching HZ will result in the *same* latencies. How much of moving
> to lower HZ values is due to the fact that everything is request 10ms
> for 1 jiffy of sleep instead of 1 ms? It's hard to tell if the gain is
> there or from the lower frequency of interrupts.

After some further consideration, I don't think that my patches would
be at all changed by adjusting HZ's default value. I just want to make
sure maintainers are still responsive to appropriate patches to split
time-based delays from tick-based delays. So, CONFIG_HZ is ok by me,
but I consider it a band-aid.

Thanks,
Nish

Re: -mm -> 2.6.13 merge status

[Andrew Morton]

Lee Revell <rlrevell@joe-job.com> wrote:
>
> On Mon, 2005-06-20 at 23:54 -0700, Andrew Morton wrote:
> > CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ
> > Kconfigurable.
> > 
> >     Will merge (will switch default to 1000 Hz later if that seems
> > necessary) 
> 
> Are you serious?  You're changing the *default* HZ in a stable kernel
> series?!?
> 
> This is a big regression, it degrades the resolution of system calls.
> 

Well we'll see what happens.  As I said, if it's determined to be a real
problem we'll put the default back to 1000Hz prior to 2.6.13 release.

Re: -mm -> 2.6.13 merge status

[Lee Revell]

On Tue, 2005-06-21 at 13:32 -0700, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > On Mon, 2005-06-20 at 23:54 -0700, Andrew Morton wrote:
> > > CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ
> > > Kconfigurable.
> > > 
> > >     Will merge (will switch default to 1000 Hz later if that seems
> > > necessary) 
> > 
> > Are you serious?  You're changing the *default* HZ in a stable kernel
> > series?!?
> > 
> > This is a big regression, it degrades the resolution of system calls.
> > 
> 
> Well we'll see what happens.  As I said, if it's determined to be a real
> problem we'll put the default back to 1000Hz prior to 2.6.13 release.
> 

I just think it's silly to merge CONFIG_HZ this late in the game, when
dynamic tick and high res timers are right around the corner.  Seems
like more trouble than it's worth.

Lee

Re: -mm -> 2.6.13 merge status (HZ -> 250?)

[Lee Revell]

On Mon, 2005-06-20 at 23:54 -0700, Andrew Morton wrote:
> CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ
> Kconfigurable.
> 
>     Will merge (will switch default to 1000 Hz later if that seems
> necessary) 

How about delaying this until the high res timers patches are ready?
That way you can save power and avoid the latency regression, in fact it
would be a huge improvement from user POV.

Consider a program with a 5ms RT constraint, like a game or mplayer.
Currently it uses the RTC on 2.4/HZ=100 systems and usleep() on
2.6/HZ=1000.  Allowing HZ to regress to 250 would force us to handle
2.4, 2.6.1 - 2.6.12, and 2.6.13+ separately.  It would be a huge mess.

Lee

Re: -mm -> 2.6.13 merge status (HZ -> 250?)

[Alan Cox]

On Maw, 2005-06-21 at 17:26, Lee Revell wrote:
> Consider a program with a 5ms RT constraint, like a game or mplayer.
> Currently it uses the RTC on 2.4/HZ=100 systems and usleep() on
> 2.6/HZ=1000.  Allowing HZ to regress to 250 would force us to handle
> 2.4, 2.6.1 - 2.6.12, and 2.6.13+ separately.  It would be a huge mess.

Vendors already ship 100Hz and 1KHz kernels. 2.4 and 2.6 are different
already. I can see the argument for not picking another new value
though.

Re: xtensa architecture (-mm -> 2.6.13 merge status)

[Chris Zankel]

Andrew Morton wrote:
> xtensa architecture
> 
>     Is xtensa now, or will it be in the future a sufficiently popular
>     architecture to justify the cost of having this code in the tree?
> 
>     Heaven knows.  Will merge.

Andrew,

I understand your concern and am glad that you give Xtensa and the other 
smaller non-mainstream architectures a chance.

The Xtensa architecture is highly configurable and usually buried inside 
an SOC device. So, if you buy a new printer, digital camera, or cell 
phone, there is a chance that there is an Xtensa inside even though you 
don't know it (sometimes as a small audio-engine or as a control CPU). 
Linux hasn't been adopted widely with Xtensa yet, but with Linux growing 
in the embedded space, I am sure it will become much more important -- 
at least this is where I bet my time (and spare time) on.
 
          To minimize the impact on other developers, I do understand 
that changes that affect all architectures will only be applied to the 
mainstream architectures and that the maintainers of the non-mainstream 
architectures then have to pick it up. Luckily, the architecture 
dependent files have their own confined space in the arch and asm 
directories.

In my opinion, as long as an architecture, driver, etc. is maintained 
and not obviously obsolete, it should be allowed to remain in the 
kernel.

I do have a few small patches in the queue but am struggling with some 
changes I want to make to the syscalls that might break some older code. 

 
                     Thanks, 
                              ~Chris 
 
 
 
 
 
 
 

 

OCFS (was Re: -mm -> 2.6.13 merge status)

[Rik Van Riel]

On Mon, 20 Jun 2005, Andrew Morton wrote:

> git-ocfs
> 
>     The OCFS2 filesystem.  OK by me, although I'm not sure it's had enough
>     review.

The only problem I can see with this is that people will want
to use OCFS together with CLVM, and both use a different cluster
infrastructure.

IMHO it would be good if they both used the same underlying
cluster infrastructure...

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics

Re: OCFS (was Re: -mm -> 2.6.13 merge status)

[Wim Coekaerts]

On Tue, Jun 21, 2005 at 11:35:10PM -0400, Rik Van Riel wrote:
> On Mon, 20 Jun 2005, Andrew Morton wrote:
> 
> > git-ocfs
> The only problem I can see with this is that people will want
> to use OCFS together with CLVM, and both use a different cluster
> infrastructure.
> 
> IMHO it would be good if they both used the same underlying
> cluster infrastructure...

as clvm stuff and infra get submitted and come around we will definitely
be working with it. but it's a feature more than a requirement. altho we
have every intend to make our stuff work with clvm and I think some
pieces of ocfs2 are already being used or looked at to be used for this
infrastructure so it's looking good.

Wim

Re: -mm -> 2.6.13 merge status (configfs)

[David Teigland]

On 6/21/05, Andrew Morton <akpm@osdl.org> wrote:
> git-ocfs
> 
>     The OCFS2 filesystem.  OK by me, although I'm not sure it's had enough
>     review.

Does this include configfs?  I'd especially like to see that sooner
rather than later.

Dave

Re: -mm -> 2.6.13 merge status (configfs)

[Andrew Morton]

David Teigland <dteigland@gmail.com> wrote:
>
> On 6/21/05, Andrew Morton <akpm@osdl.org> wrote:
> > git-ocfs
> > 
> >     The OCFS2 filesystem.  OK by me, although I'm not sure it's had enough
> >     review.
> 
> Does this include configfs?  I'd especially like to see that sooner
> rather than later.

There's not a lot of point in adding a fs which has no in-kernel users.

Re: -mm -> 2.6.13 merge status

[Christoph Hellwig]

> git-ocfs
> 
>     The OCFS2 filesystem.  OK by me, although I'm not sure it's had enough
>     review.

I'll try to take a look next week.  A major blocker is that it's not
endian-clean yet.  Even if other review items where perfect that's something
preventing it from going to mainline completely.

Re: -mm -> 2.6.13 merge status

[Christoph Hellwig]

On Mon, Jun 20, 2005 at 11:54:58PM -0700, Andrew Morton wrote:
> reiser4

So looking over the code a little for the plugin debate I found that a
reason patch introduces a ->put_inode method in reiser4.  We plan to
kill ->put_inode because it's the wrong abstraction and almost impossible
to use non-racy (reiser4 usage is racy).

I had a discussion with someone from namesys how to solve this correctly,
but I don't remember the details of either the solution or problem anymore.
Unless someone else does let's describe the problem again so we can find
a proper fix.

Re: -mm -> 2.6.13 merge status

[Vladimir Saveliev]

Hello

On Mon, 2005-06-27 at 13:06, Christoph Hellwig wrote:
> On Mon, Jun 20, 2005 at 11:54:58PM -0700, Andrew Morton wrote:
> > reiser4
> 
> So looking over the code a little for the plugin debate I found that a
> reason patch introduces a ->put_inode method in reiser4.  We plan to
> kill ->put_inode because it's the wrong abstraction and almost impossible
> to use non-racy (reiser4 usage is racy).

Sorry, would you please explain what is wrong in having the below

if (inode->i_nlink != 0 || atomic_read(&inode->i_count) > 1)

in reiser4_put_inode.


> I had a discussion with someone from namesys how to solve this correctly,

We were trying to avoid need to have reiser4_drop_inode because you said
drop_inode is a hack for hugetlbfs.

The problem is that on file removal reiser4 wants to do
truncate_inode_pages in reiser4_delete_inode. We used reiser4_drop_inode
for that. As long as drop_inode was about to die, we decided to do file
predeletion in reiser4_put_inode when inode is about to get into
iput_final with inode->i_nlink == 0.

It is a pity that put_inode is also wrong abstraction.

> but I don't remember the details of either the solution or problem anymore.

You said:
--------
So what you want is actually to move the  truncate_inode_pages out of
generic_delete_inode and into ->delete_inode?


Looking at the code another strategt makes more sense:

 - move the truncate_inode_pages at the beginning of clear_inode.
   All callers but one already do it just before that call, but
   the one that doesn't will require a full audit of all ->delete_inode
   instances
 - make the first half of clear_inode into a helper (__clear_inode or
   whatever), and make ->clear_inode responsible for calling it as first
   thing for a normal fs or call it in clear_inode if ->clear_inode
doesn't
   exist.  that way we can also move the invalidate_inode_buffers out
there
   easily later for filesystems that don't use buffer_heads at all.

p.s. please try to keep -fsdevel Cc'ed on the mail related to core
changes
-------

I hoped that we can solve the problem internally in reiser4. If
put_inode is about to be removed we will have to do ssomething like
that.


> Unless someone else does let's describe the problem again so we can find
> a proper fix.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: -mm -> 2.6.13 merge status

[Christoph Hellwig]

On Mon, Jun 27, 2005 at 06:25:43PM +0400, Vladimir Saveliev wrote:
> Sorry, would you please explain what is wrong in having the below
> 
> if (inode->i_nlink != 0 || atomic_read(&inode->i_count) > 1)
> 
> in reiser4_put_inode.

Between that atomic_read(&inode->i_count) and the
atomic_dec_and_lock(&inode->i_count, &inode_lock)) in iput someone could
have grabbed a reference.

> The problem is that on file removal reiser4 wants to do
> truncate_inode_pages in reiser4_delete_inode. We used reiser4_drop_inode
> for that. As long as drop_inode was about to die, we decided to do file

drop_inode is not going to die, we need it to support filesystems that
want to call generic_delete_inode even for a non-null i_nlink.  What's
hopefully going to die is the last instance of it that isn't either
generic_drop_inode or generic_delete_inode.

> You said:
> --------
> So what you want is actually to move the  truncate_inode_pages out of
> generic_delete_inode and into ->delete_inode?
> 
> 
> Looking at the code another strategt makes more sense:
> 
>  - move the truncate_inode_pages at the beginning of clear_inode.
>    All callers but one already do it just before that call, but
>    the one that doesn't will require a full audit of all ->delete_inode
>    instances
>  - make the first half of clear_inode into a helper (__clear_inode or
>    whatever), and make ->clear_inode responsible for calling it as first
>    thing for a normal fs or call it in clear_inode if ->clear_inode
> doesn't
>    exist.  that way we can also move the invalidate_inode_buffers out
> there
>    easily later for filesystems that don't use buffer_heads at all.
> 
> p.s. please try to keep -fsdevel Cc'ed on the mail related to core
> changes
> -------
> 
> I hoped that we can solve the problem internally in reiser4. If
> put_inode is about to be removed we will have to do ssomething like
> that.

In fact I know from some cluster filesystem folks that have a similar
problems as yours.  So getting the truncate_inode_pages under control
of the filesystems sounds like a very good choice.

Re: -mm -> 2.6.13 merge status

[Joel Becker]

On Mon, Jun 27, 2005 at 08:26:51PM +0100, Christoph Hellwig wrote:
> drop_inode is not going to die, we need it to support filesystems that
> want to call generic_delete_inode even for a non-null i_nlink.  What's
> hopefully going to die is the last instance of it that isn't either
> generic_drop_inode or generic_delete_inode.

	OCFS2 uses drop_inode as well, as it must handle last-close when
another node did the unlink.  It fixes up i_nlink in that case, then
calls generic_drop_inode().
	If there's a more elegant solution, we're all ears.

Joel

-- 

"When choosing between two evils, I always like to try the one
 I've never tried before."
        - Mae West

Joel Becker
Senior Member of Technical Staff
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: -mm -> 2.6.13 merge status

[Christoph Hellwig]

On Mon, Jun 27, 2005 at 12:44:02PM -0700, Joel Becker wrote:
> On Mon, Jun 27, 2005 at 08:26:51PM +0100, Christoph Hellwig wrote:
> > drop_inode is not going to die, we need it to support filesystems that
> > want to call generic_delete_inode even for a non-null i_nlink.  What's
> > hopefully going to die is the last instance of it that isn't either
> > generic_drop_inode or generic_delete_inode.
> 
> 	OCFS2 uses drop_inode as well, as it must handle last-close when
> another node did the unlink.  It fixes up i_nlink in that case, then
> calls generic_drop_inode().
> 	If there's a more elegant solution, we're all ears.

I think this still qualifies as calling generic_delete_inode because it's
a trivial wrapper.  Manipulating i_nlink seems rather odd to me, I'd
say you should rather call into generic_delete_inode directly if
OCFS2_INODE_MAYBE_ORPHANED is set (that's what generic_drop_inode will
do for i_nlink == 0 anyway).

In fact given every cluster and possibly many network filesystems will
need this it might make sense to take the OCFS2_INODE_MAYBE_ORPHANED into
the VFS, i.e. make it an i_state flag (after fixing can_unuse to not do
something totally stupid with i_state)

generic_drop_inode

[Mark Fasheh]

On Mon, Jun 27, 2005 at 09:26:48PM +0100, Christoph Hellwig wrote:
> I think this still qualifies as calling generic_delete_inode because it's
> a trivial wrapper.  Manipulating i_nlink seems rather odd to me, I'd
> say you should rather call into generic_delete_inode directly if
> OCFS2_INODE_MAYBE_ORPHANED is set (that's what generic_drop_inode will
> do for i_nlink == 0 anyway).
Allowing the file system to make the full decision and call the proper
function would be fine, but then we'd need generic_forget_inode exported
too :)

> In fact given every cluster and possibly many network filesystems will
> need this it might make sense to take the OCFS2_INODE_MAYBE_ORPHANED into
> the VFS, i.e. make it an i_state flag (after fixing can_unuse to not do
> something totally stupid with i_state)
Yes, the problem certainly isn't necessarily specific to OCFS2 so I'd be
more than happy to see that as a generic VFS feature. As is, the export and
callback get us past quite a few problems with tracking orphaned inodes in
the cluster.

This however, brings me to a related issue for which I'd appreciate some
advice.

If a node crashes or otherwise fails to complete the unlink(2) then the
inode in qusetion will *not* have actually been orphaned. Of course, the
local node doesn't know this and eventually gets to ocfs2_delete_inode() (via
the MAYBE_ORPHANED flag) which will do the work of determining whether the
inode is actually orphaned. The problem is that by the time we've gotten
there, generic_delete_inode() has done a truncate_inode_pages() which might
throw out data for a file which otherwise wants it on disk.

In fact, because OCFS2 supports POSIX style unlink-while-open across the
entire cluster, we might get to ocfs2_delete_inode() and during the voting
process, discover that the file is still open on another node in which case
we don't want to wipe it but would have lost file data due to the
truncate_inode_pages() anyway.

Essentially, we get to ocfs2_delete_inode() before we have a chance to
take the cluster lock and determine the true state of the inode.  By
then it is too late.  We really need to be able to make the
determination before calling generic_drop_inode(), but drop_inode() is
under the inode lock, and we can't call out to the cluster.  The
question becomes where and how to do this work?
	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
mark.fasheh@oracle.com

Re: generic_drop_inode

[Christoph Hellwig]

On Mon, Jun 27, 2005 at 03:06:12PM -0700, Mark Fasheh wrote:
> Allowing the file system to make the full decision and call the proper
> function would be fine, but then we'd need generic_forget_inode exported
> too :)

shouldn't be much of a problem if it's actually an _GPL export.

> If a node crashes or otherwise fails to complete the unlink(2) then the
> inode in qusetion will *not* have actually been orphaned. Of course, the
> local node doesn't know this and eventually gets to ocfs2_delete_inode() (via
> the MAYBE_ORPHANED flag) which will do the work of determining whether the
> inode is actually orphaned. The problem is that by the time we've gotten
> there, generic_delete_inode() has done a truncate_inode_pages() which might
> throw out data for a file which otherwise wants it on disk.

As discussed in a different subthread of the 2.6.13 merge plan we'd like to
move towards not having truncate_inode_pages in that place, because reiser4,
hugetlbfs and some other cluster filesystems don't like it either.
But that alone isn't going to help you a lot..

> In fact, because OCFS2 supports POSIX style unlink-while-open across the
> entire cluster, we might get to ocfs2_delete_inode() and during the voting
> process, discover that the file is still open on another node in which case
> we don't want to wipe it but would have lost file data due to the
> truncate_inode_pages() anyway.
> 
> Essentially, we get to ocfs2_delete_inode() before we have a chance to
> take the cluster lock and determine the true state of the inode.  By
> then it is too late.  We really need to be able to make the
> determination before calling generic_drop_inode(), but drop_inode() is
> under the inode lock, and we can't call out to the cluster.  The
> question becomes where and how to do this work?

I don't think that's doable without a major rework of that area of code.

Re: -mm -> 2.6.13 merge status

[Christoph Hellwig]

> reiser4

sparse isn't to happy about this:

hch@macfly:/work/linux-2.6.12$ make C=1 SUBDIRS=fs/reiser4/ >/dev/null 2>err.log && wc -l err.log
2286 err.log

The log is at http://verein.lst.de/~hch/linux-2.6.12-mm2-fs-reiser4-errors.log

Re: -mm -> 2.6.13 merge status

[Hans Reiser]

Christoph Hellwig wrote:

>>reiser4
>>    
>>
>
>sparse isn't to happy about this:
>
>hch@macfly:/work/linux-2.6.12$ make C=1 SUBDIRS=fs/reiser4/ >/dev/null 2>err.log && wc -l err.log
>2286 err.log
>
>The log is at http://verein.lst.de/~hch/linux-2.6.12-mm2-fs-reiser4-errors.log
>
>
>  
>
Thanks for telling us about sparse, we will work on fixing these. 
Vitaly, can you do this?

Hans

Re: -mm -> 2.6.13 merge status

[Vitaly Fertman]

On Tuesday 28 June 2005 00:37, Hans Reiser wrote:
> Christoph Hellwig wrote:
> 
> >>reiser4
> >>    
> >>
> >
> >sparse isn't to happy about this:
> >
> >hch@macfly:/work/linux-2.6.12$ make C=1 SUBDIRS=fs/reiser4/ >/dev/null 2>err.log && wc -l err.log
> >2286 err.log
> >
> >The log is at http://verein.lst.de/~hch/linux-2.6.12-mm2-fs-reiser4-errors.log
> >
> >
> >  
> >
> Thanks for telling us about sparse, we will work on fixing these. 
> Vitaly, can you do this?

fixed, will be included into the next mm kernel.

-- 
Thanks,
Vitaly Fertman

Re: -mm -> 2.6.13 merge status

[Christoph Hellwig]

> v9fs
> 
>     I'm not sure that this has a sufficiently high
>     usefulness-to-maintenance-cost ratio.

Personally I think this is very useful to have.  It provides a portable
way to have simple userland or remote filesystems, and it's been around
for a long time in others OSes.

That beeing said there's a few issues with the code still I'd like to
see fixed:

  - there's three sparse warnings still.  Two of them are easily fixed
    by moving externs to headers, one doesn't look fixable until we get
    a sane in-kernel api for socket operations
  - some dentry handling looks rather odd.  Why are you for example
    calling d_drop in v9fs_vfs_symlink, v9fs_vfs_mknod and v9fs_vfs_link?
    Shouldn't all these call d_instantatiate to actually reuse the
    dentry as in v9fs_vfs_create?  Also what's the issue with
    v9fs_fid_insert?  It would seem better and more logical to me to
    always set d_fsdata in create/mknod/symlink/open before hashing it
    and then beeing able to rely on it beeing non-NULL.
  - buf_check_sizep, buf_check_size and buf_check_sizev should be made
    inlines, and lose the implict return.  Please don't hide such
    things in macros
  - please avoid using hlist_for_each, usually hlist_for_each_entry is
    a much better choice
  - dito for list_for_each_safe vs list_for_each_entry_safe
  - can you please check whether lib/idr.c fullfills your needs so we
    can get rid of idpool.c?
  - v9fs_inode2v9ses has lots of useless checks, inode->i_sb can never
    be NULL, and inode->i_sb->s_fs_info can't be either once set in
    fill_inode, which is before the first inode on the filesystem is
    created.  Also the argument is never NULL.  Because of that you
    can also kill all the return value checks in the callers.
  - do you really need to keep v9fs_dentry_delete just for the dprintk?
  - no need to check for a NULL file in v9fs_dir_readdir, the VFS gurantees
    it's not.  And if it was you'd better be off panic because something
    is enormously fscked.
  - Dito for v9fs_file_open
  - And the inode in v9fs_file_lock
  - And dir, file, file->d_inode, sb, v9ses in v9fs_remove.
  - And dir, sb and v9ses in v9fs_vfs_lookup
  - And dir, sb and v9ses in v9fs_vfs_symlink
  - And dir, sb and v9ses in v9fs_vfs_link
  - And dir, sb and v9ses in v9fs_vfs_mknod
  - copy_from_user returns the bytes actually copied in the failure case,
    but you should return -EFAULT instead of that number in v9fs_file_write
  - No need to implement v9fs_file_mmap, do_mmap_pgoff makes sure to error
    out if it's not present (and actually returns the correct errno)
  - I think it's pretty similar for all these checks for fid (=private_data)
    checks.  You always set them in open, so they can't be NULL
  - kfree can be called with a NULL argument just fine, you can remove
    lots of ifs for that. You also often set pointers to NULL just before
    freeing a structure - that's pretty useless as slab debugging will
    catch bugs with stary references very well, and overwrites these NULLs
    ASAP.
  - The call to ->put_inode in the error case of v9fs_get_inode is very
    wrong.  You'd actually panic if you ever hit this as v9fs doesn't
    implement a ->put_inode :-)
  - All the ISDIR checks in v9fs_remove can go, VFS makes sure to only
    call ->remove and ->rmdir on directories, and only the right one
    for each kind of child.
  - Please try to use generic_readlink instead of your own
    v9fs_vfs_readlink, as you're implemting ->follow_link and ->put_link
    that should just work
  - the last error case in v9fs_get_sb needs a dput on ->s_root

Also did you look into the VFS/NFS lookup intent bits to solve your
atomic create and open issue?

Re: (v9fs) -mm -> 2.6.13 merge status

[Eric Van Hensbergen]

On 6/27/05, Christoph Hellwig <hch@infradead.org> wrote:
> 
> That beeing said there's a few issues with the code still I'd like to
> see fixed:
>

Sorry I didn't get to these quicker - was on vacation and basically
off-line for the past week and a half.  I've made 90% of the changes
suggested and committed them to my git tree, I'll combine the changes
into a single patch and then split them by file-group before sending
them to akpm to more closely match the existing patches.  The 10% I
didn't address I'll comment on below, most of them represent harder
problems that I'd like to think about a bit more.

>
>   - there's three sparse warnings still.  Two of them are easily fixed
>     by moving externs to headers, one doesn't look fixable until we get
>     a sane in-kernel api for socket operations

done

>   - some dentry handling looks rather odd.  Why are you for example
>     calling d_drop in v9fs_vfs_symlink, v9fs_vfs_mknod and v9fs_vfs_link?
>     Shouldn't all these call d_instantatiate to actually reuse the
>     dentry as in v9fs_vfs_create?  Also what's the issue with
>     v9fs_fid_insert?  It would seem better and more logical to me to
>     always set d_fsdata in create/mknod/symlink/open before hashing it
>     and then beeing able to rely on it beeing non-NULL.

All of this is kind of tricky due to the association of fids with
dentry elements and the special way we handle certain features (such
as special files and symlinks).  The current code aggressively
invalidates fids to prevent the dcache from masking operations that
may be semantically important to synthetic file systems.  If you look
in v9fs_create we actually d_drop the dentry for created directories
as well.  The only reason we don't d_drop normal files is because we
are trying to preserve the atomic create/open semantics.

I'm not 100% confident this is the right solution, but its the closest
I've been able to come so far -- there's actually been a fair amount
of discussion on this in the v9fs-developer's list.  If you want more
details, it's probably worth a separate thread to discuss the reasons
behind why we want to aggressively invalidate the dcache and how we
have tried to accomplish this -- or we could just catch up at OLS.

>   - buf_check_sizep, buf_check_size and buf_check_sizev should be made
>     inlines, and lose the implict return.  Please don't hide such
>     things in macros

done

>   - please avoid using hlist_for_each, usually hlist_for_each_entry is
>     a much better choice
>   - dito for list_for_each_safe vs list_for_each_entry_safe

done

>   - can you please check whether lib/idr.c fullfills your needs so we
>     can get rid of idpool.c?

Last time I looked idr didn't do exactly what I wanted, but looking
over it again I realize its just doing more than I want -- so I've
eliminated idpool.*, but still have wrapper functions to encapsulate
locking and retry -- it strikes me that there may be a case for
generalizing these wrapper functions and putting them in lib/idr.c,
but figured that could wait.

>   - v9fs_inode2v9ses has lots of useless checks, inode->i_sb can never
>     be NULL, and inode->i_sb->s_fs_info can't be either once set in
>     fill_inode, which is before the first inode on the filesystem is
>     created.  Also the argument is never NULL.  Because of that you
>     can also kill all the return value checks in the callers.
>   - do you really need to keep v9fs_dentry_delete just for the dprintk?
>   - no need to check for a NULL file in v9fs_dir_readdir, the VFS gurantees
>     it's not.  And if it was you'd better be off panic because something
>     is enormously fscked.
>   - Dito for v9fs_file_open
>   - And the inode in v9fs_file_lock
>   - And dir, file, file->d_inode, sb, v9ses in v9fs_remove.
>   - And dir, sb and v9ses in v9fs_vfs_lookup
>   - And dir, sb and v9ses in v9fs_vfs_symlink
>   - And dir, sb and v9ses in v9fs_vfs_link
>   - And dir, sb and v9ses in v9fs_vfs_mknod

Yeah, all of these were sanity checks during initial development while
I was still understanding the VFS API.  I think I got most of them
this time.

>   - copy_from_user returns the bytes actually copied in the failure case,
>     but you should return -EFAULT instead of that number in v9fs_file_write

fixed

>   - No need to implement v9fs_file_mmap, do_mmap_pgoff makes sure to error
>     out if it's not present (and actually returns the correct errno)
>   - I think it's pretty similar for all these checks for fid (=private_data)
>     checks.  You always set them in open, so they can't be NULL
>   - kfree can be called with a NULL argument just fine, you can remove
>     lots of ifs for that. You also often set pointers to NULL just before
>     freeing a structure - that's pretty useless as slab debugging will
>     catch bugs with stary references very well, and overwrites these NULLs
>     ASAP.
>   - The call to ->put_inode in the error case of v9fs_get_inode is very
>     wrong.  You'd actually panic if you ever hit this as v9fs doesn't
>     implement a ->put_inode :-)
>   - All the ISDIR checks in v9fs_remove can go, VFS makes sure to only
>     call ->remove and ->rmdir on directories, and only the right one
>     for each kind of child.

done

>   - Please try to use generic_readlink instead of your own
>     v9fs_vfs_readlink, as you're implemting ->follow_link and ->put_link
>     that should just work

I tried this and the kernel crashed - which may mean I've done
something wrong in follow_link or put_link.  I'll revisit this and try
to figure out what happened, but for now I've left my readlink in.

>   - the last error case in v9fs_get_sb needs a dput on ->s_root

done

> 
> Also did you look into the VFS/NFS lookup intent bits to solve your
> atomic create and open issue?
> 

I had considered the lookup intent bits to solve several different
problems associated with our fid handling - but never really felt
confident in my understanding of how the intent stuff was supposed to
work.  It is certainly worth revisiting now that things have calmed
down a bit.

           -eric

Re: (v9fs) -mm -> 2.6.13 merge status

[Christoph Hellwig]

On Wed, Jul 13, 2005 at 01:23:24PM -0500, Eric Van Hensbergen wrote:
> Sorry I didn't get to these quicker - was on vacation and basically
> off-line for the past week and a half.  I've made 90% of the changes
> suggested and committed them to my git tree, I'll combine the changes
> into a single patch and then split them by file-group before sending

normally we prefer a patch per actual change, not per file so the
description fits.  Given that all these are pretty trivial fixes one
patch would have done it aswell, though.

With these changes the code is fine for mainline in my opinion.

Re: (v9fs) -mm -> 2.6.13 merge status

[Alexey Dobriyan]

On Friday 15 July 2005 00:04, Christoph Hellwig wrote:
> normally we prefer a patch per actual change, not per file so the
> description fits.  Given that all these are pretty trivial fixes one
> patch would have done it aswell, though.
> 
> With these changes the code is fine for mainline in my opinion.

Can I make one more nitpicking comment?

All these functions can use cpu_to_le*() and le*_to_cpu().

> --- /dev/null
> +++ 25-akpm/fs/9p/conv.c

> +static inline void buf_put_int16(struct cbuf *buf, u16 val)
> +{
> +	buf_check_sizev(buf, 2);
> +
> +	buf->p[0] = val;
> +	buf->p[1] = val >> 8;
> +	buf->p += 2;
> +}
> +
> +static inline void buf_put_int32(struct cbuf *buf, u32 val)
> +{
> +	buf_check_sizev(buf, 4);
> +
> +	buf->p[0] = val;
> +	buf->p[1] = val >> 8;
> +	buf->p[2] = val >> 16;
> +	buf->p[3] = val >> 24;
> +	buf->p += 4;
> +}
> +
> +static inline void buf_put_int64(struct cbuf *buf, u64 val)
> +{
> +	buf_check_sizev(buf, 8);
> +
> +	buf->p[0] = val;
> +	buf->p[1] = val >> 8;
> +	buf->p[2] = val >> 16;
> +	buf->p[3] = val >> 24;
> +	buf->p[4] = val >> 32;
> +	buf->p[5] = val >> 40;
> +	buf->p[6] = val >> 48;
> +	buf->p[7] = val >> 56;
> +	buf->p += 8;
> +}

> +static inline u16 buf_get_int16(struct cbuf *buf)
> +{
> +	u16 ret = 0;
> +
> +	buf_check_size(buf, 2);
> +	ret = buf->p[0] | (buf->p[1] << 8);
> +
> +	buf->p += 2;
> +
> +	return ret;
> +}
> +
> +static inline u32 buf_get_int32(struct cbuf *buf)
> +{
> +	u32 ret = 0;
> +
> +	buf_check_size(buf, 4);
> +	ret =
> +	    buf->p[0] | (buf->p[1] << 8) | (buf->p[2] << 16) | (buf->
> +								p[3] << 24);
> +
> +	buf->p += 4;
> +
> +	return ret;
> +}
> +
> +static inline u64 buf_get_int64(struct cbuf *buf)
> +{
> +	u64 ret = 0;
> +
> +	buf_check_size(buf, 8);
> +	ret = (u64) buf->p[0] | ((u64) buf->p[1] << 8) |
> +	    ((u64) buf->p[2] << 16) | ((u64) buf->p[3] << 24) |
> +	    ((u64) buf->p[4] << 32) | ((u64) buf->p[5] << 40) |
> +	    ((u64) buf->p[6] << 48) | ((u64) buf->p[7] << 56);
> +
> +	buf->p += 8;
> +
> +	return ret;
> +}

Re: (v9fs) -mm -> 2.6.13 merge status

[Eric Van Hensbergen]

On 7/14/05, Alexey Dobriyan <adobriyan@gmail.com> wrote:
> On Friday 15 July 2005 00:04, Christoph Hellwig wrote:
> > normally we prefer a patch per actual change, not per file so the
> > description fits.  Given that all these are pretty trivial fixes one
> > patch would have done it aswell, though.
> >
> > With these changes the code is fine for mainline in my opinion.
> 
> Can I make one more nitpicking comment?
> 
> All these functions can use cpu_to_le*() and le*_to_cpu().
> 

I need to rethink some parts of conv.c, I'll incorporate your
suggestion during the rework.  Thanks Alexey.

         -eric