Using fuse for AFS/DFS (was Re: [OpenAFS-devel] openafs / opendfs collaboration)

Using fuse for AFS/DFS (was Re: [OpenAFS-devel] openafs / opendfs collaboration)

[Luke Kenneth Casson Leighton]

On Fri, Jan 28, 2005 at 03:45:08PM +0100, Alexander Bostr?m wrote:
> fre 2005-01-21 klockan 12:22 -0500 skrev Derrick J Brashear: 
> > On Fri, 21 Jan 2005, Matthew Miller wrote:
> > 
> > > On Fri, Jan 21, 2005 at 09:00:59AM -0500, Derrick J Brashear wrote:
> 
> > >> It seems like Arla would probably have a better model for us all to follow
> > >> if we did so.
> > >
> > > Or on Linux, something based on FUSE, which is apparently now getting
> > > merged.
> > 
> > Arla's nnpfs is actually portable, one filesystem per platform sort of 
> > sucks.
> 
> But it lacks a nice libnnpfs that one can use to implement a filesystem.
> 
> Every OS should have some kind of userland filesystem interface. Linux
> might get FUSE (and it might be adequate*), HURD has one, Dragonfly are
> aiming at it and on some systems there's nnpfs already. On top of those
> interfaces there could be a set of libraries implementing a common API
> for all the platforms.
> 
> *) Last time I looked at FUSE the security model was: If the current uid
> equals the owner of the mountpoint then forward the request to the
> userland daemon, without any authentication information like for example
> the current uid. This might have or could be changed though.

 as of 2.6.7-ish (last time i looked: 2.5 months) there was
 no forwarding of security: in fact there was nothing in any of the
 APIs about security at all: in fact, root as a user was banned (with
 good justification iirc)

 also, the xattr handling was (is?) non-existant and i had to
 add it, but it was unsuitable for selinux, and that's a design
 mismatch between fuse's way of communicating with its userspace
 daemon (err -512 "please try later") and selinux's requirement
 for instant answers (inability to cope with err -512)

 so i started to look at lufs instead, which appeared to be a much
 cleaner design.

 lufs expects the userspace daemon to handle and manage inodes,
 whereas fuse instead keeps an in-memory cache of inodes in
 the userspace daemon, does a hell of a lot of extra fstat'ing
 for you in order to guarantee file consistency, that sort of thing.

 there is an API / library which your userspace daemon is expected to
 use: this library handles the communication to the kernel and also it
 handles the inode proxy redirection and cacheing for you.

 lufs has a heck of a lot more examples available for it than fuse
 does.

 that all having been said, i don't think lufs's API has any
 security handling either.

 six of one, half a dozen of the other, but if you wanted fuse
 to support selinux and any other form of security that involves
 extended attributes, you would definitely have problems: i didn't get
 round to evaluating lufs for selinux (ran out of time and money).

 l.

--
<a href="http://lkcl.net">http://lkcl.net</a>
--

Re: Using fuse for AFS/DFS (was Re: [OpenAFS-devel] openafs / opendfs collaboration)

[Miklos Szeredi]

> > *) Last time I looked at FUSE the security model was: If the current uid
> > equals the owner of the mountpoint then forward the request to the
> > userland daemon, without any authentication information like for example
> > the current uid. This might have or could be changed though.
> 
>  as of 2.6.7-ish (last time i looked: 2.5 months) there was
>  no forwarding of security: in fact there was nothing in any of the
>  APIs about security at all: in fact, root as a user was banned (with
>  good justification iirc)

There are two choices for the security model in FUSE.  The first
choice is that the userspace filesystem does the permission checking
in each operation.  Current uid and gid is available, group list is
presently not.

The other choice is that the kernel does the normal file mode based
permission checking.  Obviously in this case the filesystem can still
implement an additional (stricter) permission policy.

The "root banning" issue is in fact orthogonal to this.  The default
operation is that only the user who mounted the filesystem is allowed
to access the contents.  This behavior can be switched off with a
mount option, to allow access to all users.

>  also, the xattr handling was (is?) non-existant and i had to add
>  it,

Looking at the changelog it was added on 2004-03-30, so you must be
using a pretty outdated version.

>  but it was unsuitable for selinux, and that's a design mismatch
>  between fuse's way of communicating with its userspace daemon (err
>  -512 "please try later") and selinux's requirement for instant
>  answers (inability to cope with err -512)

Heh?  Where did you see error value 512 (ERESTARTSYS)?  It's not
something that the userspace daemon can return.

>  so i started to look at lufs instead, which appeared to be a much
>  cleaner design.

That's pretty subjective.  Please back up your statement with concrete
examples, so maybe then I can do something about it.

>  lufs expects the userspace daemon to handle and manage inodes,
>  whereas fuse instead keeps an in-memory cache of inodes in
>  the userspace daemon, does a hell of a lot of extra fstat'ing
>  for you in order to guarantee file consistency, that sort of thing.

Well, how much "hell of a lot" actually is depends on a lot of things.
E.g. on whether the backed up filesystem is modified externally (not
just through the kernel).  If not, then it will stay consistent
without any extra messaging.  This can be set by a timeout parameter
for each looked up entry.

The extra flexibility offered by an inode based kernel interface
(FUSE) instead of a path based one (LUFS) I think outweighs the
disadvantage of having to once look up each path element.

>  there is an API / library which your userspace daemon is expected to
>  use: this library handles the communication to the kernel and also it
>  handles the inode proxy redirection and cacheing for you.

Yes, useful for some filesystems (sshfs, ftpfs) useless for others.  I
plan to add a generic caching layer to the FUSE library as well.

>  lufs has a heck of a lot more examples available for it than fuse
>  does.

In the LUFS package yes.  However I bet, currently there are much more
applications which use FUSE than LUFS.

Thanks,
Miklos

Re: Using fuse for AFS/DFS (was Re: [OpenAFS-devel] openafs / opendfs collaboration)

[Luke Kenneth Casson Leighton]

On Sun, Jan 30, 2005 at 12:13:04PM +0100, Miklos Szeredi wrote:
> > > *) Last time I looked at FUSE the security model was: If the current uid
> > > equals the owner of the mountpoint then forward the request to the
> > > userland daemon, without any authentication information like for example
> > > the current uid. This might have or could be changed though.
> > 
> >  as of 2.6.7-ish (last time i looked: 2.5 months) there was
> >  no forwarding of security: in fact there was nothing in any of the
> >  APIs about security at all: in fact, root as a user was banned (with
> >  good justification iirc)
> 
> There are two choices for the security model in FUSE.  The first
> choice is that the userspace filesystem does the permission checking
> in each operation.  Current uid and gid is available, group list is
> presently not.

> The other choice is that the kernel does the normal file mode based
> permission checking.  Obviously in this case the filesystem can still
> implement an additional (stricter) permission policy.

 if your users are okay with having to run a fuse-mount themselves,
 that's okay [to have the kernel do the file mode checking]

 the problem with that is that you can't have a "publicly accessible"
 mount point like you do on an nfs server.

 also if you have a completely different kind of file permission
 checking system (which AFS and DFS do), you're stuffed.


> The "root banning" issue is in fact orthogonal to this.  The default
> operation is that only the user who mounted the filesystem is allowed
> to access the contents.  This behavior can be switched off with a
> mount option, to allow access to all users.
> 
> >  also, the xattr handling was (is?) non-existant and i had to add
> >  it,
> 
> Looking at the changelog it was added on 2004-03-30, so you must be
> using a pretty outdated version.
 
  ... Release 1.3 - 2004-07-14.

 hm, error-at-memory-recall fault, redo from start...

> >  but it was unsuitable for selinux, and that's a design mismatch
> >  between fuse's way of communicating with its userspace daemon (err
> >  -512 "please try later") and selinux's requirement for instant
> >  answers (inability to cope with err -512)
> 
> Heh?  Where did you see error value 512 (ERESTARTSYS)?  It's not
> something that the userspace daemon can return.
 
 userspace no, kernel, yes.

 the kernel-part of fuse tells any kernel-level callers to
 "go away, come back later".

 obviously this gives time for the kernel-part to "wake up" the
 userspace daemon, obtain an answer, such that when the kernel-level
 caller _does_ come back, the information is available.

 the problem with using SELinux to obtain xattrs
 "security.selinux" in order to perform security checks
 is that the checking is done from in the kernel ITSELF
 (security/hooks.c), not by a userspace function call RESULTING
 in a kernel call.

 therefore when you even attempt to _mount_ a selinux-enabled fuse
 filesystem, hooks.c tests to see whether the filesystem supports
 xattrs, gets this silly 512 (ERESTARTSYS) error message and goes "nope,
 doesn't look like it does".

 for various reasons, the details of which i am not aware of,
 from what i can gather, getting selinux to support ERESTARTSYS
 is tricky.


> >  so i started to look at lufs instead, which appeared to be a much
> >  cleaner design.
> 
> That's pretty subjective.  Please back up your statement with concrete
> examples, so maybe then I can do something about it.

 i must apologise for not having sufficient time _at present_
 to do that: please i would therefore as you to treat my
 statement _as_ subjective until/unless demonstrated otherwise.

 sorry about that.

> >  lufs expects the userspace daemon to handle and manage inodes,
> >  whereas fuse instead keeps an in-memory cache of inodes in
> >  the userspace daemon, does a hell of a lot of extra fstat'ing
> >  for you in order to guarantee file consistency, that sort of thing.
> 
> Well, how much "hell of a lot" actually is depends on a lot of things.
> E.g. on whether the backed up filesystem is modified externally (not
> just through the kernel).  If not, then it will stay consistent
> without any extra messaging.  This can be set by a timeout parameter
> for each looked up entry.
> 
> The extra flexibility offered by an inode based kernel interface
> (FUSE) instead of a path based one (LUFS) I think outweighs the
> disadvantage of having to once look up each path element.

 mrr, yehhh... mmmm :)
 
 what about a remote NTFS filesystem which supports NT Security
 Descriptors, which are "inherited" where you not only don't
 have the concept of inodes, but also due to the security
 model, a client must look up every path element _anyway_
 and perform a conglomeration of the "inheritance" parts of
 the ACEs in each security descriptor of the path components?

 :)

 btw so people don't freak out too badly at that concept,
 there _has_ existed for a couple of decades the concept of
 "change notify" in remote NT filesystems, where the client
 can watch for any significant changes on a filesystem, so you
 don't have to end up re-reading all of the path components,
 you can get the remote server to _tell_ you when they've
 changed - cool, huh?

 [btw, nt's change notify is what spurred linux kernel's inotify and
  dnotify to be written]

 
 in a nutshell, inodes is an optimisation from a unix
 perspective: by providing an inode based interface, you are
 burdening _all_ filesystem implementers with that concept.

 l.

Re: Using fuse for AFS/DFS (was Re: [OpenAFS-devel] openafs / opendfs collaboration)

[Miklos Szeredi]

> > There are two choices for the security model in FUSE.  The first
> > choice is that the userspace filesystem does the permission checking
> > in each operation.  Current uid and gid is available, group list is
> > presently not.
> 
> > The other choice is that the kernel does the normal file mode based
> > permission checking.  Obviously in this case the filesystem can still
> > implement an additional (stricter) permission policy.
> 
>  if your users are okay with having to run a fuse-mount themselves,
>  that's okay [to have the kernel do the file mode checking]
> 
>  the problem with that is that you can't have a "publicly accessible"
>  mount point like you do on an nfs server.
> 
>  also if you have a completely different kind of file permission
>  checking system (which AFS and DFS do), you're stuffed.

No.  Just fall back on the first option (permission checking in each
operation)

> > Looking at the changelog it was added on 2004-03-30, so you must be
> > using a pretty outdated version.
>  
>   ... Release 1.3 - 2004-07-14.
> 
>  hm, error-at-memory-recall fault, redo from start...

OK.  From 1.2 onwards it was just a bugfix branch with the new
features going into the FUSE-2 release.

> > Heh?  Where did you see error value 512 (ERESTARTSYS)?  It's not
> > something that the userspace daemon can return.
>  
>  userspace no, kernel, yes.
> 
>  the kernel-part of fuse tells any kernel-level callers to
>  "go away, come back later".
> 
>  obviously this gives time for the kernel-part to "wake up" the
>  userspace daemon, obtain an answer, such that when the kernel-level
>  caller _does_ come back, the information is available.

It doesn't do that and never did.  ERESTARTSYS is only returned if the
operation is interrupted, and in that case the operation is restarted
from scratch, the answer to the old request is never used.

>  the problem with using SELinux to obtain xattrs
>  "security.selinux" in order to perform security checks
>  is that the checking is done from in the kernel ITSELF
>  (security/hooks.c), not by a userspace function call RESULTING
>  in a kernel call.
> 
>  therefore when you even attempt to _mount_ a selinux-enabled fuse
>  filesystem, hooks.c tests to see whether the filesystem supports
>  xattrs, gets this silly 512 (ERESTARTSYS) error message and goes "nope,
>  doesn't look like it does".
> 
>  for various reasons, the details of which i am not aware of,
>  from what i can gather, getting selinux to support ERESTARTSYS
>  is tricky.

Just disable signal delivery while calling FUSE, and it will never
return -ERESTARTSYS or -ERESTARTNOINTR.

> > The extra flexibility offered by an inode based kernel interface
> > (FUSE) instead of a path based one (LUFS) I think outweighs the
> > disadvantage of having to once look up each path element.
> 
>  mrr, yehhh... mmmm :)
>  
>  what about a remote NTFS filesystem which supports NT Security
>  Descriptors, which are "inherited" where you not only don't
>  have the concept of inodes, but also due to the security
>  model, a client must look up every path element _anyway_
>  and perform a conglomeration of the "inheritance" parts of
>  the ACEs in each security descriptor of the path components?

There's not that much difference between the inode and the path model.
If you say each "path component" corresponds to an inode, you have
just solved this problem.

>  btw so people don't freak out too badly at that concept,
>  there _has_ existed for a couple of decades the concept of
>  "change notify" in remote NT filesystems, where the client
>  can watch for any significant changes on a filesystem, so you
>  don't have to end up re-reading all of the path components,
>  you can get the remote server to _tell_ you when they've
>  changed - cool, huh?

Yes.  And would be pretty easy to add support for this to the FUSE
interface.  It's currently isn't there just because nobody demanded
it.

>  [btw, nt's change notify is what spurred linux kernel's inotify and
>   dnotify to be written]
> 
>  
>  in a nutshell, inodes is an optimisation from a unix
>  perspective: by providing an inode based interface, you are
>  burdening _all_ filesystem implementers with that concept.

Yes.  However I think the burden on performance (nothing else), is
justified by the better flexibility.

Thanks,
Miklos

Re: Using fuse for AFS/DFS (was Re: [OpenAFS-devel] openafs / opendfs collaboration)

[Luke Kenneth Casson Leighton]

On Sun, Jan 30, 2005 at 01:40:35PM +0100, Miklos Szeredi wrote:

> >  the kernel-part of fuse tells any kernel-level callers to
> >  "go away, come back later".
> > 
> >  obviously this gives time for the kernel-part to "wake up" the
> >  userspace daemon, obtain an answer, such that when the kernel-level
> >  caller _does_ come back, the information is available.
> 
> It doesn't do that and never did.  ERESTARTSYS is only returned if the
> operation is interrupted, and in that case the operation is restarted
> from scratch, the answer to the old request is never used.
 
 oh??

 *confused* - well that's good, then!  glad that's cleared up!
 
 [must contact you again about this when i have time]

> >  in a nutshell, inodes is an optimisation from a unix
> >  perspective: by providing an inode based interface, you are
> >  burdening _all_ filesystem implementers with that concept.
> 
> Yes.  However I think the burden on performance (nothing else), is
> justified by the better flexibility.

 i understand.

 l.