* FS: hardlinks on directories @ 2003-08-04 12:15 Stephan von Krawczynski 2003-08-04 12:45 ` Måns Rullgård ` (2 more replies) 0 siblings, 3 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 12:15 UTC (permalink / raw) To: linux-kernel Hello all, although it is very likely I am entering (again :-) an ancient discussion I would like to ask why hardlinks on directories are not allowed/no supported fs action these days. I can't think of a good reason for this, but can think of many good reasons why one would like to have such a function, amongst those: - restructuring of the fs to meet different sorting criterias (kind of a database view onto the fs) - relinking of the fs for export to other hosts via nfs or the like (enhanced security through artificially constructed, exported trees) Would a feature like that be seen as "big action" or rather small enhancement to the fs-layer? Are there any supporters for this feature out there? Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 12:15 FS: hardlinks on directories Stephan von Krawczynski @ 2003-08-04 12:45 ` Måns Rullgård 2003-08-04 13:22 ` Stephan von Krawczynski 2003-08-04 12:47 ` Nikita Danilov 2003-08-04 13:44 ` Andries Brouwer 2 siblings, 1 reply; 79+ messages in thread From: Måns Rullgård @ 2003-08-04 12:45 UTC (permalink / raw) To: linux-kernel Stephan von Krawczynski <skraw@ithnet.com> writes: > although it is very likely I am entering (again :-) an ancient > discussion I would like to ask why hardlinks on directories are not > allowed/no supported fs action these days. I can't think of a good > reason for this, but can think of many good reasons why one would > like to have such a function, amongst those: I don't know the exact reasons it isn't allowed, but you can always use "mount --bind" to get a similar effect. -- Måns Rullgård mru@users.sf.net ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 12:45 ` Måns Rullgård @ 2003-08-04 13:22 ` Stephan von Krawczynski 2003-08-04 13:37 ` Christian Reichert 2003-08-04 15:31 ` Jeff Muizelaar 0 siblings, 2 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 13:22 UTC (permalink / raw) To: Måns Rullgård; +Cc: linux-kernel On Mon, 04 Aug 2003 14:45:58 +0200 mru@users.sourceforge.net (Måns Rullgård) wrote: > Stephan von Krawczynski <skraw@ithnet.com> writes: > > > although it is very likely I am entering (again :-) an ancient > > discussion I would like to ask why hardlinks on directories are not > > allowed/no supported fs action these days. I can't think of a good > > reason for this, but can think of many good reasons why one would > > like to have such a function, amongst those: > > I don't know the exact reasons it isn't allowed, but you can always > use "mount --bind" to get a similar effect. I guess this is not really an option if talking about hundreds or thousands of "links", is it? Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 13:22 ` Stephan von Krawczynski @ 2003-08-04 13:37 ` Christian Reichert 2003-08-04 13:44 ` Stephan von Krawczynski 2003-08-04 15:31 ` Jeff Muizelaar 1 sibling, 1 reply; 79+ messages in thread From: Christian Reichert @ 2003-08-04 13:37 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Måns Rullgård, linux-kernel Hi ! the fundamental problem as i know it is that the FS design in unix is based on a directory TREE structure - however if you implement hard links for directories you are breaking this strict treeu structure and can end up with loops/graphs. (What are the cases where a symlink wouldn't be enough ?) Cheers, Chris Zitat von Stephan von Krawczynski <skraw@ithnet.com>: > On Mon, 04 Aug 2003 14:45:58 +0200 > mru@users.sourceforge.net (Måns Rullgård) wrote: > > > Stephan von Krawczynski <skraw@ithnet.com> writes: > > > > > although it is very likely I am entering (again :-) an ancient > > > discussion I would like to ask why hardlinks on directories are not > > > allowed/no supported fs action these days. I can't think of a good > > > reason for this, but can think of many good reasons why one would > > > like to have such a function, amongst those: > > > > I don't know the exact reasons it isn't allowed, but you can always > > use "mount --bind" to get a similar effect. > > I guess this is not really an option if talking about hundreds or thousands > of > "links", is it? > > Regards, > Stephan > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 13:37 ` Christian Reichert @ 2003-08-04 13:44 ` Stephan von Krawczynski 2003-08-04 14:22 ` Christian Reichert 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 13:44 UTC (permalink / raw) To: Christian Reichert; +Cc: mru, linux-kernel On Mon, 4 Aug 2003 15:37:01 +0200 Christian Reichert <c.reichert@resolution.de> wrote: > Hi ! > > the fundamental problem as i know it is that the FS design in unix is based > on a directory TREE structure - however if you implement hard links for > directories you are breaking this strict treeu structure and can end up with > loops/graphs. This is just the same with softlinks, or not? > (What are the cases where a symlink wouldn't be enough ?) fs export via nfs. It only works out if all your symlinked trees are completely visible to the client, so you have to open big wholes for no good reason at all ... Just as an example. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 13:44 ` Stephan von Krawczynski @ 2003-08-04 14:22 ` Christian Reichert 0 siblings, 0 replies; 79+ messages in thread From: Christian Reichert @ 2003-08-04 14:22 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: mru, linux-kernel Zitat von Stephan von Krawczynski <skraw@ithnet.com>: > On Mon, 4 Aug 2003 15:37:01 +0200 > Christian Reichert <c.reichert@resolution.de> wrote: > > > Hi ! > > > > the fundamental problem as i know it is that the FS design in unix is > based > > on a directory TREE structure - however if you implement hard links for > > directories you are breaking this strict treeu structure and can end up > with > > loops/graphs. > > This is just the same with softlinks, or not? > Not really, symlinks are just files which contain the location of the directory - so the 'physical' structure still stays a tree .. Cheers, Chris ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 13:22 ` Stephan von Krawczynski 2003-08-04 13:37 ` Christian Reichert @ 2003-08-04 15:31 ` Jeff Muizelaar 2003-08-04 16:15 ` Stephan von Krawczynski 1 sibling, 1 reply; 79+ messages in thread From: Jeff Muizelaar @ 2003-08-04 15:31 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: linux-kernel, Måns Rullgård Stephan von Krawczynski wrote: > >I guess this is not really an option if talking about hundreds or thousands of >"links", is it? > > actually hundreds or thounds still should be ok. See... >From: Alexander Viro <http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&safe=off&q=author:viro%40math.psu.edu+> (viro@math.psu.edu <mailto:viro%40math.psu.edu>) >Subject: Re: hundreds of mount --bind mountpoints? > >On Sun, 22 Apr 2001, David L. Parsley wrote: >> Hi, >> >> I'm still working on a packaging system for diskless (quasi-embedded) >> devices. The root filesystem is all tmpfs, and I attach packages inside >> it. Since symlinks in a tmpfs filesystem cost 4k each (ouch!), I'm >> considering using mount --bind for everything. This appears to use very >> little memory, but I'm wondering if I'll run into problems when I start >> having many hundreds of bind mountings. Any feel for this? > >Memory use is sizeof(struct vfsmount) per binding. In principle, you can get >in trouble when size of /proc/mount will get past 4Kb - you'll get only >first 4 (actually 3, IIRC) kilobytes, so stuff that relies on the contents >of said file may get unhappy. It's fixable, though. > > > The 4Kb problem has also been solved in 2.6, I just tested having about 5k mounts and things seemed fine. /proc/mounts reports all of them. -Jeff ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:31 ` Jeff Muizelaar @ 2003-08-04 16:15 ` Stephan von Krawczynski 2003-08-05 2:45 ` Neil Brown 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 16:15 UTC (permalink / raw) To: Jeff Muizelaar; +Cc: linux-kernel, mru, Neil Brown On Mon, 04 Aug 2003 11:31:47 -0400 Jeff Muizelaar <muizelaar@rogers.com> wrote: > Stephan von Krawczynski wrote: > > > > >I guess this is not really an option if talking about hundreds or thousands > >of"links", is it? > > > > > actually hundreds or thounds still should be ok. See... Hm, and I just found out that re-exporting "mount --bind" volumes does not work over nfs... Is this correct, Neil? Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 16:15 ` Stephan von Krawczynski @ 2003-08-05 2:45 ` Neil Brown 2003-08-05 9:41 ` Stephan von Krawczynski 0 siblings, 1 reply; 79+ messages in thread From: Neil Brown @ 2003-08-05 2:45 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Jeff Muizelaar, linux-kernel, mru On Monday August 4, skraw@ithnet.com wrote: > On Mon, 04 Aug 2003 11:31:47 -0400 > Jeff Muizelaar <muizelaar@rogers.com> wrote: > > > Stephan von Krawczynski wrote: > > > > > > > >I guess this is not really an option if talking about hundreds or thousands > > >of"links", is it? > > > > > > > > actually hundreds or thounds still should be ok. See... > > Hm, and I just found out that re-exporting "mount --bind" volumes does not work > over nfs... > > Is this correct, Neil? Yes, though there is a reasonable chance that it can be made to work with linux-2.6.0 and nfs-utils-1.1.0 (neither of which have been released yet:-) But I'm a big fan of auto-mounters - e.g. am-utils. If you want a filesystem that is assembled from lots of bits of other filesystem, describe it to am-utils and it will present it to you. NeilBrown ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 2:45 ` Neil Brown @ 2003-08-05 9:41 ` Stephan von Krawczynski 2003-08-06 1:12 ` Neil Brown 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 9:41 UTC (permalink / raw) To: Neil Brown; +Cc: muizelaar, linux-kernel, mru On Tue, 5 Aug 2003 12:45:29 +1000 Neil Brown <neilb@cse.unsw.edu.au> wrote: > On Monday August 4, skraw@ithnet.com wrote: > > On Mon, 04 Aug 2003 11:31:47 -0400 > > Jeff Muizelaar <muizelaar@rogers.com> wrote: > > > > > Stephan von Krawczynski wrote: > > > > > > > > > > >I guess this is not really an option if talking about hundreds or > > > >thousands of"links", is it? > > > > > > > > > > > actually hundreds or thounds still should be ok. See... > > > > Hm, and I just found out that re-exporting "mount --bind" volumes does not > > work over nfs... > > > > Is this correct, Neil? > > Yes, though there is a reasonable chance that it can be made to work > with linux-2.6.0 and nfs-utils-1.1.0 (neither of which have been > released yet:-) Is this a complex issue? Can you imagine a not-too-big sized patch can make it work in 2.4? What is the basic reason it does in fact not work? Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 9:41 ` Stephan von Krawczynski @ 2003-08-06 1:12 ` Neil Brown 2003-08-06 10:14 ` Stephan von Krawczynski 0 siblings, 1 reply; 79+ messages in thread From: Neil Brown @ 2003-08-06 1:12 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: muizelaar, linux-kernel, mru On Tuesday August 5, skraw@ithnet.com wrote: > > > Hm, and I just found out that re-exporting "mount --bind" volumes does not > > > work over nfs... > > > > > > Is this correct, Neil? > > > > Yes, though there is a reasonable chance that it can be made to work > > with linux-2.6.0 and nfs-utils-1.1.0 (neither of which have been > > released yet:-) > > Is this a complex issue? Can you imagine a not-too-big sized patch can make it > work in 2.4? What is the basic reason it does in fact not work? On reflection, it could probably work in 2.4 and current nfs-utils, but admin might be a bit clumsy. To allow knfsd to see a mountpoint, you have to export the mounted directory with the "nohide" option. Currently "nohide" only works properly for exports to specific hosts, not to wildcarded hosts or netgroups. So if your /etc/export contains: /path/to/some/--bind/mountpoint servername(nohide,....) for every mountpoint and every server, then it should work. In 2.6, you can (will be able to) just export the top level mount point with "crossmnt" and it should all work for you. Getting that functionality into 2.4 would be a very big job. NeilBrown ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-06 1:12 ` Neil Brown @ 2003-08-06 10:14 ` Stephan von Krawczynski 2003-08-07 2:27 ` Neil Brown 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-06 10:14 UTC (permalink / raw) To: Neil Brown; +Cc: muizelaar, linux-kernel, mru On Wed, 6 Aug 2003 11:12:38 +1000 Neil Brown <neilb@cse.unsw.edu.au> wrote: > On Tuesday August 5, skraw@ithnet.com wrote: > > > > Hm, and I just found out that re-exporting "mount --bind" volumes does > > > > not work over nfs... > > > > > > > > Is this correct, Neil? > > > > > > Yes, though there is a reasonable chance that it can be made to work > > > with linux-2.6.0 and nfs-utils-1.1.0 (neither of which have been > > > released yet:-) > > > > Is this a complex issue? Can you imagine a not-too-big sized patch can make > > it work in 2.4? What is the basic reason it does in fact not work? > > On reflection, it could probably work in 2.4 and current nfs-utils, > but admin might be a bit clumsy. > > To allow knfsd to see a mountpoint, you have to export the mounted > directory with the "nohide" option. Currently "nohide" only works > properly for exports to specific hosts, not to wildcarded hosts or > netgroups. > So if your /etc/export contains: > > /path/to/some/--bind/mountpoint servername(nohide,....) > > for every mountpoint and every server, then it should work. Hm, bad luck. I tried and it did not work. I used 2.4.20 kernel, are there chances a later kernel might work? Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-06 10:14 ` Stephan von Krawczynski @ 2003-08-07 2:27 ` Neil Brown 0 siblings, 0 replies; 79+ messages in thread From: Neil Brown @ 2003-08-07 2:27 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: muizelaar, linux-kernel, mru On Wednesday August 6, skraw@ithnet.com wrote: > On Wed, 6 Aug 2003 11:12:38 +1000 > Neil Brown <neilb@cse.unsw.edu.au> wrote: > > > On Tuesday August 5, skraw@ithnet.com wrote: > > > > > Hm, and I just found out that re-exporting "mount --bind" volumes does > > > > > not work over nfs... > > > > > > > > > > Is this correct, Neil? > > > > > > > > Yes, though there is a reasonable chance that it can be made to work > > > > with linux-2.6.0 and nfs-utils-1.1.0 (neither of which have been > > > > released yet:-) > > > > > > Is this a complex issue? Can you imagine a not-too-big sized patch can make > > > it work in 2.4? What is the basic reason it does in fact not work? > > > > On reflection, it could probably work in 2.4 and current nfs-utils, > > but admin might be a bit clumsy. > > > > To allow knfsd to see a mountpoint, you have to export the mounted > > directory with the "nohide" option. Currently "nohide" only works > > properly for exports to specific hosts, not to wildcarded hosts or > > netgroups. > > So if your /etc/export contains: > > > > /path/to/some/--bind/mountpoint servername(nohide,....) > > > > for every mountpoint and every server, then it should work. > > Hm, bad luck. I tried and it did not work. I used 2.4.20 kernel, are there > chances a later kernel might work? It worked for me. There is nothing in recent kernels that would affect this. What errors do you get? Can you describe your setup in a bit more detail. One thing to be careful of is that you cannot export a directory and a parent of that directory in the same filesystem to the same client. This tripped me up the first time I tried it. Here "parent" means in the real filesystem, ignoring any --bind mounts. NeilBrown ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 12:15 FS: hardlinks on directories Stephan von Krawczynski 2003-08-04 12:45 ` Måns Rullgård @ 2003-08-04 12:47 ` Nikita Danilov 2003-08-04 13:32 ` Stephan von Krawczynski 2003-08-04 13:44 ` Andries Brouwer 2 siblings, 1 reply; 79+ messages in thread From: Nikita Danilov @ 2003-08-04 12:47 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: linux-kernel Stephan von Krawczynski writes: > Hello all, > > although it is very likely I am entering (again :-) an ancient discussion I > would like to ask why hardlinks on directories are not allowed/no supported fs > action these days. I can't think of a good reason for this, but can think of > many good reasons why one would like to have such a function, amongst those: > > - restructuring of the fs to meet different sorting criterias (kind of a > database view onto the fs) > - relinking of the fs for export to other hosts via nfs or the like (enhanced > security through artificially constructed, exported trees) > > Would a feature like that be seen as "big action" or rather small enhancement > to the fs-layer? > Are there any supporters for this feature out there? Hard links on directories are hard to do in the UNIX file system model. Where ".." should point? How to implement rmdir? You can think about UNIX unlink as some form of reference counter based garbage collector---when last (persistent) reference to the object is removed it is safe to recycle it. It is well-known that reference counting GC cannot cope with cyclical references. Usually this is not a problem for the file system because all cyclical references are very well known---they always involve "." and "..". But as one allows hard links on directories, file system is no longer tree, but generic directed graph and reference counting GC wouldn't work. > > Regards, > Stephan Nikita. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 12:47 ` Nikita Danilov @ 2003-08-04 13:32 ` Stephan von Krawczynski 0 siblings, 0 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 13:32 UTC (permalink / raw) To: Nikita Danilov; +Cc: linux-kernel On Mon, 4 Aug 2003 16:47:14 +0400 Nikita Danilov <Nikita@Namesys.COM> wrote: > Hard links on directories are hard to do in the UNIX file system model. > > Where ".." should point? How to implement rmdir? You can think about > UNIX unlink as some form of reference counter based garbage > collector---when last (persistent) reference to the object is removed it > is safe to recycle it. It is well-known that reference counting GC > cannot cope with cyclical references. Usually this is not a problem for > the file system because all cyclical references are very well > known---they always involve "." and "..". But as one allows hard links > on directories, file system is no longer tree, but generic directed > graph and reference counting GC wouldn't work. If file-/directory-nodes are single-linked list nodes inside one directory, and directory-nodes pointing to the same directory are single-linked list nodes, you can: - ".." do as first node of a directory and the "shared" part of the directory follows on its next-pointer, so you have one ".." for every hard-link. - implement rmdir as throw-away dir-node and ".." node and only if pointer to next hw-linked directory-node is itself remove rest of linked node list Are there further questionable operations? Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 12:15 FS: hardlinks on directories Stephan von Krawczynski 2003-08-04 12:45 ` Måns Rullgård 2003-08-04 12:47 ` Nikita Danilov @ 2003-08-04 13:44 ` Andries Brouwer 2003-08-04 13:56 ` Stephan von Krawczynski 2003-08-04 15:42 ` Brian Pawlowski 2 siblings, 2 replies; 79+ messages in thread From: Andries Brouwer @ 2003-08-04 13:44 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: linux-kernel On Mon, Aug 04, 2003 at 02:15:48PM +0200, Stephan von Krawczynski wrote: > although it is very likely I am entering (again :-) an ancient discussion I > would like to ask why hardlinks on directories are not allowed/no supported fs > action these days. Quite a lot of software thinks that the file hierarchy is a tree, if you wish a forest. Things would break badly if the hierarchy became an arbitrary graph. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 13:44 ` Andries Brouwer @ 2003-08-04 13:56 ` Stephan von Krawczynski 2003-08-04 14:04 ` Anton Altaparmakov ` (2 more replies) 2003-08-04 15:42 ` Brian Pawlowski 1 sibling, 3 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 13:56 UTC (permalink / raw) To: Andries Brouwer; +Cc: linux-kernel On Mon, 4 Aug 2003 15:44:15 +0200 Andries Brouwer <aebr@win.tue.nl> wrote: > On Mon, Aug 04, 2003 at 02:15:48PM +0200, Stephan von Krawczynski wrote: > > > although it is very likely I am entering (again :-) an ancient discussion I > > would like to ask why hardlinks on directories are not allowed/no supported > > fs action these days. > > Quite a lot of software thinks that the file hierarchy is a tree, > if you wish a forest. > > Things would break badly if the hierarchy became an arbitrary graph. Care to name one? What exactly is the rule you see broken? Sure you can build loops, but you cannot prevent people from doing braindamaged things to their data anyway. You would not ban dd either for being able to flatten your harddisk only because of one mistyping char... Every feature can be misused and then damaging, but that is no real reason not to have it - IMHO. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 13:56 ` Stephan von Krawczynski @ 2003-08-04 14:04 ` Anton Altaparmakov 2003-08-04 14:50 ` Stephan von Krawczynski 2003-08-04 14:33 ` Jesse Pollard 2003-08-04 20:47 ` Jan Harkes 2 siblings, 1 reply; 79+ messages in thread From: Anton Altaparmakov @ 2003-08-04 14:04 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Andries Brouwer, linux-kernel On Mon, 4 Aug 2003, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 15:44:15 +0200 > Andries Brouwer <aebr@win.tue.nl> wrote: > > On Mon, Aug 04, 2003 at 02:15:48PM +0200, Stephan von Krawczynski wrote: > > > although it is very likely I am entering (again :-) an ancient discussion I > > > would like to ask why hardlinks on directories are not allowed/no supported > > > fs action these days. > > > > Quite a lot of software thinks that the file hierarchy is a tree, > > if you wish a forest. > > > > Things would break badly if the hierarchy became an arbitrary graph. > > Care to name one? What exactly is the rule you see broken? Sure you can build > loops, but you cannot prevent people from doing braindamaged things to their > data anyway. You would not ban dd either for being able to flatten your > harddisk only because of one mistyping char... > Every feature can be misused and then damaging, but that is no real reason not > to have it - IMHO. For a start the kernel VFS dcache would break because you end up with multiple entries for each inode, one entry for each parallel directory tree. Read-only you are just about able to get away with it (been there, done that, don't recommend it!) but allow files to be deleted and it will blow up in your face. You ask for examples of applications? There are millions! Anything that walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, du, any type of search and/or indexing engine, chown -R, cp -R, scp -R, chmod -R, etc... Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 14:04 ` Anton Altaparmakov @ 2003-08-04 14:50 ` Stephan von Krawczynski 2003-08-04 20:03 ` Olivier Galibert ` (2 more replies) 0 siblings, 3 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 14:50 UTC (permalink / raw) To: Anton Altaparmakov; +Cc: aebr, linux-kernel On Mon, 4 Aug 2003 15:04:28 +0100 (BST) Anton Altaparmakov <aia21@cam.ac.uk> wrote: > For a start the kernel VFS dcache would break because you end up with > multiple entries for each inode, one entry for each parallel directory > tree. Read-only you are just about able to get away with it (been there, > done that, don't recommend it!) but allow files to be deleted and it will > blow up in your face. I cannot comment, I have no inside knowledge of it. > You ask for examples of applications? There are millions! Anything that > walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, > du, any type of search and/or indexing engine, chown -R, cp -R, scp > -R, chmod -R, etc... There is a flaw in this argument. If I am told that mount --bind does just about what I want to have as a feature then these applictions must have the same problems already (if I mount braindead). So an implementation in fs cannot do any _additional_ damage to these applications, or not? My saying is not "I want to have hardlinks for creating a big mess of loops inside my filesystems". Your view simply drops the fact that there are more possibilities to create and use hardlinks without any loops... Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 14:50 ` Stephan von Krawczynski @ 2003-08-04 20:03 ` Olivier Galibert 2003-08-04 21:16 ` Jesse Pollard 2003-08-04 22:58 ` Andrew Pimlott 2 siblings, 0 replies; 79+ messages in thread From: Olivier Galibert @ 2003-08-04 20:03 UTC (permalink / raw) To: linux-kernel On Mon, Aug 04, 2003 at 04:50:02PM +0200, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 15:04:28 +0100 (BST) > Anton Altaparmakov <aia21@cam.ac.uk> wrote: > > You ask for examples of applications? There are millions! Anything that > > walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, > > du, any type of search and/or indexing engine, chown -R, cp -R, scp > > -R, chmod -R, etc... > > There is a flaw in this argument. If I am told that mount --bind does just > about what I want to have as a feature then these applictions must have the > same problems already (if I mount braindead). So an implementation in fs cannot > do any _additional_ damage to these applications, or not? There is a subtle but fundamental difference between mount --bind and hard links for directories[1]: mounts done under a bind point and mounts done under the original tree are independant. An example on a system (rh9) with the "standard" /dev/pts mounted and no devfs: root@zalem:~ #1 >mkdir x root@zalem:~ #2 >mount --bind /dev x root@zalem:~ #3 >ls -l /dev |head -5 total 305 crw------- 1 root root 10, 10 Jan 30 2003 adbmouse crw-r--r-- 1 root root 10, 175 Jan 30 2003 agpgart crw------- 1 root root 10, 4 Jan 30 2003 amigamouse crw------- 1 root root 10, 7 Jan 30 2003 amigamouse1 root@zalem:~ #4 >ls -l x | head -5 total 306 crw------- 1 root root 10, 10 Jan 30 2003 adbmouse crw-r--r-- 1 root root 10, 175 Jan 30 2003 agpgart crw------- 1 root root 10, 4 Jan 30 2003 amigamouse crw------- 1 root root 10, 7 Jan 30 2003 amigamouse1 root@zalem:~ #5 >ls -l x/pts |head -5 total 0 root@zalem:~ #6 >ls -l /dev/pts | head -5 total 0 crw--w---- 1 galibert tty 136, 0 Aug 4 21:56 0 crw--w---- 1 galibert tty 136, 1 Aug 4 21:56 1 crw--w---- 1 galibert tty 136, 2 Aug 4 21:57 2 root@zalem:~ #7 >mount --bind /tmp x/input root@zalem:~ #8 >ls -l x/input | head -5 total 25 -rw-r--r-- 1 galibert mame 0 Aug 1 23:24 Acrodqnzpn drwx------ 2 galibert mame 48 Jul 31 22:52 galibert drwx------ 2 galibert mame 72 Aug 1 22:17 gsrvdir500 srwxrwxrwx 1 wnn wnn 0 Aug 4 21:11 jd_sockV4 root@zalem:~ #9 >ls -l /dev/input | head -5 total 0 crw------- 1 root root 13, 64 Jan 30 2003 event0 crw------- 1 root root 13, 65 Jan 30 2003 event1 crw------- 1 root root 13, 74 Jan 30 2003 event10 crw------- 1 root root 13, 75 Jan 30 2003 event11 You can see that ~root/x and /dev are the same file tree but different mount spaces. Because of that you can't have a loop, because a bind mount can't point to a mount space that is a predecessor of itself. OG. [1] You can mount --bind files. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 14:50 ` Stephan von Krawczynski 2003-08-04 20:03 ` Olivier Galibert @ 2003-08-04 21:16 ` Jesse Pollard 2003-08-04 23:34 ` Stephan von Krawczynski 2003-08-04 22:58 ` Andrew Pimlott 2 siblings, 1 reply; 79+ messages in thread From: Jesse Pollard @ 2003-08-04 21:16 UTC (permalink / raw) To: Stephan von Krawczynski, Anton Altaparmakov; +Cc: aebr, linux-kernel On Monday 04 August 2003 09:50, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 15:04:28 +0100 (BST) > > Anton Altaparmakov <aia21@cam.ac.uk> wrote: > > For a start the kernel VFS dcache would break because you end up with > > multiple entries for each inode, one entry for each parallel directory > > tree. Read-only you are just about able to get away with it (been there, > > done that, don't recommend it!) but allow files to be deleted and it will > > blow up in your face. > > I cannot comment, I have no inside knowledge of it. > > > You ask for examples of applications? There are millions! Anything that > > walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, > > du, any type of search and/or indexing engine, chown -R, cp -R, scp > > -R, chmod -R, etc... > > There is a flaw in this argument. If I am told that mount --bind does just > about what I want to have as a feature then these applictions must have the > same problems already (if I mount braindead). So an implementation in fs > cannot do any _additional_ damage to these applications, or not? Mount -bind only modifies the transient memory storage of a directory. It doesn't change the filesystem. Each bind occupies memory, and on a reboot, the bind is gone. > My saying is not "I want to have hardlinks for creating a big mess of loops > inside my filesystems". Your view simply drops the fact that there are more > possibilities to create and use hardlinks without any loops... been there done that, is is a "big mess of loops". And you can't prevent the loops either, without scanning the entire graph, or keeping a graph location reference embeded with the file. Which then breaks "mv" for renaming directories... It would then have to scan the entire graph again to locate a possble creation of a loop, and regenerate the graph location for every file. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 21:16 ` Jesse Pollard @ 2003-08-04 23:34 ` Stephan von Krawczynski 2003-08-05 14:20 ` Jesse Pollard 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 23:34 UTC (permalink / raw) To: Jesse Pollard; +Cc: aia21, aebr, linux-kernel On Mon, 4 Aug 2003 16:16:39 -0500 Jesse Pollard <jesse@cats-chateau.net> wrote: > > > You ask for examples of applications? There are millions! Anything that > > > walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, > > > du, any type of search and/or indexing engine, chown -R, cp -R, scp > > > -R, chmod -R, etc... > > > > There is a flaw in this argument. If I am told that mount --bind does just > > about what I want to have as a feature then these applictions must have the > > same problems already (if I mount braindead). So an implementation in fs > > cannot do any _additional_ damage to these applications, or not? > > Mount -bind only modifies the transient memory storage of a directory. It > doesn't change the filesystem. Each bind occupies memory, and on a reboot, > the bind is gone. What kind of an argument is this? What difference can you see between a transient loop and a permanent loop for the applications? Exactly zero I guess. In my environments nil boots ought to happen. This is the reason why I would in fact be satisfied with mount -bind if only I could export it via nfs... > > My saying is not "I want to have hardlinks for creating a big mess of loops > > inside my filesystems". Your view simply drops the fact that there are more > > possibilities to create and use hardlinks without any loops... > > been there done that, is is a "big mess of loops". > > And you can't prevent the loops either, without scanning the entire graph, or > keeping a graph location reference embeded with the file. Or marking the links as type links somehow. > Which then breaks "mv" for renaming directories... It would then have to > scan the entire graph again to locate a possble creation of a loop, and > regenerate the graph location for every file. There should be no difference if only a hardlink is simply marked as such by any kind of marker you possibly can think of. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 23:34 ` Stephan von Krawczynski @ 2003-08-05 14:20 ` Jesse Pollard 2003-08-05 14:44 ` Stephan von Krawczynski 0 siblings, 1 reply; 79+ messages in thread From: Jesse Pollard @ 2003-08-05 14:20 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: aia21, aebr, linux-kernel On Monday 04 August 2003 18:34, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 16:16:39 -0500 > > Jesse Pollard <jesse@cats-chateau.net> wrote: > > > > You ask for examples of applications? There are millions! Anything > > > > that walks the directory tree for a start, e.g. ls -R, find, > > > > locatedb, medusa, du, any type of search and/or indexing engine, > > > > chown -R, cp -R, scp -R, chmod -R, etc... > > > > > > There is a flaw in this argument. If I am told that mount --bind does > > > just about what I want to have as a feature then these applictions must > > > have the same problems already (if I mount braindead). So an > > > implementation in fs cannot do any _additional_ damage to these > > > applications, or not? > > > > Mount -bind only modifies the transient memory storage of a directory. It > > doesn't change the filesystem. Each bind occupies memory, and on a > > reboot, the bind is gone. > > What kind of an argument is this? What difference can you see between a > transient loop and a permanent loop for the applications? Exactly zero I > guess. In my environments nil boots ought to happen. simple .. tar --one-file-system will not process past a mount point. > This is the reason why I would in fact be satisfied with mount -bind if > only I could export it via nfs... it's a MOUNT point. NFS doesn't export across mount points just as it doesn't allow exporting a NFS mounted directory. > > > > My saying is not "I want to have hardlinks for creating a big mess of > > > loops inside my filesystems". Your view simply drops the fact that > > > there are more possibilities to create and use hardlinks without any > > > loops... > > > > been there done that, is is a "big mess of loops". > > > > And you can't prevent the loops either, without scanning the entire > > graph, or keeping a graph location reference embeded with the file. > > Or marking the links as type links somehow. > > > Which then breaks "mv" for renaming directories... It would then have to > > scan the entire graph again to locate a possble creation of a loop, and > > regenerate the graph location for every file. > > There should be no difference if only a hardlink is simply marked as such > by any kind of marker you possibly can think of. think about what happens with a "rm -rf name". If there are two parents you cant remove the other parents link without first finding it. hard links do not have a marker. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 14:20 ` Jesse Pollard @ 2003-08-05 14:44 ` Stephan von Krawczynski 0 siblings, 0 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 14:44 UTC (permalink / raw) To: Jesse Pollard; +Cc: aia21, aebr, linux-kernel On Tue, 5 Aug 2003 09:20:41 -0500 Jesse Pollard <jesse@cats-chateau.net> wrote: > > This is the reason why I would in fact be satisfied with mount -bind if > > only I could export it via nfs... > > it's a MOUNT point. NFS doesn't export across mount points just as it doesn't > allow exporting a NFS mounted directory. Hm, I surely have seen re-exports in the past and used them myself, though I did not check under 2.4 setups lately. I definitely know old SuSE distros had a flag for that in rc.config. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 14:50 ` Stephan von Krawczynski 2003-08-04 20:03 ` Olivier Galibert 2003-08-04 21:16 ` Jesse Pollard @ 2003-08-04 22:58 ` Andrew Pimlott 2003-08-05 0:19 ` Stephan von Krawczynski 2 siblings, 1 reply; 79+ messages in thread From: Andrew Pimlott @ 2003-08-04 22:58 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Anton Altaparmakov, aebr, linux-kernel On Mon, Aug 04, 2003 at 04:50:02PM +0200, Stephan von Krawczynski wrote: > There is a flaw in this argument. If I am told that mount --bind > does just about what I want to have as a feature then these > applictions must have the same problems already (if I mount > braindead). So an implementation in fs cannot do any _additional_ > damage to these applications, or not? There is a flaw in this flaw. :-) /tmp# mkdir a /tmp# mkdir a/b /tmp# mkdir a/c /tmp# mount --bind a a/b /tmp# ls a b c /tmp# ls a/b b c /tmp# ls a/b/b/ /tmp# It is enlightening in this regard to consider the difference between using unix /etc/fstab and Hurd translators to manage your namespace. In preparing this example, I discovered that find and ls -R already have hard-link cycle "protection" built in, so they are broken in the presence of bind mounts. :-( Andrew ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 22:58 ` Andrew Pimlott @ 2003-08-05 0:19 ` Stephan von Krawczynski 2003-08-05 1:18 ` Andrew Pimlott 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 0:19 UTC (permalink / raw) To: Andrew Pimlott; +Cc: aia21, aebr, linux-kernel On Mon, 4 Aug 2003 18:58:19 -0400 Andrew Pimlott <andrew@pimlott.net> wrote: > On Mon, Aug 04, 2003 at 04:50:02PM +0200, Stephan von Krawczynski wrote: > > There is a flaw in this argument. If I am told that mount --bind > > does just about what I want to have as a feature then these > > applictions must have the same problems already (if I mount > > braindead). So an implementation in fs cannot do any _additional_ > > damage to these applications, or not? > > There is a flaw in this flaw. :-) > > /tmp# mkdir a > /tmp# mkdir a/b > /tmp# mkdir a/c > /tmp# mount --bind a a/b > /tmp# ls a > b c > /tmp# ls a/b > b c > /tmp# ls a/b/b/ > /tmp# > > It is enlightening in this regard to consider the difference between > using unix /etc/fstab and Hurd translators to manage your namespace. > > In preparing this example, I discovered that find and ls -R already > have hard-link cycle "protection" built in, so they are broken in > the presence of bind mounts. :-( Ok, so now we are at: application programmer expected hardlinks to exist, but fs programmer says they won't because they break existing applications. Now the discussion gets real interesting ;-) Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 0:19 ` Stephan von Krawczynski @ 2003-08-05 1:18 ` Andrew Pimlott 2003-08-05 8:04 ` Stephan von Krawczynski 0 siblings, 1 reply; 79+ messages in thread From: Andrew Pimlott @ 2003-08-05 1:18 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: aia21, aebr, linux-kernel On Tue, Aug 05, 2003 at 02:19:38AM +0200, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 18:58:19 -0400 > Andrew Pimlott <andrew@pimlott.net> wrote: > > > > In preparing this example, I discovered that find and ls -R already > > have hard-link cycle "protection" built in, so they are broken in > > the presence of bind mounts. :-( > > Ok, so now we are at: application programmer expected hardlinks to exist, but > fs programmer says they won't because they break existing applications. I would say that a few venerable programs, that have seen wide use on old broken versions of unix, contain workarounds for this problem. I'm sure that most applications don't contain such workarounds. And it is expensive, so I would certainly prefer that my find and ls didn't do this. > Now the discussion gets real interesting ;-) Wouldn't bind mounts solve your problem? Why are you still interested in hard links? Andrew ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 1:18 ` Andrew Pimlott @ 2003-08-05 8:04 ` Stephan von Krawczynski 2003-08-05 11:18 ` Wakko Warner 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 8:04 UTC (permalink / raw) To: Andrew Pimlott; +Cc: aia21, aebr, linux-kernel On Mon, 4 Aug 2003 21:18:20 -0400 Andrew Pimlott <andrew@pimlott.net> wrote: > On Tue, Aug 05, 2003 at 02:19:38AM +0200, Stephan von Krawczynski wrote: > Wouldn't bind mounts solve your problem? Why are you still > interested in hard links? Because mount -bind cannot be exported over nfs (to my current knowledge and testing). Else I would be very pleased to use it. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 8:04 ` Stephan von Krawczynski @ 2003-08-05 11:18 ` Wakko Warner 0 siblings, 0 replies; 79+ messages in thread From: Wakko Warner @ 2003-08-05 11:18 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Andrew Pimlott, aia21, aebr, linux-kernel > > Wouldn't bind mounts solve your problem? Why are you still > > interested in hard links? > > Because mount -bind cannot be exported over nfs (to my current knowledge and > testing). Else I would be very pleased to use it. The userspace NFS server may support this. Unfortunately, it only supports NFSv2. -- Lab tests show that use of micro$oft causes cancer in lab animals ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 13:56 ` Stephan von Krawczynski 2003-08-04 14:04 ` Anton Altaparmakov @ 2003-08-04 14:33 ` Jesse Pollard 2003-08-04 15:05 ` Stephan von Krawczynski 2003-08-04 20:47 ` Jan Harkes 2 siblings, 1 reply; 79+ messages in thread From: Jesse Pollard @ 2003-08-04 14:33 UTC (permalink / raw) To: Stephan von Krawczynski, Andries Brouwer; +Cc: linux-kernel On Monday 04 August 2003 08:56, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 15:44:15 +0200 > > Andries Brouwer <aebr@win.tue.nl> wrote: > > On Mon, Aug 04, 2003 at 02:15:48PM +0200, Stephan von Krawczynski wrote: > > > although it is very likely I am entering (again :-) an ancient > > > discussion I would like to ask why hardlinks on directories are not > > > allowed/no supported fs action these days. > > > > Quite a lot of software thinks that the file hierarchy is a tree, > > if you wish a forest. > > > > Things would break badly if the hierarchy became an arbitrary graph. > > Care to name one? What exactly is the rule you see broken? Sure you can > build loops, but you cannot prevent people from doing braindamaged things > to their data anyway. You would not ban dd either for being able to flatten > your harddisk only because of one mistyping char... > Every feature can be misused and then damaging, but that is no real reason > not to have it - IMHO. Find for one. Any application that must scan the tree in a search. Any application that must backup every file for another (I know, dump bypasses the filesystem to make backups, tar doesn't). tar, cpio, rm fsck for another... ls for another.. (ls -R) The problem is backward links. If the forward link to a directory points to a directory above the entry for that link will cause a failure. Another problem is reverse links... for example: a/b/c and a/b/d are in the same directory b, (c and d are both directories) a/e/f is hardlinked to the directory d. current directory is in "f". How do you address "../c" ? I've used systems that allowed this (VMS was one, though it wasn't really documented). The filesystem repair utilitity insisted that "directory was not properly linked...", and any attempt to fix it caused the other link to show up as "not properly". It introduces too many unique problems to be easily handled. That is why symbolic links actually work. Symbolic links are not hard links, therefore they are not processed as part of the tree. and do not cause loops. It was also done in one of the "popular" code management systems under unix. (it allowed a "mount" of the system root to be under the CVS repository to detect unauthorized modifications...). Unfortunately, the system could not be backed up anymore. 1. A dump of the CVS filesystem turned into a dump of the entire system... 2. You could not restore the backups... The dumps failed (bru at the time) because the pathnames got too long, the restore failed since it ran out of disk space due to the multiple copies of the tree being created. The KIS principle is the key. A graph is NOT simple to maintain. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 14:33 ` Jesse Pollard @ 2003-08-04 15:05 ` Stephan von Krawczynski 2003-08-04 15:57 ` Richard B. Johnson ` (4 more replies) 0 siblings, 5 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 15:05 UTC (permalink / raw) To: Jesse Pollard; +Cc: aebr, linux-kernel On Mon, 4 Aug 2003 09:33:44 -0500 Jesse Pollard <jesse@cats-chateau.net> wrote: > Find for one. Any application that must scan the tree in a search. Any > application that must backup every file for another (I know, dump bypasses > the filesystem to make backups, tar doesn't). All that can handle symlinks already have the same problem nowadays. Where is the difference? And yet again: it is no _must_ for the feature to use it for creating complete loops inside your fs. You _can_ as well dd if=/dev/zero of=/dev/hda, but of course you shouldn't. Have you therefore deleted dd from your bin ? > It introduces too many unique problems to be easily handled. That is why > symbolic links actually work. Symbolic links are not hard links, therefore > they are not processed as part of the tree. and do not cause loops. tar --dereference loops on symlinks _today_, to name an example. All you have to do is to provide a way to find out if a directory is a hardlink, nothing more. And that should be easy. > It was also done in one of the "popular" code management systems under > unix. (it allowed a "mount" of the system root to be under the CVS > repository to detect unauthorized modifications...). Unfortunately, > the system could not be backed up anymore. 1. A dump of the CVS filesystem > turned into a dump of the entire system... 2. You could not restore the > backups... The dumps failed (bru at the time) because the pathnames got > too long, the restore failed since it ran out of disk space due to the > multiple copies of the tree being created. And they never heard of "--exclude" in tar, did they? > The KIS principle is the key. A graph is NOT simple to maintain. This is true. But I am very willing to believe reiserfs is not simple either, still it is there ;-) Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:05 ` Stephan von Krawczynski @ 2003-08-04 15:57 ` Richard B. Johnson 2003-08-04 21:23 ` Jesse Pollard 2003-08-04 16:11 ` Adam Sampson ` (3 subsequent siblings) 4 siblings, 1 reply; 79+ messages in thread From: Richard B. Johnson @ 2003-08-04 15:57 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Jesse Pollard, aebr, linux-kernel On Mon, 4 Aug 2003, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 09:33:44 -0500 > Jesse Pollard <jesse@cats-chateau.net> wrote: > > > Find for one. Any application that must scan the tree in a search. Any > > application that must backup every file for another (I know, dump bypasses > > the filesystem to make backups, tar doesn't). > > All that can handle symlinks already have the same problem nowadays. Where is > the difference? And yet again: it is no _must_ for the feature to use it for > creating complete loops inside your fs. > You _can_ as well dd if=/dev/zero of=/dev/hda, but of course you shouldn't. > Have you therefore deleted dd from your bin ? > > > It introduces too many unique problems to be easily handled. That is why > > symbolic links actually work. Symbolic links are not hard links, therefore > > they are not processed as part of the tree. and do not cause loops. > > tar --dereference loops on symlinks _today_, to name an example. > All you have to do is to provide a way to find out if a directory is a > hardlink, nothing more. And that should be easy. > [SNIPPED...] Reading Denis Howe's Free Online Dictionary of Computing; http://burks.bton.ac.uk/burks/foldoc/55/51.html, we see that the chief reason for no directory hard-links is that `rm` and `rmdir` won't allow you to delete them. There is no POSIX requirement for eliminating them, and it is possible "Some systems provide link and unlink commands which give direct access to the system calls of the same name, for which no such restrictions apply." Perhaps Linux does support hard-links to directories? mkdir("foo", 0644) = 0 link("foo", "bar") = -1 EPERM (Operation not permitted) _exit(0) = ? Nah... No such luck. I'll bet it's artificial. I think you could remove that S_IFDIR check and get away with it! Cheers, Dick Johnson Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips). Note 96.31% of all statistics are fiction. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:57 ` Richard B. Johnson @ 2003-08-04 21:23 ` Jesse Pollard 0 siblings, 0 replies; 79+ messages in thread From: Jesse Pollard @ 2003-08-04 21:23 UTC (permalink / raw) To: Richard B. Johnson, Stephan von Krawczynski; +Cc: aebr, linux-kernel On Monday 04 August 2003 10:57, Richard B. Johnson wrote: > On Mon, 4 Aug 2003, Stephan von Krawczynski wrote: > > On Mon, 4 Aug 2003 09:33:44 -0500 > > > > Jesse Pollard <jesse@cats-chateau.net> wrote: > > > Find for one. Any application that must scan the tree in a search. Any > > > application that must backup every file for another (I know, dump > > > bypasses the filesystem to make backups, tar doesn't). > > > > All that can handle symlinks already have the same problem nowadays. > > Where is the difference? And yet again: it is no _must_ for the feature > > to use it for creating complete loops inside your fs. > > You _can_ as well dd if=/dev/zero of=/dev/hda, but of course you > > shouldn't. Have you therefore deleted dd from your bin ? > > > > > It introduces too many unique problems to be easily handled. That is > > > why symbolic links actually work. Symbolic links are not hard links, > > > therefore they are not processed as part of the tree. and do not cause > > > loops. > > > > tar --dereference loops on symlinks _today_, to name an example. > > All you have to do is to provide a way to find out if a directory is a > > hardlink, nothing more. And that should be easy. > > [SNIPPED...] > > Reading Denis Howe's Free Online Dictionary of Computing; > http://burks.bton.ac.uk/burks/foldoc/55/51.html, we see that > the chief reason for no directory hard-links is that `rm` > and `rmdir` won't allow you to delete them. There is no > POSIX requirement for eliminating them, and it is possible > "Some systems provide link and unlink commands which give > direct access to the system calls of the same name, for > which no such restrictions apply." > > Perhaps Linux does support hard-links to directories? > > mkdir("foo", 0644) = 0 > link("foo", "bar") = -1 EPERM (Operation not > permitted) _exit(0) = ? > > Nah... No such luck. I'll bet it's artificial. I think you > could remove that S_IFDIR check and get away with it! Nope -- what you get is a corrupted filesystem.... Usually, a lost directory gets put in lost+found, othertimes you get a "corrupted inode", and if the inode is cleared you then find every file that was contained in the directory that used to be the inode, is now in lost+found. The only place I can think of that this might not happen are those filesystems that don't use the UNIX style tree structure. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:05 ` Stephan von Krawczynski 2003-08-04 15:57 ` Richard B. Johnson @ 2003-08-04 16:11 ` Adam Sampson 2003-08-04 17:00 ` Hans Reiser ` (2 subsequent siblings) 4 siblings, 0 replies; 79+ messages in thread From: Adam Sampson @ 2003-08-04 16:11 UTC (permalink / raw) To: linux-kernel Stephan von Krawczynski <skraw@ithnet.com> writes: > All that can handle symlinks already have the same problem > nowadays. Where is the difference? lstat() will tell you if a file is a symlink; if you only walk into directories, then you're guaranteed not to get into a loop. If you've got two hardlinks to a directory, how do you make that available in stat() output, and how does a tree-walking program know which to walk into? You could do the rsync trick of keeping track of every device-inode pair you've seen to detect hardlinks, but that's horribly non-space-efficient on large directories -- particularly bad for backups. I could imagine this functionality maybe being useful for system administrators, but with normal Unixish userspace, it doesn't strike me as a good idea to give users the ability to create hardlinks to directories. -- Adam Sampson <azz@us-lot.org> <http://offog.org/> ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:05 ` Stephan von Krawczynski 2003-08-04 15:57 ` Richard B. Johnson 2003-08-04 16:11 ` Adam Sampson @ 2003-08-04 17:00 ` Hans Reiser 2003-08-04 17:18 ` Sean Neakums 2003-08-05 4:53 ` jw schultz 2003-08-04 18:50 ` jlnance 2003-08-04 21:09 ` Jesse Pollard 4 siblings, 2 replies; 79+ messages in thread From: Hans Reiser @ 2003-08-04 17:00 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Jesse Pollard, aebr, linux-kernel, god, Vitaly Stephan von Krawczynski wrote: > > >>The KIS principle is the key. A graph is NOT simple to maintain. >> >> > >This is true. But I am very willing to believe reiserfs is not simple either, >still it is there ;-) > >Regards, >Stephan >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > If you want hard linked directories, send us a patch for v4. Should be VERY easy to write. If there is some reason it is not simple, let me know. Discuss it with Vitaly though, it might affect fsck. -- Hans ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 17:00 ` Hans Reiser @ 2003-08-04 17:18 ` Sean Neakums 2003-08-05 4:53 ` jw schultz 1 sibling, 0 replies; 79+ messages in thread From: Sean Neakums @ 2003-08-04 17:18 UTC (permalink / raw) To: Hans Reiser Cc: Stephan von Krawczynski, Jesse Pollard, aebr, linux-kernel, god, Vitaly Hans Reiser <reiser@namesys.com> writes: > If you want hard linked directories, send us a patch for v4. Should > be VERY easy to write. If there is some reason it is not simple, let > me know. Discuss it with Vitaly though, it might affect fsck. The commentary above fs/namei.c:vfs_rename_dir indicates that the locking is already pretty tricky, and it seems to assume that a directory can have only one parent. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 17:00 ` Hans Reiser 2003-08-04 17:18 ` Sean Neakums @ 2003-08-05 4:53 ` jw schultz 1 sibling, 0 replies; 79+ messages in thread From: jw schultz @ 2003-08-05 4:53 UTC (permalink / raw) To: linux-kernel On Mon, Aug 04, 2003 at 09:00:53PM +0400, Hans Reiser wrote: > If you want hard linked directories, send us a patch for v4. Should be > VERY easy to write. If there is some reason it is not simple, let me > know. Discuss it with Vitaly though, it might affect fsck. I don't recommend it but if you do make sure those links can only be made by root. SVR3 and earlier allowed manual hardlinking of directories by root only. They were a real source of problems. It also confused the dickens out of fsck so it would have to be restricted or allowed by the filesystem code, not the VFS layer. I remember playing with it and it was a guarantee that fsck would have to be run manually. $mkdir A $mkdir B $mkdir C $mkdir A/A1 $ln A/A1 B/B1 $ln A/A1 C/C1 $rmdir A/A1 Assuming we can do this. A1 is an empty directory after all. Now B1 has a link count of 1, but i'll assume that is OK $rmdir A It is after all empty even though the link count is 3. $cd B/B1 $/bin/pwd cannot stat . Remember B/B1/.. is it A with a nlinks==1 $cd .. Now where are you? It used to be called A but now it has no normal path but can be reached through B/B1/.. It still has .. so what directory is it linked to that doesn't have an entry pointing back to it. Lets some fun with it. $mkdir A2 Ah, now we have a directory the path to which is B/B1/../A2 We can hide all sorts of stuff here and find will never see it. It won't get backed up but maybe that doesn't matter. What a lovely way to hide a rootkit. If on the other hand you removed A/A1/.. when removing A/A1 you have a B/B1 and C/C1 without .. Now umount the filesystem and run fsck. B/B1 and C/C1 each refer to the same directory with no .. or a .. that points to a third directory with nlinks==1 and no directory entries except a .. You can put in a slew of logic to reduce the risks somewhat. Things like only allowing rmdir somedir when somedir->nlinks <= 2 || somedir/.. != . but that still leaves the issue of where .. is linked and the potential to bypass parent directory based access controls such as Linus likes in a way that the admin will have a hard time identifying. The issues on a filesystem where .. is simulated would i imagine be different. -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:05 ` Stephan von Krawczynski ` (2 preceding siblings ...) 2003-08-04 17:00 ` Hans Reiser @ 2003-08-04 18:50 ` jlnance 2003-08-04 21:09 ` Jesse Pollard 4 siblings, 0 replies; 79+ messages in thread From: jlnance @ 2003-08-04 18:50 UTC (permalink / raw) To: linux-kernel On Mon, Aug 04, 2003 at 05:05:06PM +0200, Stephan von Krawczynski wrote: > tar --dereference loops on symlinks _today_, to name an example. > All you have to do is to provide a way to find out if a directory is a > hardlink, nothing more. And that should be easy. Actually I think that is the crux of the problem. If I do: touch a ln a b then a and b are both links to the same file. It is not like a is the real file and b is a link to it. The situation is indistinguishable from: touch b ln b a When you say that you want hardlinks to work on directories I believe people are making the assumption that you want this to hold true for directories as well. The difference between hardlinking directories and using symlinks or mount --bind is that there IS a difference between a directory and a link to the directory. Thanks, Jim ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:05 ` Stephan von Krawczynski ` (3 preceding siblings ...) 2003-08-04 18:50 ` jlnance @ 2003-08-04 21:09 ` Jesse Pollard 2003-08-04 22:13 ` Stephan von Krawczynski 2003-08-04 22:32 ` Stephan von Krawczynski 4 siblings, 2 replies; 79+ messages in thread From: Jesse Pollard @ 2003-08-04 21:09 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: aebr, linux-kernel On Monday 04 August 2003 10:05, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 09:33:44 -0500 > > Jesse Pollard <jesse@cats-chateau.net> wrote: > > Find for one. Any application that must scan the tree in a search. Any > > application that must backup every file for another (I know, dump > > bypasses the filesystem to make backups, tar doesn't). > > All that can handle symlinks already have the same problem nowadays. Where > is the difference? And yet again: it is no _must_ for the feature to use it > for creating complete loops inside your fs. > You _can_ as well dd if=/dev/zero of=/dev/hda, but of course you shouldn't. > Have you therefore deleted dd from your bin ? The difference is "SYMLINK". That is not a hard link. The are file with a mode bit that identifies them as a symlink. The contents of the file is the reference to the real file. A symlink is a file with its' own inode number. It may point to files on any filesystem (or none actually). > > > It introduces too many unique problems to be easily handled. That is why > > symbolic links actually work. Symbolic links are not hard links, > > therefore they are not processed as part of the tree. and do not cause > > loops. > > tar --dereference loops on symlinks _today_, to name an example. > All you have to do is to provide a way to find out if a directory is a > hardlink, nothing more. And that should be easy. Yup - because a symlink is NOT a hard link. it is a file. If you use hard links then there is no way to determine which is the "proper" link. > > > It was also done in one of the "popular" code management systems under > > unix. (it allowed a "mount" of the system root to be under the CVS > > repository to detect unauthorized modifications...). Unfortunately, > > the system could not be backed up anymore. 1. A dump of the CVS > > filesystem turned into a dump of the entire system... 2. You could not > > restore the backups... The dumps failed (bru at the time) because the > > pathnames got too long, the restore failed since it ran out of disk space > > due to the multiple copies of the tree being created. > > And they never heard of "--exclude" in tar, did they? Doesn't work. Remember - you have to --exclude EVERY possible loop. And unless you know ahead of time, you can't exclude it. The only way we found to reliably do the backup was to dismount the CVS. > > The KIS principle is the key. A graph is NOT simple to maintain. > > This is true. But I am very willing to believe reiserfs is not simple > either, still it is there ;-) The filesystem structure IS simple. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 21:09 ` Jesse Pollard @ 2003-08-04 22:13 ` Stephan von Krawczynski 2003-08-04 22:32 ` Stephan von Krawczynski 1 sibling, 0 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 22:13 UTC (permalink / raw) To: Jesse Pollard; +Cc: aebr, linux-kernel On Mon, 4 Aug 2003 16:09:28 -0500 Jesse Pollard <jesse@cats-chateau.net> wrote: > > > It was also done in one of the "popular" code management systems under > > > unix. (it allowed a "mount" of the system root to be under the CVS > > > repository to detect unauthorized modifications...). Unfortunately, > > > the system could not be backed up anymore. 1. A dump of the CVS > > > filesystem turned into a dump of the entire system... 2. You could not > > > restore the backups... The dumps failed (bru at the time) because the > > > pathnames got too long, the restore failed since it ran out of disk space > > > due to the multiple copies of the tree being created. > > > > And they never heard of "--exclude" in tar, did they? > > Doesn't work. Remember - you have to --exclude EVERY possible loop. And > unless you know ahead of time, you can't exclude it. The only way we found > to reliably do the backup was to dismount the CVS. I don't know the system, so I can only judge from your description: if they allow mount of something insive the cvs tree, then you have a mountpoint and can --exclude=mountpoint. You don't run into the system root and are fine, right? If you don't have known fixed mountpoints use mount -l to find out. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 21:09 ` Jesse Pollard 2003-08-04 22:13 ` Stephan von Krawczynski @ 2003-08-04 22:32 ` Stephan von Krawczynski 2003-08-04 23:00 ` Randolph Bentson ` (2 more replies) 1 sibling, 3 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 22:32 UTC (permalink / raw) To: Jesse Pollard; +Cc: aebr, linux-kernel On Mon, 4 Aug 2003 16:09:28 -0500 Jesse Pollard <jesse@cats-chateau.net> wrote: > > tar --dereference loops on symlinks _today_, to name an example. > > All you have to do is to provide a way to find out if a directory is a > > hardlink, nothing more. And that should be easy. > > Yup - because a symlink is NOT a hard link. it is a file. > > If you use hard links then there is no way to determine which is the "proper" > link. Where does it say that an fs is not allowed to give you this information in some way? > > > > > > It was also done in one of the "popular" code management systems under > > > unix. (it allowed a "mount" of the system root to be under the CVS > > > repository to detect unauthorized modifications...). Unfortunately, > > > the system could not be backed up anymore. 1. A dump of the CVS > > > filesystem turned into a dump of the entire system... 2. You could not > > > restore the backups... The dumps failed (bru at the time) because the > > > pathnames got too long, the restore failed since it ran out of disk space > > > due to the multiple copies of the tree being created. > > > > And they never heard of "--exclude" in tar, did they? > > Doesn't work. Remember - you have to --exclude EVERY possible loop. And > unless you know ahead of time, you can't exclude it. The only way we found > to reliably do the backup was to dismount the CVS. And if you completely run out of ideas in your wild-mounts-throughout-the-tree problem you should simply use "tar -l". And in just the same way fs could provide a mode bit saying "hi, I am a hardlink", and tar can then easily provide option number 1345 saying "stay away from hardlinks". If you can do the fs patch, I can do the tar patch. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 22:32 ` Stephan von Krawczynski @ 2003-08-04 23:00 ` Randolph Bentson 2003-08-05 0:10 ` Stephan von Krawczynski 2003-08-05 12:51 ` Helge Hafting 2003-08-05 14:12 ` Jesse Pollard 2 siblings, 1 reply; 79+ messages in thread From: Randolph Bentson @ 2003-08-04 23:00 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Jesse Pollard, aebr, linux-kernel On Tue, Aug 05, 2003 at 12:32:10AM +0200, Stephan von Krawczynski wrote: > And in just the same way fs could provide a mode bit saying "hi, I am a > hardlink", and tar can then easily provide option number 1345 saying > "stay away from hardlinks". Perhaps not a bit, but rather another enumerated value in the type field of an inode. (Are there any values left?) Ok, lets consider this. Suppose that /a/b and /a/c both refer to the same directory, where /a/b is a traditional link, but /a/c is a "hardlink". What happens when one executes 'rmdir /a/b'? Does the directory it references disappear? If not, how would tar ever find it? (I have this vision of a disk full of these hardlink-only directories which tar and perhaps fsck cannot find.) -- Randolph Bentson bentson@holmsjoen.com ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 23:00 ` Randolph Bentson @ 2003-08-05 0:10 ` Stephan von Krawczynski 2003-08-05 2:09 ` Edgar Toernig 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 0:10 UTC (permalink / raw) To: Randolph Bentson; +Cc: jesse, aebr, linux-kernel On Mon, 4 Aug 2003 16:00:09 -0700 Randolph Bentson <bentson@holmsjoen.com> wrote: > On Tue, Aug 05, 2003 at 12:32:10AM +0200, Stephan von Krawczynski wrote: > > And in just the same way fs could provide a mode bit saying "hi, I am a > > hardlink", and tar can then easily provide option number 1345 saying > > "stay away from hardlinks". > > Perhaps not a bit, but rather another enumerated value in the type field > of an inode. (Are there any values left?) > > Ok, lets consider this. Suppose that /a/b and /a/c both refer to the same > directory, where /a/b is a traditional link, but /a/c is a "hardlink". > > What happens when one executes 'rmdir /a/b'? Does the directory it > references disappear? If not, how would tar ever find it? (I have > this vision of a disk full of these hardlink-only directories which > tar and perhaps fsck cannot find.) The setup you describe is exactly the reason why I suggested elsewhere (in a private discussion) to single-link all directory entries pointing to the same directory in a list. In case of deletion of the "main" entry, the "main" simply can walk on to the next (former) hardlink, if there are any left the tree is deleted completely. That's it. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 0:10 ` Stephan von Krawczynski @ 2003-08-05 2:09 ` Edgar Toernig 2003-08-05 8:05 ` Stephan von Krawczynski 0 siblings, 1 reply; 79+ messages in thread From: Edgar Toernig @ 2003-08-05 2:09 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Randolph Bentson, jesse, aebr, linux-kernel Stephan von Krawczynski wrote: > > The setup you describe is exactly the reason why I suggested elsewhere (in a > private discussion) to single-link all directory entries pointing to the same > directory in a list. In case of deletion of the "main" entry, the "main" simply > can walk on to the next (former) hardlink, if there are any left the tree is > deleted completely. That's it. Amiga-FFS challenged? ;-) ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 2:09 ` Edgar Toernig @ 2003-08-05 8:05 ` Stephan von Krawczynski 0 siblings, 0 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 8:05 UTC (permalink / raw) To: Edgar Toernig; +Cc: bentson, jesse, aebr, linux-kernel On Tue, 05 Aug 2003 04:09:46 +0200 Edgar Toernig <froese@gmx.de> wrote: > Stephan von Krawczynski wrote: > > > > The setup you describe is exactly the reason why I suggested elsewhere (in a > > private discussion) to single-link all directory entries pointing to the same > > directory in a list. In case of deletion of the "main" entry, the "main" simply > > can walk on to the next (former) hardlink, if there are any left the tree is > > deleted completely. That's it. > > Amiga-FFS challenged? ;-) Ah, he knows the roots ;-) Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 22:32 ` Stephan von Krawczynski 2003-08-04 23:00 ` Randolph Bentson @ 2003-08-05 12:51 ` Helge Hafting 2003-08-05 13:03 ` Stephan von Krawczynski 2003-08-05 14:12 ` Jesse Pollard 2 siblings, 1 reply; 79+ messages in thread From: Helge Hafting @ 2003-08-05 12:51 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: linux-kernel Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 16:09:28 -0500 > Jesse Pollard <jesse@cats-chateau.net> wrote: > > >>>tar --dereference loops on symlinks _today_, to name an example. >>>All you have to do is to provide a way to find out if a directory is a >>>hardlink, nothing more. And that should be easy. >> >>Yup - because a symlink is NOT a hard link. it is a file. >> >>If you use hard links then there is no way to determine which is the "proper" >>link. > > > Where does it say that an fs is not allowed to give you this information in > some way? Because there is no such thing as the "proper" link. Every filename and directoryname is a "hard link" to some inode. The "ln" command lets you make more links to the same inode, there is _no_ difference at all between the "first" and the "second" (or third) hard link. There is no information recorded anywhere about which one where "first". You can remove the "first" and still use the file through the second link. > >>>>It was also done in one of the "popular" code management systems under >>>>unix. (it allowed a "mount" of the system root to be under the CVS >>>>repository to detect unauthorized modifications...). Unfortunately, >>>>the system could not be backed up anymore. 1. A dump of the CVS >>>>filesystem turned into a dump of the entire system... 2. You could not >>>>restore the backups... The dumps failed (bru at the time) because the >>>>pathnames got too long, the restore failed since it ran out of disk space >>>>due to the multiple copies of the tree being created. >>> >>>And they never heard of "--exclude" in tar, did they? >> >>Doesn't work. Remember - you have to --exclude EVERY possible loop. And >>unless you know ahead of time, you can't exclude it. The only way we found >>to reliably do the backup was to dismount the CVS. > > > And if you completely run out of ideas in your wild-mounts-throughout-the-tree > problem you should simply use "tar -l". > > And in just the same way fs could provide a mode bit saying "hi, I am a > hardlink", and tar can then easily provide option number 1345 saying "stay away > from hardlinks". > Then you stay away from each and every file on the system, because every file is a hard link. This is useless. Making up a new bit in the directory saying "this directory entry was created with 'ln' as opposed to 'open'" is indeed possible, but wouldn't solve your problems. Consider this: A file is created in the normal way by a user. (link count=1) Someone else finds it useful creates a link to it. (link count=2) The first user don't need the file anymore and deletes it. (link count=1) The file still exist, because disk blocks are only released when the link count goes to zero. The second user can still use the file through his link, and think it is safe. But the file is no longer backed up because your tar avoids the directory entry created with 'ln', and there is no longer any directory entry created the normal way. Similiar problems arise for all other utilities you might want to modify to use this sort of linking flags. The problem don't aries with symlinks because the file really disappear when deleted, leaving invalid symlinks. (Everybody knows that and don't think they can keep a file by making a symlink, as you can with a hardlink) The number of hard links to some inode is a reference count, the number of symlinks is not. Another post of yours: > tar --dereference loops on symlinks _today_, to name an example. > All you have to do is to provide a way to find out if a directory is a > hardlink, nothing more. And that should be easy. There is currently no way to find out, as explained above. And a "flag" solution is useless, as you then will get files (and directories) that exist only as links when the "original" link is deleted. Another post: >> Things would break badly if the hierarchy became an arbitrary graph. > Care to name one? What exactly is the rule you see broken? Sure you can build > loops, Loops are funny in more than one way. Tools can be made that detect them via inode numbers, and handle them properly. This can be costly in time and memory, but backing up or otherwise traversing a generic graph is possible. Even more fun is when you have a directory loop like this: mkdir A cd A mkdir B cd B make hard link C back to A cd ../.. rmdir A You now removed A from your home directory, but the directory itself did not disappear because it had another hard link from C in B. Expected behaviour, but there is no longer a path from _anywhere_ to the B and C directories. That means they occupy disk space but cannot be deleted, accessed or used in any way short of garbage collecting the entire directory structure. That can be a big job. The space usage can be big too, for the inaccessible B or C directories might hold some large files. This isn't easily avoidable by "not doing stupid things" either, interactions between several users who don't know what the others do can easily form loops by linking and moving some directories around. Helge Hafting ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 12:51 ` Helge Hafting @ 2003-08-05 13:03 ` Stephan von Krawczynski 2003-08-05 13:13 ` Bernd Petrovitsch ` (2 more replies) 0 siblings, 3 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 13:03 UTC (permalink / raw) To: Helge Hafting; +Cc: linux-kernel On Tue, 05 Aug 2003 14:51:46 +0200 Helge Hafting <helgehaf@aitel.hist.no> wrote: > Even more fun is when you have a directory loop like this: > > mkdir A > cd A > mkdir B > cd B > make hard link C back to A > > cd ../.. > rmdir A > > You now removed A from your home directory, but the > directory itself did not disappear because it had > another hard link from C in B. How about a truly simple idea: rmdir A says "directory in use" and is rejected Which means you simply cannot remove the first directory entry before not all other links to it are removed. This implies only two things: 1) you have to know who was first. 2) you have to be able to find out where the links are. Both sound solvable. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 13:03 ` Stephan von Krawczynski @ 2003-08-05 13:13 ` Bernd Petrovitsch 2003-08-05 13:39 ` Stephan von Krawczynski 2003-08-05 13:36 ` Richard B. Johnson 2003-08-05 22:08 ` Helge Hafting 2 siblings, 1 reply; 79+ messages in thread From: Bernd Petrovitsch @ 2003-08-05 13:13 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: linux-kernel On Die, 2003-08-05 at 15:03, Stephan von Krawczynski wrote: [...] > Both sound solvable. We are waiting for your first patch. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 13:13 ` Bernd Petrovitsch @ 2003-08-05 13:39 ` Stephan von Krawczynski 0 siblings, 0 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 13:39 UTC (permalink / raw) To: Bernd Petrovitsch; +Cc: linux-kernel On 05 Aug 2003 15:13:48 +0200 Bernd Petrovitsch <bernd@firmix.at> wrote: > We are waiting for your first patch. You might have missed it somewhen in the late last century ;-) Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 13:03 ` Stephan von Krawczynski 2003-08-05 13:13 ` Bernd Petrovitsch @ 2003-08-05 13:36 ` Richard B. Johnson 2003-08-05 14:04 ` Stephan von Krawczynski 2003-08-05 14:56 ` Jesse Pollard 2003-08-05 22:08 ` Helge Hafting 2 siblings, 2 replies; 79+ messages in thread From: Richard B. Johnson @ 2003-08-05 13:36 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Helge Hafting, linux-kernel On Tue, 5 Aug 2003, Stephan von Krawczynski wrote: > On Tue, 05 Aug 2003 14:51:46 +0200 > Helge Hafting <helgehaf@aitel.hist.no> wrote: > > > Even more fun is when you have a directory loop like this: > > > > mkdir A > > cd A > > mkdir B > > cd B > > make hard link C back to A > > > > cd ../.. > > rmdir A > > > > You now removed A from your home directory, but the > > directory itself did not disappear because it had > > another hard link from C in B. > > How about a truly simple idea: > > rmdir A says "directory in use" and is rejected > > Which means you simply cannot remove the first directory entry before not all > other links to it are removed. This implies only two things: > 1) you have to know who was first. > 2) you have to be able to find out where the links are. > > Both sound solvable. > > Regards, > Stephan > A hard-link is, by definition, indistinguishable from the original entry. In fact, with fast machines and the course granularity of file-system times, even the creation time may be exactly the same. Without making one of the files different, and therefore not a hard-link, there is no way for an operating system to distinguish them except by keeping some list of hard-link inodes in some container file or some other such atrocious misconduct. Then every significant file operation would require a search of this inode-list to determine the special handling of such hard-linked directories. A directory hard-link to the same directory will cause any recursive directory look-up to fail. It doesn't fail with sim-links because they are specifically different and can be excluded from the infinite recursion syndrome that would exist otherwise. Somebody mentioned that VAX/VMS would allow hard-linked directories. This is not true. Even files, created with SET FILE ENTER=DUA0:[1.DIR] DUA0:[000000] that seemed like hard links were not hard links. They were sim-links. The file entry contained the directory information of the file being linked, and was not a clone of the header (inode) itself as would be a hard link. Therefore, these were unique and could not cause recursion to fail. Cheers, Dick Johnson Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips). Note 96.31% of all statistics are fiction. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 13:36 ` Richard B. Johnson @ 2003-08-05 14:04 ` Stephan von Krawczynski 2003-08-05 14:57 ` Richard B. Johnson 2003-08-05 15:02 ` Jesse Pollard 2003-08-05 14:56 ` Jesse Pollard 1 sibling, 2 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 14:04 UTC (permalink / raw) To: root; +Cc: helgehaf, linux-kernel On Tue, 5 Aug 2003 09:36:37 -0400 (EDT) "Richard B. Johnson" <root@chaos.analogic.com> wrote: > A hard-link is, by definition, indistinguishable from the original > entry. In fact, with fast machines and the course granularity of > file-system times, even the creation time may be exactly the > same. Hello Richard, I really don't mind if you call the thing I am looking for a hardlink or a chicken. And I am really not sticking to creating them by ln or mount or just about anything else. I am, too, not bound to making them permanent on the media. All I really want to do is to _export_ them via nfs. And guys, looking at mount -bind makes me think someone else (before poor me) needed just about the same thing. So, instead of constantly feeding my bad conscience, can some kind soul explain the possibilities to make "mount -bind/rbind" work over a network fs of some flavor, please? Regards, Stephan PS: if you ever want to find out what *nix people are carrying guns, just enter the room and cry out loud "directory hardlinks to the left!" ;-) ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 14:04 ` Stephan von Krawczynski @ 2003-08-05 14:57 ` Richard B. Johnson 2003-08-05 15:08 ` Stephan von Krawczynski 2003-08-05 15:02 ` Jesse Pollard 1 sibling, 1 reply; 79+ messages in thread From: Richard B. Johnson @ 2003-08-05 14:57 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: helgehaf, linux-kernel On Tue, 5 Aug 2003, Stephan von Krawczynski wrote: > On Tue, 5 Aug 2003 09:36:37 -0400 (EDT) > "Richard B. Johnson" <root@chaos.analogic.com> wrote: > > > A hard-link is, by definition, indistinguishable from the original > > entry. In fact, with fast machines and the course granularity of > > file-system times, even the creation time may be exactly the > > same. > > Hello Richard, > > I really don't mind if you call the thing I am looking for a hardlink or a > chicken. And I am really not sticking to creating them by ln or mount or just > about anything else. I am, too, not bound to making them permanent on the > media. All I really want to do is to _export_ them via nfs. > And guys, looking at mount -bind makes me think someone else (before poor me) > needed just about the same thing. > So, instead of constantly feeding my bad conscience, can some kind soul explain > the possibilities to make "mount -bind/rbind" work over a network fs of some > flavor, please? > > Regards, > Stephan > > PS: if you ever want to find out what *nix people are carrying guns, just enter > the room and cry out loud "directory hardlinks to the left!" > ;-) > But symlinks work over NFS. You just have to make sure they are relative to whatever the remote mount-point is: Script started on Tue Aug 5 10:38:55 2003 # mount boneserver:/tmp /mnt # df Filesystem 1k-blocks Used Available Use% Mounted on /dev/sdb1 16603376 6560676 9199292 42% / /dev/sdc1 6356624 1217040 4816680 20% /alt /dev/sdc3 2253284 1796788 342036 84% /home/users /dev/sda1 1048272 280960 767312 27% /dos/drive_C /dev/sda5 1046224 181280 864944 17% /dos/drive_D boneserver:/tmp 3881192 2385676 1294706 65% /mnt # cd /mnt # ls # mkdir foo # ls foo # ln -s foo bar # ls bar foo # cd bar # pwd /mnt/bar # mkdir xxx # ln -s xxx ../zzz # cd .. # ls bar foo zzz # file zzz zzz: broken symbolic link to xxx # rm zzz # cd bar # ls xxx # file xxx xxx: directory # pwd /mnt/bar # ln -s /mnt/bar/xxx ../zzz # cd .. # ls bar foo zzz # file zzz zzz: symbolic link to /mnt/bar/xxx # ls zzz # home # umount /mnt # exit exit Script done on Tue Aug 5 10:41:41 2003 As you an clearly see, the symlink to the directories worked fine. You don't need a hard-link at all. I deliberately created a broken link, deleted it, then made another that works. Everything I did locally could have been done remotely on the server who's /tmp directory I mounted R/W. All you need to do to make such working links is to use the same mount-point name on each of you clients. That way, a sim-link will work the same for everybody. Cheers, Dick Johnson Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips). Note 96.31% of all statistics are fiction. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 14:57 ` Richard B. Johnson @ 2003-08-05 15:08 ` Stephan von Krawczynski 0 siblings, 0 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 15:08 UTC (permalink / raw) To: root; +Cc: helgehaf, linux-kernel On Tue, 5 Aug 2003 10:57:11 -0400 (EDT) "Richard B. Johnson" <root@chaos.analogic.com> wrote: > But symlinks work over NFS. You just have to make sure they are > relative to whatever the remote mount-point is: Yes, I know this works. But honestly I'd say that this is a very ugly thing. That's why I want to get rid of it completely (using it currently). The straight forward method for linking/remounting stuff is following the links (or whatever) on the fileserver and not on the client (like with symlinks). Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 14:04 ` Stephan von Krawczynski 2003-08-05 14:57 ` Richard B. Johnson @ 2003-08-05 15:02 ` Jesse Pollard 2003-08-05 15:12 ` Stephan von Krawczynski 2003-08-05 15:44 ` Trond Myklebust 1 sibling, 2 replies; 79+ messages in thread From: Jesse Pollard @ 2003-08-05 15:02 UTC (permalink / raw) To: Stephan von Krawczynski, root; +Cc: helgehaf, linux-kernel On Tuesday 05 August 2003 09:04, Stephan von Krawczynski wrote: > On Tue, 5 Aug 2003 09:36:37 -0400 (EDT) > > "Richard B. Johnson" <root@chaos.analogic.com> wrote: > > A hard-link is, by definition, indistinguishable from the original > > entry. In fact, with fast machines and the course granularity of > > file-system times, even the creation time may be exactly the > > same. > > Hello Richard, > > I really don't mind if you call the thing I am looking for a hardlink or a > chicken. And I am really not sticking to creating them by ln or mount or > just about anything else. I am, too, not bound to making them permanent on > the media. All I really want to do is to _export_ them via nfs. > And guys, looking at mount -bind makes me think someone else (before poor > me) needed just about the same thing. > So, instead of constantly feeding my bad conscience, can some kind soul > explain the possibilities to make "mount -bind/rbind" work over a network > fs of some flavor, please? Not sure, but I suspect there would be a problem IF the -bind mount crosses filesystems. If it doesn't cross the filesystems I wouldn't think there would be much problem. You do have to remember that any NFS export gives IMPLICIT access to the entire filesystem (it is the device number that is actually exported). If the attacker can generate device:inode number, then that file reference can be opened. I haven't read/seen anything yet that has said different. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 15:02 ` Jesse Pollard @ 2003-08-05 15:12 ` Stephan von Krawczynski 2003-08-05 15:44 ` Trond Myklebust 1 sibling, 0 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 15:12 UTC (permalink / raw) To: Jesse Pollard; +Cc: root, helgehaf, linux-kernel On Tue, 5 Aug 2003 10:02:04 -0500 Jesse Pollard <jesse@cats-chateau.net> wrote: > On Tuesday 05 August 2003 09:04, Stephan von Krawczynski wrote: > > So, instead of constantly feeding my bad conscience, can some kind soul > > explain the possibilities to make "mount -bind/rbind" work over a network > > fs of some flavor, please? > > Not sure, but I suspect there would be a problem IF the -bind mount crosses > filesystems. If it doesn't cross the filesystems I wouldn't think there > would be much problem. Hm, that would be quite ok in my eyes. Obviously it would be nice if total re-export were possible, but as a first step within the same device sounds ok. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 15:02 ` Jesse Pollard 2003-08-05 15:12 ` Stephan von Krawczynski @ 2003-08-05 15:44 ` Trond Myklebust 1 sibling, 0 replies; 79+ messages in thread From: Trond Myklebust @ 2003-08-05 15:44 UTC (permalink / raw) To: Jesse Pollard; +Cc: Stephan von Krawczynski, root, helgehaf, linux-kernel >>>>> " " == Jesse Pollard <jesse@cats-chateau.net> writes: > You do have to remember that any NFS export gives IMPLICIT > access to the entire filesystem (it is the device number that > is actually exported). If the attacker can generate > device:inode number, then that file reference can be opened. I > haven't read/seen anything yet that has said different. Not entirely true. The default under Linux is to enable subtree checks. This means that knfsd must establish that the file being accessed lies within a subtree the head of which is the export point. This wreaks havoc with cross-directory renames etc, so a lot of people disable it, however it is a slightly safer default... Of course if you start playing with the idea of hard linked directories then subtree checks are no longer an option as the path connecting an export point and the file is no longer guaranteed to be unique (and some paths may not even be finite). Cheers, Trond ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 13:36 ` Richard B. Johnson 2003-08-05 14:04 ` Stephan von Krawczynski @ 2003-08-05 14:56 ` Jesse Pollard 1 sibling, 0 replies; 79+ messages in thread From: Jesse Pollard @ 2003-08-05 14:56 UTC (permalink / raw) To: Richard B. Johnson, Stephan von Krawczynski; +Cc: Helge Hafting, linux-kernel On Tuesday 05 August 2003 08:36, Richard B. Johnson wrote: [snip] > > Somebody mentioned that VAX/VMS would allow hard-linked > directories. This is not true. Even files, created with > SET FILE ENTER=DUA0:[1.DIR] DUA0:[000000] that seemed > like hard links were not hard links. They were sim-links. > The file entry contained the directory information of the > file being linked, and was not a clone of the header > (inode) itself as would be a hard link. Therefore, these > were unique and could not cause recursion to fail. Those created with "set file enter" are. Those created with "pip file file/link" are/were hard links. (at least I think I got the command right. There was a syntax to specified a file node directory, but the number,sequence syntax is forgotten). found this on VAX 8800s... Each boot directory was a hard link to the common boot directory. One hard link per cluster node entry. That eliminated the duplicated code, and still permitted unique boots during rolling upgrades. As well as permitted different architectures to use the cluster filesystem. It had to be a hard link because the boot code couldn't/didn't handle the others. the "analyze/disk" would find these as "incorrect parent" references, and would attempt to repair them by switching the parent directory entries. A subsequent check would report the other entry as incorrect... ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 13:03 ` Stephan von Krawczynski 2003-08-05 13:13 ` Bernd Petrovitsch 2003-08-05 13:36 ` Richard B. Johnson @ 2003-08-05 22:08 ` Helge Hafting 2003-08-24 17:35 ` Hans Reiser 2 siblings, 1 reply; 79+ messages in thread From: Helge Hafting @ 2003-08-05 22:08 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: linux-kernel On Tue, Aug 05, 2003 at 03:03:51PM +0200, Stephan von Krawczynski wrote: > On Tue, 05 Aug 2003 14:51:46 +0200 > Helge Hafting <helgehaf@aitel.hist.no> wrote: > > > Even more fun is when you have a directory loop like this: > > > > mkdir A > > cd A > > mkdir B > > cd B > > make hard link C back to A > > > > cd ../.. > > rmdir A > > > > You now removed A from your home directory, but the > > directory itself did not disappear because it had > > another hard link from C in B. > > How about a truly simple idea: > > rmdir A says "directory in use" and is rejected > Then anybody can prevent you from removing your obsolete directories by creating links to them. Existing hard link don't have such problems. > Which means you simply cannot remove the first directory entry before not all > other links to it are removed. This implies only two things: > 1) you have to know who was first. This requires fs modifications. I.e. a new fs > 2) you have to be able to find out where the links are. This is trivial but io intensive - by searching the entire directory tree. Doable in userspace, a nastry DOS opportunity if in the kernel. Another option is to store pointers to all directory entries in the inode, but that means much more complicated "mv", "rmdir" and "mkdir". Remember, it shouldn't merely "work" but also scale well on io-intensive SMP machines. Complicated operations tend to need more locking in case several processes (on different processors) try to modify the same directory simultaneously. > > Both sound solvable. > Anything is possible, but in this case - why bother? Do you have a specific use in mind, or is it just a "cool" thing? People have thought about directory links long before linux, tried it, and found other solutions for their tasks. Helge Hafting ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 22:08 ` Helge Hafting @ 2003-08-24 17:35 ` Hans Reiser 2003-08-24 19:02 ` Helge Hafting 2003-08-25 8:27 ` Nikita Danilov 0 siblings, 2 replies; 79+ messages in thread From: Hans Reiser @ 2003-08-24 17:35 UTC (permalink / raw) To: Helge Hafting Cc: Stephan von Krawczynski, linux-kernel, Reiserfs List, Nikita Danilov Helge Hafting wrote: >On Tue, Aug 05, 2003 at 03:03:51PM +0200, Stephan von Krawczynski wrote: > > >>On Tue, 05 Aug 2003 14:51:46 +0200 >>Helge Hafting <helgehaf@aitel.hist.no> wrote: >> >> >> >>>Even more fun is when you have a directory loop like this: >>> >>>mkdir A >>>cd A >>>mkdir B >>>cd B >>>make hard link C back to A >>> >>>cd ../.. >>>rmdir A >>> >>>You now removed A from your home directory, but the >>>directory itself did not disappear because it had >>>another hard link from C in B. >>> >>> >>How about a truly simple idea: >> >>rmdir A says "directory in use" and is rejected >> >> >> >Then anybody can prevent you from removing your obsolete directories >by creating links to them. Existing hard link don't have >such problems. > > So, he needs links that count as references, links that don't count as references but disappear if the object disappears (without dangling like symlinks), and unlinkall(), which removes an object and all of its links. He needs for the first reference to a directory to be removable only by removing all links to the object, or designating another link to be the "first" reference. Sounds clean to me. This is not to say that I am funded to write it.;-) I'd look at a patch though.....;-) I need to write up a taxonomy of links..... after reiser4 ships..... -- Hans ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-24 17:35 ` Hans Reiser @ 2003-08-24 19:02 ` Helge Hafting 2003-08-25 8:27 ` Nikita Danilov 1 sibling, 0 replies; 79+ messages in thread From: Helge Hafting @ 2003-08-24 19:02 UTC (permalink / raw) To: Hans Reiser Cc: Helge Hafting, Stephan von Krawczynski, linux-kernel, Reiserfs List, Nikita Danilov On Sun, Aug 24, 2003 at 09:35:57PM +0400, Hans Reiser wrote: > Helge Hafting wrote: > > > So, he needs links that count as references, Plain hard links. > links that don't count as > references but disappear if the object disappears (without dangling like > symlinks), Current filesystems would need a fs-wide search for soft links, but this is avoidable by storing a list of links in the inode. That must be done carefully though, or you'll get another problem: if I move a high-level directory, will I have to update links and inodes linked to that exists deep within some subsubsubdirectory? This had better be unnecessary, meaning the in-inode list of "reverse links" can't contain paths. Perhaps the can contain the inode numbers of the referencing directories. Deleting a file with this new kind of links will then cause searches in various directories containing those links. > and unlinkall(), which removes an object and all of its > links. He needs for the first reference to a directory to be removable > only by removing all links to the object, or designating another link to > be the "first" reference. > > Sounds clean to me. This is not to say that I am funded to write > it.;-) I'd look at a patch though.....;-) > > I need to write up a taxonomy of links..... after reiser4 ships..... > This stuff is possible if we get a new type of link then. I wonder how exisitng link-aware tools will cope. Helge Hafting ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-24 17:35 ` Hans Reiser 2003-08-24 19:02 ` Helge Hafting @ 2003-08-25 8:27 ` Nikita Danilov 2003-08-25 15:48 ` Hans Reiser 1 sibling, 1 reply; 79+ messages in thread From: Nikita Danilov @ 2003-08-25 8:27 UTC (permalink / raw) To: Hans Reiser Cc: Helge Hafting, Stephan von Krawczynski, linux-kernel, Reiserfs List Hans Reiser writes: > [...] > So, he needs links that count as references, links that don't count as > references but disappear if the object disappears (without dangling like > symlinks), and unlinkall(), which removes an object and all of its > links. He needs for the first reference to a directory to be removable > only by removing all links to the object, or designating another link to > be the "first" reference. > > Sounds clean to me. Will surely continue to be this way until you start implementing. :) > This is not to say that I am funded to write > it.;-) I'd look at a patch though.....;-) > > I need to write up a taxonomy of links..... after reiser4 ships..... http://www.namesys.com/v4/links-taxonomy.html > > -- > Hans > Nikita. > ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-25 8:27 ` Nikita Danilov @ 2003-08-25 15:48 ` Hans Reiser 0 siblings, 0 replies; 79+ messages in thread From: Hans Reiser @ 2003-08-25 15:48 UTC (permalink / raw) To: Nikita Danilov Cc: Helge Hafting, Stephan von Krawczynski, linux-kernel, Reiserfs List Nikita Danilov wrote: >Hans Reiser writes: > > > >[...] > > > So, he needs links that count as references, links that don't count as > > references but disappear if the object disappears (without dangling like > > symlinks), and unlinkall(), which removes an object and all of its > > links. He needs for the first reference to a directory to be removable > > only by removing all links to the object, or designating another link to > > be the "first" reference. > > > > Sounds clean to me. > >Will surely continue to be this way until you start implementing. :) > > > This is not to say that I am funded to write > > it.;-) I'd look at a patch though.....;-) > > > > I need to write up a taxonomy of links..... after reiser4 ships..... > >http://www.namesys.com/v4/links-taxonomy.html > > > > > -- > > Hans > > > >Nikita. > > > > > > > I meant one that is intelligible to the reader and comes with a detailed explanation.;-) Also, I am sure we need another round of seminars on it. -- Hans ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 22:32 ` Stephan von Krawczynski 2003-08-04 23:00 ` Randolph Bentson 2003-08-05 12:51 ` Helge Hafting @ 2003-08-05 14:12 ` Jesse Pollard 2003-08-05 14:21 ` Stephan von Krawczynski 2 siblings, 1 reply; 79+ messages in thread From: Jesse Pollard @ 2003-08-05 14:12 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: aebr, linux-kernel On Monday 04 August 2003 17:32, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 16:09:28 -0500 > > Jesse Pollard <jesse@cats-chateau.net> wrote: > > > tar --dereference loops on symlinks _today_, to name an example. > > > All you have to do is to provide a way to find out if a directory is a > > > hardlink, nothing more. And that should be easy. > > > > Yup - because a symlink is NOT a hard link. it is a file. > > > > If you use hard links then there is no way to determine which is the > > "proper" link. > > Where does it say that an fs is not allowed to give you this information in > some way? > > > > > It was also done in one of the "popular" code management systems > > > > under unix. (it allowed a "mount" of the system root to be under the > > > > CVS repository to detect unauthorized modifications...). > > > > Unfortunately, the system could not be backed up anymore. 1. A dump > > > > of the CVS filesystem turned into a dump of the entire system... 2. > > > > You could not restore the backups... The dumps failed (bru at the > > > > time) because the pathnames got too long, the restore failed since it > > > > ran out of disk space due to the multiple copies of the tree being > > > > created. > > > > > > And they never heard of "--exclude" in tar, did they? > > > > Doesn't work. Remember - you have to --exclude EVERY possible loop. And > > unless you know ahead of time, you can't exclude it. The only way we > > found to reliably do the backup was to dismount the CVS. > > And if you completely run out of ideas in your > wild-mounts-throughout-the-tree problem you should simply use "tar -l". > > And in just the same way fs could provide a mode bit saying "hi, I am a > hardlink", and tar can then easily provide option number 1345 saying "stay > away from hardlinks". Do you know what a hard link is? Or a directory entry? A directory is a file of records with two fields: an inode number, and a name (string). Nothing else. The "hard link" is the inode number. The count of hard links is maintained in the inode data structure. One for each (inode, filename) entry in a directory. Multiple hard links are just having the same inode number used with two names. Those two names do not have to exist in the same directory. A directory is a special file. It has a bit that says "hi, i am a directory". At a minimum it contains two implicit hard links. One to itself, and one to it's parent. The parent inode is also a directory. The parent, in addition to its own two hard links, contains a (inode, name) pair where the inode number is the same as the inode number for the subdirectory. This way each directory inode ALWAYS has a minimum of two hard links - the one to itself, and the one from its' parent. This implements the tree. It permits the concept of "working directory" with relative path names ("../.." and "./name" contstructs). What you say when you want additional hard links to a directory is that a directory will have two (or more) parents. If the fs provides a mode bit to identify something different that points to a directory, then you have a SYMLINK. This is because the directory entry currently only carries inode,name pairs. -- logically carries. Specific implementations may have ANY choice implementation of a directory file, tree, hash, whatever. To the VFS it always appears as if it were a linear file containing inode,name pairs. The entry for a symlink is still (inode, name), but the inode structure referenced contains a bit that says "hi, i am a symlink". The system THEN knows that the data contents of the inode point to another file. It doesn't matter whether that file is a regular file, a directory file, or a device file. > If you can do the fs patch, I can do the tar patch. The patch is FAR too extensive. It would have to be a whole new filesystem where a directory entry would have to carry additional information. And a rather large patch to the VFS to support it. And a new fsck to try and maintain it. Directory locking would no longer be reliable. And modifications to various system calls and facilities (unlink, link, namei, open, opendir, readdir, ...) Certain Linux features would have to be disallowed ("mv dir ../otherdir" for instance could generate backward loops). Tar would have to maintain a "pathname,inode" array for each file it backed up, so that it could verify that if "inode" is already copied, the new name is the only thing copied... (some graphs will cause the array to grow without bounds... so watch it...). It would be interesting in the abstract - as in using a datafile to store the graph, utilities to insert, delete, store data, retrieve data, import/export data.. Fsck would have to be implemented with a "mark and sweep" garbage collector. Hmmmm It would support an augmented transition matrix.... (ie most computer games are ATMs... as are large simulations...) As a Unix filesystem... It doesn't fit. (which is one reason databases are used to implement ATMs... The database backup ignores the contents other than to copy data in/out. The application program uses the data as an ATM, and the application has to deal with consistancy.) ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 14:12 ` Jesse Pollard @ 2003-08-05 14:21 ` Stephan von Krawczynski 2003-08-05 15:53 ` Herbert Pötzl 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 14:21 UTC (permalink / raw) To: Jesse Pollard; +Cc: aebr, linux-kernel On Tue, 5 Aug 2003 09:12:31 -0500 Jesse Pollard <jesse@cats-chateau.net> wrote: > > If you can do the fs patch, I can do the tar patch. > > The patch is FAR too extensive. It would have to be a whole new filesystem Fine. This is a clear and straight forward word. Can you explain why I don't see the mount -bind/rbind stuff through nfs? Is this a problem with a cheaper solution? Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-05 14:21 ` Stephan von Krawczynski @ 2003-08-05 15:53 ` Herbert Pötzl 0 siblings, 0 replies; 79+ messages in thread From: Herbert Pötzl @ 2003-08-05 15:53 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Jesse Pollard, aebr, linux-kernel On Tue, Aug 05, 2003 at 04:21:52PM +0200, Stephan von Krawczynski wrote: > On Tue, 5 Aug 2003 09:12:31 -0500 > Jesse Pollard <jesse@cats-chateau.net> wrote: > > > > If you can do the fs patch, I can do the tar patch. > > > > The patch is FAR too extensive. It would have to be a whole new filesystem > > Fine. This is a clear and straight forward word. Can you explain why I don't > see the mount -bind/rbind stuff through nfs? Is this a problem with a cheaper > solution? probably because --bind mounts have a vfsmount of their own, and you do not explicitely export this submount? just an idea, Herbert > Regards, > Stephan > > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 13:56 ` Stephan von Krawczynski 2003-08-04 14:04 ` Anton Altaparmakov 2003-08-04 14:33 ` Jesse Pollard @ 2003-08-04 20:47 ` Jan Harkes 2 siblings, 0 replies; 79+ messages in thread From: Jan Harkes @ 2003-08-04 20:47 UTC (permalink / raw) To: linux-kernel On Mon, Aug 04, 2003 at 03:56:04PM +0200, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 15:44:15 +0200 Andries Brouwer <aebr@win.tue.nl> wrote: > > > > Quite a lot of software thinks that the file hierarchy is a tree, > > if you wish a forest. > > > > Things would break badly if the hierarchy became an arbitrary graph. > > Care to name one? What exactly is the rule you see broken? Sure you can build > loops, but you cannot prevent people from doing braindamaged things to their > data anyway. You would not ban dd either for being able to flatten your > harddisk only because of one mistyping char... Only root can 'dd' to my disks, but anyone can do 'mkdir a; ln a a/a' and get even the simple things really messed up. You can't even rm -rf it anymore. Currently rm doesn't need to concern itself about loops. If something doesn't go away, it is a non-empty directory that needs to be traversed, and suddenly it has to track inodes and device numbers to identify potential cycles in the path. Hundreds of simple application suddenly become more complex. Can you imagine 'rm' running your machine out of memory on a reasonably sized tree. Also all 'hardlinked name entries' point at the same object. However, '..' could be the parent directory of any one of the name entries, but clearly not more than one at any time. So we actually have two (or more) objects that do not have the identical contents but share the same supposedly unique object identifier (inode number). Should we be allowed to 'unlink' a directory that is non-empty? But then the rmdir has to check all children, count the number of directories (i.e. fetch the attributes of all children) to compensate for the additional linkcounts added by the '..' entries. And to avoid a race with another unlink, or changes to the directory while we are traversing it this has to happen while all access to the directory is locked out. Not very scalable, especially when your directories do contain tens-of-thousands of users and gigabytes of data. Now if we don't allow the unlink for non-empty directories it is possible to create a loop that can never be removed. ln / /tmp/foo Typical optimizations for directory traversal make use of the fact that the child directories increase the link count. When reading directory entries and (linkcount - 2) directories have been seen we know all other entries are files. The additional references make this optimization useless. NFS exporting a filesystem is another good example. NFS is stateless and currently identifies files with a cookie that contains the (device/ino) pair. But because inode numbers are no longer unique because we need to know who the parent is, but the parent ino number also isn't unique anymore, so we need to pass a list of device/[list of all parents]/ino. As a result we're no better of compared to sending fixed pathnames over to the NFS server, and anyone who moves a directory from a to b breaks all other clients that happen to have a reference to a file or directory anywhere in the tree below the recently moved directory. Did you want any more reasons why hardlinked directories are bad? Jan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 13:44 ` Andries Brouwer 2003-08-04 13:56 ` Stephan von Krawczynski @ 2003-08-04 15:42 ` Brian Pawlowski 2003-08-04 15:56 ` Stephan von Krawczynski 1 sibling, 1 reply; 79+ messages in thread From: Brian Pawlowski @ 2003-08-04 15:42 UTC (permalink / raw) To: Andries Brouwer; +Cc: skraw, linux-kernel I'm still waking up, but '..' obviously breaks the "no cycle" observations. It's just that '..' is well known name by utilities as opposed to arbitrary links. Symlinks as poor man's link can create unwanted cycles (but are caught again by utils?) I was always wondering what to do with all those spare CPU cycles, running around in circles in the file system will soak them up. :-) ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:42 ` Brian Pawlowski @ 2003-08-04 15:56 ` Stephan von Krawczynski 2003-08-04 16:16 ` Herbert Pötzl 2003-08-04 21:29 ` Jesse Pollard 0 siblings, 2 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 15:56 UTC (permalink / raw) To: Brian Pawlowski; +Cc: aebr, linux-kernel On Mon, 4 Aug 2003 08:42:09 -0700 (PDT) Brian Pawlowski <beepy@netapp.com> wrote: > I'm still waking up, but '..' obviously breaks the "no cycle" > observations. Hear, hear ... > It's just that '..' is well known name by utilities as opposed > to arbitrary links. Well, that leads only to the point that ".." implementation is just lousy and it should have been done right in the first place. If there is a need for a loop or a hardlink (like "..") all you have to have is a standard way to find out, be it flags or the like, whatever. But taking the filename or anything not applicable to other cases as matching factor was obviously short-sighted. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:56 ` Stephan von Krawczynski @ 2003-08-04 16:16 ` Herbert Pötzl 2003-08-04 16:35 ` Stephan von Krawczynski 2003-08-04 21:29 ` Jesse Pollard 1 sibling, 1 reply; 79+ messages in thread From: Herbert Pötzl @ 2003-08-04 16:16 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Brian Pawlowski, aebr, linux-kernel On Mon, Aug 04, 2003 at 05:56:09PM +0200, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 08:42:09 -0700 (PDT) > Brian Pawlowski <beepy@netapp.com> wrote: > > > I'm still waking up, but '..' obviously breaks the "no cycle" > > observations. > > Hear, hear ... > > > It's just that '..' is well known name by utilities as opposed > > to arbitrary links. > > Well, that leads only to the point that ".." implementation is just lousy and > it should have been done right in the first place. If there is a need for a > loop or a hardlink (like "..") all you have to have is a standard way to find > out, be it flags or the like, whatever. But taking the filename or anything not > applicable to other cases as matching factor was obviously short-sighted. hey, why not just implement it, as a proof of concept and see, what is broken, and what can be fixed ... maybe your concept has a big fute, ... give it a try! on the other hand, if you want somebody to implement this stuff for you, you'll have to provide convincing arguments for it, I for example, would be glad if hardlinks where removed from unix altogether ... best, Herbert > Regards, > Stephan > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 16:16 ` Herbert Pötzl @ 2003-08-04 16:35 ` Stephan von Krawczynski 2003-08-04 16:54 ` Herbert Pötzl ` (2 more replies) 0 siblings, 3 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 16:35 UTC (permalink / raw) To: herbert; +Cc: beepy, aebr, linux-kernel On Mon, 4 Aug 2003 18:16:57 +0200 Herbert Pötzl <herbert@13thfloor.at> wrote: > on the other hand, if you want somebody to implement > this stuff for you, you'll have to provide convincing > arguments for it, I for example, would be glad if > hardlinks where removed from unix altogether ... Huh, hard stuff! Explain your solution for a very common problem: You have a _big_ fileserver, say some SAN or the like with Gigs. Your data on it is organized according to your basic user structure, because it is very handy to have all data from one user altogether in one directory. You have lots of hosts that use parts of the users' data for a wide range of purposes, lets say web, ftp, sql, name one. If you cannot re-structure and export your data according to the requirements of your external hosts (web-trees to webserver, sql-trees to sql-server, ftp-trees to ftp-server, name-it to cool-server) you will have to export the total user tree to all your (cluster-) nodes. Do you want that? NO! Of course you don't want that in times of hacked webservers and uncontrollable sql-servers. If anything blows up you likely loose all data at once. On the other hand, if you managed to link all web-data together in one directory and exported that to your webservers and they are hacked, you just blew up all your web-data but nothing more. This is a remarkable risk reduction. And now? Name your idea to export only the data needed to the servers that need it. And keep in mind, we are talking of Gigs and tenthousands of users. You definitely don't want one mount per user per service. Can you think of a more elegant way to solve such a problem than hardlinking all web in one single webtree, all sql in one single sql tree ... and then export this single tree (with its artificial structure) to the corresponding server? I am curiously listening... Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 16:35 ` Stephan von Krawczynski @ 2003-08-04 16:54 ` Herbert Pötzl 2003-08-04 17:18 ` Stephan von Krawczynski 2003-08-04 21:38 ` Jesse Pollard 2003-08-05 3:11 ` Neil Brown 2 siblings, 1 reply; 79+ messages in thread From: Herbert Pötzl @ 2003-08-04 16:54 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: beepy, aebr, linux-kernel On Mon, Aug 04, 2003 at 06:35:45PM +0200, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 18:16:57 +0200 > Herbert Pötzl <herbert@13thfloor.at> wrote: > > > on the other hand, if you want somebody to implement > > this stuff for you, you'll have to provide convincing > > arguments for it, I for example, would be glad if > > hardlinks where removed from unix altogether ... > > Huh, hard stuff! > > Explain your solution for a very common problem: > > You have a _big_ fileserver, say some SAN or the like with Gigs. > Your data on it is organized according to your basic user > structure, because it is very handy to have all data from one > user altogether in one directory. I already do something like this, although not for thousands of users, but I guess this would scale well ... consider a storage device (maybe a partition) for each category of data you want to store/provide/serve /mnt/webspace, /mnt/database, /mnt/email, /mnt/wossname ... now each data space gets subdirectories for logical groupings (optional) and a second level for the actual users ... there could be other layers like domains for example too ... /mnt/webspace/customer/charlie /mnt/webspace/customer/jack /mnt/webspace/customer/wossname and, if required, the same structure for database, email, ... so you end up with totally separate storage trees, which can be easily mounted/exported/shared with the apropriate servers, now for the conceptional grouping, where the customer wants to have all the data in one place ... on the access server (where the customer maintains the data) you'll end up mounting all the storage categories, required for access and you do an additional restructuring for each customer (which of course is automated) /home/customer/charlie is populated with symlinks to /mnt/*/customer/charlie named by category /home/customer/charlie/webspace -> /mnt/webspace/customer/charlie /home/customer/charlie/email -> /mnt/email/customer/charlie ... this also has the advantage that any kind of service change (for example changing the webspace) can be simply done by modifying the symlinks ... > if you managed to link all web-data together in one directory and > exported that to your webservers and they are hacked, you just > blew up all your web-data but nothing more. > This is a remarkable risk reduction. > And now? Name your idea to export only the data needed to > the servers that need it. And keep in mind, we are talking of > Gigs and tenthousands of users. this is covered by the suggested approach. > You definitely don't want one mount per user per service. no, I don't ... > Can you think of a more elegant way to solve such a problem > than hardlinking all web in one single webtree, all sql in one > single sql tree ... and then export this single tree (with its > artificial structure) to the corresponding server? > I am curiously listening... best, Herbert > Regards, > Stephan > ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 16:54 ` Herbert Pötzl @ 2003-08-04 17:18 ` Stephan von Krawczynski 2003-08-04 17:25 ` Herbert Pötzl 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 17:18 UTC (permalink / raw) To: herbert; +Cc: beepy, aebr, linux-kernel On Mon, 4 Aug 2003 18:54:35 +0200 Herbert Pötzl <herbert@13thfloor.at> wrote: > On Mon, Aug 04, 2003 at 06:35:45PM +0200, Stephan von Krawczynski wrote: > > On Mon, 4 Aug 2003 18:16:57 +0200 > > Herbert Pötzl <herbert@13thfloor.at> wrote: > > > > > on the other hand, if you want somebody to implement > > > this stuff for you, you'll have to provide convincing > > > arguments for it, I for example, would be glad if > > > hardlinks where removed from unix altogether ... > > > > Huh, hard stuff! > > > > Explain your solution for a very common problem: > > > > You have a _big_ fileserver, say some SAN or the like with Gigs. > > Your data on it is organized according to your basic user > > structure, because it is very handy to have all data from one > > user altogether in one directory. > > I already do something like this, although not for thousands > of users, but I guess this would scale well ... > > consider a storage device (maybe a partition) for each > category of data you want to store/provide/serve > > /mnt/webspace, /mnt/database, /mnt/email, /mnt/wossname ... I have one objection: it is the poorest possible implementation in terms of storage usage. You may have TBs free on database and ran full on webspace, fluctuation on email is dramatic. You waste a damn lot of resources with this approach. I consider it usable but inferior in terms of costs and scalability. One can very well argue you have to waste bucks to avoid hardlinks (which you don't have at this point). Not having hardlinks costs obviously more. In fact a very good commercial argument pro hardlinks. If you have all your partitions only logically organised on one SAN you have to remount via nfs quite a bit to get things in the right trees. That does not sound like a performance setup. Did you manage to remount "mount --bind" volumes? and ... > on the access server (where the customer maintains the data) > you'll end up mounting all the storage categories, required > for access and you do an additional restructuring for each > customer (which of course is automated) > > /home/customer/charlie is populated with symlinks to > /mnt/*/customer/charlie named by category > > /home/customer/charlie/webspace -> /mnt/webspace/customer/charlie > /home/customer/charlie/email -> /mnt/email/customer/charlie > ... which basically means on your access server you have to follow symlinks which is yet another security issue in itself. Basically you avoid a clean hardlink-setup with a symlink-setup rising possible security drawbacks and a lot of remounting to get things straight... I am not really convinced. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 17:18 ` Stephan von Krawczynski @ 2003-08-04 17:25 ` Herbert Pötzl 0 siblings, 0 replies; 79+ messages in thread From: Herbert Pötzl @ 2003-08-04 17:25 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: beepy, aebr, linux-kernel On Mon, Aug 04, 2003 at 07:18:28PM +0200, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 18:54:35 +0200 > Herbert Pötzl <herbert@13thfloor.at> wrote: > > > On Mon, Aug 04, 2003 at 06:35:45PM +0200, Stephan von Krawczynski wrote: > > > On Mon, 4 Aug 2003 18:16:57 +0200 > > > Herbert Pötzl <herbert@13thfloor.at> wrote: > > > > > > > on the other hand, if you want somebody to implement > > > > this stuff for you, you'll have to provide convincing > > > > arguments for it, I for example, would be glad if > > > > hardlinks where removed from unix altogether ... > > > > > > Huh, hard stuff! > > > > > > Explain your solution for a very common problem: > > > > > > You have a _big_ fileserver, say some SAN or the like with Gigs. > > > Your data on it is organized according to your basic user > > > structure, because it is very handy to have all data from one > > > user altogether in one directory. > > > > I already do something like this, although not for thousands > > of users, but I guess this would scale well ... > > > > consider a storage device (maybe a partition) for each > > category of data you want to store/provide/serve > > > > /mnt/webspace, /mnt/database, /mnt/email, /mnt/wossname ... > > I have one objection: it is the poorest possible implementation in terms of > storage usage. You may have TBs free on database and ran full on webspace, > fluctuation on email is dramatic. You waste a damn lot of resources with this > approach. hmm, I guess this objection can be ignored, because you do not have to use a separate partition, if you do not want to, do you? > I consider it usable but inferior in terms of costs and scalability. One can > very well argue you have to waste bucks to avoid hardlinks (which you don't > have at this point). Not having hardlinks costs obviously more. In fact a very > good commercial argument pro hardlinks. > If you have all your partitions only logically organised on one SAN you have to > remount via nfs quite a bit to get things in the right trees. That does not > sound like a performance setup. Did you manage to remount "mount --bind" > volumes? > > and ... > > > on the access server (where the customer maintains the data) > > you'll end up mounting all the storage categories, required > > for access and you do an additional restructuring for each > > customer (which of course is automated) > > > > /home/customer/charlie is populated with symlinks to > > /mnt/*/customer/charlie named by category > > > > /home/customer/charlie/webspace -> /mnt/webspace/customer/charlie > > /home/customer/charlie/email -> /mnt/email/customer/charlie > > ... > > which basically means on your access server you have to follow > symlinks which is yet another security issue in itself. > > Basically you avoid a clean hardlink-setup with a symlink-setup > rising possible > security drawbacks and a lot of remounting to get things straight... > I am not really convinced. no remounting so far, and security is the same with or without symlinks, either you have access to the data or not, in any way, the symlink doesn't change this behaviour ... best, Herbert > Regards, > Stephan > ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 16:35 ` Stephan von Krawczynski 2003-08-04 16:54 ` Herbert Pötzl @ 2003-08-04 21:38 ` Jesse Pollard 2003-08-05 0:06 ` Stephan von Krawczynski 2003-08-05 3:11 ` Neil Brown 2 siblings, 1 reply; 79+ messages in thread From: Jesse Pollard @ 2003-08-04 21:38 UTC (permalink / raw) To: Stephan von Krawczynski, herbert; +Cc: beepy, aebr, linux-kernel On Monday 04 August 2003 11:35, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 18:16:57 +0200 > > Herbert Pötzl <herbert@13thfloor.at> wrote: > > on the other hand, if you want somebody to implement > > this stuff for you, you'll have to provide convincing > > arguments for it, I for example, would be glad if > > hardlinks where removed from unix altogether ... > > Huh, hard stuff! > > Explain your solution for a very common problem: > > You have a _big_ fileserver, say some SAN or the like with Gigs. > Your data on it is organized according to your basic user structure, > because it is very handy to have all data from one user altogether in one > directory. You have lots of hosts that use parts of the users' data for a > wide range of purposes, lets say web, ftp, sql, name one. > If you cannot re-structure and export your data according to the > requirements of your external hosts (web-trees to webserver, sql-trees to > sql-server, ftp-trees to ftp-server, name-it to cool-server) you will have > to export the total user tree to all your (cluster-) nodes. Do you want > that? NO! Of course you don't want that in times of hacked webservers and > uncontrollable sql-servers. If anything blows up you likely loose all data > at once. On the other hand, if you managed to link all web-data together in > one directory and exported that to your webservers and they are hacked, you > just blew up all your web-data but nothing more. This is a remarkable risk > reduction. > And now? Name your idea to export only the data needed to the servers that > need it. And keep in mind, we are talking of Gigs and tenthousands of > users. You definitely don't want one mount per user per service. > Can you think of a more elegant way to solve such a problem than > hardlinking all web in one single webtree, all sql in one single sql tree > ... and then export this single tree (with its artificial structure) to the > corresponding server? > I am curiously listening... Don't do that. It is too insecure. 1. the structure you describe is FRAGILE. Just adding one more entry could/would break the entire structure. 2. If you mix security structures like this you WILL get a problem. What you do is copy the declassified data to a nonsecure area (also known as released data). This way the user can modify internal cata without causing the web server potentially catastrophic releases. Same with the SQL. Do not attmept to mix sensitive and nonsensitive data this way. If you web server got hacked, how do you prevent the hack from ADDING more links? Or adding SQL injections to other applications... If you've get this much disk space, then you can afford to provide isolated data too. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 21:38 ` Jesse Pollard @ 2003-08-05 0:06 ` Stephan von Krawczynski 0 siblings, 0 replies; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-05 0:06 UTC (permalink / raw) To: Jesse Pollard; +Cc: herbert, beepy, aebr, linux-kernel On Mon, 4 Aug 2003 16:38:17 -0500 Jesse Pollard <jesse@cats-chateau.net> wrote: > On Monday 04 August 2003 11:35, Stephan von Krawczynski wrote: > > On Mon, 4 Aug 2003 18:16:57 +0200 > [...] > Don't do that. It is too insecure. > > 1. the structure you describe is FRAGILE. Just adding one more entry > could/would break the entire structure. > > 2. If you mix security structures like this you WILL get a problem. > > What you do is copy the declassified data to a nonsecure area (also known > as released data). This way the user can modify internal cata without > causing the web server potentially catastrophic releases. > > Same with the SQL. Do not attmept to mix sensitive and nonsensitive data > this way. Your just kidding, don't you? Definition of "problem" here is: service got corrupted. It is really of _no_ interest if the data that was corrupted is "sensitive" or "nonsensitive", because the only cure in both versions is rewriting from scratch (and dumping the server of course). So your possible downtime is just as big in both ways. And nothing else counts. > If you web server got hacked, how do you prevent the hack from ADDING > more links? Or adding SQL injections to other applications... I don't, because it is simply impossible. If you are root on a webserver everything is lost, no matter if your data is local or nfs-mounted you can delete, relink or whatever you like at will. The only thing you _can't_ do is access data that is not exported to your hacked system. And that's exactly what I am trying to do: don't give any more data away than absolutely necessary. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 16:35 ` Stephan von Krawczynski 2003-08-04 16:54 ` Herbert Pötzl 2003-08-04 21:38 ` Jesse Pollard @ 2003-08-05 3:11 ` Neil Brown 2 siblings, 0 replies; 79+ messages in thread From: Neil Brown @ 2003-08-05 3:11 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: herbert, beepy, aebr, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset=unknown, Size: 4329 bytes --] On Monday August 4, skraw@ithnet.com wrote: > On Mon, 4 Aug 2003 18:16:57 +0200 > Herbert Pötzl <herbert@13thfloor.at> wrote: > > > on the other hand, if you want somebody to implement > > this stuff for you, you'll have to provide convincing > > arguments for it, I for example, would be glad if > > hardlinks where removed from unix altogether ... > > Huh, hard stuff! > > Explain your solution for a very common problem: > > You have a _big_ fileserver, say some SAN or the like with Gigs. > Your data on it is organized according to your basic user structure, because it > is very handy to have all data from one user altogether in one directory. > You have lots of hosts that use parts of the users' data for a wide range of > purposes, lets say web, ftp, sql, name one. > If you cannot re-structure and export your data according to the requirements > of your external hosts (web-trees to webserver, sql-trees to sql-server, > ftp-trees to ftp-server, name-it to cool-server) you will have to export the > total user tree to all your (cluster-) nodes. Do you want that? NO! Of course > you don't want that in times of hacked webservers and uncontrollable > sql-servers. If anything blows up you likely loose all data at once. On the > other hand, if you managed to link all web-data together in one directory and > exported that to your webservers and they are hacked, you just blew up all your > web-data but nothing more. This is a remarkable risk reduction. > And now? Name your idea to export only the data needed to the servers that need > it. And keep in mind, we are talking of Gigs and tenthousands of users. You > definitely don't want one mount per user per service. > Can you think of a more elegant way to solve such a problem than hardlinking > all web in one single webtree, all sql in one single sql tree ... and then > export this single tree (with its artificial structure) to the corresponding > server? > I am curiously listening... I would suggest that this is where you should have started. Instead of leading people down pointless discussions about hardlinks to directories, it would be best to state the real problem. The best solution almost certainly has nothing to do with hardlinks to directories (which are simply *not* Unix). What you want to do, it would seem, is 1/ nfs-export just selected subtrees of users' accounts to selected servers and 2/ mount just those sub-trees on the relevant servers. Neither of these are particularly difficult except from an admin perspective. For (1) just put lines like: /home/neilb/public_html webserver(rw) /home/neilb/public_ftp ftpserver(rw) /home/neilb/database dbserver(rw) /home/fred/public_html webserver(rw) ... in /etc/exports and it is done. for 2, just use an automounter with a different map on each server. For the straight-forward approach you would need to modifiy /etc/exports and all the maps whenever you add a new user. which isn't ideal from an admin perspective. We do the second of these on our cgi-servers, but we have a sophisticated user database that can present arbitrary views as NIS maps, and we point amd at an appropiate NIS map and it sees new users as they are added. You would need to integrate management of the automounter maps with whatever accounts management system you have. We don't bother with the second (I'm not worried about root compomises much - I'm just worried that if someone installs an insecure cgi script, an attacker cannot get beyond that person's public_html) but if I did, it would not be hard to modify nfs_utils to understand wildcards in /etc/exports so that, e.g. /home/*/public_html webserver(rw) would be equivalent to as whole list of similar lines with explicit user names. I might even do that for an upcoming release of nfs-utils, but no promises. In summary: you cannot "hardlink directories"(*) and if you think you want to, you are probably asking the wrong question. NeilBrown (*) I put "hardlink directories" in quotes because ofcourse you can have hard links to directories. Every child has a hard link called ".." and the one parent has a hard link with some other name. What you really mean in "multiple parents of directories" and that is not possible for assorted reasons, but mostly because you cannot have multiple ".."s. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 15:56 ` Stephan von Krawczynski 2003-08-04 16:16 ` Herbert Pötzl @ 2003-08-04 21:29 ` Jesse Pollard 2003-08-04 23:42 ` Stephan von Krawczynski 1 sibling, 1 reply; 79+ messages in thread From: Jesse Pollard @ 2003-08-04 21:29 UTC (permalink / raw) To: Stephan von Krawczynski, Brian Pawlowski; +Cc: aebr, linux-kernel On Monday 04 August 2003 10:56, Stephan von Krawczynski wrote: > On Mon, 4 Aug 2003 08:42:09 -0700 (PDT) > > Brian Pawlowski <beepy@netapp.com> wrote: > > I'm still waking up, but '..' obviously breaks the "no cycle" > > observations. > > Hear, hear ... > > > It's just that '..' is well known name by utilities as opposed > > to arbitrary links. > > Well, that leads only to the point that ".." implementation is just lousy > and it should have been done right in the first place. If there is a need > for a loop or a hardlink (like "..") all you have to have is a standard way > to find out, be it flags or the like, whatever. But taking the filename or > anything not applicable to other cases as matching factor was obviously > short-sighted. Has nothing to do with the loop. It is called an AVL tree. It makes the namei lookup function extreemly simple to implement: "../file" is treated in the same way as "./file" is treated, which is the same as "file". No special case required. It allows the VERY flexable (and simple) relative path name handling. ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 21:29 ` Jesse Pollard @ 2003-08-04 23:42 ` Stephan von Krawczynski 2003-08-05 16:46 ` viro 0 siblings, 1 reply; 79+ messages in thread From: Stephan von Krawczynski @ 2003-08-04 23:42 UTC (permalink / raw) To: Jesse Pollard; +Cc: beepy, aebr, linux-kernel On Mon, 4 Aug 2003 16:29:50 -0500 Jesse Pollard <jesse@cats-chateau.net> wrote: > On Monday 04 August 2003 10:56, Stephan von Krawczynski wrote: > > On Mon, 4 Aug 2003 08:42:09 -0700 (PDT) > > > > Brian Pawlowski <beepy@netapp.com> wrote: > > > I'm still waking up, but '..' obviously breaks the "no cycle" > > > observations. > > > > Hear, hear ... > > > > > It's just that '..' is well known name by utilities as opposed > > > to arbitrary links. > > > > Well, that leads only to the point that ".." implementation is just lousy > > and it should have been done right in the first place. If there is a need > > for a loop or a hardlink (like "..") all you have to have is a standard way > > to find out, be it flags or the like, whatever. But taking the filename or > > anything not applicable to other cases as matching factor was obviously > > short-sighted. > > Has nothing to do with the loop. It is called an AVL tree. Hm, ".." points back to a directory in its parent path (in fact simply its own parent). You don't call this a loop? How come? If I write a simple program that follows all directory entries of a given directory it will simply loop, it only won't loop if I tell it explicitely _not_ to follow ".." and ".", because "." is nothing else but the shortest possible loop. Regards, Stephan ^ permalink raw reply [flat|nested] 79+ messages in thread
* Re: FS: hardlinks on directories 2003-08-04 23:42 ` Stephan von Krawczynski @ 2003-08-05 16:46 ` viro 0 siblings, 0 replies; 79+ messages in thread From: viro @ 2003-08-05 16:46 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Jesse Pollard, beepy, aebr, linux-kernel Egads... OK, here comes: a) if you have real loops in directory graph (directory being its own descendent), determining whether rename() will make it disconnected becomes hell; it's completely infeasible. b) if you allow multiple paths from root, preventing loops creation upon rename() also becomes hell. Forget about VFS, on-disk structures, etc. - just think in terms of oriented graphs and operations on them; that's the price you'd have to pay with any implementation and it's *big*. ^ permalink raw reply [flat|nested] 79+ messages in thread
end of thread, other threads:[~2003-08-25 15:56 UTC | newest] Thread overview: 79+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2003-08-04 12:15 FS: hardlinks on directories Stephan von Krawczynski 2003-08-04 12:45 ` Måns Rullgård 2003-08-04 13:22 ` Stephan von Krawczynski 2003-08-04 13:37 ` Christian Reichert 2003-08-04 13:44 ` Stephan von Krawczynski 2003-08-04 14:22 ` Christian Reichert 2003-08-04 15:31 ` Jeff Muizelaar 2003-08-04 16:15 ` Stephan von Krawczynski 2003-08-05 2:45 ` Neil Brown 2003-08-05 9:41 ` Stephan von Krawczynski 2003-08-06 1:12 ` Neil Brown 2003-08-06 10:14 ` Stephan von Krawczynski 2003-08-07 2:27 ` Neil Brown 2003-08-04 12:47 ` Nikita Danilov 2003-08-04 13:32 ` Stephan von Krawczynski 2003-08-04 13:44 ` Andries Brouwer 2003-08-04 13:56 ` Stephan von Krawczynski 2003-08-04 14:04 ` Anton Altaparmakov 2003-08-04 14:50 ` Stephan von Krawczynski 2003-08-04 20:03 ` Olivier Galibert 2003-08-04 21:16 ` Jesse Pollard 2003-08-04 23:34 ` Stephan von Krawczynski 2003-08-05 14:20 ` Jesse Pollard 2003-08-05 14:44 ` Stephan von Krawczynski 2003-08-04 22:58 ` Andrew Pimlott 2003-08-05 0:19 ` Stephan von Krawczynski 2003-08-05 1:18 ` Andrew Pimlott 2003-08-05 8:04 ` Stephan von Krawczynski 2003-08-05 11:18 ` Wakko Warner 2003-08-04 14:33 ` Jesse Pollard 2003-08-04 15:05 ` Stephan von Krawczynski 2003-08-04 15:57 ` Richard B. Johnson 2003-08-04 21:23 ` Jesse Pollard 2003-08-04 16:11 ` Adam Sampson 2003-08-04 17:00 ` Hans Reiser 2003-08-04 17:18 ` Sean Neakums 2003-08-05 4:53 ` jw schultz 2003-08-04 18:50 ` jlnance 2003-08-04 21:09 ` Jesse Pollard 2003-08-04 22:13 ` Stephan von Krawczynski 2003-08-04 22:32 ` Stephan von Krawczynski 2003-08-04 23:00 ` Randolph Bentson 2003-08-05 0:10 ` Stephan von Krawczynski 2003-08-05 2:09 ` Edgar Toernig 2003-08-05 8:05 ` Stephan von Krawczynski 2003-08-05 12:51 ` Helge Hafting 2003-08-05 13:03 ` Stephan von Krawczynski 2003-08-05 13:13 ` Bernd Petrovitsch 2003-08-05 13:39 ` Stephan von Krawczynski 2003-08-05 13:36 ` Richard B. Johnson 2003-08-05 14:04 ` Stephan von Krawczynski 2003-08-05 14:57 ` Richard B. Johnson 2003-08-05 15:08 ` Stephan von Krawczynski 2003-08-05 15:02 ` Jesse Pollard 2003-08-05 15:12 ` Stephan von Krawczynski 2003-08-05 15:44 ` Trond Myklebust 2003-08-05 14:56 ` Jesse Pollard 2003-08-05 22:08 ` Helge Hafting 2003-08-24 17:35 ` Hans Reiser 2003-08-24 19:02 ` Helge Hafting 2003-08-25 8:27 ` Nikita Danilov 2003-08-25 15:48 ` Hans Reiser 2003-08-05 14:12 ` Jesse Pollard 2003-08-05 14:21 ` Stephan von Krawczynski 2003-08-05 15:53 ` Herbert Pötzl 2003-08-04 20:47 ` Jan Harkes 2003-08-04 15:42 ` Brian Pawlowski 2003-08-04 15:56 ` Stephan von Krawczynski 2003-08-04 16:16 ` Herbert Pötzl 2003-08-04 16:35 ` Stephan von Krawczynski 2003-08-04 16:54 ` Herbert Pötzl 2003-08-04 17:18 ` Stephan von Krawczynski 2003-08-04 17:25 ` Herbert Pötzl 2003-08-04 21:38 ` Jesse Pollard 2003-08-05 0:06 ` Stephan von Krawczynski 2003-08-05 3:11 ` Neil Brown 2003-08-04 21:29 ` Jesse Pollard 2003-08-04 23:42 ` Stephan von Krawczynski 2003-08-05 16:46 ` viro
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).