'Subset' Hard Links

'Subset' Hard Links

[Micah Dombrowski]

Hello,

I couldn't think of anywhere else to ask such a question, and google  
is useless as I have no unique keywords.  I am wondering if it is  
possible with some/any filesystems to have multiple hard links to a  
file, some of which only point to a subset of the file's data.

Eg:
firstname -> all data bytes 1 to 10
secondname -> bytes 3 to 10
thirdname -> bytes 5 to 7

This would clearly require some interesting locking of the file WRT  
writes, but it seems like it should be possible, and even easy for  
read-only access.  I deal with moderately large data files (50+GB),  
and such a thing would be incredibly useful to me for pulling out  
interesting bits of my data without having to make copies of the data  
itself.

A related method I was wishing existed would allow concatenation of  
files simply by deleting all but one hard link, and changing the  
remaining one to point to all of the original files' data as  
fragments.  This would be great, as 'cat'ing together six 10GB files  
is pretty slow, and it seems silly to be copying all of that data  
around when I only need one actual instance of the full data on disk,  
and that instance already exists, albeit in a fragmented manner.

Do any tools for doing this sort of thing exist?

Thanks,
Micah Dombrowski
Department of Physics and Astronomy
Dartmouth College

Re: 'Subset' Hard Links

[Sunil Mushran]

Micah Dombrowski wrote:
> I couldn't think of anywhere else to ask such a question, and google 
> is useless as I have no unique keywords.  I am wondering if it is 
> possible with some/any filesystems to have multiple hard links to a 
> file, some of which only point to a subset of the file's data.
>
> Eg:
> firstname -> all data bytes 1 to 10
> secondname -> bytes 3 to 10
> thirdname -> bytes 5 to 7
>
> This would clearly require some interesting locking of the file WRT 
> writes, but it seems like it should be possible, and even easy for 
> read-only access.  I deal with moderately large data files (50+GB), 
> and such a thing would be incredibly useful to me for pulling out 
> interesting bits of my data without having to make copies of the data 
> itself.
>
> A related method I was wishing existed would allow concatenation of 
> files simply by deleting all but one hard link, and changing the 
> remaining one to point to all of the original files' data as 
> fragments.  This would be great, as 'cat'ing together six 10GB files 
> is pretty slow, and it seems silly to be copying all of that data 
> around when I only need one actual instance of the full data on disk, 
> and that instance already exists, albeit in a fragmented manner.
>
> Do any tools for doing this sort of thing exist?
>

btrfs should able to handle most of this.

http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commitdiff;h=c5c9cd4d1b827fe545ed2a945e91e3a6909f3886

However, note that file systems operate in terms of blocks. So the start 
offset
would need to be block aligned.

Re: 'Subset' Hard Links

[Sage Weil]

On Thu, 17 Sep 2009, Sunil Mushran wrote:

> Micah Dombrowski wrote:
> > I couldn't think of anywhere else to ask such a question, and google is
> > useless as I have no unique keywords.  I am wondering if it is possible with
> > some/any filesystems to have multiple hard links to a file, some of which
> > only point to a subset of the file's data.
> > 
> > Eg:
> > firstname -> all data bytes 1 to 10
> > secondname -> bytes 3 to 10
> > thirdname -> bytes 5 to 7
> >
> > This would clearly require some interesting locking of the file WRT writes,
> > but it seems like it should be possible, and even easy for read-only access.
> > I deal with moderately large data files (50+GB), and such a thing would be
> > incredibly useful to me for pulling out interesting bits of my data without
> > having to make copies of the data itself.
> >
> > A related method I was wishing existed would allow concatenation of files
> > simply by deleting all but one hard link, and changing the remaining one to
> > point to all of the original files' data as fragments.  This would be great,
> > as 'cat'ing together six 10GB files is pretty slow, and it seems silly to be
> > copying all of that data around when I only need one actual instance of the
> > full data on disk, and that instance already exists, albeit in a fragmented
> > manner.
> > 
> > Do any tools for doing this sort of thing exist?
> > 
> 
> btrfs should able to handle most of this.
> 
> http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commitdiff;h=c5c9cd4d1b827fe545ed2a945e91e3a6909f3886

Note that currently, you can only clone a range from one file to another.  
It should be pretty straightforward to allow cloning from one offset to 
another.  Or, you can work around it by cloning the range to a temporary 
file and then back again at a different offset.

The code can also currently fail when compression is enabled and 
you clone a subset of the file (compressed inline extents don't get split 
yet).

sage


> 
> However, note that file systems operate in terms of blocks. So the start 
> offset would need to be block aligned.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>