creating live virtual files by concatenation

creating live virtual files by concatenation

[Maciej Soltysiak]

Hello!

I have this idea about creating sort of a virtual file.

Let us say I have three text files that contain javascript code:
tooltip.js
banner.js
foo.js

Now let us say I am creating sort of a virtual text file (code.js)
that is a live-concatenation of these files:
# concatenate tooltip.js banner.js foo.js code.js

Note I am not talking about the cat(1) utility. I am thinking of
code.js be always a live concatenated version of these three, so when
I modify one file, the live-version is also modified.

What puprose I might have? Network-related. Say, I have an HTML file
that includes these three files in its code.

When a browser downloads the HTML file it will then create three threads
to download each of those javascript files.

If I had a live-concatenated file, I could reference it in the HTML file
so that the browser does not have to download three files but just one.

This would surely reduce network overhead of downloading the same amount
of data but within just one connection, reduce resource usage on the client
and possibly (depending on implementation) reduce the cost of accessing
three individual files on the server.

I am CC'ing reiserfs-list because Reiser4 would seem to be the most
robust filesystem that could have it done.

Any thoughts about the idea itself?
Would be nice if this idea could inspire some talented hackers here and there.

Best Regards,
Maciej

Re: creating live virtual files by concatenation

[Jan Engelhardt]

>
>Now let us say I am creating sort of a virtual text file (code.js)
>that is a live-concatenation of these files:
># concatenate tooltip.js banner.js foo.js code.js
>
>Note I am not talking about the cat(1) utility. I am thinking of
>code.js be always a live concatenated version of these three, so when
>I modify one file, the live-version is also modified.
>
>What puprose I might have? Network-related. Say, I have an HTML file
>that includes these three files in its code.
>
Try FUSE.

>If I had a live-concatenated file, I could reference it in the HTML file
>so that the browser does not have to download three files but just one.
>
>This would surely reduce network overhead of downloading the same amount
>of data but within just one connection, reduce resource usage on the client
>and possibly (depending on implementation) reduce the cost of accessing
>three individual files on the server.
>
Have you ever heard of persistent connections with HTTP/1.1?


Jan Engelhardt
-- 

Re: creating live virtual files by concatenation

[Jesper Juhl]

On 2/25/06, Maciej Soltysiak <solt2@dns.toxicfilms.tv> wrote:
> Hello!
>
> I have this idea about creating sort of a virtual file.
>
> Let us say I have three text files that contain javascript code:
> tooltip.js
> banner.js
> foo.js
>
> Now let us say I am creating sort of a virtual text file (code.js)
> that is a live-concatenation of these files:
> # concatenate tooltip.js banner.js foo.js code.js
>
> Note I am not talking about the cat(1) utility. I am thinking of
> code.js be always a live concatenated version of these three, so when
> I modify one file, the live-version is also modified.
>
> What puprose I might have? Network-related. Say, I have an HTML file
> that includes these three files in its code.
>
> When a browser downloads the HTML file it will then create three threads
> to download each of those javascript files.
>
> If I had a live-concatenated file, I could reference it in the HTML file
> so that the browser does not have to download three files but just one.
>

If that's what you want to accomplish, then you can easily get around
that in several ways.

1. Simply   $ cat tooltip.js banner.js foo.js > code.js  then include
code.js in your html document and remember to update it when you
change one of the 3 files (or create a script that does it.

2. use Apache's mod_include

3. Use PHP, Perl, python or watever your scripting language of choice
is - here's an example in PHP :

<?php
header('Content-type: text/javascript');
readfile('tooltip.js');
readfile('banner.js');
readfile('foo.js');
?>

save that as javascripts.php then put this in your HTML document :

<script src="javascripts.php" language="javascript"
type="text/javascript"></script>


And there are other ways ...


> This would surely reduce network overhead of downloading the same amount
> of data but within just one connection, reduce resource usage on the client
> and possibly (depending on implementation) reduce the cost of accessing
> three individual files on the server.
>

Negligible I'd say.


> I am CC'ing reiserfs-list because Reiser4 would seem to be the most
> robust filesystem that could have it done.
>
> Any thoughts about the idea itself?

Might be a cute little hack, but I don't think it's a very useful
feature really..


> Would be nice if this idea could inspire some talented hackers here and there.
>
> Best Regards,
> Maciej
>


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: creating live virtual files by concatenation

[Maciej Soltysiak]

Jesper, Jan,

thanks for your replies. What I had in mind was not a solution
to any specific problem, but a filesystem feature.

A feature that could be used anywhere to have a live version
of files, a bit of what SQL CREATE VIEW could do for databases.

Code files, DNS zones, configuration files, HTML code. We are still
dealing with lots of text files today. Sometimes we do tiresome operations
on them or have to make up solutions similar to which Jesper had
proposed (using cat and updating it, using application level features,etc)

> Might be a cute little hack, but I don't think it's a very useful
> feature really..
Well, I will give FUSE a shot. However I have a hunch that this type
of feature is actually the future of something like filesystem services
for userspace.

Filesystems providing ways for userspace applications to create temporary
files, view, collections based on metadata. But that is completely
an other story, I wanted to keep low with ideas.

-- 
Best regards,
Maciej

Re: creating live virtual files by concatenation

[Peter Foldiak]

hi, I think it is a good idea but there has been quite a long discusion of 
this issue on the reiserfs-list (some of it also on lkml) in the last few 
years, with the latest posts in May 2005 I think, most of ithem with the 
"file as directory" subject . The idea was if you have e.g. a "book" 
directory/file (object) and chapters within it, then the default action of 
"book" object when read as a file would be to give a concatenation of the 
chapter objects within it. There should be ways of specifying non-default 
actions for the objects too. There has also been a lot a resistance here to 
this idea, as it is quite a radical change to the concept of the conventional 
unstructured, serial "file". (I don't believe these are good reason not to 
try the idea at least.)
Read those mails first. Peter

On Saturday 25 February 2006 14:37, Maciej Soltysiak wrote:
> I have this idea about creating sort of a virtual file.

Re: creating live virtual files by concatenation

[Rik van Riel]

On Sat, 25 Feb 2006, Maciej Soltysiak wrote:

> Code files, DNS zones, configuration files, HTML code. We are still 
> dealing with lots of text files today.

You say it like it's a bad thing, but in truth I suspect
people often deal with text files because they're EASY
to manipulate through scripts, etc.

-- 
All Rights Reversed

Re: creating live virtual files by concatenation

[Jesper Juhl]

On 2/25/06, Rik van Riel <riel@redhat.com> wrote:
> On Sat, 25 Feb 2006, Maciej Soltysiak wrote:
>
> > Code files, DNS zones, configuration files, HTML code. We are still
> > dealing with lots of text files today.
>
> You say it like it's a bad thing, but in truth I suspect
> people often deal with text files because they're EASY
> to manipulate through scripts, etc.
>

I agree 100%, plain text serial files are *easy*.

Given the swiss-army knife of tools we have at our disposal (cat, cut,
head, tail, sed, awk, split, sort, grep  and many more), plain text
files are very easy to manipulate - not to mention write apps to
manipulate. And that is a very good thing IMHO.

I can imagine quite a mess if I open a file that is really a view of
several files and then start manipulating text in it across "actual
file" boundaries  that could blow up easily.

--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re[2]: creating live virtual files by concatenation

[Maciej Soltysiak]

Hello Jesper

Saturday, February 25, 2006, 7:52:06 PM, you wrote:

> On 2/25/06, Rik van Riel <riel@redhat.com> wrote:
>> On Sat, 25 Feb 2006, Maciej Soltysiak wrote:
>>
>> > Code files, DNS zones, configuration files, HTML code. We are still
>> > dealing with lots of text files today.
>>
>> You say it like it's a bad thing, but in truth I suspect
>> people often deal with text files because they're EASY
>> to manipulate through scripts, etc.
Well, I did not mean to sound like that. My emphasis should have been
on that it sometimes is tiresome. I have no problems with plain text
files, I still am a human being, not an binary/XML parser or whatever :-D

> I can imagine quite a mess if I open a file that is really a view of
> several files and then start manipulating text in it across "actual
> file" boundaries  that could blow up easily.
Well, I meant that file to be read-only. Just a quick concatentated view
for reading.

-- 
Best regards,
Maciej

Re: creating live virtual files by concatenation

[Peter Foldiak]

Maciej Soltysiak wrote:

>>I can imagine quite a mess if I open a file that is really a view of
>>several files and then start manipulating text in it across "actual
>>file" boundaries  that could blow up easily.
>>    
>>
>Well, I meant that file to be read-only. Just a quick concatentated view
>for reading.
>  
>
The quick hack might be useful in certain situations. But the really 
interesting way to do it would be not to distinguish beween "actual" and 
"non-actual" files. All of them should be equally "actual", it is just 
that containment (possibly even overlap) would be allowed. The tree 
structure used by a file system such as Reiser4 would make this very 
efficient with each "sub-file" corresponding to a key-range. Writing a 
chapter should change the book that the chapter is part of. That is what 
would make it really valuable. Of course it would have all sorts of 
implications (e.g. for metadata for each part) that need to be thought 
about, but it could be done properly, I think.   Peter

Re: creating live virtual files by concatenation

[Rik van Riel]

On Sat, 25 Feb 2006, Peter Foldiak wrote:

> "sub-file" corresponding to a key-range. Writing a chapter should change the
> book that the chapter is part of. That is what would make it really valuable.
> Of course it would have all sorts of implications (e.g. for metadata for each
> part) that need to be thought about, but it could be done properly, I think.

What happens if you read the first 10kB of a file,
and one of the "chapters" behind your read cursor
grows?

Do you read part of the same data again when you
continue reading?

Does the read cursor automatically advance?

Your idea changes the way userspace expects files
to behave...

-- 
All Rights Reversed

Re: creating live virtual files by concatenation

[Peter Foldiak]

Rik van Riel wrote:

>On Sat, 25 Feb 2006, Peter Foldiak wrote:
>
>>sub-file" corresponding to a key-range. Writing a chapter should change the
>>book that the chapter is part of. That is what would make it really valuable.
>>Of course it would have all sorts of implications (e.g. for metadata for each
>>part) that need to be thought about, but it could be done properly, I think.
>>    
>>
>
>What happens if you read the first 10kB of a file,
>and one of the "chapters" behind your read cursor
>grows?
>
>Do you read part of the same data again when you
>continue reading?
>
>Does the read cursor automatically advance?
>  
>
You should probably continue where you left off .

>Your idea changes the way userspace expects files
>to behave...
>  
>
Yes, I know.

Re: creating live virtual files by concatenation

[Yoanis Gil Delgado]

Jan Engelhardt wrote:

>>Now let us say I am creating sort of a virtual text file (code.js)
>>that is a live-concatenation of these files:
>># concatenate tooltip.js banner.js foo.js code.js
>>
>>Note I am not talking about the cat(1) utility. I am thinking of
>>code.js be always a live concatenated version of these three, so when
>>I modify one file, the live-version is also modified.
>>
>>What puprose I might have? Network-related. Say, I have an HTML file
>>that includes these three files in its code.
>>
>>    
>>
>Try FUSE.
>  
>
Yes that's the best solution. Email me if you have a question about how 
to accomplish this. Here at
our school we have created a fuse filesystem that "glues" files in a 
single one.

>  
>
>>If I had a live-concatenated file, I could reference it in the HTML file
>>so that the browser does not have to download three files but just one.
>>
>>This would surely reduce network overhead of downloading the same amount
>>of data but within just one connection, reduce resource usage on the client
>>and possibly (depending on implementation) reduce the cost of accessing
>>three individual files on the server.
>>
>>    
>>
>Have you ever heard of persistent connections with HTTP/1.1?
>
>
>Jan Engelhardt
>  
>

Re: creating live virtual files by concatenation

[Hans Reiser]

Pretty much any simple basic tool or extension can be lived without. 
For them to say that you can do without it is to say nothing.  There is
little one actually needs a computer to do, I remember professors
explaining to me that they felt essays written on a typewritter tended
to be better written (it was probably due to them being written by
humanities students.;-) ) than those produced using troff.

Re: creating live virtual files by concatenation

[Hans Reiser]

Rik van Riel wrote:

>On Sat, 25 Feb 2006, Peter Foldiak wrote:
>
>  
>
>>"sub-file" corresponding to a key-range. Writing a chapter should change the
>>book that the chapter is part of. That is what would make it really valuable.
>>Of course it would have all sorts of implications (e.g. for metadata for each
>>part) that need to be thought about, but it could be done properly, I think.
>>    
>>
>
>What happens if you read the first 10kB of a file,
>and one of the "chapters" behind your read cursor
>grows?
>  
>
That is why you have magic delimiters, like the colons and newlines in
/etc/passwd.  You also specify the delimiters in your inheritance
syntax.  You don't want it to really all be one file, because you want
separation of modification times and acls that let users change their
own gecos field and etc., but there are times when you want to act on
/etc/passwd as one file.

>Do you read part of the same data again when you
>continue reading?
>
>Does the read cursor automatically advance?
>
>Your idea changes the way userspace expects files
>to behave...
>
>  
>

Re: creating live virtual files by concatenation

[Jesper Juhl]

On 2/27/06, Hans Reiser <reiser@namesys.com> wrote:
> Pretty much any simple basic tool or extension can be lived without.
> For them to say that you can do without it is to say nothing.  There is

All I said was that I don't *think* it will be of much use, that's not
to say that it *won't* be.

As a read-only thing I can certainly see some uses of it - Maciej
pointed out some nice ideas, sure, but wether or not they are actually
useful when you try to do real work needs to be tested. I for one need
to see it and play with it to determine if it will actually be useful.

Let's see an implementation and see how it works out for real...

--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html