Re: [PATCH] O_STREAMING - flag for optimal streaming I/O

[Larry McVoy <lm@bitmover.com>]

> What we need is to detect the situation where someone is linearly
> walking through a file which is preposterously too large to cache,
> and just start dropping it.

...

> The tricky part is designing the algorithm which decides when to
> pull the trigger.

I did a variation of this in SunOS years ago.  It did not have the
"big file" wrinkle you are suggesting, it worked like

	if (we are sequential AND
	    we are running low on memory AND
	    file size > 256K) {
	    	invalidate the pages behind me
	}

You use the same data structures which turn on read ahead to mean 
sequential access, that's obvious.

What this didn't fix was when you read a monstor file into memory and
then didn't do anything with it.  That would fill the page cache and 
the above alg would keep you from thrashing the machine but didn't 
flush the stale memory.

If I were to do it again, I'd maintain stats in the inode about 
access pattern, # of pages in ram for the inode, time of last I/O,
time of last page fault or read.  Then when memory is getting tight
you do a 

	foreach i (ALL INODES) {
		unless (i.accesspat == SEQ) continue;
		unless ((now() - i.pagefault) > STALE_TIME) continue;
		unless ((now() - i.io) > STALE_TIME) continue;
		flush_pages();
	}

You want to be a lot more clever than that because you'd like to have 
fudging in favor of the clean pages vs dirty pages, you more or less
end up wanting to go through the loop more than once, getting more and
more eager as you are more and more desparate for ram.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm