Re: [patch 00/2] improve .text size on gcc 4.0 and newer compilers

[Martin Bligh <mbligh@mbligh.org>]

> these 30 functions involve roughly 10-15 .o files, so we've got 2-3 
> functions per .o file only. And that's typical for VFS, IO, networking 
> and lots of other syscall types. So if the full kernel image is 3MB, and 
> we've got 30 functions totalling to 3000 bytes, they are spread out in 
> 10-15 groups right now - creating 10-15 split icache lines. (in reality 
> we have in excess of 20 split icache-lines, due to weak cohesion even 
> within .o files) With 64-byte lines that's 320-480 bytes 'lost' due to 
> fragmentation alone, with 128-byte lines it's 640-960 bytes - which is 
> 10%-21% in the 64-byte case, and 21%-42% in the 128-byte case. I.e. the 
> icache bloat just due to the placement is quite significant. Adding 
> .o-level fragmentation plus inter-function inactive code to the mix can 
> easily baloon this even higher. Plus the current method of doing 
> unlikely() means the unlikely instructions are near the end of the 
> function - so they still 'spread apart' the footprint and thus have a 
> nontrivial icache cost.

mmm. will take me a little time to digest that.

But we were just discussing here ... wouldn't it be worth moving 
"unlikely" sections of code completely out of line? If they were calls 
to separate functions, all this optimisation stuff could just work at a 
function level, and would be pretty trivial to do?

ie instead of:

if (unlikely(conditon)) {
	do;
	some;
	stuff;
	BUG();
	error();
	oh_dear();
}


we'd have


if (unlikely(conditon)) {
	call_oh_shit();
}

__rarely_called void call_oh_shit()
{
	do;
	some;
	stuff;
	BUG();
	error();
	oh_dear();
}

depends how long they are, I suppose. Moving that out of line would seem 
to make more difference to icache footprint to me than just cacheline 
packing functions.

M.