64bit kernel is huge

64bit kernel is huge

[Anton Blanchard]

Hi,

I've found at least one machine that wont boot 2.6.31-rc* with a 
pseries_defconfig. If I move real-base from 0xc00000 to 0xd00000 it
boots fine.

# size vmlinux
   text	   data	    bss	    dec	    hex	filename
9812942	1982496	1105228	12900666	 c4d93a	vmlinux

Looks like we blow right through the 12MB mark. It desperately needs to eat
less and lose weight.

Here are some of the problem areas:

788224  kallsyms_names
537600  kallsyms_addresses

I guess mostly CONFIG_KALLSYMS_ALL.

262144  kstat_irqs_all
131072  irq_desc
16384   irq_stat

Could we dynamically allocate our irq structures?

151912  __start_mcount_loc
131076  map_pid_to_cmdline

ftrace stuff. With a name like map_pid_to_cmdline I'm pretty sure I'm not
going to like what it does.

131072  lppaca
65536   paca

I think we've attacked these before, not sure if there is anything left
we can trim.

131072  __log_buf

Looks like we can dynamically allocate a large log buf at runtime. Perhaps
we should default to a small log_buf and grow it at boot based on machine size
(eg max cpus).

101160  powerpc_opcodes

xmon instruction disassembly. I'd probably prefer to cut off my right hand and
debug one handed before losing this though.

87600   __start___bug_table

Can't do much about this.

77452   _fw_acenic_tg2_bin_bin

We could probably change acenic to be a module, 

46464   kmalloc_caches
32768   read_buffers
32768   mem_section
21816   hstates
20480   node_devices
14336   bootmem_node_data

mm stuff.

32096   _fw_tigon_tg3_tso5_bin_name
12184   .tg3_reset_hw
10124   .tg3_get_invariants

tg3

20804   .radeonfb_pm_init
18412   .radeon_pm_disable_dynamic_mode
15280   .radeon_write_mode

Radeon

26288   __start___ex_table
22632   PPC64_CACHES
20564   alg_test_descs
17784   ioctl_start
16384   xics_ipi_message
16384   vdso64_end
16384   stateid_hashtbl
16384   raw_devices
16384   lockstateid_hashtbl
16384   kexec_stack
16384   init_thread_union
16384   boot_pageset
15360   latency_record
14880   .do_balance
14008   .hidinput_connect
11920   kernel_config_data
10240   futex_queues
10044   .e1000_init_hw

Anton

Re: 64bit kernel is huge

[Benjamin Herrenschmidt]

On Mon, 2009-09-28 at 17:45 +1000, Anton Blanchard wrote:
> Hi,
> 
> I've found at least one machine that wont boot 2.6.31-rc* with a 
> pseries_defconfig. If I move real-base from 0xc00000 to 0xd00000 it
> boots fine.
> 
> # size vmlinux
>    text	   data	    bss	    dec	    hex	filename
> 9812942	1982496	1105228	12900666	 c4d93a	vmlinux
> 
> Looks like we blow right through the 12MB mark. It desperately needs to eat
> less and lose weight.

Depends what you have enabled tho. Known killers are CONFIG_RELOCATABLE,
the FTRACE stuff.

> Here are some of the problem areas:
> 
> 788224  kallsyms_names
> 537600  kallsyms_addresses
> 
> I guess mostly CONFIG_KALLSYMS_ALL.

Yeah, those are full on. Maybe we could compress them ? It should
compress well... but then, zImage does that already.

> 262144  kstat_irqs_all
> 131072  irq_desc
> 16384   irq_stat
> 
> Could we dynamically allocate our irq structures?

We still want one big array, unless we go to sparse IRQ numbering like
x86 but we'd have to also adapt the remapping stuff. Definitely to put
on a list somewhere for people who want to pick up something to do :-)

It's hard to properly dynamically size it. I'd rather have a "capacity"
of _lots_ and sparsely populate the array (a tree ?) because we never
know with MSIs etc... how many we'll really need.

At the -very-least- we could make NR_IRQS a CONFIG option.

> 151912  __start_mcount_loc
> 131076  map_pid_to_cmdline
> 
> ftrace stuff. With a name like map_pid_to_cmdline I'm pretty sure I'm not
> going to like what it does.

:-) No idea in fact

> 131072  lppaca
> 65536   paca
> 
> I think we've attacked these before, not sure if there is anything left
> we can trim.

Doubt it.

> 131072  __log_buf
> 
> Looks like we can dynamically allocate a large log buf at runtime. Perhaps
> we should default to a small log_buf and grow it at boot based on machine size
> (eg max cpus).

Ah, it's a new feature I have missed.

> 101160  powerpc_opcodes
> 
> xmon instruction disassembly. I'd probably prefer to cut off my right hand and
> debug one handed before losing this though.

It's already a CONFIG_OPTION for those who prefer coding with their
feet :-)

> 87600   __start___bug_table
> 
> Can't do much about this.

Appart from having no bugs :-)

> 77452   _fw_acenic_tg2_bin_bin
> 
> We could probably change acenic to be a module, 

Right.

> 46464   kmalloc_caches
> 32768   read_buffers
> 32768   mem_section
> 21816   hstates
> 20480   node_devices
> 14336   bootmem_node_data
> 
> mm stuff.

Also if ftrace is enabled, -pg is going to bloat the shit our of
everything (and slow everything down, gcc becomes really silly)

Cheers,
Ben.

Re: 64bit kernel is huge

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]

On Mon, 2009-09-28 at 18:07 +1000, Benjamin Herrenschmidt wrote:
> On Mon, 2009-09-28 at 17:45 +1000, Anton Blanchard wrote:
> > Hi,
> > 
> > I've found at least one machine that wont boot 2.6.31-rc* with a 
> > pseries_defconfig. If I move real-base from 0xc00000 to 0xd00000 it
> > boots fine.
> > 
> > # size vmlinux
> >    text	   data	    bss	    dec	    hex	filename
> > 9812942	1982496	1105228	12900666	 c4d93a	vmlinux
> > 
> > Looks like we blow right through the 12MB mark. It desperately needs to eat
> > less and lose weight.

> > 262144  kstat_irqs_all
> > 131072  irq_desc
> > 16384   irq_stat
> > 
> > Could we dynamically allocate our irq structures?
> 
> We still want one big array, unless we go to sparse IRQ numbering like
> x86 but we'd have to also adapt the remapping stuff. Definitely to put
> on a list somewhere for people who want to pick up something to do :-)

Actually I've already been looking at it. The easy step is to enable
SPARSE_IRQ, the harder step is to get rid of our irq_map. But that's not
that big.

I'll try and get them polished for the next merge window.

> It's hard to properly dynamically size it. I'd rather have a "capacity"
> of _lots_ and sparsely populate the array (a tree ?) because we never
> know with MSIs etc... how many we'll really need.
> 
> At the -very-least- we could make NR_IRQS a CONFIG option.

Yeah I have a patch for that too, I'll post it. It's no good for distros
but for custom kernels it could be quite handy, small partitions with
virtual most things don't really need many interrupts.

cheers

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: 64bit kernel is huge

[Michael Neuling]

> # size vmlinux
>    text	   data	    bss	    dec	    hex	filename
> 9812942	1982496	1105228	12900666	 c4d93a	vmlinux

... over kernel releases with pseries_defconfig:

           text		data	    bss	    dec		hex	
2.6.32-rc1 9872090      1971184 1105236 12948510        c5941e
2.6.31     9730495	1708796	1054236	12493527	bea2d7
2.6.30     9221595	1620728	1052812	11895135	b5815f
2.6.29     9104807	1569840	 785292	11459939	aedd63
2.6.28     8647080	1699460	 780472	11127012	a9c8e4
2.6.27     7461663	1505796	 774400	9741859		94a623
2.6.26     7192966	1470432	 631024	9294422		8dd256
2.6.25     7163913	1558800	 568048	9290761		8dc409

We are getting bigger pretty quickly :-(

Mikey

Re: 64bit kernel is huge

[Arnd Bergmann]

On Monday 28 September 2009, Anton Blanchard wrote:

> 262144  kstat_irqs_all
> 131072  irq_desc
> 16384   irq_stat
> 
> Could we dynamically allocate our irq structures?

There were patches floating around for that a few years ago,
but I haven't seen anyone working on it since.
 
> 131072  lppaca
> 65536   paca
> 
> I think we've attacked these before, not sure if there is anything left
> we can trim.

well, it essentially comes down to CONFIG_NR_CPUS. If you build with NR_CPUS=4,
this is really small.
 
> 131072  __log_buf
> 
> Looks like we can dynamically allocate a large log buf at runtime. Perhaps
> we should default to a small log_buf and grow it at boot based on machine size
> (eg max cpus).

I like the idea of being able to very easily find the log buffer in
dump files or in hw debuggers for bringup, but it's probably not
worth it for production systems. A config option might be nice.

> 101160  powerpc_opcodes
> 
> xmon instruction disassembly. I'd probably prefer to cut off my right hand and
> debug one handed before losing this though.

This should be very compressible though, which will help binary size, but not
runtime memory footprint.

> 77452   _fw_acenic_tg2_bin_bin
> 
> We could probably change acenic to be a module, 

CONFIG_FIRMWARE_IN_KERNEL ?

> 17784   ioctl_start

I have a patch for this one that I need to submit some day.

	Arnd <><

Re: 64bit kernel is huge

[Benjamin Herrenschmidt]

On Wed, 2009-09-30 at 16:01 +1000, Michael Neuling wrote:
> 2.6.30     9221595      1620728 1052812 11895135        b5815f
> 2.6.29     9104807      1569840  785292 11459939        aedd63 

The jump in BSS size is huge...  lockdep ?

Ben.

Re: 64bit kernel is huge

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 881 bytes --]

On Mon, 2009-09-28 at 17:45 +1000, Anton Blanchard wrote:
> Hi,
> 
> I've found at least one machine that wont boot 2.6.31-rc* with a 
> pseries_defconfig. If I move real-base from 0xc00000 to 0xd00000 it
> boots fine.
> 
> # size vmlinux
>    text	   data	    bss	    dec	    hex	filename
> 9812942	1982496	1105228	12900666	 c4d93a	vmlinux
> 
> Looks like we blow right through the 12MB mark. It desperately needs to eat
> less and lose weight.
> 
> Here are some of the problem areas:
> 
> 131072  lppaca
> 65536   paca
> 
> I think we've attacked these before, not sure if there is anything left
> we can trim.

Why can't we dynamically allocate all but one paca? I seem to recall
Mikey tried it but it didn't work?

And on !ISERIES we should be able to allocate the lppacas too I think,
the HV doesn't know about them until register_vpa().

cheers

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: 64bit kernel is huge

[Michael Neuling]

> 2.6.28     8647080	1699460	 780472	11127012	a9c8e4
> 2.6.27     7461663	1505796	 774400	9741859		94a623

If you compile 28 with the 27 pseries_config, you lose most of this
1.2MB text bloat.

If we remove just FUNCTION_TRACER and STACK_TRACER from 28 we gain back
about 600K.

Mikey

Re: 64bit kernel is huge

[Michael Neuling]

> > 2.6.28     8647080	1699460	 780472	11127012	a9c8e4
> > 2.6.27     7461663	1505796	 774400	9741859		94a623
> 
> If you compile 28 with the 27 pseries_config, you lose most of this
> 1.2MB text bloat.
> 
> If we remove just FUNCTION_TRACER and STACK_TRACER from 28 we gain back
> about 600K.

EXT4 also adds about 200K, between the 27 and 28 configs.

Mikey