larger per cpu data

larger per cpu data

[Andi Kleen]

Hi,

I've been looking at optimizing the network per CPU statistics and the best way to 
do that would be to move them back to direct per CPU data. Currently they are
allocated separately because there used to be some trouble on x86 (and ia64?) 
with limited per CPU areas. At least the x86 side has been fixed now. I heard
there were some problems with ia64 have those been fixed too?
Anybody would have a problem with increasing per CPU data in a standard kernel
by a few KB?

-Andi
 

RE: larger per cpu data

[Luck, Tony]

> I heard there were some problems with ia64 have those been fixed too?

IA64 has a 64KB page allocated for each cpu for the percpu data. Right
now we are using 28K-32K for the arch/ia64/configs/* variations.

> Anybody would have a problem with increasing per CPU data in a
> standard kernel by a few KB?

How many is "a few" ... and much more importantly is your few KB
a fixed amount, or does it scale up with the number of network cards
(or anything else that there might be a lot of on a large system)?

A fixed extra 8-10K should be fine.  #NIC * 1K would cause problems.

-Tony
 

Re: larger per cpu data

[Andi Kleen]

On Friday 20 April 2007 20:23:13 Luck, Tony wrote:
> > I heard there were some problems with ia64 have those been fixed too?
> 
> IA64 has a 64KB page allocated for each cpu for the percpu data. Right
> now we are using 28K-32K for the arch/ia64/configs/* variations.

Could that be enlarged without too much trouble?

> > Anybody would have a problem with increasing per CPU data in a
> > standard kernel by a few KB?
> 
> How many is "a few" ... and much more importantly is your few KB
> a fixed amount, or does it scale up with the number of network cards
> (or anything else that there might be a lot of on a large system)?

> A fixed extra 8-10K should be fine.  #NIC * 1K would cause problems.

There are scalable parts, but those would likely stay allocated.
==

-Andi

RE: larger per cpu data

[Luck, Tony]

> > IA64 has a 64KB page allocated for each cpu for the percpu data. Right
> > now we are using 28K-32K for the arch/ia64/configs/* variations.
>
> Could that be enlarged without too much trouble?

If we stick to mapping it with a single TLB entry, then the next
size up is 256K ... which is a pretty big jump.  But the code that
does the mapping is just about to be changed from a single locked
TLB entry to a map-on-tlb-miss version (thanks to Ken Chen). Some
small-ish additions to that code would allow an expansion to 128K
by mapping each half with a 64K TLB entry on demand.

-Tony

Re: larger per cpu data

[Eric W. Biederman]

Andi Kleen <ak@suse.de> writes:

> Hi,
>
> I've been looking at optimizing the network per CPU statistics and the best way
> to
> do that would be to move them back to direct per CPU data. Currently they are
> allocated separately because there used to be some trouble on x86 (and ia64?) 
> with limited per CPU areas. At least the x86 side has been fixed now. I heard
> there were some problems with ia64 have those been fixed too?
> Anybody would have a problem with increasing per CPU data in a standard kernel
> by a few KB?

Andi which areas are you looking at modifying.  I'm trying to get a feel
to see if your changes will have an effect on the network namespace work.

Eric

Re: larger per cpu data

[Andi Kleen]

On Saturday 21 April 2007 22:23:21 Eric W. Biederman wrote:
> Andi Kleen <ak@suse.de> writes:
> 
> > Hi,
> >
> > I've been looking at optimizing the network per CPU statistics and the best way
> > to
> > do that would be to move them back to direct per CPU data. Currently they are
> > allocated separately because there used to be some trouble on x86 (and ia64?) 
> > with limited per CPU areas. At least the x86 side has been fixed now. I heard
> > there were some problems with ia64 have those been fixed too?
> > Anybody would have a problem with increasing per CPU data in a standard kernel
> > by a few KB?
> 
> Andi which areas are you looking at modifying.  I'm trying to get a feel
> to see if your changes will have an effect on the network namespace work.


Just the internal implementation of the statistics. Not very many changes
outside those include files

-Andi

Re: larger per cpu data

[Eric W. Biederman]

Andi Kleen <ak@suse.de> writes:

> Just the internal implementation of the statistics. Not very many changes
> outside those include files

Ok. I guess I'm dense at the moment.  Which include files?

The only statistics I can think of at the moment are per network device,
and that doesn't sound like something that should be made per cpu.

Eric

Re: larger per cpu data

[Jes Sorensen]

>>>>> "Tony" == Luck, Tony <tony.luck@intel.com> writes:

>> > IA64 has a 64KB page allocated for each cpu for the percpu
>> data. Right > now we are using 28K-32K for the arch/ia64/configs/*
>> variations.
>> 
>> Could that be enlarged without too much trouble?

Tony> If we stick to mapping it with a single TLB entry, then the next
Tony> size up is 256K ... which is a pretty big jump.  But the code
Tony> that does the mapping is just about to be changed from a single
Tony> locked TLB entry to a map-on-tlb-miss version (thanks to Ken
Tony> Chen). Some small-ish additions to that code would allow an
Tony> expansion to 128K by mapping each half with a 64K TLB entry on
Tony> demand.

256KB * 1024 CPUs (or more) == verymightyhuge ..... I certainly
wouldn't like to see this.

Cheers,
Jes

RE: larger per cpu data

[Luck, Tony]

> 256KB * 1024 CPUs (or more) == verymightyhuge ..... I certainly
> wouldn't like to see this.

"Only" 256M :-)  That 1024 cpu system probably has 1TB of memory or
more ... so this isn't quite as terrible as it sounds (at least in
terms of percentage of memory used).

Are there customers buying high cpu count, low memory systems?

But unless these are some fabulously useful counters that Andi
wants to add ... I'd prefer to keep the ia64 percpu area at 64K
for a while longer.

-Tony

Re: larger per cpu data

[Jes Sorensen]

Luck, Tony wrote:
>> 256KB * 1024 CPUs (or more) == verymightyhuge ..... I certainly
>> wouldn't like to see this.
> 
> "Only" 256M :-)  That 1024 cpu system probably has 1TB of memory or
> more ... so this isn't quite as terrible as it sounds (at least in
> terms of percentage of memory used).
> 
> Are there customers buying high cpu count, low memory systems?

I doubt it, but I don't have numbers. However 256M is still a good chunk
of memory that could be put to better use.

> But unless these are some fabulously useful counters that Andi
> wants to add ... I'd prefer to keep the ia64 percpu area at 64K
> for a while longer.

Nod, my thoughts exactly.

Cheers,
Jes