tbench regression on each kernel release from 2.6.22 -> 2.6.28

tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Christoph Lameter]

It seems that the network stack becomes slower over time? Here is a list of
tbench results with various kernel versions:

2.6.22		3207.77 mb/sec
2.6.24		3185.66
2.6.25		2848.83
2.6.26		2706.09
2.6.27(rc2)	2571.03

And linux-next is:

2.6.28(l-next)	2568.74

It shows that there is still have work to be done on linux-next. Too close to
upstream in performance.

Note the KT event between 2.6.24 and 2.6.25. Why is that?

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Kok, Auke]

Christoph Lameter wrote:
> It seems that the network stack becomes slower over time? Here is a list of
> tbench results with various kernel versions:
> 
> 2.6.22		3207.77 mb/sec
> 2.6.24		3185.66
> 2.6.25		2848.83
> 2.6.26		2706.09
> 2.6.27(rc2)	2571.03
> 
> And linux-next is:
> 
> 2.6.28(l-next)	2568.74
> 
> It shows that there is still have work to be done on linux-next. Too close to
> upstream in performance.
> 
> Note the KT event between 2.6.24 and 2.6.25. Why is that?

is this with SLAB or with SLUB? SLUB has been known to impact network performance...

Auke

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Christoph Lameter]

Kok, Auke wrote:

> is this with SLAB or with SLUB? SLUB has been known to impact network performance...

The original testing config was SLAB based so it was used throughout.

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[David Miller]

From: Christoph Lameter <cl@linux-foundation.org>
Date: Mon, 11 Aug 2008 13:36:38 -0500

> It seems that the network stack becomes slower over time? Here is a list of
> tbench results with various kernel versions:
> 
> 2.6.22		3207.77 mb/sec
> 2.6.24		3185.66
> 2.6.25		2848.83
> 2.6.26		2706.09
> 2.6.27(rc2)	2571.03
> 
> And linux-next is:
> 
> 2.6.28(l-next)	2568.74
> 
> It shows that there is still have work to be done on linux-next. Too close to
> upstream in performance.
> 
> Note the KT event between 2.6.24 and 2.6.25. Why is that?

Isn't that when some major scheduler changes went in?  I'm not blaming
the scheduler, but rather I'm making the point that there are other
subsystems in the kernel that the networking interacts with that
influences performance at such a low level.  This includes the memory
allocator :-)

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Christoph Lameter]

David Miller wrote:

> Isn't that when some major scheduler changes went in?  I'm not blaming
> the scheduler, but rather I'm making the point that there are other
> subsystems in the kernel that the networking interacts with that
> influences performance at such a low level.  This includes the memory
> allocator :-)

Right this covers a significant portion of the kernel. SLAB was used since .22
was pretty early for SLUB. And around 2.6.24 we had the merges of the antifrag
logic.

.25 was the point where HR timers came in. By switching off hrtimers I can get
some (minor) portion of performance back. There must be more things in play
though.

Maybe what we are seeing is general bloat in kernel execution paths due to the
growth in complexity?

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[David Miller]

From: Christoph Lameter <cl@linux-foundation.org>
Date: Mon, 11 Aug 2008 16:33:43 -0500

> Maybe what we are seeing is general bloat in kernel execution paths
> due to the growth in complexity?

It could be, and any kind of analysis into this would be great.

I had a change that RCU destroyed sockets and this added a
tiny bit of latency, so I never added it even though it would
have allowed a lot of simplification of socket handling (which
I though would make up for RCU's latency, but it didn't).

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Kok, Auke]

David Miller wrote:
> From: Christoph Lameter <cl@linux-foundation.org>
> Date: Mon, 11 Aug 2008 16:33:43 -0500
> 
>> Maybe what we are seeing is general bloat in kernel execution paths
>> due to the growth in complexity?
> 
> It could be, and any kind of analysis into this would be great.

perhaps Rick Jones who maintains netperf could enlighten us on some historic
numbers? he usually seems to be happy to prop up new netperf numbers :)


Auke

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Rick Jones]

Kok, Auke wrote:
> David Miller wrote:
> 
>>From: Christoph Lameter <cl@linux-foundation.org>
>>Date: Mon, 11 Aug 2008 16:33:43 -0500
>>
>>
>>>Maybe what we are seeing is general bloat in kernel execution paths
>>>due to the growth in complexity?
>>
>>It could be, and any kind of analysis into this would be great.
> 
> 
> perhaps Rick Jones who maintains netperf could enlighten us on some historic
> numbers? he usually seems to be happy to prop up new netperf numbers :)

While this is an excellent opening to talk about how netperf 
top-of-trunk can now emit keyword=value results easier (ostensibly) to 
put into a database then the regular or even CSV output formats, I 
cannot fully exploit it by pointing at a database of results :(

rick jones

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Andi Kleen]

Christoph Lameter <cl@linux-foundation.org> writes:

> Maybe what we are seeing is general bloat in kernel execution paths
> due to the growth in complexity?

Wouldn't surprise me. Have you considered doing profiles? 

e.g. just oprofiling the benchmark on the different kernels and see
if there's some obvious difference in the CPU consumers? 

-Andi

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Christoph Lameter]

Andi Kleen wrote:
> Christoph Lameter <cl@linux-foundation.org> writes:
> 
>> Maybe what we are seeing is general bloat in kernel execution paths
>> due to the growth in complexity?
> 
> Wouldn't surprise me. Have you considered doing profiles? 

If I get the time I will try to do that.

Another way to understand why we are accepting the regressions here may be
that we give more consideration to real time issues and deterministic
performance these days. Hardware speed gains compensate for the additional
bloat? (I ran the old kernels on cutting edge hardware after all).

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Ilpo Järvinen]

On Mon, 11 Aug 2008, David Miller wrote:

> From: Christoph Lameter <cl@linux-foundation.org>
> Date: Mon, 11 Aug 2008 13:36:38 -0500
> 
> > It seems that the network stack becomes slower over time? Here is a list of
> > tbench results with various kernel versions:
> > 
> > 2.6.22		3207.77 mb/sec
> > 2.6.24		3185.66
> > 2.6.25		2848.83
> > 2.6.26		2706.09
> > 2.6.27(rc2)	2571.03
> > 
> > And linux-next is:
> > 
> > 2.6.28(l-next)	2568.74
> > 
> > It shows that there is still have work to be done on linux-next. Too close to
> > upstream in performance.
> > 
> > Note the KT event between 2.6.24 and 2.6.25. Why is that?
> 
> Isn't that when some major scheduler changes went in?  I'm not blaming
> the scheduler, but rather I'm making the point that there are other
> subsystems in the kernel that the networking interacts with that
> influences performance at such a low level.

...IIRC, somebody in the past did even bisect his (probably netperf) 
2.6.24-25 regression to some scheduler change (obviously it might or might 
not be related to this case of yours)...


-- 
 i.

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Zhang, Yanmin]

On Tue, 2008-08-12 at 11:13 +0300, Ilpo Järvinen wrote: 
> On Mon, 11 Aug 2008, David Miller wrote:
> 
> > From: Christoph Lameter <cl@linux-foundation.org>
> > Date: Mon, 11 Aug 2008 13:36:38 -0500
> > 
> > > It seems that the network stack becomes slower over time? Here is a list of
> > > tbench results with various kernel versions:
> > > 
> > > 2.6.22		3207.77 mb/sec
> > > 2.6.24		3185.66
> > > 2.6.25		2848.83
> > > 2.6.26		2706.09
> > > 2.6.27(rc2)	2571.03
> > > 
> > > And linux-next is:
> > > 
> > > 2.6.28(l-next)	2568.74
> > > 
> > > It shows that there is still have work to be done on linux-next. Too close to
> > > upstream in performance.
> > > 
> > > Note the KT event between 2.6.24 and 2.6.25. Why is that?
> > 
> > Isn't that when some major scheduler changes went in?  I'm not blaming
> > the scheduler, but rather I'm making the point that there are other
> > subsystems in the kernel that the networking interacts with that
> > influences performance at such a low level.
> 
> ...IIRC, somebody in the past did even bisect his (probably netperf) 
> 2.6.24-25 regression to some scheduler change (obviously it might or might 
> not be related to this case of yours)...
I did find much regression with netperf TCP-RR-1/UDP-RR-1/UDP-RR-512. I start
1 serve and 1 client while binding them to a different logical processor in
different physical cpu.

Comparing with 2.6.22, the regression of TCP-RR-1 on 16-core tigerton is:
2.6.23		6%
2.6.24		6%
2.6.25		9.7%
2.6.26		14.5%
2.6.27-rc1	22%

Other regressions on other machines are similar.

yanmin

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2118 bytes --]

On Mon, 18 Aug 2008, Zhang, Yanmin wrote:

> 
> On Tue, 2008-08-12 at 11:13 +0300, Ilpo Järvinen wrote: 
> > On Mon, 11 Aug 2008, David Miller wrote:
> > 
> > > From: Christoph Lameter <cl@linux-foundation.org>
> > > Date: Mon, 11 Aug 2008 13:36:38 -0500
> > > 
> > > > It seems that the network stack becomes slower over time? Here is a list of
> > > > tbench results with various kernel versions:
> > > > 
> > > > 2.6.22		3207.77 mb/sec
> > > > 2.6.24		3185.66
> > > > 2.6.25		2848.83
> > > > 2.6.26		2706.09
> > > > 2.6.27(rc2)	2571.03
> > > > 
> > > > And linux-next is:
> > > > 
> > > > 2.6.28(l-next)	2568.74
> > > > 
> > > > It shows that there is still have work to be done on linux-next. Too close to
> > > > upstream in performance.
> > > > 
> > > > Note the KT event between 2.6.24 and 2.6.25. Why is that?
> > > 
> > > Isn't that when some major scheduler changes went in?  I'm not blaming
> > > the scheduler, but rather I'm making the point that there are other
> > > subsystems in the kernel that the networking interacts with that
> > > influences performance at such a low level.
> > 
> > ...IIRC, somebody in the past did even bisect his (probably netperf) 
> > 2.6.24-25 regression to some scheduler change (obviously it might or might 
> > not be related to this case of yours)...
> I did find much regression with netperf TCP-RR-1/UDP-RR-1/UDP-RR-512. I start
> 1 serve and 1 client while binding them to a different logical processor in
> different physical cpu.
> 
> Comparing with 2.6.22, the regression of TCP-RR-1 on 16-core tigerton is:
> 2.6.23		6%
> 2.6.24		6%
> 2.6.25		9.7%
> 2.6.26		14.5%
> 2.6.27-rc1	22%
> 
> Other regressions on other machines are similar.

I btw reorganized tcp_sock for 2.6.26, it shouldn't cause this but it's 
not always obvious what even a small change in field ordering does for 
performance (it's b79eeeb9e48457579cb742cd02e162fcd673c4a3 in case you 
want to check that).

Also, there was this 83f36f3f35f4f83fa346bfff58a5deabc78370e5 fix to 
current -rcs but I guess it might not be that significant in your case 
(but I don't know well enough :-)).

-- 
 i.

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Zhang, Yanmin]

On Mon, 2008-08-18 at 10:53 +0300, Ilpo Järvinen wrote:
> On Mon, 18 Aug 2008, Zhang, Yanmin wrote:
> 
> > 
> > On Tue, 2008-08-12 at 11:13 +0300, Ilpo Järvinen wrote: 
> > > On Mon, 11 Aug 2008, David Miller wrote:
> > > 
> > > > From: Christoph Lameter <cl@linux-foundation.org>
> > > > Date: Mon, 11 Aug 2008 13:36:38 -0500
> > > > 
> > > > > It seems that the network stack becomes slower over time? Here is a list of
> > > > > tbench results with various kernel versions:
> > > > > 
> > > > > 2.6.22		3207.77 mb/sec
> > > > > 2.6.24		3185.66
> > > > > 2.6.25		2848.83
> > > > > 2.6.26		2706.09
> > > > > 2.6.27(rc2)	2571.03
> > > > > 
> > > > > And linux-next is:
> > > > > 
> > > > > 2.6.28(l-next)	2568.74
> > > > > 
> > > > > It shows that there is still have work to be done on linux-next. Too close to
> > > > > upstream in performance.
> > > > > 
> > > > > Note the KT event between 2.6.24 and 2.6.25. Why is that?
> > > > 
> > > > Isn't that when some major scheduler changes went in?  I'm not blaming
> > > > the scheduler, but rather I'm making the point that there are other
> > > > subsystems in the kernel that the networking interacts with that
> > > > influences performance at such a low level.
> > > 
> > > ...IIRC, somebody in the past did even bisect his (probably netperf) 
> > > 2.6.24-25 regression to some scheduler change (obviously it might or might 
> > > not be related to this case of yours)...
> > I did find much regression with netperf TCP-RR-1/UDP-RR-1/UDP-RR-512. I start
> > 1 serve and 1 client while binding them to a different logical processor in
> > different physical cpu.
> > 
> > Comparing with 2.6.22, the regression of TCP-RR-1 on 16-core tigerton is:
> > 2.6.23		6%
> > 2.6.24		6%
> > 2.6.25		9.7%
> > 2.6.26		14.5%
> > 2.6.27-rc1	22%
> > 
> > Other regressions on other machines are similar.
> 
> I btw reorganized tcp_sock for 2.6.26, it shouldn't cause this but it's 
> not always obvious what even a small change in field ordering does for 
> performance (it's b79eeeb9e48457579cb742cd02e162fcd673c4a3 in case you 
> want to check that).
> 
> Also, there was this 83f36f3f35f4f83fa346bfff58a5deabc78370e5 fix to 
> current -rcs but I guess it might not be that significant in your case 
> (but I don't know well enough :-)).
I reverted the patch against 2.6.27-rc1 and did a quick testing with netperf TCP-RR-1
and didn't find improvement. So your patch is good.
Mostly, I suspect process scheduler causes the regression. It seems when there are 
only 1 or 2 tasks running on the cpu, the performance isn't good. My netperf testing
is just one example.

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Christoph Lameter]

Zhang, Yanmin wrote:

> Other regressions on other machines are similar.

There are AIM7 regressions that are similar to tbench.

2.6.22	28436
2.6.26	23064

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Ray Lee]

On Mon, Aug 18, 2008 at 7:07 AM, Christoph Lameter
<cl@linux-foundation.org> wrote:
> Zhang, Yanmin wrote:
>
>> Other regressions on other machines are similar.
>
> There are AIM7 regressions that are similar to tbench.
>
> 2.6.22  28436
> 2.6.26  23064

Just a shot in the dark -- is this with Group Scheduling on or off?
Off is prefered for benchmarks.

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Christoph Lameter]

Ray Lee wrote:
> On Mon, Aug 18, 2008 at 7:07 AM, Christoph Lameter
> <cl@linux-foundation.org> wrote:
>> Zhang, Yanmin wrote:
>>
>>> Other regressions on other machines are similar.
>> There are AIM7 regressions that are similar to tbench.
>>
>> 2.6.22  28436
>> 2.6.26  23064
> 
> Just a shot in the dark -- is this with Group Scheduling on or off?
> Off is prefered for benchmarks.

Off

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Zhang, Yanmin]

On Mon, 2008-08-18 at 09:07 -0500, Christoph Lameter wrote:
> Zhang, Yanmin wrote:
> 
> > Other regressions on other machines are similar.
> 
> There are AIM7 regressions that are similar to tbench.
> 
> 2.6.22	28436
> 2.6.26	23064

Mostly, AIM7 has about 4~5% regression on my machines. As AIM7 result is stable,
so 4% is big. 

Re: tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Zhang, Yanmin]

On Mon, 2008-08-11 at 13:36 -0500, Christoph Lameter wrote:
> It seems that the network stack becomes slower over time? Here is a list of
> tbench results with various kernel versions:
> 
> 2.6.22		3207.77 mb/sec
> 2.6.24		3185.66
> 2.6.25		2848.83
> 2.6.26		2706.09
> 2.6.27(rc2)	2571.03
What's the hardware configuration? Is it dual-core?

I also track tbench performance with lastest kernels on a couple of quad-core machines,
and didn't find such regression while the results did have fluctuation.

What's the commandline you is using to start tbench? I start tbench with CPU_NUM*2.

BTW, I enabled CONFIG_SLUB since 2.6.22.

> 
> And linux-next is:
> 
> 2.6.28(l-next)	2568.74
> 
> It shows that there is still have work to be done on linux-next. Too close to
> upstream in performance.
> 
> Note the KT event between 2.6.24 and 2.6.25. Why is that?