Performance versus FreeBSD 7.0

Performance versus FreeBSD 7.0

[Stephen Cuppett]

Not lobbing the first artillery shell by any means, but I saw this
bullet on the FreeBSD release announcement:

# Dramatic improvements in performance and SMP scalability shown by
various database and other benchmarks, in some cases showing peak
performance improvements as high as 350% over FreeBSD 6.X under normal
loads and 1500% at high loads. When compared with the best performing
Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
from benchmarks used to analyze and improve system performance,
results with your specific work load may vary. Some of the changes
that contribute to this improvement are:

    * The 1:1 libthr threading model is now the default.
    * Finer-grained IPC, networking, and scheduler locking.
    * A major focus on optimizing the SMP architecture that was put in
place during the 5.x and 6.x branches.

Some benchmarks show linear scaling up to 8 CPUs. Many workloads see a
significant performance improvement with multicore systems.

The whole thing is available at:

http://www.freebsd.org/releases/7.0R/announce.html

However, their site is pretty scant on details about those benchmarks.
 Was just curious if anybody on this list had any input or links on
the metrics, the hardware, the workloads, how was Linux or FreeBSD
tuned, knew more information, could point me to the stats....

Re: Performance versus FreeBSD 7.0

[Diego Calleja]

El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett" <cuppett@gmail.com> escribió:

> loads and 1500% at high loads. When compared with the best performing
> Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are

There has been some performance problems with sysbench performance in linux
which made it slower than freebsd, there were some patches to speed things up,
not sure if they have been merged.

Myself I find fun that they look at Linux as the Leader That Must Be Surpassed.
All the performance highlights of freebsd 7.0 are things that linux already did some
years ago.

Re: Performance versus FreeBSD 7.0

[Andi Kleen]

"Stephen Cuppett" <cuppett@gmail.com> writes:

> Not lobbing the first artillery shell by any means, but I saw this
> bullet on the FreeBSD release announcement:

I posted a comment on that on the LWN thread, but their website 
has eaten it.

Anyways I would be very careful with any benchmarks testing mysql
especially on sysbench on larger systems (like they did with a 8 core
system). I did some similar testing on a slightly larger system (16 core) 
some time ago and found that mysql itself didn't scale at all.
With sysbench which only exercises a single table it would end up basically 
just banging on a single mysql user space lock and spend nearly all
the kernel time in the scheduler over-scheduling like crazy.

The FreeBSD people are not very clear on what they exactly benchmarked
(MySQL has lots of configuration options and engines), but I'll assume
they used the standard MyISAM.

I don't think that's a very useful benchmark to tune for in this
form at least not until MySQL is fixed to actually scale and sysbench
replaced with some a little more sane. It's worse than dbench. 

-Andi

Re: Performance versus FreeBSD 7.0

[Nick Piggin]

On Saturday 01 March 2008 01:54, Diego Calleja wrote:
> El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett" <cuppett@gmail.com> 
escribió:
> > loads and 1500% at high loads. When compared with the best performing
> > Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
>
> There has been some performance problems with sysbench performance in linux
> which made it slower than freebsd, there were some patches to speed things
> up, not sure if they have been merged.

There definitely were performance problems with threaded malloc/free
in the Linux kernel and glibc. Fixes have been merged in both packages,
and AFAIK the FreeBSD guys tested with those fixes in place.

I think these were never really run into before in part due to MySQL's
unscalable heap design makes it not scale well on higher numbers of
CPUs anyway, and also made the malloc problems more pronounced (ie.
they added a bit to the contention of the single heap lock, which is
the big killer here).

But it was very nice to be made aware of the problem and be able to
improve it.


> Myself I find fun that they look at Linux as the Leader That Must Be
> Surpassed. All the performance highlights of freebsd 7.0 are things that
> linux already did some years ago.

I don't know very much about FreeBSD nor have verified the results
for myself. But to their credit they seem to have done quite well at
least on the smaller end of the scale.

Re: Performance versus FreeBSD 7.0

[Cyrus Massoumi]

Nick Piggin wrote:
> On Saturday 01 March 2008 01:54, Diego Calleja wrote:
>> El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett" <cuppett@gmail.com> 
> escribió:
>>> loads and 1500% at high loads. When compared with the best performing
>>> Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
>> There has been some performance problems with sysbench performance in linux
>> which made it slower than freebsd, there were some patches to speed things
>> up, not sure if they have been merged.
> 
> There definitely were performance problems with threaded malloc/free
> in the Linux kernel and glibc. Fixes have been merged in both packages,
> and AFAIK the FreeBSD guys tested with those fixes in place.
> 
> I think these were never really run into before in part due to MySQL's
> unscalable heap design makes it not scale well on higher numbers of
> CPUs anyway, and also made the malloc problems more pronounced (ie.
> they added a bit to the contention of the single heap lock, which is
> the big killer here).

IIRC, going to fine-grained file locking gave them a huge boost in this 
particular benchmark (and maybe others).
As I said on lwn.net Peter Zijlstra posted a patch to break the global 
file list lock about a year ago [1], but I don't think it was ever 
merged. Here [2] are some numbers for the patchset.

[1] http://lkml.org/lkml/2007/1/28/29
[2] http://lkml.org/lkml/2007/1/28/116

> But it was very nice to be made aware of the problem and be able to
> improve it.
> 
> 
>> Myself I find fun that they look at Linux as the Leader That Must Be
>> Surpassed. All the performance highlights of freebsd 7.0 are things that
>> linux already did some years ago.
> 
> I don't know very much about FreeBSD nor have verified the results
> for myself. But to their credit they seem to have done quite well at
> least on the smaller end of the scale.

Re: Performance versus FreeBSD 7.0

[Nick Piggin]

On Tuesday 04 March 2008 06:02, Cyrus Massoumi wrote:
> Nick Piggin wrote:
> > On Saturday 01 March 2008 01:54, Diego Calleja wrote:
> >> El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett"
> >> <cuppett@gmail.com>
> >
> > escribió:
> >>> loads and 1500% at high loads. When compared with the best performing
> >>> Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
> >>
> >> There has been some performance problems with sysbench performance in
> >> linux which made it slower than freebsd, there were some patches to
> >> speed things up, not sure if they have been merged.
> >
> > There definitely were performance problems with threaded malloc/free
> > in the Linux kernel and glibc. Fixes have been merged in both packages,
> > and AFAIK the FreeBSD guys tested with those fixes in place.
> >
> > I think these were never really run into before in part due to MySQL's
> > unscalable heap design makes it not scale well on higher numbers of
> > CPUs anyway, and also made the malloc problems more pronounced (ie.
> > they added a bit to the contention of the single heap lock, which is
> > the big killer here).
>
> IIRC, going to fine-grained file locking gave them a huge boost in this
> particular benchmark (and maybe others).
> As I said on lwn.net Peter Zijlstra posted a patch to break the global
> file list lock about a year ago [1], but I don't think it was ever
> merged. Here [2] are some numbers for the patchset.
>
> [1] http://lkml.org/lkml/2007/1/28/29
> [2] http://lkml.org/lkml/2007/1/28/116

The mysql workload is not constrained by this lock. The important
part (where freebsd saw their gain) is probably fd lookups, which
are lockless in Linux for a long time.

Re: Performance versus FreeBSD 7.0

[Cyrus Massoumi]

Nick Piggin wrote:
> On Tuesday 04 March 2008 06:02, Cyrus Massoumi wrote:
>> Nick Piggin wrote:
>>> On Saturday 01 March 2008 01:54, Diego Calleja wrote:
>>>> El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett"
>>>> <cuppett@gmail.com>
>>> escribió:
>>>>> loads and 1500% at high loads. When compared with the best performing
>>>>> Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
>>>> There has been some performance problems with sysbench performance in
>>>> linux which made it slower than freebsd, there were some patches to
>>>> speed things up, not sure if they have been merged.
>>> There definitely were performance problems with threaded malloc/free
>>> in the Linux kernel and glibc. Fixes have been merged in both packages,
>>> and AFAIK the FreeBSD guys tested with those fixes in place.
>>>
>>> I think these were never really run into before in part due to MySQL's
>>> unscalable heap design makes it not scale well on higher numbers of
>>> CPUs anyway, and also made the malloc problems more pronounced (ie.
>>> they added a bit to the contention of the single heap lock, which is
>>> the big killer here).
>> IIRC, going to fine-grained file locking gave them a huge boost in this
>> particular benchmark (and maybe others).
>> As I said on lwn.net Peter Zijlstra posted a patch to break the global
>> file list lock about a year ago [1], but I don't think it was ever
>> merged. Here [2] are some numbers for the patchset.
>>
>> [1] http://lkml.org/lkml/2007/1/28/29
>> [2] http://lkml.org/lkml/2007/1/28/116
> 
> The mysql workload is not constrained by this lock. The important
> part (where freebsd saw their gain) is probably fd lookups, which
> are lockless in Linux for a long time.

Thanks for pointing this out Nick.

Re: Performance versus FreeBSD 7.0

[Andi Kleen]

Cyrus Massoumi <cyrusm@gmx.net> writes:
>
> IIRC, going to fine-grained file locking gave them a huge boost in
> this particular benchmark (and maybe others).

What locking? You mean their equivalent of i_mutex? There are 
already file systems on Linux that don't use it significantly
for write.

> As I said on lwn.net Peter Zijlstra posted a patch to break the global
> file list lock about a year ago [1], but I don't think it was ever
> merged. Here [2] are some numbers for the patchset.

The file_list_lock is only for open/close which MySQL is very unlikely
to do in its fast path. You're totally on the wrong path here I think.

-Andi

Re: Performance versus FreeBSD 7.0

[Cyrus Massoumi]

Andi Kleen wrote:
> Cyrus Massoumi <cyrusm@gmx.net> writes:
>> IIRC, going to fine-grained file locking gave them a huge boost in
>> this particular benchmark (and maybe others).
> 
> What locking? You mean their equivalent of i_mutex? There are 
> already file systems on Linux that don't use it significantly
> for write.
> 
>> As I said on lwn.net Peter Zijlstra posted a patch to break the global
>> file list lock about a year ago [1], but I don't think it was ever
>> merged. Here [2] are some numbers for the patchset.
> 
> The file_list_lock is only for open/close which MySQL is very unlikely
> to do in its fast path. You're totally on the wrong path here I think.

Yes, Nick already corrected me, sorry for the noise :)

Re: Performance versus FreeBSD 7.0

[Nick Piggin]

On Tuesday 04 March 2008 10:23, Andi Kleen wrote:

> You're totally on the wrong path here I think.

In my testing, I found that Linux is a bit faster and more scalable
than FreeBSD on MySQL sysbench.

http://www.kernel.org/pub/linux/kernel/people/npiggin/sysbench/

I think since the mmap_sem madvise problem was solved, the kernel
didn't really have any further problems with this workload.

It is doing a lot of context switches, no IO, and not a lot of
real work. The DBMS runs into some scalability problems in this
workload, but to be fair it is probably not one that the MySQL
guys care about too much. But it sometimes seems a bit sloppy...
eg. a full 35% of the syscalls the MySQL server makes are failed
sched_setparam calls that are passing in invalid values.

So I don't really consider it a big victory to be slightly faster
here; nobody really noticed until this test that we had the silly
performance bug in our malloc/free paths in glibc and the kernel.

Re: Performance versus FreeBSD 7.0

[Andi Kleen]

> I think since the mmap_sem madvise problem was solved, the kernel
> didn't really have any further problems with this workload.

Good.

> It is doing a lot of context switches, no IO, and not a lot of

Yes sysbench is a pretty poor benchmark i think, especially
on systems with a lot of cores. I ran into this on some of my
own testing too. I don't think it makes too much
sense to tune for it seriously, but of course it doesn't hurt
when Linux does well on it anyways.

-Andi

Re: Performance versus FreeBSD 7.0

[debian developer]

On Fri, Feb 29, 2008 at 7:08 PM, Stephen Cuppett <cuppett@gmail.com> wrote:
> Not lobbing the first artillery shell by any means, but I saw this
>  bullet on the FreeBSD release announcement:
>
>  # Dramatic improvements in performance and SMP scalability shown by
>  various database and other benchmarks, in some cases showing peak
>  performance improvements as high as 350% over FreeBSD 6.X under normal
>  loads and 1500% at high loads. When compared with the best performing
>  Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
>  from benchmarks used to analyze and improve system performance,
>  results with your specific work load may vary. Some of the changes
>  that contribute to this improvement are:
>
>     * The 1:1 libthr threading model is now the default.
>     * Finer-grained IPC, networking, and scheduler locking.
>     * A major focus on optimizing the SMP architecture that was put in
>  place during the 5.x and 6.x branches.
>
>  Some benchmarks show linear scaling up to 8 CPUs. Many workloads see a
>  significant performance improvement with multicore systems.
>
>  The whole thing is available at:
>
>  http://www.freebsd.org/releases/7.0R/announce.html
>
>  However, their site is pretty scant on details about those benchmarks.
>   Was just curious if anybody on this list had any input or links on
>  the metrics, the hardware, the workloads, how was Linux or FreeBSD
>  tuned, knew more information, could point me to the stats....

The benchmarks are posted here

http://people.freebsd.org/~kris/scaling/7.0%20Preview.pdf

page no 17-20.

A funny quote on CFS from above (page 19)

"The new CFS scheduler in 2.6.23 is "Completely Fair"...to
FreeBSD" :)

good to see some competition at last :D

Re: Performance versus FreeBSD 7.0

[debian developer]

On Thu, Mar 6, 2008 at 10:59 PM, debian developer <debiandev@gmail.com> wrote:

>  http://people.freebsd.org/~kris/scaling/7.0%20Preview.pdf
>
bloody spaces in file names #$%$#..

http://people.freebsd.org/~kris/scaling/7.0 Preview.pdf

Re: Performance versus FreeBSD 7.0

[Lx]

>
> Myself I find fun that they look at Linux as the Leader That Must Be Surpassed.
> All the performance highlights of freebsd 7.0 are things that linux already did some
> years ago.

GNU : Got Nothing Unique !

So it must be GNU/Linux. What a giant lock and what a benchmark ?