Linux TPC-C performance aided by kernel features

Linux TPC-C performance aided by kernel features

[Bond, Andrew]

This past Monday HP released the very first TPC-C benchmark result done using Linux.  When comparing it to a previous HP result using basically the same hardware and same database but with Windows 2000, the Linux result is faster and cheaper.  The benchmark is a proof point that shows Linux can run well in an enterprise environment.

Of more interest to this mailing list however would be some of the kernel features that gave us a performance boost.

1. Async IO  - We saw about a 5% increase in performance when using Ben's asynchronous IO with Oracle.  The peformance gain was because of reduced overhead on the Oracle and OS process front.  By using async IO we were able to run with only 2 Oracle dbwriter processes rather than the 15+ we would have needed without it.  This made the Oracle environment more efficient, and reduced our context switch overhead on the OS side.

2. Large memory support (16GB) - The Oracle processes used about 14GB of shared memory which was allocated using shmfs and managed through a mapping window in the Oracle process space.  Databases always love more memory, however in an IA-32 architecture the gains definitely diminish once you get past 4GB because of overhead.  Our gains going from 8GB to 16GB of memory in the system were in the 10% range.  

3. Raw device access - I don't have a percentage gain for this one since we have been using this feature since the beginning.  Initial testing last year showed that raw device access was definitely superior to other methods of device access.  On a single node, O_DIRECT would be an option now, but for cluster configurations multinode filesystem access would be needed.  One limitation we kept running into was the 256 raw device limit imposed by raw having a single major number.  This issue has been taken care of in 2.5, but for 2.4 based kernels it is a limitation for larger configurations.

4. 2MB PTE's for Oracle memory - We saw an 8% gain when we switched from using 4k PTE's to 2MB PTE's for the shared memory Oracle was allocating.  14GB of the 16GB of memory on the system was set aside for this "bigpage" usage.

Other features such as IO elevator tuning, process priority adjustment, and TCP tuning were also used.


For those that are interested in more detail about the benchmark configuration I would refer you to this TPC web page where #7 is the Linux benchmark and #8 is the Windows one.  You can select the benchmark and pull up the full disclosure report for more gory detail.

http://www.tpc.org/tpcc/results/tpcc_perf_results.asp?resulttype=cluster

Regards,
Andy

Re: Linux TPC-C performance aided by kernel features

[Matthew Kirkwood]

On Wed, 18 Sep 2002, Bond, Andrew wrote:

> 2. Large memory support (16GB) - The Oracle processes used about 14GB
> of shared memory which was allocated using shmfs and managed through a
> mapping window in the Oracle process space.  Databases always love
> more memory, however in an IA-32 architecture the gains definitely
> diminish once you get past 4GB because of overhead.  Our gains going
> from 8GB to 16GB of memory in the system were in the 10% range.

What did the gains from 4 to 8GB look like?

Could going above 4GB be a performance loss on less busy systems?
#
Matthew.

RE: Linux TPC-C performance aided by kernel features

[Bond, Andrew]

I don't have exact numbers, but I believe the gains were in the 12-15% range going from 4 to 8GB.

Yes, I believe that there are situations where you could add more memory >4GB and lose performance.  When we went from 8GB to 16GB we lost 5% performance just because the memory was in the system.  We ran tests that used the same about of memory in both cases and performance went down when the system had 16GB.  

Seeing a slowdown adding memory >4GB is also very possible if you don't have a kernel/driver/controller that supports 64-bit DMA.  The bounce buffer activity can be very costly.  For instance, in the benchmark we just published we could not use 64-bit DMA so we spent about 40% of our time in the kernel when we should be spending only about 20%.

Andy

> -----Original Message-----
> From: Matthew Kirkwood [mailto:matthew@hairy.beasts.org]
> Sent: Thursday, September 19, 2002 6:27 AM
> To: Bond, Andrew
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: Linux TPC-C performance aided by kernel features
> 
> 
> On Wed, 18 Sep 2002, Bond, Andrew wrote:
> 
> > 2. Large memory support (16GB) - The Oracle processes used 
> about 14GB
> > of shared memory which was allocated using shmfs and 
> managed through a
> > mapping window in the Oracle process space.  Databases always love
> > more memory, however in an IA-32 architecture the gains definitely
> > diminish once you get past 4GB because of overhead.  Our gains going
> > from 8GB to 16GB of memory in the system were in the 10% range.
> 
> What did the gains from 4 to 8GB look like?
> 
> Could going above 4GB be a performance loss on less busy systems?
> #
> Matthew.
> 
>