From: david@lang.hm
To: Ray Lee <ray-lk@madrabbit.org>
Cc: Ingo Molnar <mingo@elte.hu>,
Linus Torvalds <torvalds@linux-foundation.org>,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
miklos@szeredi.hu, akpm@linux-foundation.org, neilb@suse.de,
dgc@sgi.com, tomoki.sekiyama.qu@hitachi.com,
nikita@clusterfs.com, trond.myklebust@fys.uio.no,
yingchao.zhou@gmail.com, richard@rsk.demon.co.uk,
netdev@vger.kernel.org
Subject: Re: [PATCH 00/23] per device dirty throttling -v8
Date: Sat, 4 Aug 2007 10:15:05 -0700 (PDT) [thread overview]
Message-ID: <Pine.LNX.4.64.0708041014020.6905@asgard.lang.hm> (raw)
In-Reply-To: <2c0942db0708040901x7ada0fe2mf71f37ecba51005b@mail.gmail.com>
On Sat, 4 Aug 2007, Ray Lee wrote:
> (adding netdev cc:)
>
> On 8/4/07, david@lang.hm <david@lang.hm> wrote:
>> On Sat, 4 Aug 2007, Ingo Molnar wrote:
>>
>>> * Ingo Molnar <mingo@elte.hu> wrote:
>>>
>>>> There are positive reports in the never-ending "my system crawls like
>>>> an XT when copying large files" bugzilla entry:
>>>>
>>>> http://bugzilla.kernel.org/show_bug.cgi?id=7372
>>>
>>> i forgot this entry:
>>>
>>> " We recently upgraded our office to gigabit Ethernet and got some big
>>> AMD64 / 3ware boxes for file and vmware servers... only to find them
>>> almost useless under any kind of real load. I've built some patched
>>> 2.6.21.6 kernels (using the bdi throttling patch you mentioned) to
>>> see if our various Debian Etch boxes run better. So far my testing
>>> shows a *great* improvement over the stock Debian 2.6.18 kernel on
>>> our configurations. "
>>>
>>> and bdi has been in -mm in the past i think, so we also know (to a
>>> certain degree) that it does not hurt those workloads that are fine
>>> either.
>>>
>>> [ my personal interest in this is the following regression: every time i
>>> start a large kernel build with DEBUG_INFO on a quad-core 4GB RAM box,
>>> i get up to 30 seconds complete pauses in Vim (and most other tasks),
>>> during plain editing of the source code. (which happens when Vim tries
>>> to write() to its swap/undo-file.) ]
>>
>> I have an issue that sounds like it's related.
>>
>> I've got a syslog server that's got two Opteron 246 cpu's, 16G ram, 2x140G
>> 15k rpm drives (fusion MPT hardware mirroring), 16x500G 7200rpm SATA
>> drives on 3ware 9500 cards (software raid6) running 2.6.20.3 with hz set
>> at default and preempt turned off.
>>
>> I have syslog doing buffered writes to the SCSI drives and every 5 min a
>> cron job copies the data to the raid array.
>>
>> I've found that if I do anything significant on the large raid array that
>> the system looses a significant amount of the UDP syslog traffic, even
>> though there should be pleanty of ram and cpu (and the spindles involved
>> in the writes are not being touched), even a grep can cause up to 40%
>> losses in the syslog traffic. I've experimented with nice levels (nicing
>> down the grep and nicing up the syslogd) without a noticable effect on the
>> losses.
>>
>> I've been planning to try a new kernel with hz=1000 to see if that would
>> help, and after that experiment with the various preempt settings, but it
>> sounds like the per-device queues may actually be more relavent to the
>> problem.
>>
>> what would you suggest I test, and in what order and combination?
>
> At least on a surface level, your report has some similarities to
> http://lkml.org/lkml/2007/5/21/84 . In that message, John Miller
> mentions several things he tried without effect:
>
> < - I increased the max allowed receive buffer through
> < proc/sys/net/core/rmem_max and the application calls the right
> < syscall. "netstat -su" does not show any "packet receive errors".
> <
> < - After getting "kernel: swapper: page allocation failure.
> < order:0, mode:0x20", I increased /proc/sys/vm/min_free_kbytes
> <
> < - ixgb.txt in kernel network documentation suggests to increase
> < net.core.netdev_max_backlog to 300000. This did not help.
> <
> < - I also had to increase net.core.optmem_max, because the default
> < value was too small for 700 multicast groups.
>
> As they're all pretty simple to test, it may be worthwhile to give
> them a shot just to rule things out.
I will try them later today.
I forgot to mention that the filesystems are ext2 for the mirrored high
speed disks and xfs for the 8TB array.
David Lang
next prev parent reply other threads:[~2007-08-04 17:17 UTC|newest]
Thread overview: 189+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-08-03 12:37 [PATCH 00/23] per device dirty throttling -v8 Peter Zijlstra
2007-08-03 12:37 ` [PATCH 01/23] nfs: remove congestion_end() Peter Zijlstra
2007-08-03 12:37 ` [PATCH 02/23] lib: percpu_counter_add Peter Zijlstra
2007-08-03 12:37 ` [PATCH 03/23] lib: percpu_counter variable batch Peter Zijlstra
2007-08-03 12:37 ` [PATCH 04/23] lib: make percpu_counter_add take s64 Peter Zijlstra
2007-08-03 12:37 ` [PATCH 05/23] lib: percpu_counter_set Peter Zijlstra
2007-08-03 12:37 ` [PATCH 06/23] lib: percpu_counter_sum_positive Peter Zijlstra
2007-08-03 12:37 ` [PATCH 07/23] lib: percpu_count_sum() Peter Zijlstra
2007-08-03 12:37 ` [PATCH 08/23] lib: percpu_counter_init error handling Peter Zijlstra
2007-08-03 12:37 ` [PATCH 09/23] lib: percpu_counter_init_irq Peter Zijlstra
2007-08-03 12:37 ` [PATCH 10/23] mm: bdi init hooks Peter Zijlstra
2007-08-03 12:37 ` [PATCH 11/23] containers: " Peter Zijlstra
2007-08-03 12:37 ` [PATCH 12/23] mtd: " Peter Zijlstra
2007-08-03 12:37 ` [PATCH 13/23] mtd: clean up the backing_dev_info usage Peter Zijlstra
2007-08-03 12:37 ` [PATCH 14/23] mtd: give mtdconcat devices their own backing_dev_info Peter Zijlstra
2007-08-03 12:37 ` [PATCH 15/23] mm: scalable bdi statistics counters Peter Zijlstra
2007-08-03 12:37 ` [PATCH 16/23] mm: count reclaimable pages per BDI Peter Zijlstra
2007-08-03 12:37 ` [PATCH 17/23] mm: count writeback " Peter Zijlstra
2007-08-09 19:15 ` Christoph Lameter
2007-08-09 19:23 ` Peter Zijlstra
2007-08-09 19:27 ` Christoph Lameter
2007-08-13 8:36 ` Peter Zijlstra
2007-08-03 12:37 ` [PATCH 18/23] mm: expose BDI statistics in sysfs Peter Zijlstra
2007-08-03 12:37 ` [PATCH 19/23] lib: floating proportions Peter Zijlstra
2007-08-03 12:37 ` [PATCH 20/23] lib: floating proportions _single Peter Zijlstra
2007-08-03 12:37 ` [PATCH 21/23] mm: per device dirty threshold Peter Zijlstra
2007-08-03 12:37 ` [PATCH 22/23] mm: dirty balancing for tasks Peter Zijlstra
2007-08-03 12:37 ` [PATCH 23/23] debug: sysfs files for the current ratio/size/total Peter Zijlstra
2007-08-03 22:21 ` [PATCH 00/23] per device dirty throttling -v8 Linus Torvalds
2007-08-04 6:32 ` Ingo Molnar
2007-08-04 7:07 ` Ingo Molnar
2007-08-04 7:44 ` david
2007-08-04 16:01 ` Ray Lee
2007-08-04 17:15 ` david [this message]
2007-08-09 5:11 ` david
2007-08-04 10:33 ` Ingo Molnar
2007-08-04 16:17 ` Linus Torvalds
2007-08-04 16:37 ` Ingo Molnar
2007-08-04 16:51 ` Andrew Morton
2007-08-04 16:56 ` Ingo Molnar
2007-08-04 20:23 ` Alan Cox
2007-08-04 17:02 ` Diego Calleja
2007-08-04 17:17 ` Ingo Molnar
2007-08-04 17:38 ` Diego Calleja
2007-08-04 17:51 ` Diego Calleja
2007-08-08 10:43 ` Karel Zak
2007-08-04 17:39 ` Linus Torvalds
2007-08-04 18:08 ` Jeff Garzik
2007-08-04 19:12 ` Jörn Engel
2007-08-04 19:21 ` Ingo Molnar
2007-08-04 19:26 ` Jörn Engel
2007-08-04 19:42 ` Jörn Engel
2007-08-05 20:36 ` Christoph Hellwig
2007-08-06 18:03 ` Chuck Ebbert
2007-08-06 18:53 ` Jeff Garzik
2007-08-06 19:37 ` Alan Cox
2007-08-06 19:46 ` Chuck Ebbert
2007-08-07 7:05 ` Ingo Molnar
2007-08-08 21:10 ` Martin J. Bligh
2007-08-08 21:21 ` Andrew Morton
2007-08-09 0:54 ` Martin Bligh
2007-08-11 23:14 ` Valerie Henson
2007-08-10 0:21 ` Bill Davidsen
2007-08-14 9:57 ` Helge Hafting
2007-08-04 19:47 ` Linus Torvalds
2007-08-04 19:49 ` Linus Torvalds
2007-08-04 20:00 ` Ingo Molnar
2007-08-04 20:11 ` Ingo Molnar
2007-08-04 20:13 ` Arjan van de Ven
2007-08-05 8:18 ` [patch] add noatime/atime boot options, CONFIG_DEFAULT_NOATIME Ingo Molnar
2007-08-04 20:13 ` [PATCH 00/23] per device dirty throttling -v8 Arjan van de Ven
2007-08-04 21:48 ` Theodore Tso
2007-08-05 18:01 ` Arjan van de Ven
2007-08-05 20:34 ` Christoph Hellwig
[not found] ` <fa.7rstQpXif2z9y2n2HD+qxLFnueg@ifi.uio.no>
[not found] ` <fa.6VOZrceT65Vh8CIRIta0zSg2V38@ifi.uio.no>
[not found] ` <fa.xcZCTa5cHDOhrcyXZ2gZbzbu7g0@ifi.uio.no>
[not found] ` <fa.JjZwG90x+07YaOx8h5VLN+9AL/8@ifi.uio.no>
[not found] ` <fa.V9U4mAEXVjNqblhzu7GRmxif7Uw@ifi.uio.no>
[not found] ` <fa.uq0BQtrgp66a08hpsF+vrqXUNC4@ifi.uio.no>
2007-08-15 18:16 ` david.balazic
2007-08-04 20:11 ` [PATCH 00/23] " Alan Cox
2007-08-04 20:28 ` Jeff Garzik
2007-08-04 21:47 ` Alan Cox
2007-08-04 23:51 ` Claudio Martins
2007-08-05 0:49 ` Alan Cox
2007-08-05 7:28 ` Ingo Molnar
2007-08-05 10:29 ` Jakob Oestergaard
2007-08-05 12:46 ` Alan Cox
2007-08-05 12:58 ` Ingo Molnar
2007-08-05 13:29 ` Willy Tarreau
2007-08-06 6:57 ` Ingo Molnar
2007-08-06 13:12 ` Willy Tarreau
2007-08-05 14:46 ` Theodore Tso
2007-08-05 17:55 ` Ingo Molnar
2007-08-05 17:59 ` Jeff Garzik
2007-08-05 18:09 ` Ingo Molnar
2007-08-05 18:08 ` Arjan van de Ven
2007-08-07 21:20 ` Bill Davidsen
2007-08-05 7:18 ` Ingo Molnar
2007-08-07 18:55 ` Bill Davidsen
2007-08-07 19:35 ` Alan Cox
2007-08-08 17:44 ` Bill Davidsen
2007-08-04 20:28 ` Ingo Molnar
2007-08-04 20:34 ` Arjan van de Ven
2007-08-04 21:03 ` Ingo Molnar
2007-08-04 21:51 ` Alan Cox
2007-08-05 7:21 ` Ingo Molnar
2007-08-05 7:29 ` Andrew Morton
2007-08-05 7:39 ` Ingo Molnar
2007-08-05 8:53 ` Willy Tarreau
2007-08-05 14:17 ` Jörn Engel
2007-08-05 18:02 ` Arjan van de Ven
2007-08-05 18:37 ` Jörn Engel
2007-08-05 20:21 ` Jörn Engel
2007-08-05 20:33 ` Andrew Morton
2007-08-05 12:47 ` Alan Cox
2007-08-05 12:56 ` Ingo Molnar
2007-08-05 18:44 ` Dave Jones
2007-08-05 18:58 ` adi
2007-08-06 6:39 ` Ingo Molnar
2007-08-06 15:59 ` Dave Jones
2007-08-06 16:16 ` Ingo Molnar
2007-08-05 7:37 ` Ingo Molnar
2007-08-05 9:04 ` Jeff Garzik
2007-08-05 12:43 ` Alan Cox
2007-08-05 12:54 ` Ingo Molnar
2007-08-05 13:37 ` Alan Cox
2007-08-05 18:08 ` Ingo Molnar
2007-08-05 19:11 ` Alan Cox
2007-08-08 18:22 ` Bill Davidsen
2007-08-08 19:39 ` Jeff Garzik
2007-08-08 20:31 ` Bill Davidsen
2007-08-08 23:18 ` Alan Cox
2007-08-07 19:09 ` Bill Davidsen
2007-08-04 21:48 ` Alan Cox
2007-08-05 7:13 ` Ingo Molnar
2007-08-05 13:22 ` Diego Calleja
2007-08-05 19:03 ` david
2007-08-06 6:52 ` Ingo Molnar
2007-08-10 4:04 ` Bill Davidsen
2007-08-11 5:19 ` Valdis.Kletnieks
2007-08-06 6:58 ` Ingo Molnar
2007-08-09 0:57 ` Greg Trounson
2007-08-09 1:26 ` david
2007-08-09 2:33 ` Andi Kleen
2007-08-04 22:39 ` Ilpo Järvinen
2007-08-05 10:20 ` Jakob Oestergaard
2007-08-05 10:42 ` Jeff Garzik
2007-08-05 10:58 ` Jakob Oestergaard
2007-08-05 12:46 ` Ingo Molnar
2007-08-05 13:46 ` Jakob Oestergaard
2007-08-05 16:45 ` Linus Torvalds
2007-08-05 19:09 ` Ingo Molnar
2007-08-05 19:22 ` [patch] implement smarter atime updates support Ingo Molnar
2007-08-05 19:28 ` [patch] implement smarter atime updates support, v2 Ingo Molnar
2007-08-05 20:42 ` Theodore Tso
2007-08-06 5:36 ` Ingo Molnar
2007-08-05 19:53 ` [patch] implement smarter atime updates support Arjan van de Ven
2007-08-05 20:04 ` Alan Cox
2007-08-05 20:22 ` Arjan van de Ven
2007-08-05 19:29 ` [PATCH 00/23] per device dirty throttling -v8 Alan Cox
2007-08-05 19:32 ` Ingo Molnar
2007-08-05 23:43 ` David Chinner
2007-08-05 0:26 ` Andi Kleen
2007-08-05 15:00 ` Theodore Tso
2007-08-06 13:47 ` Chris Mason
2007-08-17 0:45 ` Dave Jones
2007-08-05 20:41 ` Christoph Hellwig
2007-08-06 10:42 ` Andi Kleen
2007-08-16 10:18 ` Helge Hafting
2007-08-09 6:25 ` Lionel Elie Mamane
2007-08-09 15:02 ` Chuck Ebbert
2007-08-09 16:22 ` Diego Calleja
2007-08-04 16:41 ` Andrew Morton
2007-08-04 17:26 ` Nikita Danilov
2007-08-04 19:16 ` Florian Weimer
2007-08-05 6:00 ` Andrew Morton
2007-08-05 7:57 ` Florian Weimer
2007-08-05 20:43 ` Christoph Hellwig
2007-08-05 22:46 ` Theodore Tso
2007-08-06 0:24 ` David Chinner
2007-08-05 0:28 ` Andi Kleen
2007-08-04 16:15 ` Linus Torvalds
2007-08-05 17:22 ` Brice Figureau
2007-08-05 22:17 ` Andi Kleen
2007-08-06 8:40 ` Brice Figureau
2007-08-14 1:44 ` Stewart Smith
2007-08-14 2:25 ` Andi Kleen
2007-08-14 7:59 ` Brice Figureau
2007-08-06 20:26 ` Miklos Szeredi
2007-08-08 12:25 ` richard kennedy
2007-08-08 13:54 ` Andi Kleen
2007-08-10 4:17 ` Bill Davidsen
2007-08-10 16:00 pointman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Pine.LNX.4.64.0708041014020.6905@asgard.lang.hm \
--to=david@lang.hm \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=dgc@sgi.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=miklos@szeredi.hu \
--cc=mingo@elte.hu \
--cc=neilb@suse.de \
--cc=netdev@vger.kernel.org \
--cc=nikita@clusterfs.com \
--cc=ray-lk@madrabbit.org \
--cc=richard@rsk.demon.co.uk \
--cc=tomoki.sekiyama.qu@hitachi.com \
--cc=torvalds@linux-foundation.org \
--cc=trond.myklebust@fys.uio.no \
--cc=yingchao.zhou@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).