From: Ralf-Peter Rohbeck <Ralf-Peter.Rohbeck@quantum.com>
To: Michal Hocko <mhocko@suse.cz>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>,
Vlastimil Babka <vbabka@suse.cz>
Subject: Re: OOM killer changes
Date: Mon, 1 Aug 2016 13:16:49 -0700 [thread overview]
Message-ID: <3c022d92-9c96-9022-8496-aa8738fb7358@quantum.com> (raw)
In-Reply-To: <20160801200926.GF31957@dhcp22.suse.cz>
On 08/01/16 13:09, Michal Hocko wrote:
> On Mon 01-08-16 12:52:40, Ralf-Peter Rohbeck wrote:
>> On 01.08.2016 12:43, Michal Hocko wrote:
>>> On Mon 01-08-16 12:35:51, Ralf-Peter Rohbeck wrote:
>>>> On 01.08.2016 12:26, Michal Hocko wrote:
>>> [...]
>>>>> the amount of dirty pages is much smaller as well as the anonymous
>>>>> memory. The biggest portion seems to be in the page cache. The memory
>>>> The page cache will always be full if I'm writing at full steam to multiple
>>>> drives, no?
>>> Yes, the memory full of page cache is not unusual. The large portion of
>>> that memory being dirty/writeback can be a problem. That is why we have
>>> a dirty memory throttling which slows down (throttles) writers to keep
>>> the amount reasonable. What is your dirty throttling setup?
>>> $ grep . /proc/sys/vm/dirty*
>>>
>>> and what is your storage setup?
>> root@fs:~# grep . /proc/sys/vm/dirty*
>> /proc/sys/vm/dirty_background_bytes:0
>> /proc/sys/vm/dirty_background_ratio:10
>> /proc/sys/vm/dirty_bytes:0
>> /proc/sys/vm/dirty_expire_centisecs:3000
>> /proc/sys/vm/dirty_ratio:20
> With your 8G of RAM this can be quite a lot of dirty data at once. Is
> your storage able to write that back in a reasonable time? I mean this
> shouldn't cause the OOM killer but it can lead to some unexpected stalls
> especially when there are a lot of writers AFAIU. dirty_bytes knob
> should help to define a better cap.
The main filesystems are on the MegaRAID and can do 500-600 MB/s.
Writing to the USB drives only pushes about 90MB/s per drive.
>
>> /proc/sys/vm/dirtytime_expire_seconds:43200
>> /proc/sys/vm/dirty_writeback_centisecs:500
>>
>>
>> Storage setup:
>>
>> root@fs:~# lsscsi
>> [0:2:0:0] disk LSI MR9271-8iCC 3.29 /dev/sda
>> [0:2:1:0] disk LSI MR9271-8iCC 3.29 /dev/sdb
>> [9:0:0:0] disk TOSHIBA External USB 3.0 5438 /dev/sdf
>> [10:0:0:0] disk Seagate Backup+ Desk 050B /dev/sdc
>> [11:0:0:0] disk Seagate Expansion Desk 9400 /dev/sdd
>> [12:0:0:0] disk Seagate Backup+ Desk 050B /dev/sde
>> [13:0:0:0] disk Seagate Expansion Desk 9400 /dev/sdg
>> [14:0:0:0] disk TOSHIBA External USB 3.0 5438 /dev/sdl
>> [15:0:0:0] disk Seagate Expansion Desk 9400 /dev/sdh
>> [16:0:0:0] disk Seagate Expansion Desk 9400 /dev/sdi
>> [17:0:0:0] disk TOSHIBA External USB 3.0 5438 /dev/sdm
>> [18:0:0:0] disk Seagate Expansion Desk 9400 /dev/sdj
>> [19:0:0:0] disk Seagate Expansion Desk 9400 /dev/sdk
>>
>> sda is a 6x 1TB RAID5 and sdb is a single 480GB SSD, both on a MegaRAID
>> controller.
>>
>> The rest are 4TB USB drives that I'm experimenting with.
> Which devices did you write when hitting the OOM killer?
sdc, sdd and sde each at max speed, with a little bit of garden variety
IO on sda and sdb.
----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2016-08-01 20:16 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <d8f3adcc-3607-1ef6-9ec5-82b2e125eef2@quantum.com>
2016-08-01 6:16 ` OOM killer changes Michal Hocko
[not found] ` <b1a39756-a0b5-1900-6575-d6e1f502cb26@Quantum.com>
[not found] ` <20160801182358.GB31957@dhcp22.suse.cz>
[not found] ` <30dbabc4-585c-55a5-9f3a-4e243c28356a@Quantum.com>
2016-08-01 19:26 ` Michal Hocko
2016-08-01 19:35 ` Ralf-Peter Rohbeck
2016-08-01 19:43 ` Michal Hocko
2016-08-01 19:52 ` Ralf-Peter Rohbeck
2016-08-01 20:09 ` Michal Hocko
2016-08-01 20:16 ` Ralf-Peter Rohbeck [this message]
2016-08-01 20:26 ` Michal Hocko
2016-08-01 21:14 ` Ralf-Peter Rohbeck
2016-08-01 21:27 ` Ralf-Peter Rohbeck
2016-08-02 7:10 ` Michal Hocko
2016-08-02 19:25 ` Ralf-Peter Rohbeck
2016-08-15 4:48 ` Ralf-Peter Rohbeck
2016-08-15 9:16 ` Vlastimil Babka
2016-08-15 15:01 ` Michal Hocko
2016-08-15 18:42 ` Ralf-Peter Rohbeck
2016-08-16 7:32 ` Michal Hocko
2016-08-16 7:43 ` Michal Hocko
2016-08-17 9:14 ` Ralf-Peter Rohbeck
2016-08-17 9:23 ` Vlastimil Babka
2016-08-17 9:28 ` Ralf-Peter Rohbeck
2016-08-17 9:33 ` Michal Hocko
2016-08-17 23:37 ` Ralf-Peter Rohbeck
2016-08-18 6:57 ` Vlastimil Babka
2016-08-18 20:01 ` Ralf-Peter Rohbeck
2016-08-18 20:12 ` Vlastimil Babka
2016-08-19 2:42 ` Ralf-Peter Rohbeck
2016-08-19 6:27 ` Vlastimil Babka
2016-08-19 7:33 ` Michal Hocko
2016-08-19 7:47 ` Vlastimil Babka
2016-08-19 8:26 ` Michal Hocko
2016-08-24 18:13 ` Ralf-Peter Rohbeck
2016-08-25 7:22 ` Michal Hocko
2016-08-25 20:35 ` Ralf-Peter Rohbeck
2016-08-26 8:35 ` Michal Hocko
2016-09-06 11:09 ` Vlastimil Babka
2016-08-23 5:02 ` Joonsoo Kim
2016-08-23 7:45 ` Michal Hocko
2016-08-17 0:26 ` Ralf-Peter Rohbeck
2016-08-17 7:43 ` Vlastimil Babka
2016-08-16 3:12 ` Joonsoo Kim
2016-08-16 7:44 ` Vlastimil Babka
2016-08-17 4:48 ` Ralf-Peter Rohbeck
2016-08-17 7:56 ` Vlastimil Babka
2016-08-17 8:16 ` Joonsoo Kim
2016-08-17 9:21 ` Ralf-Peter Rohbeck
2016-08-17 9:11 ` Ralf-Peter Rohbeck
2016-08-17 9:20 ` Vlastimil Babka
2016-08-02 7:11 ` Vlastimil Babka
2016-08-02 9:02 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3c022d92-9c96-9022-8496-aa8738fb7358@quantum.com \
--to=ralf-peter.rohbeck@quantum.com \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.cz \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).