All of lore.kernel.org
 help / color / mirror / Atom feed
From: Vlastimil Babka <vbabka@suse.cz>
To: Simon Kirby <sim@hostway.ca>
Cc: Marc MERLIN <marc@merlins.org>,
	linux-mm@kvack.org, Michal Hocko <mhocko@kernel.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Joonsoo Kim <iamjoonsoo.kim@lge.com>, Tejun Heo <tj@kernel.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Subject: Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free
Date: Mon, 28 Nov 2016 09:06:58 +0100	[thread overview]
Message-ID: <312fa85c-19b3-46e6-fd7f-c8070eab5d08@suse.cz> (raw)
In-Reply-To: <20161122214607.GA11962@hostway.ca>

On 11/22/2016 10:46 PM, Simon Kirby wrote:
> On Tue, Nov 22, 2016 at 05:14:02PM +0100, Vlastimil Babka wrote:
>
>> On 11/22/2016 05:06 PM, Marc MERLIN wrote:
>>> On Mon, Nov 21, 2016 at 01:56:39PM -0800, Marc MERLIN wrote:
>>>> On Mon, Nov 21, 2016 at 10:50:20PM +0100, Vlastimil Babka wrote:
>>>>>> 4.9rc5 however seems to be doing better, and is still running after 18
>>>>>> hours. However, I got a few page allocation failures as per below, but the
>>>>>> system seems to recover.
>>>>>> Vlastimil, do you want me to continue the copy on 4.9 (may take 3-5 days)
>>>>>> or is that good enough, and i should go back to 4.8.8 with that patch applied?
>>>>>> https://marc.info/?l=linux-mm&m=147423605024993
>>>>>
>>>>> Hi, I think it's enough for 4.9 for now and I would appreciate trying
>>>>> 4.8 with that patch, yeah.
>>>>
>>>> So the good news is that it's been running for almost 5H and so far so good.
>>>
>>> And the better news is that the copy is still going strong, 4.4TB and
>>> going. So 4.8.8 is fixed with that one single patch as far as I'm
>>> concerned.
>>>
>>> So thanks for that, looks good to me to merge.
>>
>> Thanks a lot for the testing. So what do we do now about 4.8? (4.7 is
>> already EOL AFAICS).
>>
>> - send the patch [1] as 4.8-only stable. Greg won't like that, I expect.
>>   - alternatively a simpler (againm 4.8-only) patch that just outright
>> prevents OOM for 0 < order < costly, as Michal already suggested.
>> - backport 10+ compaction patches to 4.8 stable
>> - something else?
>>
>> Michal? Linus?
>>
>> [1] https://marc.info/?l=linux-mm&m=147423605024993
>
> Sorry for my molasses rate of feedback. I found a workaround, setting
> vm/watermark_scale_factor to 500, and threw that in sysctl. This was on
> the MythTV box that OOMs everything after about a day on 4.8 otherwise.
>
> I've been running [1] for 9 days on it (4.8.4 + [1]) without issue, but
> just realized I forgot to remove the watermark_scale_factor workaround.
> I've restored that now, so I'll see if it becomes unhappy by tomorrow.

Thanks for the testing. Could you now try Michal's stable candidate [1] 
from this thread please?

[1] http://marc.info/?l=linux-mm&m=147988285831283&w=2

> I also threw up a few other things you had asked for (vmstat, zoneinfo
> before and after the first OOM on 4.8.4): http://0x.ca/sim/ref/4.8.4/
> (that was before booting into a rebuild with [1] applied)
>
> Simon-
>

WARNING: multiple messages have this Message-ID (diff)
From: Vlastimil Babka <vbabka@suse.cz>
To: Simon Kirby <sim@hostway.ca>
Cc: Marc MERLIN <marc@merlins.org>,
	linux-mm@kvack.org, Michal Hocko <mhocko@kernel.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Joonsoo Kim <iamjoonsoo.kim@lge.com>, Tejun Heo <tj@kernel.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Subject: Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free
Date: Mon, 28 Nov 2016 09:06:58 +0100	[thread overview]
Message-ID: <312fa85c-19b3-46e6-fd7f-c8070eab5d08@suse.cz> (raw)
In-Reply-To: <20161122214607.GA11962@hostway.ca>

On 11/22/2016 10:46 PM, Simon Kirby wrote:
> On Tue, Nov 22, 2016 at 05:14:02PM +0100, Vlastimil Babka wrote:
>
>> On 11/22/2016 05:06 PM, Marc MERLIN wrote:
>>> On Mon, Nov 21, 2016 at 01:56:39PM -0800, Marc MERLIN wrote:
>>>> On Mon, Nov 21, 2016 at 10:50:20PM +0100, Vlastimil Babka wrote:
>>>>>> 4.9rc5 however seems to be doing better, and is still running after 18
>>>>>> hours. However, I got a few page allocation failures as per below, but the
>>>>>> system seems to recover.
>>>>>> Vlastimil, do you want me to continue the copy on 4.9 (may take 3-5 days)
>>>>>> or is that good enough, and i should go back to 4.8.8 with that patch applied?
>>>>>> https://marc.info/?l=linux-mm&m=147423605024993
>>>>>
>>>>> Hi, I think it's enough for 4.9 for now and I would appreciate trying
>>>>> 4.8 with that patch, yeah.
>>>>
>>>> So the good news is that it's been running for almost 5H and so far so good.
>>>
>>> And the better news is that the copy is still going strong, 4.4TB and
>>> going. So 4.8.8 is fixed with that one single patch as far as I'm
>>> concerned.
>>>
>>> So thanks for that, looks good to me to merge.
>>
>> Thanks a lot for the testing. So what do we do now about 4.8? (4.7 is
>> already EOL AFAICS).
>>
>> - send the patch [1] as 4.8-only stable. Greg won't like that, I expect.
>>   - alternatively a simpler (againm 4.8-only) patch that just outright
>> prevents OOM for 0 < order < costly, as Michal already suggested.
>> - backport 10+ compaction patches to 4.8 stable
>> - something else?
>>
>> Michal? Linus?
>>
>> [1] https://marc.info/?l=linux-mm&m=147423605024993
>
> Sorry for my molasses rate of feedback. I found a workaround, setting
> vm/watermark_scale_factor to 500, and threw that in sysctl. This was on
> the MythTV box that OOMs everything after about a day on 4.8 otherwise.
>
> I've been running [1] for 9 days on it (4.8.4 + [1]) without issue, but
> just realized I forgot to remove the watermark_scale_factor workaround.
> I've restored that now, so I'll see if it becomes unhappy by tomorrow.

Thanks for the testing. Could you now try Michal's stable candidate [1] 
from this thread please?

[1] http://marc.info/?l=linux-mm&m=147988285831283&w=2

> I also threw up a few other things you had asked for (vmstat, zoneinfo
> before and after the first OOM on 4.8.4): http://0x.ca/sim/ref/4.8.4/
> (that was before booting into a rebuild with [1] applied)
>
> Simon-
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

  reply	other threads:[~2016-11-28  8:07 UTC|newest]

Thread overview: 113+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-11-21 15:43 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free Marc MERLIN
2016-11-21 16:30 ` Marc MERLIN
2016-11-21 21:50 ` Vlastimil Babka
2016-11-21 21:50   ` Vlastimil Babka
2016-11-21 21:56   ` Marc MERLIN
2016-11-21 21:56     ` Marc MERLIN
2016-11-21 23:03     ` [PATCH] block,blkcg: use __GFP_NOWARN for best-effort allocations in blkcg Tejun Heo
2016-11-21 23:03       ` Tejun Heo
2016-11-22 15:47       ` Vlastimil Babka
2016-11-22 15:47         ` Vlastimil Babka
2016-11-22 16:48         ` Tejun Heo
2016-11-22 16:48           ` Tejun Heo
2016-11-22 22:13           ` Linus Torvalds
2016-11-22 22:13             ` Linus Torvalds
2016-11-23  8:50             ` Vlastimil Babka
2016-11-23  8:50               ` Vlastimil Babka
2016-11-28 17:19               ` Tejun Heo
2016-11-28 17:19                 ` Tejun Heo
2016-11-29  7:25                 ` Michal Hocko
2016-11-29  7:25                   ` Michal Hocko
2016-11-29 16:38                   ` Tejun Heo
2016-11-29 16:38                     ` Tejun Heo
2016-11-29 16:57                     ` Vlastimil Babka
2016-11-29 16:57                       ` Vlastimil Babka
2016-11-29 17:13                       ` Michal Hocko
2016-11-29 17:13                         ` Michal Hocko
2016-11-29 17:17                         ` Linus Torvalds
2016-11-29 17:17                           ` Linus Torvalds
2016-11-29 17:28                           ` Michal Hocko
2016-11-29 17:28                             ` Michal Hocko
2016-11-29 17:48                             ` Linus Torvalds
2016-11-29 17:48                               ` Linus Torvalds
2016-11-22 16:00       ` Jens Axboe
2016-11-22 16:00         ` Jens Axboe
2016-11-22 16:06     ` 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free Marc MERLIN
2016-11-22 16:06       ` Marc MERLIN
2016-11-22 16:14       ` Vlastimil Babka
2016-11-22 16:14         ` Vlastimil Babka
2016-11-22 16:25         ` Michal Hocko
2016-11-22 16:25           ` Michal Hocko
2016-11-22 16:47           ` Marc MERLIN
2016-11-22 16:47             ` Marc MERLIN
2016-11-22 16:38         ` Greg Kroah-Hartman
2016-11-22 16:38           ` Greg Kroah-Hartman
2016-11-29 16:25           ` Michal Hocko
2016-11-29 16:25             ` Michal Hocko
2016-11-29 16:25             ` Michal Hocko
2016-11-29 16:43             ` Patch "mm, oom: stop pre-mature high-order OOM killer invocations" has been added to the 4.8-stable tree gregkh
2016-11-29 16:43             ` 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free Greg Kroah-Hartman
2016-11-29 16:43               ` Greg Kroah-Hartman
2016-11-22 19:38         ` Linus Torvalds
2016-11-22 19:38           ` Linus Torvalds
2016-11-23  6:34           ` Michal Hocko
2016-11-23  6:34             ` Michal Hocko
2016-11-23  6:53             ` Hillf Danton
2016-11-23  6:53               ` Hillf Danton
2016-11-23  7:00               ` Michal Hocko
2016-11-23  7:00                 ` Michal Hocko
2016-11-23  9:18             ` Vlastimil Babka
2016-11-23  9:18               ` Vlastimil Babka
2016-11-28  7:23             ` Michal Hocko
2016-11-28  7:23               ` Michal Hocko
2016-11-28 20:55               ` Marc MERLIN
2016-11-29 15:55               ` Marc MERLIN
2016-11-29 15:55                 ` Marc MERLIN
2016-11-29 16:07                 ` Michal Hocko
2016-11-29 16:07                   ` Michal Hocko
2016-11-29 16:34                   ` Marc MERLIN
2016-11-29 16:34                     ` Marc MERLIN
2016-11-29 17:07                     ` Linus Torvalds
2016-11-29 17:07                       ` Linus Torvalds
2016-11-29 17:40                       ` Marc MERLIN
2016-11-29 17:40                         ` Marc MERLIN
2016-11-29 18:01                         ` Linus Torvalds
2016-11-29 18:01                           ` Linus Torvalds
2016-11-30 17:47                           ` Marc MERLIN
2016-11-30 17:47                             ` Marc MERLIN
2016-11-30 18:14                             ` Linus Torvalds
2016-11-30 18:21                               ` Marc MERLIN
2016-11-30 18:21                                 ` Marc MERLIN
2016-11-30 18:27                               ` Jens Axboe
2016-11-30 18:27                                 ` Jens Axboe
2016-11-30 20:30                               ` Tejun Heo
2016-11-30 20:30                                 ` Tejun Heo
2016-12-01 13:50                                 ` Kent Overstreet
2016-12-01 13:50                                   ` Kent Overstreet
2016-12-01 18:16                                   ` Linus Torvalds
2016-12-01 18:16                                     ` Linus Torvalds
2016-12-01 18:30                                     ` Jens Axboe
2016-12-01 18:30                                       ` Jens Axboe
2016-12-01 18:37                                       ` Linus Torvalds
2016-12-01 18:37                                         ` Linus Torvalds
2016-12-01 18:46                                         ` Jens Axboe
2016-12-01 18:46                                           ` Jens Axboe
2016-11-29 20:11                         ` Holger Hoffstätte
2016-11-29 23:01                         ` Marc MERLIN
2016-11-29 23:01                           ` Marc MERLIN
2016-11-30 13:58                           ` Tetsuo Handa
2016-11-30 13:58                             ` Tetsuo Handa
2017-05-02  4:12                           ` Marc MERLIN
2017-05-02  4:12                             ` Marc MERLIN
2017-05-02  7:44                             ` Michal Hocko
2017-05-02  7:44                               ` Michal Hocko
2017-05-02 14:15                               ` Marc MERLIN
2017-05-02 14:15                                 ` Marc MERLIN
2017-05-02 10:44                             ` Tetsuo Handa
2017-05-02 10:44                               ` Tetsuo Handa
2016-11-29 16:15               ` Marc MERLIN
2016-11-29 16:15                 ` Marc MERLIN
2016-11-22 21:46         ` Simon Kirby
2016-11-22 21:46           ` Simon Kirby
2016-11-28  8:06           ` Vlastimil Babka [this message]
2016-11-28  8:06             ` Vlastimil Babka

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=312fa85c-19b3-46e6-fd7f-c8070eab5d08@suse.cz \
    --to=vbabka@suse.cz \
    --cc=gregkh@linuxfoundation.org \
    --cc=iamjoonsoo.kim@lge.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=marc@merlins.org \
    --cc=mhocko@kernel.org \
    --cc=sim@hostway.ca \
    --cc=tj@kernel.org \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.