From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB4F3C433E1 for ; Fri, 17 Jul 2020 01:36:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 761EC2070E for ; Fri, 17 Jul 2020 01:36:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qTta5OTH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 761EC2070E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B27518D0006; Thu, 16 Jul 2020 21:36:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD7968D0003; Thu, 16 Jul 2020 21:36:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A21F8D0006; Thu, 16 Jul 2020 21:36:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0099.hostedemail.com [216.40.44.99]) by kanga.kvack.org (Postfix) with ESMTP id 809C98D0003 for ; Thu, 16 Jul 2020 21:36:14 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 125628248D7C for ; Fri, 17 Jul 2020 01:36:14 +0000 (UTC) X-FDA: 77045852268.11.time21_511820a26f06 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id D7D38180F9825 for ; Fri, 17 Jul 2020 01:36:13 +0000 (UTC) X-HE-Tag: time21_511820a26f06 X-Filterd-Recvd-Size: 7050 Received: from mail-io1-f67.google.com (mail-io1-f67.google.com [209.85.166.67]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Fri, 17 Jul 2020 01:36:13 +0000 (UTC) Received: by mail-io1-f67.google.com with SMTP id k23so8588407iom.10 for ; Thu, 16 Jul 2020 18:36:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5rer2JfW0saMht8JVGss/XammRKDxp4PpmFY9JEScvo=; b=qTta5OTHFJKonVojSkxrO7c767omkkF7otL09AAy0YMaq1ZTrww3aB7m2GWsVvwkVU CGanZX3vc2XlowGJNDaodW+KcoPtBYRsxtjxmWhPie7ZL6mRJaPpkA/+QGh7YLUYy3il ecMBP9+AZJURS5A+xt8HS8My5qILzuTk7TiU/ZNcJ5/prV3g2BgXdvcQQelLQMihyLzI Q9BnjyDZIEz/tLeWtVIgfgyqQekAUBAFv7O8+4l90iKquVHt9HoaybV2SZqRh+YPq0Ek BhawERcOHttf0ITQ97dN260IS8Cr5rlKK2x3pWOQXUh5UGx7tHCPK/1j1RXZjWzRbv3g 2zVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5rer2JfW0saMht8JVGss/XammRKDxp4PpmFY9JEScvo=; b=KRorcNAYLY/cT4YCGJ1H67b0JRqn5vk2wDl2UXvp52xFlNTFjIlxMjFpzjDlBnXptG SDIKj3Kw8oi0gLjgs8vgFbr61ZLJ8EdG/CtWH2BWTeAbBmzLWSOUGMv4wjcCJo6+8T2b U+u9x6n7G+Q2McxwsEy5POllTKabJ6RuIuKWk+WkESsm2EpcSHNMeWmVcWE7R1VimrWQ G4LP2el30asAJfibpvBypKQLcIPmuZlEFoGaBPmsOAvN9siL3vdXWHh2OTC4CRO5KA5f 1aVHRArNwfAwhpfHh7H7YhafM1Cdz3e4SExoE16/mQGjPhzouotxJgqZo0UdBVj6flKt TwVg== X-Gm-Message-State: AOAM531DKIwcY9wRL36PXAqy2boXZRI5wxjqJVMunok7T2+d2om5sqoT S9zUXIz016gco+UsqLicmNgKsXSzGwMB5pu2p+Y= X-Google-Smtp-Source: ABdhPJxTBinJaZX93mrndWhMn0TRpfO/sBP3reUXIWc7oVPU6XLvL3TnztYEfB6TAgB/4KOkgO8XRCumQsdgmdEwRLU= X-Received: by 2002:a02:c50d:: with SMTP id s13mr8005300jam.109.1594949772694; Thu, 16 Jul 2020 18:36:12 -0700 (PDT) MIME-Version: 1.0 References: <1594735034-19190-1-git-send-email-laoar.shao@gmail.com> In-Reply-To: From: Yafang Shao Date: Fri, 17 Jul 2020 09:35:36 +0800 Message-ID: Subject: Re: [PATCH v2] memcg, oom: check memcg margin for parallel oom To: David Rientjes Cc: Michal Hocko , Tetsuo Handa , Andrew Morton , Johannes Weiner , Linux MM Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: D7D38180F9825 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jul 17, 2020 at 3:53 AM David Rientjes wrote: > > On Thu, 16 Jul 2020, Yafang Shao wrote: > > > > It's likely a misunderstanding: I wasn't necessarily concerned about > > > showing 60MB in a memcg limited to 100MB, that part we can deal with, the > > > concern was after dumping all of that great information that instead of > > > getting a "Killed process..." we get a "Oh, there's memory now, just > > > kidding about everything we just dumped" ;) > > > > > > > Actually the kernel is doing it now, see bellow, > > > > dump_header() <<<< dump lots of information > > __oom_kill_process > > p = find_lock_task_mm(victim); > > if (!p) > > return; <<<< without killing any process. > > > > Ah, this is catching an instance where the chosen process has already done > exit_mm(), good catch -- I can find examples of this by scraping kernel > logs from our fleet. > > So it appears there is precedence for dumping all the oom info but not > actually performing any action for it and I made the earlier point that > diagnostic information in the kernel log here is still useful. I think it > is still preferable that the kernel at least tell us why it didn't do > anything, but as you mention that already happens today. > > Would you like to send a patch that checks for mem_cgroup_margin() here as > well? A second patch could make the possible inaction more visibile, > something like "Process ${pid} (${comm}) is already exiting" for the above > check or "Memcg ${memcg} is no longer out of memory". > > Another thing that these messages indicate, beyond telling us why the oom > killer didn't actually SIGKILL anything, is that we can expect some skew > in the memory stats that shows an availability of memory. > Agreed, these messages would be helpful. I will send a patch for it. > > > > > We could likely enlighten userspace about that so that we don't consider > > > that to be an actual oom kill. But I also very much agree that after > > > dump_header() would be appropriate as well since the goal is to prevent > > > unnecessary oom killing. > > > > > > Would you mind sending a patch to check mem_cgroup_margin() on > > > is_memcg_oom() prior to sending the SIGKILL to the victim and printing the > > > "Killed process..." line? We'd need a line that says "xKB of memory now > > > available -- suppressing oom kill" or something along those lines so > > > userspace understands what happened. But the memory info that it emits > > > both for the state of the memcg and system RAM may also be helpful to > > > understand why we got to the oom kill in the first place, which is also a > > > good thing. > > > > > > I'd happy ack that patch since it would be a comprehensive solution that > > > avoids oom kill of user processes at all costs, which is a goal I think we > > > can all rally behind. > > > > I'd prefer to put dump_header() behind do_send_sig_info(), for example, > > > > __oom_kill_process() > > do_send_sig_info() > > dump_header() <<<< may better put it behind wake_oom_reaper(), but > > it may loses some information to dump... > > pr_err("%s: Killed process %d (%s)....") > > > > I agree with Michal here that dump_header() after the actual kill would no > longer represent the state of the system (or cpuset or memcg, depending on > context) at the time of the oom kill so it's best to dump relevant > information before the actual kill. -- Thanks Yafang