From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93EB1C433E9 for ; Mon, 1 Mar 2021 21:34:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 41B3560241 for ; Mon, 1 Mar 2021 21:34:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244229AbhCAVdK (ORCPT ); Mon, 1 Mar 2021 16:33:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47472 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235320AbhCARTA (ORCPT ); Mon, 1 Mar 2021 12:19:00 -0500 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4C49C061756 for ; Mon, 1 Mar 2021 09:18:03 -0800 (PST) Received: by mail-ej1-x62b.google.com with SMTP id hs11so29648077ejc.1 for ; Mon, 01 Mar 2021 09:18:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=vv6t5g8xmRQrU3t4nwEI1adaYgpKEkMqFy3Pe4ng6DE=; b=LG5630hxJ/RU1Vhwdq9zzm9Y94EvZi4itREzN4F+bHXi9ZMDLn5fk/NhUoOeDQ92Yd RaaVkDDZrgitJ1gYmrJY2fzvTGU7Yce18FWzKGE3qN7xR6jbrM075E9XrvWw6uxbcugF tOuTJAs4Cdn9QeZeeWqMo+NCDRdTcnAy4OpLY8JEZBNV+vBRxgxIdxxbVB+/NyeSOkAK dqR4Tysj/lpoLQFDfgLSJX5NO7k4WclYROWTo7zIW2vc9YNr81iWx4/QiQF5yzt8iErt uvMQz0b5eomR+5ZominYN7ZcTEn8/+v9o7CVF6au5HruHQC0wkWXdZoGyqpQOefiwUI9 2Frg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vv6t5g8xmRQrU3t4nwEI1adaYgpKEkMqFy3Pe4ng6DE=; b=hFuit7mLros+Ck7WIZcKa9WR6U0+VlxgT2wsytDRtb4gnwI2af45i/HMIaScmkqm98 JowbQVR68ak53pXy97Z622sMrymf+o6i0sa3Xad8evzepcFeMV0PMubSixzy+Lm/HuhI Lvm3CIN4Xxub7mFqkjDb+yihntVJcTsL6d/LhgJPXE7RCeAPMVOBSv3W+HDY0RkvwaUZ tONf5NLUF4Vmrv18VcHodxkzkE3d9bY7+NlZxGgTmHi7S9TaqNWSyJyO6vToMcBc6V9A S6MTSTbjUnWAXPPqamjrN2ASBEDqwwxYjCU8jYKATH//Cr2mxE7BkSPcXFLGAVAi5bF/ SK3A== X-Gm-Message-State: AOAM530kt61OECLRgThZDlyPWy6UEvtdBP0BJ223HEv6FGRiPMaRUlpY DBZLjC/4Z05n0xuljMQ+YoZqgG4c9oG9uIJpxOo= X-Google-Smtp-Source: ABdhPJx31N4JWJjQv43bkttTGqAs8q4uS7GeLxKU4ia9HhxWJv7MDghBLNRkrkoxDyZL1rgsGhbvTg6XleVLGNjVwe8= X-Received: by 2002:a17:907:2bf6:: with SMTP id gv54mr17343031ejc.514.1614619082651; Mon, 01 Mar 2021 09:18:02 -0800 (PST) MIME-Version: 1.0 References: <20210226021254.3980-1-shy828301@gmail.com> In-Reply-To: From: Yang Shi Date: Mon, 1 Mar 2021 09:17:50 -0800 Message-ID: Subject: Re: [PATCH] doc: memcontrol: add description for oom_kill To: Michal Hocko Cc: Johannes Weiner , Roman Gushchin , Shakeel Butt , Andrew Morton , Jonathan Corbet , Linux MM , Linux Kernel Mailing List Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Mar 1, 2021 at 4:24 AM Michal Hocko wrote: > > On Fri 26-02-21 11:19:51, Yang Shi wrote: > > On Fri, Feb 26, 2021 at 8:42 AM Yang Shi wrote: > > > > > > On Thu, Feb 25, 2021 at 11:30 PM Michal Hocko wrote: > > > > > > > > On Thu 25-02-21 18:12:54, Yang Shi wrote: > > > > > When debugging an oom issue, I found the oom_kill counter of memcg is > > > > > confusing. At the first glance without checking document, I thought it > > > > > just counts for memcg oom, but it turns out it counts both global and > > > > > memcg oom. > > > > > > > > Yes, this is the case indeed. The point of the counter was to count oom > > > > victims from the memcg rather than matching that to the source of the > > > > oom. Rememeber that this could have been a memcg oom up in the > > > > hierarchy as well. Counting victims on the oom origin could be equally > > > > > > Yes, it is updated hierarchically on v2, but not on v1. I'm supposed > > > this is because v1 may work in non-hierarchcal mode? If this is the > > > only reason we may be able to remove this to get aligned with v2 since > > > non-hierarchal mode is no longer supported. > > > > BTW, having the counter recorded hierarchically may help out one of > > our usecases. We want to monitor the oom_kill for some services, but > > systemd would wipe out the cgroup if the service is oom killed then > > restart the service from scratch (it means create a brand new cgroup > > with the same name). So this systemd behavior makes the counter > > useless if it is not recorded hierarchically. > > Just to make sure I understand correctly. You have a setup where memcg > for a service has a hard limit configured and it is destroyed when oom > happens inside that memcg. A new instance is created at the same place > of the hierarchy with a new memcg. Your problem is that the oom killed > memcg will not be recorded in its parent oom event and the information > will get lost with the torn down memcg. Correct? Yes. But global oom instead of memcg oom. > > If yes then how do you tell which of the child cgroup was killed from > the parent counter? Or is there only a single child? Not only a single child, but our case is that oom-killed child consumes 90% memory, then global oom would kill it. This definitely doesn't prevent from accounting oom from other children, but we don't have to have a very accurate counter and in our case we can tell 99% oom kill happens with that specific memcg. > > Anyway, cgroup v2 will offer the hierarchical behavior. Do you have any > strong reasons that you cannot use v2? I do prefer to migrate to cgroup v2 personally. But it incurs significant work for orchestration tools, infrastructure configuration, monitoring tools, etc which are out of my control. > -- > Michal Hocko > SUSE Labs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35FF2C433DB for ; Mon, 1 Mar 2021 17:18:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A503261606 for ; Mon, 1 Mar 2021 17:18:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A503261606 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2CD568D008B; Mon, 1 Mar 2021 12:18:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 27D368D0063; Mon, 1 Mar 2021 12:18:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16C1B8D008B; Mon, 1 Mar 2021 12:18:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0173.hostedemail.com [216.40.44.173]) by kanga.kvack.org (Postfix) with ESMTP id F102E8D0063 for ; Mon, 1 Mar 2021 12:18:04 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B51964DB3 for ; Mon, 1 Mar 2021 17:18:04 +0000 (UTC) X-FDA: 77871963288.19.C554542 Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by imf29.hostedemail.com (Postfix) with ESMTP id E430F2BCE for ; Mon, 1 Mar 2021 17:18:03 +0000 (UTC) Received: by mail-ej1-f52.google.com with SMTP id r17so29594074ejy.13 for ; Mon, 01 Mar 2021 09:18:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=vv6t5g8xmRQrU3t4nwEI1adaYgpKEkMqFy3Pe4ng6DE=; b=LG5630hxJ/RU1Vhwdq9zzm9Y94EvZi4itREzN4F+bHXi9ZMDLn5fk/NhUoOeDQ92Yd RaaVkDDZrgitJ1gYmrJY2fzvTGU7Yce18FWzKGE3qN7xR6jbrM075E9XrvWw6uxbcugF tOuTJAs4Cdn9QeZeeWqMo+NCDRdTcnAy4OpLY8JEZBNV+vBRxgxIdxxbVB+/NyeSOkAK dqR4Tysj/lpoLQFDfgLSJX5NO7k4WclYROWTo7zIW2vc9YNr81iWx4/QiQF5yzt8iErt uvMQz0b5eomR+5ZominYN7ZcTEn8/+v9o7CVF6au5HruHQC0wkWXdZoGyqpQOefiwUI9 2Frg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vv6t5g8xmRQrU3t4nwEI1adaYgpKEkMqFy3Pe4ng6DE=; b=fK0iE8A9BCUCsBWp8Dt0OqB2bHcMUX0glVs5ffmNedx/NmPVi1q1+sa/coUZ4qLry0 zw/C3VUVSLHAeRsgkL85MgzHz4sifJuCBhOI8NZd/zS1LhO62ldIHemso53FVmixHuZd CmG11Mh27NMVWpDj+RlXgcZRGP7Z7CrktCessnnDMQsWMEGKECytK2w1DqGPiKr7h59k WFOHxJ1pw5/XEMzckk06Y44L0gMGSsUs0y3FvZglFxhDg9S7oRY44QuuUFXclns005b3 xNG/stOPVu1zLtX3LPVaNyM/uWfTn7xtwOK7tw/8fqCSxC2Fz+dOwE7dAdC5SXMUAe3J f/fg== X-Gm-Message-State: AOAM531UiQFB/mcCKoStAwsmuUuJL6Z/OiGak6USEu2Wu7+LBQlWSuXy ExJPnOy/NZYDDO/Ltw/f787f354YnHfrl2yVDoc= X-Google-Smtp-Source: ABdhPJx31N4JWJjQv43bkttTGqAs8q4uS7GeLxKU4ia9HhxWJv7MDghBLNRkrkoxDyZL1rgsGhbvTg6XleVLGNjVwe8= X-Received: by 2002:a17:907:2bf6:: with SMTP id gv54mr17343031ejc.514.1614619082651; Mon, 01 Mar 2021 09:18:02 -0800 (PST) MIME-Version: 1.0 References: <20210226021254.3980-1-shy828301@gmail.com> In-Reply-To: From: Yang Shi Date: Mon, 1 Mar 2021 09:17:50 -0800 Message-ID: Subject: Re: [PATCH] doc: memcontrol: add description for oom_kill To: Michal Hocko Cc: Johannes Weiner , Roman Gushchin , Shakeel Butt , Andrew Morton , Jonathan Corbet , Linux MM , Linux Kernel Mailing List Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E430F2BCE X-Stat-Signature: i7sq8pg84u4nk4d1q87mzhizmx3ywcmb Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=mail-ej1-f52.google.com; client-ip=209.85.218.52 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614619083-736984 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Mar 1, 2021 at 4:24 AM Michal Hocko wrote: > > On Fri 26-02-21 11:19:51, Yang Shi wrote: > > On Fri, Feb 26, 2021 at 8:42 AM Yang Shi wrote: > > > > > > On Thu, Feb 25, 2021 at 11:30 PM Michal Hocko wrote: > > > > > > > > On Thu 25-02-21 18:12:54, Yang Shi wrote: > > > > > When debugging an oom issue, I found the oom_kill counter of memcg is > > > > > confusing. At the first glance without checking document, I thought it > > > > > just counts for memcg oom, but it turns out it counts both global and > > > > > memcg oom. > > > > > > > > Yes, this is the case indeed. The point of the counter was to count oom > > > > victims from the memcg rather than matching that to the source of the > > > > oom. Rememeber that this could have been a memcg oom up in the > > > > hierarchy as well. Counting victims on the oom origin could be equally > > > > > > Yes, it is updated hierarchically on v2, but not on v1. I'm supposed > > > this is because v1 may work in non-hierarchcal mode? If this is the > > > only reason we may be able to remove this to get aligned with v2 since > > > non-hierarchal mode is no longer supported. > > > > BTW, having the counter recorded hierarchically may help out one of > > our usecases. We want to monitor the oom_kill for some services, but > > systemd would wipe out the cgroup if the service is oom killed then > > restart the service from scratch (it means create a brand new cgroup > > with the same name). So this systemd behavior makes the counter > > useless if it is not recorded hierarchically. > > Just to make sure I understand correctly. You have a setup where memcg > for a service has a hard limit configured and it is destroyed when oom > happens inside that memcg. A new instance is created at the same place > of the hierarchy with a new memcg. Your problem is that the oom killed > memcg will not be recorded in its parent oom event and the information > will get lost with the torn down memcg. Correct? Yes. But global oom instead of memcg oom. > > If yes then how do you tell which of the child cgroup was killed from > the parent counter? Or is there only a single child? Not only a single child, but our case is that oom-killed child consumes 90% memory, then global oom would kill it. This definitely doesn't prevent from accounting oom from other children, but we don't have to have a very accurate counter and in our case we can tell 99% oom kill happens with that specific memcg. > > Anyway, cgroup v2 will offer the hierarchical behavior. Do you have any > strong reasons that you cannot use v2? I do prefer to migrate to cgroup v2 personally. But it incurs significant work for orchestration tools, infrastructure configuration, monitoring tools, etc which are out of my control. > -- > Michal Hocko > SUSE Labs