From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58F6DC433F5 for ; Tue, 22 Mar 2022 08:14:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230245AbiCVIPq (ORCPT ); Tue, 22 Mar 2022 04:15:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229718AbiCVIPo (ORCPT ); Tue, 22 Mar 2022 04:15:44 -0400 Received: from mail-vs1-xe2b.google.com (mail-vs1-xe2b.google.com [IPv6:2607:f8b0:4864:20::e2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B68833A13 for ; Tue, 22 Mar 2022 01:14:17 -0700 (PDT) Received: by mail-vs1-xe2b.google.com with SMTP id 2so10265213vse.4 for ; Tue, 22 Mar 2022 01:14:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ejYqrEpcN+YIapqTaek6UuldGWcXdxvGA3dM/VeJRBw=; b=BUFmL4heEnQlaoWLUylgiBVgyrJVVkojiCMeAXJFkN+klvf/L4GwYq1ryCISPtRUGS si+JBR7yCvBp7J4j0vIyULINRl51j4D2xaSkm1tVlpAqr9ogkF+C20iqQWzgRJKDf6xi 1M1mqaHI6o7a5+6Yz7jK1F1rJL5JiJwjft7xaB1CBGWFhb62E9R6oyLDzXkkzC3kDjcn ZRn5V0IZBgmPnaC5K5gxyqsWjQoKO32xo7oY4B415L1eK0tIF9nzSwNhbzEasO4BBShX VTjugLCAiW2YsPV+ybWh1e1Uw+wun1LOiGG/FxAAD80r/3EsiSoGEhgcnIbXMnNCJNGK M2eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ejYqrEpcN+YIapqTaek6UuldGWcXdxvGA3dM/VeJRBw=; b=R8CmRJpVsPDkgP327aYi/JqbCBZqjP4Tp2dlVmswHrN3PwhEn9fDXQkyXXK9rpyYu0 rfJP9eASxMx1Lk0M/pFnxjsY4bdJoPcAZmda7eJdUZvilJDOKXdP6vLAXSLQceQU0i/q UyY0lBYylt9/iesCeprXX7/dSRRlCGDRXy/XJRkRSnr2hVUR1lnQN5BMAhoKaxKvlWbs HTpfdJZQ7sgHUDTkiSg6BeCiwGHQkUb1QoXG7eSph1Qp8+AU4bquNLl0aFBh9+IjhTCF 4Xejg+2K1VCotvKwYidqjxuh4uRcT8GqYVkSxv1TIWUJ8G1gthwYQ3LYsID+5v1Vf8hn HMsw== X-Gm-Message-State: AOAM533Ehm19ErIHCLaqxtN52aeSI87jomSF9ZgYnClg5rCFnvmflQ/h 10nZdjn021YzPk1kW+h+U8J8JYOFeRoeNoCgCnvXqA== X-Google-Smtp-Source: ABdhPJyS4rjE3E91EMtptKGTaCGpsQomyOoF1sxK53qyi6zy6Jm6Br78Kyg19z+h1BvSAwQv9ZiyDuNmYhVUsBl/EOw= X-Received: by 2002:a67:f956:0:b0:324:eb38:52fb with SMTP id u22-20020a67f956000000b00324eb3852fbmr5339929vsq.22.1647936855962; Tue, 22 Mar 2022 01:14:15 -0700 (PDT) MIME-Version: 1.0 References: <20220309021230.721028-1-yuzhao@google.com> <20220309021230.721028-12-yuzhao@google.com> In-Reply-To: From: Yu Zhao Date: Tue, 22 Mar 2022 02:14:04 -0600 Message-ID: Subject: Re: [PATCH v9 11/14] mm: multi-gen LRU: thrashing prevention To: Barry Song <21cnbao@gmail.com> Cc: Andrew Morton , Linus Torvalds , Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Jesse Barnes , Johannes Weiner , Jonathan Corbet , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Rik van Riel , Vlastimil Babka , Will Deacon , Ying Huang , LAK , Linux Doc Mailing List , LKML , Linux-MM , Kernel Page Reclaim v2 , x86 , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , =?UTF-8?Q?Holger_Hoffst=C3=A4tte?= , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 22, 2022 at 1:23 AM Barry Song <21cnbao@gmail.com> wrote: > > On Wed, Mar 9, 2022 at 3:48 PM Yu Zhao wrote: > > > > Add /sys/kernel/mm/lru_gen/min_ttl_ms for thrashing prevention, as > > requested by many desktop users [1]. > > > > When set to value N, it prevents the working set of N milliseconds > > from getting evicted. The OOM killer is triggered if this working set > > cannot be kept in memory. Based on the average human detectable lag > > (~100ms), N=1000 usually eliminates intolerable lags due to thrashing. > > Larger values like N=3000 make lags less noticeable at the risk of > > premature OOM kills. > > > > Compared with the size-based approach, e.g., [2], this time-based > > approach has the following advantages: > > 1. It is easier to configure because it is agnostic to applications > > and memory sizes. > > 2. It is more reliable because it is directly wired to the OOM killer. > > > > how are userspace oom daemons like android lmkd, systemd-oomd supposed > to work with this time-based oom killer? > only one of min_ttl_ms and userspace daemon should be enabled? or both > should be enabled at the same time? Generally we just need one. lmkd and oomd are more flexible but 1) they need customizations 2) not all distros have them 3) they might be stuck in direct reclaim as well. The last remark is not just a theoretical problem: a) we had many servers under extremely heavy (global) memory pressure, that 200+ direct reclaimers on each CPU competed for resources and userspace livelocked for 2 hours. Eventually hardware watchdogs kicked in. b) on Chromebooks we have something similar to lmkd, and we still frequently observe crashes due to heavy memory pressure, meaning some Chrome tabs were stuck in direct reclaim for 120 seconds (hung_task_timeout_secs=120). From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7EA9FC433EF for ; Tue, 22 Mar 2022 08:15:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OPQs4Wv9bXT64AQM2xpwWd6Lpcim6yr+y9+Ux/T9V+4=; b=Yhl3SfG4q0bgvj BvE1z8ezZUXs9RvLz/5M8hx49v9EurazEMe8TUmdp2fHcXlEd+RjgOpuZ3H2+ntOSb/eXQyVmBLmM AZvkGVNMMWgfbub6FJogEr1Ox1BPgBdLepeJWBYZq+gznmGthP8sWhvgS0+IfE4GdtCe02agzAV5l Fau6Rzfhz8Dew6VK/CEYQTqVwEl01/bt5E5J4d/IzZqFlgzpgUUQyeYW+J5hpcNAtaGBVAIxhthr/ Audua7oVWEBAh7iq9bH7wb/iPKGw0bhaUszFGWYf//rZstPh31m5vQvQPck5XOXStRvViGkvnBx1/ 8s74oDz+SvR/VmEzJRiQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nWZen-00ALor-Cb; Tue, 22 Mar 2022 08:14:21 +0000 Received: from mail-vs1-xe36.google.com ([2607:f8b0:4864:20::e36]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nWZej-00ALnK-Le for linux-arm-kernel@lists.infradead.org; Tue, 22 Mar 2022 08:14:19 +0000 Received: by mail-vs1-xe36.google.com with SMTP id 2so10265214vse.4 for ; Tue, 22 Mar 2022 01:14:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ejYqrEpcN+YIapqTaek6UuldGWcXdxvGA3dM/VeJRBw=; b=BUFmL4heEnQlaoWLUylgiBVgyrJVVkojiCMeAXJFkN+klvf/L4GwYq1ryCISPtRUGS si+JBR7yCvBp7J4j0vIyULINRl51j4D2xaSkm1tVlpAqr9ogkF+C20iqQWzgRJKDf6xi 1M1mqaHI6o7a5+6Yz7jK1F1rJL5JiJwjft7xaB1CBGWFhb62E9R6oyLDzXkkzC3kDjcn ZRn5V0IZBgmPnaC5K5gxyqsWjQoKO32xo7oY4B415L1eK0tIF9nzSwNhbzEasO4BBShX VTjugLCAiW2YsPV+ybWh1e1Uw+wun1LOiGG/FxAAD80r/3EsiSoGEhgcnIbXMnNCJNGK M2eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ejYqrEpcN+YIapqTaek6UuldGWcXdxvGA3dM/VeJRBw=; b=r4cAjrhYn5KDBilEtHAOAJuxGuzvuiWkW1o7WWRz63epKq3IbQl9qYK5524rBPgigs RkjJ7u9UhGUmrXv5EzYRo/B5vXhMHp6QlOZk6LGj24Bl+f8PERnk0K4Pp+BiL7OSNYna lSbneuyGEKqelK+WfF03YPobeOMImqgXM41fzZ/W00yXaVWed1jfgCiJ9JZekksknd2v eUToxtz08Sbdqh8w2kWc0vskLHEY9757Tku0GcvgGvMu620AXyKSMJdrHQ3smbi+3JIA 37ebPhQ6SXIFPofjn7rxs8tOKYqms0qbO3Q9ifORAqa8OIw6aNWm4Wrv52jToM7LqZPD uoUQ== X-Gm-Message-State: AOAM530HW+cqvCPg/LF77ycaINRLtb/X1XzNQXJq/I0HN+T4Xs4TkAz0 Wp/+FOvEyk7bMbV0ZZa4UChfeVdgJIJXBHYjlfRSLw== X-Google-Smtp-Source: ABdhPJyS4rjE3E91EMtptKGTaCGpsQomyOoF1sxK53qyi6zy6Jm6Br78Kyg19z+h1BvSAwQv9ZiyDuNmYhVUsBl/EOw= X-Received: by 2002:a67:f956:0:b0:324:eb38:52fb with SMTP id u22-20020a67f956000000b00324eb3852fbmr5339929vsq.22.1647936855962; Tue, 22 Mar 2022 01:14:15 -0700 (PDT) MIME-Version: 1.0 References: <20220309021230.721028-1-yuzhao@google.com> <20220309021230.721028-12-yuzhao@google.com> In-Reply-To: From: Yu Zhao Date: Tue, 22 Mar 2022 02:14:04 -0600 Message-ID: Subject: Re: [PATCH v9 11/14] mm: multi-gen LRU: thrashing prevention To: Barry Song <21cnbao@gmail.com> Cc: Andrew Morton , Linus Torvalds , Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Jesse Barnes , Johannes Weiner , Jonathan Corbet , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Rik van Riel , Vlastimil Babka , Will Deacon , Ying Huang , LAK , Linux Doc Mailing List , LKML , Linux-MM , Kernel Page Reclaim v2 , x86 , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , =?UTF-8?Q?Holger_Hoffst=C3=A4tte?= , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220322_011417_741265_F40E4541 X-CRM114-Status: GOOD ( 21.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Mar 22, 2022 at 1:23 AM Barry Song <21cnbao@gmail.com> wrote: > > On Wed, Mar 9, 2022 at 3:48 PM Yu Zhao wrote: > > > > Add /sys/kernel/mm/lru_gen/min_ttl_ms for thrashing prevention, as > > requested by many desktop users [1]. > > > > When set to value N, it prevents the working set of N milliseconds > > from getting evicted. The OOM killer is triggered if this working set > > cannot be kept in memory. Based on the average human detectable lag > > (~100ms), N=1000 usually eliminates intolerable lags due to thrashing. > > Larger values like N=3000 make lags less noticeable at the risk of > > premature OOM kills. > > > > Compared with the size-based approach, e.g., [2], this time-based > > approach has the following advantages: > > 1. It is easier to configure because it is agnostic to applications > > and memory sizes. > > 2. It is more reliable because it is directly wired to the OOM killer. > > > > how are userspace oom daemons like android lmkd, systemd-oomd supposed > to work with this time-based oom killer? > only one of min_ttl_ms and userspace daemon should be enabled? or both > should be enabled at the same time? Generally we just need one. lmkd and oomd are more flexible but 1) they need customizations 2) not all distros have them 3) they might be stuck in direct reclaim as well. The last remark is not just a theoretical problem: a) we had many servers under extremely heavy (global) memory pressure, that 200+ direct reclaimers on each CPU competed for resources and userspace livelocked for 2 hours. Eventually hardware watchdogs kicked in. b) on Chromebooks we have something similar to lmkd, and we still frequently observe crashes due to heavy memory pressure, meaning some Chrome tabs were stuck in direct reclaim for 120 seconds (hung_task_timeout_secs=120). _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel