From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DAC8C433E0 for ; Thu, 11 Feb 2021 22:17:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C6A1D64E38 for ; Thu, 11 Feb 2021 22:17:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C6A1D64E38 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 310B18D0002; Thu, 11 Feb 2021 17:17:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C2576B009B; Thu, 11 Feb 2021 17:17:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1631B8D0002; Thu, 11 Feb 2021 17:17:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id 0115F6B0096 for ; Thu, 11 Feb 2021 17:17:09 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C11D5181AF5C2 for ; Thu, 11 Feb 2021 22:17:09 +0000 (UTC) X-FDA: 77807398578.22.2A1DD85 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf29.hostedemail.com (Postfix) with ESMTP id E3A09DA for ; Thu, 11 Feb 2021 22:17:07 +0000 (UTC) Received: by mail-pg1-f175.google.com with SMTP id o21so4913081pgn.12 for ; Thu, 11 Feb 2021 14:17:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=7PN+xYXlj8kgRJMJ+MQ+hUbkTJOWzNeBYDPvx/DNPXc=; b=uIHFK7L11lzphzThVk+l5AtSJBNPtuSElxYohoXVmVYPiwk7q0GXcj90UYsEhB94f2 Ckf2i5GCVpn1v937R4iAzifVyD6bLi2eEZZIrKxaM7b/0KAZRssti7nWRtE3LULXgMIr GLuY72i7q9sJiseE8KGlblhLMTb9TjdzhGsU3D3bpCSVsc3Vp5kwXZPSoDmPrZBA8gUG pAkzWVkDgS2uXz9QHyHZkMBjyNQ1dpGbsgYXVCS0wfc7+eKOEpPolKjk46ALNGx5LzD/ 1wIWxHmaPW86jy8l2TjVHZAfgqAwF3stFJV2i+p8KEFjsTMo2cgexZDOiHcRfaJ1PCkH ZbrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=7PN+xYXlj8kgRJMJ+MQ+hUbkTJOWzNeBYDPvx/DNPXc=; b=mTyFMC8she0F2VzgDateB7wYqB6Os4noYR3E/DcOLHy3QmUp/bOIzSuIUsnQa8W4K1 uvytkwWgmTyilyKhD3uT/9SSB0q1aoB95sMLwMhM8Xw18K5C1t2txdrhhCC2/mM63nq7 cKJI4QbHX+JXE2FA6tqyNenvKd40rB7yvzXLXwq/b/+2ti1c2X1e25x8/khLGFf96l86 LcQDWVOI0mZCj8Z8T8qrCJb4VaKq6vLtTp8VZQYa7jrBWdnWJMKoL0hl/XNWiFyUQ+7I 5cmaUwy8NI+vKHsFLTNHm0KeWz78r0333qUQic9cuGUgPYZzpTSOREKHFqXVGLq3YEH/ L6lg== X-Gm-Message-State: AOAM530Vjt6sreBH4Be1U+D9E08EAyDC0PakVeCxseWD/zdRotr0JIJH MFB4cObF4IpVpPmOf0e3nRQ= X-Google-Smtp-Source: ABdhPJzuLRt1n1vVb4iLiq5BZOj8WzwMxglXa7GYOwN6rCdNwjJuNzJzAbBPB+U81a5WST4PNONbJA== X-Received: by 2002:aa7:8d0d:0:b029:1d7:3c52:e1f6 with SMTP id j13-20020aa78d0d0000b02901d73c52e1f6mr145911pfe.39.1613081828162; Thu, 11 Feb 2021 14:17:08 -0800 (PST) Received: from google.com ([2620:15c:211:201:2149:cbd5:4673:bc93]) by smtp.gmail.com with ESMTPSA id t9sm5959026pji.34.2021.02.11.14.17.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Feb 2021 14:17:07 -0800 (PST) Date: Thu, 11 Feb 2021 14:17:05 -0800 From: Minchan Kim To: Chris Goldsworthy Cc: Andrew Morton , Alexander Viro , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Matthew Wilcox Subject: Re: [PATCH v2] [RFC] mm: fs: Invalidate BH LRU during page migration Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E3A09DA X-Stat-Signature: kbjeu7cb7u3iihis6h1z3aegugeceorr Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=mail-pg1-f175.google.com; client-ip=209.85.215.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1613081827-139827 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Feb 10, 2021 at 09:35:40PM -0800, Chris Goldsworthy wrote: > Pages containing buffer_heads that are in one of the per-CPU > buffer_head LRU caches will be pinned and thus cannot be migrated. > This can prevent CMA allocations from succeeding, which are often used > on platforms with co-processors (such as a DSP) that can only use > physically contiguous memory. It can also prevent memory > hot-unplugging from succeeding, which involves migrating at least > MIN_MEMORY_BLOCK_SIZE bytes of memory, which ranges from 8 MiB to 1 > GiB based on the architecture in use. > > Correspondingly, invalidate the BH LRU caches before a migration > starts and stop any buffer_head from being cached in the LRU caches, > until migration has finished. > > Signed-off-by: Chris Goldsworthy > Cc: Minchan Kim > Cc: Matthew Wilcox > Cc: Andrew Morton > --- > fs/buffer.c | 54 +++++++++++++++++++++++++++++++++++++++++++-- > include/linux/buffer_head.h | 8 +++++++ > include/linux/migrate.h | 2 ++ > mm/migrate.c | 19 ++++++++++++++++ > mm/page_alloc.c | 3 +++ > mm/swap.c | 7 +++++- > 6 files changed, 90 insertions(+), 3 deletions(-) > > diff --git a/fs/buffer.c b/fs/buffer.c > index 96c7604..634e474 100644 > --- a/fs/buffer.c > +++ b/fs/buffer.c > @@ -1274,6 +1274,10 @@ struct bh_lru { > > static DEFINE_PER_CPU(struct bh_lru, bh_lrus) = {{ NULL }}; > > +/* These are used to control the BH LRU invalidation during page migration */ > +static struct cpumask lru_needs_invalidation; > +static bool bh_lru_disabled = false; > + > #ifdef CONFIG_SMP > #define bh_lru_lock() local_irq_disable() > #define bh_lru_unlock() local_irq_enable() > @@ -1292,7 +1296,9 @@ static inline void check_irqs_on(void) > /* > * Install a buffer_head into this cpu's LRU. If not already in the LRU, it is > * inserted at the front, and the buffer_head at the back if any is evicted. > - * Or, if already in the LRU it is moved to the front. > + * Or, if already in the LRU it is moved to the front. Note that if LRU is > + * disabled because of an ongoing page migration, we won't insert bh into the > + * LRU. > */ > static void bh_lru_install(struct buffer_head *bh) > { > @@ -1303,6 +1309,9 @@ static void bh_lru_install(struct buffer_head *bh) > check_irqs_on(); > bh_lru_lock(); > > + if (bh_lru_disabled) > + goto out; > + > b = this_cpu_ptr(&bh_lrus); > for (i = 0; i < BH_LRU_SIZE; i++) { > swap(evictee, b->bhs[i]); > @@ -1313,6 +1322,7 @@ static void bh_lru_install(struct buffer_head *bh) > } > > get_bh(bh); > +out: > bh_lru_unlock(); > brelse(evictee); > } > @@ -1328,6 +1338,10 @@ lookup_bh_lru(struct block_device *bdev, sector_t block, unsigned size) > > check_irqs_on(); > bh_lru_lock(); > + > + if (bh_lru_disabled) > + goto out; > + > for (i = 0; i < BH_LRU_SIZE; i++) { > struct buffer_head *bh = __this_cpu_read(bh_lrus.bhs[i]); > > @@ -1346,6 +1360,7 @@ lookup_bh_lru(struct block_device *bdev, sector_t block, unsigned size) > break; > } > } > +out: > bh_lru_unlock(); > return ret; > } > @@ -1446,7 +1461,7 @@ EXPORT_SYMBOL(__bread_gfp); > * This doesn't race because it runs in each cpu either in irq > * or with preempt disabled. > */ > -static void invalidate_bh_lru(void *arg) > +void invalidate_bh_lru(void *arg) > { > struct bh_lru *b = &get_cpu_var(bh_lrus); > int i; > @@ -1477,6 +1492,41 @@ void invalidate_bh_lrus(void) > } > EXPORT_SYMBOL_GPL(invalidate_bh_lrus); > > +bool need_bh_lru_invalidation(int cpu) > +{ > + return cpumask_test_cpu(cpu, &lru_needs_invalidation); > +} > + > +void bh_lru_disable(void) > +{ > + int cpu; > + > + bh_lru_disabled = true; What happens if the function is nested? Shouldn't we make it count? So only disble when the count is zero.