From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D14FC433DB for ; Tue, 5 Jan 2021 07:06:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 631D122482 for ; Tue, 5 Jan 2021 07:06:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 631D122482 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E1BC58D0058; Tue, 5 Jan 2021 02:06:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DCBD78D0036; Tue, 5 Jan 2021 02:06:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6E628D0058; Tue, 5 Jan 2021 02:06:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0028.hostedemail.com [216.40.44.28]) by kanga.kvack.org (Postfix) with ESMTP id AA1468D0036 for ; Tue, 5 Jan 2021 02:06:01 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6F23F1EE6 for ; Tue, 5 Jan 2021 07:06:01 +0000 (UTC) X-FDA: 77670836922.18.ice87_4611d27274d7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 3CBB1100EC668 for ; Tue, 5 Jan 2021 07:06:01 +0000 (UTC) X-HE-Tag: ice87_4611d27274d7 X-Filterd-Recvd-Size: 7954 Received: from so254-31.mailgun.net (so254-31.mailgun.net [198.61.254.31]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Tue, 5 Jan 2021 07:05:58 +0000 (UTC) DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1609830360; h=References: In-Reply-To: References: In-Reply-To: Message-Id: Date: Subject: Cc: To: From: Sender; bh=PSD4T8ASrr1pfYU48t/ftAURhWN9+rkwMmpAWedYLBE=; b=j9ZtZjPfmXXXh4MJ5t2kF3G5RIumbC99L64g99jpp6JfNMXjDlSKPJaOnX5pTK7iN+MF4gcp 12myAoxAevHqsE/yKDkgJPc2PGelGdb0sX7iwYFwUOUdEgAjccApjl7wYdGYQVP4PclKFeL5 db+xQ41rIUX4WKUPhfYle5OtfKM= X-Mailgun-Sending-Ip: 198.61.254.31 X-Mailgun-Sid: WyIwY2Q3OCIsICJsaW51eC1tbUBrdmFjay5vcmciLCAiYmU5ZTRhIl0= Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n05.prod.us-west-2.postgun.com with SMTP id 5ff40fcad3eb3c36b475231f (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Tue, 05 Jan 2021 07:05:46 GMT Received: by smtp.codeaurora.org (Postfix, from userid 1001) id 4209EC43464; Tue, 5 Jan 2021 07:05:46 +0000 (UTC) Received: from cgoldswo-linux.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: cgoldswo) by smtp.codeaurora.org (Postfix) with ESMTPSA id 89BFBC433C6; Tue, 5 Jan 2021 07:05:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 89BFBC433C6 Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; spf=fail smtp.mailfrom=cgoldswo@codeaurora.org From: Chris Goldsworthy To: Alexander Viro , linux-fsdevel@vger.kernel.org, Matthew Wilcox , linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Laura Abbott , Chris Goldsworthy Subject: [PATCH v2] fs/buffer.c: Revoke LRU when trying to drop buffers Date: Mon, 4 Jan 2021 23:05:33 -0800 Message-Id: X-Mailer: git-send-email 2.7.4 In-Reply-To: References: In-Reply-To: References: X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Laura Abbott When a buffer is added to the LRU list, a reference is taken which is not dropped until the buffer is evicted from the LRU list. This is the correct behavior, however this LRU reference will prevent the buffer from being dropped. This means that the buffer can't actually be dropped until it is selected for eviction. There's no bound on the time spent on the LRU list, which means that the buffer may be undroppable for very long periods of time. Given that migration involves dropping buffers, the associated page is now unmigratible for long periods of time as well. CMA relies on being able to migrate a specific range of pages, so these types of failures make CMA significantly less reliable, especially under high filesystem usage. Rather than waiting for the LRU algorithm to eventually kick out the buffer, explicitly remove the buffer from the LRU list when trying to drop it. There is still the possibility that the buffer could be added back on the list, but that indicates the buffer is still in use and would probably have other 'in use' indicates to prevent dropping. Signed-off-by: Laura Abbott Signed-off-by: Chris Goldsworthy Cc: Matthew Wilcox --- fs/buffer.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++---- fs/internal.h | 5 ++++ 2 files changed, 85 insertions(+), 5 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 96c7604..536fb5b 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -48,6 +48,7 @@ #include #include #include +#include #include "internal.h" @@ -1471,12 +1472,63 @@ static bool has_bh_in_lru(int cpu, void *dummy) return false; } +static void __evict_bhs_lru(void *arg) +{ + struct bh_lru *b = &get_cpu_var(bh_lrus); + struct busy_bhs_container *busy_bhs = arg; + struct buffer_head *bh; + int i; + + XA_STATE(xas, &busy_bhs->xarray, 0); + + xas_for_each(&xas, bh, busy_bhs->size) { + for (i = 0; i < BH_LRU_SIZE; i++) { + if (b->bhs[i] == bh) { + brelse(b->bhs[i]); + b->bhs[i] = NULL; + break; + } + } + + bh = bh->b_this_page; + } + + put_cpu_var(bh_lrus); +} + +static bool page_has_bhs_in_lru(int cpu, void *arg) +{ + struct bh_lru *b = per_cpu_ptr(&bh_lrus, cpu); + struct busy_bhs_container *busy_bhs = arg; + struct buffer_head *bh; + int i; + + XA_STATE(xas, &busy_bhs->xarray, 0); + + xas_for_each(&xas, bh, busy_bhs->size) { + for (i = 0; i < BH_LRU_SIZE; i++) { + if (b->bhs[i] == bh) + return true; + } + + bh = bh->b_this_page; + } + + return false; + +} void invalidate_bh_lrus(void) { on_each_cpu_cond(has_bh_in_lru, invalidate_bh_lru, NULL, 1); } EXPORT_SYMBOL_GPL(invalidate_bh_lrus); +static void evict_bh_lrus(struct busy_bhs_container *busy_bhs) +{ + on_each_cpu_cond(page_has_bhs_in_lru, __evict_bhs_lru, + busy_bhs, 1); +} + void set_bh_page(struct buffer_head *bh, struct page *page, unsigned long offset) { @@ -3242,14 +3294,36 @@ drop_buffers(struct page *page, struct buffer_head **buffers_to_free) { struct buffer_head *head = page_buffers(page); struct buffer_head *bh; + struct busy_bhs_container busy_bhs; + int xa_ret, ret = 0; + + xa_init(&busy_bhs.xarray); + busy_bhs.size = 0; bh = head; do { - if (buffer_busy(bh)) - goto failed; + if (buffer_busy(bh)) { + xa_ret = xa_err(xa_store(&busy_bhs.xarray, busy_bhs.size++, + bh, GFP_ATOMIC)); + if (xa_ret) + goto out; + } bh = bh->b_this_page; } while (bh != head); + if (busy_bhs.size) { + /* + * Check if the busy failure was due to an outstanding + * LRU reference + */ + evict_bh_lrus(&busy_bhs); + do { + if (buffer_busy(bh)) + goto out; + } while (bh != head); + } + + ret = 1; do { struct buffer_head *next = bh->b_this_page; @@ -3259,9 +3333,10 @@ drop_buffers(struct page *page, struct buffer_head **buffers_to_free) } while (bh != head); *buffers_to_free = head; detach_page_private(page); - return 1; -failed: - return 0; +out: + xa_destroy(&busy_bhs.xarray); + + return ret; } int try_to_free_buffers(struct page *page) diff --git a/fs/internal.h b/fs/internal.h index 77c50be..00f17c4 100644 --- a/fs/internal.h +++ b/fs/internal.h @@ -15,6 +15,7 @@ struct mount; struct shrink_control; struct fs_context; struct user_namespace; +struct xarray; /* * block_dev.c @@ -49,6 +50,10 @@ static inline int emergency_thaw_bdev(struct super_block *sb) */ extern int __block_write_begin_int(struct page *page, loff_t pos, unsigned len, get_block_t *get_block, struct iomap *iomap); +struct busy_bhs_container { + struct xarray xarray; + int size; +}; /* * char_dev.c -- The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project