From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4DC8C433F5 for ; Thu, 2 Dec 2021 02:01:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350264AbhLBCFQ (ORCPT ); Wed, 1 Dec 2021 21:05:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36956 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231330AbhLBCEy (ORCPT ); Wed, 1 Dec 2021 21:04:54 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B705C061574 for ; Wed, 1 Dec 2021 18:01:32 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 66F68CE2146 for ; Thu, 2 Dec 2021 02:01:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 05484C53FCD; Thu, 2 Dec 2021 02:01:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638410488; bh=lPoQ0AgBRO5yDAwAe1LtBJ93RGqyiwFuQSqf/evEv2A=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=bUFZeL978wSPNwFDkPJUlek4br9n+54FBbeIJssvEpxX2mF/04yTyB1nFVOR7MGVj oswpkeEVv951Py9zdSDOAcXXaZ0Ww1xCUTWf1WCZvVRR5W/AJCH3iMTeGLP8AfXb8k LCAul9/GVovTjWBTKe7/PhCVjAza+IvgmYc/oOAge05ziDjg1u1xNKv0bVM8dLyNi3 cgjcMy8ex2ra0vlnCKPxyyWWKEyQEdSF7iKMqMWDd47C8ITqd2FY+hSBE86rmYdxu4 pwfLhINrl1kx0gf+bXa1B7AxHgw6T1RzI49YdVnDgSQTMJpVPKUsBiU6x0VC7ojdME xXeIih3hdu2tA== Message-ID: Date: Thu, 2 Dec 2021 10:01:19 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.3.1 Subject: Re: [f2fs-dev] [PATCH Vx 1/1] f2fs: Avoid deadlock between writeback and checkpoint Content-Language: en-US To: Jaegeuk Kim Cc: niuzhiguo84@gmail.com, Jing.Xia@unisoc.com, linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net References: <1636438608-27597-1-git-send-email-niuzhiguo84@gmail.com> From: Chao Yu In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2021/12/2 6:20, Jaegeuk Kim wrote: > On 11/20, Chao Yu wrote: >> On 2021/11/18 14:46, Chao Yu wrote: >>> On 2021/11/18 0:56, Jaegeuk Kim wrote: >>>> On 11/09, niuzhiguo84@gmail.com wrote: >>>>> From: Zhiguo Niu >>>>> >>>>> There could be a scenario as following: >>>>> The inodeA and inodeB are in b_io queue of writeback >>>>> inodeA : f2fs's node inode >>>>> inodeB : a dir inode with only one dirty pages, and the node page >>>>> of inodeB cached into inodeA >>>>> >>>>> writeback: >>>>> >>>>> wb_workfn >>>>> wb_writeback >>>>> blk_start_plug >>>>> loop { >>>>> queue_io >>>>> progress=__writeback_inodes_wb >>>>> __writeback_single_inode >>>>> do_writepages >>>>> f2fs_write_data_pages >>>>> wbc->pages_skipped +=get_dirty_pages >>>>> inode->i_state &= ~dirty >>>>> wrote++ >>>>> requeue_inode >>>>> } >>>>> blk_finish_plug >>>>> >>>>> checkpoint: >>>>> >>>>> f2fs_write_checkpoint >>>>> f2fs_sync_dirty_inodes >>>>> filemap_fdatawrite >>>>> do_writepages >>>>> f2fs_write_data_pages >>>>> f2fs_write_single_data_page >>>>> f2fs_do_write_data_page >>>>> set_page_writeback >>>>> f2fs_outplace_write_data >>>>> f2fs_update_data_blkaddr >>>>> f2fs_wait_on_page_writeback >>>>> inode_dec_dirty_pages >>>>> >>>>> 1. Writeback thread flush inodeA, and push it's bio request in task's plug; >>>>> 2. Checkpoint thread writes inodeB's dirty page, and then wait its node >>>>> page writeback cached into inodeA which is in writeback task's plug >>>>> 3. Writeback thread flush inodeB and skip writing the dirty page as >>>>> wb_sync_req[DATA] > 0. >>>>> 4. As none of the inodeB's page is marked as PAGECACHE_TAG_DIRTY, writeback >>>>> thread clear inodeB's dirty state. >>>>> 5. Then inodeB is moved from b_io to b_dirty because of pages_skipped > 0 >>>>> as checkpoint thread is stuck before dec dirty_pages. >>>>> >>>>> This patch collect correct pages_skipped according to the tag state in >>>>> page tree of inode >>>>> >>>>> Signed-off-by: Zhiguo Niu >>>>> Signed-off-by: Jing Xia >>>>> --- >>>>> fs/f2fs/data.c | 4 +++- >>>>> 1 file changed, 3 insertions(+), 1 deletion(-) >>>>> >>>>> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c >>>>> index f4fd6c246c9a..e98628e3868c 100644 >>>>> --- a/fs/f2fs/data.c >>>>> +++ b/fs/f2fs/data.c >>>>> @@ -3237,7 +3237,9 @@ static int __f2fs_write_data_pages(struct address_space *mapping, >>>>> return ret; >>>>> skip_write: >>>>> - wbc->pages_skipped += get_dirty_pages(inode); >>>>> + wbc->pages_skipped += >>>>> + mapping_tagged(inode->i_mapping, PAGECACHE_TAG_DIRTY) ? >>>> >>>> Is there any race condition to get 0, if there's any dirty page? IOWs, it >>> >>> Quoted from Jing Xia's explanation: >>> >>> [T:writeback] [T:checkpoint] >> >> My bad, [1] should be here: >> >> bio contains NodeA was plugged in writeback threads >> >> Thanks, >> >>> - do_writepages -- sync write inodeB, inc wb_sync_req[DATA] >>> - f2fs_write_data_pages >>> - f2fs_write_single_data_page -- write last dirty page >>> - f2fs_do_write_data_page >>> - set_page_writeback -- clear page dirty flag and >>> PAGECACHE_TAG_DIRTY tag in radix tree >>> - f2fs_outplace_write_data >>> - f2fs_update_data_blkaddr >>> - f2fs_wait_on_page_writeback -- wait NodeA to writeback here >>> - inode_dec_dirty_pages >> >>> bio contains NodeA was plugged in writeback threads >> >> [1] >> >> Thanks, >> >>> - writeback_sb_inodes >>> - writeback_single_inode >>> - do_writepages >>> - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA] >>> - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one >>> - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped > > So, my question was why this is the problem? kworker will loop writebacking this requeued inode. Thanks, > >>> >>>> seems the current condition is just requeuing the inode as dirty, but next >>>> flushing time will remove it from dirty list. Is this giving too much overheads? >>> >>> I prefer to let writeback thread call blk_flush_plug() after skipping >>> writepages() due to wb_sync_req[DATA/NODE] check condition, thoughts? >>> >>> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c >>> index 9f754aaef558..b6e1ed73f8f5 100644 >>> --- a/fs/f2fs/data.c >>> +++ b/fs/f2fs/data.c >>> @@ -3087,6 +3087,8 @@ static int f2fs_write_cache_pages(struct address_space *mapping, >>> /* give a priority to WB_SYNC threads */ >>> if (atomic_read(&sbi->wb_sync_req[DATA]) && >>> wbc->sync_mode == WB_SYNC_NONE) { >>> + if (current->plug) >>> + blk_flush_plug(current->plug, false); >>> done = 1; >>> break; >>> } >>> diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c >>> index 556fcd8457f3..dd9a817d8dab 100644 >>> --- a/fs/f2fs/node.c >>> +++ b/fs/f2fs/node.c >>> @@ -1946,6 +1946,8 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, >>> if (atomic_read(&sbi->wb_sync_req[NODE]) && >>> wbc->sync_mode == WB_SYNC_NONE) { >>> done = 1; >>> + if (current->plug) >>> + blk_flush_plug(current->plug, false); >>> break; >>> } >>> >>> >>> >>> Thanks, >>> >>>> >>>>> + get_dirty_pages(inode) : 0; >>>>> trace_f2fs_writepages(mapping->host, wbc, DATA); >>>>> return 0; >>>>> } >>>>> -- >>>>> 2.28.0 >>> >>> >>> _______________________________________________ >>> Linux-f2fs-devel mailing list >>> Linux-f2fs-devel@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel >>> From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63D1AC433F5 for ; Thu, 2 Dec 2021 02:01:52 +0000 (UTC) Received: from [127.0.0.1] (helo=sfs-ml-1.v29.lw.sourceforge.com) by sfs-ml-1.v29.lw.sourceforge.com with esmtp (Exim 4.94.2) (envelope-from ) id 1msbPv-0005E5-7O; Thu, 02 Dec 2021 02:01:48 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-1.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1msbPt-0005Dz-Am for linux-f2fs-devel@lists.sourceforge.net; Thu, 02 Dec 2021 02:01:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: From:References:Cc:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=zivQdiHLvRNXvOghLUDLL7SVwqg5WgyPsBXYrEtLIOI=; b=nUR6xocRctRiqm3exuRVkQqh2C 6WtcE74f9TjNpZTUmKGHEBKec/Aaue5UTp2h1l5gOOoJhirKeqtQg7Fh35hSaNr8h1wo0ZL+RjKuA 0B07Jboh/4bQj+h9W6IQJLw/raDDIRFfNblrfOV+7oaphCZD/PWM+mgGVD9jCN5/y6nQ=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=zivQdiHLvRNXvOghLUDLL7SVwqg5WgyPsBXYrEtLIOI=; b=M1aMBRRqT2OuoX42ARwaePrirt Jv0oJ5R1gsGE4+2wrQjTBAVTDfiqsehx7NVzN5KG8HAhVx2iw5mL21Kq/AxCHcqly9VHZgwUBwWSd OliXf8Ge0ycRfvAfd9aphDcq9RjYA7t6YuYCTk9T+Bjm/Bx4nA0NslI7omlajtxL3Lms=; Received: from ams.source.kernel.org ([145.40.68.75]) by sfi-mx-1.v28.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92.3) id 1msbPs-000v0C-7i for linux-f2fs-devel@lists.sourceforge.net; Thu, 02 Dec 2021 02:01:46 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DC322B821BB; Thu, 2 Dec 2021 02:01:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 05484C53FCD; Thu, 2 Dec 2021 02:01:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638410488; bh=lPoQ0AgBRO5yDAwAe1LtBJ93RGqyiwFuQSqf/evEv2A=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=bUFZeL978wSPNwFDkPJUlek4br9n+54FBbeIJssvEpxX2mF/04yTyB1nFVOR7MGVj oswpkeEVv951Py9zdSDOAcXXaZ0Ww1xCUTWf1WCZvVRR5W/AJCH3iMTeGLP8AfXb8k LCAul9/GVovTjWBTKe7/PhCVjAza+IvgmYc/oOAge05ziDjg1u1xNKv0bVM8dLyNi3 cgjcMy8ex2ra0vlnCKPxyyWWKEyQEdSF7iKMqMWDd47C8ITqd2FY+hSBE86rmYdxu4 pwfLhINrl1kx0gf+bXa1B7AxHgw6T1RzI49YdVnDgSQTMJpVPKUsBiU6x0VC7ojdME xXeIih3hdu2tA== Message-ID: Date: Thu, 2 Dec 2021 10:01:19 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.3.1 Content-Language: en-US To: Jaegeuk Kim References: <1636438608-27597-1-git-send-email-niuzhiguo84@gmail.com> From: Chao Yu In-Reply-To: X-Headers-End: 1msbPs-000v0C-7i Subject: Re: [f2fs-dev] [PATCH Vx 1/1] f2fs: Avoid deadlock between writeback and checkpoint X-BeenThere: linux-f2fs-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jing.Xia@unisoc.com, niuzhiguo84@gmail.com, linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net On 2021/12/2 6:20, Jaegeuk Kim wrote: > On 11/20, Chao Yu wrote: >> On 2021/11/18 14:46, Chao Yu wrote: >>> On 2021/11/18 0:56, Jaegeuk Kim wrote: >>>> On 11/09, niuzhiguo84@gmail.com wrote: >>>>> From: Zhiguo Niu >>>>> >>>>> There could be a scenario as following: >>>>> The inodeA and inodeB are in b_io queue of writeback >>>>> inodeA : f2fs's node inode >>>>> inodeB : a dir inode with only one dirty pages, and the node page >>>>> of inodeB cached into inodeA >>>>> >>>>> writeback: >>>>> >>>>> wb_workfn >>>>> wb_writeback >>>>> blk_start_plug >>>>> loop { >>>>> queue_io >>>>> progress=__writeback_inodes_wb >>>>> __writeback_single_inode >>>>> do_writepages >>>>> f2fs_write_data_pages >>>>> wbc->pages_skipped +=get_dirty_pages >>>>> inode->i_state &= ~dirty >>>>> wrote++ >>>>> requeue_inode >>>>> } >>>>> blk_finish_plug >>>>> >>>>> checkpoint: >>>>> >>>>> f2fs_write_checkpoint >>>>> f2fs_sync_dirty_inodes >>>>> filemap_fdatawrite >>>>> do_writepages >>>>> f2fs_write_data_pages >>>>> f2fs_write_single_data_page >>>>> f2fs_do_write_data_page >>>>> set_page_writeback >>>>> f2fs_outplace_write_data >>>>> f2fs_update_data_blkaddr >>>>> f2fs_wait_on_page_writeback >>>>> inode_dec_dirty_pages >>>>> >>>>> 1. Writeback thread flush inodeA, and push it's bio request in task's plug; >>>>> 2. Checkpoint thread writes inodeB's dirty page, and then wait its node >>>>> page writeback cached into inodeA which is in writeback task's plug >>>>> 3. Writeback thread flush inodeB and skip writing the dirty page as >>>>> wb_sync_req[DATA] > 0. >>>>> 4. As none of the inodeB's page is marked as PAGECACHE_TAG_DIRTY, writeback >>>>> thread clear inodeB's dirty state. >>>>> 5. Then inodeB is moved from b_io to b_dirty because of pages_skipped > 0 >>>>> as checkpoint thread is stuck before dec dirty_pages. >>>>> >>>>> This patch collect correct pages_skipped according to the tag state in >>>>> page tree of inode >>>>> >>>>> Signed-off-by: Zhiguo Niu >>>>> Signed-off-by: Jing Xia >>>>> --- >>>>> fs/f2fs/data.c | 4 +++- >>>>> 1 file changed, 3 insertions(+), 1 deletion(-) >>>>> >>>>> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c >>>>> index f4fd6c246c9a..e98628e3868c 100644 >>>>> --- a/fs/f2fs/data.c >>>>> +++ b/fs/f2fs/data.c >>>>> @@ -3237,7 +3237,9 @@ static int __f2fs_write_data_pages(struct address_space *mapping, >>>>> return ret; >>>>> skip_write: >>>>> - wbc->pages_skipped += get_dirty_pages(inode); >>>>> + wbc->pages_skipped += >>>>> + mapping_tagged(inode->i_mapping, PAGECACHE_TAG_DIRTY) ? >>>> >>>> Is there any race condition to get 0, if there's any dirty page? IOWs, it >>> >>> Quoted from Jing Xia's explanation: >>> >>> [T:writeback] [T:checkpoint] >> >> My bad, [1] should be here: >> >> bio contains NodeA was plugged in writeback threads >> >> Thanks, >> >>> - do_writepages -- sync write inodeB, inc wb_sync_req[DATA] >>> - f2fs_write_data_pages >>> - f2fs_write_single_data_page -- write last dirty page >>> - f2fs_do_write_data_page >>> - set_page_writeback -- clear page dirty flag and >>> PAGECACHE_TAG_DIRTY tag in radix tree >>> - f2fs_outplace_write_data >>> - f2fs_update_data_blkaddr >>> - f2fs_wait_on_page_writeback -- wait NodeA to writeback here >>> - inode_dec_dirty_pages >> >>> bio contains NodeA was plugged in writeback threads >> >> [1] >> >> Thanks, >> >>> - writeback_sb_inodes >>> - writeback_single_inode >>> - do_writepages >>> - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA] >>> - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one >>> - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped > > So, my question was why this is the problem? kworker will loop writebacking this requeued inode. Thanks, > >>> >>>> seems the current condition is just requeuing the inode as dirty, but next >>>> flushing time will remove it from dirty list. Is this giving too much overheads? >>> >>> I prefer to let writeback thread call blk_flush_plug() after skipping >>> writepages() due to wb_sync_req[DATA/NODE] check condition, thoughts? >>> >>> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c >>> index 9f754aaef558..b6e1ed73f8f5 100644 >>> --- a/fs/f2fs/data.c >>> +++ b/fs/f2fs/data.c >>> @@ -3087,6 +3087,8 @@ static int f2fs_write_cache_pages(struct address_space *mapping, >>> /* give a priority to WB_SYNC threads */ >>> if (atomic_read(&sbi->wb_sync_req[DATA]) && >>> wbc->sync_mode == WB_SYNC_NONE) { >>> + if (current->plug) >>> + blk_flush_plug(current->plug, false); >>> done = 1; >>> break; >>> } >>> diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c >>> index 556fcd8457f3..dd9a817d8dab 100644 >>> --- a/fs/f2fs/node.c >>> +++ b/fs/f2fs/node.c >>> @@ -1946,6 +1946,8 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, >>> if (atomic_read(&sbi->wb_sync_req[NODE]) && >>> wbc->sync_mode == WB_SYNC_NONE) { >>> done = 1; >>> + if (current->plug) >>> + blk_flush_plug(current->plug, false); >>> break; >>> } >>> >>> >>> >>> Thanks, >>> >>>> >>>>> + get_dirty_pages(inode) : 0; >>>>> trace_f2fs_writepages(mapping->host, wbc, DATA); >>>>> return 0; >>>>> } >>>>> -- >>>>> 2.28.0 >>> >>> >>> _______________________________________________ >>> Linux-f2fs-devel mailing list >>> Linux-f2fs-devel@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel >>> _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel