From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from cn.fujitsu.com ([59.151.112.132]:4750 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1751505AbcGSFGo (ORCPT ); Tue, 19 Jul 2016 01:06:44 -0400 Subject: Re: [PATCH] fstests: btrfs: Test send on heavily deduped file References: <20160719024402.19324-1-quwenruo@cn.fujitsu.com> From: Qu Wenruo Message-ID: <2245ccaf-643f-ef25-2ef7-84ffbf97e927@cn.fujitsu.com> Date: Tue, 19 Jul 2016 13:06:35 +0800 MIME-Version: 1.0 In-Reply-To: <20160719024402.19324-1-quwenruo@cn.fujitsu.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Sender: fstests-owner@vger.kernel.org To: linux-btrfs@vger.kernel.org, fstests@vger.kernel.org, Filipe Manana List-ID: Add Filipe to the reception, as "To:" doesn't add him automatically. Thanks, Qu At 07/19/2016 10:44 AM, Qu Wenruo wrote: > For fully deduped file, whose file extents are all pointing to the same > extent, btrfs backref walk can be very time consuming, long enough to > trigger softlock. > > Unfortunately, btrfs send is one of the caller of such backref walk > under an O(n) loop, making the total time complexity to O(n^3) or more. > > And even worse, btrfs send will allocate memory in such loop, to trigger > OOM on system with small memory(<4G). > > This test case will check if btrfs send will cause these problems. > > Reporeted-by: Tsutomu Itoh > To: Filipe Manana > Signed-off-by: Qu Wenruo > --- > To Filipe: > For the soft lockup, I will try my best to figure out some method to > avoid such lockup (but it will still be very time consuming though). > > But for the OOM problem, would you mind disabling clone/reflink > detection in btrfs send? > > In fact we should really avoid doing full backref walk inside an O(n) > loop (just like previous fiemap ioctl test case), and avoid any full > backref walk if possible. > So I'm afraid that's the only solution yet. > > Thanks, > Qu > --- > tests/btrfs/127 | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > tests/btrfs/127.out | 3 ++ > tests/btrfs/group | 1 + > 3 files changed, 93 insertions(+) > create mode 100755 tests/btrfs/127 > create mode 100644 tests/btrfs/127.out > > diff --git a/tests/btrfs/127 b/tests/btrfs/127 > new file mode 100755 > index 0000000..a31a653 > --- /dev/null > +++ b/tests/btrfs/127 > @@ -0,0 +1,89 @@ > +#! /bin/bash > +# FS QA Test 127 > +# > +# Check if btrfs send can handle large deduped file, whose file extents > +# are all pointing to one extent. > +# Such file structure will cause quite large pressure to any operation which > +# iterates all backref of one extent. > +# And unfortunately, btrfs send is one of these operations, and will cause > +# softlock or OOM on systems with small memory(<4G). > +# > +#----------------------------------------------------------------------- > +# Copyright (c) 2016 Fujitsu. All Rights Reserved. > +# > +# This program is free software; you can redistribute it and/or > +# modify it under the terms of the GNU General Public License as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it would be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write the Free Software Foundation, > +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > +#----------------------------------------------------------------------- > +# > + > +seq=`basename $0` > +seqres=$RESULT_DIR/$seq > +echo "QA output created by $seq" > + > +here=`pwd` > +tmp=/tmp/$$ > +status=1 # failure is the default! > +trap "_cleanup; exit \$status" 0 1 2 3 15 > + > +_cleanup() > +{ > + cd / > + rm -f $tmp.* > +} > + > +# get standard environment, filters and checks > +. ./common/rc > +. ./common/filter > +. ./common/reflink > + > +# remove previous $seqres.full before test > +rm -f $seqres.full > + > +# real QA test starts here > + > +# Modify as appropriate. > +_supported_fs btrfs > +_supported_os Linux > +_require_scratch > +_require_scratch_reflink > + > +_scratch_mkfs > /dev/null 2>&1 > +_scratch_mount > + > +nr_extents=$((4096 * $LOAD_FACTOR)) > + > +# Use 128K blocksize, the default value of both deduperemove or > +# inband dedupe > +blocksize=$((128 * 1024)) > +file=$SCRATCH_MNT/foobar > + > +# create the initial file, whose file extents are all point to one extent > +_pwrite_byte 0xcdcdcdcd 0 $blocksize $file | _filter_xfs_io > + > +for i in $(seq 1 $(($nr_extents - 1))); do > + _reflink_range $file 0 $file $(($i * $blocksize)) $blocksize \ > + > /dev/null 2>&1 > +done > + > +# create a RO snapshot, so we can send out the snapshot > +_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/ro_snap > + > +# send out the subvolume, and it will either: > +# 1) OOM since memory is allocated inside a O(n^3) loop > +# 2) Softlock since time consuming backref walk is called without scheduling. > +# the send destination is not important, just send will cause the problem > +_run_btrfs_util_prog send $SCRATCH_MNT/ro_snap > /dev/null 2>&1 > + > +# success, all done > +status=0 > +exit > diff --git a/tests/btrfs/127.out b/tests/btrfs/127.out > new file mode 100644 > index 0000000..8b08bf8 > --- /dev/null > +++ b/tests/btrfs/127.out > @@ -0,0 +1,3 @@ > +QA output created by 127 > +wrote 131072/131072 bytes at offset 0 > +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) > diff --git a/tests/btrfs/group b/tests/btrfs/group > index a21a80a..d9174b5 100644 > --- a/tests/btrfs/group > +++ b/tests/btrfs/group > @@ -129,3 +129,4 @@ > 124 auto replace > 125 auto replace > 126 auto quick qgroup > +127 auto clone send >