From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A615C678D4 for ; Mon, 6 Mar 2023 16:56:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229906AbjCFQ4V (ORCPT ); Mon, 6 Mar 2023 11:56:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229556AbjCFQ4T (ORCPT ); Mon, 6 Mar 2023 11:56:19 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B6A94393E for ; Mon, 6 Mar 2023 08:55:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678121651; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=NXGyu/ZMecSXYvQInMYDbVFPC7KRhmoWOHqQ2qxDhxM=; b=CuDxbCZrEkRT9/qei48kYEsOqtthg8IRRktQxjV1d8Nh86KlFd93LT8YIu3C6hYq2z10rt 27n7mIrMFz26MokWmc8+95bogiLj0wevxnnEEBsUHhvbnWDQn5GWBnZK5IQyw+56xSH+Zx 0bB8CDxrAsUqfOEOwclQsSrcsC/vkDc= Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-449-bE6LIWKpMdqoopSxhTlNOQ-1; Mon, 06 Mar 2023 11:54:09 -0500 X-MC-Unique: bE6LIWKpMdqoopSxhTlNOQ-1 Received: by mail-pf1-f198.google.com with SMTP id b8-20020aa78708000000b005eaa50faa35so5846753pfo.20 for ; Mon, 06 Mar 2023 08:54:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678121648; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=NXGyu/ZMecSXYvQInMYDbVFPC7KRhmoWOHqQ2qxDhxM=; b=ogMVuxzl/wrHGcZuKKMkhyusMohUP9b7hDZXIxOExEuXz+Pzr/WWChFoVnTo5NJN4W AYBDf8oXdoMOliIZIXFoMmOw3Th5mHdQxSvBE4jLL6tBq8PvmC3rq0VU8OTxzuijx6wT q7eLH3HMI53nUG4oXVa22BSxtoHQK6YCVoo9kTPyijxQ3fWLJKrCzkDrIBbQ6BTlnOjb /VOXt1nOkRRcm05g7856fRocQz70pQCIg57SBdn1Jx6l0j17BBVKOPcA11G3XR1HzEiQ bZNS22lfPD/ozJJa9Rv7E9lFldaKXfLwOz+O/ShohinAMzz1WWK5GTjsGpjkg4pMnjeh cMvA== X-Gm-Message-State: AO0yUKVH9Fi/svE3zlprjRv63gfrsRNQlA3qCJcEpM3MIpsKu4Ji6Xtb azTSlpUaMtg5hnmNiNsyJ48Dmcr/cqj+R5+eDtS58mDAbO4lSt3pOiCJ+RrDAn9udZ6PAsILgUF e6I1/fi4e8UBq/ARvjA== X-Received: by 2002:a17:90b:164a:b0:234:a88e:d67e with SMTP id il10-20020a17090b164a00b00234a88ed67emr11935999pjb.34.1678121648450; Mon, 06 Mar 2023 08:54:08 -0800 (PST) X-Google-Smtp-Source: AK7set+S/ylxWdjkNcJY8RDSITSwSDWSXu75cGdp3jDQ1pXRbEM0LtBRyYnjTHLUSjmF6rBDbyHF1A== X-Received: by 2002:a17:90b:164a:b0:234:a88e:d67e with SMTP id il10-20020a17090b164a00b00234a88ed67emr11935980pjb.34.1678121647989; Mon, 06 Mar 2023 08:54:07 -0800 (PST) Received: from zlang-mailbox ([43.228.180.230]) by smtp.gmail.com with ESMTPSA id k11-20020a17090a590b00b0023747b0445fsm8077316pji.14.2023.03.06.08.54.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 08:54:06 -0800 (PST) Date: Tue, 7 Mar 2023 00:54:02 +0800 From: Zorro Lang To: "Darrick J. Wong" Cc: linux-xfs@vger.kernel.org, fstests@vger.kernel.org Subject: Re: [PATCH 1/1] xfs: test upgrading old features Message-ID: <20230306165402.nrkm5jkowiorucmn@zlang-mailbox> References: <167243882949.736459.9627363155663418213.stgit@magnolia> <167243882961.736459.4302174338740527578.stgit@magnolia> <20230306155611.6w6mn3k7owsc2jz7@zlang-mailbox> <20230306164155.GA1637786@frogsfrogsfrogs> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230306164155.GA1637786@frogsfrogsfrogs> Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org On Mon, Mar 06, 2023 at 08:41:55AM -0800, Darrick J. Wong wrote: > On Mon, Mar 06, 2023 at 11:56:11PM +0800, Zorro Lang wrote: > > On Fri, Dec 30, 2022 at 02:20:29PM -0800, Darrick J. Wong wrote: > > > From: Darrick J. Wong > > > > > > Test the ability to add older v5 features. > > > > > > Signed-off-by: Darrick J. Wong > > > --- > > > tests/xfs/769 | 248 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > tests/xfs/769.out | 2 > > > 2 files changed, 250 insertions(+) > > > create mode 100755 tests/xfs/769 > > > create mode 100644 tests/xfs/769.out > > > > > > > > > diff --git a/tests/xfs/769 b/tests/xfs/769 > > > new file mode 100755 > > > index 0000000000..7613048f52 > > > --- /dev/null > > > +++ b/tests/xfs/769 > > > @@ -0,0 +1,248 @@ > > > +#! /bin/bash > > > +# SPDX-License-Identifier: GPL-2.0-or-later > > > +# Copyright (c) 2022 Oracle. All Rights Reserved. > > > +# > > > +# FS QA Test No. 769 > > > +# > > > +# Test upgrading filesystems with new features. > > > +# > > > +. ./common/preamble > > > +_begin_fstest auto mkfs repair > > > + > > > +# Import common functions. > > > +. ./common/filter > > > +. ./common/populate > > > + > > > +# real QA test starts here > > > +_supported_fs xfs > > > + > > > +test -w /dev/ttyprintk || _notrun "test requires writable /dev/ttyprintk" > > > > Hi Darrick, > > > > I'm not sure why /dev/ttyprintk is necessary. I think sometimes we might not > > have this driver, but has /dev/kmsg. Can /dev/kmsg be a replacement of > > ttyprintk ? > > The kernel logging here is for debugging purposes -- if the upgrade > corrupts the fs and the kernel splats, whoever triages the failure can > have at least some clue as to where things went wrong. > > I think the _notrun line should go away though. If only for debugging, better to let this case keep running without ttyprintk. Or it'll _notrun on many systems without ttyprintk driver. There's a log_dmesg() helper in common/rc, the /dev/kmsg might be more often. As you've called *_require_check_dmesg*, it helps to check kmsg file. So how about use it? Or making a helper to write messages to ttyprintk, then fallback to kmsg if ttyprintk is missing ? Thanks, Zorro > > --D > > > Thanks, > > Zorro > > > > > > > +_require_check_dmesg > > > +_require_scratch_nocheck > > > +_require_scratch_xfs_crc > > > + > > > +# Does repair know how to add a particular feature to a filesystem? > > > +check_repair_upgrade() > > > +{ > > > + $XFS_REPAIR_PROG -c "$1=narf" 2>&1 | \ > > > + grep -q 'unknown option' && return 1 > > > + return 0 > > > +} > > > + > > > +# Are we configured for realtime? > > > +rt_configured() > > > +{ > > > + test "$USE_EXTERNAL" = "yes" && test -n "$SCRATCH_RTDEV" > > > +} > > > + > > > +# Compute the MKFS_OPTIONS string for a particular feature upgrade test > > > +compute_mkfs_options() > > > +{ > > > + local m_opts="" > > > + local caller_options="$MKFS_OPTIONS" > > > + > > > + for feat in "${FEATURES[@]}"; do > > > + local feat_state="${FEATURE_STATE["${feat}"]}" > > > + > > > + if echo "$caller_options" | grep -E -w -q "${feat}=[0-9]*"; then > > > + # Change the caller's options > > > + caller_options="$(echo "$caller_options" | \ > > > + sed -e "s/\([^[:alnum:]]\)${feat}=[0-9]*/\1${feat}=${feat_state}/g")" > > > + else > > > + # Add it to our list of new mkfs flags > > > + m_opts="${feat}=${feat_state},${m_opts}" > > > + fi > > > + done > > > + > > > + test -n "$m_opts" && m_opts=" -m $m_opts" > > > + > > > + echo "$caller_options$m_opts" > > > +} > > > + > > > +# Log the start of an upgrade. > > > +function upgrade_start_message() > > > +{ > > > + local feat="$1" > > > + > > > + echo "Add $feat to filesystem" > > > +} > > > + > > > +# Find dmesg log messages since we started a particular upgrade test > > > +function dmesg_since_feature_upgrade_start() > > > +{ > > > + local feat_logmsg="$(upgrade_start_message "$1")" > > > + > > > + # search the dmesg log of last run of $seqnum for possible failures > > > + # use sed \cregexpc address type, since $seqnum contains "/" > > > + dmesg | \ > > > + tac | \ > > > + sed -ne "0,\#run fstests $seqnum at $date_time#p" | \ > > > + sed -ne "0,\#${feat_logmsg}#p" | \ > > > + tac > > > +} > > > + > > > +# Did the mount fail because this feature is not supported? > > > +function feature_unsupported() > > > +{ > > > + local feat="$1" > > > + > > > + dmesg_since_feature_upgrade_start "$feat" | \ > > > + grep -q 'has unknown.*features' > > > +} > > > + > > > +# Exercise the scratch fs > > > +function scratch_fsstress() > > > +{ > > > + echo moo > $SCRATCH_MNT/sample.txt > > > + $FSSTRESS_PROG -n $((TIME_FACTOR * 1000)) -p $((LOAD_FACTOR * 4)) \ > > > + -d $SCRATCH_MNT/data >> $seqres.full > > > +} > > > + > > > +# Exercise the filesystem a little bit and emit a manifest. > > > +function pre_exercise() > > > +{ > > > + local feat="$1" > > > + > > > + _try_scratch_mount &> $tmp.mount > > > + res=$? > > > + # If the kernel doesn't support the filesystem even after a > > > + # fresh format, skip the rest of the upgrade test quietly. > > > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > > > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > > > + return 1 > > > + fi > > > + if [ $res -ne 0 ]; then > > > + cat $tmp.mount > > > + echo "mount failed with $res before upgrading to $feat" | \ > > > + tee -a $seqres.full > > > + return 1 > > > + fi > > > + > > > + scratch_fsstress > > > + find $SCRATCH_MNT -type f -print0 | xargs -r -0 md5sum > $tmp.manifest > > > + _scratch_unmount > > > + return 0 > > > +} > > > + > > > +# Check the manifest and exercise the filesystem more > > > +function post_exercise() > > > +{ > > > + local feat="$1" > > > + > > > + _try_scratch_mount &> $tmp.mount > > > + res=$? > > > + # If the kernel doesn't support the filesystem even after a > > > + # fresh format, skip the rest of the upgrade test quietly. > > > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > > > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > > > + return 1 > > > + fi > > > + if [ $res -ne 0 ]; then > > > + cat $tmp.mount > > > + echo "mount failed with $res after upgrading to $feat" | \ > > > + tee -a $seqres.full > > > + return 1 > > > + fi > > > + > > > + md5sum --quiet -c $tmp.manifest || \ > > > + echo "fs contents ^^^ changed after adding $feat" > > > + > > > + iam="check" _check_scratch_fs || \ > > > + echo "scratch fs check failed after adding $feat" > > > + > > > + # Try to mount the fs in case the check unmounted it > > > + _try_scratch_mount &>> $seqres.full > > > + > > > + scratch_fsstress > > > + > > > + iam="check" _check_scratch_fs || \ > > > + echo "scratch fs check failed after exercising $feat" > > > + > > > + # Try to unmount the fs in case the check didn't > > > + _scratch_unmount &>> $seqres.full > > > + return 0 > > > +} > > > + > > > +# Create a list of fs features in the order that support for them was added > > > +# to the kernel driver. For each feature upgrade test, we enable all the > > > +# features that came before it and none of the ones after, which means we're > > > +# testing incremental migrations. We start each run with a clean fs so that > > > +# errors and unsatisfied requirements (log size, root ino position, etc) in one > > > +# upgrade don't spread failure to the rest of the tests. > > > +FEATURES=() > > > +if rt_configured; then > > > + check_repair_upgrade finobt && FEATURES+=("finobt") > > > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > > > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > > > +else > > > + check_repair_upgrade finobt && FEATURES+=("finobt") > > > + check_repair_upgrade rmapbt && FEATURES+=("rmapbt") > > > + check_repair_upgrade reflink && FEATURES+=("reflink") > > > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > > > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > > > +fi > > > + > > > +test "${#FEATURES[@]}" -eq 0 && \ > > > + _notrun "xfs_repair does not know how to add V5 features" > > > + > > > +declare -A FEATURE_STATE > > > +for f in "${FEATURES[@]}"; do > > > + FEATURE_STATE["$f"]=0 > > > +done > > > + > > > +for feat in "${FEATURES[@]}"; do > > > + echo "-----------------------" >> $seqres.full > > > + > > > + upgrade_start_message "$feat" | tee -a $seqres.full /dev/ttyprintk > /dev/null > > > + > > > + opts="$(compute_mkfs_options)" > > > + echo "mkfs.xfs $opts" >> $seqres.full > > > + > > > + # Format filesystem > > > + MKFS_OPTIONS="$opts" _scratch_mkfs &>> $seqres.full > > > + res=$? > > > + outcome="mkfs returns $res for $feat upgrade test" > > > + echo "$outcome" >> $seqres.full > > > + if [ $res -ne 0 ]; then > > > + echo "$outcome" > > > + continue > > > + fi > > > + > > > + # Create some files to make things interesting. > > > + pre_exercise "$feat" || break > > > + > > > + # Upgrade the fs > > > + _scratch_xfs_repair -c "${feat}=1" &> $tmp.upgrade > > > + res=$? > > > + cat $tmp.upgrade >> $seqres.full > > > + grep -q "^Adding" $tmp.upgrade || \ > > > + echo "xfs_repair ignored command to add $feat" > > > + > > > + outcome="xfs_repair returns $res while adding $feat" > > > + echo "$outcome" >> $seqres.full > > > + if [ $res -ne 0 ]; then > > > + # Couldn't upgrade filesystem, move on to the next feature. > > > + FEATURE_STATE["$feat"]=1 > > > + continue > > > + fi > > > + > > > + # Make sure repair runs cleanly afterwards > > > + _scratch_xfs_repair -n &>> $seqres.full > > > + res=$? > > > + outcome="xfs_repair -n returns $res after adding $feat" > > > + echo "$outcome" >> $seqres.full > > > + if [ $res -ne 0 ]; then > > > + echo "$outcome" > > > + fi > > > + > > > + # Make sure we can still exercise the filesystem. > > > + post_exercise "$feat" || break > > > + > > > + # Update feature state for next run > > > + FEATURE_STATE["$feat"]=1 > > > +done > > > + > > > +# success, all done > > > +echo Silence is golden. > > > +status=0 > > > +exit > > > diff --git a/tests/xfs/769.out b/tests/xfs/769.out > > > new file mode 100644 > > > index 0000000000..332432db97 > > > --- /dev/null > > > +++ b/tests/xfs/769.out > > > @@ -0,0 +1,2 @@ > > > +QA output created by 769 > > > +Silence is golden. > > > > > >