From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AD0DC433F5 for ; Thu, 7 Apr 2022 00:19:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9DA446B0071; Wed, 6 Apr 2022 20:19:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 989296B0073; Wed, 6 Apr 2022 20:19:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 803526B0074; Wed, 6 Apr 2022 20:19:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0058.hostedemail.com [216.40.44.58]) by kanga.kvack.org (Postfix) with ESMTP id 6AE0A6B0071 for ; Wed, 6 Apr 2022 20:19:23 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1EC828248D52 for ; Thu, 7 Apr 2022 00:19:13 +0000 (UTC) X-FDA: 79328173386.25.4C16285 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf14.hostedemail.com (Postfix) with ESMTP id 9B8A0100005 for ; Thu, 7 Apr 2022 00:19:12 +0000 (UTC) Received: by mail-qt1-f176.google.com with SMTP id o15so6884745qtv.8 for ; Wed, 06 Apr 2022 17:19:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=3mWEnyekxUAKBntA2OPgIAq5gAuaD6bF8X9P4fQWVHY=; b=mCm4aZc8V0CZVgKvFBLDEaytsHuexVkWhnUTk3QdlrLI6zwX1CjwjYlp6z815SV608 c6jVJgFYnqSXX2yBAAtTUhjGggSusRwcGvHP88sPtTIk5Mk6wFLWsga9TqCM8JPbINpE SPYRrNM2ej7K5ICemsSQnq9IKwioRPeU4dUlrw70zUCfOKFu4kX2HJ5gUgMqZSzUAc2A c/MbPU2BsjIrB4GmXJiD9KxOmOwPPHTHSKRipjGdFOgZmWuLr2xBXNI7QB5uAE2mg3SE rUUFGJnBjJdQ+e3YFRXW4dr8KhDrztU5t4AzsHCDthBxcVxH0RQoHukOG9t+02xrBiCE 59JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=3mWEnyekxUAKBntA2OPgIAq5gAuaD6bF8X9P4fQWVHY=; b=nXxqD5rwUX/D7CFbnxQe3r+KjzgAPvXUoi7Kr+IhwblMn7JX2o3CFDIIoaSyuy5aKb Xom7U2/E7n6G7vVHUKEEipBG2SgYPlW4By7ASejCp5tyO7AJqa2x7TjRzX3cohzN8UL4 XN+19f2Go7pnILQz+b6JIzQxcXivMWBVddW9NPAG+vR5FjbJ3hEG3jAvf/dWxVkQdbL5 I/9DXcFsTv6BLVADjzFQKeCJvxgu1MiDSdjU37NgjjzWvB0x9v0VqoZA8W1aAwBZDwzR JJSjIO3d69b1fiphrRQWGoknbxDRmphsTRqr1xVN1kbGJ0i9SSSUGAKmV8O8hHtL9JLI 2/NQ== X-Gm-Message-State: AOAM530BiOkkZryt/80gQmnFtsHfmpTWvkKU4qXmobHtwiiLmfAYewxd cSgKLh3ZssWVcc2zwRF/ObPEPw== X-Google-Smtp-Source: ABdhPJx8VVHyff4a+1iNZKsR51aHhj1HJx23vesqDtCuIEm5GjNaDF1HUownJz1K8ZiNQWfhjJS7Mw== X-Received: by 2002:a05:622a:38c:b0:2e2:32de:4d64 with SMTP id j12-20020a05622a038c00b002e232de4d64mr9824145qtx.402.1649290751642; Wed, 06 Apr 2022 17:19:11 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id f17-20020ac87f11000000b002e1e831366asm14290048qtk.77.2022.04.06.17.19.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Apr 2022 17:19:09 -0700 (PDT) Date: Wed, 6 Apr 2022 17:18:55 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Chuck Lever III cc: Hugh Dickins , Andrew Morton , Patrice CHOTARD , Mikulas Patocka , Lukas Czerner , Christoph Hellwig , "Darrick J. Wong" , Linux-MM , Linux NFS Mailing List , linux-fsdevel@vger.kernel.org Subject: Re: Regression in xfstests on tmpfs-backed NFS exports In-Reply-To: <673D708E-2DFA-4812-BB63-6A437E0C72EE@oracle.com> Message-ID: <11f319-c9a-4648-bfbb-dc5a83c774@google.com> References: <673D708E-2DFA-4812-BB63-6A437E0C72EE@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Stat-Signature: bragmz6j5sxakwseys9hbc9qr1bu1zqa Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mCm4aZc8; spf=pass (imf14.hostedemail.com: domain of hughd@google.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9B8A0100005 X-HE-Tag: 1649290752-738420 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 6 Apr 2022, Chuck Lever III wrote: > Good day, Hugh- Huh! If you were really wishing me a good day, would you tell me this ;-? > > I noticed that several fsx-related tests in the xfstests suite are > failing after updating my NFS server to v5.18-rc1. I normally test > against xfs, ext4, btrfs, and tmpfs exports. tmpfs is the only export > that sees these new failures: > > generic/075 2s ... [failed, exit status 1]- output mismatch (see /home/cel/src/xfstests/results//generic/075.out.bad) > --- tests/generic/075.out 2014-02-13 15:40:45.000000000 -0500 > +++ /home/cel/src/xfstests/results//generic/075.out.bad 2022-04-05 16:39:59.145991520 -0400 > @@ -4,15 +4,5 @@ > ----------------------------------------------- > fsx.0 : -d -N numops -S 0 > ----------------------------------------------- > - > ------------------------------------------------ > -fsx.1 : -d -N numops -S 0 -x > ------------------------------------------------ > ... > (Run 'diff -u /home/cel/src/xfstests/tests/generic/075.out /home/cel/src/xfstests/results//generic/075.out.bad' to see the entire diff) > > generic/091 9s ... [failed, exit status 1]- output mismatch (see /home/cel/src/xfstests/results//generic/091.out.bad) > --- tests/generic/091.out 2014-02-13 15:40:45.000000000 -0500 > +++ /home/cel/src/xfstests/results//generic/091.out.bad 2022-04-05 16:41:24.329063277 -0400 > @@ -1,7 +1,75 @@ > QA output created by 091 > fsx -N 10000 -l 500000 -r PSIZE -t BSIZE -w BSIZE -Z -R -W > -fsx -N 10000 -o 8192 -l 500000 -r PSIZE -t BSIZE -w BSIZE -Z -R -W > -fsx -N 10000 -o 32768 -l 500000 -r PSIZE -t BSIZE -w BSIZE -Z -R -W > -fsx -N 10000 -o 8192 -l 500000 -r PSIZE -t BSIZE -w BSIZE -Z -R -W > -fsx -N 10000 -o 32768 -l 500000 -r PSIZE -t BSIZE -w BSIZE -Z -R -W > -fsx -N 10000 -o 128000 -l 500000 -r PSIZE -t BSIZE -w BSIZE -Z -W > ... > (Run 'diff -u /home/cel/src/xfstests/tests/generic/091.out /home/cel/src/xfstests/results//generic/091.out.bad' to see the entire diff) > > generic/112 2s ... [failed, exit status 1]- output mismatch (see /home/cel/src/xfstests/results//generic/112.out.bad) > --- tests/generic/112.out 2014-02-13 15:40:45.000000000 -0500 > +++ /home/cel/src/xfstests/results//generic/112.out.bad 2022-04-05 16:41:38.511075170 -0400 > @@ -4,15 +4,4 @@ > ----------------------------------------------- > fsx.0 : -A -d -N numops -S 0 > ----------------------------------------------- > - > ------------------------------------------------ > -fsx.1 : -A -d -N numops -S 0 -x > ------------------------------------------------ > ... > (Run 'diff -u /home/cel/src/xfstests/tests/generic/112.out /home/cel/src/xfstests/results//generic/112.out.bad' to see the entire diff) > > generic/127 49s ... - output mismatch (see /home/cel/src/xfstests/results//generic/127.out.bad) > --- tests/generic/127.out 2016-08-28 12:16:20.000000000 -0400 > +++ /home/cel/src/xfstests/results//generic/127.out.bad 2022-04-05 16:42:07.655099652 -0400 > @@ -4,10 +4,198 @@ > === FSX Light Mode, Memory Mapping === > All 100000 operations completed A-OK! > === FSX Standard Mode, No Memory Mapping === > -All 100000 operations completed A-OK! > +ltp/fsx -q -l 262144 -o 65536 -S 191110531 -N 100000 -R -W fsx_std_nommap > +READ BAD DATA: offset = 0x9cb7, size = 0xfae3, fname = /tmp/mnt/manet.ib-2323703/fsx_std_nommap > +OFFSET GOOD BAD RANGE > ... > (Run 'diff -u /home/cel/src/xfstests/tests/generic/127.out /home/cel/src/xfstests/results//generic/127.out.bad' to see the entire diff) > > I bisected the problem to: > > 56a8c8eb1eaf ("tmpfs: do not allocate pages on read") > > generic/075 fails almost immediately without any NFS-level errors. > Likely this is data corruption rather than an overt I/O error. That's sad. Thanks for bisecting and reporting. Sorry for the nuisance. I suspect this patch is heading for a revert, because I shall not have time to debug and investigate. Cc'ing fsdevel and a few people who have an interest in it, to warn of that likely upcoming revert. But if it's okay with everyone, please may we leave it in for -rc2? Given that having it in -rc1 already smoked out another issue (problem of SetPageUptodate(ZERO_PAGE(0)) without CONFIG_MMU), I think keeping it in a little longer might smoke out even more. The xfstests info above doesn't actually tell very much, beyond that generic/075 generic/091 generic/112 generic/127, each a test with fsx, all fall at their first hurdle. If you have time, please rerun and tar up the results/generic directory (maybe filter just those failing) and send as attachment. But don't go to any trouble, it's unlikely that I shall even untar it - it would be mainly to go on record if anyone has time to look into it later. And, frankly, it's unlikely to tell us anything more enlightening, than that the data seen was not as expected: which we do already know. I've had no problem with xfstests generic 075,091,112,127 testing tmpfs here, not before and not in the month or two I've had that patch in: so it's something in the way that NFS exercises tmpfs that reveals it. If I had time to duplicate your procedure, I'd be asking for detailed instructions: but no, I won't have a chance. But I can sit here and try to guess. I notice fs/nfsd checks file->f_op->splice_read, and employs fallback if not available: if you have time, please try rerunning those xfstests on an -rc1 kernel, but with mm/shmem.c's .splice_read line commented out. My guess is that will then pass the tests, and we shall know more. What could be going wrong there? I've thought of two possibilities. A minor, hopefully easily fixed, issue would be if fs/nfsd has trouble with seeing the same page twice in a row: since tmpfs is now using the ZERO_PAGE(0) for all pages of a hole, and I think I caught sight of code which looks to see if the latest page is the same as the one before. It's easy to imagine that might go wrong. A more difficult issue would be, if fsx is racing writes and reads, in a way that it can guarantee the correct result, but that correct result is no longer delivered: because the writes go into freshly allocated tmpfs cache pages, while reads are still delivering stale ZERO_PAGEs from the pipe. I'm hazy on the guarantees there. But unless someone has time to help out, we're heading for a revert. Thanks, Hugh