Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
From: Chris Murphy <lists@colorremedies.com>
To: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>,
	Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Subject: Re: Reproducer for "compressed data + hole data corruption bug, 2018 edition" still works on 4.20.7
Date: Tue, 12 Feb 2019 15:53:53 -0700
Message-ID: <CAJCQCtQcU0CvVf-NGt3ZHZo773meYeW3UVyPS6iP+2C_Kd-jZQ@mail.gmail.com> (raw)
In-Reply-To: <20190212221107.GC23918@hungrycats.org>

On Tue, Feb 12, 2019 at 3:11 PM Zygo Blaxell
<ce3g8jdj@umail.furryterror.org> wrote:
>
> On Tue, Feb 12, 2019 at 02:48:38PM -0700, Chris Murphy wrote:
> > Is it possibly related to the zlib library being used on
> > Debian/Ubuntu? That you've got even one reproducer with the exact same
> > hash for the transient error case means it's not hardware or random
> > error; let alone two independent reproducers.
>
> The errors are not consistent between runs.  The above pattern is quite
> common, but it is not the only possible output.  Add in other processes
> reading the 'am' file at the same time and it gets very random.
>
> The bad data tends to have entire extents missing, replaced with zeros.
> That leads to a small number of possible outputs (the choices seem to be
> only to have the data or have the zeros).  It does seem to be a lot more
> consistent in recent (post 4.14.80) kernels, which may be interesting.
>
> Here is an example of a diff between two copies of the 'am' file copied
> while the repro script was running, filtered through hd:
>
>         # diff -u /tmp/f1 /tmp/f2
>         --- /tmp/f1     2019-02-12 17:05:14.861844871 -0500
>         +++ /tmp/f2     2019-02-12 17:05:16.883868402 -0500
>         @@ -56,10 +56,6 @@
>          *
>          00020000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>          *
>         -00021000  11 11 11 11 11 11 11 11  11 11 11 11 11 11 11 11  |................|
>         -*
>         -00022000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         -*
>          00023000  12 12 12 12 12 12 12 12  12 12 12 12 12 12 12 12  |................|
>          *
>          00024000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         @@ -268,10 +264,6 @@
>          *
>          000a0000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>          *
>         -000a1000  11 11 11 11 11 11 11 11  11 11 11 11 11 11 11 11  |................|
>         -*
>         -000a2000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         -*
>          000a3000  12 12 12 12 12 12 12 12  12 12 12 12 12 12 12 12  |................|
>          *
>          000a4000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         @@ -688,10 +680,6 @@
>          *
>          001a0000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>          *
>         -001a1000  11 11 11 11 11 11 11 11  11 11 11 11 11 11 11 11  |................|
>         -*
>         -001a2000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         -*
>          001a3000  12 12 12 12 12 12 12 12  12 12 12 12 12 12 12 12  |................|
>          *
>          001a4000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         @@ -1524,10 +1512,6 @@
>          *
>          003a0000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>          *
>         -003a1000  11 11 11 11 11 11 11 11  11 11 11 11 11 11 11 11  |................|
>         -*
>         -003a2000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         -*
>          003a3000  12 12 12 12 12 12 12 12  12 12 12 12 12 12 12 12  |................|
>          *
>          003a4000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         @@ -3192,10 +3176,6 @@
>          *
>          007a0000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>          *
>         -007a1000  11 11 11 11 11 11 11 11  11 11 11 11 11 11 11 11  |................|
>         -*
>         -007a2000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         -*
>          007a3000  12 12 12 12 12 12 12 12  12 12 12 12 12 12 12 12  |................|
>          *
>          007a4000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         @@ -5016,10 +4996,6 @@
>          *
>          00c00000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>          *
>         -00c01000  11 11 11 11 11 11 11 11  11 11 11 11 11 11 11 11  |................|
>         -*
>         -00c02000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
>         -*
>         [etc...you get the idea]

And yet the file is delivered to user space, despite the changes, as
if it's immune to checksum computation or matching. The data is
clearly difference so how is it bypassing checksumming? Data csums are
based on original uncompressed data, correct? So any holes are zeros,
there are still csums for those holes?

>
> I'm not sure how the zlib library is involved--sha1sum doesn't use one.
>
> > And then what happens if you do the exact same test but change to zstd
> > or lzo? No error? Strictly zlib?
>
> Same errors on all three btrfs compression algorithms (as mentioned in
> the original post from August 2018).

Obviously there is a pattern. It's not random. I just don't know what
it looks like. I use compression, for years now, mostly zstd lately
and a mix of lzo and zlib before that, but never any errors or
corruptions. But I also never use holes, no punched holes, and rarely
use fallocated files which I guess isn't quite the same thing as hole
punching.

So the bug you're reproducing is for sure 100% not on the media
itself, it's somehow transiently being interpreted differently roughly
1 in 10 reads, but with a pattern. What about scrub? Do you get errors
every 1 in 10 scrubs? Or how does it manifest? No scrub errors?

I know very little about what parts of the kernel a file system
depends on outside of its own code (e.g. page cache) but I wonder if
there's something outside of Btrfs that's the source but it never gets
triggered because no other file systems use compression. Huh - what
file system uses compression *and* hole punching? squashfs? Is sparse
file support different than hole punching?


-- 
Chris Murphy

  reply index

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-08-23  3:11 Reproducer for "compressed data + hole data corruption bug, 2018 editiion" Zygo Blaxell
2018-08-23  5:10 ` Qu Wenruo
2018-08-23 16:44   ` Zygo Blaxell
2018-08-23 23:50     ` Qu Wenruo
2019-02-12  3:09 ` Reproducer for "compressed data + hole data corruption bug, 2018 edition" still works on 4.20.7 Zygo Blaxell
2019-02-12 15:33   ` Christoph Anton Mitterer
2019-02-12 15:35   ` Filipe Manana
2019-02-12 17:01     ` Zygo Blaxell
2019-02-12 17:56       ` Filipe Manana
2019-02-12 18:13         ` Zygo Blaxell
2019-02-13  7:24           ` Qu Wenruo
2019-02-13 17:36           ` Filipe Manana
2019-02-13 18:14             ` Filipe Manana
2019-02-14  1:22               ` Filipe Manana
2019-02-14  5:00                 ` Zygo Blaxell
2019-02-14 12:21                 ` Christoph Anton Mitterer
2019-02-15  5:40                   ` Zygo Blaxell
2019-03-04 15:34                     ` Christoph Anton Mitterer
2019-03-07 20:07                       ` Zygo Blaxell
2019-03-08 10:37                         ` Filipe Manana
2019-03-14 18:58                           ` Christoph Anton Mitterer
2019-03-14 20:22                           ` Christoph Anton Mitterer
2019-03-14 22:39                             ` Filipe Manana
2019-03-08 12:20                         ` Austin S. Hemmelgarn
2019-03-14 18:58                           ` Christoph Anton Mitterer
2019-03-14 18:58                         ` Christoph Anton Mitterer
2019-03-15  5:28                           ` Zygo Blaxell
2019-03-16 22:11                             ` Christoph Anton Mitterer
2019-03-17  2:54                               ` Zygo Blaxell
2019-02-15 12:02                   ` Filipe Manana
2019-03-04 15:46                     ` Christoph Anton Mitterer
2019-02-12 18:58       ` Andrei Borzenkov
2019-02-12 21:48         ` Chris Murphy
2019-02-12 22:11           ` Zygo Blaxell
2019-02-12 22:53             ` Chris Murphy [this message]
2019-02-13  2:46               ` Zygo Blaxell
2019-02-13  7:47   ` Roman Mamedov
2019-02-13  8:04     ` Qu Wenruo

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAJCQCtQcU0CvVf-NGt3ZHZo773meYeW3UVyPS6iP+2C_Kd-jZQ@mail.gmail.com \
    --to=lists@colorremedies.com \
    --cc=ce3g8jdj@umail.furryterror.org \
    --cc=linux-btrfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org linux-btrfs@archiver.kernel.org
	public-inbox-index linux-btrfs


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/ public-inbox