Re: [PATCH v12 09/17] btrfs: add BTRFS_IOC_ENCODED_READ

From: Omar Sandoval <osandov@osandov.com>
To: dsterba@suse.cz, linux-btrfs@vger.kernel.org, kernel-team@fb.com
Subject: Re: [PATCH v12 09/17] btrfs: add BTRFS_IOC_ENCODED_READ
Date: Mon, 24 Jan 2022 14:33:04 -0800	[thread overview]
Message-ID: <Ye8ovb+nABMXgksN@relinquished.localdomain> (raw)
In-Reply-To: <20220124215435.GK14046@twin.jikos.cz>

On Mon, Jan 24, 2022 at 10:54:35PM +0100, David Sterba wrote:
> On Wed, Nov 17, 2021 at 12:19:19PM -0800, Omar Sandoval wrote:
> > From: Omar Sandoval <osandov@fb.com>
> 
> > - We don't do read repair, because it turns out that read repair is
> >   currently broken for compressed data.
> 
> Is there a reproducer, and a fix?

Reproducer:

# Create a filesystem with data duplicated at 2 physical locations on the same
# disk.
$ mkfs.btrfs -f -d dup /dev/vdb
# Mount it with compression enabled.
$ mount -o compress /dev/vdb /mnt
# Write some compressible data (octal dump of random data).
$ dd if=/dev/urandom bs=4k count=1 | od > /mnt/foo
# Force it on disk.
$ sync
# Get the locations it was written to on disk.
$ ~/repos/osandov-linux/scripts/btrfs_map_physical /mnt/foo | column -ts $'\t'
FILE OFFSET  FILE SIZE  EXTENT OFFSET  EXTENT TYPE                   LOGICAL SIZE  LOGICAL OFFSET  PHYSICAL SIZE  DEVID  PHYSICAL OFFSET
0            20480      0              regular,compression=zlib,dup  20480         298844160       8192           1      575668224
                                                                                                                  1      1005125632
# Corrupt one of the copies.
$ dd if=/dev/zero of=/dev/vdb bs=4k count=1 seek=575668224 oflag=seek_bytes
$ sync
# Now, re-read the file until we read it from the corrupted copy. To make sure
# we read from disk, we drop the page cache between each read.
$ while ! btrfs device stats /dev/vdb | grep -q 'corruption_errs\s\+[1-9]'; do echo 1 > /proc/sys/vm/drop_caches; cat /mnt/foo > /dev/null; done
$ dmesg | tail
[ 3240.222922] BTRFS info (device vdb): has skinny extents
[ 3240.235245] BTRFS info (device vdb): checking UUID tree
[ 3298.885372] bash (481): drop_caches: 1
[ 3298.924648] BTRFS warning (device vdb): csum failed root 5 ino 257 off 575676416 csum 0x8941f998 expected csum 0x05a3d0cd mirror 1
[ 3298.924657] BTRFS error (device vdb): bdev /dev/vdb errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[ 3298.926221] BTRFS info (device vdb): read error corrected: ino 257 off 0 (dev /dev/vdb sector 1124352)
[ 3298.926473] BTRFS info (device vdb): read error corrected: ino 257 off 4096 (dev /dev/vdb sector 1124352)
[ 3298.926516] BTRFS info (device vdb): read error corrected: ino 257 off 16384 (dev /dev/vdb sector 1124352)
[ 3298.926555] BTRFS info (device vdb): read error corrected: ino 257 off 8192 (dev /dev/vdb sector 1124352)
[ 3298.926614] BTRFS info (device vdb): read error corrected: ino 257 off 12288 (dev /dev/vdb sector 1124352)
# Now check that the copies match.
$ dd if=/dev/vdb bs=4k count=1 skip=575668224 iflag=skip_bytes status=none | sha256sum
6ffd32b49d77b9e4ae07fd1c598b8407bc4cbb2fdb7244420589703f35605996  -
$ dd if=/dev/vdb bs=4k count=1 skip=1005125632 iflag=skip_bytes status=none | sha256sum
3646164006a08d908f5cbd6131ce413c8de49566560bf7ac6bc9432ac792605d  -
# Oops, they don't. Check the corrupted copy.
$ dd if=/dev/vdb bs=4k count=1 skip=575668224 iflag=skip_bytes status=none | head
0006000 006760 154362 010427 177511 151544 074422 116513 105472
0006020 045224 041623 063647 150022 006147 155332 053640 077304
0006040 173561 102237 155233 021641 165413 114564 004351 006141
0006060 075766 007723 017005 142265 175347 110221 071117 004421
0006100 045713 040102 005447 127414 173546 075206 042537 176547
0006120 120721 117062 177257 171012 130114 031767 165144 103776
0006140 054556 152447 123212 023000 062570 103502 057476 065541
0006160 064364 151221 117125 133463 076760 133756 133026 171622
0006200 164705 031677 005544 027201 130024 177437 102433 005170
0006220 144353 136645 147416 035017 173440 121605 050052 050325
# It contains decompressed data, and not even for the correct offset: this is
# the 4th block of the file.

I don't have a fix. Rohit Singh from my team is ramping up and working
on a fix.

This bug has been present as far back as I could follow the history. I
think it's just been masked by the fact that 1) repairing the bad copies
with bad data doesn't make things "worse" and 2) the page cache caches
the good data, so once you've repaired it once, you don't see it again
until you reboot or the page is evicted.