Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
From: Qu Wenruo <quwenruo.btrfs@gmx.com>
To: Justin Brown <Justin.Brown@fandingo.org>,
	linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: Access Beyond End of Device & Input/Output Errors
Date: Sat, 1 Aug 2020 14:58:58 +0800
Message-ID: <8e17a4d1-6555-15ba-808c-dd867d7ecbcb@gmx.com> (raw)
In-Reply-To: <CAKZK7uwRs_tf6htRtJvw3kNhyNPMJ-juA6_WSJo+PbQA7f40Cg@mail.gmail.com>

[-- Attachment #1.1: Type: text/plain, Size: 6999 bytes --]



On 2020/8/1 下午2:51, Justin Brown wrote:
> Hello,
> 
> I've run into a strange problem that I haven't seen before, and I need
> some help. I started getting generic "input/output" errors on a couple
> of files, and when I looked deeper, the kernel logs are full of
> messages like:
> 
>     sd 5:0:0:0: [sdf] tag#29 access beyond end of device

We had a new fix for trim. But according to your kernel message, it
doesn't look like the case.

(No obvious tag showing it's trim/discard)

> 
> I've never seen anything like this before with any FS, so I figured it
> was worth asking before I consider running the standard btrfs tools.
> (I briefly started a scrub, but it was going crazy with uncorrectable
> errors, so I cancelled it.)
> 
> Here's my system info:
> 
> Fedora 32, kernel 5.7.7-200.fc32.x86_64
> btrfs-progs v5.7
> 
> /etc/fstab entry:
> LABEL=media /var/media btrfs subvol=media,discard 0 2
> 
> btrfs fi show /var/media/
> Label: 'media' uuid: 51eef0c7-2977-4037-b271-3270ea22c7d9
> Total devices 6 FS bytes used 4.68TiB
> devid 1 size 1.82TiB used 963.00GiB path /dev/sdf1
> devid 2 size 1.82TiB used 962.00GiB path /dev/sde1
> devid 4 size 1.82TiB used 963.00GiB path /dev/sdg1
> devid 6 size 1.82TiB used 962.03GiB path /dev/sda1
> devid 7 size 7.28TiB used 967.03GiB path /dev/sdb1
> devid 8 size 7.28TiB used 967.03GiB path /dev/sdd1
> 
> btrfs fi df /var/media/
> Data, RAID5: total=4.69TiB, used=4.68TiB
> System, RAID1C3: total=32.00MiB, used=304.00KiB
> Metadata, RAID1C3: total=6.00GiB, used=4.94GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> I can only mount -o degraded now. Here are the logs when mounting:
> 
> Aug 01 01:15:26 spaceman.fandingo.org sudo[275572]: justin : TTY=pts/0
> ; PWD=/home/justin ; USER=root ; COMMAND=/usr/bin/mount -t btrfs -o
> degraded /dev/sda1 /var/media/
> Aug 01 01:15:26 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
> access beyond end of device
> Aug 01 01:15:26 spaceman.fandingo.org kernel: blk_update_request: I/O
> error, dev sdf, sector 2176 op 0x0:(READ) flags 0x0 phys_seg 1 prio
> class 0

OK, it's read, not DISCARD, thus a completely different problem.


> Aug 01 01:15:26 spaceman.fandingo.org kernel: Buffer I/O error on dev
> sdf1, logical block 16, async page read
> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> sde1): allowing degraded mounts
> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> sde1): disk space caching is enabled
> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): devid 1 uuid cb05aae6-6c03-49d3-b46d-bf51a0eb8cd0 is missing
> Aug 01 01:15:26 spaceman.fandingo.org kernel: BTRFS info (device
> sde1): bdev /dev/sdf1 errs: wr 4458026, rd 14571, flush 0, corrupt 0,
> gen 0
> 
> It seems like only relatively recently written files are encountering
> I/O errors. If I `cat` one of the problematic files when the FS is
> mounted normally, I see a ton of this:
> 
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#26
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#27
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#28
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#29
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#30
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#0
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#1
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#13
> access beyond end of device
> Aug 01 01:13:49 spaceman.fandingo.org kernel: sd 5:0:0:0: [sdf] tag#2
> access beyond end of device
> 
> Now that I'm remounted in -o degraded, I'm getting more comprehensible
> warnings, but it still results in I/O read failures:
> 
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99942400 csum 0x8941f998
> expected csum 0xbe3f80a4 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99946496 csum 0x8941f998
> expected csum 0x9c36a6b4 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99950592 csum 0x8941f998
> expected csum 0x44d30ca2 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99958784 csum 0x8941f998
> expected csum 0xc0f08acc mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99954688 csum 0x8941f998
> expected csum 0xcb11db59 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99962880 csum 0x8941f998
> expected csum 0x8a4ee0aa mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99971072 csum 0x8941f998
> expected csum 0xdfb79e85 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99966976 csum 0x8941f998
> expected csum 0xc14921a0 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99975168 csum 0x8941f998
> expected csum 0xf2fe8774 mirror 2
> Aug 01 01:31:53 spaceman.fandingo.org kernel: BTRFS warning (device
> sde1): csum failed root 2820 ino 747435 off 99979264 csum 0x8941f998
> expected csum 0xae1cafd6 mirror 2
> 
> Why trying to research this problem, I came across a Github issue
> https://github.com/kdave/btrfs-progs/issues/282 and a patch from Qu
> from yesterday ([PATCH] btrfs: trim: fix underflow in trim length to
> prevent access beyond device boundary). I do use the discard mount
> option, and I have a weekly fstrim.timer enabled. I did replace 2x2TB
> drives with the 2x8TB drives about 1 month ago, which involved a
> conversion to -d raid5 -m raid1c3, which I suppose could hit the same
> code paths that resize2fs would?

The problem doesn't look like a trim one, but more likely some device
boundary bug.

Would you please provide the following info?
- btrfs ins dump-tree -t chunk /dev/sde1
  This contains the device info and chunk tree dump. Doesn't contain
  any confidential info.
  We can use this info to determine if there is some chunk really beyond
  device boundary.
  I guess some chunks are already beyond device boundary by somehow.

Thanks,
Qu

> 
> Any advice on how to proceed would be greatly appreciated.
> 
> Thanks,
> Justin
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  reply index

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-08-01  6:51 Justin Brown
2020-08-01  6:58 ` Qu Wenruo [this message]
2020-08-01  7:02   ` Qu Wenruo
     [not found]     ` <CAKZK7uzmg19NDjGPPAxXKu7LJ-7ZdHu2cad22csj_chr2qxMJg@mail.gmail.com>
2020-08-01  9:31       ` Qu Wenruo
2020-08-01 11:56         ` Justin Brown
2020-08-01 23:30           ` Qu Wenruo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8e17a4d1-6555-15ba-808c-dd867d7ecbcb@gmx.com \
    --to=quwenruo.btrfs@gmx.com \
    --cc=Justin.Brown@fandingo.org \
    --cc=linux-btrfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org
	public-inbox-index linux-btrfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git