Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
From: Chris Murphy <lists@colorremedies.com>
To: Gard Vaaler <gardv@megacandy.net>
Cc: Chris Murphy <lists@colorremedies.com>,
	Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Subject: Re: Unrecoverable corruption after loss of cache
Date: Wed, 4 Dec 2019 14:09:14 -0700
Message-ID: <CAJCQCtQW+-VyATVzi47vtBvN34Ev8j704tiQDVZKxHqT15qccw@mail.gmail.com> (raw)
In-Reply-To: <B154F1B0-C80A-4E7E-B105-B0E654279E28@megacandy.net>

On Wed, Dec 4, 2019 at 1:17 PM Gard Vaaler <gardv@megacandy.net> wrote:
>
> 4. des. 2019 kl. 20:08 skrev Chris Murphy <lists@colorremedies.com>:
> Why do you think it's complaining about the journal? I'm not seeing
> tree log related messages here.
>
>
> Thanks for the reply! That must be a misunderstanding on my part (it's called "transid", which suggested something in the journal to me).

Gotcha, yeah transid is just a way Btrfs keeps track of separate
commits over time. In effect the file system itself is the journal,
there is no separate dedicated journal on Btrfs like you see on ext4
or XFS. For fsync performance enhancement there is a log tree which
might be somewhat like a journal, which is what zero log is wiping
away. Pretty much on all file systems, it's best to allow log replay
to happen before zeroing it, and only zeroing it if there's a problem
reported about it, rather than as an early trouble shooting step.


>
> Is the output provided complete or are
> there additional messages?
>
>
> No, that's it.
>
> What do you get for:
>
> btrfs insp dump-s /dev/X

OK so no log tree, therefore not related.

>
>
> Attached.
>
> What kernel version was being used at the time of the first problem instance?
>
>
> Fedora's 5.2.8-300 kernel.

There's a decent chance this is the cause of the problem. That kernel
does not have the fix for this bug:
https://www.spinics.net/lists/stable-commits/msg129532.html
https://bugzilla.redhat.com/show_bug.cgi?id=1751901

As far as I'm aware the corruption isn't fixable. You might still be
able to mount the file system ro to get data out; if not then decent
chance you can extract data with btrfs restore, which is an offline
scraping tool, but it is a bit tedious to use.
https://btrfs.wiki.kernel.org/index.php/Restore

The real fix it to make a new Btrfs file system, and don't use kernels
5.2.0-5.2.14. Fedora 29, 30, 31 all are long since on 5.3.x series
kernels which do have this fix incorporated. But the fix found it's
way into 5.2.15 pretty soon after discovery so I'm gonna guess you've
got updates disabled and just got unlucky to get hit by this bug.


>
> The transid messages above suggest some kind of failure to actually
> commit what should have ended up on stable media. Also please provide:
>
> btrfs-find-root /dev/
>
>
> Attached (compressed).
>
> btrfs check --mode=lowmem /dev/
>
>
> Attached.
>
> The latter will take a while and since it is an offline check will
> need to be done in initramfs, or better from Live media which will
> make it easier to capture the output. I recommend btrfs-progs not
> older than 5.1.1 if possible. It is only for check, not with --repair,
> so the version matters somewhat less if it's not too old.
>
>
> As you can see, it terminates almost immediately with an IO error. However, there's no error in the dmesg on the underlying device, which makes me think there's a bad bounds check or something similar.



--
Chris Murphy

  parent reply index

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-01 17:27 Gard Vaaler
2019-12-01 18:51 ` Nikolay Borisov
2019-12-02 21:27   ` Gard Vaaler
2019-12-05  2:45     ` Zygo Blaxell
2019-12-04 15:50 ` Gard Vaaler
2019-12-04 19:08   ` Chris Murphy
2019-12-04 20:21     ` Gard Vaaler
     [not found]     ` <B154F1B0-C80A-4E7E-B105-B0E654279E28@megacandy.net>
2019-12-04 21:09       ` Chris Murphy [this message]
2019-12-05  0:34         ` Gard Vaaler
2019-12-05  1:26           ` Chris Murphy

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAJCQCtQW+-VyATVzi47vtBvN34Ev8j704tiQDVZKxHqT15qccw@mail.gmail.com \
    --to=lists@colorremedies.com \
    --cc=gardv@megacandy.net \
    --cc=linux-btrfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org
	public-inbox-index linux-btrfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git