Re: Massive filesystem corruption since kernel 5.2 (ARCH)

From: "Swâmi Petaramesh" <swami@petaramesh.org>
To: Qu Wenruo <quwenruo.btrfs@gmx.com>
Cc: Anand Jain <anand.jain@oracle.com>,
	Lionel Bouton <lionel-subscription@bouton.name>,
	linux-btrfs@vger.kernel.org
Subject: Re: Massive filesystem corruption since kernel 5.2 (ARCH)
Date: Thu, 1 Aug 2019 10:07:49 +0200	[thread overview]
Message-ID: <e73421f5-444b-2daa-4c28-45f3b5db007c@petaramesh.org> (raw)
In-Reply-To: <a199a382-3ea4-e061-e5fc-dc8c2cc66e73@gmx.com>

On 8/1/19 8:36 AM, Qu Wenruo wrote:
> Could you give more detailed history, including each reboot?
> Like:
>
> CASE 1
> # Upgrade kernel (running 5.1)
> # Reboot
> # Kernel mount failure (running 5.2)

No, it never was a “kernel mount failure”, it was more of :

- Running 5.1 OK

- Upgrade to 5.2

- Reboot without noticing problem on kernel 5.2.1-arch1-1

- Performed usual remote rsync backup using kernel 5.2.1-arch1-1 WITHOUT
any error at 23:20 on july, 16

- Quite unfortunately I do not backup /var/log in frequent rsync backups...

- Machine does its usual cronned snapper snapshots auto-delete

- Turned off machine for the night

- Next days, boot machine as usual (without paying attention to
scrolling messages)

- Machine boots. Cinnamon GUI fails loading. Wonders. Reboot.

- Notice BTRFS error messages on console at boot. Still no GUI.

- Reboot in systemd rescue mode. Run "btrfs check -f" in read-only mode.

- Get LOADS of error messages.

- Tells myself « Jeez the damn thing screwed up ! »

- Reboot in multi-user.target console mode

- Notice BTRFS errors again.

- Connect external USB HD for performing an emergency full backup of
what can be.

- Lack enough space on external USB HD. Delete a load of old snapshots
to make enough space.

- Perform full backup (rsync onto external HD). Everything goes well
except for a few recently modified files that fail. Either temp or cache
files I can live without, or files that are OK in the remote backup
performed the evening before.

- Wait a few days before restoring the machine - lack of time.

- Reformat and restore the machine, reverting to kernel 5.1.

- Want to perform more backups onto the external USB HD.

- Get BTRFS errors on the external HD (posted here previously).

- Eventually decide to reformat the external HD completely as the FS
seems to be beyond salvation by “btrfs check”.

- The machine and involved disks seems stable and have been checked
healthy now with kernel 5.1.

- As you can see, the damaged filesystems have been reformatted, and I'm
afraid I don't have useful logs available.

> (It's a really pity that the original corrupted leaf kernel message
> can't be preserved, that could really help a lot to detect memory
> corruption or things like that

Well I'm sorry..

Kind regards.

-- 

ॐ

Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E