Re: btrfs rare silent data corruption with kernel data leak

[Kai Krakow <hurikhan77@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4124 bytes --]

Am Wed, 21 Sep 2016 09:02:09 -0400
schrieb Zygo Blaxell <ce3g8jdj@umail.furryterror.org>:

> On Wed, Sep 21, 2016 at 11:14:35AM +0000, Paul Jones wrote:
> > > -----Original Message-----
> > > From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
> > > owner@vger.kernel.org] On Behalf Of Zygo Blaxell
> > > Sent: Wednesday, 21 September 2016 2:56 PM
> > > To: linux-btrfs@vger.kernel.org
> > > Subject: btrfs rare silent data corruption with kernel data leak
> > > 
> > > Summary:
> > > 
> > > There seem to be two btrfs bugs here: one loses data on writes,
> > > and the other leaks data from the kernel to replace it on reads.
> > > It all happens after checksums are verified, so the corruption is
> > > entirely silent--no EIO errors, kernel messages, or device event
> > > statistics.
> > > 
> > > Compressed extents are corrupted with kernel data leak.
> > > Uncompressed extents may not be corrupted, or may be corrupted by
> > > deterministically replacing data bytes with zero, or may not be
> > > corrupted.  No preconditions for corruption are known.  Less than
> > > one file per hundred thousand seems to be affected.  Only
> > > specific parts of any file can be affected. Kernels v4.0..v4.5.7
> > > tested, all have the issue.  
> > 
> > Funny you should bring this up - I think I just suffered from this,
> > or something similar.  
> [...snip...]
> > [ 1311.456545] BTRFS warning (device sda1): csum failed ino
> > 44628192 off 3222405120 csum 2566472073 expected csum 3548745998
> > [ 1311.456551] BTRFS warning (device sda1): csum failed ino
> > 44628192 off 3222470656 csum 2566472073 expected csum 2988893031
> > 
> > I'm seeing a lot of checksum 2566472073 - Is that the checksum of
> > blank space I wonder?  
> 
> The issue I found occurs after successful checksum validation, and
> can affect only the first 4096 bytes of a file.  It produces no kernel
> messages and no checksum failures.  If you run 'filefrag -v' on the
> file, the corrupted extent *must* have the 'inline' flag; otherwise,
> it's a different problem.
> 
> You have csum failures, and the affected offsets are way beyond 4096.
> 
> Holes are not checksummed in general because they are gaps in the
> file offset address space.  Metadata records that describe holes are
> checksummed, but when those checksums fail the kernel messages look
> different.
> 
> I don't recognize the symptoms you are having.  After eliminating
> hardware problems and making sure your kernel is up to date, I'd try
> changing mount options...
> 
> > Here are the details of the filesystem concerned:  
> [...]
> > /dev/sda1 on / type btrfs
> > (rw,noatime,ssd,discard,noacl,space_cache=v2,subvolid=5,subvol=/)
> > (compress was enabled previously)  
> 
> ...discard and space_cache=v2 in particular.  Free space tree is a new
> feature, and discard has had a few bug fixes over the years.  Both are
> non-default options.
> 
> Speaking of mount options, 'max_inline=0' may prevent the issue I
> found.

This sounds a lot like problems I've seen with my virtualbox images.
They just accumulate csum errors over time, especially after
defragmenting. No matter if discard was enabled or not.

The only common option is compression, I'm using compress=lzo.

I made the files nocow which of course disables compression and
probably results in not triggering autodefrag. Of course this also
disables checksums so you won't see data corruption as a direct error
in the FS. But the vm images are running fine so far without any
problems. Data integrity and data safety looks healthy.

You probably want to nonow mysql anyways. I think mysql data files have
their own checksumming, copy on write, and transactions. There's really
no point in doing it twice.

So it seems that btrfs either can damage files on defragmentation or
that compression has a bug with files that are heavily modified inline
(thus breaking the file into a lot of extents) - which probably has bad
effects on compressed extents, too.

-- 
Regards,
Kai

Replies to list-only preferred.

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 181 bytes --]