csum errors in VirtualBox VDI files

csum errors in VirtualBox VDI files

[Kai Krakow]

Hello!

Since one of the last kernel updates (I don't know which exactly), I'm
experiencing csum errors within VDI files when running VirtualBox. A
side effect of this is, as soon as dmesg shows these errors, commands
like "du" and "df" hang until reboot.

I've now restored the file from backup but it happens over and over
again.

On another machine I'm also seeing errors with big files in the
following scenario (apparently an older kernel, 4.1.x I afair):

# ntfsclone --save /dev/md126p2 -o rescue.ntfs.img
                   ^ big NTFS partition   ^ file on btrfs

results in a write error and the file system goes read-only.

Both systems have in common they are using btrfs on bcache with
compress=lzo,autodefrag,nossd,discard (mraid=1,draid=0 and
mraid=1,draid=single).

The system mentioned first is running Kernel 4.5.0 with Gentoo
patch-set. I upgraded from the last 4.4.x kernel when I first
experienced this problem. The first time the problem resulted in a
duplicate extent which btrfsck wasn't able to fix, that's when I first
restored from backup. But now I'm getting csum errors in this file over
a over again, plus when rsync has run for backup, the system no longer
responds to "du" and "df" commands - it just hangs.

Known problem? Does it help if I send debug info? If so, please
instruct.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Tue, 22 Mar 2016 09:03:42 +0100
schrieb Kai Krakow <hurikhan77@gmail.com>:

> Hello!
> 
> Since one of the last kernel updates (I don't know which exactly), I'm
> experiencing csum errors within VDI files when running VirtualBox. A
> side effect of this is, as soon as dmesg shows these errors, commands
> like "du" and "df" hang until reboot.
> 
> I've now restored the file from backup but it happens over and over
> again.
> 
> On another machine I'm also seeing errors with big files in the
> following scenario (apparently an older kernel, 4.1.x I afair):
> 
> # ntfsclone --save /dev/md126p2 -o rescue.ntfs.img
>                    ^ big NTFS partition   ^ file on btrfs
> 
> results in a write error and the file system goes read-only.
> 
> Both systems have in common they are using btrfs on bcache with
> compress=lzo,autodefrag,nossd,discard (mraid=1,draid=0 and
> mraid=1,draid=single).
> 
> The system mentioned first is running Kernel 4.5.0 with Gentoo
> patch-set. I upgraded from the last 4.4.x kernel when I first
> experienced this problem. The first time the problem resulted in a
> duplicate extent which btrfsck wasn't able to fix, that's when I first
> restored from backup. But now I'm getting csum errors in this file
> over a over again, plus when rsync has run for backup, the system no
> longer responds to "du" and "df" commands - it just hangs.
> 
> Known problem? Does it help if I send debug info? If so, please
> instruct.

Interestingly, ddrescue just skips over these csum errors without
counting an error...


-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Tue, 22 Mar 2016 09:03:42 +0100
schrieb Kai Krakow <hurikhan77@gmail.com>:

> Hello!
> 
> Since one of the last kernel updates (I don't know which exactly), I'm
> experiencing csum errors within VDI files when running VirtualBox. A
> side effect of this is, as soon as dmesg shows these errors, commands
> like "du" and "df" hang until reboot.
> 
> I've now restored the file from backup but it happens over and over
> again.
> 
> On another machine I'm also seeing errors with big files in the
> following scenario (apparently an older kernel, 4.1.x I afair):
> 
> # ntfsclone --save /dev/md126p2 -o rescue.ntfs.img
>                    ^ big NTFS partition   ^ file on btrfs
> 
> results in a write error and the file system goes read-only.
> 
> Both systems have in common they are using btrfs on bcache with
> compress=lzo,autodefrag,nossd,discard (mraid=1,draid=0 and
> mraid=1,draid=single).
> 
> The system mentioned first is running Kernel 4.5.0 with Gentoo
> patch-set. I upgraded from the last 4.4.x kernel when I first
> experienced this problem. The first time the problem resulted in a
> duplicate extent which btrfsck wasn't able to fix, that's when I first
> restored from backup. But now I'm getting csum errors in this file
> over a over again, plus when rsync has run for backup, the system no
> longer responds to "du" and "df" commands - it just hangs.
> 
> Known problem? Does it help if I send debug info? If so, please
> instruct.
> 

[ 3073.426785] BTRFS info (device bcache2): no csum found for inode 7528856 start 79614263296
[ 3073.447613] BTRFS info (device bcache2): csum failed ino 7528856 extent 128363245568 csum 3730946112 wanted 0 mirror 0
[ 3073.554746] BTRFS info (device bcache2): no csum found for inode 7528856 start 79614263296
[ 3073.590864] BTRFS info (device bcache2): csum failed ino 7528856 extent 128363245568 csum 3730946112 wanted 0 mirror 0
[ 3073.590930] BTRFS info (device bcache2): no csum found for inode 7528856 start 79614263296
[ 3073.617864] BTRFS info (device bcache2): csum failed ino 7528856 extent 128363245568 csum 3730946112 wanted 0 mirror 0
[ 3073.618057] BTRFS info (device bcache2): no csum found for inode 7528856 start 79614263296
[ 3073.644597] BTRFS info (device bcache2): csum failed ino 7528856 extent 412938412032 csum 3730946112 wanted 0 mirror 0


-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Qu Wenruo]

Hi,

Kai Krakow wrote on 2016/03/22 09:03 +0100:
> Hello!
>
> Since one of the last kernel updates (I don't know which exactly), I'm
> experiencing csum errors within VDI files when running VirtualBox. A
> side effect of this is, as soon as dmesg shows these errors, commands
> like "du" and "df" hang until reboot.
>
> I've now restored the file from backup but it happens over and over
> again.
>
> On another machine I'm also seeing errors with big files in the
> following scenario (apparently an older kernel, 4.1.x I afair):
>
> # ntfsclone --save /dev/md126p2 -o rescue.ntfs.img
>                     ^ big NTFS partition   ^ file on btrfs
>
> results in a write error and the file system goes read-only.

When it goes RO, it must have some warning in kernel log.
Would you please paste the kernel log?

>
> Both systems have in common they are using btrfs on bcache with
> compress=lzo,autodefrag,nossd,discard (mraid=1,draid=0 and
> mraid=1,draid=single).
>
> The system mentioned first is running Kernel 4.5.0 with Gentoo
> patch-set. I upgraded from the last 4.4.x kernel when I first
> experienced this problem. The first time the problem resulted in a
> duplicate extent which btrfsck wasn't able to fix, that's when I first
> restored from backup. But now I'm getting csum errors in this file over
> a over again, plus when rsync has run for backup, the system no longer
> responds to "du" and "df" commands - it just hangs.
>
> Known problem? Does it help if I send debug info? If so, please
> instruct.
>
Does btrfs check report anything wrong?

Thanks,
Qu

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Tue, 22 Mar 2016 16:47:10 +0800
schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:

> Hi,
> 
> Kai Krakow wrote on 2016/03/22 09:03 +0100:
> > Hello!
> >
> > Since one of the last kernel updates (I don't know which exactly),
> > I'm experiencing csum errors within VDI files when running
> > VirtualBox. A side effect of this is, as soon as dmesg shows these
> > errors, commands like "du" and "df" hang until reboot.
> >
> > I've now restored the file from backup but it happens over and over
> > again.
> >
> > On another machine I'm also seeing errors with big files in the
> > following scenario (apparently an older kernel, 4.1.x I afair):
> >
> > # ntfsclone --save /dev/md126p2 -o rescue.ntfs.img
> >                     ^ big NTFS partition   ^ file on btrfs
> >
> > results in a write error and the file system goes read-only.  
> 
> When it goes RO, it must have some warning in kernel log.
> Would you please paste the kernel log?

Apparently, that system does not boot now due to errors in bcache
b-tree. That being that, it may well be some bcache error and not
btrfs' fault. Apparently I couldn't catch the output, I've been in a
hurry. It said "write error" and had some backtrace. I will come to
this back later.

Let's go to the system I currently care about (that one with the
always breaking VDI file):

> > Both systems have in common they are using btrfs on bcache with
> > compress=lzo,autodefrag,nossd,discard (mraid=1,draid=0 and
> > mraid=1,draid=single).
> >
> > The system mentioned first is running Kernel 4.5.0 with Gentoo
> > patch-set. I upgraded from the last 4.4.x kernel when I first
> > experienced this problem. The first time the problem resulted in a
> > duplicate extent which btrfsck wasn't able to fix, that's when I
> > first restored from backup. But now I'm getting csum errors in this
> > file over a over again, plus when rsync has run for backup, the
> > system no longer responds to "du" and "df" commands - it just hangs.
> >
> > Known problem? Does it help if I send debug info? If so, please
> > instruct.
> >  
> Does btrfs check report anything wrong?

After the error occured?

Yes, some text about the extent being compressed and btrfs repair
doesn't currently handle that case (I tried --repair as I'm having a
backup). I simply decided not to investigate that further at that point
but delete and restore the affected file from backup. However, this is
the message from dmesg (tho, I didn't catch the backtrace):

btrfs_run_delayed_refs:2927: errno=-17 Object already exists

After this, the system went RO and I had to reboot. I ran btrfs check
and it told about a duplicate extent. I identified the file (using
btrfs inspect and the inode number) being the VDI file, and restored it.
Afterwards, I upgraded from latest 4.4 to 4.5. Currently, I'm now
watching closer since this incident, and the file becomes damaged
without any message in the kernel log when doing some more than usual
IO in VirtualBox. When my backup script then runs over the file, I get
errors about missing csums - the block is not readable. I now ran
ddrescue, and replaced the file to get a current and slightly damaged
VDI image back (my backup uses time rotation, so no problem). But
running chkdsk in VirtualBox damages the VDI again.

Regarding the other error on the other machine, I'm not completely
convinced bcache ain't involved in this problem.

As soon as I "produced" csum errors again, I'll run btrfs check. Or
should I do it now without forcing the csum error to occur?


-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Chris Murphy]

This is kinda confusing.

So the gist is that the guest OS is Windows, so the VDI contains an
NTFS file system. Correct? And that VDI is on a Btrfs formatted bcache
device. Correct? Does the VDI have +C xattr set?


Chris Murphy

Re: csum errors in VirtualBox VDI files

[Henk Slager]

On Tue, Mar 22, 2016 at 9:03 AM, Kai Krakow <hurikhan77@gmail.com> wrote:
> Hello!
>
> Since one of the last kernel updates (I don't know which exactly), I'm
> experiencing csum errors within VDI files when running VirtualBox. A
> side effect of this is, as soon as dmesg shows these errors, commands
> like "du" and "df" hang until reboot.
>
> I've now restored the file from backup but it happens over and over
> again.
>
> On another machine I'm also seeing errors with big files in the
> following scenario (apparently an older kernel, 4.1.x I afair):
>
> # ntfsclone --save /dev/md126p2 -o rescue.ntfs.img
>                    ^ big NTFS partition   ^ file on btrfs
>
> results in a write error and the file system goes read-only.
>
> Both systems have in common they are using btrfs on bcache with
> compress=lzo,autodefrag,nossd,discard (mraid=1,draid=0 and
> mraid=1,draid=single).

autodefrag,nossd:  I am using these too on bcached btrfs, so far no
issues (including kernel 4.5.0). I have been writing bigfiles (> 50G)
without problems.
compress=lzo: I have used it on bcached btrfs in the past, worked fine
discard: I am not sure what this should do on top of /dev/bcacheX ; I
don't know how this relates to the bcache discard; I currently have
bcache setting:  Discard?  False

I am not saying that the last mount option is the direct cause of the
problems you experience, it is just that I don't know its impact
currently.

> The system mentioned first is running Kernel 4.5.0 with Gentoo
> patch-set. I upgraded from the last 4.4.x kernel when I first
> experienced this problem. The first time the problem resulted in a
> duplicate extent which btrfsck wasn't able to fix, that's when I first
> restored from backup. But now I'm getting csum errors in this file over
> a over again, plus when rsync has run for backup, the system no longer
> responds to "du" and "df" commands - it just hangs.
>
> Known problem? Does it help if I send debug info? If so, please
> instruct.
>
> --
> Regards,
> Kai
>
> Replies to list-only preferred.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Tue, 22 Mar 2016 13:42:00 -0600
schrieb Chris Murphy <lists@colorremedies.com>:

> This is kinda confusing.

Err yes maybe... ;-D

> So the gist is that the guest OS is Windows, so the VDI contains an
> NTFS file system. Correct? And that VDI is on a Btrfs formatted bcache
> device. Correct? Does the VDI have +C xattr set?

Yes:

VirtualBox runs guest OS "Windows 7 32-bit"
VDI file is stored on btrfs running on bcache device
chattr +C is not set

I used chattr +C until a few weeks ago when I decided to flip
autodefrag back on and realised that +C won't allow compressing file
contents. The VM actually ran faster afterwards. And it ran fine until
lately when those errors occurred.

But I have to admit I didn't use the VM a lot in a while. So the first
time the error occurred may be a few more days back. According to my
snapshot backlog on the backup partition, the last successful backup of
this VDI file was on 2016-03-13 when I was running kernel 4.4.4
according to my emerge log.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Tue, 22 Mar 2016 21:07:35 +0100
schrieb Henk Slager <eye1tm@gmail.com>:

> On Tue, Mar 22, 2016 at 9:03 AM, Kai Krakow <hurikhan77@gmail.com>
> wrote:
> > Hello!
> >
> > Since one of the last kernel updates (I don't know which exactly),
> > I'm experiencing csum errors within VDI files when running
> > VirtualBox. A side effect of this is, as soon as dmesg shows these
> > errors, commands like "du" and "df" hang until reboot.
> >
> > I've now restored the file from backup but it happens over and over
> > again.
> >
> > On another machine I'm also seeing errors with big files in the
> > following scenario (apparently an older kernel, 4.1.x I afair):
> >
> > # ntfsclone --save /dev/md126p2 -o rescue.ntfs.img
> >                    ^ big NTFS partition   ^ file on btrfs
> >
> > results in a write error and the file system goes read-only.
> >
> > Both systems have in common they are using btrfs on bcache with
> > compress=lzo,autodefrag,nossd,discard (mraid=1,draid=0 and
> > mraid=1,draid=single).  
> 
> autodefrag,nossd:  I am using these too on bcached btrfs, so far no
> issues (including kernel 4.5.0). I have been writing bigfiles (> 50G)
> without problems.
> compress=lzo: I have used it on bcached btrfs in the past, worked fine
> discard: I am not sure what this should do on top of /dev/bcacheX ; I
> don't know how this relates to the bcache discard; I currently have
> bcache setting:  Discard?  False
> 
> I am not saying that the last mount option is the direct cause of the
> problems you experience, it is just that I don't know its impact
> currently.

Well, maybe interesting... My backup drive doesn't use discard and
shows no such misbehavior. I try disabling it as soon as I'm having
some time elaborating on that.

I guess only the devs can tell how discard interacts with bcache, and
if it makes sense to enable it for this setup.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Qu Wenruo]

Kai Krakow wrote on 2016/03/22 19:48 +0100:
> Am Tue, 22 Mar 2016 16:47:10 +0800
> schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:
>
>> Hi,
>>
>> Kai Krakow wrote on 2016/03/22 09:03 +0100:
>>> Hello!
>>>
>>> Since one of the last kernel updates (I don't know which exactly),
>>> I'm experiencing csum errors within VDI files when running
>>> VirtualBox. A side effect of this is, as soon as dmesg shows these
>>> errors, commands like "du" and "df" hang until reboot.
>>>
>>> I've now restored the file from backup but it happens over and over
>>> again.
>>>
>>> On another machine I'm also seeing errors with big files in the
>>> following scenario (apparently an older kernel, 4.1.x I afair):
>>>
>>> # ntfsclone --save /dev/md126p2 -o rescue.ntfs.img
>>>                      ^ big NTFS partition   ^ file on btrfs
>>>
>>> results in a write error and the file system goes read-only.
>>
>> When it goes RO, it must have some warning in kernel log.
>> Would you please paste the kernel log?
>
> Apparently, that system does not boot now due to errors in bcache
> b-tree. That being that, it may well be some bcache error and not
> btrfs' fault. Apparently I couldn't catch the output, I've been in a
> hurry. It said "write error" and had some backtrace. I will come to
> this back later.
>
> Let's go to the system I currently care about (that one with the
> always breaking VDI file):
>
>>> Both systems have in common they are using btrfs on bcache with
>>> compress=lzo,autodefrag,nossd,discard (mraid=1,draid=0 and
>>> mraid=1,draid=single).
>>>
>>> The system mentioned first is running Kernel 4.5.0 with Gentoo
>>> patch-set. I upgraded from the last 4.4.x kernel when I first
>>> experienced this problem. The first time the problem resulted in a
>>> duplicate extent which btrfsck wasn't able to fix, that's when I
>>> first restored from backup. But now I'm getting csum errors in this
>>> file over a over again, plus when rsync has run for backup, the
>>> system no longer responds to "du" and "df" commands - it just hangs.
>>>
>>> Known problem? Does it help if I send debug info? If so, please
>>> instruct.
>>>
>> Does btrfs check report anything wrong?
>
> After the error occured?
>
> Yes, some text about the extent being compressed and btrfs repair
> doesn't currently handle that case (I tried --repair as I'm having a
> backup). I simply decided not to investigate that further at that point
> but delete and restore the affected file from backup. However, this is
> the message from dmesg (tho, I didn't catch the backtrace):
>
> btrfs_run_delayed_refs:2927: errno=-17 Object already exists

That's nice, at least we have some clue.

It's almost sure, it's a bug either in btrfs kernel which doesn't handle 
delayed refs well(low possibility), or, corrupted fs which create 
something kernel can't handle(I bet that's the case).

>
> After this, the system went RO and I had to reboot. I ran btrfs check
> and it told about a duplicate extent.

If output of btrfsck can be posted, it would help a lot to locate the 
problem and enhance btrfsck.

> I identified the file (using
> btrfs inspect and the inode number) being the VDI file, and restored it.
> Afterwards, I upgraded from latest 4.4 to 4.5. Currently, I'm now
> watching closer since this incident, and the file becomes damaged
> without any message in the kernel log when doing some more than usual
> IO in VirtualBox. When my backup script then runs over the file, I get
> errors about missing csums - the block is not readable.

If no other problem reported by btrfsck after your fix, --init-csum 
would handle such case.

> I now ran
> ddrescue, and replaced the file to get a current and slightly damaged
> VDI image back (my backup uses time rotation, so no problem). But
> running chkdsk in VirtualBox damages the VDI again.
>
> Regarding the other error on the other machine, I'm not completely
> convinced bcache ain't involved in this problem.
>
> As soon as I "produced" csum errors again, I'll run btrfs check. Or
> should I do it now without forcing the csum error to occur?
>
>
If it's possible, btrfsck now with all its output posted is recommended.

Thanks,
Qu

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Wed, 23 Mar 2016 12:16:24 +0800
schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:

> Kai Krakow wrote on 2016/03/22 19:48 +0100:
> > Am Tue, 22 Mar 2016 16:47:10 +0800
> > schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:
> >  
> >> Hi,
> >>
> >> Kai Krakow wrote on 2016/03/22 09:03 +0100:  
>  [...]  
> >>
> >> When it goes RO, it must have some warning in kernel log.
> >> Would you please paste the kernel log?  
> >
> > Apparently, that system does not boot now due to errors in bcache
> > b-tree. That being that, it may well be some bcache error and not
> > btrfs' fault. Apparently I couldn't catch the output, I've been in a
> > hurry. It said "write error" and had some backtrace. I will come to
> > this back later.
> >
> > Let's go to the system I currently care about (that one with the
> > always breaking VDI file):
> >  
>  [...]  
> >> Does btrfs check report anything wrong?  
> >
> > After the error occured?
> >
> > Yes, some text about the extent being compressed and btrfs repair
> > doesn't currently handle that case (I tried --repair as I'm having a
> > backup). I simply decided not to investigate that further at that
> > point but delete and restore the affected file from backup.
> > However, this is the message from dmesg (tho, I didn't catch the
> > backtrace):
> >
> > btrfs_run_delayed_refs:2927: errno=-17 Object already exists  
> 
> That's nice, at least we have some clue.
> 
> It's almost sure, it's a bug either in btrfs kernel which doesn't
> handle delayed refs well(low possibility), or, corrupted fs which
> create something kernel can't handle(I bet that's the case).

[kernel 4.5.0 gentoo, btrfs-progs 4.4.1]

Well, this time it hit me on the USB backup drive which uses no bcache
and no other fancy options except compress-force=zlib. Apparently, I've
only got a (real) screenshot which I'm going to link here:

https://www.dropbox.com/s/9qbc7np23y8lrii/IMG_20160326_200033.jpg?dl=0

The same drive has no problems except "bad metadata crossing stripe
boundary" - but a lot of them. This drive was never converted, it was
freshly generated several months ago.

---8<---
$ sudo btrfsck /dev/disk/by-label/usb-backup 
Checking filesystem on /dev/disk/by-label/usb-backup
UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
checking extents
bad metadata [156041216, 156057600) crossing stripe boundary
bad metadata [181403648, 181420032) crossing stripe boundary
bad metadata [392167424, 392183808) crossing stripe boundary
bad metadata [783482880, 783499264) crossing stripe boundary
bad metadata [784924672, 784941056) crossing stripe boundary
bad metadata [130151612416, 130151628800) crossing stripe boundary
bad metadata [162826813440, 162826829824) crossing stripe boundary
bad metadata [162927083520, 162927099904) crossing stripe boundary
bad metadata [619740659712, 619740676096) crossing stripe boundary
bad metadata [619781947392, 619781963776) crossing stripe boundary
bad metadata [619795644416, 619795660800) crossing stripe boundary
bad metadata [619816091648, 619816108032) crossing stripe boundary
bad metadata [620011388928, 620011405312) crossing stripe boundary
bad metadata [890992459776, 890992476160) crossing stripe boundary
bad metadata [891022737408, 891022753792) crossing stripe boundary
bad metadata [891101773824, 891101790208) crossing stripe boundary
bad metadata [891301199872, 891301216256) crossing stripe boundary
[...]
--->8---

My main drive (which this thread was about) has a huge amount of
different problems according to btrfsck. Repair doesn't work: it says
something about overlapping extents and that it needs a careful
thought. I wanted to catch the output when the above problem occured. So
I'd like to defer that until later and first fix my backup drive. If I
lose my main drive, I simply restore from backup. It is very old anyway
(still using 4k node size). Only downside it takes 24+ hours to restore.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Chris Murphy]

On Sat, Mar 26, 2016 at 1:30 PM, Kai Krakow <hurikhan77@gmail.com> wrote:

> Well, this time it hit me on the USB backup drive which uses no bcache
> and no other fancy options except compress-force=zlib. Apparently, I've
> only got a (real) screenshot which I'm going to link here:
>
> https://www.dropbox.com/s/9qbc7np23y8lrii/IMG_20160326_200033.jpg?dl=0

This is a curious screen shot. It's a dracut pre-mount shell, so
nothing should be mounted yet. And btrfs check only works on an
unmounted file system. And yet the bottom part of the trace shows a
Btrfs volume being made read only, as if it was mounted read write and
is still mounted. Huh?


Chris Murphy

Re: csum errors in VirtualBox VDI files

[Chris Murphy]

On Sat, Mar 26, 2016 at 2:28 PM, Chris Murphy <lists@colorremedies.com> wrote:
> On Sat, Mar 26, 2016 at 1:30 PM, Kai Krakow <hurikhan77@gmail.com> wrote:
>
>> Well, this time it hit me on the USB backup drive which uses no bcache
>> and no other fancy options except compress-force=zlib. Apparently, I've
>> only got a (real) screenshot which I'm going to link here:
>>
>> https://www.dropbox.com/s/9qbc7np23y8lrii/IMG_20160326_200033.jpg?dl=0
>
> This is a curious screen shot. It's a dracut pre-mount shell, so
> nothing should be mounted yet. And btrfs check only works on an
> unmounted file system. And yet the bottom part of the trace shows a
> Btrfs volume being made read only, as if it was mounted read write and
> is still mounted. Huh?

Wait. You said no bcache, and yet in this screen shot it shows 'btrfs
check /dev/bcache2 ...' right before the back trace.

This thread is confusing. You're talking about two different btrfs
volumes intermixed, one uses bcache the other doesn't, yet they both
have corruption. I think it's hardware related: bad cable bad ram bad
power, something.



-- 
Chris Murphy

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Sat, 26 Mar 2016 14:28:22 -0600
schrieb Chris Murphy <lists@colorremedies.com>:

> On Sat, Mar 26, 2016 at 1:30 PM, Kai Krakow <hurikhan77@gmail.com>
> wrote:
> 
> > Well, this time it hit me on the USB backup drive which uses no
> > bcache and no other fancy options except compress-force=zlib.
> > Apparently, I've only got a (real) screenshot which I'm going to
> > link here:
> >
> > https://www.dropbox.com/s/9qbc7np23y8lrii/IMG_20160326_200033.jpg?dl=0  
> 
> This is a curious screen shot. It's a dracut pre-mount shell, so
> nothing should be mounted yet. And btrfs check only works on an
> unmounted file system. And yet the bottom part of the trace shows a
> Btrfs volume being made read only, as if it was mounted read write and
> is still mounted. Huh?

It's a pre-mount shell because I wanted to check the rootfs from there.
I mounted it once (and unmounted) before checking (that's
bcache{0,1,2}). Yeah, you can get there forcibly by using
rd.break=pre-mount - and yeah, nothing "should" be mounted unless I did
so previously. But I cut that away as it contained unrelated errors to
this problem and would be even more confusing.

The file system that failed then was the one I just mounted to put the
stdout of btrfsck on (sde1). That one showed these (screenshot) kernel
console logs just in the middle of typing the command - so a few
seconds after mounting.

What may be consusing to you: I use more than one btrfs. ;-)

bcache{0,1,2} = my rootfs (plus subvolumes)
sde = my USB3 backup drive (or whatever the kernel assigns)

Both run btrfs. The bcache*'s have their own problems currently, I'd
like to set those aside first and get the backup drive back in good
shape. The latter seems easier to fix.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Sat, 26 Mar 2016 15:04:13 -0600
schrieb Chris Murphy <lists@colorremedies.com>:

> On Sat, Mar 26, 2016 at 2:28 PM, Chris Murphy
> <lists@colorremedies.com> wrote:
> > On Sat, Mar 26, 2016 at 1:30 PM, Kai Krakow <hurikhan77@gmail.com>
> > wrote: 
> >> Well, this time it hit me on the USB backup drive which uses no
> >> bcache and no other fancy options except compress-force=zlib.
> >> Apparently, I've only got a (real) screenshot which I'm going to
> >> link here:
> >>
> >> https://www.dropbox.com/s/9qbc7np23y8lrii/IMG_20160326_200033.jpg?dl=0  
> >
> > This is a curious screen shot. It's a dracut pre-mount shell, so
> > nothing should be mounted yet. And btrfs check only works on an
> > unmounted file system. And yet the bottom part of the trace shows a
> > Btrfs volume being made read only, as if it was mounted read write
> > and is still mounted. Huh?  
> 
> Wait. You said no bcache, and yet in this screen shot it shows 'btrfs
> check /dev/bcache2 ...' right before the back trace.
> 
> This thread is confusing. You're talking about two different btrfs
> volumes intermixed, one uses bcache the other doesn't, yet they both
> have corruption. I think it's hardware related: bad cable bad ram bad
> power, something.

No it's not, it's tested. That system ran rock stable until somewhere
in the 4.4 kernel series (probably). It ran high loads without problems
(loadavg >50), it ran huge IO copies concurrently without problems, it
survived unintentional reboots without FS corruption, it ran
VirtualBox VMs without problems. And the system still runs almost
without problems: Except for the "object already exists" which forced
my rootfs RO, I did not even take note that the FS has corruptions:
Nothing in dmesg, everything fine. There's just VirtualBox crashing a
VM now, and I see csum errors in that very VDI file - even after
recovering the file from backup, it happens again and again. Qu
mentioned that this may be a follow-up of other corruption - and tada:
Yes, there are lots of them now (my last check was back in 4.1 or 4.2
series). But because I still can rsync all my important files, I'd like
to get my backup drive in sane state again first.

Both filesystems on this PC show similar corruption now - but they are
connected to completely different buses (SATA3 bcache + 3x SATA2
backing store bache{0,1,2}, and USB3 without bcache = sde), use
different compression (compress=lzo vs. compress-force=zlib), but
similar redundancy scheme (draid=0,mraid=1 vs. draid=single,mraid=dup).
A hardware problem would induce completely random errors on these
pathes.

Completely different hardware shows similar problems - but that system
is currently not available to me, and will stay there for a while
(it's a non-production installation at my workplace). Why would similar
errors show up here, if it'd be a hardware error of the first system?

Meanwhile, I conclude we can rule out bcache or hardware - three file
systems show similar errors:

1. bcache on Crucial MX100 SATA3, 3x SATA2 backing HDD
2. bcache on Samsung Evo 850 SATA2, 1x SATA1 backing HDD
3. 1x plain USB3 btrfs (no bcache)

Not even the SSD hardware is in common, just system configuration in
general (Gentoo kernel, rootfs on btrfs) and workload (I do lot's of
similar things on both machines).

I need to grab the errors for machine setup 2 - tho I can't do that
currently, that system is offline and will be for a while.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Sat, 26 Mar 2016 20:30:35 +0100
schrieb Kai Krakow <hurikhan77@gmail.com>:

> Am Wed, 23 Mar 2016 12:16:24 +0800
> schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:
> 
> > Kai Krakow wrote on 2016/03/22 19:48 +0100:  
> > > Am Tue, 22 Mar 2016 16:47:10 +0800
> > > schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:
> > >    
>  [...]  
> >  [...]    
>  [...]  
> > >
> > > Apparently, that system does not boot now due to errors in bcache
> > > b-tree. That being that, it may well be some bcache error and not
> > > btrfs' fault. Apparently I couldn't catch the output, I've been
> > > in a hurry. It said "write error" and had some backtrace. I will
> > > come to this back later.
> > >
> > > Let's go to the system I currently care about (that one with the
> > > always breaking VDI file):
> > >    
> >  [...]    
>  [...]  
> > >
> > > After the error occured?
> > >
> > > Yes, some text about the extent being compressed and btrfs repair
> > > doesn't currently handle that case (I tried --repair as I'm
> > > having a backup). I simply decided not to investigate that
> > > further at that point but delete and restore the affected file
> > > from backup. However, this is the message from dmesg (tho, I
> > > didn't catch the backtrace):
> > >
> > > btrfs_run_delayed_refs:2927: errno=-17 Object already exists    
> > 
> > That's nice, at least we have some clue.
> > 
> > It's almost sure, it's a bug either in btrfs kernel which doesn't
> > handle delayed refs well(low possibility), or, corrupted fs which
> > create something kernel can't handle(I bet that's the case).  
> 
> [kernel 4.5.0 gentoo, btrfs-progs 4.4.1]
> 
> Well, this time it hit me on the USB backup drive which uses no bcache
> and no other fancy options except compress-force=zlib. Apparently,
> I've only got a (real) screenshot which I'm going to link here:
> 
> https://www.dropbox.com/s/9qbc7np23y8lrii/IMG_20160326_200033.jpg?dl=0
> 
> The same drive has no problems except "bad metadata crossing stripe
> boundary" - but a lot of them. This drive was never converted, it was
> freshly generated several months ago.
> [...]

I finally got copy&paste data:

# before mounting let's check the FS:

$ sudo btrfsck /dev/disk/by-label/usb-backup 
Checking filesystem on /dev/disk/by-label/usb-backup
UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
checking extents
bad metadata [156041216, 156057600) crossing stripe boundary
bad metadata [181403648, 181420032) crossing stripe boundary
bad metadata [392167424, 392183808) crossing stripe boundary
bad metadata [783482880, 783499264) crossing stripe boundary
bad metadata [784924672, 784941056) crossing stripe boundary
bad metadata [130151612416, 130151628800) crossing stripe boundary
bad metadata [162826813440, 162826829824) crossing stripe boundary
bad metadata [162927083520, 162927099904) crossing stripe boundary
bad metadata [619740659712, 619740676096) crossing stripe boundary
bad metadata [619781947392, 619781963776) crossing stripe boundary
bad metadata [619795644416, 619795660800) crossing stripe boundary
bad metadata [619816091648, 619816108032) crossing stripe boundary
bad metadata [620011388928, 620011405312) crossing stripe boundary
bad metadata [890992459776, 890992476160) crossing stripe boundary
bad metadata [891022737408, 891022753792) crossing stripe boundary
bad metadata [891101773824, 891101790208) crossing stripe boundary
bad metadata [891301199872, 891301216256) crossing stripe boundary
bad metadata [1012219314176, 1012219330560) crossing stripe boundary
bad metadata [1017202409472, 1017202425856) crossing stripe boundary
bad metadata [1017365397504, 1017365413888) crossing stripe boundary
bad metadata [1020764422144, 1020764438528) crossing stripe boundary
bad metadata [1251103342592, 1251103358976) crossing stripe boundary
bad metadata [1251144695808, 1251144712192) crossing stripe boundary
bad metadata [1251147055104, 1251147071488) crossing stripe boundary
bad metadata [1259271225344, 1259271241728) crossing stripe boundary
bad metadata [1266223611904, 1266223628288) crossing stripe boundary
bad metadata [1304750063616, 1304750080000) crossing stripe boundary
bad metadata [1304790106112, 1304790122496) crossing stripe boundary
bad metadata [1304850792448, 1304850808832) crossing stripe boundary
bad metadata [1304869928960, 1304869945344) crossing stripe boundary
bad metadata [1305089540096, 1305089556480) crossing stripe boundary
bad metadata [1309561651200, 1309561667584) crossing stripe boundary
bad metadata [1309581443072, 1309581459456) crossing stripe boundary
bad metadata [1309583671296, 1309583687680) crossing stripe boundary
bad metadata [1309942808576, 1309942824960) crossing stripe boundary
bad metadata [1310050549760, 1310050566144) crossing stripe boundary
bad metadata [1313031585792, 1313031602176) crossing stripe boundary
bad metadata [1313232912384, 1313232928768) crossing stripe boundary
bad metadata [1555210764288, 1555210780672) crossing stripe boundary
bad metadata [1555395182592, 1555395198976) crossing stripe boundary
bad metadata [2050576744448, 2050576760832) crossing stripe boundary
bad metadata [2050803957760, 2050803974144) crossing stripe boundary
bad metadata [2050969108480, 2050969124864) crossing stripe boundary
checking free space tree cache and super generation don't match, space
cache will be invalidated checking fs roots
checking csums
checking root refs
found 1860217443214 bytes used err is 0
total csum bytes: 1805105116
total tree bytes: 11793776640
total fs tree bytes: 8220835840
total extent tree bytes: 1443315712
btree space waste bytes: 2307850845
file data blocks allocated: 2137151094784
 referenced 2706830905344

# now let's wait for the backup to mount the FS and look at dmesg:

[21375.606479] BTRFS info (device sde1): force zlib compression
[21375.606483] BTRFS info (device sde1): using free space tree
[21375.606485] BTRFS: has skinny extents
[21383.710725] BTRFS: checking UUID tree
[21388.531267] ------------[ cut here ]------------
[21388.531279] WARNING: CPU: 0 PID: 27085 at
fs/btrfs/extent-tree.c:2946 btrfs_run_delayed_refs+0x279/0x2b0()
[21388.531281] BTRFS: Transaction aborted (error -17) [21388.531282]
Modules linked in: nvidia_drm(PO) uas usb_storage vboxnetadp(O)
vboxnetflt(O) vboxdrv(O) nvidia_modeset(PO) nvidia(PO) [21388.531293]
CPU: 0 PID: 27085 Comm: kworker/u8:3 Tainted: P           O
4.5.0-gentoo #1 [21388.531295] Hardware name: To Be Filled By O.E.M. To
Be Filled By O.E.M./Z68 Pro3, BIOS L2.16A 02/22/2013 [21388.531300]
Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [21388.531302]
0000000000000000 ffffffff8159eaa9 ffff88029d723d68 ffffffff81ea1cf6
[21388.531306]  ffffffff810c6e37 ffff880407be5960 ffff88029d723db8
0000000000000020 [21388.531308]  ffff8802bd886510 ffff8803cc846d00
ffffffff810c6eb7 ffffffff81e8a088 [21388.531311] Call Trace:
[21388.531317]  [<ffffffff8159eaa9>] ? dump_stack+0x46/0x5d
[21388.531322]  [<ffffffff810c6e37>] ? warn_slowpath_common+0x77/0xb0
[21388.531326]  [<ffffffff810c6eb7>] ? warn_slowpath_fmt+0x47/0x50
[21388.531329]  [<ffffffff814923d9>] ?
btrfs_run_delayed_refs+0x279/0x2b0 [21388.531332]
[<ffffffff8149243d>] ? delayed_ref_async_start+0x2d/0x70
[21388.531337]  [<ffffffff810da8c3>] ? process_one_work+0x133/0x350
[21388.531340]  [<ffffffff810dae25>] ? worker_thread+0x45/0x450
[21388.531343]  [<ffffffff810dade0>] ? rescuer_thread+0x300/0x300
[21388.531347]  [<ffffffff810df3f8>] ? kthread+0xb8/0xd0
[21388.531350]  [<ffffffff810e301f>] ? finish_task_switch+0x6f/0x1f0
[21388.531352]  [<ffffffff810df340>] ? kthread_park+0x50/0x50
[21388.531356]  [<ffffffff81bdb31f>] ? ret_from_fork+0x3f/0x70
[21388.531359]  [<ffffffff810df340>] ? kthread_park+0x50/0x50
[21388.531361] ---[ end trace 69a4c78997ef63b2 ]--- [21388.531364]
BTRFS: error (device sde1) in btrfs_run_delayed_refs:2946: errno=-17
Object already exists [21388.531367] BTRFS info (device sde1): forced
readonly

This FS has been okay just a few reboots earlier (except the stripe
boundary thing).

History of kernel versions used on the system (back to the date when I
think the FS was still clean/not corrupted):

     Wed Aug  5 08:12:01 2015 >>> sys-kernel/gentoo-sources-4.1.4
     Sat Aug 15 19:04:34 2015 >>> sys-kernel/gentoo-sources-4.1.5
     Sat Aug 22 11:10:53 2015 >>> sys-kernel/gentoo-sources-4.1.6
     Tue Sep  1 00:03:38 2015 >>> sys-kernel/gentoo-sources-4.2.0
     Sat Sep  5 17:45:11 2015 >>> sys-kernel/gentoo-sources-4.2.0-r1
     Wed Sep 30 04:55:55 2015 >>> sys-kernel/gentoo-sources-4.2.2
     Tue Oct  6 22:55:19 2015 >>> sys-kernel/gentoo-sources-4.2.3
     Sun Oct 25 03:45:03 2015 >>> sys-kernel/gentoo-sources-4.2.4
     Wed Nov  4 04:21:38 2015 >>> sys-kernel/gentoo-sources-4.2.5
     Mon Nov  9 21:36:31 2015 >>> sys-kernel/gentoo-sources-4.3.0
     Sat Nov 14 10:48:03 2015 >>> sys-kernel/gentoo-sources-4.2.6
     Sun Dec 13 13:03:12 2015 >>> sys-kernel/gentoo-sources-4.2.7
     Tue Dec 15 22:37:53 2015 >>> sys-kernel/gentoo-sources-4.2.8
     Tue Dec 22 14:37:01 2015 >>> sys-kernel/gentoo-sources-4.3.3
     Fri Jan 22 20:38:32 2016 >>> sys-kernel/gentoo-sources-4.3.3-r1
     Wed Jan 27 09:27:57 2016 >>> sys-kernel/gentoo-sources-4.3.4
     Sun Jan 31 10:25:00 2016 >>> sys-kernel/gentoo-sources-4.4.0-r1
     Mon Feb  1 20:40:17 2016 >>> sys-kernel/gentoo-sources-4.4.1
     Fri Feb 19 22:11:45 2016 >>> sys-kernel/gentoo-sources-4.4.2
     Sat Feb 27 15:10:45 2016 >>> sys-kernel/gentoo-sources-4.4.3
     Sun Mar  6 14:58:05 2016 >>> sys-kernel/gentoo-sources-4.4.4
     Sat Mar 12 19:21:15 2016 >>> sys-kernel/gentoo-sources-4.4.5
     Sat Mar 19 06:13:09 2016 >>> sys-kernel/gentoo-sources-4.4.6
     Sun Mar 20 20:45:21 2016 >>> sys-kernel/gentoo-sources-4.5.0

I only saw unreliable behavior with 4.4.5, 4.4.6, and 4.5.0 tho the
problem may exist longer in my FS.

$ sudo btrfs-show-super /dev/sde1
superblock: bytenr=65536, device=/dev/sde1
---------------------------------------------------------
csum                    0xcc976d97 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    1318ec21-c421-4e36-a44a-7be3d41f9c3f
label                   usb-backup
generation              50814
root                    1251250159616
sys_array_size          129
chunk_root_generation   50784
root_level              1
chunk_root              2516518567936
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             2000397864960
bytes_used              1860398493696
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x1
incompat_flags          0x169
                        ( MIXED_BACKREF |
                          COMPRESS_LZO |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
csum_type               0
csum_size               4
cache_generation        50208
uuid_tree_generation    50742
dev_item.uuid           9008d5a0-ac7b-4505-8193-27428429f953
dev_item.fsid           1318ec21-c421-4e36-a44a-7be3d41f9c3f [match]
dev_item.type           0
dev_item.total_bytes    2000397864960
dev_item.bytes_used     1912308039680
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0



BTW: btrfsck thinks that the space tree is invalid every time it is
run, no matter if cleanly unmounted, uncleanly unmounted, or "btrfsck
--repair" and then ran a second time.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Chris Murphy]

On Sat, Mar 26, 2016 at 7:50 PM, Kai Krakow <hurikhan77@gmail.com> wrote:

>
> # now let's wait for the backup to mount the FS and look at dmesg:
>
> [21375.606479] BTRFS info (device sde1): force zlib compression
> [21375.606483] BTRFS info (device sde1): using free space tree

You're using space_cache=v2. You're aware new free space tree option
sets a read only incompat feature flag on the file system? You've got
quite a few non-default mount options on this backup volume. Hopefully
Qu has some idea what to try next or if you're better off just
starting over with a new file system.



> I only saw unreliable behavior with 4.4.5, 4.4.6, and 4.5.0 tho the
> problem may exist longer in my FS.
>
> $ sudo btrfs-show-super /dev/sde1
> superblock: bytenr=65536, device=/dev/sde1
> ---------------------------------------------------------
> csum                    0xcc976d97 [match]
> bytenr                  65536
> flags                   0x1
>                         ( WRITTEN )
> magic                   _BHRfS_M [match]
> fsid                    1318ec21-c421-4e36-a44a-7be3d41f9c3f
> label                   usb-backup
> generation              50814
> root                    1251250159616
> sys_array_size          129
> chunk_root_generation   50784
> root_level              1
> chunk_root              2516518567936
> chunk_root_level        1
> log_root                0
> log_root_transid        0
> log_root_level          0
> total_bytes             2000397864960
> bytes_used              1860398493696
> sectorsize              4096
> nodesize                16384
> leafsize                16384
> stripesize              4096
> root_dir                6
> num_devices             1
> compat_flags            0x0
> compat_ro_flags         0x1
> incompat_flags          0x169
>                         ( MIXED_BACKREF |
>                           COMPRESS_LZO |
>                           BIG_METADATA |
>                           EXTENDED_IREF |
>                           SKINNY_METADATA )
> csum_type               0
> csum_size               4
> cache_generation        50208
> uuid_tree_generation    50742
> dev_item.uuid           9008d5a0-ac7b-4505-8193-27428429f953
> dev_item.fsid           1318ec21-c421-4e36-a44a-7be3d41f9c3f [match]
> dev_item.type           0
> dev_item.total_bytes    2000397864960
> dev_item.bytes_used     1912308039680
> dev_item.io_align       4096
> dev_item.io_width       4096
> dev_item.sector_size    4096
> dev_item.devid          1
> dev_item.dev_group      0
> dev_item.seek_speed     0
> dev_item.bandwidth      0
> dev_item.generation     0
>
>
>
> BTW: btrfsck thinks that the space tree is invalid every time it is
> run, no matter if cleanly unmounted, uncleanly unmounted, or "btrfsck
> --repair" and then ran a second time.


-- 
Chris Murphy

Re: csum errors in VirtualBox VDI files

[Chris Murphy]

On Sat, Mar 26, 2016 at 7:30 PM, Kai Krakow <hurikhan77@gmail.com> wrote:

> Both filesystems on this PC show similar corruption now - but they are
> connected to completely different buses (SATA3 bcache + 3x SATA2
> backing store bache{0,1,2}, and USB3 without bcache = sde), use
> different compression (compress=lzo vs. compress-force=zlib), but
> similar redundancy scheme (draid=0,mraid=1 vs. draid=single,mraid=dup).
> A hardware problem would induce completely random errors on these
> pathes.
>
> Completely different hardware shows similar problems - but that system
> is currently not available to me, and will stay there for a while
> (it's a non-production installation at my workplace). Why would similar
> errors show up here, if it'd be a hardware error of the first system?

Then there's something about the particular combination of mount
options you're using with the workload that's inducing this, if it's
reproducing on two different systems. What's the workload and what's
the full history of the mount options? Looks like it started life as
compress lzo and then later compress-force zlib and then after that
the addition of space_cache=v2?

Hopefully Qu has some advice on what's next. It might not be a bad
idea to get a btrfs-image going.



-- 
Chris Murphy

Re: csum errors in VirtualBox VDI files

[Martin Steigerwald]

On Dienstag, 22. März 2016 09:03:42 CEST Kai Krakow wrote:
> Hello!
> 
> Since one of the last kernel updates (I don't know which exactly), I'm
> experiencing csum errors within VDI files when running VirtualBox. A
> side effect of this is, as soon as dmesg shows these errors, commands
> like "du" and "df" hang until reboot.
> 
> I've now restored the file from backup but it happens over and over
> again.

Just as another data point I am irregularily using a VM with Virtualbox in a 
VDI file on a BTRFS RAID 1 on two SSDs and had no such issues so far up to 
kernel 4.5.

Thanks,
-- 
Martin

Re: csum errors in VirtualBox VDI files

[Qu Wenruo]

On 03/27/2016 03:30 AM, Kai Krakow wrote:
> Am Wed, 23 Mar 2016 12:16:24 +0800
> schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:
>
>> Kai Krakow wrote on 2016/03/22 19:48 +0100:
>>> Am Tue, 22 Mar 2016 16:47:10 +0800
>>> schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:
>>>
>>>> Hi,
>>>>
>>>> Kai Krakow wrote on 2016/03/22 09:03 +0100:
>>   [...]
>>>>
>>>> When it goes RO, it must have some warning in kernel log.
>>>> Would you please paste the kernel log?
>>>
>>> Apparently, that system does not boot now due to errors in bcache
>>> b-tree. That being that, it may well be some bcache error and not
>>> btrfs' fault. Apparently I couldn't catch the output, I've been in a
>>> hurry. It said "write error" and had some backtrace. I will come to
>>> this back later.
>>>
>>> Let's go to the system I currently care about (that one with the
>>> always breaking VDI file):
>>>
>>   [...]
>>>> Does btrfs check report anything wrong?
>>>
>>> After the error occured?
>>>
>>> Yes, some text about the extent being compressed and btrfs repair
>>> doesn't currently handle that case (I tried --repair as I'm having a
>>> backup). I simply decided not to investigate that further at that
>>> point but delete and restore the affected file from backup.
>>> However, this is the message from dmesg (tho, I didn't catch the
>>> backtrace):
>>>
>>> btrfs_run_delayed_refs:2927: errno=-17 Object already exists
>>
>> That's nice, at least we have some clue.
>>
>> It's almost sure, it's a bug either in btrfs kernel which doesn't
>> handle delayed refs well(low possibility), or, corrupted fs which
>> create something kernel can't handle(I bet that's the case).
>
> [kernel 4.5.0 gentoo, btrfs-progs 4.4.1]
>
> Well, this time it hit me on the USB backup drive which uses no bcache
> and no other fancy options except compress-force=zlib. Apparently, I've
> only got a (real) screenshot which I'm going to link here:
>
> https://www.dropbox.com/s/9qbc7np23y8lrii/IMG_20160326_200033.jpg?dl=0

Nothing new.
The needed thing is not the warning/error part, but the info part.

Which will output the extent tree leaf with what run_delayed_refs is 
going to do.

>
> The same drive has no problems except "bad metadata crossing stripe
> boundary" - but a lot of them. This drive was never converted, it was
> freshly generated several months ago.
>
> ---8<---
> $ sudo btrfsck /dev/disk/by-label/usb-backup
> Checking filesystem on /dev/disk/by-label/usb-backup
> UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
> checking extents
> bad metadata [156041216, 156057600) crossing stripe boundary
> bad metadata [181403648, 181420032) crossing stripe boundary
> bad metadata [392167424, 392183808) crossing stripe boundary
> bad metadata [783482880, 783499264) crossing stripe boundary
> bad metadata [784924672, 784941056) crossing stripe boundary
> bad metadata [130151612416, 130151628800) crossing stripe boundary
> bad metadata [162826813440, 162826829824) crossing stripe boundary
> bad metadata [162927083520, 162927099904) crossing stripe boundary
> bad metadata [619740659712, 619740676096) crossing stripe boundary
> bad metadata [619781947392, 619781963776) crossing stripe boundary
> bad metadata [619795644416, 619795660800) crossing stripe boundary
> bad metadata [619816091648, 619816108032) crossing stripe boundary
> bad metadata [620011388928, 620011405312) crossing stripe boundary
> bad metadata [890992459776, 890992476160) crossing stripe boundary
> bad metadata [891022737408, 891022753792) crossing stripe boundary
> bad metadata [891101773824, 891101790208) crossing stripe boundary
> bad metadata [891301199872, 891301216256) crossing stripe boundary
> [...]
> --->8---

Normally false alert, just old btrfs-progs.
Or your fs is converted from ext*.

Update to latest btrfs-progs to see what it output now.
>
> My main drive (which this thread was about) has a huge amount of
> different problems according to btrfsck. Repair doesn't work:

Don't use --repair until you know the meaning of the error.

I just found your full fsck output, and will comment there.

Thanks,
Qu

> it says
> something about overlapping extents and that it needs a careful
> thought. I wanted to catch the output when the above problem occured. So
> I'd like to defer that until later and first fix my backup drive. If I
> lose my main drive, I simply restore from backup. It is very old anyway
> (still using 4k node size). Only downside it takes 24+ hours to restore.
>

Re: csum errors in VirtualBox VDI files

[Qu Wenruo]

On 03/27/2016 09:50 AM, Kai Krakow wrote:
> Am Sat, 26 Mar 2016 20:30:35 +0100
> schrieb Kai Krakow <hurikhan77@gmail.com>:
>
>> Am Wed, 23 Mar 2016 12:16:24 +0800
>> schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:
>>
>>> Kai Krakow wrote on 2016/03/22 19:48 +0100:
>>>> Am Tue, 22 Mar 2016 16:47:10 +0800
>>>> schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:
>>>>
>>   [...]
>>>   [...]
>>   [...]
>>>>
>>>> Apparently, that system does not boot now due to errors in bcache
>>>> b-tree. That being that, it may well be some bcache error and not
>>>> btrfs' fault. Apparently I couldn't catch the output, I've been
>>>> in a hurry. It said "write error" and had some backtrace. I will
>>>> come to this back later.
>>>>
>>>> Let's go to the system I currently care about (that one with the
>>>> always breaking VDI file):
>>>>
>>>   [...]
>>   [...]
>>>>
>>>> After the error occured?
>>>>
>>>> Yes, some text about the extent being compressed and btrfs repair
>>>> doesn't currently handle that case (I tried --repair as I'm
>>>> having a backup). I simply decided not to investigate that
>>>> further at that point but delete and restore the affected file
>>>> from backup. However, this is the message from dmesg (tho, I
>>>> didn't catch the backtrace):
>>>>
>>>> btrfs_run_delayed_refs:2927: errno=-17 Object already exists
>>>
>>> That's nice, at least we have some clue.
>>>
>>> It's almost sure, it's a bug either in btrfs kernel which doesn't
>>> handle delayed refs well(low possibility), or, corrupted fs which
>>> create something kernel can't handle(I bet that's the case).
>>
>> [kernel 4.5.0 gentoo, btrfs-progs 4.4.1]
>>
>> Well, this time it hit me on the USB backup drive which uses no bcache
>> and no other fancy options except compress-force=zlib. Apparently,
>> I've only got a (real) screenshot which I'm going to link here:
>>
>> https://www.dropbox.com/s/9qbc7np23y8lrii/IMG_20160326_200033.jpg?dl=0
>>
>> The same drive has no problems except "bad metadata crossing stripe
>> boundary" - but a lot of them. This drive was never converted, it was
>> freshly generated several months ago.
>> [...]
>
> I finally got copy&paste data:
>
> # before mounting let's check the FS:
>
> $ sudo btrfsck /dev/disk/by-label/usb-backup
> Checking filesystem on /dev/disk/by-label/usb-backup
> UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
> checking extents
> bad metadata [156041216, 156057600) crossing stripe boundary
> bad metadata [181403648, 181420032) crossing stripe boundary
> bad metadata [392167424, 392183808) crossing stripe boundary
> bad metadata [783482880, 783499264) crossing stripe boundary
> bad metadata [784924672, 784941056) crossing stripe boundary
> bad metadata [130151612416, 130151628800) crossing stripe boundary
> bad metadata [162826813440, 162826829824) crossing stripe boundary
> bad metadata [162927083520, 162927099904) crossing stripe boundary
> bad metadata [619740659712, 619740676096) crossing stripe boundary
> bad metadata [619781947392, 619781963776) crossing stripe boundary
> bad metadata [619795644416, 619795660800) crossing stripe boundary
> bad metadata [619816091648, 619816108032) crossing stripe boundary
> bad metadata [620011388928, 620011405312) crossing stripe boundary
> bad metadata [890992459776, 890992476160) crossing stripe boundary
> bad metadata [891022737408, 891022753792) crossing stripe boundary
> bad metadata [891101773824, 891101790208) crossing stripe boundary
> bad metadata [891301199872, 891301216256) crossing stripe boundary
> bad metadata [1012219314176, 1012219330560) crossing stripe boundary
> bad metadata [1017202409472, 1017202425856) crossing stripe boundary
> bad metadata [1017365397504, 1017365413888) crossing stripe boundary
> bad metadata [1020764422144, 1020764438528) crossing stripe boundary
> bad metadata [1251103342592, 1251103358976) crossing stripe boundary
> bad metadata [1251144695808, 1251144712192) crossing stripe boundary
> bad metadata [1251147055104, 1251147071488) crossing stripe boundary
> bad metadata [1259271225344, 1259271241728) crossing stripe boundary
> bad metadata [1266223611904, 1266223628288) crossing stripe boundary
> bad metadata [1304750063616, 1304750080000) crossing stripe boundary
> bad metadata [1304790106112, 1304790122496) crossing stripe boundary
> bad metadata [1304850792448, 1304850808832) crossing stripe boundary
> bad metadata [1304869928960, 1304869945344) crossing stripe boundary
> bad metadata [1305089540096, 1305089556480) crossing stripe boundary
> bad metadata [1309561651200, 1309561667584) crossing stripe boundary
> bad metadata [1309581443072, 1309581459456) crossing stripe boundary
> bad metadata [1309583671296, 1309583687680) crossing stripe boundary
> bad metadata [1309942808576, 1309942824960) crossing stripe boundary
> bad metadata [1310050549760, 1310050566144) crossing stripe boundary
> bad metadata [1313031585792, 1313031602176) crossing stripe boundary
> bad metadata [1313232912384, 1313232928768) crossing stripe boundary
> bad metadata [1555210764288, 1555210780672) crossing stripe boundary
> bad metadata [1555395182592, 1555395198976) crossing stripe boundary
> bad metadata [2050576744448, 2050576760832) crossing stripe boundary
> bad metadata [2050803957760, 2050803974144) crossing stripe boundary
> bad metadata [2050969108480, 2050969124864) crossing stripe boundary

Already mentioned in another reply, this *seems* to be false alert.
Latest btrfs-progs would help.

> checking free space tree cache and super generation don't match, space
> cache will be invalidated checking fs roots
Err, I found a missing '\n' before "checking fs roots".

And it seems that fs roots and extent tree are all OK.

Quite surprising.
The only possible problem seems to be outdated space cache.

Maybe mount with "-o clear_cache" will help, but I don't think that's 
the cause.


> checking csums
> checking root refs
> found 1860217443214 bytes used err is 0
> total csum bytes: 1805105116
> total tree bytes: 11793776640
> total fs tree bytes: 8220835840
> total extent tree bytes: 1443315712
> btree space waste bytes: 2307850845
> file data blocks allocated: 2137151094784
>   referenced 2706830905344
>
> # now let's wait for the backup to mount the FS and look at dmesg:
>
> [21375.606479] BTRFS info (device sde1): force zlib compression
> [21375.606483] BTRFS info (device sde1): using free space tree

Thanks to Chris Murphy, I almost ignored such line.
Not familiar with the new free space cache tree, sorry.

If "clear_cache" doesn't help, then try "nospace_cache" to disable the 
entire space cache.


> [21375.606485] BTRFS: has skinny extents
> [21383.710725] BTRFS: checking UUID tree
> [21388.531267] ------------[ cut here ]------------
> [21388.531279] WARNING: CPU: 0 PID: 27085 at
> fs/btrfs/extent-tree.c:2946 btrfs_run_delayed_refs+0x279/0x2b0()
> [21388.531281] BTRFS: Transaction aborted (error -17) [21388.531282]
> Modules linked in: nvidia_drm(PO) uas usb_storage vboxnetadp(O)
> vboxnetflt(O) vboxdrv(O) nvidia_modeset(PO) nvidia(PO) [21388.531293]
> CPU: 0 PID: 27085 Comm: kworker/u8:3 Tainted: P           O
> 4.5.0-gentoo #1 [21388.531295] Hardware name: To Be Filled By O.E.M. To
> Be Filled By O.E.M./Z68 Pro3, BIOS L2.16A 02/22/2013 [21388.531300]
> Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [21388.531302]
> 0000000000000000 ffffffff8159eaa9 ffff88029d723d68 ffffffff81ea1cf6
> [21388.531306]  ffffffff810c6e37 ffff880407be5960 ffff88029d723db8
> 0000000000000020 [21388.531308]  ffff8802bd886510 ffff8803cc846d00
> ffffffff810c6eb7 ffffffff81e8a088 [21388.531311] Call Trace:
> [21388.531317]  [<ffffffff8159eaa9>] ? dump_stack+0x46/0x5d
> [21388.531322]  [<ffffffff810c6e37>] ? warn_slowpath_common+0x77/0xb0
> [21388.531326]  [<ffffffff810c6eb7>] ? warn_slowpath_fmt+0x47/0x50
> [21388.531329]  [<ffffffff814923d9>] ?
> btrfs_run_delayed_refs+0x279/0x2b0 [21388.531332]
> [<ffffffff8149243d>] ? delayed_ref_async_start+0x2d/0x70
> [21388.531337]  [<ffffffff810da8c3>] ? process_one_work+0x133/0x350
> [21388.531340]  [<ffffffff810dae25>] ? worker_thread+0x45/0x450
> [21388.531343]  [<ffffffff810dade0>] ? rescuer_thread+0x300/0x300
> [21388.531347]  [<ffffffff810df3f8>] ? kthread+0xb8/0xd0
> [21388.531350]  [<ffffffff810e301f>] ? finish_task_switch+0x6f/0x1f0
> [21388.531352]  [<ffffffff810df340>] ? kthread_park+0x50/0x50
> [21388.531356]  [<ffffffff81bdb31f>] ? ret_from_fork+0x3f/0x70
> [21388.531359]  [<ffffffff810df340>] ? kthread_park+0x50/0x50
> [21388.531361] ---[ end trace 69a4c78997ef63b2 ]--- [21388.531364]
> BTRFS: error (device sde1) in btrfs_run_delayed_refs:2946: errno=-17
> Object already exists [21388.531367] BTRFS info (device sde1): forced
> readonly
>
> This FS has been okay just a few reboots earlier (except the stripe
> boundary thing).
>
> History of kernel versions used on the system (back to the date when I
> think the FS was still clean/not corrupted):
>
>       Wed Aug  5 08:12:01 2015 >>> sys-kernel/gentoo-sources-4.1.4
>       Sat Aug 15 19:04:34 2015 >>> sys-kernel/gentoo-sources-4.1.5
>       Sat Aug 22 11:10:53 2015 >>> sys-kernel/gentoo-sources-4.1.6
>       Tue Sep  1 00:03:38 2015 >>> sys-kernel/gentoo-sources-4.2.0
>       Sat Sep  5 17:45:11 2015 >>> sys-kernel/gentoo-sources-4.2.0-r1
>       Wed Sep 30 04:55:55 2015 >>> sys-kernel/gentoo-sources-4.2.2
>       Tue Oct  6 22:55:19 2015 >>> sys-kernel/gentoo-sources-4.2.3
>       Sun Oct 25 03:45:03 2015 >>> sys-kernel/gentoo-sources-4.2.4
>       Wed Nov  4 04:21:38 2015 >>> sys-kernel/gentoo-sources-4.2.5
>       Mon Nov  9 21:36:31 2015 >>> sys-kernel/gentoo-sources-4.3.0
>       Sat Nov 14 10:48:03 2015 >>> sys-kernel/gentoo-sources-4.2.6
>       Sun Dec 13 13:03:12 2015 >>> sys-kernel/gentoo-sources-4.2.7
>       Tue Dec 15 22:37:53 2015 >>> sys-kernel/gentoo-sources-4.2.8
>       Tue Dec 22 14:37:01 2015 >>> sys-kernel/gentoo-sources-4.3.3
>       Fri Jan 22 20:38:32 2016 >>> sys-kernel/gentoo-sources-4.3.3-r1
>       Wed Jan 27 09:27:57 2016 >>> sys-kernel/gentoo-sources-4.3.4
>       Sun Jan 31 10:25:00 2016 >>> sys-kernel/gentoo-sources-4.4.0-r1
>       Mon Feb  1 20:40:17 2016 >>> sys-kernel/gentoo-sources-4.4.1
>       Fri Feb 19 22:11:45 2016 >>> sys-kernel/gentoo-sources-4.4.2
>       Sat Feb 27 15:10:45 2016 >>> sys-kernel/gentoo-sources-4.4.3
>       Sun Mar  6 14:58:05 2016 >>> sys-kernel/gentoo-sources-4.4.4
>       Sat Mar 12 19:21:15 2016 >>> sys-kernel/gentoo-sources-4.4.5
>       Sat Mar 19 06:13:09 2016 >>> sys-kernel/gentoo-sources-4.4.6
>       Sun Mar 20 20:45:21 2016 >>> sys-kernel/gentoo-sources-4.5.0
>
> I only saw unreliable behavior with 4.4.5, 4.4.6, and 4.5.0 tho the
> problem may exist longer in my FS.
>
> $ sudo btrfs-show-super /dev/sde1
> superblock: bytenr=65536, device=/dev/sde1
> ---------------------------------------------------------
> csum                    0xcc976d97 [match]
> bytenr                  65536
> flags                   0x1
>                          ( WRITTEN )
> magic                   _BHRfS_M [match]
> fsid                    1318ec21-c421-4e36-a44a-7be3d41f9c3f
> label                   usb-backup
> generation              50814
> root                    1251250159616
> sys_array_size          129
> chunk_root_generation   50784
> root_level              1
> chunk_root              2516518567936
> chunk_root_level        1
> log_root                0
> log_root_transid        0
> log_root_level          0
> total_bytes             2000397864960
> bytes_used              1860398493696
> sectorsize              4096
> nodesize                16384
> leafsize                16384
> stripesize              4096
> root_dir                6
> num_devices             1
> compat_flags            0x0
> compat_ro_flags         0x1

OK, you're using the new free space cache tree option, update 
btrfs-progs to latest version will help further debugging.

> incompat_flags          0x169
>                          ( MIXED_BACKREF |
>                            COMPRESS_LZO |
>                            BIG_METADATA |
>                            EXTENDED_IREF |
>                            SKINNY_METADATA )
> csum_type               0
> csum_size               4
> cache_generation        50208
> uuid_tree_generation    50742
> dev_item.uuid           9008d5a0-ac7b-4505-8193-27428429f953
> dev_item.fsid           1318ec21-c421-4e36-a44a-7be3d41f9c3f [match]
> dev_item.type           0
> dev_item.total_bytes    2000397864960
> dev_item.bytes_used     1912308039680
> dev_item.io_align       4096
> dev_item.io_width       4096
> dev_item.sector_size    4096
> dev_item.devid          1
> dev_item.dev_group      0
> dev_item.seek_speed     0
> dev_item.bandwidth      0
> dev_item.generation     0
>
>
>
> BTW: btrfsck thinks that the space tree is invalid every time it is
> run, no matter if cleanly unmounted, uncleanly unmounted, or "btrfsck
> --repair" and then ran a second time.
>
Because your btrfs-progs is too old to understand the new free space 
cache tree.

Thanks,
Qu

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Sun, 27 Mar 2016 14:18:26 +0200
schrieb Martin Steigerwald <martin@lichtvoll.de>:

> On Dienstag, 22. März 2016 09:03:42 CEST Kai Krakow wrote:
> > Hello!
> > 
> > Since one of the last kernel updates (I don't know which exactly),
> > I'm experiencing csum errors within VDI files when running
> > VirtualBox. A side effect of this is, as soon as dmesg shows these
> > errors, commands like "du" and "df" hang until reboot.
> > 
> > I've now restored the file from backup but it happens over and over
> > again.  
> 
> Just as another data point I am irregularily using a VM with
> Virtualbox in a VDI file on a BTRFS RAID 1 on two SSDs and had no
> such issues so far up to kernel 4.5.

Meanwhile I think this is a side effect of some other corruption in the
filesystem.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Sat, 26 Mar 2016 22:57:53 -0600
schrieb Chris Murphy <lists@colorremedies.com>:

> On Sat, Mar 26, 2016 at 7:30 PM, Kai Krakow <hurikhan77@gmail.com>
> wrote:
> 
> > Both filesystems on this PC show similar corruption now - but they
> > are connected to completely different buses (SATA3 bcache + 3x SATA2
> > backing store bache{0,1,2}, and USB3 without bcache = sde), use
> > different compression (compress=lzo vs. compress-force=zlib), but
> > similar redundancy scheme (draid=0,mraid=1 vs.
> > draid=single,mraid=dup). A hardware problem would induce completely
> > random errors on these pathes.
> >
> > Completely different hardware shows similar problems - but that
> > system is currently not available to me, and will stay there for a
> > while (it's a non-production installation at my workplace). Why
> > would similar errors show up here, if it'd be a hardware error of
> > the first system?  
> 
> Then there's something about the particular combination of mount
> options you're using with the workload that's inducing this, if it's
> reproducing on two different systems. What's the workload and what's
> the full history of the mount options? Looks like it started life as
> compress lzo and then later compress-force zlib and then after that
> the addition of space_cache=v2?

Still, that's two (or three) different filesystems:

The first (my main system) had compress=lzo forever, I never used
compress-force or something different than lzo.

The second (my main system backup) had compress-force=zlib forever,
never used a different compression option.

The third (the currently offline system) had compress=lzo like the
first one. It has no backup, system can be rebuild from scratch, no
important data there. I don't bother about that currently.

> Hopefully Qu has some advice on what's next. It might not be a bad
> idea to get a btrfs-image going.

I first upgraded to btrfs-progs 4.5 and removed the space_cache=v2
option (space tree has been removed, ro-incompat flag was reset
according to dmesg). I only activated that to see if it changes things,
and I made sure beforehand that this can be removed. Looks like that
works as documented.

I'll come back to the other thread as soon as I've run the check. It
takes a while (it contains a few weeks worth of snapshots). Meanwhile
I'll see if the main fs looks any different with btrfs-progs 4.5. I
need to get into dracut pre-mount for that.

At least, with space_cache=v2 removed, the delayed_refs problem there
is gone - so that code obviously has problems.

The main system didn't use space_cache=v2, tho, when the "object
already exists" problem hit me first.

I'll prepare for btrfs-image. How big is that going to be? I'd prefer
to make it as sparse as possible. I could hook it up to a 100mbit
upload but I need to get storage for that first.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Chris Murphy]

On Sun, Mar 27, 2016 at 11:31 AM, Kai Krakow <hurikhan77@gmail.com> wrote:
> Am Sat, 26 Mar 2016 22:57:53 -0600
> schrieb Chris Murphy <lists@colorremedies.com>:
>
>> On Sat, Mar 26, 2016 at 7:30 PM, Kai Krakow <hurikhan77@gmail.com>
>> wrote:
>>
>> > Both filesystems on this PC show similar corruption now - but they
>> > are connected to completely different buses (SATA3 bcache + 3x SATA2
>> > backing store bache{0,1,2}, and USB3 without bcache = sde), use
>> > different compression (compress=lzo vs. compress-force=zlib), but
>> > similar redundancy scheme (draid=0,mraid=1 vs.
>> > draid=single,mraid=dup). A hardware problem would induce completely
>> > random errors on these pathes.
>> >
>> > Completely different hardware shows similar problems - but that
>> > system is currently not available to me, and will stay there for a
>> > while (it's a non-production installation at my workplace). Why
>> > would similar errors show up here, if it'd be a hardware error of
>> > the first system?
>>
>> Then there's something about the particular combination of mount
>> options you're using with the workload that's inducing this, if it's
>> reproducing on two different systems. What's the workload and what's
>> the full history of the mount options? Looks like it started life as
>> compress lzo and then later compress-force zlib and then after that
>> the addition of space_cache=v2?
>
> Still, that's two (or three) different filesystems:
>
> The first (my main system) had compress=lzo forever, I never used
> compress-force or something different than lzo.
>
> The second (my main system backup) had compress-force=zlib forever,
> never used a different compression option.
>
> The third (the currently offline system) had compress=lzo like the
> first one. It has no backup, system can be rebuild from scratch, no
> important data there. I don't bother about that currently.
>
>> Hopefully Qu has some advice on what's next. It might not be a bad
>> idea to get a btrfs-image going.
>
> I first upgraded to btrfs-progs 4.5 and removed the space_cache=v2
> option (space tree has been removed, ro-incompat flag was reset
> according to dmesg). I only activated that to see if it changes things,
> and I made sure beforehand that this can be removed. Looks like that
> works as documented.
>
> I'll come back to the other thread as soon as I've run the check. It
> takes a while (it contains a few weeks worth of snapshots). Meanwhile
> I'll see if the main fs looks any different with btrfs-progs 4.5. I
> need to get into dracut pre-mount for that.
>
> At least, with space_cache=v2 removed, the delayed_refs problem there
> is gone - so that code obviously has problems.
>
> The main system didn't use space_cache=v2, tho, when the "object
> already exists" problem hit me first.
>
> I'll prepare for btrfs-image. How big is that going to be? I'd prefer
> to make it as sparse as possible. I could hook it up to a 100mbit
> upload but I need to get storage for that first.

Before you go to the trouble of uploading btrfs-image, see if 'btrfs
check' with progs 4.5 clears up the noisy messages that Qu thinks are
false alarms.

As for the csum errors with this one single VDI file, you're going to
have to come up with a way to reproduce it consistently. You'll need
to have a good copy on a filesystem that comes up clean with btrfs
check and scrub. And then reproduce the corruption somehow. One hint
based on the other two users with similar setups or workload is they
aren't using the discard mount option and you are. I'd say unless you
have a newer SSD that supports queued trim, it probably shouldn't be
used, it's known to cause the kinds of hangs you report with drives
that only support non-queued trim. Those drives are better off getting
fstrim e.g. once a week on a timer.

And something that I find annoying is when you say in the first post
"results in a write error and the file system goes read-only" you're
asked by one of the developers to provide the kernel log, and then you
paste only some csum errors not kernel log. And not even the write
error. I think you need to collect more data, this whole thread is
like throwing spaghetti at a wall or throwing darts in a pitch black
room. You need to do more testing to find out what the pattern is,
when it happens and doesn't happen. The traces you're posting don't
help. And they're also incomplete. That's my two cents.


-- 
Chris Murphy

bad metadata crossing stripe boundary (was: csum errors in VirtualBox VDI files)

[Kai Krakow]

Changing subject to reflect the current topic...

Am Sun, 27 Mar 2016 21:55:40 +0800
schrieb Qu Wenruo <quwenruo.btrfs@gmx.com>:

> > I finally got copy&paste data:
> >
> > # before mounting let's check the FS:
> >
> > $ sudo btrfsck /dev/disk/by-label/usb-backup
> > Checking filesystem on /dev/disk/by-label/usb-backup
> > UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
> > checking extents
> > bad metadata [156041216, 156057600) crossing stripe boundary
> > bad metadata [181403648, 181420032) crossing stripe boundary
> > bad metadata [392167424, 392183808) crossing stripe boundary
> > bad metadata [783482880, 783499264) crossing stripe boundary
> > bad metadata [784924672, 784941056) crossing stripe boundary
> > bad metadata [130151612416, 130151628800) crossing stripe boundary
> > bad metadata [162826813440, 162826829824) crossing stripe boundary
> > bad metadata [162927083520, 162927099904) crossing stripe boundary
> > bad metadata [619740659712, 619740676096) crossing stripe boundary
> > bad metadata [619781947392, 619781963776) crossing stripe boundary
> > bad metadata [619795644416, 619795660800) crossing stripe boundary
> > bad metadata [619816091648, 619816108032) crossing stripe boundary
> > bad metadata [620011388928, 620011405312) crossing stripe boundary
> > bad metadata [890992459776, 890992476160) crossing stripe boundary
> > bad metadata [891022737408, 891022753792) crossing stripe boundary
> > bad metadata [891101773824, 891101790208) crossing stripe boundary
> > bad metadata [891301199872, 891301216256) crossing stripe boundary
> > bad metadata [1012219314176, 1012219330560) crossing stripe boundary
> > bad metadata [1017202409472, 1017202425856) crossing stripe boundary
> > bad metadata [1017365397504, 1017365413888) crossing stripe boundary
> > bad metadata [1020764422144, 1020764438528) crossing stripe boundary
> > bad metadata [1251103342592, 1251103358976) crossing stripe boundary
> > bad metadata [1251144695808, 1251144712192) crossing stripe boundary
> > bad metadata [1251147055104, 1251147071488) crossing stripe boundary
> > bad metadata [1259271225344, 1259271241728) crossing stripe boundary
> > bad metadata [1266223611904, 1266223628288) crossing stripe boundary
> > bad metadata [1304750063616, 1304750080000) crossing stripe boundary
> > bad metadata [1304790106112, 1304790122496) crossing stripe boundary
> > bad metadata [1304850792448, 1304850808832) crossing stripe boundary
> > bad metadata [1304869928960, 1304869945344) crossing stripe boundary
> > bad metadata [1305089540096, 1305089556480) crossing stripe boundary
> > bad metadata [1309561651200, 1309561667584) crossing stripe boundary
> > bad metadata [1309581443072, 1309581459456) crossing stripe boundary
> > bad metadata [1309583671296, 1309583687680) crossing stripe boundary
> > bad metadata [1309942808576, 1309942824960) crossing stripe boundary
> > bad metadata [1310050549760, 1310050566144) crossing stripe boundary
> > bad metadata [1313031585792, 1313031602176) crossing stripe boundary
> > bad metadata [1313232912384, 1313232928768) crossing stripe boundary
> > bad metadata [1555210764288, 1555210780672) crossing stripe boundary
> > bad metadata [1555395182592, 1555395198976) crossing stripe boundary
> > bad metadata [2050576744448, 2050576760832) crossing stripe boundary
> > bad metadata [2050803957760, 2050803974144) crossing stripe boundary
> > bad metadata [2050969108480, 2050969124864) crossing stripe
> > boundary  
> 
> Already mentioned in another reply, this *seems* to be false alert.
> Latest btrfs-progs would help.

No, btrfs-progs 4.5 reports those, too (as far as I understood, this
includes the fixes for bogus "bad metadata" errors, tho I thought this
has already been fixed in 4.2.1, I used 4.4.1). There were some nbytes
wrong errors before which I already repaired using "--repair". I think
that's okay, I had those in the past and it looks like btrfsck can
repair those now (and I don't have to delete and recreate the files).
It caused problems with "du" and "df" in the past, a problem that I'm
currently facing too. So I better fixed them.

With that done, the backup fs now only reports "bad metadata" which
have been there before space cache v2. Full output below.

> > checking free space tree cache and super generation don't match,
> > space cache will be invalidated checking fs roots  
> Err, I found a missing '\n' before "checking fs roots".

Copy and paste problem. Claws mail pretends to be smarter than me
- I missed to fix that one. ;-)

> And it seems that fs roots and extent tree are all OK.
> 
> Quite surprising.
> The only possible problem seems to be outdated space cache.
> 
> Maybe mount with "-o clear_cache" will help, but I don't think that's 
> the cause.

Helped, it automatically reverted the FS back to space cache v1 with
incompat flag cleared. (I wouldn't have enabled v2 if it wasn't
documented that this is possible)

> > checking csums
> > checking root refs
> > found 1860217443214 bytes used err is 0
> > total csum bytes: 1805105116
> > total tree bytes: 11793776640
> > total fs tree bytes: 8220835840
> > total extent tree bytes: 1443315712
> > btree space waste bytes: 2307850845
> > file data blocks allocated: 2137151094784
> >   referenced 2706830905344
> >
> > # now let's wait for the backup to mount the FS and look at dmesg:
> >
> > [21375.606479] BTRFS info (device sde1): force zlib compression
> > [21375.606483] BTRFS info (device sde1): using free space tree  
> 
> Thanks to Chris Murphy, I almost ignored such line.
> Not familiar with the new free space cache tree, sorry.
> 
> If "clear_cache" doesn't help, then try "nospace_cache" to disable
> the entire space cache.

It's gone now, ignore that. It's back to the situation before space
cache v2. Minus some "nbytes wrong" errors I had and fixed.

Nevertheless, I'm now using btrfs-progs 4.5. Here's the full output:
(the lines seem to be partly out of order, probably due to the
redirection)

$ sudo btrfsck /dev/sde1 2>&1 | tee btrfsck-label-usb-backup.txt
checking extents
bad metadata [156041216, 156057600) crossing stripe boundary
bad metadata [181403648, 181420032) crossing stripe boundary
bad metadata [392167424, 392183808) crossing stripe boundary
bad metadata [783482880, 783499264) crossing stripe boundary
bad metadata [784924672, 784941056) crossing stripe boundary
bad metadata [130151612416, 130151628800) crossing stripe boundary
bad metadata [162826813440, 162826829824) crossing stripe boundary
bad metadata [162927083520, 162927099904) crossing stripe boundary
bad metadata [619740659712, 619740676096) crossing stripe boundary
bad metadata [619781947392, 619781963776) crossing stripe boundary
bad metadata [619795644416, 619795660800) crossing stripe boundary
bad metadata [619816091648, 619816108032) crossing stripe boundary
bad metadata [620011388928, 620011405312) crossing stripe boundary
bad metadata [890992459776, 890992476160) crossing stripe boundary
bad metadata [891022737408, 891022753792) crossing stripe boundary
bad metadata [891101773824, 891101790208) crossing stripe boundary
bad metadata [891301199872, 891301216256) crossing stripe boundary
bad metadata [1012219314176, 1012219330560) crossing stripe boundary
bad metadata [1017202409472, 1017202425856) crossing stripe boundary
bad metadata [1017365397504, 1017365413888) crossing stripe boundary
bad metadata [1020764422144, 1020764438528) crossing stripe boundary
bad metadata [1251103342592, 1251103358976) crossing stripe boundary
bad metadata [1251145809920, 1251145826304) crossing stripe boundary
bad metadata [1251147055104, 1251147071488) crossing stripe boundary
bad metadata [1259271225344, 1259271241728) crossing stripe boundary
bad metadata [1266223611904, 1266223628288) crossing stripe boundary
bad metadata [1304750063616, 1304750080000) crossing stripe boundary
bad metadata [1304790106112, 1304790122496) crossing stripe boundary
bad metadata [1304850792448, 1304850808832) crossing stripe boundary
bad metadata [1304869928960, 1304869945344) crossing stripe boundary
bad metadata [1305089540096, 1305089556480) crossing stripe boundary
bad metadata [1309581443072, 1309581459456) crossing stripe boundary
bad metadata [1309583671296, 1309583687680) crossing stripe boundary
bad metadata [1309942808576, 1309942824960) crossing stripe boundary
bad metadata [1310050549760, 1310050566144) crossing stripe boundary
bad metadata [1313031585792, 1313031602176) crossing stripe boundary
bad metadata [1313232912384, 1313232928768) crossing stripe boundary
bad metadata [1555210764288, 1555210780672) crossing stripe boundary
bad metadata [1555395182592, 1555395198976) crossing stripe boundary
bad metadata [2050576744448, 2050576760832) crossing stripe boundary
bad metadata [2050803957760, 2050803974144) crossing stripe boundary
bad metadata [2050969108480, 2050969124864) crossing stripe boundary
checking free space cache
checking fs roots
checking csums
checking root refs
Checking filesystem on /dev/sde1
UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
found 1860212384661 bytes used err is 0
total csum bytes: 1805105124
total tree bytes: 11788713984
total fs tree bytes: 8220835840
total extent tree bytes: 1443282944
btree space waste bytes: 2306453698
file data blocks allocated: 2137152151552
 referenced 2706831974400

That drive was not converted so either "bad metadata" is still a false
alert or there is something else going wrong in btrfs.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: csum errors in VirtualBox VDI files

[Kai Krakow]

Am Sun, 27 Mar 2016 13:04:25 -0600
schrieb Chris Murphy <lists@colorremedies.com>:

> As for the csum errors with this one single VDI file, you're going to
> have to come up with a way to reproduce it consistently. You'll need
> to have a good copy on a filesystem that comes up clean with btrfs
> check and scrub. And then reproduce the corruption somehow. One hint
> based on the other two users with similar setups or workload is they
> aren't using the discard mount option and you are. I'd say unless you
> have a newer SSD that supports queued trim, it probably shouldn't be
> used, it's known to cause the kinds of hangs you report with drives
> that only support non-queued trim. Those drives are better off getting
> fstrim e.g. once a week on a timer.

Let's get back to the csum errors later - that's only on the main drive
which has other corruptions, too, as I found out. So the csum errors
may well be a side effect.

I'm currently trying to fix the remaining problems of the backup drive
which uses no discard at all (it's no SSD).

I'd like to help you out of the confusion, here's the output of:

$ lsblk -o NAME,MODEL,FSTYPE,LABEL,MOUNTPOINT
NAME        MODEL            FSTYPE LABEL      MOUNTPOINT
sda         Crucial_CT128MX1
├─sda1                       vfat   ESP        /boot
├─sda2
└─sda3                       bcache
  ├─bcache0                  btrfs  system
  ├─bcache1                  btrfs  system
  └─bcache2                  btrfs  system     /
sdb         SAMSUNG HD103SJ
├─sdb1                       swap   swap0      [SWAP]
└─sdb2                       bcache
  └─bcache2                  btrfs  system     /
sdc         SAMSUNG HD103SJ
├─sdc1                       swap   swap1      [SWAP]
└─sdc2                       bcache
  └─bcache0                  btrfs  system
sdd         SAMSUNG HD103UJ
├─sdd1                       swap   swap2      [SWAP]
└─sdd2                       bcache
  └─bcache1                  btrfs  system
sde         003-9VT166
└─sde1                       btrfs  usb-backup

(the mountpoint is pretty bogus due to multiple subvolumes, so I
corrected it)

BTW: This discard option ran smooth for the last 12 months or so
(apparently, the SSD drive is soon to die - smartctl lifetime counter
is almost used up, bcache + btrfs can be pretty stressful I think). I'm
not even sure if btrfs mount option "discard" has any effect at all if
it is mounted through bcache.

BTW2: Even fstrim will issue queued trim if it is supported by the
drive and will get you in the same trouble. It needs to be disabled in
the kernel, according to [1]. Side note: I have model MX100 with
firmware update applied which is supposed to fix the problem. I never
experienced the libata fault messages in dmesg.

[1]:
http://forums.crucial.com/t5/Crucial-SSDs/M500-M5x0-QUEUED-TRIM-data-corruption-alert-mostly-for-Linux/td-p/151028

-- 
Regards,
Kai

Replies to list-only preferred.

Re: bad metadata crossing stripe boundary

[Qu Wenruo]

Kai Krakow wrote on 2016/03/28 12:02 +0200:
> Changing subject to reflect the current topic...
>
> Am Sun, 27 Mar 2016 21:55:40 +0800
> schrieb Qu Wenruo <quwenruo.btrfs@gmx.com>:
>
>>> I finally got copy&paste data:
>>>
>>> # before mounting let's check the FS:
>>>
>>> $ sudo btrfsck /dev/disk/by-label/usb-backup
>>> Checking filesystem on /dev/disk/by-label/usb-backup
>>> UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
>>> checking extents
>>> bad metadata [156041216, 156057600) crossing stripe boundary
>>> bad metadata [181403648, 181420032) crossing stripe boundary
>>> bad metadata [392167424, 392183808) crossing stripe boundary
>>> bad metadata [783482880, 783499264) crossing stripe boundary
>>> bad metadata [784924672, 784941056) crossing stripe boundary
>>> bad metadata [130151612416, 130151628800) crossing stripe boundary
>>> bad metadata [162826813440, 162826829824) crossing stripe boundary
>>> bad metadata [162927083520, 162927099904) crossing stripe boundary
>>> bad metadata [619740659712, 619740676096) crossing stripe boundary
>>> bad metadata [619781947392, 619781963776) crossing stripe boundary
>>> bad metadata [619795644416, 619795660800) crossing stripe boundary
>>> bad metadata [619816091648, 619816108032) crossing stripe boundary
>>> bad metadata [620011388928, 620011405312) crossing stripe boundary
>>> bad metadata [890992459776, 890992476160) crossing stripe boundary
>>> bad metadata [891022737408, 891022753792) crossing stripe boundary
>>> bad metadata [891101773824, 891101790208) crossing stripe boundary
>>> bad metadata [891301199872, 891301216256) crossing stripe boundary
>>> bad metadata [1012219314176, 1012219330560) crossing stripe boundary
>>> bad metadata [1017202409472, 1017202425856) crossing stripe boundary
>>> bad metadata [1017365397504, 1017365413888) crossing stripe boundary
>>> bad metadata [1020764422144, 1020764438528) crossing stripe boundary
>>> bad metadata [1251103342592, 1251103358976) crossing stripe boundary
>>> bad metadata [1251144695808, 1251144712192) crossing stripe boundary
>>> bad metadata [1251147055104, 1251147071488) crossing stripe boundary
>>> bad metadata [1259271225344, 1259271241728) crossing stripe boundary
>>> bad metadata [1266223611904, 1266223628288) crossing stripe boundary
>>> bad metadata [1304750063616, 1304750080000) crossing stripe boundary
>>> bad metadata [1304790106112, 1304790122496) crossing stripe boundary
>>> bad metadata [1304850792448, 1304850808832) crossing stripe boundary
>>> bad metadata [1304869928960, 1304869945344) crossing stripe boundary
>>> bad metadata [1305089540096, 1305089556480) crossing stripe boundary
>>> bad metadata [1309561651200, 1309561667584) crossing stripe boundary
>>> bad metadata [1309581443072, 1309581459456) crossing stripe boundary
>>> bad metadata [1309583671296, 1309583687680) crossing stripe boundary
>>> bad metadata [1309942808576, 1309942824960) crossing stripe boundary
>>> bad metadata [1310050549760, 1310050566144) crossing stripe boundary
>>> bad metadata [1313031585792, 1313031602176) crossing stripe boundary
>>> bad metadata [1313232912384, 1313232928768) crossing stripe boundary
>>> bad metadata [1555210764288, 1555210780672) crossing stripe boundary
>>> bad metadata [1555395182592, 1555395198976) crossing stripe boundary
>>> bad metadata [2050576744448, 2050576760832) crossing stripe boundary
>>> bad metadata [2050803957760, 2050803974144) crossing stripe boundary
>>> bad metadata [2050969108480, 2050969124864) crossing stripe
>>> boundary
>>
>> Already mentioned in another reply, this *seems* to be false alert.
>> Latest btrfs-progs would help.
>
> No, btrfs-progs 4.5 reports those, too (as far as I understood, this
> includes the fixes for bogus "bad metadata" errors, tho I thought this
> has already been fixed in 4.2.1, I used 4.4.1). There were some nbytes
> wrong errors before which I already repaired using "--repair". I think
> that's okay, I had those in the past and it looks like btrfsck can
> repair those now (and I don't have to delete and recreate the files).
> It caused problems with "du" and "df" in the past, a problem that I'm
> currently facing too. So I better fixed them.
>
> With that done, the backup fs now only reports "bad metadata" which
> have been there before space cache v2. Full output below.
>
>>> checking free space tree cache and super generation don't match,
>>> space cache will be invalidated checking fs roots
>> Err, I found a missing '\n' before "checking fs roots".
>
> Copy and paste problem. Claws mail pretends to be smarter than me
> - I missed to fix that one. ;-)

I was searching for the missing '\n' and hopes to find any chance to 
submit a new patch.
What a pity. :(

>
>> And it seems that fs roots and extent tree are all OK.
>>
>> Quite surprising.
>> The only possible problem seems to be outdated space cache.
>>
>> Maybe mount with "-o clear_cache" will help, but I don't think that's
>> the cause.
>
> Helped, it automatically reverted the FS back to space cache v1 with
> incompat flag cleared. (I wouldn't have enabled v2 if it wasn't
> documented that this is possible)
>
>>> checking csums
>>> checking root refs
>>> found 1860217443214 bytes used err is 0
>>> total csum bytes: 1805105116
>>> total tree bytes: 11793776640
>>> total fs tree bytes: 8220835840
>>> total extent tree bytes: 1443315712
>>> btree space waste bytes: 2307850845
>>> file data blocks allocated: 2137151094784
>>>    referenced 2706830905344
>>>
>>> # now let's wait for the backup to mount the FS and look at dmesg:
>>>
>>> [21375.606479] BTRFS info (device sde1): force zlib compression
>>> [21375.606483] BTRFS info (device sde1): using free space tree
>>
>> Thanks to Chris Murphy, I almost ignored such line.
>> Not familiar with the new free space cache tree, sorry.
>>
>> If "clear_cache" doesn't help, then try "nospace_cache" to disable
>> the entire space cache.
>
> It's gone now, ignore that. It's back to the situation before space
> cache v2. Minus some "nbytes wrong" errors I had and fixed.

Nice to see it works.

>
> Nevertheless, I'm now using btrfs-progs 4.5. Here's the full output:
> (the lines seem to be partly out of order, probably due to the
> redirection)
>
> $ sudo btrfsck /dev/sde1 2>&1 | tee btrfsck-label-usb-backup.txt
> checking extents
> bad metadata [156041216, 156057600) crossing stripe boundary
> bad metadata [181403648, 181420032) crossing stripe boundary
> bad metadata [392167424, 392183808) crossing stripe boundary
> bad metadata [783482880, 783499264) crossing stripe boundary
> bad metadata [784924672, 784941056) crossing stripe boundary
> bad metadata [130151612416, 130151628800) crossing stripe boundary
> bad metadata [162826813440, 162826829824) crossing stripe boundary
> bad metadata [162927083520, 162927099904) crossing stripe boundary
> bad metadata [619740659712, 619740676096) crossing stripe boundary
> bad metadata [619781947392, 619781963776) crossing stripe boundary
> bad metadata [619795644416, 619795660800) crossing stripe boundary
> bad metadata [619816091648, 619816108032) crossing stripe boundary
> bad metadata [620011388928, 620011405312) crossing stripe boundary
> bad metadata [890992459776, 890992476160) crossing stripe boundary
> bad metadata [891022737408, 891022753792) crossing stripe boundary
> bad metadata [891101773824, 891101790208) crossing stripe boundary
> bad metadata [891301199872, 891301216256) crossing stripe boundary
> bad metadata [1012219314176, 1012219330560) crossing stripe boundary
> bad metadata [1017202409472, 1017202425856) crossing stripe boundary
> bad metadata [1017365397504, 1017365413888) crossing stripe boundary
> bad metadata [1020764422144, 1020764438528) crossing stripe boundary
> bad metadata [1251103342592, 1251103358976) crossing stripe boundary
> bad metadata [1251145809920, 1251145826304) crossing stripe boundary
> bad metadata [1251147055104, 1251147071488) crossing stripe boundary
> bad metadata [1259271225344, 1259271241728) crossing stripe boundary
> bad metadata [1266223611904, 1266223628288) crossing stripe boundary
> bad metadata [1304750063616, 1304750080000) crossing stripe boundary
> bad metadata [1304790106112, 1304790122496) crossing stripe boundary
> bad metadata [1304850792448, 1304850808832) crossing stripe boundary
> bad metadata [1304869928960, 1304869945344) crossing stripe boundary
> bad metadata [1305089540096, 1305089556480) crossing stripe boundary
> bad metadata [1309581443072, 1309581459456) crossing stripe boundary
> bad metadata [1309583671296, 1309583687680) crossing stripe boundary
> bad metadata [1309942808576, 1309942824960) crossing stripe boundary
> bad metadata [1310050549760, 1310050566144) crossing stripe boundary
> bad metadata [1313031585792, 1313031602176) crossing stripe boundary
> bad metadata [1313232912384, 1313232928768) crossing stripe boundary
> bad metadata [1555210764288, 1555210780672) crossing stripe boundary
> bad metadata [1555395182592, 1555395198976) crossing stripe boundary
> bad metadata [2050576744448, 2050576760832) crossing stripe boundary
> bad metadata [2050803957760, 2050803974144) crossing stripe boundary
> bad metadata [2050969108480, 2050969124864) crossing stripe boundary
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> Checking filesystem on /dev/sde1
> UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
> found 1860212384661 bytes used err is 0
> total csum bytes: 1805105124
> total tree bytes: 11788713984
> total fs tree bytes: 8220835840
> total extent tree bytes: 1443282944
> btree space waste bytes: 2306453698
> file data blocks allocated: 2137152151552
>   referenced 2706831974400
>
> That drive was not converted so either "bad metadata" is still a false
> alert or there is something else going wrong in btrfs.
>
It seems to be a false alert.

Just pick the last [2050969108480, 2050969124864) as an example,
In fact, they are inside the same 64K block:
2050969108480 / 64K == (2050969124864 - 1) / 64K == 31295305.

I'd better double check the codes to fix the false alert before more 
such report.

Thanks,
Qu

Re: bad metadata crossing stripe boundary

[Qu Wenruo]

Qu Wenruo wrote on 2016/03/31 09:33 +0800:
>
>
> Kai Krakow wrote on 2016/03/28 12:02 +0200:
>> Changing subject to reflect the current topic...
>>
>> Am Sun, 27 Mar 2016 21:55:40 +0800
>> schrieb Qu Wenruo <quwenruo.btrfs@gmx.com>:
>>
>>>> I finally got copy&paste data:
>>>>
>>>> # before mounting let's check the FS:
>>>>
>>>> $ sudo btrfsck /dev/disk/by-label/usb-backup
>>>> Checking filesystem on /dev/disk/by-label/usb-backup
>>>> UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
>>>> checking extents
>>>> bad metadata [156041216, 156057600) crossing stripe boundary
>>>> bad metadata [181403648, 181420032) crossing stripe boundary
>>>> bad metadata [392167424, 392183808) crossing stripe boundary
>>>> bad metadata [783482880, 783499264) crossing stripe boundary
>>>> bad metadata [784924672, 784941056) crossing stripe boundary
>>>> bad metadata [130151612416, 130151628800) crossing stripe boundary
>>>> bad metadata [162826813440, 162826829824) crossing stripe boundary
>>>> bad metadata [162927083520, 162927099904) crossing stripe boundary
>>>> bad metadata [619740659712, 619740676096) crossing stripe boundary
>>>> bad metadata [619781947392, 619781963776) crossing stripe boundary
>>>> bad metadata [619795644416, 619795660800) crossing stripe boundary
>>>> bad metadata [619816091648, 619816108032) crossing stripe boundary
>>>> bad metadata [620011388928, 620011405312) crossing stripe boundary
>>>> bad metadata [890992459776, 890992476160) crossing stripe boundary
>>>> bad metadata [891022737408, 891022753792) crossing stripe boundary
>>>> bad metadata [891101773824, 891101790208) crossing stripe boundary
>>>> bad metadata [891301199872, 891301216256) crossing stripe boundary
>>>> bad metadata [1012219314176, 1012219330560) crossing stripe boundary
>>>> bad metadata [1017202409472, 1017202425856) crossing stripe boundary
>>>> bad metadata [1017365397504, 1017365413888) crossing stripe boundary
>>>> bad metadata [1020764422144, 1020764438528) crossing stripe boundary
>>>> bad metadata [1251103342592, 1251103358976) crossing stripe boundary
>>>> bad metadata [1251144695808, 1251144712192) crossing stripe boundary
>>>> bad metadata [1251147055104, 1251147071488) crossing stripe boundary
>>>> bad metadata [1259271225344, 1259271241728) crossing stripe boundary
>>>> bad metadata [1266223611904, 1266223628288) crossing stripe boundary
>>>> bad metadata [1304750063616, 1304750080000) crossing stripe boundary
>>>> bad metadata [1304790106112, 1304790122496) crossing stripe boundary
>>>> bad metadata [1304850792448, 1304850808832) crossing stripe boundary
>>>> bad metadata [1304869928960, 1304869945344) crossing stripe boundary
>>>> bad metadata [1305089540096, 1305089556480) crossing stripe boundary
>>>> bad metadata [1309561651200, 1309561667584) crossing stripe boundary
>>>> bad metadata [1309581443072, 1309581459456) crossing stripe boundary
>>>> bad metadata [1309583671296, 1309583687680) crossing stripe boundary
>>>> bad metadata [1309942808576, 1309942824960) crossing stripe boundary
>>>> bad metadata [1310050549760, 1310050566144) crossing stripe boundary
>>>> bad metadata [1313031585792, 1313031602176) crossing stripe boundary
>>>> bad metadata [1313232912384, 1313232928768) crossing stripe boundary
>>>> bad metadata [1555210764288, 1555210780672) crossing stripe boundary
>>>> bad metadata [1555395182592, 1555395198976) crossing stripe boundary
>>>> bad metadata [2050576744448, 2050576760832) crossing stripe boundary
>>>> bad metadata [2050803957760, 2050803974144) crossing stripe boundary
>>>> bad metadata [2050969108480, 2050969124864) crossing stripe
>>>> boundary
>>>
>>> Already mentioned in another reply, this *seems* to be false alert.
>>> Latest btrfs-progs would help.
>>
>> No, btrfs-progs 4.5 reports those, too (as far as I understood, this
>> includes the fixes for bogus "bad metadata" errors, tho I thought this
>> has already been fixed in 4.2.1, I used 4.4.1). There were some nbytes
>> wrong errors before which I already repaired using "--repair". I think
>> that's okay, I had those in the past and it looks like btrfsck can
>> repair those now (and I don't have to delete and recreate the files).
>> It caused problems with "du" and "df" in the past, a problem that I'm
>> currently facing too. So I better fixed them.
>>
>> With that done, the backup fs now only reports "bad metadata" which
>> have been there before space cache v2. Full output below.
>>
>>>> checking free space tree cache and super generation don't match,
>>>> space cache will be invalidated checking fs roots
>>> Err, I found a missing '\n' before "checking fs roots".
>>
>> Copy and paste problem. Claws mail pretends to be smarter than me
>> - I missed to fix that one. ;-)
>
> I was searching for the missing '\n' and hopes to find any chance to
> submit a new patch.
> What a pity. :(
>
>>
>>> And it seems that fs roots and extent tree are all OK.
>>>
>>> Quite surprising.
>>> The only possible problem seems to be outdated space cache.
>>>
>>> Maybe mount with "-o clear_cache" will help, but I don't think that's
>>> the cause.
>>
>> Helped, it automatically reverted the FS back to space cache v1 with
>> incompat flag cleared. (I wouldn't have enabled v2 if it wasn't
>> documented that this is possible)
>>
>>>> checking csums
>>>> checking root refs
>>>> found 1860217443214 bytes used err is 0
>>>> total csum bytes: 1805105116
>>>> total tree bytes: 11793776640
>>>> total fs tree bytes: 8220835840
>>>> total extent tree bytes: 1443315712
>>>> btree space waste bytes: 2307850845
>>>> file data blocks allocated: 2137151094784
>>>>    referenced 2706830905344
>>>>
>>>> # now let's wait for the backup to mount the FS and look at dmesg:
>>>>
>>>> [21375.606479] BTRFS info (device sde1): force zlib compression
>>>> [21375.606483] BTRFS info (device sde1): using free space tree
>>>
>>> Thanks to Chris Murphy, I almost ignored such line.
>>> Not familiar with the new free space cache tree, sorry.
>>>
>>> If "clear_cache" doesn't help, then try "nospace_cache" to disable
>>> the entire space cache.
>>
>> It's gone now, ignore that. It's back to the situation before space
>> cache v2. Minus some "nbytes wrong" errors I had and fixed.
>
> Nice to see it works.
>
>>
>> Nevertheless, I'm now using btrfs-progs 4.5. Here's the full output:
>> (the lines seem to be partly out of order, probably due to the
>> redirection)
>>
>> $ sudo btrfsck /dev/sde1 2>&1 | tee btrfsck-label-usb-backup.txt
>> checking extents
>> bad metadata [156041216, 156057600) crossing stripe boundary
>> bad metadata [181403648, 181420032) crossing stripe boundary
>> bad metadata [392167424, 392183808) crossing stripe boundary
>> bad metadata [783482880, 783499264) crossing stripe boundary
>> bad metadata [784924672, 784941056) crossing stripe boundary
>> bad metadata [130151612416, 130151628800) crossing stripe boundary
>> bad metadata [162826813440, 162826829824) crossing stripe boundary
>> bad metadata [162927083520, 162927099904) crossing stripe boundary
>> bad metadata [619740659712, 619740676096) crossing stripe boundary
>> bad metadata [619781947392, 619781963776) crossing stripe boundary
>> bad metadata [619795644416, 619795660800) crossing stripe boundary
>> bad metadata [619816091648, 619816108032) crossing stripe boundary
>> bad metadata [620011388928, 620011405312) crossing stripe boundary
>> bad metadata [890992459776, 890992476160) crossing stripe boundary
>> bad metadata [891022737408, 891022753792) crossing stripe boundary
>> bad metadata [891101773824, 891101790208) crossing stripe boundary
>> bad metadata [891301199872, 891301216256) crossing stripe boundary
>> bad metadata [1012219314176, 1012219330560) crossing stripe boundary
>> bad metadata [1017202409472, 1017202425856) crossing stripe boundary
>> bad metadata [1017365397504, 1017365413888) crossing stripe boundary
>> bad metadata [1020764422144, 1020764438528) crossing stripe boundary
>> bad metadata [1251103342592, 1251103358976) crossing stripe boundary
>> bad metadata [1251145809920, 1251145826304) crossing stripe boundary
>> bad metadata [1251147055104, 1251147071488) crossing stripe boundary
>> bad metadata [1259271225344, 1259271241728) crossing stripe boundary
>> bad metadata [1266223611904, 1266223628288) crossing stripe boundary
>> bad metadata [1304750063616, 1304750080000) crossing stripe boundary
>> bad metadata [1304790106112, 1304790122496) crossing stripe boundary
>> bad metadata [1304850792448, 1304850808832) crossing stripe boundary
>> bad metadata [1304869928960, 1304869945344) crossing stripe boundary
>> bad metadata [1305089540096, 1305089556480) crossing stripe boundary
>> bad metadata [1309581443072, 1309581459456) crossing stripe boundary
>> bad metadata [1309583671296, 1309583687680) crossing stripe boundary
>> bad metadata [1309942808576, 1309942824960) crossing stripe boundary
>> bad metadata [1310050549760, 1310050566144) crossing stripe boundary
>> bad metadata [1313031585792, 1313031602176) crossing stripe boundary
>> bad metadata [1313232912384, 1313232928768) crossing stripe boundary
>> bad metadata [1555210764288, 1555210780672) crossing stripe boundary
>> bad metadata [1555395182592, 1555395198976) crossing stripe boundary
>> bad metadata [2050576744448, 2050576760832) crossing stripe boundary
>> bad metadata [2050803957760, 2050803974144) crossing stripe boundary
>> bad metadata [2050969108480, 2050969124864) crossing stripe boundary
>> checking free space cache
>> checking fs roots
>> checking csums
>> checking root refs
>> Checking filesystem on /dev/sde1
>> UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
>> found 1860212384661 bytes used err is 0
>> total csum bytes: 1805105124
>> total tree bytes: 11788713984
>> total fs tree bytes: 8220835840
>> total extent tree bytes: 1443282944
>> btree space waste bytes: 2306453698
>> file data blocks allocated: 2137152151552
>>   referenced 2706831974400
>>
>> That drive was not converted so either "bad metadata" is still a false
>> alert or there is something else going wrong in btrfs.
>>
> It seems to be a false alert.
>
> Just pick the last [2050969108480, 2050969124864) as an example,
> In fact, they are inside the same 64K block:
> 2050969108480 / 64K == (2050969124864 - 1) / 64K == 31295305.
>
> I'd better double check the codes to fix the false alert before more
> such report.
>
> Thanks,
> Qu
>

Would you please try the following patch based on v4.5 btrfs-progs?
https://patchwork.kernel.org/patch/8706891/

According to your output, all the output is false alert.
All the extent starting bytenr can be divided by 64K, and I think at 
initial time, its 'max_size' may be set to 0, causing "start + 0 - 1" to 
be inside previous 64K range.

The patch would update cross_stripe every time the extent is updated, so 
such temporary false alert should disappear.

Thanks,
Qu
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Thu, 31 Mar 2016 10:31:49 +0800
schrieb Qu Wenruo <quwenruo@cn.fujitsu.com>:

> Qu Wenruo wrote on 2016/03/31 09:33 +0800:
> >
> >
> > Kai Krakow wrote on 2016/03/28 12:02 +0200:  
> >> Changing subject to reflect the current topic...
> >>
> >> Am Sun, 27 Mar 2016 21:55:40 +0800
> >> schrieb Qu Wenruo <quwenruo.btrfs@gmx.com>:
> >>  
>  [...]  
>  [...]  
> >>
> >> No, btrfs-progs 4.5 reports those, too (as far as I understood,
> >> this includes the fixes for bogus "bad metadata" errors, tho I
> >> thought this has already been fixed in 4.2.1, I used 4.4.1). There
> >> were some nbytes wrong errors before which I already repaired
> >> using "--repair". I think that's okay, I had those in the past and
> >> it looks like btrfsck can repair those now (and I don't have to
> >> delete and recreate the files). It caused problems with "du" and
> >> "df" in the past, a problem that I'm currently facing too. So I
> >> better fixed them.
> >>
> >> With that done, the backup fs now only reports "bad metadata" which
> >> have been there before space cache v2. Full output below.
> >>  
>  [...]  
>  [...]  
> >>
> >> Copy and paste problem. Claws mail pretends to be smarter than me
> >> - I missed to fix that one. ;-)  
> >
> > I was searching for the missing '\n' and hopes to find any chance to
> > submit a new patch.
> > What a pity. :(
> >  
> >>  
>  [...]  
> >>
> >> Helped, it automatically reverted the FS back to space cache v1
> >> with incompat flag cleared. (I wouldn't have enabled v2 if it
> >> wasn't documented that this is possible)
> >>  
>  [...]  
>  [...]  
> >>
> >> It's gone now, ignore that. It's back to the situation before space
> >> cache v2. Minus some "nbytes wrong" errors I had and fixed.  
> >
> > Nice to see it works.
> >  
> >>
> >> Nevertheless, I'm now using btrfs-progs 4.5. Here's the full
> >> output: (the lines seem to be partly out of order, probably due to
> >> the redirection)
> >>
> >> $ sudo btrfsck /dev/sde1 2>&1 | tee btrfsck-label-usb-backup.txt
> >> checking extents
> >> bad metadata [156041216, 156057600) crossing stripe boundary
> >> bad metadata [181403648, 181420032) crossing stripe boundary
> >> bad metadata [392167424, 392183808) crossing stripe boundary
> >> bad metadata [783482880, 783499264) crossing stripe boundary
> >> bad metadata [784924672, 784941056) crossing stripe boundary
> >> bad metadata [130151612416, 130151628800) crossing stripe boundary
> >> bad metadata [162826813440, 162826829824) crossing stripe boundary
> >> bad metadata [162927083520, 162927099904) crossing stripe boundary
> >> bad metadata [619740659712, 619740676096) crossing stripe boundary
> >> bad metadata [619781947392, 619781963776) crossing stripe boundary
> >> bad metadata [619795644416, 619795660800) crossing stripe boundary
> >> bad metadata [619816091648, 619816108032) crossing stripe boundary
> >> bad metadata [620011388928, 620011405312) crossing stripe boundary
> >> bad metadata [890992459776, 890992476160) crossing stripe boundary
> >> bad metadata [891022737408, 891022753792) crossing stripe boundary
> >> bad metadata [891101773824, 891101790208) crossing stripe boundary
> >> bad metadata [891301199872, 891301216256) crossing stripe boundary
> >> bad metadata [1012219314176, 1012219330560) crossing stripe
> >> boundary bad metadata [1017202409472, 1017202425856) crossing
> >> stripe boundary bad metadata [1017365397504, 1017365413888)
> >> crossing stripe boundary bad metadata [1020764422144,
> >> 1020764438528) crossing stripe boundary bad metadata
> >> [1251103342592, 1251103358976) crossing stripe boundary bad
> >> metadata [1251145809920, 1251145826304) crossing stripe boundary
> >> bad metadata [1251147055104, 1251147071488) crossing stripe
> >> boundary bad metadata [1259271225344, 1259271241728) crossing
> >> stripe boundary bad metadata [1266223611904, 1266223628288)
> >> crossing stripe boundary bad metadata [1304750063616,
> >> 1304750080000) crossing stripe boundary bad metadata
> >> [1304790106112, 1304790122496) crossing stripe boundary bad
> >> metadata [1304850792448, 1304850808832) crossing stripe boundary
> >> bad metadata [1304869928960, 1304869945344) crossing stripe
> >> boundary bad metadata [1305089540096, 1305089556480) crossing
> >> stripe boundary bad metadata [1309581443072, 1309581459456)
> >> crossing stripe boundary bad metadata [1309583671296,
> >> 1309583687680) crossing stripe boundary bad metadata
> >> [1309942808576, 1309942824960) crossing stripe boundary bad
> >> metadata [1310050549760, 1310050566144) crossing stripe boundary
> >> bad metadata [1313031585792, 1313031602176) crossing stripe
> >> boundary bad metadata [1313232912384, 1313232928768) crossing
> >> stripe boundary bad metadata [1555210764288, 1555210780672)
> >> crossing stripe boundary bad metadata [1555395182592,
> >> 1555395198976) crossing stripe boundary bad metadata
> >> [2050576744448, 2050576760832) crossing stripe boundary bad
> >> metadata [2050803957760, 2050803974144) crossing stripe boundary
> >> bad metadata [2050969108480, 2050969124864) crossing stripe
> >> boundary checking free space cache checking fs roots checking
> >> csums checking root refs Checking filesystem on /dev/sde1 UUID:
> >> 1318ec21-c421-4e36-a44a-7be3d41f9c3f found 1860212384661 bytes
> >> used err is 0 total csum bytes: 1805105124
> >> total tree bytes: 11788713984
> >> total fs tree bytes: 8220835840
> >> total extent tree bytes: 1443282944
> >> btree space waste bytes: 2306453698
> >> file data blocks allocated: 2137152151552
> >>   referenced 2706831974400
> >>
> >> That drive was not converted so either "bad metadata" is still a
> >> false alert or there is something else going wrong in btrfs.
> >>  
> > It seems to be a false alert.
> >
> > Just pick the last [2050969108480, 2050969124864) as an example,
> > In fact, they are inside the same 64K block:
> > 2050969108480 / 64K == (2050969124864 - 1) / 64K == 31295305.
> >
> > I'd better double check the codes to fix the false alert before more
> > such report.
> >
> > Thanks,
> > Qu
> >  
> 
> Would you please try the following patch based on v4.5 btrfs-progs?
> https://patchwork.kernel.org/patch/8706891/
> 
> According to your output, all the output is false alert.
> All the extent starting bytenr can be divided by 64K, and I think at 
> initial time, its 'max_size' may be set to 0, causing "start + 0 - 1"
> to be inside previous 64K range.
> 
> The patch would update cross_stripe every time the extent is updated,
> so such temporary false alert should disappear.

Applied and no more reports of crossing stripe boundary - thanks.

Will this go into 4.5.1 or 4.5.2?

-- 
Regards,
Kai

Replies to list-only preferred.

Re: bad metadata crossing stripe boundary

[Henk Slager]

>> Would you please try the following patch based on v4.5 btrfs-progs?
>> https://patchwork.kernel.org/patch/8706891/
>>
>> According to your output, all the output is false alert.
>> All the extent starting bytenr can be divided by 64K, and I think at
>> initial time, its 'max_size' may be set to 0, causing "start + 0 - 1"
>> to be inside previous 64K range.
>>
>> The patch would update cross_stripe every time the extent is updated,
>> so such temporary false alert should disappear.
>
> Applied and no more reports of crossing stripe boundary - thanks.
>
> Will this go into 4.5.1 or 4.5.2?

It is not in 4.5.1

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Thu, Mar 31, 2016 at 10:31:49AM +0800, Qu Wenruo wrote:
> Would you please try the following patch based on v4.5 btrfs-progs?
> https://patchwork.kernel.org/patch/8706891/

This also fixes the "bad metadata crossing stripe boundary" on my pet
patient.

I find it somewhere between funny and disturbing that the first call
of btrfs check made my kernel log the following:
Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted filesystem with ordered data mode. Opts: (null)
Mar 31 22:45:38 fan kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid 67526 /dev/dm-31

No, the filesystem was not converted, it was directly created as
btrfs, and no, I didn't try mounting it.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Thu, 31 Mar 2016 23:00:04 +0200
schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:

> On Thu, Mar 31, 2016 at 10:31:49AM +0800, Qu Wenruo wrote:
> > Would you please try the following patch based on v4.5 btrfs-progs?
> > https://patchwork.kernel.org/patch/8706891/  
> 
> This also fixes the "bad metadata crossing stripe boundary" on my pet
> patient.
> 
> I find it somewhere between funny and disturbing that the first call
> of btrfs check made my kernel log the following:
> Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
> filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
> kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
> 67526 /dev/dm-31
> 
> No, the filesystem was not converted, it was directly created as
> btrfs, and no, I didn't try mounting it.

I suggest that your partition contained ext4 before, and you didn't run
wipefs before running mkfs.btrfs. Now, the ext4 superblock is still
detected because no btrfs structure or block did overwrite it. I had a
similar problem when I first tried btrfs.

I think there is some magic dd-fu to damage the ext4 superblock without
hurting the btrfs itself. But I leave this to the fs devs, they could
properly tell you.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Thu, 31 Mar 2016 23:16:30 +0200
schrieb Kai Krakow <hurikhan77@gmail.com>:

> Am Thu, 31 Mar 2016 23:00:04 +0200
> schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:
> 
> > On Thu, Mar 31, 2016 at 10:31:49AM +0800, Qu Wenruo wrote:  
> > > Would you please try the following patch based on v4.5
> > > btrfs-progs? https://patchwork.kernel.org/patch/8706891/    
> > 
> > This also fixes the "bad metadata crossing stripe boundary" on my
> > pet patient.
> > 
> > I find it somewhere between funny and disturbing that the first call
> > of btrfs check made my kernel log the following:
> > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
> > filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
> > kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
> > 67526 /dev/dm-31
> > 
> > No, the filesystem was not converted, it was directly created as
> > btrfs, and no, I didn't try mounting it.  
> 
> I suggest that your partition contained ext4 before, and you didn't
> run wipefs before running mkfs.btrfs. Now, the ext4 superblock is
> still detected because no btrfs structure or block did overwrite it.
> I had a similar problem when I first tried btrfs.
> 
> I think there is some magic dd-fu to damage the ext4 superblock
> without hurting the btrfs itself. But I leave this to the fs devs,
> they could properly tell you.

Tho, you could also try to force detecting btrfs before ext4 by
modifying /etc/filesystems.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
> Am Thu, 31 Mar 2016 23:00:04 +0200
> schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:
> > I find it somewhere between funny and disturbing that the first call
> > of btrfs check made my kernel log the following:
> > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
> > filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
> > kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
> > 67526 /dev/dm-31
> > 
> > No, the filesystem was not converted, it was directly created as
> > btrfs, and no, I didn't try mounting it.
> 
> I suggest that your partition contained ext4 before, and you didn't run
> wipefs before running mkfs.btrfs.

I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
That should do a much better job than wipefsing it, shouldnt it?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Fri, 1 Apr 2016 07:57:25 +0200
schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:

> On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
> > Am Thu, 31 Mar 2016 23:00:04 +0200
> > schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:  
> > > I find it somewhere between funny and disturbing that the first
> > > call of btrfs check made my kernel log the following:
> > > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31):
> > > mounted filesystem with ordered data mode. Opts: (null) Mar 31
> > > 22:45:38 fan kernel: [ 6255.361328] BTRFS: device label fanbtr
> > > devid 1 transid 67526 /dev/dm-31
> > > 
> > > No, the filesystem was not converted, it was directly created as
> > > btrfs, and no, I didn't try mounting it.  
> > 
> > I suggest that your partition contained ext4 before, and you didn't
> > run wipefs before running mkfs.btrfs.  
> 
> I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
> That should do a much better job than wipefsing it, shouldnt it?

Not sure how luksFormat works. If it encrypts what is already on the
device, it would also encrypt orphan superblocks.

If it actually wipes the device, the orphan superblock should be gone.

This suggests it does not wipe the device:

https://wiki.archlinux.org/index.php/Dm-crypt/Device_encryption:

> Encryption options for LUKS mode
> The cryptsetup action to set up a new dm-crypt device in LUKS
> encryption mode is luksFormat. Unlike the name implies, it does not
> format the device, but sets up the LUKS device header and encrypts
> the master-key with the desired cryptographic options.

Thus, I may very well have an orphan superblock lying around.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Sat, Apr 02, 2016 at 11:03:53AM +0200, Kai Krakow wrote:
> Am Fri, 1 Apr 2016 07:57:25 +0200
> schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:
> > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
> > > Am Thu, 31 Mar 2016 23:00:04 +0200
> > > schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:  
> > > > I find it somewhere between funny and disturbing that the first
> > > > call of btrfs check made my kernel log the following:
> > > > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31):
> > > > mounted filesystem with ordered data mode. Opts: (null) Mar 31
> > > > 22:45:38 fan kernel: [ 6255.361328] BTRFS: device label fanbtr
> > > > devid 1 transid 67526 /dev/dm-31
> > > > 
> > > > No, the filesystem was not converted, it was directly created as
> > > > btrfs, and no, I didn't try mounting it.  
> > > 
> > > I suggest that your partition contained ext4 before, and you didn't
> > > run wipefs before running mkfs.btrfs.  
> > 
> > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
> > That should do a much better job than wipefsing it, shouldnt it?
> 
> Not sure how luksFormat works. If it encrypts what is already on the
> device, it would also encrypt orphan superblocks.

It overwrites the LUKS metadata including the symmetric key that was
used to encrypt the existing data. Short of Shor's Algorithm and
Quantum Computers, after that operation it is no longer possible to
even guess what was on the disk before.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Sat, 2 Apr 2016 11:44:32 +0200
schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:

> On Sat, Apr 02, 2016 at 11:03:53AM +0200, Kai Krakow wrote:
> > Am Fri, 1 Apr 2016 07:57:25 +0200
> > schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:  
> > > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:  
>  [...]  
>  [...]  
>  [...]  
> > > 
> > > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed
> > > it. That should do a much better job than wipefsing it, shouldnt
> > > it?  
> > 
> > Not sure how luksFormat works. If it encrypts what is already on the
> > device, it would also encrypt orphan superblocks.  
> 
> It overwrites the LUKS metadata including the symmetric key that was
> used to encrypt the existing data. Short of Shor's Algorithm and
> Quantum Computers, after that operation it is no longer possible to
> even guess what was on the disk before.

If it was encrypted before... ;-)

-- 
Regards,
Kai

Replies to list-only preferred.

Re: bad metadata crossing stripe boundary

[Patrik Lundquist]

On 2 April 2016 at 20:31, Kai Krakow <hurikhan77@gmail.com> wrote:
> Am Sat, 2 Apr 2016 11:44:32 +0200
> schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:
>
>> On Sat, Apr 02, 2016 at 11:03:53AM +0200, Kai Krakow wrote:
>> > Am Fri, 1 Apr 2016 07:57:25 +0200
>> > schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:
>> > > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
>>  [...]
>>  [...]
>>  [...]
>> > >
>> > > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed
>> > > it. That should do a much better job than wipefsing it, shouldnt
>> > > it?
>> >
>> > Not sure how luksFormat works. If it encrypts what is already on the
>> > device, it would also encrypt orphan superblocks.
>>
>> It overwrites the LUKS metadata including the symmetric key that was
>> used to encrypt the existing data. Short of Shor's Algorithm and
>> Quantum Computers, after that operation it is no longer possible to
>> even guess what was on the disk before.
>
> If it was encrypted before... ;-)

What does wipefs -n find?

Re: bad metadata crossing stripe boundary

[Chris Murphy]

On Thu, Mar 31, 2016 at 11:57 PM, Marc Haber
<mh+linux-btrfs@zugschlus.de> wrote:
> On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
>> Am Thu, 31 Mar 2016 23:00:04 +0200
>> schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:
>> > I find it somewhere between funny and disturbing that the first call
>> > of btrfs check made my kernel log the following:
>> > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
>> > filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
>> > kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
>> > 67526 /dev/dm-31
>> >
>> > No, the filesystem was not converted, it was directly created as
>> > btrfs, and no, I didn't try mounting it.
>>
>> I suggest that your partition contained ext4 before, and you didn't run
>> wipefs before running mkfs.btrfs.
>
> I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
> That should do a much better job than wipefsing it, shouldnt it?

Not really. The first btrfs super is at 64K. The second at 64M. The
third at 256G. While wipefs will remove the magic only on the first,
mkfs.btrfs will take care of all three. And luksFormat only overwrites
the first 132K of a block device. There's a scant chance of bugs
related to previous filesystems not being erased, I think this is more
likely when mixing and matching filesystems just because the
superblocks for each filesystem aren't in the same location.

If you're concerned about traces of previous file systems, then use
the dmcrypt device itself, rather than merely using the original block
device where merely 132K at the beginning has been overwritten.
Everytime you format a device, the resulting dmcrypt logical device is
in effect full of completely random data. A new random key is
generated each time you use luksFormat, even if you're using the same
passphrase.

-- 
Chris Murphy

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Sat, Apr 02, 2016 at 08:31:17PM +0200, Kai Krakow wrote:
> Am Sat, 2 Apr 2016 11:44:32 +0200
> schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:
> 
> > On Sat, Apr 02, 2016 at 11:03:53AM +0200, Kai Krakow wrote:
> > > Am Fri, 1 Apr 2016 07:57:25 +0200
> > > schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:  
> > > > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:  
> >  [...]  
> >  [...]  
> >  [...]  
> > > > 
> > > > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed
> > > > it. That should do a much better job than wipefsing it, shouldnt
> > > > it?  
> > > 
> > > Not sure how luksFormat works. If it encrypts what is already on the
> > > device, it would also encrypt orphan superblocks.  
> > 
> > It overwrites the LUKS metadata including the symmetric key that was
> > used to encrypt the existing data. Short of Shor's Algorithm and
> > Quantum Computers, after that operation it is no longer possible to
> > even guess what was on the disk before.
> 
> If it was encrypted before... ;-)

First, it was.

Second, cleartext found on the block device is quite unlikely to be
readable from the unlocked crypto device. I would be very worried if
that were the case.

I must be missing something here.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Sat, Apr 02, 2016 at 01:41:53PM -0600, Chris Murphy wrote:
> On Thu, Mar 31, 2016 at 11:57 PM, Marc Haber
> <mh+linux-btrfs@zugschlus.de> wrote:
> > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
> >> Am Thu, 31 Mar 2016 23:00:04 +0200
> >> schrieb Marc Haber <mh+linux-btrfs@zugschlus.de>:
> >> > I find it somewhere between funny and disturbing that the first call
> >> > of btrfs check made my kernel log the following:
> >> > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
> >> > filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
> >> > kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
> >> > 67526 /dev/dm-31
> >> >
> >> > No, the filesystem was not converted, it was directly created as
> >> > btrfs, and no, I didn't try mounting it.
> >>
> >> I suggest that your partition contained ext4 before, and you didn't run
> >> wipefs before running mkfs.btrfs.
> >
> > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
> > That should do a much better job than wipefsing it, shouldnt it?
> 
> Not really. The first btrfs super is at 64K. The second at 64M. The
> third at 256G. While wipefs will remove the magic only on the first,
> mkfs.btrfs will take care of all three. And luksFormat only overwrites
> the first 132K of a block device. There's a scant chance of bugs
> related to previous filesystems not being erased, I think this is more
> likely when mixing and matching filesystems just because the
> superblocks for each filesystem aren't in the same location.

If I do:

umount /dev/mapper/foo
cryptsetup close /dev/mapper/foo
cryptsetup luksFormat /dev/mapper/pv-c_foo
cryptsetup open /dev/mapper/pv-c_foo foo

and the contents of /dev/mapper/foo would randomly resemble its
previous contents afterwards, I would be _very_ disturbed. During the
luksFormat process, a new random symmetric key is created, and
overwrites the old random symmetric key in the LUKS header. Therefore,
the following crypto operations are _very_ unlikely to produce
something that resembles an ext4 fileystem.

Even if I did:

umount /dev/mapper/foo
cryptsetup close /dev/mapper/foo
mkfs.btrfs /dev/mapper/pv-c_foo

(assuming I previously did cryptsetup open /dev/mapper/pv-c_foo foo)

I would be _very_ surprised if the kernel would find something
resembling and ext4 file system on /dev/mapper/pv-c_foo.

> If you're concerned about traces of previous file systems, then use
> the dmcrypt device itself, rather than merely using the original block
> device where merely 132K at the beginning has been overwritten.
> Everytime you format a device, the resulting dmcrypt logical device is
> in effect full of completely random data. A new random key is
> generated each time you use luksFormat, even if you're using the same
> passphrase.

That's what I am saying.

I must be missing something.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: bad metadata crossing stripe boundary

[Chris Murphy]

On Sun, Apr 3, 2016 at 2:51 AM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> On Sat, Apr 02, 2016 at 01:41:53PM -0600, Chris Murphy wrote:

>> > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
>> > That should do a much better job than wipefsing it, shouldnt it?
>>
>> Not really. The first btrfs super is at 64K. The second at 64M. The
>> third at 256G. While wipefs will remove the magic only on the first,
>> mkfs.btrfs will take care of all three. And luksFormat only overwrites
>> the first 132K of a block device. There's a scant chance of bugs
>> related to previous filesystems not being erased, I think this is more
>> likely when mixing and matching filesystems just because the
>> superblocks for each filesystem aren't in the same location.
>
> If I do:
>
> umount /dev/mapper/foo
> cryptsetup close /dev/mapper/foo
> cryptsetup luksFormat /dev/mapper/pv-c_foo
> cryptsetup open /dev/mapper/pv-c_foo foo
>
> and the contents of /dev/mapper/foo would randomly resemble its
> previous contents afterwards, I would be _very_ disturbed.


You wrote "luksFormat the partition, then mkfs.btrfs it" and then also
"wipefs it" where in each case "it" sounded to me like you're
referring to the same partition. But your example makes it more clear
you're referring to mkfs.btrfs being done not on the partition, but on
the dmcrypt device.



> During the
> luksFormat process, a new random symmetric key is created, and
> overwrites the old random symmetric key in the LUKS header. Therefore,
> the following crypto operations are _very_ unlikely to produce
> something that resembles an ext4 fileystem.

All the primates on the planet have a better chance of writing
Shakespeare, this year.

>
> Even if I did:
>
> umount /dev/mapper/foo
> cryptsetup close /dev/mapper/foo
> mkfs.btrfs /dev/mapper/pv-c_foo
>
> (assuming I previously did cryptsetup open /dev/mapper/pv-c_foo foo)
>
> I would be _very_ surprised if the kernel would find something
> resembling and ext4 file system on /dev/mapper/pv-c_foo.

I don't follow that at all, really. It's a sequence, vaguely modified
by a parenthetical in a way that I do not understand, with an outcome
that can't be confirmed or denied because mkfs.ext4 isn't in your
sequence anywhere.

If you  were to see (encrypted) writes to the LV, you could identify
the file system being used based on the write pattern. While the
writes themselves are ciphertext and therefore you don't know what is
being written, the write pattern is not obfuscated by dmcrypt so you
could infer what is doing the writing by that pattern (probably both
application and the filesystem could be identified).


-- 
Chris Murphy