linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Unrecoverable btrfs corruption (backref bytes do not match extent backref)
@ 2019-01-03 22:06 Nazar Mokrynskyi
  2019-01-04  1:15 ` Chris Murphy
  0 siblings, 1 reply; 10+ messages in thread
From: Nazar Mokrynskyi @ 2019-01-03 22:06 UTC (permalink / raw)
  To: linux-btrfs

Today I have faced yet another BTRFS corruption, the only major software change in recent days was upgrade to Linux 4.20, so this is something to keep in mind.
Previous corruption happened with kernel 4.14 a bit over a year ago I think, so filesystem was with high probability created in December 2017.

This time I saved an image of corrupted partition, so I can make more experiments with it.

A bit of history:
1) several days ago I was running scrub and found some inode errors, it boiled down to /var/lib/mysql/ibdata1 file (/var/lib/mysql have CoW disabled), so I've replaced that file with file from backup, made fresh snapshot and removed old snapshots that had corrupted file in it, scrub was happy after that
2) today everything worked fine until on reboot system booted into initramfs complaining about something with btrfs...

My setup: BTRFS under full-disk LUKS, no partition table:
/dev/mapper/system / btrfs compress=lzo,noatime,ssd,subvol=/root 0 1

btrfs-progs v4.19.1, Ubuntu 19.04 (develoment branch with proposed packages enabled), kernel 4.20.0 with ACS override patch

After inspection turned out that scrub is still perfectly fine, no complains, however:

root@ubuntu:~# btrfsck /dev/mapper/luks-739967f1-9770-470a-a031-8d8b8bcdb350
warning, bad space info total_bytes 2155872256 used 2155876352
warning, bad space info total_bytes 3229614080 used 3229618176
warning, bad space info total_bytes 4303355904 used 4303360000
warning, bad space info total_bytes 5377097728 used 5377101824
warning, bad space info total_bytes 6450839552 used 6450843648
warning, bad space info total_bytes 7524581376 used 7524585472
warning, bad space info total_bytes 8598323200 used 8598327296
warning, bad space info total_bytes 9672065024 used 9672069120
warning, bad space info total_bytes 10745806848 used 10745810944
warning, bad space info total_bytes 11819548672 used 11819552768
warning, bad space info total_bytes 12893290496 used 12893294592
warning, bad space info total_bytes 13967032320 used 13967036416
warning, bad space info total_bytes 15040774144 used 15040778240
warning, bad space info total_bytes 16114515968 used 16114520064
warning, bad space info total_bytes 17188257792 used 17188261888
warning, bad space info total_bytes 18261999616 used 18262003712
warning, bad space info total_bytes 19335741440 used 19335745536
warning, bad space info total_bytes 20409483264 used 20409487360
warning, bad space info total_bytes 21483225088 used 21483229184
warning, bad space info total_bytes 22556966912 used 22556971008
warning, bad space info total_bytes 23630708736 used 23630712832
warning, bad space info total_bytes 24704450560 used 24704454656
warning, bad space info total_bytes 25778192384 used 25778196480
warning, bad space info total_bytes 26851934208 used 26851938304
warning, bad space info total_bytes 27925676032 used 27925680128
warning, bad space info total_bytes 28999417856 used 28999421952
warning, bad space info total_bytes 30073159680 used 30073163776
warning, bad space info total_bytes 31146901504 used 31146905600
warning, bad space info total_bytes 32220643328 used 32220647424
Checking filesystem on /dev/mapper/luks-739967f1-9770-470a-a031-8d8b8bcdb350
UUID: 5170aca4-061a-4c6c-ab00-bd7fc8ae6030
checking extents
extent item 3114475520 has multiple extent items
ref mismatch on [3114475520 4096] extent item 1, found 2
backref bytes do not match extent backref, bytenr=3114475520, ref bytes=4096, backref bytes=36864
backpointer mismatch on [3114475520 4096]
ERROR: errors found in extent allocation tree or chunk allocation
checking free space cache
checking fs roots
checking csums
checking root refs
found 39409483813 bytes used, error(s) found
total csum bytes: 35990412
total tree bytes: 2395095040
total fs tree bytes: 2249408512
total extent tree bytes: 96534528
btree space waste bytes: 456622616
file data blocks allocated: 174319587328
 referenced 61677670400

I've removed all secondary subvolumes (like for Docker) and snapshots, but issue persisted.

Then I've tried `btrfs balance` and it failed at around 10-20% remaining.

I think after that scrub started complaining about errors or just stop increasing scrub progress.

This is where I made 256G partition image.

Since I have proper backups, I gave `btrfsck --repair` a try just in case it does anything useful at least once, but it just repeatedly prints following without any meaningful CPU or disk activity:

ref mismatch on [3114475520 36864] extent item 1, found 2
backref bytes do not match extent backref, bytenr=3114475520, ref bytes=36864, backref bytes=4096
backpointer mismatch on [3114475520 36864]
attempting to repair backref discrepency for bytenr 3114475520

The actual corruption happened in /docker subvolume somewhere amongst regular files. I get segfaults when I try to do anything there, sometimes kernel can lock up. Other files seem to be perfectly fine.

Removing /docker subvolume locked the kernel if I recall correctly.


If this seems anything important and you want me to run some commands to check what happened exactly, I can start VMs with this partition image connected and do whatever is needed. I can't send image anywhere though, since it contains sensitive information.

NOTE: I don't need help with partition or data recovery, I'm used to these kinds of crashes and have backups, so no data were lost.

P.S. I really wish BTRFS can stop accidentally corrupting itself one day.

-- 
Sincerely, Nazar Mokrynskyi
github.com/nazar-pc


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Unrecoverable btrfs corruption (backref bytes do not match extent backref)
  2019-01-03 22:06 Unrecoverable btrfs corruption (backref bytes do not match extent backref) Nazar Mokrynskyi
@ 2019-01-04  1:15 ` Chris Murphy
  2019-01-04  1:32   ` Qu Wenruo
  2019-01-05  0:13   ` Nazar Mokrynskyi
  0 siblings, 2 replies; 10+ messages in thread
From: Chris Murphy @ 2019-01-04  1:15 UTC (permalink / raw)
  To: Nazar Mokrynskyi, Qu Wenruo; +Cc: linux-btrfs

On Thu, Jan 3, 2019 at 3:13 PM Nazar Mokrynskyi <nazar@mokrynskyi.com> wrote:

> root@ubuntu:~# btrfsck /dev/mapper/luks-739967f1-9770-470a-a031-8d8b8bcdb350
> warning, bad space info total_bytes 2155872256 used 2155876352
> warning, bad space info total_bytes 3229614080 used 3229618176
> warning, bad space info total_bytes 4303355904 used 4303360000
> warning, bad space info total_bytes 5377097728 used 5377101824
> warning, bad space info total_bytes 6450839552 used 6450843648
> warning, bad space info total_bytes 7524581376 used 7524585472
> warning, bad space info total_bytes 8598323200 used 8598327296
> warning, bad space info total_bytes 9672065024 used 9672069120
> warning, bad space info total_bytes 10745806848 used 10745810944
> warning, bad space info total_bytes 11819548672 used 11819552768
> warning, bad space info total_bytes 12893290496 used 12893294592
> warning, bad space info total_bytes 13967032320 used 13967036416
> warning, bad space info total_bytes 15040774144 used 15040778240
> warning, bad space info total_bytes 16114515968 used 16114520064
> warning, bad space info total_bytes 17188257792 used 17188261888
> warning, bad space info total_bytes 18261999616 used 18262003712
> warning, bad space info total_bytes 19335741440 used 19335745536
> warning, bad space info total_bytes 20409483264 used 20409487360
> warning, bad space info total_bytes 21483225088 used 21483229184
> warning, bad space info total_bytes 22556966912 used 22556971008
> warning, bad space info total_bytes 23630708736 used 23630712832
> warning, bad space info total_bytes 24704450560 used 24704454656
> warning, bad space info total_bytes 25778192384 used 25778196480
> warning, bad space info total_bytes 26851934208 used 26851938304
> warning, bad space info total_bytes 27925676032 used 27925680128
> warning, bad space info total_bytes 28999417856 used 28999421952
> warning, bad space info total_bytes 30073159680 used 30073163776
> warning, bad space info total_bytes 31146901504 used 31146905600
> warning, bad space info total_bytes 32220643328 used 32220647424
> Checking filesystem on /dev/mapper/luks-739967f1-9770-470a-a031-8d8b8bcdb350
> UUID: 5170aca4-061a-4c6c-ab00-bd7fc8ae6030
> checking extents
> extent item 3114475520 has multiple extent items
> ref mismatch on [3114475520 4096] extent item 1, found 2
> backref bytes do not match extent backref, bytenr=3114475520, ref bytes=4096, backref bytes=36864
> backpointer mismatch on [3114475520 4096]
> ERROR: errors found in extent allocation tree or chunk allocation
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> found 39409483813 bytes used, error(s) found
> total csum bytes: 35990412
> total tree bytes: 2395095040
> total fs tree bytes: 2249408512
> total extent tree bytes: 96534528
> btree space waste bytes: 456622616
> file data blocks allocated: 174319587328
>  referenced 61677670400

I haven't seen "bad space info" before and searching the list archive
I only came up with one other report of it and no developer replied.

What do you get with 'btrfs check --mode=lowmem' ? This is a different
implementation of check, and might reveal some additional information
useful to developers. It is wickedly slow however.


> If this seems anything important and you want me to run some commands to check what happened exactly, I can start VMs with this partition image connected and do whatever is needed. I can't send image anywhere though, since it contains sensitive information.

If you use btrfs-image -ss option, there won't be any sensitive
information included. Files are hashed. Some short name files or dirs
can't be hashed (you'll see a warning) and those are replaced with
random garbage instead. There is only metadata with the image, no user
data.

> P.S. I really wish BTRFS can stop accidentally corrupting itself one day.

I have several Btrfs file systems that I use constantly, all but one
is at least three years old. I've never experienced corruption on any
of them. I did discover some isolated corruption of file data
(systemd-journald and also a VM image) but those were bugs that got
fixed, and also the file system metadata was fine.

So I guess my bigger complaint with Btrfs isn't so much that
corruption happens, but that it sometimes happens in a way we can't
track down and know what to blame for the problem. Knowing what to
blame is important for fixing bugs and also working around them in the
meantime. So for sure it's frustrating, even if I haven't experienced
it myself.

As Btrfs file systems get bigger, the more we're running into the very
scalability complaints that existed pre-Btrfs. The idea of Btrfs is
that it should always be consistent, and not need offline repair.
Doing an offline consistency check is just too expensive. Doing a
check and repair that ends up making the situation worse, is really
expensive. That's a bad user experience that reasonably turns into a
lost user. And in order to fix bugs and make Btrfs better, we need
more and good bug reports, not less.

So... yeah. I'm not sure if there is more tracing or debugging
information that's needed by default in Btrfs, so we can have a better
chance of understanding these corruptions when they occur? Or what. We
can't expect people to leave integrity checking always on, it's too
expensive.

--
Chris Murphy

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Unrecoverable btrfs corruption (backref bytes do not match extent backref)
  2019-01-04  1:15 ` Chris Murphy
@ 2019-01-04  1:32   ` Qu Wenruo
  2019-01-04  1:36     ` Hans van Kranenburg
                       ` (2 more replies)
  2019-01-05  0:13   ` Nazar Mokrynskyi
  1 sibling, 3 replies; 10+ messages in thread
From: Qu Wenruo @ 2019-01-04  1:32 UTC (permalink / raw)
  To: Chris Murphy, Nazar Mokrynskyi; +Cc: linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 6377 bytes --]



On 2019/1/4 上午9:15, Chris Murphy wrote:
> On Thu, Jan 3, 2019 at 3:13 PM Nazar Mokrynskyi <nazar@mokrynskyi.com> wrote:
> 
>> root@ubuntu:~# btrfsck /dev/mapper/luks-739967f1-9770-470a-a031-8d8b8bcdb350
>> warning, bad space info total_bytes 2155872256 used 2155876352
>> warning, bad space info total_bytes 3229614080 used 3229618176
>> warning, bad space info total_bytes 4303355904 used 4303360000
>> warning, bad space info total_bytes 5377097728 used 5377101824
>> warning, bad space info total_bytes 6450839552 used 6450843648
>> warning, bad space info total_bytes 7524581376 used 7524585472
>> warning, bad space info total_bytes 8598323200 used 8598327296
>> warning, bad space info total_bytes 9672065024 used 9672069120
>> warning, bad space info total_bytes 10745806848 used 10745810944
>> warning, bad space info total_bytes 11819548672 used 11819552768
>> warning, bad space info total_bytes 12893290496 used 12893294592
>> warning, bad space info total_bytes 13967032320 used 13967036416
>> warning, bad space info total_bytes 15040774144 used 15040778240
>> warning, bad space info total_bytes 16114515968 used 16114520064
>> warning, bad space info total_bytes 17188257792 used 17188261888
>> warning, bad space info total_bytes 18261999616 used 18262003712
>> warning, bad space info total_bytes 19335741440 used 19335745536
>> warning, bad space info total_bytes 20409483264 used 20409487360
>> warning, bad space info total_bytes 21483225088 used 21483229184
>> warning, bad space info total_bytes 22556966912 used 22556971008
>> warning, bad space info total_bytes 23630708736 used 23630712832
>> warning, bad space info total_bytes 24704450560 used 24704454656
>> warning, bad space info total_bytes 25778192384 used 25778196480
>> warning, bad space info total_bytes 26851934208 used 26851938304
>> warning, bad space info total_bytes 27925676032 used 27925680128
>> warning, bad space info total_bytes 28999417856 used 28999421952
>> warning, bad space info total_bytes 30073159680 used 30073163776
>> warning, bad space info total_bytes 31146901504 used 31146905600
>> warning, bad space info total_bytes 32220643328 used 32220647424

I'm not sure what this means.
I thought it's free space cache, but code doesn't prove that.


>> Checking filesystem on /dev/mapper/luks-739967f1-9770-470a-a031-8d8b8bcdb350
>> UUID: 5170aca4-061a-4c6c-ab00-bd7fc8ae6030
>> checking extents
>> extent item 3114475520 has multiple extent items
>> ref mismatch on [3114475520 4096] extent item 1, found 2
>> backref bytes do not match extent backref, bytenr=3114475520, ref bytes=4096, backref bytes=36864
>> backpointer mismatch on [3114475520 4096]

So only one corruption, the only good news.

Please provide the dump of the following command:

# btrfs ins dump-tree -t extent | grep 3114475520 -C 20

Feel free to hide any sensitive output, but for this command there
should be no sensitive info at all.

But from what the output, I'm wondering it's some kernel bug, maybe from
v4.14.

>> ERROR: errors found in extent allocation tree or chunk allocation
>> checking free space cache
>> checking fs roots
>> checking csums
>> checking root refs
>> found 39409483813 bytes used, error(s) found
>> total csum bytes: 35990412
>> total tree bytes: 2395095040
>> total fs tree bytes: 2249408512
>> total extent tree bytes: 96534528
>> btree space waste bytes: 456622616
>> file data blocks allocated: 174319587328
>>  referenced 61677670400
> 
> I haven't seen "bad space info" before and searching the list archive
> I only came up with one other report of it and no developer replied.
> 
> What do you get with 'btrfs check --mode=lowmem' ? This is a different
> implementation of check, and might reveal some additional information
> useful to developers. It is wickedly slow however.

Yes, --mode=lowmem is also preferred.

> 
> 
>> If this seems anything important and you want me to run some commands to check what happened exactly, I can start VMs with this partition image connected and do whatever is needed. I can't send image anywhere though, since it contains sensitive information.
> 
> If you use btrfs-image -ss option, there won't be any sensitive
> information included. Files are hashed. Some short name files or dirs
> can't be hashed (you'll see a warning) and those are replaced with
> random garbage instead. There is only metadata with the image, no user
> data.

Please don't advice -ss for btrfs-image. It's super slow and will easily
cause tons of noise when running btrfs check on it.

It's either btrfs-image, or nothing (manual btrfs ins dump-tree check).

Thanks,
Qu

> 
>> P.S. I really wish BTRFS can stop accidentally corrupting itself one day.
> 
> I have several Btrfs file systems that I use constantly, all but one
> is at least three years old. I've never experienced corruption on any
> of them. I did discover some isolated corruption of file data
> (systemd-journald and also a VM image) but those were bugs that got
> fixed, and also the file system metadata was fine.
> 
> So I guess my bigger complaint with Btrfs isn't so much that
> corruption happens, but that it sometimes happens in a way we can't
> track down and know what to blame for the problem. Knowing what to
> blame is important for fixing bugs and also working around them in the
> meantime. So for sure it's frustrating, even if I haven't experienced
> it myself.
> 
> As Btrfs file systems get bigger, the more we're running into the very
> scalability complaints that existed pre-Btrfs. The idea of Btrfs is
> that it should always be consistent, and not need offline repair.
> Doing an offline consistency check is just too expensive. Doing a
> check and repair that ends up making the situation worse, is really
> expensive. That's a bad user experience that reasonably turns into a
> lost user. And in order to fix bugs and make Btrfs better, we need
> more and good bug reports, not less.
> 
> So... yeah. I'm not sure if there is more tracing or debugging
> information that's needed by default in Btrfs, so we can have a better
> chance of understanding these corruptions when they occur? Or what. We
> can't expect people to leave integrity checking always on, it's too
> expensive.
> 
> --
> Chris Murphy
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Unrecoverable btrfs corruption (backref bytes do not match extent backref)
  2019-01-04  1:32   ` Qu Wenruo
@ 2019-01-04  1:36     ` Hans van Kranenburg
  2019-01-04  3:43     ` Chris Murphy
  2019-01-04 23:04     ` Nazar Mokrynskyi
  2 siblings, 0 replies; 10+ messages in thread
From: Hans van Kranenburg @ 2019-01-04  1:36 UTC (permalink / raw)
  To: Qu Wenruo, Chris Murphy, Nazar Mokrynskyi; +Cc: linux-btrfs

On 1/4/19 2:32 AM, Qu Wenruo wrote:
> 
> 
> On 2019/1/4 上午9:15, Chris Murphy wrote:
>> On Thu, Jan 3, 2019 at 3:13 PM Nazar Mokrynskyi <nazar@mokrynskyi.com> wrote:
>>
>>> root@ubuntu:~# btrfsck /dev/mapper/luks-739967f1-9770-470a-a031-8d8b8bcdb350
>>> warning, bad space info total_bytes 2155872256 used 2155876352
>>> warning, bad space info total_bytes 3229614080 used 3229618176
>>> warning, bad space info total_bytes 4303355904 used 4303360000
>>> warning, bad space info total_bytes 5377097728 used 5377101824
>>> warning, bad space info total_bytes 6450839552 used 6450843648
>>> warning, bad space info total_bytes 7524581376 used 7524585472
>>> warning, bad space info total_bytes 8598323200 used 8598327296
>>> warning, bad space info total_bytes 9672065024 used 9672069120
>>> warning, bad space info total_bytes 10745806848 used 10745810944
>>> warning, bad space info total_bytes 11819548672 used 11819552768
>>> warning, bad space info total_bytes 12893290496 used 12893294592
>>> warning, bad space info total_bytes 13967032320 used 13967036416
>>> warning, bad space info total_bytes 15040774144 used 15040778240
>>> warning, bad space info total_bytes 16114515968 used 16114520064
>>> warning, bad space info total_bytes 17188257792 used 17188261888
>>> warning, bad space info total_bytes 18261999616 used 18262003712
>>> warning, bad space info total_bytes 19335741440 used 19335745536
>>> warning, bad space info total_bytes 20409483264 used 20409487360
>>> warning, bad space info total_bytes 21483225088 used 21483229184
>>> warning, bad space info total_bytes 22556966912 used 22556971008
>>> warning, bad space info total_bytes 23630708736 used 23630712832
>>> warning, bad space info total_bytes 24704450560 used 24704454656
>>> warning, bad space info total_bytes 25778192384 used 25778196480
>>> warning, bad space info total_bytes 26851934208 used 26851938304
>>> warning, bad space info total_bytes 27925676032 used 27925680128
>>> warning, bad space info total_bytes 28999417856 used 28999421952
>>> warning, bad space info total_bytes 30073159680 used 30073163776
>>> warning, bad space info total_bytes 31146901504 used 31146905600
>>> warning, bad space info total_bytes 32220643328 used 32220647424
> 
> I'm not sure what this means.
> I thought it's free space cache, but code doesn't prove that.

I was thinking about a 'space', which in btrfs, is a collection of all
block groups which have a specific combination of type/profile, like
'Data, RAID1'.

Note that the difference between the two numbers in every line seems to
be 4096 bytes.

> [...]

-- 
Hans van Kranenburg

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Unrecoverable btrfs corruption (backref bytes do not match extent backref)
  2019-01-04  1:32   ` Qu Wenruo
  2019-01-04  1:36     ` Hans van Kranenburg
@ 2019-01-04  3:43     ` Chris Murphy
  2019-01-04  4:31       ` Qu Wenruo
  2019-01-04 23:04     ` Nazar Mokrynskyi
  2 siblings, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2019-01-04  3:43 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Chris Murphy, Nazar Mokrynskyi, linux-btrfs

On Thu, Jan 3, 2019 at 6:32 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/1/4 上午9:15, Chris Murphy wrote:
> > If you use btrfs-image -ss option, there won't be any sensitive
> > information included. Files are hashed. Some short name files or dirs
> > can't be hashed (you'll see a warning) and those are replaced with
> > random garbage instead. There is only metadata with the image, no user
> > data.
>
> Please don't advice -ss for btrfs-image. It's super slow and will easily
> cause tons of noise when running btrfs check on it.

I thought in recent btrfs-progs that -ss was quite a bit faster. The
last time I used it I don't think it was much slower than -s. Anyway I
don't have a preference between hash or garbage, but I don't think
it's realistic for users to give out filenames in images. So is -s OK?



-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Unrecoverable btrfs corruption (backref bytes do not match extent backref)
  2019-01-04  3:43     ` Chris Murphy
@ 2019-01-04  4:31       ` Qu Wenruo
  0 siblings, 0 replies; 10+ messages in thread
From: Qu Wenruo @ 2019-01-04  4:31 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Nazar Mokrynskyi, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 1453 bytes --]



On 2019/1/4 上午11:43, Chris Murphy wrote:
> On Thu, Jan 3, 2019 at 6:32 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2019/1/4 上午9:15, Chris Murphy wrote:
>>> If you use btrfs-image -ss option, there won't be any sensitive
>>> information included. Files are hashed. Some short name files or dirs
>>> can't be hashed (you'll see a warning) and those are replaced with
>>> random garbage instead. There is only metadata with the image, no user
>>> data.
>>
>> Please don't advice -ss for btrfs-image. It's super slow and will easily
>> cause tons of noise when running btrfs check on it.
> 
> I thought in recent btrfs-progs that -ss was quite a bit faster. The
> last time I used it I don't think it was much slower than -s. Anyway I
> don't have a preference between hash or garbage, but I don't think
> it's realistic for users to give out filenames in images. So is -s OK?

For -s/-ss the problem is, btrfs check will report tons of hash
mismatch, and recent kernel will even refuse to mount due to enhanced
dir item hash check.

The extra problem of -ss is, it can't ensure all filenames to have a
conflict hash, so it's not that much different from -s.


For User don't want to give any filenames, then don't give any files names.
We could find a way to get needed info just using btrfs ins dump-tree,
especially for this case where only one error is reported.

Thanks,
Qu

> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Unrecoverable btrfs corruption (backref bytes do not match extent backref)
  2019-01-04  1:32   ` Qu Wenruo
  2019-01-04  1:36     ` Hans van Kranenburg
  2019-01-04  3:43     ` Chris Murphy
@ 2019-01-04 23:04     ` Nazar Mokrynskyi
  2019-01-05  1:18       ` Qu Wenruo
  2 siblings, 1 reply; 10+ messages in thread
From: Nazar Mokrynskyi @ 2019-01-04 23:04 UTC (permalink / raw)
  To: Qu Wenruo, Chris Murphy; +Cc: linux-btrfs

04.01.19 03:32, Qu Wenruo пише:
> Please provide the dump of the following command:
>
> # btrfs ins dump-tree -t extent | grep 3114475520 -C 20
root@nazarpc-Standard-PC-Q35-ICH9-2009:~# btrfs ins dump-tree -t extent /dev/vdc | grep 3114475520 -C 20
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 146 key (3114242048 EXTENT_ITEM 36864) itemoff 10402 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 147 key (3114278912 EXTENT_ITEM 36864) itemoff 10365 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 148 key (3114315776 EXTENT_ITEM 36864) itemoff 10328 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 149 key (3114352640 EXTENT_ITEM 45056) itemoff 10291 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 150 key (3114397696 EXTENT_ITEM 40960) itemoff 10254 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 151 key (3114438656 EXTENT_ITEM 36864) itemoff 10217 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 152 key (3114475520 EXTENT_ITEM 4096) itemoff 10193 itemsize 24
        refs 2 gen 1701147 flags DATA
    item 153 key (3114475520 EXTENT_ITEM 36864) itemoff 10169 itemsize 24
        refs 1 gen 1712966 flags DATA
    item 154 key (3114475520 SHARED_DATA_REF 311408951296) itemoff 10165 itemsize 4
        shared data backref count 1
    item 155 key (3114475520 SHARED_DATA_REF 342561947648) itemoff 10161 itemsize 4
        shared data backref count 1
    item 156 key (3114475520 SHARED_DATA_REF 348547874816) itemoff 10157 itemsize 4
        shared data backref count 1
    item 157 key (3114508288 EXTENT_ITEM 4096) itemoff 10120 itemsize 37
        refs 1 gen 1713581 flags DATA
        shared data backref parent 311693983744 count 1
    item 158 key (3114512384 EXTENT_ITEM 45056) itemoff 10083 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 159 key (3114557440 EXTENT_ITEM 110592) itemoff 10046 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 160 key (3114668032 EXTENT_ITEM 102400) itemoff 10009 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 161 key (3114770432 EXTENT_ITEM 36864) itemoff 9972 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 162 key (3114807296 EXTENT_ITEM 40960) itemoff 9935 itemsize 37
        refs 1 gen 1712966 flags DATA
        shared data backref parent 311408951296 count 1
    item 163 key (3114848256 EXTENT_ITEM 36864) itemoff 9898 itemsize 37
--
    item 194 key (311447486464 METADATA_ITEM 0) itemoff 13136 itemsize 33
        refs 1 gen 1713594 flags TREE_BLOCK|FULL_BACKREF
        tree block skinny level 0
        shared block backref parent 348437413888
    item 195 key (311447502848 METADATA_ITEM 0) itemoff 13094 itemsize 42
        refs 2 gen 1713200 flags TREE_BLOCK|FULL_BACKREF
        tree block skinny level 0
        shared block backref parent 924172288
        shared block backref parent 871956480
    item 196 key (311447519232 METADATA_ITEM 0) itemoff 13043 itemsize 51
        refs 3 gen 1713594 flags TREE_BLOCK|FULL_BACKREF
        tree block skinny level 0
        shared block backref parent 348542287872
        shared block backref parent 348542271488
        shared block backref parent 348542238720
    item 197 key (311447535616 METADATA_ITEM 0) itemoff 13001 itemsize 42
        refs 2 gen 1713311 flags TREE_BLOCK|FULL_BACKREF
        tree block skinny level 0
        shared block backref parent 311441817600
        shared block backref parent 773259264
    item 198 key (311447552000 METADATA_ITEM 0) itemoff 12959 itemsize 42
        refs 2 gen 1713594 flags TREE_BLOCK|FULL_BACKREF
        tree block skinny level 0
        shared block backref parent 1068089344
        shared block backref parent 1067450368
    item 199 key (311447568384 METADATA_ITEM 0) itemoff 12917 itemsize 42
        refs 2 gen 1713499 flags TREE_BLOCK|FULL_BACKREF
        tree block skinny level 0
        shared block backref parent 310977855488
        shared block backref parent 941277184
    item 200 key (311447584768 METADATA_ITEM 0) itemoff 12884 itemsize 33
        refs 1 gen 1713594 flags TREE_BLOCK|FULL_BACKREF
        tree block skinny level 0
        shared block backref parent 310736814080
    item 201 key (311447601152 METADATA_ITEM 0) itemoff 12842 itemsize 42
        refs 2 gen 1713499 flags TREE_BLOCK|FULL_BACKREF
        tree block skinny level 0
        shared block backref parent 310977855488
        shared block backref parent 941277184
    item 202 key (311447617536 METADATA_ITEM 0) itemoff 12800 itemsize 42
        refs 2 gen 1713594 flags TREE_BLOCK|FULL_BACKREF

Actually, I removed Docker's snapshots after making image, but most others are not there anymore.

This is how many snapshots are in saved image:

root@nazarpc-Standard-PC-Q35-ICH9-2009:~# btrfs subvolume list /media/nazar-pc/5170aca4-061a-4c6c-ab00-bd7fc8ae6030
ID 260 gen 1713629 top level 5 path docker
ID 262 gen 1713624 top level 5 path backup
ID 276 gen 1713629 top level 5 path root
ID 292 gen 1713630 top level 5 path home
ID 304 gen 1713631 top level 5 path web
ID 95835 gen 1713630 top level 260 path docker/btrfs/subvolumes/ffdbac568451ccb1815901d943757292bdf7070c0e4e69d0b7e503de8c8d109a
ID 95836 gen 1713630 top level 260 path docker/btrfs/subvolumes/f712c6a7e463ef960c271d347bcf68b9d7a0d79f4c0ffcb9800592ca03617511
ID 95837 gen 1713630 top level 260 path docker/btrfs/subvolumes/c00c2e1ae11b73a65a14b82487dcb8da6998b99592d2d0c1a085ced8cd14aed3
ID 95838 gen 1713630 top level 260 path docker/btrfs/subvolumes/af2539cc2aea09cab2af9ec66641ebd30ab3fb694dcf06a5bb1ab60d79f83d80
ID 95839 gen 1713630 top level 260 path docker/btrfs/subvolumes/877184e84a9e36e0f12d6141192dfb48bcf636dd06380316203d64654ece1e6c
ID 95840 gen 1713630 top level 260 path docker/btrfs/subvolumes/86ddd718f65a77f69d1f3b86cee75a0fdb46fbc1e428e5b84c6b9eab602f97dc
ID 95841 gen 1713630 top level 260 path docker/btrfs/subvolumes/6b9be1d8e13a9d325156c01cf49701dc6c81a38c2ff3678569d8995e05e2bd31
ID 95842 gen 1713630 top level 260 path docker/btrfs/subvolumes/a46b4cbac9ea7e4fba6b6251ca14b0f91a53f978f8afb135a123217dae000539
ID 95843 gen 1713630 top level 260 path docker/btrfs/subvolumes/51f8c4ceeef20fe9f42fb08f392a2f36414b7f77aa6d204ec9355c34a4b32ff2
ID 95844 gen 1713630 top level 260 path docker/btrfs/subvolumes/4faf351616578a66f90d76af02370fbc0637a74b21d6c3229d396c2bd39445a8
ID 95845 gen 1713630 top level 260 path docker/btrfs/subvolumes/9bdccb99424776101554f95ee1d37d407c478828809d2dd090cf96d59f37e1ef
ID 95846 gen 1713630 top level 260 path docker/btrfs/subvolumes/43892654913d5eba5177409bdf7efb20095528b5d16aa40632cf7eb1f237d9a3
ID 95847 gen 1713630 top level 260 path docker/btrfs/subvolumes/bb6a2bc39630444f7ceca7de77bfc2655f71f690e999863726c4fb11c9a4efe9
ID 95848 gen 1713630 top level 260 path docker/btrfs/subvolumes/e50824c751f4ea9c1a8a85110ef2b7e988d0e65838e315f8dac5fbb04cef2088
ID 95849 gen 1713630 top level 260 path docker/btrfs/subvolumes/3213e15c2b0ccd9e77b4c2d6006588d1bb098a7adf24aa8f3f1d584264bb9cb4
ID 95850 gen 1713630 top level 260 path docker/btrfs/subvolumes/863f0d6874f176b4970fcb801688f11dd766ad6c6087e49c95629161831315bd
ID 95851 gen 1713629 top level 260 path docker/btrfs/subvolumes/74f50cda809faa390beb8f6fc16104a63ee31730dc38511cd7eba2313da672db
ID 95852 gen 1713630 top level 260 path docker/btrfs/subvolumes/04f5135c877ca52de01824225a53592ecf07805c6bc589b849340d9118c74d18
ID 95853 gen 1713630 top level 260 path docker/btrfs/subvolumes/5826d6e8a99b07bd56d5ccc7577a6e45cbbf1c8154150a12eb50014e8bacf7f5
ID 95854 gen 1713630 top level 260 path docker/btrfs/subvolumes/052412577db829d8c141f70a45c36cf4bac74a04fe489a0ffda730f7cb056b34
ID 95855 gen 1713630 top level 260 path docker/btrfs/subvolumes/b62abd664c21f05e01ad474c4006388d6484260f79ea20070862cbb4fc9fbb21
ID 95856 gen 1713630 top level 260 path docker/btrfs/subvolumes/95de383dca2eea1fc341f3a288e782380199a27b40870af48ac9317555a6ee1b
ID 95857 gen 1713630 top level 260 path docker/btrfs/subvolumes/7ad705b5dc4a0ec63f5c024091737a834df790f88d382f2361312368532c8cf4
ID 95858 gen 1713630 top level 260 path docker/btrfs/subvolumes/303e07881095fb33f66bc060bcb66f61848d7d9802554bcc6a449706df5c8b8d
ID 95859 gen 1713630 top level 260 path docker/btrfs/subvolumes/29727603e0672b5718ca75e1a43cdf944478504d8ba9714d924c2cd915acb992
ID 95860 gen 1713630 top level 260 path docker/btrfs/subvolumes/7f79c537b052c65586a320faf876a035146d921f30d7f0274122ffb4cf4034c5
ID 95861 gen 1713630 top level 260 path docker/btrfs/subvolumes/51888ae75a9a68feb3d7a153389326478b41ef54636b27633171a231ab41f13b
ID 95862 gen 1713630 top level 260 path docker/btrfs/subvolumes/76dfcd85bfc6913a35793994338b204ef4a317667a2dc610a4386b6941ba7d70
ID 95863 gen 1713630 top level 260 path docker/btrfs/subvolumes/beccb3af18174edc2ab53623c91220f95a35d3e6815720fc785b104f7d595cdf
ID 95864 gen 1713630 top level 260 path docker/btrfs/subvolumes/90afc60c1562a138675126571bb5e6e4a21aac7426d4bc2437745bfb22706864
ID 95865 gen 1713630 top level 260 path docker/btrfs/subvolumes/57230f7b480ea75a9c7b1d151c6ab3033aed2dedbc66a661159e18a8c9f9ad11
ID 95866 gen 1713630 top level 260 path docker/btrfs/subvolumes/1033d6b473339f8c892955b0f4c95b994c592fba4820cb46bfcd143a011ddc2a
ID 95867 gen 1713630 top level 260 path docker/btrfs/subvolumes/39c801419e4364e40f0ec17ce4d9fdbaf0ed391c3d5a448ae8e98f008c0de9b6
ID 95868 gen 1713630 top level 260 path docker/btrfs/subvolumes/a4c391b646efcbe81cdeb42c8fee106489c5748d766c715e65043c2ba65f270c
ID 95869 gen 1713630 top level 260 path docker/btrfs/subvolumes/e3f018295759a8d341a62001d660684eca4f5d364968904400c882c5f3aadddb
ID 95870 gen 1713630 top level 260 path docker/btrfs/subvolumes/98a168aebd68791197960fe00dcd81daf67544d4ab501cf25fac17acdd217ef9
ID 95871 gen 1713630 top level 260 path docker/btrfs/subvolumes/53893766624eddb1fbf634f7ce96d1bda6206f4eb8736dca9ddbdfbf6fcdcef3
ID 95872 gen 1713630 top level 260 path docker/btrfs/subvolumes/6ae644dde3329c27731942c5b7ce7b1c671d4736035ac24daee01e5b955430c2
ID 95873 gen 1713630 top level 260 path docker/btrfs/subvolumes/7151db1ee08596c29912927eb0fc363b90d2697318aa092af958ef9f32525a2f
ID 95874 gen 1713630 top level 260 path docker/btrfs/subvolumes/a8f7dd0af203b9c32b40a2e5251e418b8db0c15699eba1ffea29eb341ac2598d
ID 95875 gen 1713630 top level 260 path docker/btrfs/subvolumes/12e4cd882f4ff355096533cba3aa07482546aa2936532d14fab9262e26fbfa9e
ID 95876 gen 1713630 top level 260 path docker/btrfs/subvolumes/b6d6c46e3f12f77be010875343b0b8fa1ce49f2c178e3a85db05118add45955d
ID 95877 gen 1713630 top level 260 path docker/btrfs/subvolumes/dfdba1e04975d6dd51a436416729729242cc1cba4ee316764915c6ce9db24147
ID 95878 gen 1713630 top level 260 path docker/btrfs/subvolumes/43d7988074f8f24a32ffd63f4064ee566c304b4f14f4d3b9727b9afe62441cbb
ID 95879 gen 1713630 top level 260 path docker/btrfs/subvolumes/18ea4a7ba915d16c34fee930e68f78a0a40881b5337faca6a796f686cd677258
ID 95880 gen 1713630 top level 260 path docker/btrfs/subvolumes/a897caeb0cc8d0b50bfa411023ac90ca7732556480db6e5e35fe7e418a132e9a
ID 95881 gen 1713630 top level 260 path docker/btrfs/subvolumes/a61f155a2ab9bcb7ba9db87615412a254a997c17631d3dcef20d4de62b1884aa
ID 95882 gen 1713630 top level 260 path docker/btrfs/subvolumes/0fd0826bf2ea8ba911cee310a2fdb8442e67145019f3b05c17ad63bebb95a04c
ID 95883 gen 1713630 top level 260 path docker/btrfs/subvolumes/e6852947f89f311169160b37c8ab68bc7c8d6e5b02c32009e32d954f0a98d0e8
ID 95884 gen 1713630 top level 260 path docker/btrfs/subvolumes/2881cba9266fc5b3b91f7483c44f6a23ca82d274349c382d37157827cf93a712
ID 95885 gen 1713630 top level 260 path docker/btrfs/subvolumes/064033de33aa3f4ae56264cfc17abd023fceea42c11a10bb40f4e4365f38d991
ID 95886 gen 1713630 top level 260 path docker/btrfs/subvolumes/b637b72b1fbe011f3fe394e8541b45f799da6363bae5eaaa98eba522efad102d
ID 95887 gen 1713630 top level 260 path docker/btrfs/subvolumes/a9e19edc40fbccb05d974050bf0c6b879120f93e2542f7242d728a5f9560cc92
ID 95888 gen 1713630 top level 260 path docker/btrfs/subvolumes/5c4728344c197f4345b8852d47b4f08cda2248515018ca9e67e057c7798fee15
ID 95889 gen 1713612 top level 260 path docker/btrfs/subvolumes/239d27fd2d240d9ce0cc06c7cb2d9423da0fcbf78eb20727d5fe7ae0773afd10
ID 95890 gen 1713612 top level 260 path docker/btrfs/subvolumes/a5fcd71d7ebcb5f08ecad2bb19248fe4c5d2bd508a522b0d4989a14f5821ad9d
ID 95891 gen 1713612 top level 260 path docker/btrfs/subvolumes/e3868763aa1c5e58966ea6546630f66f9b5d2be4618631210e626bed2852e7ba
ID 95892 gen 1713612 top level 260 path docker/btrfs/subvolumes/a4c2bea28c984b15394a2aacc46c562c18b73b78dd8c8885275bcdb868a61db8
ID 95893 gen 1713629 top level 260 path docker/btrfs/subvolumes/e53197d394796fb2014d88bb36a2afd495d5388610933fd33df59ebf959ae373
ID 95894 gen 1713629 top level 260 path docker/btrfs/subvolumes/59f63759167cd047cbc865cd312f4e9db11ab331d83d97bade10aea7a5eca040
ID 95895 gen 1713629 top level 260 path docker/btrfs/subvolumes/fd334a41a4b69882766a9edf8dbcb6f94620a537be28b11a352d7171f98a92d7
ID 95896 gen 1713629 top level 260 path docker/btrfs/subvolumes/983bf99b2469983aa1a48133e0ce4aa539bd0c13f9731bf63738f138f127f4c9
ID 95897 gen 1713629 top level 260 path docker/btrfs/subvolumes/2cb616b528e1db233c3072e36743087b06ccd4897b48d3ea251e703bd74c92f3
ID 95898 gen 1713629 top level 260 path docker/btrfs/subvolumes/d24d294e693fd69ef9d93800fd79c6c4a9932edd5a984529f129c9aa0b039996
ID 95899 gen 1713629 top level 260 path docker/btrfs/subvolumes/01ab2e1f2dd98b649b986cb629ceeafa64645574f177691759ff59b809e8d6f9
ID 95900 gen 1713629 top level 260 path docker/btrfs/subvolumes/05d8a2c9102f4f3733d5bbc7ee798ea111e883813e01555b1e8ee6fe553e2aae
ID 95901 gen 1713629 top level 260 path docker/btrfs/subvolumes/e20ae2d38e20fd3b116d2ef30adc5aed769cd44498eeb41597dc808db3ed354f
ID 95902 gen 1713629 top level 260 path docker/btrfs/subvolumes/e3994ccb36b37c60f1492a9d4451c38854c7d56870a666fd236418bb7ea58b86
ID 95940 gen 1713612 top level 260 path docker/btrfs/subvolumes/89827536e353822e12af490df3b905202ec19ceab461f768093427316f11db25
ID 95941 gen 1713612 top level 260 path docker/btrfs/subvolumes/6582fd4bda2613b57e701075453d5da26fc43428d8f89366aeb894182076e4c1
ID 95942 gen 1713612 top level 260 path docker/btrfs/subvolumes/0ef11bb800cf623c638e35631ce3380bc54ed52416c3b12d4453bdb8e9e72f36
ID 95943 gen 1713612 top level 260 path docker/btrfs/subvolumes/d872917688a59499315d99d8956fe61269b37442b6f5d7009810ef70aa975f5b
ID 95946 gen 1713612 top level 260 path docker/btrfs/subvolumes/9e3189b388349b86e7a90c0009f1bc29c9cdf2b23555c7915e57b88fc8f5e58e
ID 95948 gen 1713612 top level 260 path docker/btrfs/subvolumes/24c8eee289cdeb0eeba82d108a140eb3014e8ad3f705be976dd4d07724e0dac7
ID 95962 gen 1713612 top level 260 path docker/btrfs/subvolumes/fd90fececfc124c75d25b6a45d7de190a75873b1826229bbd0ffa43dbb9ea9f0
ID 95963 gen 1713612 top level 260 path docker/btrfs/subvolumes/971894906b3215eee0aa68ef7c10b5ba9409c9139642dfbf7d9fc11c1fa340e8-init
ID 95964 gen 1713612 top level 260 path docker/btrfs/subvolumes/971894906b3215eee0aa68ef7c10b5ba9409c9139642dfbf7d9fc11c1fa340e8
ID 95969 gen 1713612 top level 260 path docker/btrfs/subvolumes/0e77a13302d101993fdb4c183ed51d4fe88e54a54300aeeadedbe6c0b06a0668
ID 95980 gen 1713612 top level 260 path docker/btrfs/subvolumes/40f5cdf8f31975a7a4ed2f7ff90cb38dfac1cda7f21c3e9bfec63cd61cfd6962
ID 95982 gen 1713612 top level 260 path docker/btrfs/subvolumes/f72edaf8d9aeefb5baf0b517b8ebf5db162e4319c09dff99cf586d30cfb224ce
ID 95985 gen 1713612 top level 260 path docker/btrfs/subvolumes/c409c1681b3ffa5b07398614b19565b03c1d6120eea00a53a49e076a5acd07b7
ID 95993 gen 1713612 top level 260 path docker/btrfs/subvolumes/650ed417ed4043c8d645da64f25fdd721753562e1268f4b22a28c7ee0bd93a61
ID 95998 gen 1713612 top level 260 path docker/btrfs/subvolumes/a286c89dc14486449cf876fee1ef74c5861a3c7695427fa2ba7e7c44e5d9127d
ID 96000 gen 1713612 top level 260 path docker/btrfs/subvolumes/5a60ed8e9010472b000bb447593a7b33a4814a3bb2f18b152bf96f15faa4c716
ID 96004 gen 1713612 top level 260 path docker/btrfs/subvolumes/9edb19a4b7f55c61f86b270545261c11adba2253a54c921e6c4693a02c016552
ID 96010 gen 1713612 top level 260 path docker/btrfs/subvolumes/51750877baa64ce5837bd4abac43bd2c7f2acb0d6751329a3f40e58527d10030
ID 96024 gen 1713612 top level 260 path docker/btrfs/subvolumes/63ef0af970ebbbcc187a189f7776995073e0d93b67679b5e4cf51fa062522487
ID 96028 gen 1713612 top level 260 path docker/btrfs/subvolumes/7f1a97cf12a0466043ef034f6511856ad792c413deeb8d35bfa5ba4e461a45c6
ID 96034 gen 1713612 top level 260 path docker/btrfs/subvolumes/accb661f6cc963b8499459d84769f8dacae64bc297c22dd0001d548e563a004f
ID 96038 gen 1713612 top level 260 path docker/btrfs/subvolumes/e906249d00ace48dc62d712c3fe58480e369eb40385747b78b60b29cabea1ed9
ID 96045 gen 1713612 top level 260 path docker/btrfs/subvolumes/f7db0c3e6b9a1066858c6725899f320861af4c9565ecc4aa75cfd0144b9547a2
ID 96052 gen 1713612 top level 260 path docker/btrfs/subvolumes/816939329badd8e467fda11160d08ad41f564b3b727196688315fafb7d52c565
ID 96063 gen 1713612 top level 260 path docker/btrfs/subvolumes/40ed03dadfdfbf64ca087429d6eb54a46c3bcf6c2d97c109a7a3e6b98ba52cd5
ID 96070 gen 1713612 top level 260 path docker/btrfs/subvolumes/fc72dbaeaa83f6ec13051d089a6a0643387ea90dd8ca99ca86ab8237bbf42e90
ID 96077 gen 1713612 top level 260 path docker/btrfs/subvolumes/d5819337c3f0dc70db9ae83983521000dfc4ea2b4f3db88a6beb78b763032cd8
ID 96102 gen 1713612 top level 260 path docker/btrfs/subvolumes/ad0a4de93ec60744aa279f9744ac88034bb283fac3c4b3c1c53069907645729c
ID 96104 gen 1713612 top level 260 path docker/btrfs/subvolumes/b58bf85e21f03584c4a37d4fb784d93adb76da56d9f358b5100c4382cc01f392
ID 96108 gen 1713612 top level 260 path docker/btrfs/subvolumes/5b9c35e3b7be76d54b47ac33d8bf3593186055a199ce1c13b6e4579efc15e52d
ID 96109 gen 1713612 top level 260 path docker/btrfs/subvolumes/c9736aa0a2879577d23a9eb73c02c6a96922f5e5d6f56ebe2b72ae825cd6e0df
ID 96110 gen 1713612 top level 260 path docker/btrfs/subvolumes/ae1df26b60a43b00167cc20abc908cd506b3aefa67a9d859be6f7d0ea0f34d9b
ID 96111 gen 1713612 top level 260 path docker/btrfs/subvolumes/ad0c8141c44664862ffafa498222e89009417c4964fcc676335ed1f73b3fc381
ID 96112 gen 1713612 top level 260 path docker/btrfs/subvolumes/bcef81f50aa1eeaec04e722a46f8ac2e1c27ff5d78aa4eead71ff5cdafa91a72
ID 96113 gen 1713612 top level 260 path docker/btrfs/subvolumes/bc829c21d30155c1ceaaea62a22244e3f9cc164ba55b148385b1d1519047b113
ID 96114 gen 1713612 top level 260 path docker/btrfs/subvolumes/85b100156d651e7b86435bd87e7a23b8f29f27e55325d4e8e87ab4399628a67e
ID 96115 gen 1713612 top level 260 path docker/btrfs/subvolumes/62b4c9d0fcfe27e00d85b67e8e21055c7a6edb027573f04d00fe4009ba4069ba
ID 96116 gen 1713612 top level 260 path docker/btrfs/subvolumes/2360dcbfe43b9ce4b005cda4fac95e5a779121307bf945ab5142809083d6fa83
ID 96117 gen 1713615 top level 260 path docker/btrfs/subvolumes/a4bf81f0e52b592284f4ed6ef345c81a2d40d72ad1cccf4664ee905d33929a00
ID 96118 gen 1713615 top level 260 path docker/btrfs/subvolumes/0220264b4198349182c0a646873b33a04130796b8edc46eb0616d684bd69b14e
ID 96377 gen 1713612 top level 260 path docker/btrfs/subvolumes/ce058663b4673033d540b13434c0bb8ce4338c0adc554cbbc0492a25aad8ac56
ID 96383 gen 1713612 top level 260 path docker/btrfs/subvolumes/ff2722dffcd342d8671f2d0a351a0be07dcb533d47a42f903006042ad33d5aeb
ID 96385 gen 1713612 top level 260 path docker/btrfs/subvolumes/a9b842e7820f58fe5592311c67c7c8b4e55047b3745ec01419199678e753de49
ID 96394 gen 1713612 top level 260 path docker/btrfs/subvolumes/c5ea8cf2bcc0264a7e0f0da277c4d2c6ada96a5bce2a3d70ad4c07a68b309beb
ID 96396 gen 1713612 top level 260 path docker/btrfs/subvolumes/5fbbaa493e80243dfc2fd8d47eb0cd77d043a4a70a1e45e4a9a8139e8378dd95
ID 96403 gen 1713612 top level 260 path docker/btrfs/subvolumes/736845c54caf6dc869c3f927b221164147c761d0e0dd307a47193a05e8888474
ID 96413 gen 1713629 top level 260 path docker/btrfs/subvolumes/b71cb06bb700164f1fe74b6266f2be60e8814d55e6c219fc7cc35da66c0ff13e
ID 96416 gen 1713629 top level 260 path docker/btrfs/subvolumes/d460afdfbc3a4387ebb51af06a510f75e1fc83299dfc104fdb351d5deef57ef7
ID 96418 gen 1713629 top level 260 path docker/btrfs/subvolumes/4d10756cdbb6f32019df4e1b79b28ff03a38f28450188c6920f58277e3b43423
ID 96463 gen 1713612 top level 260 path docker/btrfs/subvolumes/d8de08aa7f39e4bea0dca018ad82772eb5f5875a3a02b146c783214fc1b16b54
ID 96480 gen 1713612 top level 260 path docker/btrfs/subvolumes/7ade6192077ebfda9342fe3c90be99516f1d1bd37db22c73b7f44a38f3d26662
ID 97017 gen 1713630 top level 260 path docker/btrfs/subvolumes/a18d99e9187bc8694e95998f452a95326e15081a1bc1ca201070cdee70aa84e6-init
ID 97018 gen 1713630 top level 260 path docker/btrfs/subvolumes/a18d99e9187bc8694e95998f452a95326e15081a1bc1ca201070cdee70aa84e6
ID 97019 gen 1713630 top level 260 path docker/btrfs/subvolumes/521dcbc07fe6ca8a0a57c54b1b6e091707c11a0a113fc8df0da78d85e5d9030e-init
ID 97020 gen 1713630 top level 260 path docker/btrfs/subvolumes/521dcbc07fe6ca8a0a57c54b1b6e091707c11a0a113fc8df0da78d85e5d9030e
ID 97021 gen 1713630 top level 260 path docker/btrfs/subvolumes/0eede4f5eae0b6484f1822f53461745e8b0db207645d487d3d5495256a81c173-init
ID 97022 gen 1713630 top level 260 path docker/btrfs/subvolumes/0eede4f5eae0b6484f1822f53461745e8b0db207645d487d3d5495256a81c173
ID 97023 gen 1713630 top level 260 path docker/btrfs/subvolumes/f41d2d4083b4e64fbbbb67b8dfdde80fa7016019115f68c9082176fc5a307726-init
ID 97024 gen 1713630 top level 260 path docker/btrfs/subvolumes/f41d2d4083b4e64fbbbb67b8dfdde80fa7016019115f68c9082176fc5a307726
ID 97025 gen 1713630 top level 260 path docker/btrfs/subvolumes/8b662c13372e0f5cc179de1053641e80b13adafe8a9127c3da37518bdc70ab11-init
ID 97026 gen 1713630 top level 260 path docker/btrfs/subvolumes/8b662c13372e0f5cc179de1053641e80b13adafe8a9127c3da37518bdc70ab11
ID 97027 gen 1713629 top level 260 path docker/btrfs/subvolumes/aa451019b78cd5093a8cf744ea05695f0acbec591ddb30b3da046373b34b6601-init
ID 97028 gen 1713629 top level 260 path docker/btrfs/subvolumes/aa451019b78cd5093a8cf744ea05695f0acbec591ddb30b3da046373b34b6601
ID 97029 gen 1713630 top level 260 path docker/btrfs/subvolumes/a0a668bcb9f35351a8a071644675de88b23e322f22a1685b7063b885583b538c-init
ID 97030 gen 1713630 top level 260 path docker/btrfs/subvolumes/a0a668bcb9f35351a8a071644675de88b23e322f22a1685b7063b885583b538c
ID 97031 gen 1713630 top level 260 path docker/btrfs/subvolumes/1e2e83415bf68b273ff1e2000dbf45c8f97ad00a8f2c0bd9fb3dc53e9aa68f46-init
ID 97032 gen 1713630 top level 260 path docker/btrfs/subvolumes/1e2e83415bf68b273ff1e2000dbf45c8f97ad00a8f2c0bd9fb3dc53e9aa68f46
ID 97033 gen 1713630 top level 260 path docker/btrfs/subvolumes/7491bbb0ced8fbec44bd88aef8d3518deebde66ffffa43c2abc745c5d634a7fa-init
ID 97034 gen 1713630 top level 260 path docker/btrfs/subvolumes/7491bbb0ced8fbec44bd88aef8d3518deebde66ffffa43c2abc745c5d634a7fa
ID 97035 gen 1713630 top level 260 path docker/btrfs/subvolumes/41db7414816485ac4f37ab7c8be6ef0d453987163b1a0678d79d95a913b03785-init
ID 97036 gen 1713630 top level 260 path docker/btrfs/subvolumes/41db7414816485ac4f37ab7c8be6ef0d453987163b1a0678d79d95a913b03785
ID 97037 gen 1713630 top level 260 path docker/btrfs/subvolumes/bb001e6f3528d42a90cf319ef3fc6387bada9fc87873378dc13fe9eab28c7881-init
ID 97038 gen 1713630 top level 260 path docker/btrfs/subvolumes/bb001e6f3528d42a90cf319ef3fc6387bada9fc87873378dc13fe9eab28c7881
ID 97039 gen 1713630 top level 260 path docker/btrfs/subvolumes/0bc2d3d88d4ea68253a0f98d899e3f357e02161d98b9d1345bfbc6934f8be571-init
ID 97040 gen 1713630 top level 260 path docker/btrfs/subvolumes/0bc2d3d88d4ea68253a0f98d899e3f357e02161d98b9d1345bfbc6934f8be571
ID 97041 gen 1713630 top level 260 path docker/btrfs/subvolumes/f849c133333629d7a160d509454b7a9531c6ac7769eea8acbb797d19987c2a94-init
ID 97042 gen 1713630 top level 260 path docker/btrfs/subvolumes/f849c133333629d7a160d509454b7a9531c6ac7769eea8acbb797d19987c2a94
ID 97043 gen 1713630 top level 260 path docker/btrfs/subvolumes/81b1db978e6418feb97dd8239f43316190fad4ffdf84d4862409aaaac38f8297-init
ID 97044 gen 1713630 top level 260 path docker/btrfs/subvolumes/81b1db978e6418feb97dd8239f43316190fad4ffdf84d4862409aaaac38f8297
ID 97045 gen 1713630 top level 260 path docker/btrfs/subvolumes/be1a4c4d2604747b5e4a53c3802ba76e0cf3b508bdffae025f5f99afee7959ac-init
ID 97046 gen 1713630 top level 260 path docker/btrfs/subvolumes/be1a4c4d2604747b5e4a53c3802ba76e0cf3b508bdffae025f5f99afee7959ac
ID 97047 gen 1713630 top level 260 path docker/btrfs/subvolumes/e285e267b9cfe7be2bb006ad77e96d722b0170acdfe6fc14adb09df5b6f39ed0-init
ID 97048 gen 1713630 top level 260 path docker/btrfs/subvolumes/e285e267b9cfe7be2bb006ad77e96d722b0170acdfe6fc14adb09df5b6f39ed0
ID 97049 gen 1713630 top level 260 path docker/btrfs/subvolumes/f733d4105d01a2c756c3851e40cacbae15c73d5dda7b6b4581defc4c21afbc51-init
ID 97050 gen 1713630 top level 260 path docker/btrfs/subvolumes/f733d4105d01a2c756c3851e40cacbae15c73d5dda7b6b4581defc4c21afbc51
ID 97069 gen 1713630 top level 260 path docker/btrfs/subvolumes/9061d1e447444cc393b98296da15d04bb46df7b2eb08dd07f24d13f1c9b02979-init
ID 97070 gen 1713630 top level 260 path docker/btrfs/subvolumes/9061d1e447444cc393b98296da15d04bb46df7b2eb08dd07f24d13f1c9b02979
ID 97071 gen 1713612 top level 260 path docker/btrfs/subvolumes/25400bf8c6282f65acea92a7a9df96c3d685791d8af1d8ab82e02ea0c2a5da9c-init
ID 97072 gen 1713629 top level 260 path docker/btrfs/subvolumes/25400bf8c6282f65acea92a7a9df96c3d685791d8af1d8ab82e02ea0c2a5da9c
ID 97073 gen 1713630 top level 260 path docker/btrfs/subvolumes/7e08f83629bc85fa992f81d8427ac55783983da05259279b23fa23ec7017f74b-init
ID 97074 gen 1713630 top level 260 path docker/btrfs/subvolumes/7e08f83629bc85fa992f81d8427ac55783983da05259279b23fa23ec7017f74b
ID 97075 gen 1713630 top level 260 path docker/btrfs/subvolumes/0aeb9a675273486d11322032b3557bf4ee71890ed63cf6c0a01c1a7473dda0c5-init
ID 97076 gen 1713630 top level 260 path docker/btrfs/subvolumes/0aeb9a675273486d11322032b3557bf4ee71890ed63cf6c0a01c1a7473dda0c5
ID 97077 gen 1713612 top level 260 path docker/btrfs/subvolumes/79bd3fc05fd102d1558e17abfe5bc03cc9e78677910568b86bf45699b0f8bc85-init
ID 97078 gen 1713629 top level 260 path docker/btrfs/subvolumes/79bd3fc05fd102d1558e17abfe5bc03cc9e78677910568b86bf45699b0f8bc85
ID 97079 gen 1713629 top level 260 path docker/btrfs/subvolumes/0bd9e6f43d09cee4b12fc3ad688e9257c4c563126aef1d0516dd2e7942f53232-init
ID 97080 gen 1713629 top level 260 path docker/btrfs/subvolumes/0bd9e6f43d09cee4b12fc3ad688e9257c4c563126aef1d0516dd2e7942f53232
ID 97081 gen 1713630 top level 260 path docker/btrfs/subvolumes/fa1eed61e46c9e6f54df9e6ed14b2bc278fd7b102bc993498fc80beec4a61b98-init
ID 97082 gen 1713630 top level 260 path docker/btrfs/subvolumes/fa1eed61e46c9e6f54df9e6ed14b2bc278fd7b102bc993498fc80beec4a61b98
ID 97083 gen 1713630 top level 260 path docker/btrfs/subvolumes/33ff574664aaaf20fa197a24eb0c653efad9c59320173e56ecebd2acfad82f90-init
ID 97084 gen 1713630 top level 260 path docker/btrfs/subvolumes/33ff574664aaaf20fa197a24eb0c653efad9c59320173e56ecebd2acfad82f90
ID 97085 gen 1713630 top level 260 path docker/btrfs/subvolumes/9ca22798d12749bf5b32c6f8e4b2574fb408afd6b3d660dd4e166ff03fec6660-init
ID 97086 gen 1713630 top level 260 path docker/btrfs/subvolumes/9ca22798d12749bf5b32c6f8e4b2574fb408afd6b3d660dd4e166ff03fec6660
ID 97087 gen 1713630 top level 260 path docker/btrfs/subvolumes/046c86c737d574cb602208100fe97a3df827b3901819b7004c49700b651f5b1a-init
ID 97088 gen 1713630 top level 260 path docker/btrfs/subvolumes/046c86c737d574cb602208100fe97a3df827b3901819b7004c49700b651f5b1a
ID 98029 gen 1713630 top level 260 path docker/btrfs/subvolumes/7ede9a05f08d9a6f74643ff4d03808976f3af218783f9e88232d2b3bc203905b
ID 98030 gen 1713630 top level 260 path docker/btrfs/subvolumes/97db17d35087b337a08acf0de5841fa9a83cefb3eaa9d207a02553164798e10c
ID 98031 gen 1713630 top level 260 path docker/btrfs/subvolumes/7149efca44a091a3dd22f9cfcf2138ca1289fe441e6552980c6f84a09461637d
ID 98032 gen 1713630 top level 260 path docker/btrfs/subvolumes/352741e50b6cffae6e38e4180e80b5c24a819977c5bb367977b7477905feb01a
ID 98033 gen 1713630 top level 260 path docker/btrfs/subvolumes/db2ed5c843d602a425a859f5e10f12df85ce97501265aaa1efe676d80f805b73
ID 98034 gen 1713630 top level 260 path docker/btrfs/subvolumes/7b240c2ee5538f115028a3f4253fce4db9313bfbd3391302ae15efdd9ff78f75
ID 98035 gen 1713630 top level 260 path docker/btrfs/subvolumes/cd8d1eea5f8b046403e0250121e45cde0f27bcd91151c23fe782aedd09be5568
ID 98038 gen 1713630 top level 260 path docker/btrfs/subvolumes/b4c3214c119614c0edb97d471661d81f85a6b775cb8e1b421ce0934eda20cb6c
ID 98041 gen 1713630 top level 260 path docker/btrfs/subvolumes/983901834ecbd5dd746976655114492cda9e8a1094b9b2791411e9b990534bf8
ID 98044 gen 1713630 top level 260 path docker/btrfs/subvolumes/c662dd78b55de2709090d5e933c7447dc32405ad88d7cbfda36b4d924e865afb
ID 98046 gen 1713630 top level 260 path docker/btrfs/subvolumes/b05fc1c9be4a315905e1d458cf60e4952442ffab98f13e969a1d4e2cfa981dfb
ID 98048 gen 1713630 top level 260 path docker/btrfs/subvolumes/ddc00531fc0d988df9854e72be3e502520568f97294f7447d29933dbe73b6eb1
ID 98050 gen 1713630 top level 260 path docker/btrfs/subvolumes/7e66be870ef9d2430d2df11e2415b913828971bf463f81a2c2c2a3358fa7ff7a
ID 98051 gen 1713630 top level 260 path docker/btrfs/subvolumes/998e34932910864bee6731ab9dd74e1013932db2d3be905013eb084b002a236c-init
ID 98052 gen 1713630 top level 260 path docker/btrfs/subvolumes/998e34932910864bee6731ab9dd74e1013932db2d3be905013eb084b002a236c
ID 98055 gen 1713630 top level 260 path docker/btrfs/subvolumes/1d486f3d11fc499b91b06a5de0909c55cf79948255433a5ee3659f348b273e84
ID 98058 gen 1713630 top level 260 path docker/btrfs/subvolumes/bef8b5a1399e996f7ae6891ca6c30805cfc7efe02d34c5e63ad7468578c28599
ID 98061 gen 1713630 top level 260 path docker/btrfs/subvolumes/2489e86e1774c5c2493866c73d03bd5f001b54a11f95f308b04a58402f76ab88
ID 98064 gen 1713630 top level 260 path docker/btrfs/subvolumes/6e127a4764ceda2036eee475935d5429aaa4e94da71648b6214bc3c804d82d1f
ID 98067 gen 1713630 top level 260 path docker/btrfs/subvolumes/71f3c1bf7a9bac331d293e1eb88b93efd9b1b4443d3f2c1ac1b9a0d2a9fa2119
ID 98070 gen 1713630 top level 260 path docker/btrfs/subvolumes/cd7245595ad10ed466166190c8112466a9040fd2c1be1358563c881cbce6f589
ID 98073 gen 1713630 top level 260 path docker/btrfs/subvolumes/1eb13b66b83e2595361a0bac5879ba2277b10abbb56fac6035a4a12a2831d84b
ID 98074 gen 1713630 top level 260 path docker/btrfs/subvolumes/178fdaf7497b0d4ae18f967856ed9f6e3a8b9dd56b38dcd6035608b34e58a049-init
ID 98075 gen 1713630 top level 260 path docker/btrfs/subvolumes/178fdaf7497b0d4ae18f967856ed9f6e3a8b9dd56b38dcd6035608b34e58a049
ID 98370 gen 1713630 top level 260 path docker/btrfs/subvolumes/fc9b51ff7064d7aec9f5d75182125286a0b2f934f232c31751d93fa731172ce2
ID 98371 gen 1713630 top level 260 path docker/btrfs/subvolumes/7079c54ad525ff9d3b79db3da1bcc5c412635def2ead8e153c022edbb6491b69
ID 98372 gen 1713630 top level 260 path docker/btrfs/subvolumes/a2a397976eb3ee917897d3310fe4a42f4c430104313a046886f2c7e08006446a
ID 98373 gen 1713630 top level 260 path docker/btrfs/subvolumes/45d8a3e056f9b4ac72417697fbf239818254cb4c26c52517243e365c55458954
ID 98374 gen 1713630 top level 260 path docker/btrfs/subvolumes/8ab37b2c06a479c88bdee97b5fb37b021590e54ee6cfca1efca4c011c069f3be
ID 100759 gen 1713612 top level 260 path docker/btrfs/subvolumes/3b6ee88ffb0f8b13bf258b20510a6ebf0dda6b3bf781251adc669afc839804fc
ID 100760 gen 1713612 top level 260 path docker/btrfs/subvolumes/462b65f9961c335a161407eb2359a158a128c132cc26f1f5189bc97f164a7e39
ID 100761 gen 1713612 top level 260 path docker/btrfs/subvolumes/853fe117c50e2a8da30f05381f2c9f768f26a6c1ce89ddfecbc0228542f3d4c8
ID 100762 gen 1713612 top level 260 path docker/btrfs/subvolumes/1b7fafaac58cdfb75779e5b92346ef7cdd3337b0077ac63a92e56fa1eee55f88
ID 100763 gen 1713629 top level 260 path docker/btrfs/subvolumes/c8ce4ebb8a13a6bd82b02e1b67e3672e3bbdca5a0bdf8dfa8d129cc8417e5316
ID 100764 gen 1713629 top level 260 path docker/btrfs/subvolumes/0085ec91bdd612a5d680ec8e94c593dc7b8392a305c985dec57edea5fb4a28a6
ID 100765 gen 1713629 top level 260 path docker/btrfs/subvolumes/2d866553722081e0d601ed07074b6e71003aa66c1d93f4a676624f93f67abdda
ID 100766 gen 1713629 top level 260 path docker/btrfs/subvolumes/9a23c7a047aa4ca56f5ac96839c84399c095e7a5bd49a7274d1d764ea0f405b1
ID 100768 gen 1713629 top level 260 path docker/btrfs/subvolumes/b5714e7b1a0fc8ab5ddedf95ef9018690f0fae663c682a82f3a92d9d7c6dbf7c
ID 100770 gen 1713629 top level 260 path docker/btrfs/subvolumes/4e43a0e4a2df7371857970c46afe36df66986e55dac30a8e7094ba18fad9ec40
ID 100772 gen 1713629 top level 260 path docker/btrfs/subvolumes/e00bba7c8a35bb2555e77c93e39ff32d901b6563dd9d7a26f0ed3054504d50fb
ID 100773 gen 1713629 top level 260 path docker/btrfs/subvolumes/150309de0ff712a965c57ed405f2bac2bc6e98a6af4b77242d6f86e59adfc411-init
ID 100774 gen 1713629 top level 260 path docker/btrfs/subvolumes/150309de0ff712a965c57ed405f2bac2bc6e98a6af4b77242d6f86e59adfc411
ID 100777 gen 1713629 top level 260 path docker/btrfs/subvolumes/d567d9486e0157a126f5b3815e30d0847d18215ed47b80a5c53288c007ff4e18
ID 100779 gen 1713629 top level 260 path docker/btrfs/subvolumes/0d488a155bb1b47296a5d66f057c60be18dfe251c3972f9721b8888c372538ff
ID 100781 gen 1713629 top level 260 path docker/btrfs/subvolumes/5721df60be0ebb2acde2fc33502be35078ef57f65ad55be5b7f19f4dd42dc07c
ID 100783 gen 1713629 top level 260 path docker/btrfs/subvolumes/5b54f53aa0194f11819d457349728eaa34f4143ce84e35126c912022e8b67045
ID 100787 gen 1713629 top level 260 path docker/btrfs/subvolumes/f33ba2397195918fd008f248a107361f1c494b187516f08bb642e835b213ec36
ID 100789 gen 1713629 top level 260 path docker/btrfs/subvolumes/a844e3cb51cd751e40a47e75ae9a178e983e09245ab868daab48dd1424499dc4
ID 100791 gen 1713629 top level 260 path docker/btrfs/subvolumes/9fd12b318b1f5fde03c6ae7f3efcaa87899fd418d208f7141be786611157bc7a
ID 100799 gen 1713629 top level 260 path docker/btrfs/subvolumes/6b85e65b59efc041c10f6d157d7ee12bd1a2281151d1913281978414a11b3873
ID 100800 gen 1713629 top level 260 path docker/btrfs/subvolumes/6daedaa6c6fa8aaa6f87c636c4a25c775e8db6aeb3a5c25b75fd3580205bb4b4-init
ID 100801 gen 1713629 top level 260 path docker/btrfs/subvolumes/6daedaa6c6fa8aaa6f87c636c4a25c775e8db6aeb3a5c25b75fd3580205bb4b4
ID 100832 gen 1713629 top level 260 path docker/btrfs/subvolumes/5bde65bead718cf02cfb4db7c2ea8a8b5aa3dd10ef49101c371a8bb18849d98d
ID 100835 gen 1713629 top level 260 path docker/btrfs/subvolumes/bfdec46b70d82890bd082f4f5a3719ed35aa64ecfae39bade08e76c1ac444692
ID 100838 gen 1713629 top level 260 path docker/btrfs/subvolumes/799d5431b8d43d34b40a78fdf610e81c535d86ecbf2397049195604e9f0f2572
ID 101682 gen 1713612 top level 276 path root/srv
ID 101719 gen 1713629 top level 262 path backup/root/2019-01-03_19:30:01
ID 101720 gen 1713630 top level 262 path backup/home/2019-01-03_19:30:01
ID 101721 gen 1713631 top level 262 path backup/web/2019-01-03_19:30:03

There were many more in backup/* directories, but I only left latest hoping to avoid complete re-creation of filesystem.

I'm using qcow2 with snapshots, so I can do multiple experiments with image at the same time if necessary. Also started `btrfs check --mode=lowmem`, will report once it finishes.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Unrecoverable btrfs corruption (backref bytes do not match extent backref)
  2019-01-04  1:15 ` Chris Murphy
  2019-01-04  1:32   ` Qu Wenruo
@ 2019-01-05  0:13   ` Nazar Mokrynskyi
  1 sibling, 0 replies; 10+ messages in thread
From: Nazar Mokrynskyi @ 2019-01-05  0:13 UTC (permalink / raw)
  To: Chris Murphy, Qu Wenruo; +Cc: linux-btrfs

04.01.19 03:15, Chris Murphy пише:
> What do you get with 'btrfs check --mode=lowmem' ? This is a different
> implementation of check, and might reveal some additional information
> useful to developers. It is wickedly slow however.
root@nazarpc-Standard-PC-Q35-ICH9-2009:~# btrfs check --mode=lowmem /dev/vdb
warning, bad space info total_bytes 2155872256 used 2155876352
warning, bad space info total_bytes 3229614080 used 3229618176
warning, bad space info total_bytes 4303355904 used 4303360000
warning, bad space info total_bytes 5377097728 used 5377101824
warning, bad space info total_bytes 6450839552 used 6450843648
warning, bad space info total_bytes 7524581376 used 7524585472
warning, bad space info total_bytes 8598323200 used 8598327296
warning, bad space info total_bytes 9672065024 used 9672069120
warning, bad space info total_bytes 10745806848 used 10745810944
warning, bad space info total_bytes 11819548672 used 11819552768
warning, bad space info total_bytes 12893290496 used 12893294592
warning, bad space info total_bytes 13967032320 used 13967036416
warning, bad space info total_bytes 15040774144 used 15040778240
warning, bad space info total_bytes 16114515968 used 16114520064
warning, bad space info total_bytes 17188257792 used 17188261888
warning, bad space info total_bytes 18261999616 used 18262003712
warning, bad space info total_bytes 19335741440 used 19335745536
warning, bad space info total_bytes 20409483264 used 20409487360
warning, bad space info total_bytes 21483225088 used 21483229184
warning, bad space info total_bytes 22556966912 used 22556971008
warning, bad space info total_bytes 23630708736 used 23630712832
warning, bad space info total_bytes 24704450560 used 24704454656
warning, bad space info total_bytes 25778192384 used 25778196480
warning, bad space info total_bytes 26851934208 used 26851938304
warning, bad space info total_bytes 27925676032 used 27925680128
warning, bad space info total_bytes 28999417856 used 28999421952
warning, bad space info total_bytes 30073159680 used 30073163776
warning, bad space info total_bytes 31146901504 used 31146905600
warning, bad space info total_bytes 32220643328 used 32220647424
Checking filesystem on /dev/vdb
UUID: 5170aca4-061a-4c6c-ab00-bd7fc8ae6030
checking extents
checking free space cache
checking fs roots
ERROR: root 304 INODE REF[274921, 256895] name 25da95e3e893bb2fa69a2f0acd77bfe725626a1e filetype 1 missing
ERROR: root 304 EXTENT_DATA[910393 4096] gap exists, expected: EXTENT_DATA[910393 25]
ERROR: root 304 EXTENT_DATA[910393 8192] gap exists, expected: EXTENT_DATA[910393 4121]
ERROR: root 304 EXTENT_DATA[910393 16384] gap exists, expected: EXTENT_DATA[910393 12313]
ERROR: root 304 EXTENT_DATA[910400 4096] gap exists, expected: EXTENT_DATA[910400 25]
ERROR: root 304 EXTENT_DATA[910400 8192] gap exists, expected: EXTENT_DATA[910400 4121]
ERROR: root 304 EXTENT_DATA[910400 16384] gap exists, expected: EXTENT_DATA[910400 12313]
ERROR: root 304 EXTENT_DATA[910401 4096] gap exists, expected: EXTENT_DATA[910401 25]
ERROR: root 304 EXTENT_DATA[910401 8192] gap exists, expected: EXTENT_DATA[910401 4121]
ERROR: root 304 EXTENT_DATA[910401 16384] gap exists, expected: EXTENT_DATA[910401 12313]
ERROR: root 101721 INODE REF[274921, 256895] name 25da95e3e893bb2fa69a2f0acd77bfe725626a1e filetype 1 missing
ERROR: root 101721 EXTENT_DATA[910393 4096] gap exists, expected: EXTENT_DATA[910393 25]
ERROR: root 101721 EXTENT_DATA[910393 8192] gap exists, expected: EXTENT_DATA[910393 4121]
ERROR: root 101721 EXTENT_DATA[910393 16384] gap exists, expected: EXTENT_DATA[910393 12313]
ERROR: root 101721 EXTENT_DATA[910400 4096] gap exists, expected: EXTENT_DATA[910400 25]
ERROR: root 101721 EXTENT_DATA[910400 8192] gap exists, expected: EXTENT_DATA[910400 4121]
ERROR: root 101721 EXTENT_DATA[910400 16384] gap exists, expected: EXTENT_DATA[910400 12313]
ERROR: root 101721 EXTENT_DATA[910401 4096] gap exists, expected: EXTENT_DATA[910401 25]
ERROR: root 101721 EXTENT_DATA[910401 8192] gap exists, expected: EXTENT_DATA[910401 4121]
ERROR: root 101721 EXTENT_DATA[910401 16384] gap exists, expected: EXTENT_DATA[910401 12313]
ERROR: errors found in fs roots
found 39410126848 bytes used, error(s) found
total csum bytes: 35990412
total tree bytes: 196955471872
total fs tree bytes: 196809785344
total extent tree bytes: 96534528
btree space waste bytes: 33486070155
file data blocks allocated: 1705720172544
 referenced 2238568390656


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Unrecoverable btrfs corruption (backref bytes do not match extent backref)
  2019-01-04 23:04     ` Nazar Mokrynskyi
@ 2019-01-05  1:18       ` Qu Wenruo
  2019-01-05  1:26         ` Nazar Mokrynskyi
  0 siblings, 1 reply; 10+ messages in thread
From: Qu Wenruo @ 2019-01-05  1:18 UTC (permalink / raw)
  To: Nazar Mokrynskyi, Chris Murphy; +Cc: linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 1660 bytes --]



On 2019/1/5 上午7:04, Nazar Mokrynskyi wrote:
> 04.01.19 03:32, Qu Wenruo пише:
>> Please provide the dump of the following command:
>>
>> # btrfs ins dump-tree -t extent | grep 3114475520 -C 20
> root@nazarpc-Standard-PC-Q35-ICH9-2009:~# btrfs ins dump-tree -t extent /dev/vdc | grep 3114475520 -C 20
[snip]
>     item 152 key (3114475520 EXTENT_ITEM 4096) itemoff 10193 itemsize 24
>         refs 2 gen 1701147 flags DATA
>     item 153 key (3114475520 EXTENT_ITEM 36864) itemoff 10169 itemsize 24
>         refs 1 gen 1712966 flags DATA

Two extent backref for one bytenr, this is definitely a problem.

>     item 154 key (3114475520 SHARED_DATA_REF 311408951296) itemoff 10165 itemsize 4
>         shared data backref count 1
>     item 155 key (3114475520 SHARED_DATA_REF 342561947648) itemoff 10161 itemsize 4
>         shared data backref count 1
>     item 156 key (3114475520 SHARED_DATA_REF 348547874816) itemoff 10157 itemsize 4
>         shared data backref count 1

And total of 3 refs, 1 or 2 of them are the cause.

I'd say either it's caused by kernel bug or corrupted space cache/block
group item.

Please don't mount the fs RW, and copy your data out.

In fact I don't have any idea how this happened, no clue to fix either.

Thanks,
Qu

> 
> Actually, I removed Docker's snapshots after making image, but most others are not there anymore.
> 
<snip>
> I'm using qcow2 with snapshots, so I can do multiple experiments with image at the same time if necessary. Also started `btrfs check --mode=lowmem`, will report once it finishes.



> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Unrecoverable btrfs corruption (backref bytes do not match extent backref)
  2019-01-05  1:18       ` Qu Wenruo
@ 2019-01-05  1:26         ` Nazar Mokrynskyi
  0 siblings, 0 replies; 10+ messages in thread
From: Nazar Mokrynskyi @ 2019-01-05  1:26 UTC (permalink / raw)
  To: Qu Wenruo, Chris Murphy; +Cc: linux-btrfs

05.01.19 03:18, Qu Wenruo пише:
> Please don't mount the fs RW, and copy your data out.

I have regular automated backups and wrote initial message from restored system already, lesson learned a long time ago.

Next time I'll be smarter and will make partition image prior to doing any operations on it, maybe `balance` did something to it.

It is quite unfortunate that information provided is not useful this time. If there any other ideas for tests to run - I'm willing to help.


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2019-01-05  1:26 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-03 22:06 Unrecoverable btrfs corruption (backref bytes do not match extent backref) Nazar Mokrynskyi
2019-01-04  1:15 ` Chris Murphy
2019-01-04  1:32   ` Qu Wenruo
2019-01-04  1:36     ` Hans van Kranenburg
2019-01-04  3:43     ` Chris Murphy
2019-01-04  4:31       ` Qu Wenruo
2019-01-04 23:04     ` Nazar Mokrynskyi
2019-01-05  1:18       ` Qu Wenruo
2019-01-05  1:26         ` Nazar Mokrynskyi
2019-01-05  0:13   ` Nazar Mokrynskyi

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).