btrfs check lowmem vs original

btrfs check lowmem vs original

[Chris Murphy]

All of my Btrfs file systems, including new ones, have errors
according to lowmem mode, and no errors reported at all for original
mode. So which is correct? If lowmem mode is correct, then there are
obviously kernel bugs that are causing problems right away, even on
minutes old file systems.

I can't tell from the output whether these are serious errors or minor
errors either, because the output from check is totally
incomprehensible. Further the usage summary at the end, extent bytes,
and fs bytes, etc. are different, sometimes by an order of magnitude,
between lowmem and original.

Errors like this:

ERROR: block group[46200258560 1073741824] used 1073741824 but extent
items used 1144422400
ERROR: block group[85928706048 1073741824] used 1073741824 but extent
items used 0
ERROR: block group[178270502912 1073741824] used 1073741824 but extent
items used 1083211776
ERROR: block group[360806612992 1073741824] used 1073479680 but extent
items used 1074769920
ERROR: extent[366498091008, 134217728] referencer count mismatch
(root: 818, owner: 73782, offset: 134217728) wanted: 4, have: 26

One file system has over 100 lines (exactly the same thing, no difference)

ERROR: data extent[16913485824 7577600] backref lost

Another file system, 15 minutes old with two mounts in its whole
lifetime, and only written with kernel 4.10-rc3 has over 30 lines of
varying numbers:

ERROR: root 257 EXTENT DATA[150134 11317248] prealloc shouln't have datasum

That file system should have no preallocated extents (It's a clean
installation of Fedora Rawhide, using only rsync)

Again, zero errors with original mode; but all file systems of all
ages have errors with lowmem. This is kindof annoying to say the
least, it's like we're adding dice to the check and repair situation
on Btrfs which is already incredibly more complicated and unhelpful
than any other file system.

btrfs-progs 4.9.
kernels vary from 4.4 to 4.10-rc4, but one fs is only a month old
having used 4.7 and 4.8 kernels; and another one just 4.10-rc3 as I
said.


-- 
Chris Murphy

Re: btrfs check lowmem vs original

[Qu Wenruo]

At 01/20/2017 12:38 PM, Chris Murphy wrote:
> All of my Btrfs file systems, including new ones, have errors
> according to lowmem mode, and no errors reported at all for original
> mode. So which is correct? If lowmem mode is correct, then there are
> obviously kernel bugs that are causing problems right away, even on
> minutes old file systems.
>
> I can't tell from the output whether these are serious errors or minor
> errors either, because the output from check is totally
> incomprehensible. Further the usage summary at the end, extent bytes,
> and fs bytes, etc. are different, sometimes by an order of magnitude,
> between lowmem and original.
>
> Errors like this:
>
> ERROR: block group[46200258560 1073741824] used 1073741824 but extent
> items used 1144422400
> ERROR: block group[85928706048 1073741824] used 1073741824 but extent
> items used 0
> ERROR: block group[178270502912 1073741824] used 1073741824 but extent
> items used 1083211776
> ERROR: block group[360806612992 1073741824] used 1073479680 but extent
> items used 1074769920

Christoph Anton Mitterer also reported this bug.

And it turns out to be a false alert.

Patch under test, should be out soon.
(But the image is not easy to create, so no test case yet)

> ERROR: extent[366498091008, 134217728] referencer count mismatch
> (root: 818, owner: 73782, offset: 134217728) wanted: 4, have: 26

This is a little different.
Not sure which fsck is wrong until btrfs-debug-tree extent and fs tree 
dump is provided.

>
> One file system has over 100 lines (exactly the same thing, no difference)
>
> ERROR: data extent[16913485824 7577600] backref lost

Same, need btrfs-debug-tree extent and fs tree dump.

>
> Another file system, 15 minutes old with two mounts in its whole
> lifetime, and only written with kernel 4.10-rc3 has over 30 lines of
> varying numbers:
>
> ERROR: root 257 EXTENT DATA[150134 11317248] prealloc shouln't have datasum
>
> That file system should have no preallocated extents (It's a clean
> installation of Fedora Rawhide, using only rsync)

btrfs-debug-tree will help to make sure what is wrong.

>
> Again, zero errors with original mode; but all file systems of all
> ages have errors with lowmem. This is kindof annoying to say the
> least, it's like we're adding dice to the check and repair situation
> on Btrfs which is already incredibly more complicated and unhelpful
> than any other file system.

That's why lowmem mode is still not the default option.

The problem os original mode is, if you're checking a TB level fs and 
only 2 or 4G ram, then it's quite possible you ran out of memory and 
won't be able to check the fs forever, more several than annoying.

Thanks,
Qu

>
> btrfs-progs 4.9.
> kernels vary from 4.4 to 4.10-rc4, but one fs is only a month old
> having used 4.7 and 4.8 kernels; and another one just 4.10-rc3 as I
> said.
>
>

Re: btrfs check lowmem vs original

[Chris Murphy]

On Thu, Jan 19, 2017 at 10:45 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:

>> Another file system, 15 minutes old with two mounts in its whole
>> lifetime, and only written with kernel 4.10-rc3 has over 30 lines of
>> varying numbers:
>>
>> ERROR: root 257 EXTENT DATA[150134 11317248] prealloc shouln't have
>> datasum
>>
>> That file system should have no preallocated extents (It's a clean
>> installation of Fedora Rawhide, using only rsync)
>
>
> btrfs-debug-tree will help to make sure what is wrong.

6b187fa6.logs.tar.gz 20M
https://drive.google.com/open?id=0B_2Asp8DGjJ9SlRvZ2plNXVmTUU


That's the small recent one, generic content. The others are bigger,
and I should probably sanitize the filenames from debug-tree but can't
find in the archives how to do that. Is btrfs-image useful for this?




> That's why lowmem mode is still not the default option.
>
> The problem os original mode is, if you're checking a TB level fs and only 2
> or 4G ram, then it's quite possible you ran out of memory and won't be able
> to check the fs forever, more several than annoying.

Fair enough.



-- 
Chris Murphy

Re: btrfs check lowmem vs original

[Chris Murphy]

For sanitizing the debug tree I tried this backwards method:

# btrfs-image -c5 -t4 -s /dev/mapper/brick1 > btrfsimage_30f4724a.bin
# lvcreate -V 603440807936b -T vg/thintastic -n btr1
# btrfs-image -r btrfsimage_30f4724a.bin /dev/vg/btr1
# btrfs inspect-internal dump-tree /dev/vg/btr1 > btrfsdebugtree_30f4724a.log

But I see by default, -r makes modifications to the chunk tree, so
maybe it's not a useful debug tree? Anyway, the two btrfs checks
(original and lowmem) and the debug tree are here:

30f4724a-first.logs.tar 254MiB
https://drive.google.com/open?id=0B_2Asp8DGjJ9QmZOV3pNS212Yk0

There are 4 more file systems, 2 are single device, 2 are two device
(raid1) fs's. I'm not sure how to use this indirect btrfs-image
approach with raid1.


Chris Murphy

Re: btrfs check lowmem vs original

[Qu Wenruo]

At 01/20/2017 11:10 PM, Chris Murphy wrote:
> On Thu, Jan 19, 2017 at 10:45 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>>> Another file system, 15 minutes old with two mounts in its whole
>>> lifetime, and only written with kernel 4.10-rc3 has over 30 lines of
>>> varying numbers:
>>>
>>> ERROR: root 257 EXTENT DATA[150134 11317248] prealloc shouln't have
>>> datasum
>>>
>>> That file system should have no preallocated extents (It's a clean
>>> installation of Fedora Rawhide, using only rsync)
>>
>>
>> btrfs-debug-tree will help to make sure what is wrong.
>
> 6b187fa6.logs.tar.gz 20M
> https://drive.google.com/open?id=0B_2Asp8DGjJ9SlRvZ2plNXVmTUU
>
>
> That's the small recent one, generic content. The others are bigger,
> and I should probably sanitize the filenames from debug-tree but can't
> find in the archives how to do that. Is btrfs-image useful for this?

Normally, btrfs-image is the best format, as we can just test if the fix 
is good or not directly on the image.

Not to mention btrfs-image has -s option to sanitize filenames. And 
btrfs-image is normally a little smaller than debug tree output.

Thanks,
Qu

>
>
>
>
>> That's why lowmem mode is still not the default option.
>>
>> The problem os original mode is, if you're checking a TB level fs and only 2
>> or 4G ram, then it's quite possible you ran out of memory and won't be able
>> to check the fs forever, more several than annoying.
>
> Fair enough.
>
>
>

Re: btrfs check lowmem vs original

[Chris Murphy]

OK so all of these pass original check, but have problems reported by
lowmem. Separate notes about each inline.

~500MiB each, these three are data volumes, first two are raid1, third
one is single.
https://drive.google.com/open?id=0B_2Asp8DGjJ9Z3UzWnFKT3A0clU
https://drive.google.com/open?id=0B_2Asp8DGjJ9V0ROdHNoMW1BVE0
https://drive.google.com/open?id=0B_2Asp8DGjJ9Zmd1LXl6MU5WeXc

19MiB, about 15 minutes old, rootfs, OS installation only
https://drive.google.com/open?id=0B_2Asp8DGjJ9TF9LVkFlcDBzOG8

55MiB, about 1 month old, rootfs, not much activity
https://drive.google.com/open?id=0B_2Asp8DGjJ9bkJFc01qcVJxNnM

324MiB, about 5 months old, used as rootfs, all read-write snapshots
used as rootfs are forced readonly, a regression previously reported
without any dev response
https://drive.google.com/open?id=0B_2Asp8DGjJ9ZmNxdEw1RDBPcTA
http://www.spinics.net/lists/linux-btrfs/msg61817.html
https://bugzilla.kernel.org/show_bug.cgi?id=191761

Re: btrfs check lowmem vs original

[Qu Wenruo]

At 01/24/2017 05:14 AM, Chris Murphy wrote:
> OK so all of these pass original check, but have problems reported by
> lowmem. Separate notes about each inline.

Thanks for your images!

It really helps a lot.

I tested my patches against these images.
Feel free to test them:
https://github.com/adam900710/btrfs-progs/tree/lowmem_fixes

>
> ~500MiB each, these three are data volumes, first two are raid1, third
> one is single.
> https://drive.google.com/open?id=0B_2Asp8DGjJ9Z3UzWnFKT3A0clU
> https://drive.google.com/open?id=0B_2Asp8DGjJ9V0ROdHNoMW1BVE0
> https://drive.google.com/open?id=0B_2Asp8DGjJ9Zmd1LXl6MU5WeXc

RAID1 ones are not stable for us to check, as it already contains some 
chunk tree error after recovery.
The single one I didn't download, after 324M one encounters some 
btrfs-image recovery problem.

>
> 19MiB, about 15 minutes old, rootfs, OS installation only
> https://drive.google.com/open?id=0B_2Asp8DGjJ9TF9LVkFlcDBzOG8

Passed now.

>
> 55MiB, about 1 month old, rootfs, not much activity
> https://drive.google.com/open?id=0B_2Asp8DGjJ9bkJFc01qcVJxNnM

Passed too.

>
> 324MiB, about 5 months old, used as rootfs, all read-write snapshots
> used as rootfs are forced readonly, a regression previously reported
> without any dev response
> https://drive.google.com/open?id=0B_2Asp8DGjJ9ZmNxdEw1RDBPcTA
> http://www.spinics.net/lists/linux-btrfs/msg61817.html
> https://bugzilla.kernel.org/show_bug.cgi?id=191761

Recovery caused quite a lot of false alert on chunk tree.
Still digging if the remaining errors are valid or not.

Thanks,
Qu

> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: btrfs check lowmem vs original

[Chris Murphy]

With kernel 4.10.3, and btrfs-progs 4.10, I'm still seeing the
following errors only with --mode=lowmem with a single device volume.


ERROR: data extent[16913485824 7577600] backref lost
(hundreds of lines of the same thing)

btrfs-image (no errors reported during capture) 518MiB
https://drive.google.com/open?id=0B7NttHfq6wAJd3ZXM3AxS0dVQlU


Chris Murphy

Re: btrfs check lowmem vs original

[Qu Wenruo]

At 03/17/2017 07:22 AM, Chris Murphy wrote:
> With kernel 4.10.3, and btrfs-progs 4.10, I'm still seeing the
> following errors only with --mode=lowmem with a single device volume.
>
>
> ERROR: data extent[16913485824 7577600] backref lost
> (hundreds of lines of the same thing)
>
> btrfs-image (no errors reported during capture) 518MiB
> https://drive.google.com/open?id=0B7NttHfq6wAJd3ZXM3AxS0dVQlU

Thanks a lot for the image.

Turns out to be that, lowmem mode only checks EXTENT_DATA_REF key, and 
doesn't check SHARED_DATA_REF_KEY, this leads to the error message.

Fix will be sent out soon.

Thanks,
Qu
>
>
> Chris Murphy
>
>

Re: btrfs check lowmem vs original

[Chris Murphy]

Resurrecting this old thread because I'm still seeing these errors.

On Thu, Mar 16, 2017 at 6:54 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> At 03/17/2017 07:22 AM, Chris Murphy wrote:
>>
>> With kernel 4.10.3, and btrfs-progs 4.10, I'm still seeing the
>> following errors only with --mode=lowmem with a single device volume.
>>
>>
>> ERROR: data extent[16913485824 7577600] backref lost
>> (hundreds of lines of the same thing)
>>
>> btrfs-image (no errors reported during capture) 518MiB
>> https://drive.google.com/open?id=0B7NttHfq6wAJd3ZXM3AxS0dVQlU
>
>
> Thanks a lot for the image.
>
> Turns out to be that, lowmem mode only checks EXTENT_DATA_REF key, and
> doesn't check SHARED_DATA_REF_KEY, this leads to the error message.
>
> Fix will be sent out soon.


I'm using btrfs-progs 4.13.3 and still see these errors. Do you need
an updated sanitized image, or newer version or patchset to test?

This example error is happening on a single device volume with DUP
metadata, and also ends in a crash:.

ERROR: data extent[16913485824 7577600] backref lost
...
backref.c:466: __add_missing_keys: Assertion `ref->root_id` failed, value 0
...

Complete output, with filtered btrfs-debug-tree info for extent
address 16913485824
https://drive.google.com/open?id=1WgHdIz7zu_bO6evUBkG4wPxpJfwaFCxv

------

This example error is happening on a two device raid1 data and raid1
metadata volume:

ERROR: extent[366498091008, 134217728] referencer count mismatch
(root: 827, owner: 73782, offset: 134217728) wanted: 4, have: 26

Complete output, with filtered btrfs-debug-tree info for extent
address 366498091008.
https://drive.google.com/open?id=1LUMtLIc1LcXhYN3twFcxTyFuJ6ScMlyQ


Both examples have multiple extents reported in the error but I've
only included debug tree info for the first extent listed by lowmem
check. So it's an incomplete report. I can provide sanitized
btrfs-image on request, or test with patches.

-- 
Chris Murphy

Re: btrfs check lowmem vs original

[Su Yue]

Hi

On 11/11/2017 07:16 AM, Chris Murphy wrote:
> Resurrecting this old thread because I'm still seeing these errors.
> 
> On Thu, Mar 16, 2017 at 6:54 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>
>>
>> At 03/17/2017 07:22 AM, Chris Murphy wrote:
>>>
>>> With kernel 4.10.3, and btrfs-progs 4.10, I'm still seeing the
>>> following errors only with --mode=lowmem with a single device volume.
>>>
>>>
>>> ERROR: data extent[16913485824 7577600] backref lost
>>> (hundreds of lines of the same thing)
>>>
>>> btrfs-image (no errors reported during capture) 518MiB
>>> https://drive.google.com/open?id=0B7NttHfq6wAJd3ZXM3AxS0dVQlU
>>
>>
>> Thanks a lot for the image.
>>
>> Turns out to be that, lowmem mode only checks EXTENT_DATA_REF key, and
>> doesn't check SHARED_DATA_REF_KEY, this leads to the error message.
>>
>> Fix will be sent out soon.
> 
> 
> I'm using btrfs-progs 4.13.3 and still see these errors. Do you need
> an updated sanitized image, or newer version or patchset to test?
> 
Actually, lowmem mode in 4.13.3 still has some problem while checking
extent-tree.

I have some patches for lowmem mode. But they are not fully tested and
unstable.
It will be so nice if you can provide some images.

Thanks,
Su

> This example error is happening on a single device volume with DUP
> metadata, and also ends in a crash:.
> 
> ERROR: data extent[16913485824 7577600] backref lost
> ...
> backref.c:466: __add_missing_keys: Assertion `ref->root_id` failed, value 0
> ...
> 
> Complete output, with filtered btrfs-debug-tree info for extent
> address 16913485824
> https://drive.google.com/open?id=1WgHdIz7zu_bO6evUBkG4wPxpJfwaFCxv
> 
> ------
> 
> This example error is happening on a two device raid1 data and raid1
> metadata volume:
> 
> ERROR: extent[366498091008, 134217728] referencer count mismatch
> (root: 827, owner: 73782, offset: 134217728) wanted: 4, have: 26
> 
> Complete output, with filtered btrfs-debug-tree info for extent
> address 366498091008.
> https://drive.google.com/open?id=1LUMtLIc1LcXhYN3twFcxTyFuJ6ScMlyQ
> 
> 
> Both examples have multiple extents reported in the error but I've
> only included debug tree info for the first extent listed by lowmem
> check. So it's an incomplete report. I can provide sanitized
> btrfs-image on request, or test with patches.
> 

Re: btrfs check lowmem vs original

[Chris Murphy]

On Sun, Nov 12, 2017 at 7:25 PM, Su Yue <suy.fnst@cn.fujitsu.com> wrote:

>
> I have some patches for lowmem mode. But they are not fully tested and
> unstable.
> It will be so nice if you can provide some images.

Sorry, Qu got the image offline, I wasn't paying attention that btrfs
list wasn't cc'd.  The -ss option was taking too long and during the
imaging a systemd unit mounted the file system so I figured the image
might be useless or contain extra confusion.

Here is the -s image I send to Qu.

https://drive.google.com/open?id=1cIAANqMicIMp9NfgnvN_SA90kjM0kP0C




-- 
Chris Murphy

Re: btrfs check lowmem vs original

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1221 bytes --]



On 2017年11月13日 11:37, Chris Murphy wrote:
> On Sun, Nov 12, 2017 at 7:25 PM, Su Yue <suy.fnst@cn.fujitsu.com> wrote:
> 
>>
>> I have some patches for lowmem mode. But they are not fully tested and
>> unstable.
>> It will be so nice if you can provide some images.
> 
> Sorry, Qu got the image offline, I wasn't paying attention that btrfs
> list wasn't cc'd.  The -ss option was taking too long and during the
> imaging a systemd unit mounted the file system so I figured the image
> might be useless or contain extra confusion.
> 
> Here is the -s image I send to Qu.
> 
> https://drive.google.com/open?id=1cIAANqMicIMp9NfgnvN_SA90kjM0kP0C

Patches for backref lost false warning Cced to you.

Since the reason is clear and test case added, I think it should solve
the "backref lost" false alerts.

However for BUG_ON(), it's tree walk code who can't handle tree reloc
tree, this may be a little complex compared to backref lost.
But since it's a different story, I'll send the fix as another patch(set).


The final bug about backref counts mismatch, without the image I'm not
really sure what's going on.

(But extent tree verification is really buggy though)

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: btrfs check lowmem vs original

[Chris Murphy]

On Sun, Nov 12, 2017 at 9:37 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:

>
> The final bug about backref counts mismatch, without the image I'm not
> really sure what's going on.
>
> (But extent tree verification is really buggy though)

Ahh I imaged the wrong file system! Sorry about that:


>>ERROR: extent[366498091008, 134217728] referencer count mismatch
>>(root: 827, owner: 73782, offset: 134217728) wanted: 4, have: 26

>>Complete output, with filtered btrfs-debug-tree info for extent
>>address 366498091008.
>>https://drive.google.com/open?id=1LUMtLIc1LcXhYN3twFcxTyFuJ6ScMlyQ

Here is the image for that two device raid1:

https://drive.google.com/open?id=1LO96omP3BQh9C3ZYVXoUDT4-H8P5lZ6D



-- 
Chris Murphy

Re: btrfs check lowmem vs original

[Chris Murphy]

On Mon, Nov 13, 2017 at 11:40 AM, Chris Murphy <chris@colorremedies.com> wrote:
> On Sun, Nov 12, 2017 at 9:37 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>>
>> The final bug about backref counts mismatch, without the image I'm not
>> really sure what's going on.
>>
>> (But extent tree verification is really buggy though)
>
> Ahh I imaged the wrong file system! Sorry about that:
>
>
>>>ERROR: extent[366498091008, 134217728] referencer count mismatch
>>>(root: 827, owner: 73782, offset: 134217728) wanted: 4, have: 26
>
>>>Complete output, with filtered btrfs-debug-tree info for extent
>>>address 366498091008.
>>>https://drive.google.com/open?id=1LUMtLIc1LcXhYN3twFcxTyFuJ6ScMlyQ
>
> Here is the image for that two device raid1:
>
> https://drive.google.com/open?id=1LO96omP3BQh9C3ZYVXoUDT4-H8P5lZ6D
>
By the way, this one is done with -ss. But it has many failures to
compute collisions, hundreds of files/dirs with 4 characters or less.

-- 
Chris Murphy

Re: btrfs check lowmem vs original

[Chris Murphy]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

I've retested btrfs-progs 4.17 lowmem mode and still get a bunch of
'referencer count mismatch' that I don't see with original mode.

e.g.
ERROR: extent[1299275636736, 2654208] referencer count mismatch (root:
2192, owner: 317900, offset: 0) wanted: 5, have: 21

Complete output attached as a text file, since the MUA will mess up
the line wrap.


--
Chris Murphy

[-- Attachment #2: btrfs-check-lowmem.txt --]
[-- Type: text/plain, Size: 30086 bytes --]


[chris@f28s ~]$ sudo ~/Applications/btrfs-progs/btrfs --version
btrfs-progs v4.17 
[chris@f28s ~]$ sudo ~/Applications/btrfs-progs/btrfs check --mode=lowmem /dev/mapper/first
Checking filesystem on /dev/mapper/first
UUID: 0f43f49d-6e63-4b1b-bc8c-c54da409872d
checking extents
ERROR: extent[1258793181184, 81903616] referencer count mismatch (root: 2192, owner: 323075, offset: 0) wanted: 551, have: 625
ERROR: extent[1258899460096, 82878464] referencer count mismatch (root: 2192, owner: 323115, offset: 0) wanted: 521, have: 633
ERROR: extent[1259036663808, 64319488] referencer count mismatch (root: 2192, owner: 323125, offset: 0) wanted: 376, have: 491
ERROR: extent[1259102384128, 64552960] referencer count mismatch (root: 2192, owner: 323137, offset: 0) wanted: 439, have: 493
ERROR: extent[1259171381248, 113295360] referencer count mismatch (root: 2192, owner: 323147, offset: 0) wanted: 763, have: 865
ERROR: extent[1259307835392, 2576384] referencer count mismatch (root: 2192, owner: 323191, offset: 0) wanted: 9, have: 20
ERROR: extent[1259327455232, 30089216] referencer count mismatch (root: 2192, owner: 323203, offset: 0) wanted: 204, have: 230
ERROR: extent[1259372060672, 1974272] referencer count mismatch (root: 2192, owner: 323218, offset: 0) wanted: 13, have: 16
ERROR: extent[1259924090880, 22102016] referencer count mismatch (root: 2192, owner: 323199, offset: 427819008) wanted: 52, have: 169
ERROR: extent[1259960025088, 2211840] referencer count mismatch (root: 2192, owner: 323244, offset: 0) wanted: 11, have: 17
ERROR: extent[1260158959616, 53563392] referencer count mismatch (root: 2192, owner: 323328, offset: 0) wanted: 328, have: 409
ERROR: extent[1260258131968, 134217728] referencer count mismatch (root: 2192, owner: 323609, offset: 0) wanted: 959, have: 1024
ERROR: extent[1260410531840, 1716224] referencer count mismatch (root: 2192, owner: 323625, offset: 0) wanted: 4, have: 14
ERROR: extent[1261505388544, 1839104] referencer count mismatch (root: 2192, owner: 323654, offset: 0) wanted: 10, have: 15
ERROR: extent[1261846781952, 56676352] referencer count mismatch (root: 2192, owner: 323731, offset: 0) wanted: 354, have: 433
ERROR: extent[1262246526976, 2256896] referencer count mismatch (root: 2192, owner: 323750, offset: 0) wanted: 10, have: 18
ERROR: extent[1262765875200, 765952] referencer count mismatch (root: 2192, owner: 319045, offset: 0) wanted: 5, have: 6
ERROR: extent[1262984384512, 1642496] referencer count mismatch (root: 2192, owner: 322213, offset: 0) wanted: 6, have: 13
ERROR: extent[1263573057536, 1159168] referencer count mismatch (root: 2192, owner: 323009, offset: 0) wanted: 2, have: 9
ERROR: extent[1263615098880, 1220608] referencer count mismatch (root: 2192, owner: 323062, offset: 0) wanted: 1, have: 10
ERROR: extent[1263647588352, 2310144] referencer count mismatch (root: 2192, owner: 330026, offset: 0) wanted: 6, have: 18
ERROR: extent[1263665254400, 3538944] referencer count mismatch (root: 2192, owner: 330035, offset: 0) wanted: 16, have: 27
ERROR: extent[1263695650816, 1773568] referencer count mismatch (root: 2192, owner: 330054, offset: 0) wanted: 8, have: 14
ERROR: extent[1263894016000, 1536000] referencer count mismatch (root: 2192, owner: 319139, offset: 0) wanted: 9, have: 12
ERROR: extent[1263904493568, 1122304] referencer count mismatch (root: 2192, owner: 319151, offset: 0) wanted: 3, have: 9
ERROR: extent[1263915298816, 540672] referencer count mismatch (root: 2192, owner: 319176, offset: 0) wanted: 1, have: 5
ERROR: extent[1264166404096, 679936] referencer count mismatch (root: 2192, owner: 319672, offset: 0) wanted: 3, have: 6
ERROR: extent[1264205729792, 2166784] referencer count mismatch (root: 2192, owner: 319706, offset: 0) wanted: 13, have: 17
ERROR: extent[1264443080704, 327680] referencer count mismatch (root: 2192, owner: 319874, offset: 0) wanted: 2, have: 3
ERROR: extent[1264474132480, 1277952] referencer count mismatch (root: 2192, owner: 319926, offset: 0) wanted: 6, have: 10
ERROR: extent[1264494592000, 1622016] referencer count mismatch (root: 2192, owner: 319943, offset: 0) wanted: 12, have: 13
ERROR: extent[1264528855040, 2527232] referencer count mismatch (root: 2192, owner: 319965, offset: 0) wanted: 16, have: 20
ERROR: extent[1264556707840, 901120] referencer count mismatch (root: 2192, owner: 319974, offset: 0) wanted: 3, have: 7
ERROR: extent[1265866297344, 651264] referencer count mismatch (root: 2192, owner: 320062, offset: 0) wanted: 2, have: 5
ERROR: extent[1266110365696, 929792] referencer count mismatch (root: 2192, owner: 320074, offset: 0) wanted: 7, have: 8
ERROR: extent[1266197757952, 2420736] referencer count mismatch (root: 2192, owner: 320101, offset: 0) wanted: 12, have: 19
ERROR: extent[1266310610944, 2519040] referencer count mismatch (root: 2192, owner: 320139, offset: 0) wanted: 3, have: 20
ERROR: extent[1266373623808, 2641920] referencer count mismatch (root: 2192, owner: 320171, offset: 0) wanted: 6, have: 21
ERROR: extent[1266563051520, 1077248] referencer count mismatch (root: 2192, owner: 320185, offset: 0) wanted: 3, have: 9
ERROR: extent[1266759434240, 958464] referencer count mismatch (root: 2192, owner: 320203, offset: 0) wanted: 4, have: 8
ERROR: extent[1266968154112, 995328] referencer count mismatch (root: 2192, owner: 320237, offset: 0) wanted: 7, have: 8
ERROR: extent[1267030790144, 143360] referencer count mismatch (root: 2192, owner: 320262, offset: 0) wanted: 1, have: 2
ERROR: extent[1267294498816, 2236416] referencer count mismatch (root: 2192, owner: 320603, offset: 0) wanted: 7, have: 18
ERROR: extent[1267580084224, 2674688] referencer count mismatch (root: 2192, owner: 320658, offset: 0) wanted: 8, have: 21
ERROR: extent[1267642232832, 1830912] referencer count mismatch (root: 2192, owner: 320670, offset: 0) wanted: 9, have: 14
ERROR: extent[1267759599616, 827392] referencer count mismatch (root: 2192, owner: 320779, offset: 0) wanted: 1, have: 7
ERROR: extent[1267773407232, 749568] referencer count mismatch (root: 2192, owner: 320798, offset: 0) wanted: 5, have: 6
ERROR: extent[1267807432704, 794624] referencer count mismatch (root: 2192, owner: 320825, offset: 0) wanted: 3, have: 7
ERROR: extent[1268002275328, 2113536] referencer count mismatch (root: 2192, owner: 321004, offset: 0) wanted: 10, have: 17
ERROR: extent[1268018139136, 487424] referencer count mismatch (root: 2192, owner: 321022, offset: 0) wanted: 3, have: 4
ERROR: extent[1275709833216, 1851392] referencer count mismatch (root: 2192, owner: 297626, offset: 0) wanted: 3, have: 15
ERROR: extent[1276156461056, 1363968] referencer count mismatch (root: 2192, owner: 298850, offset: 0) wanted: 4, have: 11
ERROR: extent[1276216307712, 1200128] referencer count mismatch (root: 2192, owner: 298910, offset: 0) wanted: 5, have: 10
ERROR: extent[1276231864320, 1441792] referencer count mismatch (root: 2192, owner: 298925, offset: 0) wanted: 3, have: 11
ERROR: extent[1276248236032, 1175552] referencer count mismatch (root: 2192, owner: 298937, offset: 0) wanted: 7, have: 9
ERROR: extent[1276260663296, 499712] referencer count mismatch (root: 2192, owner: 298955, offset: 0) wanted: 3, have: 4
ERROR: extent[1276349661184, 1343488] referencer count mismatch (root: 2192, owner: 299029, offset: 0) wanted: 2, have: 11
ERROR: extent[1276361928704, 679936] referencer count mismatch (root: 2192, owner: 299040, offset: 0) wanted: 4, have: 6
ERROR: extent[1276678701056, 442368] referencer count mismatch (root: 2192, owner: 298248, offset: 0) wanted: 1, have: 4
ERROR: extent[1276703105024, 1273856] referencer count mismatch (root: 2192, owner: 299421, offset: 0) wanted: 1, have: 10
ERROR: extent[1276746051584, 614400] referencer count mismatch (root: 2192, owner: 299487, offset: 0) wanted: 1, have: 5
ERROR: extent[1276758941696, 1392640] referencer count mismatch (root: 2192, owner: 299870, offset: 0) wanted: 8, have: 11
ERROR: extent[1276758941696, 1392640] referencer count mismatch (root: 2192, owner: 299983, offset: 0) wanted: 7, have: 10
ERROR: extent[1276772073472, 1224704] referencer count mismatch (root: 2192, owner: 299881, offset: 0) wanted: 9, have: 10
ERROR: extent[1276789489664, 1355776] referencer count mismatch (root: 2192, owner: 299896, offset: 0) wanted: 3, have: 11
ERROR: extent[1276805271552, 1290240] referencer count mismatch (root: 2192, owner: 299908, offset: 0) wanted: 4, have: 10
ERROR: extent[1276822466560, 1085440] referencer count mismatch (root: 2192, owner: 299923, offset: 0) wanted: 6, have: 9
ERROR: extent[1276839735296, 1454080] referencer count mismatch (root: 2192, owner: 299938, offset: 0) wanted: 6, have: 12
ERROR: extent[1276873445376, 1318912] referencer count mismatch (root: 2192, owner: 299968, offset: 0) wanted: 7, have: 11
ERROR: extent[1276898889728, 1146880] referencer count mismatch (root: 2192, owner: 299991, offset: 0) wanted: 2, have: 9
ERROR: extent[1277025230848, 1191936] referencer count mismatch (root: 2192, owner: 300096, offset: 0) wanted: 3, have: 10
ERROR: extent[1277102907392, 1318912] referencer count mismatch (root: 2192, owner: 300163, offset: 0) wanted: 8, have: 11
ERROR: extent[1277121216512, 1327104] referencer count mismatch (root: 2192, owner: 300178, offset: 0) wanted: 5, have: 11
ERROR: extent[1277188087808, 1179648] referencer count mismatch (root: 2192, owner: 300235, offset: 0) wanted: 7, have: 9
ERROR: extent[1277263859712, 1060864] referencer count mismatch (root: 2192, owner: 300300, offset: 0) wanted: 3, have: 9
ERROR: extent[1277330145280, 1019904] referencer count mismatch (root: 2192, owner: 300361, offset: 0) wanted: 5, have: 8
ERROR: extent[1277343236096, 1044480] referencer count mismatch (root: 2192, owner: 300378, offset: 0) wanted: 6, have: 8
ERROR: extent[1277425733632, 868352] referencer count mismatch (root: 2192, owner: 300464, offset: 0) wanted: 2, have: 7
ERROR: extent[1277739409408, 1138688] referencer count mismatch (root: 2192, owner: 300989, offset: 0) wanted: 1, have: 9
ERROR: extent[1277751787520, 1306624] referencer count mismatch (root: 2192, owner: 301002, offset: 0) wanted: 4, have: 10
ERROR: extent[1278093537280, 925696] referencer count mismatch (root: 2192, owner: 301572, offset: 0) wanted: 7, have: 8
ERROR: extent[1278124670976, 2674688] referencer count mismatch (root: 2192, owner: 301605, offset: 0) wanted: 5, have: 21
ERROR: extent[1278146277376, 2650112] referencer count mismatch (root: 2192, owner: 301615, offset: 0) wanted: 2, have: 21
ERROR: extent[1278179549184, 1032192] referencer count mismatch (root: 2192, owner: 301639, offset: 0) wanted: 7, have: 8
ERROR: extent[1278216667136, 2318336] referencer count mismatch (root: 2192, owner: 301660, offset: 0) wanted: 10, have: 18
ERROR: extent[1278231187456, 2211840] referencer count mismatch (root: 2192, owner: 301671, offset: 0) wanted: 12, have: 17
ERROR: extent[1278264672256, 1740800] referencer count mismatch (root: 2192, owner: 301691, offset: 0) wanted: 10, have: 14
ERROR: extent[1278339145728, 2289664] referencer count mismatch (root: 2192, owner: 302038, offset: 0) wanted: 6, have: 18
ERROR: extent[1278385999872, 1118208] referencer count mismatch (root: 2192, owner: 302074, offset: 0) wanted: 3, have: 9
ERROR: extent[1278455382016, 2969600] referencer count mismatch (root: 2192, owner: 302127, offset: 0) wanted: 20, have: 23
ERROR: extent[1278572425216, 761856] referencer count mismatch (root: 2192, owner: 301121, offset: 0) wanted: 1, have: 6
ERROR: extent[1278588706816, 1871872] referencer count mismatch (root: 2192, owner: 301134, offset: 0) wanted: 3, have: 15
ERROR: extent[1278617640960, 2121728] referencer count mismatch (root: 2192, owner: 301161, offset: 0) wanted: 16, have: 17
ERROR: extent[1278657347584, 1265664] referencer count mismatch (root: 2192, owner: 301188, offset: 0) wanted: 2, have: 10
ERROR: extent[1278674927616, 1822720] referencer count mismatch (root: 2192, owner: 301203, offset: 0) wanted: 9, have: 14
ERROR: extent[1278685224960, 1097728] referencer count mismatch (root: 2192, owner: 301208, offset: 0) wanted: 5, have: 9
ERROR: extent[1278730657792, 2580480] referencer count mismatch (root: 2192, owner: 301243, offset: 0) wanted: 15, have: 20
ERROR: extent[1278749044736, 3244032] referencer count mismatch (root: 2192, owner: 301256, offset: 0) wanted: 4, have: 25
ERROR: extent[1278770667520, 2633728] referencer count mismatch (root: 2192, owner: 301267, offset: 0) wanted: 3, have: 21
ERROR: extent[1278789185536, 364544] referencer count mismatch (root: 2192, owner: 301280, offset: 0) wanted: 1, have: 3
ERROR: extent[1279092916224, 1806336] referencer count mismatch (root: 2192, owner: 302234, offset: 0) wanted: 9, have: 14
ERROR: extent[1279329845248, 2109440] referencer count mismatch (root: 2192, owner: 302421, offset: 0) wanted: 10, have: 17
ERROR: extent[1279415287808, 1380352] referencer count mismatch (root: 2192, owner: 302465, offset: 0) wanted: 10, have: 11
ERROR: extent[1279732752384, 286720] referencer count mismatch (root: 2192, owner: 302526, offset: 0) wanted: 1, have: 3
ERROR: extent[1282148016128, 262144] referencer count mismatch (root: 2192, owner: 305352, offset: 0) wanted: 1, have: 2
ERROR: extent[1282167455744, 512000] referencer count mismatch (root: 2192, owner: 305401, offset: 0) wanted: 1, have: 4
ERROR: extent[1282197204992, 401408] referencer count mismatch (root: 2192, owner: 305500, offset: 0) wanted: 3, have: 4
ERROR: extent[1282218409984, 479232] referencer count mismatch (root: 2192, owner: 305576, offset: 0) wanted: 1, have: 4
ERROR: extent[1282224476160, 446464] referencer count mismatch (root: 2192, owner: 305590, offset: 0) wanted: 1, have: 4
ERROR: extent[1282228514816, 348160] referencer count mismatch (root: 2192, owner: 305606, offset: 0) wanted: 2, have: 3
ERROR: extent[1282247221248, 397312] referencer count mismatch (root: 2192, owner: 305652, offset: 0) wanted: 2, have: 4
ERROR: extent[1282264170496, 16564224] referencer count mismatch (root: 2192, owner: 305700, offset: 0) wanted: 103, have: 127
ERROR: extent[1282293092352, 33533952] referencer count mismatch (root: 2192, owner: 305712, offset: 0) wanted: 134, have: 256
ERROR: extent[1282355109888, 14557184] referencer count mismatch (root: 2192, owner: 305747, offset: 0) wanted: 78, have: 112
ERROR: extent[1282395701248, 36519936] referencer count mismatch (root: 2192, owner: 305768, offset: 0) wanted: 202, have: 279
ERROR: extent[1282460270592, 18821120] referencer count mismatch (root: 2192, owner: 305801, offset: 0) wanted: 79, have: 144
ERROR: extent[1282479751168, 44199936] referencer count mismatch (root: 2192, owner: 305787, offset: 0) wanted: 315, have: 338
ERROR: extent[1282537447424, 4571136] referencer count mismatch (root: 2192, owner: 305803, offset: 0) wanted: 7, have: 35
ERROR: extent[1282574532608, 1294336] referencer count mismatch (root: 2192, owner: 305825, offset: 0) wanted: 9, have: 10
ERROR: extent[1282636775424, 24395776] referencer count mismatch (root: 2192, owner: 306200, offset: 0) wanted: 88, have: 187
ERROR: extent[1282663895040, 29925376] referencer count mismatch (root: 2192, owner: 306211, offset: 0) wanted: 107, have: 229
ERROR: extent[1282727006208, 1732608] referencer count mismatch (root: 2192, owner: 306284, offset: 0) wanted: 4, have: 14
ERROR: extent[1283804733440, 2121728] referencer count mismatch (root: 2192, owner: 307509, offset: 0) wanted: 7, have: 17
ERROR: extent[1285734670336, 1257472] referencer count mismatch (root: 2192, owner: 309442, offset: 0) wanted: 4, have: 10
ERROR: extent[1285748789248, 1019904] referencer count mismatch (root: 2192, owner: 309453, offset: 0) wanted: 3, have: 8
ERROR: extent[1285771227136, 1200128] referencer count mismatch (root: 2192, owner: 309482, offset: 0) wanted: 3, have: 10
ERROR: extent[1285781032960, 1413120] referencer count mismatch (root: 2192, owner: 309491, offset: 0) wanted: 3, have: 11
ERROR: extent[1285794684928, 1384448] referencer count mismatch (root: 2192, owner: 309511, offset: 0) wanted: 10, have: 11
ERROR: extent[1285803339776, 35004416] referencer count mismatch (root: 2192, owner: 309519, offset: 0) wanted: 103, have: 268
ERROR: extent[1285898678272, 74964992] referencer count mismatch (root: 2192, owner: 309539, offset: 0) wanted: 553, have: 572
ERROR: extent[1286063390720, 901120] referencer count mismatch (root: 2192, owner: 309818, offset: 0) wanted: 2, have: 7
ERROR: extent[1286080565248, 581632] referencer count mismatch (root: 2192, owner: 309832, offset: 0) wanted: 2, have: 5
ERROR: extent[1286091714560, 55832576] referencer count mismatch (root: 2192, owner: 309846, offset: 0) wanted: 421, have: 426
ERROR: extent[1286224744448, 1101824] referencer count mismatch (root: 2192, owner: 309914, offset: 0) wanted: 1, have: 9
ERROR: extent[1286245924864, 46239744] referencer count mismatch (root: 2192, owner: 309950, offset: 0) wanted: 208, have: 353
ERROR: extent[1286345191424, 61583360] referencer count mismatch (root: 2192, owner: 310410, offset: 0) wanted: 467, have: 470
ERROR: extent[1286411018240, 946176] referencer count mismatch (root: 2192, owner: 310415, offset: 0) wanted: 3, have: 8
ERROR: extent[1286445035520, 1765376] referencer count mismatch (root: 2192, owner: 310447, offset: 0) wanted: 12, have: 14
ERROR: extent[1286493294592, 1101824] referencer count mismatch (root: 2192, owner: 310497, offset: 0) wanted: 7, have: 9
ERROR: extent[1286503321600, 1622016] referencer count mismatch (root: 2192, owner: 310505, offset: 0) wanted: 4, have: 13
ERROR: extent[1286561673216, 1445888] referencer count mismatch (root: 2192, owner: 310586, offset: 0) wanted: 5, have: 12
ERROR: extent[1286589431808, 1449984] referencer count mismatch (root: 2192, owner: 310602, offset: 0) wanted: 2, have: 12
ERROR: extent[1286600970240, 1220608] referencer count mismatch (root: 2192, owner: 310616, offset: 0) wanted: 1, have: 10
ERROR: extent[1286639788032, 1437696] referencer count mismatch (root: 2192, owner: 310651, offset: 0) wanted: 1, have: 11
ERROR: extent[1286727856128, 1028096] referencer count mismatch (root: 2192, owner: 310773, offset: 0) wanted: 5, have: 8
ERROR: extent[1286823809024, 1056768] referencer count mismatch (root: 2192, owner: 310818, offset: 0) wanted: 3, have: 9
ERROR: extent[1286846889984, 60305408] referencer count mismatch (root: 2192, owner: 310860, offset: 0) wanted: 163, have: 461
ERROR: extent[1286965108736, 12251136] referencer count mismatch (root: 2192, owner: 310890, offset: 0) wanted: 52, have: 94
ERROR: extent[1286995849216, 37302272] referencer count mismatch (root: 2192, owner: 310896, offset: 0) wanted: 267, have: 285
ERROR: extent[1287054004224, 59039744] referencer count mismatch (root: 2192, owner: 310921, offset: 0) wanted: 291, have: 451
ERROR: extent[1287136403456, 2404352] referencer count mismatch (root: 2192, owner: 310936, offset: 0) wanted: 3, have: 19
ERROR: extent[1287167025152, 700416] referencer count mismatch (root: 2192, owner: 310973, offset: 0) wanted: 5, have: 6
ERROR: extent[1287178797056, 831488] referencer count mismatch (root: 2192, owner: 310993, offset: 0) wanted: 4, have: 7
ERROR: extent[1287217319936, 610304] referencer count mismatch (root: 2192, owner: 308913, offset: 0) wanted: 3, have: 5
ERROR: extent[1287247785984, 819200] referencer count mismatch (root: 2192, owner: 309283, offset: 0) wanted: 4, have: 7
ERROR: extent[1287278882816, 671744] referencer count mismatch (root: 2192, owner: 309344, offset: 0) wanted: 3, have: 6
ERROR: extent[1288091054080, 524288] referencer count mismatch (root: 2192, owner: 312832, offset: 0) wanted: 1, have: 4
ERROR: extent[1288558190592, 134217728] referencer count mismatch (root: 2192, owner: 313603, offset: 0) wanted: 979, have: 1024
ERROR: extent[1288755884032, 1441792] referencer count mismatch (root: 2192, owner: 313655, offset: 0) wanted: 4, have: 11
ERROR: extent[1289352884224, 1601536] referencer count mismatch (root: 2192, owner: 313787, offset: 0) wanted: 2, have: 13
ERROR: extent[1289358970880, 134217728] referencer count mismatch (root: 2192, owner: 313792, offset: 0) wanted: 886, have: 1024
ERROR: extent[1289960189952, 1089536] referencer count mismatch (root: 2192, owner: 313879, offset: 0) wanted: 3, have: 9
ERROR: extent[1290752475136, 2215936] referencer count mismatch (root: 2192, owner: 313932, offset: 0) wanted: 4, have: 17
ERROR: extent[1290786590720, 1318912] referencer count mismatch (root: 2192, owner: 313959, offset: 0) wanted: 2, have: 11
ERROR: extent[1290945564672, 1662976] referencer count mismatch (root: 2192, owner: 314025, offset: 0) wanted: 6, have: 13
ERROR: extent[1291138732032, 593920] referencer count mismatch (root: 2192, owner: 315673, offset: 0) wanted: 1, have: 5
ERROR: extent[1291221585920, 1675264] referencer count mismatch (root: 2192, owner: 315837, offset: 0) wanted: 9, have: 13
ERROR: extent[1291272941568, 1454080] referencer count mismatch (root: 2192, owner: 315873, offset: 0) wanted: 8, have: 12
ERROR: extent[1291338227712, 2686976] referencer count mismatch (root: 2192, owner: 315984, offset: 0) wanted: 5, have: 21
ERROR: extent[1291353747456, 2375680] referencer count mismatch (root: 2192, owner: 315990, offset: 0) wanted: 8, have: 19
ERROR: extent[1291470073856, 389120] referencer count mismatch (root: 2192, owner: 316139, offset: 0) wanted: 2, have: 3
ERROR: extent[1291477278720, 2109440] referencer count mismatch (root: 2192, owner: 316456, offset: 0) wanted: 13, have: 17
ERROR: extent[1291489882112, 134217728] referencer count mismatch (root: 2192, owner: 316466, offset: 0) wanted: 969, have: 1024
ERROR: extent[1291647651840, 2347008] referencer count mismatch (root: 2192, owner: 316490, offset: 0) wanted: 11, have: 18
ERROR: extent[1291662057472, 2035712] referencer count mismatch (root: 2192, owner: 316499, offset: 0) wanted: 13, have: 16
ERROR: extent[1292130070528, 1155072] referencer count mismatch (root: 2192, owner: 316627, offset: 0) wanted: 5, have: 9
ERROR: extent[1292336484352, 360448] referencer count mismatch (root: 2192, owner: 316719, offset: 0) wanted: 1, have: 3
ERROR: extent[1292668243968, 1474560] referencer count mismatch (root: 2192, owner: 316952, offset: 0) wanted: 4, have: 12
ERROR: extent[1292695543808, 1359872] referencer count mismatch (root: 2192, owner: 316976, offset: 0) wanted: 10, have: 11
ERROR: extent[1292714188800, 1687552] referencer count mismatch (root: 2192, owner: 316988, offset: 0) wanted: 2, have: 13
ERROR: extent[1292765691904, 1769472] referencer count mismatch (root: 2192, owner: 317017, offset: 0) wanted: 10, have: 14
ERROR: extent[1292781031424, 892928] referencer count mismatch (root: 2192, owner: 317032, offset: 0) wanted: 1, have: 7
ERROR: extent[1292795318272, 929792] referencer count mismatch (root: 2192, owner: 317041, offset: 0) wanted: 7, have: 8
ERROR: extent[1292812013568, 2830336] referencer count mismatch (root: 2192, owner: 317052, offset: 0) wanted: 11, have: 22
ERROR: extent[1292875067392, 1970176] referencer count mismatch (root: 2192, owner: 317098, offset: 0) wanted: 3, have: 16
ERROR: extent[1292889546752, 134217728] referencer count mismatch (root: 2192, owner: 317212, offset: 0) wanted: 856, have: 1024
ERROR: extent[1293217169408, 25329664] referencer count mismatch (root: 2192, owner: 317212, offset: 310116352) wanted: 59, have: 194
ERROR: extent[1293396553728, 2338816] referencer count mismatch (root: 2192, owner: 317168, offset: 0) wanted: 3, have: 18
ERROR: extent[1293473275904, 33972224] referencer count mismatch (root: 2192, owner: 317183, offset: 0) wanted: 230, have: 260
ERROR: extent[1293507248128, 134217728] referencer count mismatch (root: 2192, owner: 317186, offset: 0) wanted: 883, have: 1024
ERROR: extent[1294790914048, 1032192] referencer count mismatch (root: 2192, owner: 314542, offset: 0) wanted: 4, have: 8
ERROR: extent[1295180226560, 2105344] referencer count mismatch (root: 2192, owner: 314715, offset: 0) wanted: 3, have: 17
ERROR: extent[1295194308608, 1912832] referencer count mismatch (root: 2192, owner: 314732, offset: 0) wanted: 12, have: 15
ERROR: extent[1295208464384, 1626112] referencer count mismatch (root: 2192, owner: 314740, offset: 0) wanted: 3, have: 13
ERROR: extent[1295395008512, 68243456] referencer count mismatch (root: 2192, owner: 314787, offset: 0) wanted: 203, have: 521
ERROR: extent[1295463251968, 44236800] referencer count mismatch (root: 2192, owner: 314742, offset: 134217728) wanted: 294, have: 338
ERROR: extent[1295510691840, 1417216] referencer count mismatch (root: 2192, owner: 314797, offset: 0) wanted: 7, have: 11
ERROR: extent[1295590973440, 2424832] referencer count mismatch (root: 2192, owner: 314875, offset: 0) wanted: 5, have: 19
ERROR: extent[1295600427008, 82325504] referencer count mismatch (root: 2192, owner: 314890, offset: 0) wanted: 566, have: 629
ERROR: extent[1295719088128, 2453504] referencer count mismatch (root: 2192, owner: 314911, offset: 0) wanted: 11, have: 19
ERROR: extent[1295735943168, 134217728] referencer count mismatch (root: 2192, owner: 314922, offset: 0) wanted: 995, have: 1024
ERROR: extent[1296418197504, 585728] referencer count mismatch (root: 2192, owner: 314944, offset: 0) wanted: 3, have: 5
ERROR: extent[1296969801728, 68718592] referencer count mismatch (root: 2192, owner: 314933, offset: 0) wanted: 410, have: 525
ERROR: extent[1297821396992, 4706304] referencer count mismatch (root: 2192, owner: 315554, offset: 0) wanted: 3, have: 36
ERROR: extent[1297894973440, 507904] referencer count mismatch (root: 2192, owner: 317614, offset: 0) wanted: 1, have: 4
ERROR: extent[1297908019200, 2043904] referencer count mismatch (root: 2192, owner: 317628, offset: 0) wanted: 3, have: 16
ERROR: extent[1297939410944, 2150400] referencer count mismatch (root: 2192, owner: 317661, offset: 0) wanted: 16, have: 17
ERROR: extent[1298589421568, 2215936] referencer count mismatch (root: 2192, owner: 317759, offset: 0) wanted: 3, have: 17
ERROR: extent[1298620256256, 30285824] referencer count mismatch (root: 2192, owner: 317787, offset: 0) wanted: 127, have: 232
ERROR: extent[1298674737152, 27983872] referencer count mismatch (root: 2192, owner: 329099, offset: 0) wanted: 207, have: 214
ERROR: extent[1298702852096, 471040] referencer count mismatch (root: 2192, owner: 317815, offset: 0) wanted: 3, have: 4
ERROR: extent[1298703323136, 134217728] referencer count mismatch (root: 2192, owner: 329098, offset: 0) wanted: 844, have: 1024
ERROR: extent[1298845536256, 53420032] referencer count mismatch (root: 2192, owner: 317816, offset: 134217728) wanted: 310, have: 408
ERROR: extent[1298900983808, 10428416] referencer count mismatch (root: 2192, owner: 329096, offset: 0) wanted: 66, have: 80
ERROR: extent[1298900983808, 10428416] referencer count mismatch (root: 2192, owner: 317825, offset: 0) wanted: 51, have: 80
ERROR: extent[1298911412224, 36126720] referencer count mismatch (root: 2192, owner: 317826, offset: 0) wanted: 117, have: 276
ERROR: extent[1298911412224, 36126720] referencer count mismatch (root: 2192, owner: 329095, offset: 0) wanted: 137, have: 276
ERROR: extent[1298953486336, 90873856] referencer count mismatch (root: 2192, owner: 317843, offset: 0) wanted: 596, have: 694
ERROR: extent[1298953486336, 90873856] referencer count mismatch (root: 2192, owner: 329093, offset: 0) wanted: 521, have: 694
ERROR: extent[1299215028224, 2027520] referencer count mismatch (root: 2192, owner: 317878, offset: 0) wanted: 4, have: 16
ERROR: extent[1299275636736, 2654208] referencer count mismatch (root: 2192, owner: 317900, offset: 0) wanted: 5, have: 21
ERROR: extent[1299303247872, 4628480] referencer count mismatch (root: 2192, owner: 317931, offset: 0) wanted: 7, have: 36
ERROR: extent[1299368595456, 19660800] referencer count mismatch (root: 2192, owner: 317850, offset: 134217728) wanted: 88, have: 150
ERROR: extent[1299406565376, 888832] referencer count mismatch (root: 2192, owner: 317986, offset: 0) wanted: 6, have: 7
ERROR: extent[1299640987648, 2568192] referencer count mismatch (root: 2192, owner: 318047, offset: 0) wanted: 16, have: 20
ERROR: extent[1300195487744, 2138112] referencer count mismatch (root: 2192, owner: 318075, offset: 0) wanted: 5, have: 17
ERROR: extent[1300212133888, 27443200] referencer count mismatch (root: 2192, owner: 318089, offset: 0) wanted: 190, have: 210
ERROR: extent[1334735675392, 106651648] referencer count mismatch (root: 2192, owner: 329091, offset: 0) wanted: 593, have: 814
ERROR: extent[1334842327040, 103424000] referencer count mismatch (root: 2192, owner: 329089, offset: 0) wanted: 701, have: 790
ERROR: extent[1342594707456, 2465792] referencer count mismatch (root: 2192, owner: 328711, offset: 0) wanted: 18, have: 19
ERROR: extent[1386454097920, 1404928] referencer count mismatch (root: 2192, owner: 338307, offset: 0) wanted: 2, have: 10
ERROR: extent[1386556035072, 1564672] referencer count mismatch (root: 2192, owner: 338195, offset: 0) wanted: 5, have: 11
ERROR: extent[1386676543488, 1310720] referencer count mismatch (root: 2192, owner: 340273, offset: 0) wanted: 5, have: 9
ERROR: extent[1405711015936, 3756032] referencer count mismatch (root: 2192, owner: 354260, offset: 0) wanted: 2, have: 29
ERROR: extent[1406415671296, 1679360] referencer count mismatch (root: 2192, owner: 354255, offset: 0) wanted: 1, have: 13
ERROR: extent[1419697254400, 1732608] referencer count mismatch (root: 2192, owner: 327333, offset: 0) wanted: 8, have: 14
ERROR: extent[1419709677568, 1703936] referencer count mismatch (root: 2192, owner: 327635, offset: 0) wanted: 9, have: 13
ERROR: errors found in extent allocation tree or chunk allocation
checking free space cache
checking fs roots
checking only csum items (without verifying data)
checking root refs
found 828150579200 bytes used, error(s) found
total csum bytes: 803625824
total tree bytes: 5035229184
total fs tree bytes: 3999711232
total extent tree bytes: 179011584
btree space waste bytes: 781874592
file data blocks allocated: 20051143835648
 referenced 3869872529408
[chris@f28s ~]$
[chris@f28s ~]$ 

Re: btrfs check lowmem vs original

[Su Yue]

On 06/20/2018 10:19 AM, Chris Murphy wrote:
> I've retested btrfs-progs 4.17 lowmem mode and still get a bunch of
> 'referencer count mismatch' that I don't see with original mode.
> 
My bad. The bug is existed for long time since I changed lowmem mode
to traverse all trees.

Due to laziness then time passed, I have lost images you provided and
links are invalid.
It will be so nice of you if a image can be provided again.

I will look for causes immediately.

Thanks for the reports.

> e.g.
> ERROR: extent[1299275636736, 2654208] referencer count mismatch (root:
> 2192, owner: 317900, offset: 0) wanted: 5, have: 21
> 
> Complete output attached as a text file, since the MUA will mess up
> the line wrap.
> 
> 
> --
> Chris Murphy
> 
>