Need help: super_total_bytes mismatch with fs_devices total_rw_bytes

Need help: super_total_bytes mismatch with fs_devices total_rw_bytes

[Patrick Dijkgraaf]

Hi all,

My server hung this morning, and I had to hard-reset is. I did not
apply any updates. After the reboot, my FS won't mount:

[Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): super_total_bytes
92017957797888 mismatch with fs_devices total_rw_bytes 184035915595776
[Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to read
chunk tree: -22
[Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): open_ctree failed

However, running btrfs rescue shows:
root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2
No device size related problem found

FS config is shown below:
[root@cornelis ~]# btrfs fi show
Label: 'cornelis-btrfs'  uuid: ac643516-670e-40f3-aa4c-f329fc3795fd
Total devices 1 FS bytes used 536.05GiB
devid    1 size 800.00GiB used 630.02GiB path /dev/mapper/cornelis-
cornelis--btrfs

Label: 'data'  uuid: 43472491-7bb3-418c-b476-874a52e8b2b0
Total devices 16 FS bytes used 36.61TiB
devid    1 size 7.28TiB used 2.65TiB path /dev/sde2
devid    2 size 3.64TiB used 2.65TiB path /dev/sdf2
devid    3 size 3.64TiB used 2.65TiB path /dev/sdg2
devid    4 size 7.28TiB used 2.65TiB path /dev/sdh2
devid    5 size 3.64TiB used 2.65TiB path /dev/sdi2
devid    6 size 7.28TiB used 2.65TiB path /dev/sdj2
devid    7 size 3.64TiB used 2.65TiB path /dev/sdk2
devid    8 size 3.64TiB used 2.65TiB path /dev/sdl2
devid    9 size 7.28TiB used 2.65TiB path /dev/sdm2
devid   10 size 3.64TiB used 2.65TiB path /dev/sdn2
devid   11 size 7.28TiB used 2.65TiB path /dev/sdo2
devid   12 size 3.64TiB used 2.65TiB path /dev/sdp2
devid   13 size 7.28TiB used 2.65TiB path /dev/sdq2
devid   14 size 7.28TiB used 2.65TiB path /dev/sdr2
devid   15 size 3.64TiB used 2.65TiB path /dev/sds2
devid   16 size 3.64TiB used 2.65TiB path /dev/sdt2

Other info:
[root@cornelis ~]# uname -r
4.18.16-arch1-1-ARCH

I was able to mount is using:
[root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 /mnt/data

Now updating my backup, but I REALLY hope to get this fixed on the
production server!

-- 
Cheers,
Patrick Dijkgraaf

Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2701 bytes --]



On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote:
> Hi all,
> 
> My server hung this morning, and I had to hard-reset is. I did not
> apply any updates. After the reboot, my FS won't mount:
> 
> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): super_total_bytes
> 92017957797888 mismatch with fs_devices total_rw_bytes 184035915595776
> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to read
> chunk tree: -22
> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): open_ctree failed
> 
> However, running btrfs rescue shows:
> root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2
> No device size related problem found

That's strange.

Would you please dump the chunk tree and super blocks?
# btrfs ins dump-super -fFa /dev/sdh2
# btrfs ins dump-tree -t chunk /dev/sdh2

And, have you tried to mount using different devices? If it's some super
blocks get corrupted, using a different device to mount may help.
(With that said, it's better to call that dump-super for each device)

> 
> FS config is shown below:
> [root@cornelis ~]# btrfs fi show
> Label: 'cornelis-btrfs'  uuid: ac643516-670e-40f3-aa4c-f329fc3795fd
> Total devices 1 FS bytes used 536.05GiB
> devid    1 size 800.00GiB used 630.02GiB path /dev/mapper/cornelis-
> cornelis--btrfs
> 
> Label: 'data'  uuid: 43472491-7bb3-418c-b476-874a52e8b2b0
> Total devices 16 FS bytes used 36.61TiB
> devid    1 size 7.28TiB used 2.65TiB path /dev/sde2
> devid    2 size 3.64TiB used 2.65TiB path /dev/sdf2
> devid    3 size 3.64TiB used 2.65TiB path /dev/sdg2
> devid    4 size 7.28TiB used 2.65TiB path /dev/sdh2
> devid    5 size 3.64TiB used 2.65TiB path /dev/sdi2
> devid    6 size 7.28TiB used 2.65TiB path /dev/sdj2
> devid    7 size 3.64TiB used 2.65TiB path /dev/sdk2
> devid    8 size 3.64TiB used 2.65TiB path /dev/sdl2
> devid    9 size 7.28TiB used 2.65TiB path /dev/sdm2
> devid   10 size 3.64TiB used 2.65TiB path /dev/sdn2
> devid   11 size 7.28TiB used 2.65TiB path /dev/sdo2
> devid   12 size 3.64TiB used 2.65TiB path /dev/sdp2
> devid   13 size 7.28TiB used 2.65TiB path /dev/sdq2
> devid   14 size 7.28TiB used 2.65TiB path /dev/sdr2
> devid   15 size 3.64TiB used 2.65TiB path /dev/sds2
> devid   16 size 3.64TiB used 2.65TiB path /dev/sdt2

What's the profile used on so many devices?
RAID10?

The simplest way to fix it is to just update the

Thanks,
Qu
> 
> Other info:
> [root@cornelis ~]# uname -r
> 4.18.16-arch1-1-ARCH
> 
> I was able to mount is using:
> [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 /mnt/data
> 
> Now updating my backup, but I REALLY hope to get this fixed on the
> production server!
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes

[Patrick Dijkgraaf]

Thanks for the quick reply!
See responses inline.

On Sat, 2019-08-24 at 19:01 +0800, Qu Wenruo wrote:
> On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote:
> > Hi all,
> > 
> > My server hung this morning, and I had to hard-reset is. I did not
> > apply any updates. After the reboot, my FS won't mount:
> > 
> > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2):
> > super_total_bytes
> > 92017957797888 mismatch with fs_devices total_rw_bytes
> > 184035915595776
> > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to
> > read
> > chunk tree: -22
> > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): open_ctree
> > failed
> > 
> > However, running btrfs rescue shows:
> > root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2
> > No device size related problem found
> 
> That's strange.
> 
> Would you please dump the chunk tree and super blocks?
> # btrfs ins dump-super -fFa /dev/sdh2

See: https://pastebin.com/f5Wn15sx

> # btrfs ins dump-tree -t chunk /dev/sdh2

This output is too large for pastebin. The output is
viewable/downloadable here: https://kwek.duckstad.net/tree.txt

> And, have you tried to mount using different devices? If it's some
> super
> blocks get corrupted, using a different device to mount may help.
> (With that said, it's better to call that dump-super for each device)

Tried it with sde and sdh. Both give the same error.

> > FS config is shown below:
> > [root@cornelis ~]# btrfs fi show
> > Label: 'cornelis-btrfs'  uuid: ac643516-670e-40f3-aa4c-f329fc3795fd
> > Total devices 1 FS bytes used 536.05GiB
> > devid    1 size 800.00GiB used 630.02GiB path /dev/mapper/cornelis-
> > cornelis--btrfs
> > 
> > Label: 'data'  uuid: 43472491-7bb3-418c-b476-874a52e8b2b0
> > Total devices 16 FS bytes used 36.61TiB
> > devid    1 size 7.28TiB used 2.65TiB path /dev/sde2
> > devid    2 size 3.64TiB used 2.65TiB path /dev/sdf2
> > devid    3 size 3.64TiB used 2.65TiB path /dev/sdg2
> > devid    4 size 7.28TiB used 2.65TiB path /dev/sdh2
> > devid    5 size 3.64TiB used 2.65TiB path /dev/sdi2
> > devid    6 size 7.28TiB used 2.65TiB path /dev/sdj2
> > devid    7 size 3.64TiB used 2.65TiB path /dev/sdk2
> > devid    8 size 3.64TiB used 2.65TiB path /dev/sdl2
> > devid    9 size 7.28TiB used 2.65TiB path /dev/sdm2
> > devid   10 size 3.64TiB used 2.65TiB path /dev/sdn2
> > devid   11 size 7.28TiB used 2.65TiB path /dev/sdo2
> > devid   12 size 3.64TiB used 2.65TiB path /dev/sdp2
> > devid   13 size 7.28TiB used 2.65TiB path /dev/sdq2
> > devid   14 size 7.28TiB used 2.65TiB path /dev/sdr2
> > devid   15 size 3.64TiB used 2.65TiB path /dev/sds2
> > devid   16 size 3.64TiB used 2.65TiB path /dev/sdt2
> 
> What's the profile used on so many devices?
> RAID10?

It's RAID6. I know the risk, although I believe that should be minimal
nowadays.

> The simplest way to fix it is to just update the

Nice teaser! 😉 What should I update?

> Thanks,
> Qu
> > Other info:
> > [root@cornelis ~]# uname -r
> > 4.18.16-arch1-1-ARCH
> > 
> > I was able to mount is using:
> > [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 /mnt/data
> > 
> > Now updating my backup, but I REALLY hope to get this fixed on the
> > production server!
> > 

Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2504 bytes --]



On 2019/8/24 下午8:05, Patrick Dijkgraaf wrote:
> Thanks for the quick reply!
> See responses inline.
> 
> On Sat, 2019-08-24 at 19:01 +0800, Qu Wenruo wrote:
>> On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote:
>>> Hi all,
>>>
>>> My server hung this morning, and I had to hard-reset is. I did not
>>> apply any updates. After the reboot, my FS won't mount:
>>>
>>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2):
>>> super_total_bytes
>>> 92017957797888 mismatch with fs_devices total_rw_bytes
>>> 184035915595776
>>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to
>>> read
>>> chunk tree: -22
>>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): open_ctree
>>> failed
>>>
>>> However, running btrfs rescue shows:
>>> root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2
>>> No device size related problem found
>>
>> That's strange.
>>
>> Would you please dump the chunk tree and super blocks?
>> # btrfs ins dump-super -fFa /dev/sdh2
> 
> See: https://pastebin.com/f5Wn15sx

Did a quick calculation, from your fi show result, it's 83.72TiB, thus
the super total_bytes is correct.

It's the kernel doing incorrect calculation for its
fs_devices->total_rw_bytes.

This matches the output of dump-super. No wonder why btrfs-progs refuse
to fix.
> 
>> # btrfs ins dump-tree -t chunk /dev/sdh2
> 
> This output is too large for pastebin. The output is
> viewable/downloadable here: https://kwek.duckstad.net/tree.txt

This also proves your chunk tree is CORRECT!
The sum of all devices is 92017957797888, which matches with super block.
[...]
> 
>> The simplest way to fix it is to just update the
> 
> Nice teaser! 😉 What should I update?

Sorry, I meant to say just update the "superblock", but it turns out,
it's something wrong with your kernel. Probably some old bug we fixed
before.

Would you try to use latest ARCH kernel from an Archiso to try to mount
it RO (just to be safe)?

I checked latest v5.3-rc kernels, haven't found an obvious problem with
the fs_devices->total_rw_bytes update routines.

So it may be an old bug which is already fixed.

Thanks,
Qu

> 
>> Thanks,
>> Qu
>>> Other info:
>>> [root@cornelis ~]# uname -r
>>> 4.18.16-arch1-1-ARCH
>>>
>>> I was able to mount is using:
>>> [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 /mnt/data
>>>
>>> Now updating my backup, but I REALLY hope to get this fixed on the
>>> production server!
>>>
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes

[Patrick Dijkgraaf]

Hi Qu,

At the end of my first initial post, I mentioned that I finally was
able to mount the volume using:

mount -o usebackuproot,ro /dev/sdh2 /mnt/data

The chunk tree and super blocks dumps were taken after that.

Now I noticed that I was able to mount the volume without special
options (same kernel version). YAY! ☺
Could it be that the "usebackuproot,ro" mount options already fixed the
issue?

Cheers,
Patrick


On Sat, 2019-08-24 at 21:24 +0800, Qu Wenruo wrote:
> On 2019/8/24 下午8:05, Patrick Dijkgraaf wrote:
> > Thanks for the quick reply!
> > See responses inline.
> > 
> > On Sat, 2019-08-24 at 19:01 +0800, Qu Wenruo wrote:
> > > On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote:
> > > > Hi all,
> > > > 
> > > > My server hung this morning, and I had to hard-reset is. I did
> > > > not
> > > > apply any updates. After the reboot, my FS won't mount:
> > > > 
> > > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2):
> > > > super_total_bytes
> > > > 92017957797888 mismatch with fs_devices total_rw_bytes
> > > > 184035915595776
> > > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to
> > > > read
> > > > chunk tree: -22
> > > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2):
> > > > open_ctree
> > > > failed
> > > > 
> > > > However, running btrfs rescue shows:
> > > > root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2
> > > > No device size related problem found
> > > 
> > > That's strange.
> > > 
> > > Would you please dump the chunk tree and super blocks?
> > > # btrfs ins dump-super -fFa /dev/sdh2
> > 
> > See: 
> > https://pastebin.com/f5Wn15sx
> > 
> 
> Did a quick calculation, from your fi show result, it's 83.72TiB,
> thus
> the super total_bytes is correct.
> 
> It's the kernel doing incorrect calculation for its
> fs_devices->total_rw_bytes.
> 
> This matches the output of dump-super. No wonder why btrfs-progs
> refuse
> to fix.
> > > # btrfs ins dump-tree -t chunk /dev/sdh2
> > 
> > This output is too large for pastebin. The output is
> > viewable/downloadable here: 
> > https://kwek.duckstad.net/tree.txt
> > 
> 
> This also proves your chunk tree is CORRECT!
> The sum of all devices is 92017957797888, which matches with super
> block.
> [...]
> > > The simplest way to fix it is to just update the
> > 
> > Nice teaser! 😉 What should I update?
> 
> Sorry, I meant to say just update the "superblock", but it turns out,
> it's something wrong with your kernel. Probably some old bug we fixed
> before.
> 
> Would you try to use latest ARCH kernel from an Archiso to try to
> mount
> it RO (just to be safe)?
> 
> I checked latest v5.3-rc kernels, haven't found an obvious problem
> with
> the fs_devices->total_rw_bytes update routines.
> 
> So it may be an old bug which is already fixed.
> 
> Thanks,
> Qu
> 
> > > Thanks,
> > > Qu
> > > > Other info:
> > > > [root@cornelis ~]# uname -r
> > > > 4.18.16-arch1-1-ARCH
> > > > 
> > > > I was able to mount is using:
> > > > [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2
> > > > /mnt/data
> > > > 
> > > > Now updating my backup, but I REALLY hope to get this fixed on
> > > > the
> > > > production server!
> > > > 
> > 
> > 

Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3700 bytes --]



On 2019/8/25 下午2:41, Patrick Dijkgraaf wrote:
> Hi Qu,
> 
> At the end of my first initial post, I mentioned that I finally was
> able to mount the volume using:
> 
> mount -o usebackuproot,ro /dev/sdh2 /mnt/data
> 
> The chunk tree and super blocks dumps were taken after that.
> 
> Now I noticed that I was able to mount the volume without special
> options (same kernel version). YAY! ☺
> Could it be that the "usebackuproot,ro" mount options already fixed the
> issue?

Nope. It should be something wrong with the kernel code verifying the
total devices space.
As long as all your later mount is using ro, and no log tree replay
happened, the fs should not be modified.
It may be a race, but anyway, your on-disk data is completely valid, so
as long as your kernel is doing something correct, it should mount the fs.

Thanks,
Qu

> 
> Cheers,
> Patrick
> 
> 
> On Sat, 2019-08-24 at 21:24 +0800, Qu Wenruo wrote:
>> On 2019/8/24 下午8:05, Patrick Dijkgraaf wrote:
>>> Thanks for the quick reply!
>>> See responses inline.
>>>
>>> On Sat, 2019-08-24 at 19:01 +0800, Qu Wenruo wrote:
>>>> On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote:
>>>>> Hi all,
>>>>>
>>>>> My server hung this morning, and I had to hard-reset is. I did
>>>>> not
>>>>> apply any updates. After the reboot, my FS won't mount:
>>>>>
>>>>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2):
>>>>> super_total_bytes
>>>>> 92017957797888 mismatch with fs_devices total_rw_bytes
>>>>> 184035915595776
>>>>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to
>>>>> read
>>>>> chunk tree: -22
>>>>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2):
>>>>> open_ctree
>>>>> failed
>>>>>
>>>>> However, running btrfs rescue shows:
>>>>> root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2
>>>>> No device size related problem found
>>>>
>>>> That's strange.
>>>>
>>>> Would you please dump the chunk tree and super blocks?
>>>> # btrfs ins dump-super -fFa /dev/sdh2
>>>
>>> See: 
>>> https://pastebin.com/f5Wn15sx
>>>
>>
>> Did a quick calculation, from your fi show result, it's 83.72TiB,
>> thus
>> the super total_bytes is correct.
>>
>> It's the kernel doing incorrect calculation for its
>> fs_devices->total_rw_bytes.
>>
>> This matches the output of dump-super. No wonder why btrfs-progs
>> refuse
>> to fix.
>>>> # btrfs ins dump-tree -t chunk /dev/sdh2
>>>
>>> This output is too large for pastebin. The output is
>>> viewable/downloadable here: 
>>> https://kwek.duckstad.net/tree.txt
>>>
>>
>> This also proves your chunk tree is CORRECT!
>> The sum of all devices is 92017957797888, which matches with super
>> block.
>> [...]
>>>> The simplest way to fix it is to just update the
>>>
>>> Nice teaser! 😉 What should I update?
>>
>> Sorry, I meant to say just update the "superblock", but it turns out,
>> it's something wrong with your kernel. Probably some old bug we fixed
>> before.
>>
>> Would you try to use latest ARCH kernel from an Archiso to try to
>> mount
>> it RO (just to be safe)?
>>
>> I checked latest v5.3-rc kernels, haven't found an obvious problem
>> with
>> the fs_devices->total_rw_bytes update routines.
>>
>> So it may be an old bug which is already fixed.
>>
>> Thanks,
>> Qu
>>
>>>> Thanks,
>>>> Qu
>>>>> Other info:
>>>>> [root@cornelis ~]# uname -r
>>>>> 4.18.16-arch1-1-ARCH
>>>>>
>>>>> I was able to mount is using:
>>>>> [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2
>>>>> /mnt/data
>>>>>
>>>>> Now updating my backup, but I REALLY hope to get this fixed on
>>>>> the
>>>>> production server!
>>>>>
>>>
>>>
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]