* Need help: super_total_bytes mismatch with fs_devices total_rw_bytes @ 2019-08-24 6:48 Patrick Dijkgraaf 2019-08-24 11:01 ` Qu Wenruo 0 siblings, 1 reply; 6+ messages in thread From: Patrick Dijkgraaf @ 2019-08-24 6:48 UTC (permalink / raw) To: linux-btrfs Hi all, My server hung this morning, and I had to hard-reset is. I did not apply any updates. After the reboot, my FS won't mount: [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): super_total_bytes 92017957797888 mismatch with fs_devices total_rw_bytes 184035915595776 [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to read chunk tree: -22 [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): open_ctree failed However, running btrfs rescue shows: root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2 No device size related problem found FS config is shown below: [root@cornelis ~]# btrfs fi show Label: 'cornelis-btrfs' uuid: ac643516-670e-40f3-aa4c-f329fc3795fd Total devices 1 FS bytes used 536.05GiB devid 1 size 800.00GiB used 630.02GiB path /dev/mapper/cornelis- cornelis--btrfs Label: 'data' uuid: 43472491-7bb3-418c-b476-874a52e8b2b0 Total devices 16 FS bytes used 36.61TiB devid 1 size 7.28TiB used 2.65TiB path /dev/sde2 devid 2 size 3.64TiB used 2.65TiB path /dev/sdf2 devid 3 size 3.64TiB used 2.65TiB path /dev/sdg2 devid 4 size 7.28TiB used 2.65TiB path /dev/sdh2 devid 5 size 3.64TiB used 2.65TiB path /dev/sdi2 devid 6 size 7.28TiB used 2.65TiB path /dev/sdj2 devid 7 size 3.64TiB used 2.65TiB path /dev/sdk2 devid 8 size 3.64TiB used 2.65TiB path /dev/sdl2 devid 9 size 7.28TiB used 2.65TiB path /dev/sdm2 devid 10 size 3.64TiB used 2.65TiB path /dev/sdn2 devid 11 size 7.28TiB used 2.65TiB path /dev/sdo2 devid 12 size 3.64TiB used 2.65TiB path /dev/sdp2 devid 13 size 7.28TiB used 2.65TiB path /dev/sdq2 devid 14 size 7.28TiB used 2.65TiB path /dev/sdr2 devid 15 size 3.64TiB used 2.65TiB path /dev/sds2 devid 16 size 3.64TiB used 2.65TiB path /dev/sdt2 Other info: [root@cornelis ~]# uname -r 4.18.16-arch1-1-ARCH I was able to mount is using: [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 /mnt/data Now updating my backup, but I REALLY hope to get this fixed on the production server! -- Cheers, Patrick Dijkgraaf ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes 2019-08-24 6:48 Need help: super_total_bytes mismatch with fs_devices total_rw_bytes Patrick Dijkgraaf @ 2019-08-24 11:01 ` Qu Wenruo 2019-08-24 12:05 ` Patrick Dijkgraaf 0 siblings, 1 reply; 6+ messages in thread From: Qu Wenruo @ 2019-08-24 11:01 UTC (permalink / raw) To: Patrick Dijkgraaf, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 2701 bytes --] On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote: > Hi all, > > My server hung this morning, and I had to hard-reset is. I did not > apply any updates. After the reboot, my FS won't mount: > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): super_total_bytes > 92017957797888 mismatch with fs_devices total_rw_bytes 184035915595776 > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to read > chunk tree: -22 > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): open_ctree failed > > However, running btrfs rescue shows: > root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2 > No device size related problem found That's strange. Would you please dump the chunk tree and super blocks? # btrfs ins dump-super -fFa /dev/sdh2 # btrfs ins dump-tree -t chunk /dev/sdh2 And, have you tried to mount using different devices? If it's some super blocks get corrupted, using a different device to mount may help. (With that said, it's better to call that dump-super for each device) > > FS config is shown below: > [root@cornelis ~]# btrfs fi show > Label: 'cornelis-btrfs' uuid: ac643516-670e-40f3-aa4c-f329fc3795fd > Total devices 1 FS bytes used 536.05GiB > devid 1 size 800.00GiB used 630.02GiB path /dev/mapper/cornelis- > cornelis--btrfs > > Label: 'data' uuid: 43472491-7bb3-418c-b476-874a52e8b2b0 > Total devices 16 FS bytes used 36.61TiB > devid 1 size 7.28TiB used 2.65TiB path /dev/sde2 > devid 2 size 3.64TiB used 2.65TiB path /dev/sdf2 > devid 3 size 3.64TiB used 2.65TiB path /dev/sdg2 > devid 4 size 7.28TiB used 2.65TiB path /dev/sdh2 > devid 5 size 3.64TiB used 2.65TiB path /dev/sdi2 > devid 6 size 7.28TiB used 2.65TiB path /dev/sdj2 > devid 7 size 3.64TiB used 2.65TiB path /dev/sdk2 > devid 8 size 3.64TiB used 2.65TiB path /dev/sdl2 > devid 9 size 7.28TiB used 2.65TiB path /dev/sdm2 > devid 10 size 3.64TiB used 2.65TiB path /dev/sdn2 > devid 11 size 7.28TiB used 2.65TiB path /dev/sdo2 > devid 12 size 3.64TiB used 2.65TiB path /dev/sdp2 > devid 13 size 7.28TiB used 2.65TiB path /dev/sdq2 > devid 14 size 7.28TiB used 2.65TiB path /dev/sdr2 > devid 15 size 3.64TiB used 2.65TiB path /dev/sds2 > devid 16 size 3.64TiB used 2.65TiB path /dev/sdt2 What's the profile used on so many devices? RAID10? The simplest way to fix it is to just update the Thanks, Qu > > Other info: > [root@cornelis ~]# uname -r > 4.18.16-arch1-1-ARCH > > I was able to mount is using: > [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 /mnt/data > > Now updating my backup, but I REALLY hope to get this fixed on the > production server! > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes 2019-08-24 11:01 ` Qu Wenruo @ 2019-08-24 12:05 ` Patrick Dijkgraaf 2019-08-24 13:24 ` Qu Wenruo 0 siblings, 1 reply; 6+ messages in thread From: Patrick Dijkgraaf @ 2019-08-24 12:05 UTC (permalink / raw) To: Qu Wenruo, linux-btrfs Thanks for the quick reply! See responses inline. On Sat, 2019-08-24 at 19:01 +0800, Qu Wenruo wrote: > On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote: > > Hi all, > > > > My server hung this morning, and I had to hard-reset is. I did not > > apply any updates. After the reboot, my FS won't mount: > > > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): > > super_total_bytes > > 92017957797888 mismatch with fs_devices total_rw_bytes > > 184035915595776 > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to > > read > > chunk tree: -22 > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): open_ctree > > failed > > > > However, running btrfs rescue shows: > > root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2 > > No device size related problem found > > That's strange. > > Would you please dump the chunk tree and super blocks? > # btrfs ins dump-super -fFa /dev/sdh2 See: https://pastebin.com/f5Wn15sx > # btrfs ins dump-tree -t chunk /dev/sdh2 This output is too large for pastebin. The output is viewable/downloadable here: https://kwek.duckstad.net/tree.txt > And, have you tried to mount using different devices? If it's some > super > blocks get corrupted, using a different device to mount may help. > (With that said, it's better to call that dump-super for each device) Tried it with sde and sdh. Both give the same error. > > FS config is shown below: > > [root@cornelis ~]# btrfs fi show > > Label: 'cornelis-btrfs' uuid: ac643516-670e-40f3-aa4c-f329fc3795fd > > Total devices 1 FS bytes used 536.05GiB > > devid 1 size 800.00GiB used 630.02GiB path /dev/mapper/cornelis- > > cornelis--btrfs > > > > Label: 'data' uuid: 43472491-7bb3-418c-b476-874a52e8b2b0 > > Total devices 16 FS bytes used 36.61TiB > > devid 1 size 7.28TiB used 2.65TiB path /dev/sde2 > > devid 2 size 3.64TiB used 2.65TiB path /dev/sdf2 > > devid 3 size 3.64TiB used 2.65TiB path /dev/sdg2 > > devid 4 size 7.28TiB used 2.65TiB path /dev/sdh2 > > devid 5 size 3.64TiB used 2.65TiB path /dev/sdi2 > > devid 6 size 7.28TiB used 2.65TiB path /dev/sdj2 > > devid 7 size 3.64TiB used 2.65TiB path /dev/sdk2 > > devid 8 size 3.64TiB used 2.65TiB path /dev/sdl2 > > devid 9 size 7.28TiB used 2.65TiB path /dev/sdm2 > > devid 10 size 3.64TiB used 2.65TiB path /dev/sdn2 > > devid 11 size 7.28TiB used 2.65TiB path /dev/sdo2 > > devid 12 size 3.64TiB used 2.65TiB path /dev/sdp2 > > devid 13 size 7.28TiB used 2.65TiB path /dev/sdq2 > > devid 14 size 7.28TiB used 2.65TiB path /dev/sdr2 > > devid 15 size 3.64TiB used 2.65TiB path /dev/sds2 > > devid 16 size 3.64TiB used 2.65TiB path /dev/sdt2 > > What's the profile used on so many devices? > RAID10? It's RAID6. I know the risk, although I believe that should be minimal nowadays. > The simplest way to fix it is to just update the Nice teaser! 😉 What should I update? > Thanks, > Qu > > Other info: > > [root@cornelis ~]# uname -r > > 4.18.16-arch1-1-ARCH > > > > I was able to mount is using: > > [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 /mnt/data > > > > Now updating my backup, but I REALLY hope to get this fixed on the > > production server! > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes 2019-08-24 12:05 ` Patrick Dijkgraaf @ 2019-08-24 13:24 ` Qu Wenruo 2019-08-25 6:41 ` Patrick Dijkgraaf 0 siblings, 1 reply; 6+ messages in thread From: Qu Wenruo @ 2019-08-24 13:24 UTC (permalink / raw) To: Patrick Dijkgraaf, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 2504 bytes --] On 2019/8/24 下午8:05, Patrick Dijkgraaf wrote: > Thanks for the quick reply! > See responses inline. > > On Sat, 2019-08-24 at 19:01 +0800, Qu Wenruo wrote: >> On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote: >>> Hi all, >>> >>> My server hung this morning, and I had to hard-reset is. I did not >>> apply any updates. After the reboot, my FS won't mount: >>> >>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): >>> super_total_bytes >>> 92017957797888 mismatch with fs_devices total_rw_bytes >>> 184035915595776 >>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to >>> read >>> chunk tree: -22 >>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): open_ctree >>> failed >>> >>> However, running btrfs rescue shows: >>> root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2 >>> No device size related problem found >> >> That's strange. >> >> Would you please dump the chunk tree and super blocks? >> # btrfs ins dump-super -fFa /dev/sdh2 > > See: https://pastebin.com/f5Wn15sx Did a quick calculation, from your fi show result, it's 83.72TiB, thus the super total_bytes is correct. It's the kernel doing incorrect calculation for its fs_devices->total_rw_bytes. This matches the output of dump-super. No wonder why btrfs-progs refuse to fix. > >> # btrfs ins dump-tree -t chunk /dev/sdh2 > > This output is too large for pastebin. The output is > viewable/downloadable here: https://kwek.duckstad.net/tree.txt This also proves your chunk tree is CORRECT! The sum of all devices is 92017957797888, which matches with super block. [...] > >> The simplest way to fix it is to just update the > > Nice teaser! 😉 What should I update? Sorry, I meant to say just update the "superblock", but it turns out, it's something wrong with your kernel. Probably some old bug we fixed before. Would you try to use latest ARCH kernel from an Archiso to try to mount it RO (just to be safe)? I checked latest v5.3-rc kernels, haven't found an obvious problem with the fs_devices->total_rw_bytes update routines. So it may be an old bug which is already fixed. Thanks, Qu > >> Thanks, >> Qu >>> Other info: >>> [root@cornelis ~]# uname -r >>> 4.18.16-arch1-1-ARCH >>> >>> I was able to mount is using: >>> [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 /mnt/data >>> >>> Now updating my backup, but I REALLY hope to get this fixed on the >>> production server! >>> > > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes 2019-08-24 13:24 ` Qu Wenruo @ 2019-08-25 6:41 ` Patrick Dijkgraaf 2019-08-25 13:36 ` Qu Wenruo 0 siblings, 1 reply; 6+ messages in thread From: Patrick Dijkgraaf @ 2019-08-25 6:41 UTC (permalink / raw) To: Qu Wenruo, linux-btrfs Hi Qu, At the end of my first initial post, I mentioned that I finally was able to mount the volume using: mount -o usebackuproot,ro /dev/sdh2 /mnt/data The chunk tree and super blocks dumps were taken after that. Now I noticed that I was able to mount the volume without special options (same kernel version). YAY! ☺ Could it be that the "usebackuproot,ro" mount options already fixed the issue? Cheers, Patrick On Sat, 2019-08-24 at 21:24 +0800, Qu Wenruo wrote: > On 2019/8/24 下午8:05, Patrick Dijkgraaf wrote: > > Thanks for the quick reply! > > See responses inline. > > > > On Sat, 2019-08-24 at 19:01 +0800, Qu Wenruo wrote: > > > On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote: > > > > Hi all, > > > > > > > > My server hung this morning, and I had to hard-reset is. I did > > > > not > > > > apply any updates. After the reboot, my FS won't mount: > > > > > > > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): > > > > super_total_bytes > > > > 92017957797888 mismatch with fs_devices total_rw_bytes > > > > 184035915595776 > > > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to > > > > read > > > > chunk tree: -22 > > > > [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): > > > > open_ctree > > > > failed > > > > > > > > However, running btrfs rescue shows: > > > > root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2 > > > > No device size related problem found > > > > > > That's strange. > > > > > > Would you please dump the chunk tree and super blocks? > > > # btrfs ins dump-super -fFa /dev/sdh2 > > > > See: > > https://pastebin.com/f5Wn15sx > > > > Did a quick calculation, from your fi show result, it's 83.72TiB, > thus > the super total_bytes is correct. > > It's the kernel doing incorrect calculation for its > fs_devices->total_rw_bytes. > > This matches the output of dump-super. No wonder why btrfs-progs > refuse > to fix. > > > # btrfs ins dump-tree -t chunk /dev/sdh2 > > > > This output is too large for pastebin. The output is > > viewable/downloadable here: > > https://kwek.duckstad.net/tree.txt > > > > This also proves your chunk tree is CORRECT! > The sum of all devices is 92017957797888, which matches with super > block. > [...] > > > The simplest way to fix it is to just update the > > > > Nice teaser! 😉 What should I update? > > Sorry, I meant to say just update the "superblock", but it turns out, > it's something wrong with your kernel. Probably some old bug we fixed > before. > > Would you try to use latest ARCH kernel from an Archiso to try to > mount > it RO (just to be safe)? > > I checked latest v5.3-rc kernels, haven't found an obvious problem > with > the fs_devices->total_rw_bytes update routines. > > So it may be an old bug which is already fixed. > > Thanks, > Qu > > > > Thanks, > > > Qu > > > > Other info: > > > > [root@cornelis ~]# uname -r > > > > 4.18.16-arch1-1-ARCH > > > > > > > > I was able to mount is using: > > > > [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 > > > > /mnt/data > > > > > > > > Now updating my backup, but I REALLY hope to get this fixed on > > > > the > > > > production server! > > > > > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Need help: super_total_bytes mismatch with fs_devices total_rw_bytes 2019-08-25 6:41 ` Patrick Dijkgraaf @ 2019-08-25 13:36 ` Qu Wenruo 0 siblings, 0 replies; 6+ messages in thread From: Qu Wenruo @ 2019-08-25 13:36 UTC (permalink / raw) To: Patrick Dijkgraaf, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 3700 bytes --] On 2019/8/25 下午2:41, Patrick Dijkgraaf wrote: > Hi Qu, > > At the end of my first initial post, I mentioned that I finally was > able to mount the volume using: > > mount -o usebackuproot,ro /dev/sdh2 /mnt/data > > The chunk tree and super blocks dumps were taken after that. > > Now I noticed that I was able to mount the volume without special > options (same kernel version). YAY! ☺ > Could it be that the "usebackuproot,ro" mount options already fixed the > issue? Nope. It should be something wrong with the kernel code verifying the total devices space. As long as all your later mount is using ro, and no log tree replay happened, the fs should not be modified. It may be a race, but anyway, your on-disk data is completely valid, so as long as your kernel is doing something correct, it should mount the fs. Thanks, Qu > > Cheers, > Patrick > > > On Sat, 2019-08-24 at 21:24 +0800, Qu Wenruo wrote: >> On 2019/8/24 下午8:05, Patrick Dijkgraaf wrote: >>> Thanks for the quick reply! >>> See responses inline. >>> >>> On Sat, 2019-08-24 at 19:01 +0800, Qu Wenruo wrote: >>>> On 2019/8/24 下午2:48, Patrick Dijkgraaf wrote: >>>>> Hi all, >>>>> >>>>> My server hung this morning, and I had to hard-reset is. I did >>>>> not >>>>> apply any updates. After the reboot, my FS won't mount: >>>>> >>>>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): >>>>> super_total_bytes >>>>> 92017957797888 mismatch with fs_devices total_rw_bytes >>>>> 184035915595776 >>>>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): failed to >>>>> read >>>>> chunk tree: -22 >>>>> [Sat Aug 24 08:16:31 2019] BTRFS error (device sde2): >>>>> open_ctree >>>>> failed >>>>> >>>>> However, running btrfs rescue shows: >>>>> root@cornelis ~]# btrfs rescue fix-device-size /dev/sdh2 >>>>> No device size related problem found >>>> >>>> That's strange. >>>> >>>> Would you please dump the chunk tree and super blocks? >>>> # btrfs ins dump-super -fFa /dev/sdh2 >>> >>> See: >>> https://pastebin.com/f5Wn15sx >>> >> >> Did a quick calculation, from your fi show result, it's 83.72TiB, >> thus >> the super total_bytes is correct. >> >> It's the kernel doing incorrect calculation for its >> fs_devices->total_rw_bytes. >> >> This matches the output of dump-super. No wonder why btrfs-progs >> refuse >> to fix. >>>> # btrfs ins dump-tree -t chunk /dev/sdh2 >>> >>> This output is too large for pastebin. The output is >>> viewable/downloadable here: >>> https://kwek.duckstad.net/tree.txt >>> >> >> This also proves your chunk tree is CORRECT! >> The sum of all devices is 92017957797888, which matches with super >> block. >> [...] >>>> The simplest way to fix it is to just update the >>> >>> Nice teaser! 😉 What should I update? >> >> Sorry, I meant to say just update the "superblock", but it turns out, >> it's something wrong with your kernel. Probably some old bug we fixed >> before. >> >> Would you try to use latest ARCH kernel from an Archiso to try to >> mount >> it RO (just to be safe)? >> >> I checked latest v5.3-rc kernels, haven't found an obvious problem >> with >> the fs_devices->total_rw_bytes update routines. >> >> So it may be an old bug which is already fixed. >> >> Thanks, >> Qu >> >>>> Thanks, >>>> Qu >>>>> Other info: >>>>> [root@cornelis ~]# uname -r >>>>> 4.18.16-arch1-1-ARCH >>>>> >>>>> I was able to mount is using: >>>>> [root@cornelis ~]# mount -o usebackuproot,ro /dev/sdh2 >>>>> /mnt/data >>>>> >>>>> Now updating my backup, but I REALLY hope to get this fixed on >>>>> the >>>>> production server! >>>>> >>> >>> > > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-08-25 13:37 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-24 6:48 Need help: super_total_bytes mismatch with fs_devices total_rw_bytes Patrick Dijkgraaf 2019-08-24 11:01 ` Qu Wenruo 2019-08-24 12:05 ` Patrick Dijkgraaf 2019-08-24 13:24 ` Qu Wenruo 2019-08-25 6:41 ` Patrick Dijkgraaf 2019-08-25 13:36 ` Qu Wenruo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).