Ran into "invalid block group size" bug, unclear how to proceed.

Ran into "invalid block group size" bug, unclear how to proceed.

[Mike Javorski]

Need a bit of advice here ladies / gents. I am running into an issue
which Qu Wenruo seems to have posted a patch for several weeks ago
(see https://patchwork.kernel.org/patch/10694997/).

Here is the relevant dmesg output which led me to Qu's patch.
----
[   10.032475] BTRFS critical (device sdb): corrupt leaf: root=2
block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104,
invalid block group size, have 10804527104 expect (0, 10737418240]
[   10.032493] BTRFS error (device sdb): failed to read block groups: -5
[   10.053365] BTRFS error (device sdb): open_ctree failed
----

This server has a 16 disk btrfs filesystem (RAID6) which I boot
periodically to btrfs-send snapshots to. This machine is running
ArchLinux and I had just updated  to their latest 4.19.4 kernel
package (from 4.18.10 which was working fine). I've tried updating to
the 4.19.6 kernel that is in testing, but that doesn't seem to resolve
the issue. From what I can see on kernel.org, the patch above is not
pushed to stable or to Linus' tree.

At this point the question is what to do. Is my FS toast? Could I
revert to the 4.18.10 kernel and boot safely? I don't know if the 4.19
boot process may have flipped some bits which would make reverting
problematic.

Thanks much,

- mike

Re: Ran into "invalid block group size" bug, unclear how to proceed.

[Mike Javorski]

Apologies for not scouring the mailing list completely (just
subscribed in fact) as It appears that Patrick Dijkgraaf also ran into
this issue. He went ahead with a volume rebuild, whereas I am hoping I
can recover having not run anything more than a "btrfs scan" and
"btrfs device ready" on this machine to this point since the kernel
upgrade and the FS was intact up to that point.

- mike

On Mon, Dec 3, 2018 at 7:32 PM Mike Javorski <mike.javorski@gmail.com> wrote:
>
> Need a bit of advice here ladies / gents. I am running into an issue
> which Qu Wenruo seems to have posted a patch for several weeks ago
> (see https://patchwork.kernel.org/patch/10694997/).
>
> Here is the relevant dmesg output which led me to Qu's patch.
> ----
> [   10.032475] BTRFS critical (device sdb): corrupt leaf: root=2
> block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104,
> invalid block group size, have 10804527104 expect (0, 10737418240]
> [   10.032493] BTRFS error (device sdb): failed to read block groups: -5
> [   10.053365] BTRFS error (device sdb): open_ctree failed
> ----
>
> This server has a 16 disk btrfs filesystem (RAID6) which I boot
> periodically to btrfs-send snapshots to. This machine is running
> ArchLinux and I had just updated  to their latest 4.19.4 kernel
> package (from 4.18.10 which was working fine). I've tried updating to
> the 4.19.6 kernel that is in testing, but that doesn't seem to resolve
> the issue. From what I can see on kernel.org, the patch above is not
> pushed to stable or to Linus' tree.
>
> At this point the question is what to do. Is my FS toast? Could I
> revert to the 4.18.10 kernel and boot safely? I don't know if the 4.19
> boot process may have flipped some bits which would make reverting
> problematic.
>
> Thanks much,
>
> - mike

Re: Ran into "invalid block group size" bug, unclear how to proceed.

[Chris Murphy]

On Mon, Dec 3, 2018 at 8:32 PM Mike Javorski <mike.javorski@gmail.com> wrote:
>
> Need a bit of advice here ladies / gents. I am running into an issue
> which Qu Wenruo seems to have posted a patch for several weeks ago
> (see https://patchwork.kernel.org/patch/10694997/).
>
> Here is the relevant dmesg output which led me to Qu's patch.
> ----
> [   10.032475] BTRFS critical (device sdb): corrupt leaf: root=2
> block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104,
> invalid block group size, have 10804527104 expect (0, 10737418240]
> [   10.032493] BTRFS error (device sdb): failed to read block groups: -5
> [   10.053365] BTRFS error (device sdb): open_ctree failed
> ----
>
> This server has a 16 disk btrfs filesystem (RAID6) which I boot
> periodically to btrfs-send snapshots to. This machine is running
> ArchLinux and I had just updated  to their latest 4.19.4 kernel
> package (from 4.18.10 which was working fine). I've tried updating to
> the 4.19.6 kernel that is in testing, but that doesn't seem to resolve
> the issue. From what I can see on kernel.org, the patch above is not
> pushed to stable or to Linus' tree.
>
> At this point the question is what to do. Is my FS toast? Could I
> revert to the 4.18.10 kernel and boot safely? I don't know if the 4.19
> boot process may have flipped some bits which would make reverting
> problematic.

That patch is not yet merged in linux-next so to use it, you'd need to
apply yourself and compile a kernel. I can't tell for sure if it'd
help.

But, the less you change the file system, the better chance of saving
it. I have no idea why there'd be a corrupt leaf just due to a kernel
version change, though.

Needless to say, raid56 just seems fragile once it runs into any kind
of trouble. I personally wouldn't boot off it at all. I would only
mount it from another system, ideally an installed system but a live
system with the kernel versions you need would also work. That way you
can get more information without changes, and booting will almost
immediately mount rw, if mount succeeds at all, and will write a bunch
of changes to the file system.

Whether it's a case of 4.18.10 not detecting corruption that 4.19
sees, or if 4.19 already caused it, the best chance is to not mount it
rw, and not run check --repair, until you get some feedback from a
developer.

The thing I'd like to see is
# btrfs rescue super -v /anydevice/
# btrfs insp dump-s -f /anydevice/

First command will tell us if all the supers are the same and valid
across all devices. And the second one, hopefully it's pointed to a
device with valid super, will tell us if there's a log root value
other than 0. Both of those are read only commands.


-- 
Chris Murphy

Re: Ran into "invalid block group size" bug, unclear how to proceed.

[Mike Javorski]

Apologies for the dupe Chris, I neglected to hit Reply-All.. Comments below.

On Mon, Dec 3, 2018 at 9:56 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Mon, Dec 3, 2018 at 8:32 PM Mike Javorski <mike.javorski@gmail.com> wrote:
> >
> > Need a bit of advice here ladies / gents. I am running into an issue
> > which Qu Wenruo seems to have posted a patch for several weeks ago
> > (see https://patchwork.kernel.org/patch/10694997/).
> >
> > Here is the relevant dmesg output which led me to Qu's patch.
> > ----
> > [   10.032475] BTRFS critical (device sdb): corrupt leaf: root=2
> > block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104,
> > invalid block group size, have 10804527104 expect (0, 10737418240]
> > [   10.032493] BTRFS error (device sdb): failed to read block groups: -5
> > [   10.053365] BTRFS error (device sdb): open_ctree failed
> > ----
> >
> > This server has a 16 disk btrfs filesystem (RAID6) which I boot
> > periodically to btrfs-send snapshots to. This machine is running
> > ArchLinux and I had just updated  to their latest 4.19.4 kernel
> > package (from 4.18.10 which was working fine). I've tried updating to
> > the 4.19.6 kernel that is in testing, but that doesn't seem to resolve
> > the issue. From what I can see on kernel.org, the patch above is not
> > pushed to stable or to Linus' tree.
> >
> > At this point the question is what to do. Is my FS toast? Could I
> > revert to the 4.18.10 kernel and boot safely? I don't know if the 4.19
> > boot process may have flipped some bits which would make reverting
> > problematic.
>
> That patch is not yet merged in linux-next so to use it, you'd need to
> apply yourself and compile a kernel. I can't tell for sure if it'd
> help.
>
> But, the less you change the file system, the better chance of saving
> it. I have no idea why there'd be a corrupt leaf just due to a kernel
> version change, though.
>
> Needless to say, raid56 just seems fragile once it runs into any kind
> of trouble. I personally wouldn't boot off it at all. I would only
> mount it from another system, ideally an installed system but a live
> system with the kernel versions you need would also work. That way you
> can get more information without changes, and booting will almost
> immediately mount rw, if mount succeeds at all, and will write a bunch
> of changes to the file system.
>

If the boot could corrupt the disk, that ship has already sailed as I
have previously attempted to mount the volume with the 4.19.4 and
4.19.6 kernels, but they both failed reporting the log lines in my
original message. I am hoping that Qu notices this thread at some
point as they are the author of the original patch which introduced
the check which is now failing, as well as the un-merged patch linked
earlier that adjusts the check condition.

What I don't know is if the checks up until that mount failure have
been read-only and thus I can revert to the older kernel, or if
something would have been written to disk prior to the mount call
failing. I don't want to risk the 23 TiB of snapshot data stored if
it's an easy workaround :-).  I realize there are risks with the
RAID56 code, but I've done my best to mitigate with server-grade
hardware, ECC memory, a UPS and a redundant copy of the data via
btrfs-send to this machine. Losing this snapshot volume is not the end
of the world, but I am about to upgrade the primary server (which is
currently running 4.19.4 without issue btw) and want to have a best
effort snapshot/backup in place before I do so.

> Whether it's a case of 4.18.10 not detecting corruption that 4.19
> sees, or if 4.19 already caused it, the best chance is to not mount it
> rw, and not run check --repair, until you get some feedback from a
> developer.
>

+1

> The thing I'd like to see is
> # btrfs rescue super -v /anydevice/
> # btrfs insp dump-s -f /anydevice/
>
> First command will tell us if all the supers are the same and valid
> across all devices. And the second one, hopefully it's pointed to a
> device with valid super, will tell us if there's a log root value
> other than 0. Both of those are read only commands.
>

It was my understanding that rescue writes to the disk (the man page
indicates it recovers superblocks which would imply writes). If you
are sure they both leave the on-disk data structures completely
intact, I would be willing to run them.

- mike

Re: Ran into "invalid block group size" bug, unclear how to proceed.

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1927 bytes --]



On 2018/12/4 上午11:32, Mike Javorski wrote:
> Need a bit of advice here ladies / gents. I am running into an issue
> which Qu Wenruo seems to have posted a patch for several weeks ago
> (see https://patchwork.kernel.org/patch/10694997/).
> 
> Here is the relevant dmesg output which led me to Qu's patch.
> ----
> [   10.032475] BTRFS critical (device sdb): corrupt leaf: root=2
> block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104,
> invalid block group size, have 10804527104 expect (0, 10737418240]
> [   10.032493] BTRFS error (device sdb): failed to read block groups: -5
> [   10.053365] BTRFS error (device sdb): open_ctree failed
> ----

Exactly the same symptom.

> 
> This server has a 16 disk btrfs filesystem (RAID6) which I boot
> periodically to btrfs-send snapshots to. This machine is running
> ArchLinux and I had just updated  to their latest 4.19.4 kernel
> package (from 4.18.10 which was working fine). I've tried updating to
> the 4.19.6 kernel that is in testing, but that doesn't seem to resolve
> the issue. From what I can see on kernel.org, the patch above is not
> pushed to stable or to Linus' tree.
> 
> At this point the question is what to do. Is my FS toast?

If there is no other problem at all, your fs is just fine.
It's my original patch too sensitive (the excuse for not checking chunk
allocator carefully enough).

But since you have the down time, it's never a bad idea to run a btrfs
check --readonly to see if your fs is really OK.

> Could I
> revert to the 4.18.10 kernel and boot safely?

If your btrfs check --readonly doesn't report any problem, then you're
completely fine to do so.
Although I still recommend to go RAID10 other than RAID5/6.

Thanks,
Qu

> I don't know if the 4.19
> boot process may have flipped some bits which would make reverting
> problematic.
> 
> Thanks much,
> 
> - mike
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Ran into "invalid block group size" bug, unclear how to proceed.

[Mike Javorski]

On Tue, Dec 4, 2018 at 2:18 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2018/12/4 上午11:32, Mike Javorski wrote:
> > Need a bit of advice here ladies / gents. I am running into an issue
> > which Qu Wenruo seems to have posted a patch for several weeks ago
> > (see https://patchwork.kernel.org/patch/10694997/).
> >
> > Here is the relevant dmesg output which led me to Qu's patch.
> > ----
> > [   10.032475] BTRFS critical (device sdb): corrupt leaf: root=2
> > block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104,
> > invalid block group size, have 10804527104 expect (0, 10737418240]
> > [   10.032493] BTRFS error (device sdb): failed to read block groups: -5
> > [   10.053365] BTRFS error (device sdb): open_ctree failed
> > ----
>
> Exactly the same symptom.
>
> >
> > This server has a 16 disk btrfs filesystem (RAID6) which I boot
> > periodically to btrfs-send snapshots to. This machine is running
> > ArchLinux and I had just updated  to their latest 4.19.4 kernel
> > package (from 4.18.10 which was working fine). I've tried updating to
> > the 4.19.6 kernel that is in testing, but that doesn't seem to resolve
> > the issue. From what I can see on kernel.org, the patch above is not
> > pushed to stable or to Linus' tree.
> >
> > At this point the question is what to do. Is my FS toast?
>
> If there is no other problem at all, your fs is just fine.
> It's my original patch too sensitive (the excuse for not checking chunk
> allocator carefully enough).
>
> But since you have the down time, it's never a bad idea to run a btrfs
> check --readonly to see if your fs is really OK.
>

After running for 4 hours...

UUID: 25b16375-b90b-408e-b592-fb07ed116d58
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups
found 24939616169984 bytes used, no error found
total csum bytes: 24321980768
total tree bytes: 41129721856
total fs tree bytes: 9854648320
total extent tree bytes: 737804288
btree space waste bytes: 7483785005
file data blocks allocated: 212883520618496
 referenced 212876546314240

So things appear good to go. I will keep an eye out for the patch to
land before upgrading the kernel again.

> > Could I
> > revert to the 4.18.10 kernel and boot safely?
>
> If your btrfs check --readonly doesn't report any problem, then you're
> completely fine to do so.
> Although I still recommend to go RAID10 other than RAID5/6.

I understand the risk, but don't have the funds to buy sufficient
disks to operate in RAID10. The data is mostly large files and
activity is predominantly reads, so risk is currently acceptable given
the backup server. All super critical data is backed up to (very slow)
cloud storage.

>
> Thanks,
> Qu
>
> > I don't know if the 4.19
> > boot process may have flipped some bits which would make reverting
> > problematic.
> >
> > Thanks much,
> >
> > - mike
> >
>

Re: Ran into "invalid block group size" bug, unclear how to proceed.

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3503 bytes --]



On 2018/12/5 上午6:33, Mike Javorski wrote:
> On Tue, Dec 4, 2018 at 2:18 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2018/12/4 上午11:32, Mike Javorski wrote:
>>> Need a bit of advice here ladies / gents. I am running into an issue
>>> which Qu Wenruo seems to have posted a patch for several weeks ago
>>> (see https://patchwork.kernel.org/patch/10694997/).
>>>
>>> Here is the relevant dmesg output which led me to Qu's patch.
>>> ----
>>> [   10.032475] BTRFS critical (device sdb): corrupt leaf: root=2
>>> block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104,
>>> invalid block group size, have 10804527104 expect (0, 10737418240]
>>> [   10.032493] BTRFS error (device sdb): failed to read block groups: -5
>>> [   10.053365] BTRFS error (device sdb): open_ctree failed
>>> ----
>>
>> Exactly the same symptom.
>>
>>>
>>> This server has a 16 disk btrfs filesystem (RAID6) which I boot
>>> periodically to btrfs-send snapshots to. This machine is running
>>> ArchLinux and I had just updated  to their latest 4.19.4 kernel
>>> package (from 4.18.10 which was working fine). I've tried updating to
>>> the 4.19.6 kernel that is in testing, but that doesn't seem to resolve
>>> the issue. From what I can see on kernel.org, the patch above is not
>>> pushed to stable or to Linus' tree.
>>>
>>> At this point the question is what to do. Is my FS toast?
>>
>> If there is no other problem at all, your fs is just fine.
>> It's my original patch too sensitive (the excuse for not checking chunk
>> allocator carefully enough).
>>
>> But since you have the down time, it's never a bad idea to run a btrfs
>> check --readonly to see if your fs is really OK.
>>
> 
> After running for 4 hours...
> 
> UUID: 25b16375-b90b-408e-b592-fb07ed116d58
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups
> found 24939616169984 bytes used, no error found
> total csum bytes: 24321980768
> total tree bytes: 41129721856
> total fs tree bytes: 9854648320
> total extent tree bytes: 737804288
> btree space waste bytes: 7483785005
> file data blocks allocated: 212883520618496
>  referenced 212876546314240
> 
> So things appear good to go. I will keep an eye out for the patch to
> land before upgrading the kernel again.
> 
>>> Could I
>>> revert to the 4.18.10 kernel and boot safely?
>>
>> If your btrfs check --readonly doesn't report any problem, then you're
>> completely fine to do so.
>> Although I still recommend to go RAID10 other than RAID5/6.
> 
> I understand the risk, but don't have the funds to buy sufficient
> disks to operate in RAID10.

Then my advice would be, for any powerloss, please run a full-disk scrub
(and of course ensure there is not another powerloss during scrubbing).

I know this sounds silly and slow, but at least it should workaround the
write hole problem.

Thanks,
Qu

> The data is mostly large files and
> activity is predominantly reads, so risk is currently acceptable given
> the backup server. All super critical data is backed up to (very slow)
> cloud storage.
> 
>>
>> Thanks,
>> Qu
>>
>>> I don't know if the 4.19
>>> boot process may have flipped some bits which would make reverting
>>> problematic.
>>>
>>> Thanks much,
>>>
>>> - mike
>>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Ran into "invalid block group size" bug, unclear how to proceed.

[Mike Javorski]

Will do, thanks!

- mike
On Tue, Dec 4, 2018 at 9:24 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2018/12/5 上午6:33, Mike Javorski wrote:
> > On Tue, Dec 4, 2018 at 2:18 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2018/12/4 上午11:32, Mike Javorski wrote:
> >>> Need a bit of advice here ladies / gents. I am running into an issue
> >>> which Qu Wenruo seems to have posted a patch for several weeks ago
> >>> (see https://patchwork.kernel.org/patch/10694997/).
> >>>
> >>> Here is the relevant dmesg output which led me to Qu's patch.
> >>> ----
> >>> [   10.032475] BTRFS critical (device sdb): corrupt leaf: root=2
> >>> block=24655027060736 slot=20 bg_start=13188988928 bg_len=10804527104,
> >>> invalid block group size, have 10804527104 expect (0, 10737418240]
> >>> [   10.032493] BTRFS error (device sdb): failed to read block groups: -5
> >>> [   10.053365] BTRFS error (device sdb): open_ctree failed
> >>> ----
> >>
> >> Exactly the same symptom.
> >>
> >>>
> >>> This server has a 16 disk btrfs filesystem (RAID6) which I boot
> >>> periodically to btrfs-send snapshots to. This machine is running
> >>> ArchLinux and I had just updated  to their latest 4.19.4 kernel
> >>> package (from 4.18.10 which was working fine). I've tried updating to
> >>> the 4.19.6 kernel that is in testing, but that doesn't seem to resolve
> >>> the issue. From what I can see on kernel.org, the patch above is not
> >>> pushed to stable or to Linus' tree.
> >>>
> >>> At this point the question is what to do. Is my FS toast?
> >>
> >> If there is no other problem at all, your fs is just fine.
> >> It's my original patch too sensitive (the excuse for not checking chunk
> >> allocator carefully enough).
> >>
> >> But since you have the down time, it's never a bad idea to run a btrfs
> >> check --readonly to see if your fs is really OK.
> >>
> >
> > After running for 4 hours...
> >
> > UUID: 25b16375-b90b-408e-b592-fb07ed116d58
> > [1/7] checking root items
> > [2/7] checking extents
> > [3/7] checking free space cache
> > [4/7] checking fs roots
> > [5/7] checking only csums items (without verifying data)
> > [6/7] checking root refs
> > [7/7] checking quota groups
> > found 24939616169984 bytes used, no error found
> > total csum bytes: 24321980768
> > total tree bytes: 41129721856
> > total fs tree bytes: 9854648320
> > total extent tree bytes: 737804288
> > btree space waste bytes: 7483785005
> > file data blocks allocated: 212883520618496
> >  referenced 212876546314240
> >
> > So things appear good to go. I will keep an eye out for the patch to
> > land before upgrading the kernel again.
> >
> >>> Could I
> >>> revert to the 4.18.10 kernel and boot safely?
> >>
> >> If your btrfs check --readonly doesn't report any problem, then you're
> >> completely fine to do so.
> >> Although I still recommend to go RAID10 other than RAID5/6.
> >
> > I understand the risk, but don't have the funds to buy sufficient
> > disks to operate in RAID10.
>
> Then my advice would be, for any powerloss, please run a full-disk scrub
> (and of course ensure there is not another powerloss during scrubbing).
>
> I know this sounds silly and slow, but at least it should workaround the
> write hole problem.
>
> Thanks,
> Qu
>
> > The data is mostly large files and
> > activity is predominantly reads, so risk is currently acceptable given
> > the backup server. All super critical data is backed up to (very slow)
> > cloud storage.
> >
> >>
> >> Thanks,
> >> Qu
> >>
> >>> I don't know if the 4.19
> >>> boot process may have flipped some bits which would make reverting
> >>> problematic.
> >>>
> >>> Thanks much,
> >>>
> >>> - mike
> >>>
> >>
>