Root FS damaged

Root FS damaged

[Robert Klemme]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

Hi folks,

I have an issue with one of my desktop systems. Besides the usual
information below I have attached output of btrfsck and dmesg. The
system did not crash but was up for about a week.

My questions:
1. And ideas what is wrong?
2. Should I file a bug
3. can I safely repair with --repair or what else do I have to do to repair?

Thank you!

Kind regards

robert

This is a Xubuntu and I am using btrfs on top of lvm on top of LUKS.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic
$ uname -a
Linux robunt-01 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28
UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ btrfs --version
btrfs-progs v4.15.1
$ sudo btrfs fi show
Label: none  uuid: 0da6c6f7-d42e-4096-8690-97daf14d70e7
        Total devices 1 FS bytes used 12.64GiB
        devid    1 size 30.00GiB used 15.54GiB path
/dev/mapper/main--vg-main--root

Label: 'home'  uuid: cfb8c776-0dab-4596-af5b-276f0db46f79
        Total devices 1 FS bytes used 50.73GiB
        devid    1 size 161.57GiB used 53.07GiB path
/dev/mapper/main--vg-main--home

$ sudo btrfs fi df /
Data, single: total=14.01GiB, used=11.83GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.50GiB, used=820.30MiB
GlobalReserve, single: total=39.19MiB, used=0.00B

[-- Attachment #2: btrfscheck-errors.txt --]
[-- Type: text/plain, Size: 1013 bytes --]

$ sudo btrfsck /dev/mapper/main--vg-main--root 
Checking filesystem on /dev/mapper/main--vg-main--root
UUID: 0da6c6f7-d42e-4096-8690-97daf14d70e7
checking extents
data backref 5658767360 root 275 owner 19075 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 5658767360 root 275 owner 19075 offset 0 found 1 wanted 0 back 0x562deceb1250
incorrect local backref count on 5658767360 root 275 owner 19075 offset 131072 found 0 wanted 1 back 0x562deae7d0d0
backref disk bytenr does not match extent record, bytenr=5658767360, ref bytenr=0
backpointer mismatch on [5658767360 12288]
ERROR: errors found in extent allocation tree or chunk allocation
checking free space cache
checking fs roots
checking csums
checking root refs
found 13525364736 bytes used, error(s) found
total csum bytes: 10847980
total tree bytes: 825819136
total fs tree bytes: 784613376
total extent tree bytes: 26755072
btree space waste bytes: 166697454
file data blocks allocated: 40856240128
 referenced 34213076992

[-- Attachment #3: dmesg.txt.gz --]
[-- Type: application/x-gzip, Size: 23662 bytes --]

Re: Root FS damaged

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2441 bytes --]



On 2020/2/3 下午9:33, Robert Klemme wrote:
> Hi folks,
> 
> I have an issue with one of my desktop systems. Besides the usual
> information below I have attached output of btrfsck and dmesg. The
> system did not crash but was up for about a week.
> 
> My questions:
> 1. And ideas what is wrong?

One data extent lost its backref in extent tree.
So btrfs is unable to delete it, and will fallback to RO, to avoid
further corruption.

I have no idea how this happened, but I'm pretty confident it's caused
by btrfs itself, not some hardware nor disk problems.

Any history about the fs? It may be caused by some older btrfs bug.

> 2. Should I file a bug

If you have an idea how to reproduce such problem.
Or we can only help you to fix the fs, not really to locate the cause.

> 3. can I safely repair with --repair or what else do I have to do to repair?

Btrfs check --repair should be able to repair that, but not recommended
for your btrfs-progs version.

There is a bug that any power loss or transaction abort in btrfs-progs
can further screw up your fs.
That bug is solved in v5.1 btrfs-progs.
I doubt it's backported for any btrfs-progs at all.

So please use latest btrfs-progs to fix it.
A liveiso from some rolling distro would help.

Thanks,
Qu

> 
> Thank you!
> 
> Kind regards
> 
> robert
> 
> This is a Xubuntu and I am using btrfs on top of lvm on top of LUKS.
> 
> $ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description:    Ubuntu 18.04.3 LTS
> Release:        18.04
> Codename:       bionic
> $ uname -a
> Linux robunt-01 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28
> UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> $ btrfs --version
> btrfs-progs v4.15.1
> $ sudo btrfs fi show
> Label: none  uuid: 0da6c6f7-d42e-4096-8690-97daf14d70e7
>         Total devices 1 FS bytes used 12.64GiB
>         devid    1 size 30.00GiB used 15.54GiB path
> /dev/mapper/main--vg-main--root
> 
> Label: 'home'  uuid: cfb8c776-0dab-4596-af5b-276f0db46f79
>         Total devices 1 FS bytes used 50.73GiB
>         devid    1 size 161.57GiB used 53.07GiB path
> /dev/mapper/main--vg-main--home
> 
> $ sudo btrfs fi df /
> Data, single: total=14.01GiB, used=11.83GiB
> System, single: total=32.00MiB, used=16.00KiB
> Metadata, single: total=1.50GiB, used=820.30MiB
> GlobalReserve, single: total=39.19MiB, used=0.00B
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Root FS damaged

[Robert Klemme]

Hey,

thank you! That was quick! Some comments inline below.

On Mon, Feb 3, 2020 at 2:44 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
> On 2020/2/3 下午9:33, Robert Klemme wrote:

> > I have an issue with one of my desktop systems. Besides the usual
> > information below I have attached output of btrfsck and dmesg. The
> > system did not crash but was up for about a week.
> >
> > My questions:
> > 1. And ideas what is wrong?
>
> One data extent lost its backref in extent tree.
> So btrfs is unable to delete it, and will fallback to RO, to avoid
> further corruption.
>
> I have no idea how this happened, but I'm pretty confident it's caused
> by btrfs itself, not some hardware nor disk problems.

I would assume as much as there were no power outages or crashes. I
read about a bug recently (probably on
https://www.reddit.com/r/btrfs/) that had to do with btrfs on LUKS and
/ or LVM. Could this be an explanation?

> Any history about the fs? It may be caused by some older btrfs bug.
>
> > 2. Should I file a bug
>
> If you have an idea how to reproduce such problem.

Not at the moment as I did not observe any unusual circumstances.
Having the system up and running for a while is probably not a useful
test. :-)

> Or we can only help you to fix the fs, not really to locate the cause.

OK, let's take that route.

> > 3. can I safely repair with --repair or what else do I have to do to repair?
>
> Btrfs check --repair should be able to repair that, but not recommended
> for your btrfs-progs version.
>
> There is a bug that any power loss or transaction abort in btrfs-progs
> can further screw up your fs.

That explains why a repair I recently attempted elsewhere did make
things worse...

> That bug is solved in v5.1 btrfs-progs.
> I doubt it's backported for any btrfs-progs at all.
>
> So please use latest btrfs-progs to fix it.
> A liveiso from some rolling distro would help.

Is there a PPA? I could not find one so far.

Thank you!

robert

>
> Thanks,
> Qu
>
> >
> > Thank you!
> >
> > Kind regards
> >
> > robert
> >
> > This is a Xubuntu and I am using btrfs on top of lvm on top of LUKS.
> >
> > $ lsb_release -a
> > No LSB modules are available.
> > Distributor ID: Ubuntu
> > Description:    Ubuntu 18.04.3 LTS
> > Release:        18.04
> > Codename:       bionic
> > $ uname -a
> > Linux robunt-01 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28
> > UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> > $ btrfs --version
> > btrfs-progs v4.15.1
> > $ sudo btrfs fi show
> > Label: none  uuid: 0da6c6f7-d42e-4096-8690-97daf14d70e7
> >         Total devices 1 FS bytes used 12.64GiB
> >         devid    1 size 30.00GiB used 15.54GiB path
> > /dev/mapper/main--vg-main--root
> >
> > Label: 'home'  uuid: cfb8c776-0dab-4596-af5b-276f0db46f79
> >         Total devices 1 FS bytes used 50.73GiB
> >         devid    1 size 161.57GiB used 53.07GiB path
> > /dev/mapper/main--vg-main--home
> >
> > $ sudo btrfs fi df /
> > Data, single: total=14.01GiB, used=11.83GiB
> > System, single: total=32.00MiB, used=16.00KiB
> > Metadata, single: total=1.50GiB, used=820.30MiB
> > GlobalReserve, single: total=39.19MiB, used=0.00B
> >
>


-- 
[guy, jim, charlie, sho].each {|him| remember.him do |as, often|
as.you_can - without end}
http://blog.rubybestpractices.com/

Re: Root FS damaged

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3633 bytes --]



On 2020/2/3 下午9:58, Robert Klemme wrote:
> Hey,
> 
> thank you! That was quick! Some comments inline below.
> 
> On Mon, Feb 3, 2020 at 2:44 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>> On 2020/2/3 下午9:33, Robert Klemme wrote:
> 
>>> I have an issue with one of my desktop systems. Besides the usual
>>> information below I have attached output of btrfsck and dmesg. The
>>> system did not crash but was up for about a week.
>>>
>>> My questions:
>>> 1. And ideas what is wrong?
>>
>> One data extent lost its backref in extent tree.
>> So btrfs is unable to delete it, and will fallback to RO, to avoid
>> further corruption.
>>
>> I have no idea how this happened, but I'm pretty confident it's caused
>> by btrfs itself, not some hardware nor disk problems.
> 
> I would assume as much as there were no power outages or crashes. I
> read about a bug recently (probably on
> https://www.reddit.com/r/btrfs/) that had to do with btrfs on LUKS and
> / or LVM. Could this be an explanation?

No, it should only be some bug inside btrfs, nothing to do with lower stack.

> 
>> Any history about the fs? It may be caused by some older btrfs bug.
>>
>>> 2. Should I file a bug
>>
>> If you have an idea how to reproduce such problem.
> 
> Not at the moment as I did not observe any unusual circumstances.
> Having the system up and running for a while is probably not a useful
> test. :-)
> 
>> Or we can only help you to fix the fs, not really to locate the cause.
> 
> OK, let's take that route.
> 
>>> 3. can I safely repair with --repair or what else do I have to do to repair?
>>
>> Btrfs check --repair should be able to repair that, but not recommended
>> for your btrfs-progs version.
>>
>> There is a bug that any power loss or transaction abort in btrfs-progs
>> can further screw up your fs.
> 
> That explains why a repair I recently attempted elsewhere did make
> things worse...
> 
>> That bug is solved in v5.1 btrfs-progs.
>> I doubt it's backported for any btrfs-progs at all.
>>
>> So please use latest btrfs-progs to fix it.
>> A liveiso from some rolling distro would help.
> 
> Is there a PPA? I could not find one so far.

For "some rolling distro", I mean Arch...

Since you're just going to repair the fs, no need to stick to Ubuntu.

Thanks,
Qu

> 
> Thank you!
> 
> robert
> 
>>
>> Thanks,
>> Qu
>>
>>>
>>> Thank you!
>>>
>>> Kind regards
>>>
>>> robert
>>>
>>> This is a Xubuntu and I am using btrfs on top of lvm on top of LUKS.
>>>
>>> $ lsb_release -a
>>> No LSB modules are available.
>>> Distributor ID: Ubuntu
>>> Description:    Ubuntu 18.04.3 LTS
>>> Release:        18.04
>>> Codename:       bionic
>>> $ uname -a
>>> Linux robunt-01 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28
>>> UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>>> $ btrfs --version
>>> btrfs-progs v4.15.1
>>> $ sudo btrfs fi show
>>> Label: none  uuid: 0da6c6f7-d42e-4096-8690-97daf14d70e7
>>>         Total devices 1 FS bytes used 12.64GiB
>>>         devid    1 size 30.00GiB used 15.54GiB path
>>> /dev/mapper/main--vg-main--root
>>>
>>> Label: 'home'  uuid: cfb8c776-0dab-4596-af5b-276f0db46f79
>>>         Total devices 1 FS bytes used 50.73GiB
>>>         devid    1 size 161.57GiB used 53.07GiB path
>>> /dev/mapper/main--vg-main--home
>>>
>>> $ sudo btrfs fi df /
>>> Data, single: total=14.01GiB, used=11.83GiB
>>> System, single: total=32.00MiB, used=16.00KiB
>>> Metadata, single: total=1.50GiB, used=820.30MiB
>>> GlobalReserve, single: total=39.19MiB, used=0.00B
>>>
>>
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Root FS damaged

[Remi Gauvin]

[-- Attachment #1.1: Type: text/plain, Size: 478 bytes --]

On 2020-02-03 8:58 a.m., Robert Klemme wrote:

>> If you have an idea how to reproduce such problem.
> 
> Not at the moment as I did not observe any unusual circumstances.
> Having the system up and running for a while is probably not a useful
> test. :-)
> 

Have you recently converted to space_cache=v2 or otherwise tried to
clear / manipulate space cache?

On the 2 systems I've seen this error so far, it was shortly after
converting space_cache to v2.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Root FS damaged

[Robert Klemme]

Hi!

On Mon, Feb 3, 2020 at 3:12 PM Remi Gauvin <remi@georgianit.com> wrote:
>
> On 2020-02-03 8:58 a.m., Robert Klemme wrote:
>
> >> If you have an idea how to reproduce such problem.
> >
> > Not at the moment as I did not observe any unusual circumstances.
> > Having the system up and running for a while is probably not a useful
> > test. :-)
> >
>
> Have you recently converted to space_cache=v2 or otherwise tried to
> clear / manipulate space cache?

No. Unless some package update would have done this. The system was
set up in 2016 and I did no FS tweaking.

> On the 2 systems I've seen this error so far, it was shortly after
> converting space_cache to v2.

Kind regards

robert

-- 
[guy, jim, charlie, sho].each {|him| remember.him do |as, often|
as.you_can - without end}
http://blog.rubybestpractices.com/

Re: Root FS damaged

[Robert Klemme]

On Mon, Feb 3, 2020 at 3:10 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> On 2020/2/3 下午9:58, Robert Klemme wrote:

> > I would assume as much as there were no power outages or crashes. I
> > read about a bug recently (probably on
> > https://www.reddit.com/r/btrfs/) that had to do with btrfs on LUKS and
> > / or LVM. Could this be an explanation?
>
> No, it should only be some bug inside btrfs, nothing to do with lower stack.

Unfortunately I have not found the source again. IIRC that was about a
bug in btrfs code but showed only in combination with either LUKS and
/ or LVM.

> >> Btrfs check --repair should be able to repair that, but not recommended
> >> for your btrfs-progs version.
> >>
> >> There is a bug that any power loss or transaction abort in btrfs-progs
> >> can further screw up your fs.
> >
> > That explains why a repair I recently attempted elsewhere did make
> > things worse...
> >
> >> That bug is solved in v5.1 btrfs-progs.
> >> I doubt it's backported for any btrfs-progs at all.
> >>
> >> So please use latest btrfs-progs to fix it.
> >> A liveiso from some rolling distro would help.
> >
> > Is there a PPA? I could not find one so far.
>
> For "some rolling distro", I mean Arch...

:-) Understood, but I would have liked to also install the newer tool
version on that system. Anyway, I pulled a Xubuntu 19.10 and that has
more recent btrfs tools. Repair went smoothly and system is up and
running. As additional measure I installed Ubuntu's HWE packages to
get a newer kernel (from 4.15 to 5.3.0).

$ sudo apt install linux-generic-hwe-18.04 xserver-xorg-hwe-18.04

Just wanted to give you an update how it went. Again, thank you for
your support!

Kind regards

robert

-- 
[guy, jim, charlie, sho].each {|him| remember.him do |as, often|
as.you_can - without end}
http://blog.rubybestpractices.com/