Next steps in recovery?

Next steps in recovery?

[Robert Wyrick]

I cannot mount my btrfs filesystem.
$ uname -a
Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ btrfs version
btrfs-progs v5.4.1

I'm seeing the following from check:
$ btrfs check -p /dev/sda
Opening filesystem to check...
Checking filesystem on /dev/sda
UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
[1/7] checking root items                      (0:00:59 elapsed,
2649102 items checked)
ERROR: invalid generation for extent 38179182174208, have
140737491486755 expect (0, 4057084]
[2/7] checking extents                         (0:02:17 elapsed,
1116143 items checked)
ERROR: errors found in extent allocation tree or chunk allocation
cache and super generation don't match, space cache will be invalidated
[3/7] checking free space cache                (0:00:00 elapsed)
[4/7] chunresolved ref dir 8348950 index 3 namelen 7 name posters
filetype 2 errors 2, no dir index
unresolved ref dir 8348950 index 3 namelen 7 name poSters filetype 2
errors 5, no dir item, no inode ref
[4/7] checking fs roots                        (0:00:42 elapsed,
108894 items checked)
ERROR: errors found in fs roots
found 15729059057664 bytes used, error(s) found
total csum bytes: 15313288548
total tree bytes: 18286739456
total fs tree bytes: 1791819776
total extent tree bytes: 229130240
btree space waste bytes: 1018844959
file data blocks allocated: 51587230502912
 referenced 15627926712320

I've tried everything I've found on the internet, but haven't
attempted to repair based on the warnings...

What more info do you need to help me diagnose/fix this?

Thanks!
-Rob

Re: Next steps in recovery?

[Robert Wyrick]

One more thing of note.  I was running linux kernel 4.15.? when things
went bad.  I upgraded hoping that the newer tools could fix things.

On Thu, Sep 2, 2021 at 8:43 PM Robert Wyrick <rob@wyrick.org> wrote:
>
> I cannot mount my btrfs filesystem.
> $ uname -a
> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> $ btrfs version
> btrfs-progs v5.4.1
>
> I'm seeing the following from check:
> $ btrfs check -p /dev/sda
> Opening filesystem to check...
> Checking filesystem on /dev/sda
> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
> [1/7] checking root items                      (0:00:59 elapsed,
> 2649102 items checked)
> ERROR: invalid generation for extent 38179182174208, have
> 140737491486755 expect (0, 4057084]
> [2/7] checking extents                         (0:02:17 elapsed,
> 1116143 items checked)
> ERROR: errors found in extent allocation tree or chunk allocation
> cache and super generation don't match, space cache will be invalidated
> [3/7] checking free space cache                (0:00:00 elapsed)
> [4/7] chunresolved ref dir 8348950 index 3 namelen 7 name posters
> filetype 2 errors 2, no dir index
> unresolved ref dir 8348950 index 3 namelen 7 name poSters filetype 2
> errors 5, no dir item, no inode ref
> [4/7] checking fs roots                        (0:00:42 elapsed,
> 108894 items checked)
> ERROR: errors found in fs roots
> found 15729059057664 bytes used, error(s) found
> total csum bytes: 15313288548
> total tree bytes: 18286739456
> total fs tree bytes: 1791819776
> total extent tree bytes: 229130240
> btree space waste bytes: 1018844959
> file data blocks allocated: 51587230502912
>  referenced 15627926712320
>
> I've tried everything I've found on the internet, but haven't
> attempted to repair based on the warnings...
>
> What more info do you need to help me diagnose/fix this?
>
> Thanks!
> -Rob

Re: Next steps in recovery?

[Qu Wenruo]

On 2021/9/3 上午10:43, Robert Wyrick wrote:
> I cannot mount my btrfs filesystem.
> $ uname -a
> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> $ btrfs version
> btrfs-progs v5.4.1

The tool is a little too old, thus if you're going to repair, you'd
better to update the progs.
>
> I'm seeing the following from check:
> $ btrfs check -p /dev/sda
> Opening filesystem to check...
> Checking filesystem on /dev/sda
> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
> [1/7] checking root items                      (0:00:59 elapsed,
> 2649102 items checked)
> ERROR: invalid generation for extent 38179182174208, have
> 140737491486755 expect (0, 4057084]

This is a repairable problem.

We have test case for exactly the same case in tests/fsck-test/044 for it.


> [2/7] checking extents                         (0:02:17 elapsed,
> 1116143 items checked)
> ERROR: errors found in extent allocation tree or chunk allocation
> cache and super generation don't match, space cache will be invalidated
> [3/7] checking free space cache                (0:00:00 elapsed)
> [4/7] chunresolved ref dir 8348950 index 3 namelen 7 name posters
> filetype 2 errors 2, no dir index

No dir index can also be repaired.

The dir index will be added back.

> unresolved ref dir 8348950 index 3 namelen 7 name poSters filetype 2
> errors 5, no dir item, no inode ref

No dir item nor inode ref can also be repaired, but with dir item and
inode ref removed.

But the problem here looks very strange.

It's the same dir and the same index, but different name.
posters vs poSters.

'S' is 0x53 and 's' is 0x73, I'm wondering if your system had a bad
memory which caused a bitflip and the problem.

Thus I prefer to do a full memtest before running btrfs check --repair.

Thanks,
Qu

> [4/7] checking fs roots                        (0:00:42 elapsed,
> 108894 items checked)
> ERROR: errors found in fs roots
> found 15729059057664 bytes used, error(s) found
> total csum bytes: 15313288548
> total tree bytes: 18286739456
> total fs tree bytes: 1791819776
> total extent tree bytes: 229130240
> btree space waste bytes: 1018844959
> file data blocks allocated: 51587230502912
>   referenced 15627926712320
>
> I've tried everything I've found on the internet, but haven't
> attempted to repair based on the warnings...
>
> What more info do you need to help me diagnose/fix this?
>
> Thanks!
> -Rob
>

Re: Next steps in recovery?

[Qu Wenruo]

On 2021/9/3 下午2:48, Qu Wenruo wrote:
>
>
> On 2021/9/3 上午10:43, Robert Wyrick wrote:
>> I cannot mount my btrfs filesystem.
>> $ uname -a
>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>> $ btrfs version
>> btrfs-progs v5.4.1
>
> The tool is a little too old, thus if you're going to repair, you'd
> better to update the progs.
>>
>> I'm seeing the following from check:
>> $ btrfs check -p /dev/sda
>> Opening filesystem to check...
>> Checking filesystem on /dev/sda
>> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
>> [1/7] checking root items                      (0:00:59 elapsed,
>> 2649102 items checked)
>> ERROR: invalid generation for extent 38179182174208, have
>> 140737491486755 expect (0, 4057084]
>
> This is a repairable problem.
>
> We have test case for exactly the same case in tests/fsck-test/044 for it.

Oh, this invalid extent generation is already a more direct indication
of memory bitflip.

140737491486755 = 0x8000002fc823

Without the high 0x8 bit, the remaining part is completely valid
generation, 0x2fc823, which is inside the expectation.

So, a memtest is a must before doing any repair.
You won't want another bitflip to ruin your perfectly repairable fs.

Thanks,
Qu
>
>
>> [2/7] checking extents                         (0:02:17 elapsed,
>> 1116143 items checked)
>> ERROR: errors found in extent allocation tree or chunk allocation
>> cache and super generation don't match, space cache will be invalidated
>> [3/7] checking free space cache                (0:00:00 elapsed)
>> [4/7] chunresolved ref dir 8348950 index 3 namelen 7 name posters
>> filetype 2 errors 2, no dir index
>
> No dir index can also be repaired.
>
> The dir index will be added back.
>
>> unresolved ref dir 8348950 index 3 namelen 7 name poSters filetype 2
>> errors 5, no dir item, no inode ref
>
> No dir item nor inode ref can also be repaired, but with dir item and
> inode ref removed.
>
> But the problem here looks very strange.
>
> It's the same dir and the same index, but different name.
> posters vs poSters.
>
> 'S' is 0x53 and 's' is 0x73, I'm wondering if your system had a bad
> memory which caused a bitflip and the problem.
>
> Thus I prefer to do a full memtest before running btrfs check --repair.
>
> Thanks,
> Qu
>
>> [4/7] checking fs roots                        (0:00:42 elapsed,
>> 108894 items checked)
>> ERROR: errors found in fs roots
>> found 15729059057664 bytes used, error(s) found
>> total csum bytes: 15313288548
>> total tree bytes: 18286739456
>> total fs tree bytes: 1791819776
>> total extent tree bytes: 229130240
>> btree space waste bytes: 1018844959
>> file data blocks allocated: 51587230502912
>>   referenced 15627926712320
>>
>> I've tried everything I've found on the internet, but haven't
>> attempted to repair based on the warnings...
>>
>> What more info do you need to help me diagnose/fix this?
>>
>> Thanks!
>> -Rob
>>

Re: Next steps in recovery?

[Qu Wenruo]

On 2021/9/6 上午12:00, Robert Wyrick wrote:
> Running memtest86+ now....  20 hours in.  No errors yet.
> Thanks for the analysis.  I'll let this run for another day or so.

Just to mention, since 5.11 btrfs kernel module has the ability to
detect most high bitflip before writing tree blocks to disks.

Thus even with less reliable RAM, it's still more reliable than nothing.

But still, with the existing errors, the RAM test is still an essential
one before doing anything.

Thanks,
Qu
>
>
> On Fri, Sep 3, 2021 at 12:53 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2021/9/3 下午2:48, Qu Wenruo wrote:
>>>
>>>
>>> On 2021/9/3 上午10:43, Robert Wyrick wrote:
>>>> I cannot mount my btrfs filesystem.
>>>> $ uname -a
>>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
>>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>>> $ btrfs version
>>>> btrfs-progs v5.4.1
>>>
>>> The tool is a little too old, thus if you're going to repair, you'd
>>> better to update the progs.
>>>>
>>>> I'm seeing the following from check:
>>>> $ btrfs check -p /dev/sda
>>>> Opening filesystem to check...
>>>> Checking filesystem on /dev/sda
>>>> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
>>>> [1/7] checking root items                      (0:00:59 elapsed,
>>>> 2649102 items checked)
>>>> ERROR: invalid generation for extent 38179182174208, have
>>>> 140737491486755 expect (0, 4057084]
>>>
>>> This is a repairable problem.
>>>
>>> We have test case for exactly the same case in tests/fsck-test/044 for it.
>>
>> Oh, this invalid extent generation is already a more direct indication
>> of memory bitflip.
>>
>> 140737491486755 = 0x8000002fc823
>>
>> Without the high 0x8 bit, the remaining part is completely valid
>> generation, 0x2fc823, which is inside the expectation.
>>
>> So, a memtest is a must before doing any repair.
>> You won't want another bitflip to ruin your perfectly repairable fs.
>>
>> Thanks,
>> Qu
>>>
>>>
>>>> [2/7] checking extents                         (0:02:17 elapsed,
>>>> 1116143 items checked)
>>>> ERROR: errors found in extent allocation tree or chunk allocation
>>>> cache and super generation don't match, space cache will be invalidated
>>>> [3/7] checking free space cache                (0:00:00 elapsed)
>>>> [4/7] chunresolved ref dir 8348950 index 3 namelen 7 name posters
>>>> filetype 2 errors 2, no dir index
>>>
>>> No dir index can also be repaired.
>>>
>>> The dir index will be added back.
>>>
>>>> unresolved ref dir 8348950 index 3 namelen 7 name poSters filetype 2
>>>> errors 5, no dir item, no inode ref
>>>
>>> No dir item nor inode ref can also be repaired, but with dir item and
>>> inode ref removed.
>>>
>>> But the problem here looks very strange.
>>>
>>> It's the same dir and the same index, but different name.
>>> posters vs poSters.
>>>
>>> 'S' is 0x53 and 's' is 0x73, I'm wondering if your system had a bad
>>> memory which caused a bitflip and the problem.
>>>
>>> Thus I prefer to do a full memtest before running btrfs check --repair.
>>>
>>> Thanks,
>>> Qu
>>>
>>>> [4/7] checking fs roots                        (0:00:42 elapsed,
>>>> 108894 items checked)
>>>> ERROR: errors found in fs roots
>>>> found 15729059057664 bytes used, error(s) found
>>>> total csum bytes: 15313288548
>>>> total tree bytes: 18286739456
>>>> total fs tree bytes: 1791819776
>>>> total extent tree bytes: 229130240
>>>> btree space waste bytes: 1018844959
>>>> file data blocks allocated: 51587230502912
>>>>    referenced 15627926712320
>>>>
>>>> I've tried everything I've found on the internet, but haven't
>>>> attempted to repair based on the warnings...
>>>>
>>>> What more info do you need to help me diagnose/fix this?
>>>>
>>>> Thanks!
>>>> -Rob
>>>>

Re: Next steps in recovery?

[Robert Wyrick]

42+ hours of memtest86+, no errors detected.  4 passes complete.
Is that good enough?

On Sun, Sep 5, 2021 at 4:03 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/6 上午12:00, Robert Wyrick wrote:
> > Running memtest86+ now....  20 hours in.  No errors yet.
> > Thanks for the analysis.  I'll let this run for another day or so.
>
> Just to mention, since 5.11 btrfs kernel module has the ability to
> detect most high bitflip before writing tree blocks to disks.
>
> Thus even with less reliable RAM, it's still more reliable than nothing.
>
> But still, with the existing errors, the RAM test is still an essential
> one before doing anything.
>
> Thanks,
> Qu
> >
> >
> > On Fri, Sep 3, 2021 at 12:53 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2021/9/3 下午2:48, Qu Wenruo wrote:
> >>>
> >>>
> >>> On 2021/9/3 上午10:43, Robert Wyrick wrote:
> >>>> I cannot mount my btrfs filesystem.
> >>>> $ uname -a
> >>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> >>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> >>>> $ btrfs version
> >>>> btrfs-progs v5.4.1
> >>>
> >>> The tool is a little too old, thus if you're going to repair, you'd
> >>> better to update the progs.
> >>>>
> >>>> I'm seeing the following from check:
> >>>> $ btrfs check -p /dev/sda
> >>>> Opening filesystem to check...
> >>>> Checking filesystem on /dev/sda
> >>>> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
> >>>> [1/7] checking root items                      (0:00:59 elapsed,
> >>>> 2649102 items checked)
> >>>> ERROR: invalid generation for extent 38179182174208, have
> >>>> 140737491486755 expect (0, 4057084]
> >>>
> >>> This is a repairable problem.
> >>>
> >>> We have test case for exactly the same case in tests/fsck-test/044 for it.
> >>
> >> Oh, this invalid extent generation is already a more direct indication
> >> of memory bitflip.
> >>
> >> 140737491486755 = 0x8000002fc823
> >>
> >> Without the high 0x8 bit, the remaining part is completely valid
> >> generation, 0x2fc823, which is inside the expectation.
> >>
> >> So, a memtest is a must before doing any repair.
> >> You won't want another bitflip to ruin your perfectly repairable fs.
> >>
> >> Thanks,
> >> Qu
> >>>
> >>>
> >>>> [2/7] checking extents                         (0:02:17 elapsed,
> >>>> 1116143 items checked)
> >>>> ERROR: errors found in extent allocation tree or chunk allocation
> >>>> cache and super generation don't match, space cache will be invalidated
> >>>> [3/7] checking free space cache                (0:00:00 elapsed)
> >>>> [4/7] chunresolved ref dir 8348950 index 3 namelen 7 name posters
> >>>> filetype 2 errors 2, no dir index
> >>>
> >>> No dir index can also be repaired.
> >>>
> >>> The dir index will be added back.
> >>>
> >>>> unresolved ref dir 8348950 index 3 namelen 7 name poSters filetype 2
> >>>> errors 5, no dir item, no inode ref
> >>>
> >>> No dir item nor inode ref can also be repaired, but with dir item and
> >>> inode ref removed.
> >>>
> >>> But the problem here looks very strange.
> >>>
> >>> It's the same dir and the same index, but different name.
> >>> posters vs poSters.
> >>>
> >>> 'S' is 0x53 and 's' is 0x73, I'm wondering if your system had a bad
> >>> memory which caused a bitflip and the problem.
> >>>
> >>> Thus I prefer to do a full memtest before running btrfs check --repair.
> >>>
> >>> Thanks,
> >>> Qu
> >>>
> >>>> [4/7] checking fs roots                        (0:00:42 elapsed,
> >>>> 108894 items checked)
> >>>> ERROR: errors found in fs roots
> >>>> found 15729059057664 bytes used, error(s) found
> >>>> total csum bytes: 15313288548
> >>>> total tree bytes: 18286739456
> >>>> total fs tree bytes: 1791819776
> >>>> total extent tree bytes: 229130240
> >>>> btree space waste bytes: 1018844959
> >>>> file data blocks allocated: 51587230502912
> >>>>    referenced 15627926712320
> >>>>
> >>>> I've tried everything I've found on the internet, but haven't
> >>>> attempted to repair based on the warnings...
> >>>>
> >>>> What more info do you need to help me diagnose/fix this?
> >>>>
> >>>> Thanks!
> >>>> -Rob
> >>>>

Re: Next steps in recovery?

[Qu Wenruo]

On 2021/9/6 下午10:42, Robert Wyrick wrote:
> 42+ hours of memtest86+, no errors detected.  4 passes complete.
> Is that good enough?

That's strange, such obvious bitflip should be easily detected.

Is the fs only mounted on that computer?

Anyway, you can continue try to repair with *latest* btrfs-progs.

Thanks,
Qu
>
> On Sun, Sep 5, 2021 at 4:03 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2021/9/6 上午12:00, Robert Wyrick wrote:
>>> Running memtest86+ now....  20 hours in.  No errors yet.
>>> Thanks for the analysis.  I'll let this run for another day or so.
>>
>> Just to mention, since 5.11 btrfs kernel module has the ability to
>> detect most high bitflip before writing tree blocks to disks.
>>
>> Thus even with less reliable RAM, it's still more reliable than nothing.
>>
>> But still, with the existing errors, the RAM test is still an essential
>> one before doing anything.
>>
>> Thanks,
>> Qu
>>>
>>>
>>> On Fri, Sep 3, 2021 at 12:53 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2021/9/3 下午2:48, Qu Wenruo wrote:
>>>>>
>>>>>
>>>>> On 2021/9/3 上午10:43, Robert Wyrick wrote:
>>>>>> I cannot mount my btrfs filesystem.
>>>>>> $ uname -a
>>>>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
>>>>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>>>>> $ btrfs version
>>>>>> btrfs-progs v5.4.1
>>>>>
>>>>> The tool is a little too old, thus if you're going to repair, you'd
>>>>> better to update the progs.
>>>>>>
>>>>>> I'm seeing the following from check:
>>>>>> $ btrfs check -p /dev/sda
>>>>>> Opening filesystem to check...
>>>>>> Checking filesystem on /dev/sda
>>>>>> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
>>>>>> [1/7] checking root items                      (0:00:59 elapsed,
>>>>>> 2649102 items checked)
>>>>>> ERROR: invalid generation for extent 38179182174208, have
>>>>>> 140737491486755 expect (0, 4057084]
>>>>>
>>>>> This is a repairable problem.
>>>>>
>>>>> We have test case for exactly the same case in tests/fsck-test/044 for it.
>>>>
>>>> Oh, this invalid extent generation is already a more direct indication
>>>> of memory bitflip.
>>>>
>>>> 140737491486755 = 0x8000002fc823
>>>>
>>>> Without the high 0x8 bit, the remaining part is completely valid
>>>> generation, 0x2fc823, which is inside the expectation.
>>>>
>>>> So, a memtest is a must before doing any repair.
>>>> You won't want another bitflip to ruin your perfectly repairable fs.
>>>>
>>>> Thanks,
>>>> Qu
>>>>>
>>>>>
>>>>>> [2/7] checking extents                         (0:02:17 elapsed,
>>>>>> 1116143 items checked)
>>>>>> ERROR: errors found in extent allocation tree or chunk allocation
>>>>>> cache and super generation don't match, space cache will be invalidated
>>>>>> [3/7] checking free space cache                (0:00:00 elapsed)
>>>>>> [4/7] chunresolved ref dir 8348950 index 3 namelen 7 name posters
>>>>>> filetype 2 errors 2, no dir index
>>>>>
>>>>> No dir index can also be repaired.
>>>>>
>>>>> The dir index will be added back.
>>>>>
>>>>>> unresolved ref dir 8348950 index 3 namelen 7 name poSters filetype 2
>>>>>> errors 5, no dir item, no inode ref
>>>>>
>>>>> No dir item nor inode ref can also be repaired, but with dir item and
>>>>> inode ref removed.
>>>>>
>>>>> But the problem here looks very strange.
>>>>>
>>>>> It's the same dir and the same index, but different name.
>>>>> posters vs poSters.
>>>>>
>>>>> 'S' is 0x53 and 's' is 0x73, I'm wondering if your system had a bad
>>>>> memory which caused a bitflip and the problem.
>>>>>
>>>>> Thus I prefer to do a full memtest before running btrfs check --repair.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>> [4/7] checking fs roots                        (0:00:42 elapsed,
>>>>>> 108894 items checked)
>>>>>> ERROR: errors found in fs roots
>>>>>> found 15729059057664 bytes used, error(s) found
>>>>>> total csum bytes: 15313288548
>>>>>> total tree bytes: 18286739456
>>>>>> total fs tree bytes: 1791819776
>>>>>> total extent tree bytes: 229130240
>>>>>> btree space waste bytes: 1018844959
>>>>>> file data blocks allocated: 51587230502912
>>>>>>     referenced 15627926712320
>>>>>>
>>>>>> I've tried everything I've found on the internet, but haven't
>>>>>> attempted to repair based on the warnings...
>>>>>>
>>>>>> What more info do you need to help me diagnose/fix this?
>>>>>>
>>>>>> Thanks!
>>>>>> -Rob
>>>>>>
>

Re: Next steps in recovery?

[Robert Wyrick]

Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.

$ cat /etc/os-release
NAME="Linux Mint"
VERSION="20.2 (Uma)"
ID=linuxmint
ID_LIKE=ubuntu
PRETTY_NAME="Linux Mint 20.2"
VERSION_ID="20.2"
HOME_URL="https://www.linuxmint.com/"
SUPPORT_URL="https://forums.linuxmint.com/"
BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
PRIVACY_POLICY_URL="https://www.linuxmint.com/"
VERSION_CODENAME=uma
UBUNTU_CODENAME=focal

$ uname -a
Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ ./configure
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether C compiler accepts -std=gnu90... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for an ANSI C-conforming const... yes
checking for working volatile... yes
checking whether byte ordering is bigendian... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for a BSD-compatible install... /usr/bin/install -c
checking whether ln -s works... yes
checking for ar... ar
checking for rm... /bin/rm
checking for rmdir... /bin/rmdir
checking for openat... yes
checking for reallocarray... yes
checking for clock_gettime... yes
checking linux/perf_event.h usability... yes
checking linux/perf_event.h presence... yes
checking for linux/perf_event.h... yes
checking linux/hw_breakpoint.h usability... yes
checking linux/hw_breakpoint.h presence... yes
checking for linux/hw_breakpoint.h... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking execinfo.h usability... yes
checking execinfo.h presence... yes
checking for execinfo.h... yes
checking for backtrace... yes
checking for backtrace_symbols_fd... yes
checking for xmlto... /usr/bin/xmlto
checking for mv... /bin/mv
checking for a sed that does not truncate output... /bin/sed
checking for asciidoc... /usr/bin/asciidoc
checking for asciidoctor... no
checking for EXT2FS... yes
checking for COM_ERR... yes
checking for REISERFS... yes
checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
checking linux/blkzoned.h usability... yes
checking linux/blkzoned.h presence... yes
checking for linux/blkzoned.h... yes
checking for struct blk_zone.capacity... no
checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
configure: error: linux/blkzoned.h does not provide blk_zone.capacity

---

Info on the file in question (linux/blkzoned.h):

$ dpkg -S /usr/include/linux/blkzoned.h
linux-libc-dev:amd64: /usr/include/linux/blkzoned.h

$ dpkg -l linux-libc-dev
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                 Version      Architecture Description
+++-====================-============-============-====================================
ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
Headers for development


So it appears that linux-libc-dev is way out-dated compared to my
kernel.  I don't know how to update it, though... there doesn't appear
to be a newer version available.

-Rob

On Mon, Sep 6, 2021 at 5:26 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/6 下午10:42, Robert Wyrick wrote:
> > 42+ hours of memtest86+, no errors detected.  4 passes complete.
> > Is that good enough?
>
> That's strange, such obvious bitflip should be easily detected.
>
> Is the fs only mounted on that computer?
>
> Anyway, you can continue try to repair with *latest* btrfs-progs.
>
> Thanks,
> Qu
> >
> > On Sun, Sep 5, 2021 at 4:03 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2021/9/6 上午12:00, Robert Wyrick wrote:
> >>> Running memtest86+ now....  20 hours in.  No errors yet.
> >>> Thanks for the analysis.  I'll let this run for another day or so.
> >>
> >> Just to mention, since 5.11 btrfs kernel module has the ability to
> >> detect most high bitflip before writing tree blocks to disks.
> >>
> >> Thus even with less reliable RAM, it's still more reliable than nothing.
> >>
> >> But still, with the existing errors, the RAM test is still an essential
> >> one before doing anything.
> >>
> >> Thanks,
> >> Qu
> >>>
> >>>
> >>> On Fri, Sep 3, 2021 at 12:53 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2021/9/3 下午2:48, Qu Wenruo wrote:
> >>>>>
> >>>>>
> >>>>> On 2021/9/3 上午10:43, Robert Wyrick wrote:
> >>>>>> I cannot mount my btrfs filesystem.
> >>>>>> $ uname -a
> >>>>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> >>>>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> >>>>>> $ btrfs version
> >>>>>> btrfs-progs v5.4.1
> >>>>>
> >>>>> The tool is a little too old, thus if you're going to repair, you'd
> >>>>> better to update the progs.
> >>>>>>
> >>>>>> I'm seeing the following from check:
> >>>>>> $ btrfs check -p /dev/sda
> >>>>>> Opening filesystem to check...
> >>>>>> Checking filesystem on /dev/sda
> >>>>>> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
> >>>>>> [1/7] checking root items                      (0:00:59 elapsed,
> >>>>>> 2649102 items checked)
> >>>>>> ERROR: invalid generation for extent 38179182174208, have
> >>>>>> 140737491486755 expect (0, 4057084]
> >>>>>
> >>>>> This is a repairable problem.
> >>>>>
> >>>>> We have test case for exactly the same case in tests/fsck-test/044 for it.
> >>>>
> >>>> Oh, this invalid extent generation is already a more direct indication
> >>>> of memory bitflip.
> >>>>
> >>>> 140737491486755 = 0x8000002fc823
> >>>>
> >>>> Without the high 0x8 bit, the remaining part is completely valid
> >>>> generation, 0x2fc823, which is inside the expectation.
> >>>>
> >>>> So, a memtest is a must before doing any repair.
> >>>> You won't want another bitflip to ruin your perfectly repairable fs.
> >>>>
> >>>> Thanks,
> >>>> Qu
> >>>>>
> >>>>>
> >>>>>> [2/7] checking extents                         (0:02:17 elapsed,
> >>>>>> 1116143 items checked)
> >>>>>> ERROR: errors found in extent allocation tree or chunk allocation
> >>>>>> cache and super generation don't match, space cache will be invalidated
> >>>>>> [3/7] checking free space cache                (0:00:00 elapsed)
> >>>>>> [4/7] chunresolved ref dir 8348950 index 3 namelen 7 name posters
> >>>>>> filetype 2 errors 2, no dir index
> >>>>>
> >>>>> No dir index can also be repaired.
> >>>>>
> >>>>> The dir index will be added back.
> >>>>>
> >>>>>> unresolved ref dir 8348950 index 3 namelen 7 name poSters filetype 2
> >>>>>> errors 5, no dir item, no inode ref
> >>>>>
> >>>>> No dir item nor inode ref can also be repaired, but with dir item and
> >>>>> inode ref removed.
> >>>>>
> >>>>> But the problem here looks very strange.
> >>>>>
> >>>>> It's the same dir and the same index, but different name.
> >>>>> posters vs poSters.
> >>>>>
> >>>>> 'S' is 0x53 and 's' is 0x73, I'm wondering if your system had a bad
> >>>>> memory which caused a bitflip and the problem.
> >>>>>
> >>>>> Thus I prefer to do a full memtest before running btrfs check --repair.
> >>>>>
> >>>>> Thanks,
> >>>>> Qu
> >>>>>
> >>>>>> [4/7] checking fs roots                        (0:00:42 elapsed,
> >>>>>> 108894 items checked)
> >>>>>> ERROR: errors found in fs roots
> >>>>>> found 15729059057664 bytes used, error(s) found
> >>>>>> total csum bytes: 15313288548
> >>>>>> total tree bytes: 18286739456
> >>>>>> total fs tree bytes: 1791819776
> >>>>>> total extent tree bytes: 229130240
> >>>>>> btree space waste bytes: 1018844959
> >>>>>> file data blocks allocated: 51587230502912
> >>>>>>     referenced 15627926712320
> >>>>>>
> >>>>>> I've tried everything I've found on the internet, but haven't
> >>>>>> attempted to repair based on the warnings...
> >>>>>>
> >>>>>> What more info do you need to help me diagnose/fix this?
> >>>>>>
> >>>>>> Thanks!
> >>>>>> -Rob
> >>>>>>
> >

Re: Next steps in recovery?

[Anand Jain]

On 07/09/2021 10:36, Robert Wyrick wrote:
> Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
> 
> $ cat /etc/os-release
> NAME="Linux Mint"
> VERSION="20.2 (Uma)"
> ID=linuxmint
> ID_LIKE=ubuntu
> PRETTY_NAME="Linux Mint 20.2"
> VERSION_ID="20.2"
> HOME_URL="https://www.linuxmint.com/"
> SUPPORT_URL="https://forums.linuxmint.com/"
> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
> VERSION_CODENAME=uma
> UBUNTU_CODENAME=focal
> 
> $ uname -a
> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> 
> $ ./configure
> checking for gcc... gcc
> checking whether the C compiler works... yes
> checking for C compiler default output file name... a.out
> checking for suffix of executables...
> checking whether we are cross compiling... no
> checking for suffix of object files... o
> checking whether we are using the GNU C compiler... yes
> checking whether gcc accepts -g... yes
> checking for gcc option to accept ISO C89... none needed
> checking how to run the C preprocessor... gcc -E
> checking for grep that handles long lines and -e... /bin/grep
> checking for egrep... /bin/grep -E
> checking for ANSI C header files... yes
> checking for sys/types.h... yes
> checking for sys/stat.h... yes
> checking for stdlib.h... yes
> checking for string.h... yes
> checking for memory.h... yes
> checking for strings.h... yes
> checking for inttypes.h... yes
> checking for stdint.h... yes
> checking for unistd.h... yes
> checking minix/config.h usability... no
> checking minix/config.h presence... no
> checking for minix/config.h... no
> checking whether it is safe to define __EXTENSIONS__... yes
> checking for gcc... (cached) gcc
> checking whether we are using the GNU C compiler... (cached) yes
> checking whether gcc accepts -g... (cached) yes
> checking for gcc option to accept ISO C89... (cached) none needed
> checking whether C compiler accepts -std=gnu90... yes
> checking build system type... x86_64-pc-linux-gnu
> checking host system type... x86_64-pc-linux-gnu
> checking for an ANSI C-conforming const... yes
> checking for working volatile... yes
> checking whether byte ordering is bigendian... no
> checking for special C compiler options needed for large files... no
> checking for _FILE_OFFSET_BITS value needed for large files... no
> checking for a BSD-compatible install... /usr/bin/install -c
> checking whether ln -s works... yes
> checking for ar... ar
> checking for rm... /bin/rm
> checking for rmdir... /bin/rmdir
> checking for openat... yes
> checking for reallocarray... yes
> checking for clock_gettime... yes
> checking linux/perf_event.h usability... yes
> checking linux/perf_event.h presence... yes
> checking for linux/perf_event.h... yes
> checking linux/hw_breakpoint.h usability... yes
> checking linux/hw_breakpoint.h presence... yes
> checking for linux/hw_breakpoint.h... yes
> checking for pkg-config... /usr/bin/pkg-config
> checking pkg-config is at least version 0.9.0... yes
> checking execinfo.h usability... yes
> checking execinfo.h presence... yes
> checking for execinfo.h... yes
> checking for backtrace... yes
> checking for backtrace_symbols_fd... yes
> checking for xmlto... /usr/bin/xmlto
> checking for mv... /bin/mv
> checking for a sed that does not truncate output... /bin/sed
> checking for asciidoc... /usr/bin/asciidoc
> checking for asciidoctor... no
> checking for EXT2FS... yes
> checking for COM_ERR... yes
> checking for REISERFS... yes
> checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
> checking linux/blkzoned.h usability... yes
> checking linux/blkzoned.h presence... yes
> checking for linux/blkzoned.h... yes
> checking for struct blk_zone.capacity... no
> checking for BLKGETZONESZ defined in linux/blkzoned.h... yes

> configure: error: linux/blkzoned.h does not provide blk_zone.capacity


> 
> ---
> 
> Info on the file in question (linux/blkzoned.h):
> 
> $ dpkg -S /usr/include/linux/blkzoned.h
> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
> 
> $ dpkg -l linux-libc-dev
> Desired=Unknown/Install/Remove/Purge/Hold
> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name                 Version      Architecture Description
> +++-====================-============-============-====================================
> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
> Headers for development
> 
> 
> So it appears that linux-libc-dev is way out-dated compared to my
> kernel.  I don't know how to update it, though... there doesn't appear
> to be a newer version available.

You could disable the zoned.

   ./configure --disable-zoned

Re: Next steps in recovery?

[Robert Wyrick]

What exactly would i be disabling?  I don't know what zoned does.

On Mon, Sep 6, 2021, 9:07 PM Anand Jain <anand.jain@oracle.com> wrote:
>
> On 07/09/2021 10:36, Robert Wyrick wrote:
> > Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
> >
> > $ cat /etc/os-release
> > NAME="Linux Mint"
> > VERSION="20.2 (Uma)"
> > ID=linuxmint
> > ID_LIKE=ubuntu
> > PRETTY_NAME="Linux Mint 20.2"
> > VERSION_ID="20.2"
> > HOME_URL="https://www.linuxmint.com/"
> > SUPPORT_URL="https://forums.linuxmint.com/"
> > BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
> > PRIVACY_POLICY_URL="https://www.linuxmint.com/"
> > VERSION_CODENAME=uma
> > UBUNTU_CODENAME=focal
> >
> > $ uname -a
> > Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> > 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ ./configure
> > checking for gcc... gcc
> > checking whether the C compiler works... yes
> > checking for C compiler default output file name... a.out
> > checking for suffix of executables...
> > checking whether we are cross compiling... no
> > checking for suffix of object files... o
> > checking whether we are using the GNU C compiler... yes
> > checking whether gcc accepts -g... yes
> > checking for gcc option to accept ISO C89... none needed
> > checking how to run the C preprocessor... gcc -E
> > checking for grep that handles long lines and -e... /bin/grep
> > checking for egrep... /bin/grep -E
> > checking for ANSI C header files... yes
> > checking for sys/types.h... yes
> > checking for sys/stat.h... yes
> > checking for stdlib.h... yes
> > checking for string.h... yes
> > checking for memory.h... yes
> > checking for strings.h... yes
> > checking for inttypes.h... yes
> > checking for stdint.h... yes
> > checking for unistd.h... yes
> > checking minix/config.h usability... no
> > checking minix/config.h presence... no
> > checking for minix/config.h... no
> > checking whether it is safe to define __EXTENSIONS__... yes
> > checking for gcc... (cached) gcc
> > checking whether we are using the GNU C compiler... (cached) yes
> > checking whether gcc accepts -g... (cached) yes
> > checking for gcc option to accept ISO C89... (cached) none needed
> > checking whether C compiler accepts -std=gnu90... yes
> > checking build system type... x86_64-pc-linux-gnu
> > checking host system type... x86_64-pc-linux-gnu
> > checking for an ANSI C-conforming const... yes
> > checking for working volatile... yes
> > checking whether byte ordering is bigendian... no
> > checking for special C compiler options needed for large files... no
> > checking for _FILE_OFFSET_BITS value needed for large files... no
> > checking for a BSD-compatible install... /usr/bin/install -c
> > checking whether ln -s works... yes
> > checking for ar... ar
> > checking for rm... /bin/rm
> > checking for rmdir... /bin/rmdir
> > checking for openat... yes
> > checking for reallocarray... yes
> > checking for clock_gettime... yes
> > checking linux/perf_event.h usability... yes
> > checking linux/perf_event.h presence... yes
> > checking for linux/perf_event.h... yes
> > checking linux/hw_breakpoint.h usability... yes
> > checking linux/hw_breakpoint.h presence... yes
> > checking for linux/hw_breakpoint.h... yes
> > checking for pkg-config... /usr/bin/pkg-config
> > checking pkg-config is at least version 0.9.0... yes
> > checking execinfo.h usability... yes
> > checking execinfo.h presence... yes
> > checking for execinfo.h... yes
> > checking for backtrace... yes
> > checking for backtrace_symbols_fd... yes
> > checking for xmlto... /usr/bin/xmlto
> > checking for mv... /bin/mv
> > checking for a sed that does not truncate output... /bin/sed
> > checking for asciidoc... /usr/bin/asciidoc
> > checking for asciidoctor... no
> > checking for EXT2FS... yes
> > checking for COM_ERR... yes
> > checking for REISERFS... yes
> > checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
> > checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
> > checking linux/blkzoned.h usability... yes
> > checking linux/blkzoned.h presence... yes
> > checking for linux/blkzoned.h... yes
> > checking for struct blk_zone.capacity... no
> > checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
>
> > configure: error: linux/blkzoned.h does not provide blk_zone.capacity
>
>
> >
> > ---
> >
> > Info on the file in question (linux/blkzoned.h):
> >
> > $ dpkg -S /usr/include/linux/blkzoned.h
> > linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
> >
> > $ dpkg -l linux-libc-dev
> > Desired=Unknown/Install/Remove/Purge/Hold
> > | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> > |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> > ||/ Name                 Version      Architecture Description
> > +++-====================-============-============-====================================
> > ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
> > Headers for development
> >
> >
> > So it appears that linux-libc-dev is way out-dated compared to my
> > kernel.  I don't know how to update it, though... there doesn't appear
> > to be a newer version available.
>
> You could disable the zoned.
>
>    ./configure --disable-zoned
>
>
>
>
>

Re: Next steps in recovery?

[Qu Wenruo]

On 2021/9/7 下午12:36, Robert Wyrick wrote:
> What exactly would i be disabling?  I don't know what zoned does.

The zoned device support.

If you don't have any host-managed zoned device, there is no reason you
would like to enable it.

https://zonedstorage.io/introduction/

Thanks,
Qu

>
> On Mon, Sep 6, 2021, 9:07 PM Anand Jain <anand.jain@oracle.com> wrote:
>>
>> On 07/09/2021 10:36, Robert Wyrick wrote:
>>> Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
>>>
>>> $ cat /etc/os-release
>>> NAME="Linux Mint"
>>> VERSION="20.2 (Uma)"
>>> ID=linuxmint
>>> ID_LIKE=ubuntu
>>> PRETTY_NAME="Linux Mint 20.2"
>>> VERSION_ID="20.2"
>>> HOME_URL="https://www.linuxmint.com/"
>>> SUPPORT_URL="https://forums.linuxmint.com/"
>>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
>>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
>>> VERSION_CODENAME=uma
>>> UBUNTU_CODENAME=focal
>>>
>>> $ uname -a
>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> $ ./configure
>>> checking for gcc... gcc
>>> checking whether the C compiler works... yes
>>> checking for C compiler default output file name... a.out
>>> checking for suffix of executables...
>>> checking whether we are cross compiling... no
>>> checking for suffix of object files... o
>>> checking whether we are using the GNU C compiler... yes
>>> checking whether gcc accepts -g... yes
>>> checking for gcc option to accept ISO C89... none needed
>>> checking how to run the C preprocessor... gcc -E
>>> checking for grep that handles long lines and -e... /bin/grep
>>> checking for egrep... /bin/grep -E
>>> checking for ANSI C header files... yes
>>> checking for sys/types.h... yes
>>> checking for sys/stat.h... yes
>>> checking for stdlib.h... yes
>>> checking for string.h... yes
>>> checking for memory.h... yes
>>> checking for strings.h... yes
>>> checking for inttypes.h... yes
>>> checking for stdint.h... yes
>>> checking for unistd.h... yes
>>> checking minix/config.h usability... no
>>> checking minix/config.h presence... no
>>> checking for minix/config.h... no
>>> checking whether it is safe to define __EXTENSIONS__... yes
>>> checking for gcc... (cached) gcc
>>> checking whether we are using the GNU C compiler... (cached) yes
>>> checking whether gcc accepts -g... (cached) yes
>>> checking for gcc option to accept ISO C89... (cached) none needed
>>> checking whether C compiler accepts -std=gnu90... yes
>>> checking build system type... x86_64-pc-linux-gnu
>>> checking host system type... x86_64-pc-linux-gnu
>>> checking for an ANSI C-conforming const... yes
>>> checking for working volatile... yes
>>> checking whether byte ordering is bigendian... no
>>> checking for special C compiler options needed for large files... no
>>> checking for _FILE_OFFSET_BITS value needed for large files... no
>>> checking for a BSD-compatible install... /usr/bin/install -c
>>> checking whether ln -s works... yes
>>> checking for ar... ar
>>> checking for rm... /bin/rm
>>> checking for rmdir... /bin/rmdir
>>> checking for openat... yes
>>> checking for reallocarray... yes
>>> checking for clock_gettime... yes
>>> checking linux/perf_event.h usability... yes
>>> checking linux/perf_event.h presence... yes
>>> checking for linux/perf_event.h... yes
>>> checking linux/hw_breakpoint.h usability... yes
>>> checking linux/hw_breakpoint.h presence... yes
>>> checking for linux/hw_breakpoint.h... yes
>>> checking for pkg-config... /usr/bin/pkg-config
>>> checking pkg-config is at least version 0.9.0... yes
>>> checking execinfo.h usability... yes
>>> checking execinfo.h presence... yes
>>> checking for execinfo.h... yes
>>> checking for backtrace... yes
>>> checking for backtrace_symbols_fd... yes
>>> checking for xmlto... /usr/bin/xmlto
>>> checking for mv... /bin/mv
>>> checking for a sed that does not truncate output... /bin/sed
>>> checking for asciidoc... /usr/bin/asciidoc
>>> checking for asciidoctor... no
>>> checking for EXT2FS... yes
>>> checking for COM_ERR... yes
>>> checking for REISERFS... yes
>>> checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
>>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
>>> checking linux/blkzoned.h usability... yes
>>> checking linux/blkzoned.h presence... yes
>>> checking for linux/blkzoned.h... yes
>>> checking for struct blk_zone.capacity... no
>>> checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
>>
>>> configure: error: linux/blkzoned.h does not provide blk_zone.capacity
>>
>>
>>>
>>> ---
>>>
>>> Info on the file in question (linux/blkzoned.h):
>>>
>>> $ dpkg -S /usr/include/linux/blkzoned.h
>>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
>>>
>>> $ dpkg -l linux-libc-dev
>>> Desired=Unknown/Install/Remove/Purge/Hold
>>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
>>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>>> ||/ Name                 Version      Architecture Description
>>> +++-====================-============-============-====================================
>>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
>>> Headers for development
>>>
>>>
>>> So it appears that linux-libc-dev is way out-dated compared to my
>>> kernel.  I don't know how to update it, though... there doesn't appear
>>> to be a newer version available.
>>
>> You could disable the zoned.
>>
>>     ./configure --disable-zoned
>>
>>
>>
>>
>>

Re: Next steps in recovery?

[Robert Wyrick]

Ran a repair:

$ sudo ./btrfs check --repair -p /dev/sda  # I did NOT make install,
just ran from the compiled directory
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no
fsck can successfully repair all types of filesystem corruption. Eg.
some software or hardware bugs can fatally damage a volume.
The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/sda
UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
[1/7] checking root items                      (0:00:59 elapsed,
2649102 items checked)
Fixed 0 roots.
Reset extent item (38179182174208) generation to 4057084elapsed,
1116143 items checked)
No device size related problem found           (0:02:22 elapsed,
1116143 items checked)
[2/7] checking extents                         (0:02:23 elapsed,
1116143 items checked)
cache and super generation don't match, space cache will be invalidated
[3/7] checking free space cache                (0:00:00 elapsed)
Deleting bad dir index [8348950,96,3] root 259 (0:00:25 elapsed,
106695 items checked)
repairing missing dir index item for inode 834922400:26 elapsed,
108893 items checked)
[4/7] checking fs roots                        (0:01:04 elapsed,
217787 items checked)
[5/7] checking csums (without verifying data)  (0:00:04 elapsed,
12350321 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 4
items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 15729059057664 bytes used, no error found
total csum bytes: 15313288548
total tree bytes: 18286739456
total fs tree bytes: 1791819776
total extent tree bytes: 229130240
btree space waste bytes: 1018844959
file data blocks allocated: 51587230502912
 referenced 15627926712320

I can now mount the filesystem successfully!  Thank you for your help.

I do have some additional questions if you don't mind...
I am already using RAID 1 to handle single disk outages.  I assume
things could have gone much worse and I could have lost the whole
filesystem.  Aside from backups (I know, I know), is there anything
else I can do to prevent such issues or make them easier to recover
from?  Could this problem have been avoided/detected earlier?  This
wasn't a disk failure and according to memtest86+, it wasn't due to
bad memory either....  I don't run scrubs very often.  Should I?  I
guess the more general question is:  What are the best practices for
maintaining a healthy btrfs file system?

Thanks again!

On Mon, Sep 6, 2021 at 10:53 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/7 下午12:36, Robert Wyrick wrote:
> > What exactly would i be disabling?  I don't know what zoned does.
>
> The zoned device support.
>
> If you don't have any host-managed zoned device, there is no reason you
> would like to enable it.
>
> https://zonedstorage.io/introduction/
>
> Thanks,
> Qu
>
> >
> > On Mon, Sep 6, 2021, 9:07 PM Anand Jain <anand.jain@oracle.com> wrote:
> >>
> >> On 07/09/2021 10:36, Robert Wyrick wrote:
> >>> Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
> >>>
> >>> $ cat /etc/os-release
> >>> NAME="Linux Mint"
> >>> VERSION="20.2 (Uma)"
> >>> ID=linuxmint
> >>> ID_LIKE=ubuntu
> >>> PRETTY_NAME="Linux Mint 20.2"
> >>> VERSION_ID="20.2"
> >>> HOME_URL="https://www.linuxmint.com/"
> >>> SUPPORT_URL="https://forums.linuxmint.com/"
> >>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
> >>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
> >>> VERSION_CODENAME=uma
> >>> UBUNTU_CODENAME=focal
> >>>
> >>> $ uname -a
> >>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> >>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> >>>
> >>> $ ./configure
> >>> checking for gcc... gcc
> >>> checking whether the C compiler works... yes
> >>> checking for C compiler default output file name... a.out
> >>> checking for suffix of executables...
> >>> checking whether we are cross compiling... no
> >>> checking for suffix of object files... o
> >>> checking whether we are using the GNU C compiler... yes
> >>> checking whether gcc accepts -g... yes
> >>> checking for gcc option to accept ISO C89... none needed
> >>> checking how to run the C preprocessor... gcc -E
> >>> checking for grep that handles long lines and -e... /bin/grep
> >>> checking for egrep... /bin/grep -E
> >>> checking for ANSI C header files... yes
> >>> checking for sys/types.h... yes
> >>> checking for sys/stat.h... yes
> >>> checking for stdlib.h... yes
> >>> checking for string.h... yes
> >>> checking for memory.h... yes
> >>> checking for strings.h... yes
> >>> checking for inttypes.h... yes
> >>> checking for stdint.h... yes
> >>> checking for unistd.h... yes
> >>> checking minix/config.h usability... no
> >>> checking minix/config.h presence... no
> >>> checking for minix/config.h... no
> >>> checking whether it is safe to define __EXTENSIONS__... yes
> >>> checking for gcc... (cached) gcc
> >>> checking whether we are using the GNU C compiler... (cached) yes
> >>> checking whether gcc accepts -g... (cached) yes
> >>> checking for gcc option to accept ISO C89... (cached) none needed
> >>> checking whether C compiler accepts -std=gnu90... yes
> >>> checking build system type... x86_64-pc-linux-gnu
> >>> checking host system type... x86_64-pc-linux-gnu
> >>> checking for an ANSI C-conforming const... yes
> >>> checking for working volatile... yes
> >>> checking whether byte ordering is bigendian... no
> >>> checking for special C compiler options needed for large files... no
> >>> checking for _FILE_OFFSET_BITS value needed for large files... no
> >>> checking for a BSD-compatible install... /usr/bin/install -c
> >>> checking whether ln -s works... yes
> >>> checking for ar... ar
> >>> checking for rm... /bin/rm
> >>> checking for rmdir... /bin/rmdir
> >>> checking for openat... yes
> >>> checking for reallocarray... yes
> >>> checking for clock_gettime... yes
> >>> checking linux/perf_event.h usability... yes
> >>> checking linux/perf_event.h presence... yes
> >>> checking for linux/perf_event.h... yes
> >>> checking linux/hw_breakpoint.h usability... yes
> >>> checking linux/hw_breakpoint.h presence... yes
> >>> checking for linux/hw_breakpoint.h... yes
> >>> checking for pkg-config... /usr/bin/pkg-config
> >>> checking pkg-config is at least version 0.9.0... yes
> >>> checking execinfo.h usability... yes
> >>> checking execinfo.h presence... yes
> >>> checking for execinfo.h... yes
> >>> checking for backtrace... yes
> >>> checking for backtrace_symbols_fd... yes
> >>> checking for xmlto... /usr/bin/xmlto
> >>> checking for mv... /bin/mv
> >>> checking for a sed that does not truncate output... /bin/sed
> >>> checking for asciidoc... /usr/bin/asciidoc
> >>> checking for asciidoctor... no
> >>> checking for EXT2FS... yes
> >>> checking for COM_ERR... yes
> >>> checking for REISERFS... yes
> >>> checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
> >>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
> >>> checking linux/blkzoned.h usability... yes
> >>> checking linux/blkzoned.h presence... yes
> >>> checking for linux/blkzoned.h... yes
> >>> checking for struct blk_zone.capacity... no
> >>> checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
> >>
> >>> configure: error: linux/blkzoned.h does not provide blk_zone.capacity
> >>
> >>
> >>>
> >>> ---
> >>>
> >>> Info on the file in question (linux/blkzoned.h):
> >>>
> >>> $ dpkg -S /usr/include/linux/blkzoned.h
> >>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
> >>>
> >>> $ dpkg -l linux-libc-dev
> >>> Desired=Unknown/Install/Remove/Purge/Hold
> >>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> >>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> >>> ||/ Name                 Version      Architecture Description
> >>> +++-====================-============-============-====================================
> >>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
> >>> Headers for development
> >>>
> >>>
> >>> So it appears that linux-libc-dev is way out-dated compared to my
> >>> kernel.  I don't know how to update it, though... there doesn't appear
> >>> to be a newer version available.
> >>
> >> You could disable the zoned.
> >>
> >>     ./configure --disable-zoned
> >>
> >>
> >>
> >>
> >>

Re: Next steps in recovery?

[Robert Wyrick]

Looks like I spoke too soon.

I can now mount the FS readonly.

I see this error in dmesg:
[58995.896369] CPU: 10 PID: 83845 Comm: btrfs-transacti Tainted: P
      OE     5.11.0-27-generic #29~20.04.1-Ubuntu
[58995.896373] Hardware name: System manufacturer System Product
Name/PRIME X370-PRO, BIOS 0515 03/30/2017
[58995.896376] RIP: 0010:btrfs_run_delayed_refs+0x1af/0x200 [btrfs]
[58995.896422] Code: 8b 55 50 f0 48 0f ba aa 48 0a 00 00 03 72 20 83
f8 fb 74 3c 83 f8 e2 74 37 89 c6 48 c7 c7 50 7e 77 c0 89 45 d0 e8 96
d4 4d d3 <0f> 0b 8b 45 d0 89 c1 ba 4c 08 00 00 4c 89 ef 89 45 d0 48 c7
c6 c0
[58995.896425] RSP: 0018:ffffb89a4a0dfdf8 EFLAGS: 00010282
[58995.896428] RAX: 0000000000000000 RBX: ffffffffffffffff RCX: 0000000000000027
[58995.896430] RDX: 0000000000000027 RSI: 00000000ffffdfff RDI: ffff960fdf098ac8
[58995.896432] RBP: ffffb89a4a0dfe40 R08: ffff960fdf098ac0 R09: ffffb89a4a0dfbb8
[58995.896434] R10: 0000000000000001 R11: 0000000000000001 R12: ffff96036a7d5378
[58995.896435] R13: ffff960115028888 R14: ffff96036a7d5200 R15: 0000000000000000
[58995.896437] FS:  0000000000000000(0000) GS:ffff960fdf080000(0000)
knlGS:0000000000000000
[58995.896439] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[58995.896441] CR2: 00005616936891e8 CR3: 000000010fe46000 CR4: 00000000003506e0
[58995.896444] Call Trace:
[58995.896450]  btrfs_commit_transaction+0x2c3/0xa80 [btrfs]
[58995.896500]  ? start_transaction+0xd5/0x590 [btrfs]
[58995.896549]  transaction_kthread+0x138/0x1b0 [btrfs]
[58995.896596]  kthread+0x114/0x150
[58995.896604]  ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
[58995.896649]  ? kthread_park+0x90/0x90
[58995.896653]  ret_from_fork+0x22/0x30
[58995.896661] ---[ end trace c8ba04bdf2113cae ]---
[58995.896664] BTRFS: error (device sdf) in
btrfs_run_delayed_refs:2124: errno=-17 Object already exists
[58995.896669] BTRFS info (device sdf): forced readonly
[58995.896672] BTRFS warning (device sdf): Skipping commit of aborted
transaction.
[58995.896674] BTRFS: error (device sdf) in cleanup_transaction:1939:
errno=-17 Object already exists

Read-only is better than nothing, but what would be my next steps?

On Tue, Sep 7, 2021 at 11:02 AM Robert Wyrick <rob@wyrick.org> wrote:
>
> Ran a repair:
>
> $ sudo ./btrfs check --repair -p /dev/sda  # I did NOT make install,
> just ran from the compiled directory
> enabling repair mode
> WARNING:
>
> Do not use --repair unless you are advised to do so by a developer
> or an experienced user, and then only after having accepted that no
> fsck can successfully repair all types of filesystem corruption. Eg.
> some software or hardware bugs can fatally damage a volume.
> The operation will start in 10 seconds.
> Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting repair.
> Opening filesystem to check...
> Checking filesystem on /dev/sda
> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
> [1/7] checking root items                      (0:00:59 elapsed,
> 2649102 items checked)
> Fixed 0 roots.
> Reset extent item (38179182174208) generation to 4057084elapsed,
> 1116143 items checked)
> No device size related problem found           (0:02:22 elapsed,
> 1116143 items checked)
> [2/7] checking extents                         (0:02:23 elapsed,
> 1116143 items checked)
> cache and super generation don't match, space cache will be invalidated
> [3/7] checking free space cache                (0:00:00 elapsed)
> Deleting bad dir index [8348950,96,3] root 259 (0:00:25 elapsed,
> 106695 items checked)
> repairing missing dir index item for inode 834922400:26 elapsed,
> 108893 items checked)
> [4/7] checking fs roots                        (0:01:04 elapsed,
> 217787 items checked)
> [5/7] checking csums (without verifying data)  (0:00:04 elapsed,
> 12350321 items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 4
> items checked)
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 15729059057664 bytes used, no error found
> total csum bytes: 15313288548
> total tree bytes: 18286739456
> total fs tree bytes: 1791819776
> total extent tree bytes: 229130240
> btree space waste bytes: 1018844959
> file data blocks allocated: 51587230502912
>  referenced 15627926712320
>
> I can now mount the filesystem successfully!  Thank you for your help.
>
> I do have some additional questions if you don't mind...
> I am already using RAID 1 to handle single disk outages.  I assume
> things could have gone much worse and I could have lost the whole
> filesystem.  Aside from backups (I know, I know), is there anything
> else I can do to prevent such issues or make them easier to recover
> from?  Could this problem have been avoided/detected earlier?  This
> wasn't a disk failure and according to memtest86+, it wasn't due to
> bad memory either....  I don't run scrubs very often.  Should I?  I
> guess the more general question is:  What are the best practices for
> maintaining a healthy btrfs file system?
>
> Thanks again!
>
> On Mon, Sep 6, 2021 at 10:53 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> >
> >
> > On 2021/9/7 下午12:36, Robert Wyrick wrote:
> > > What exactly would i be disabling?  I don't know what zoned does.
> >
> > The zoned device support.
> >
> > If you don't have any host-managed zoned device, there is no reason you
> > would like to enable it.
> >
> > https://zonedstorage.io/introduction/
> >
> > Thanks,
> > Qu
> >
> > >
> > > On Mon, Sep 6, 2021, 9:07 PM Anand Jain <anand.jain@oracle.com> wrote:
> > >>
> > >> On 07/09/2021 10:36, Robert Wyrick wrote:
> > >>> Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
> > >>>
> > >>> $ cat /etc/os-release
> > >>> NAME="Linux Mint"
> > >>> VERSION="20.2 (Uma)"
> > >>> ID=linuxmint
> > >>> ID_LIKE=ubuntu
> > >>> PRETTY_NAME="Linux Mint 20.2"
> > >>> VERSION_ID="20.2"
> > >>> HOME_URL="https://www.linuxmint.com/"
> > >>> SUPPORT_URL="https://forums.linuxmint.com/"
> > >>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
> > >>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
> > >>> VERSION_CODENAME=uma
> > >>> UBUNTU_CODENAME=focal
> > >>>
> > >>> $ uname -a
> > >>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> > >>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> > >>>
> > >>> $ ./configure
> > >>> checking for gcc... gcc
> > >>> checking whether the C compiler works... yes
> > >>> checking for C compiler default output file name... a.out
> > >>> checking for suffix of executables...
> > >>> checking whether we are cross compiling... no
> > >>> checking for suffix of object files... o
> > >>> checking whether we are using the GNU C compiler... yes
> > >>> checking whether gcc accepts -g... yes
> > >>> checking for gcc option to accept ISO C89... none needed
> > >>> checking how to run the C preprocessor... gcc -E
> > >>> checking for grep that handles long lines and -e... /bin/grep
> > >>> checking for egrep... /bin/grep -E
> > >>> checking for ANSI C header files... yes
> > >>> checking for sys/types.h... yes
> > >>> checking for sys/stat.h... yes
> > >>> checking for stdlib.h... yes
> > >>> checking for string.h... yes
> > >>> checking for memory.h... yes
> > >>> checking for strings.h... yes
> > >>> checking for inttypes.h... yes
> > >>> checking for stdint.h... yes
> > >>> checking for unistd.h... yes
> > >>> checking minix/config.h usability... no
> > >>> checking minix/config.h presence... no
> > >>> checking for minix/config.h... no
> > >>> checking whether it is safe to define __EXTENSIONS__... yes
> > >>> checking for gcc... (cached) gcc
> > >>> checking whether we are using the GNU C compiler... (cached) yes
> > >>> checking whether gcc accepts -g... (cached) yes
> > >>> checking for gcc option to accept ISO C89... (cached) none needed
> > >>> checking whether C compiler accepts -std=gnu90... yes
> > >>> checking build system type... x86_64-pc-linux-gnu
> > >>> checking host system type... x86_64-pc-linux-gnu
> > >>> checking for an ANSI C-conforming const... yes
> > >>> checking for working volatile... yes
> > >>> checking whether byte ordering is bigendian... no
> > >>> checking for special C compiler options needed for large files... no
> > >>> checking for _FILE_OFFSET_BITS value needed for large files... no
> > >>> checking for a BSD-compatible install... /usr/bin/install -c
> > >>> checking whether ln -s works... yes
> > >>> checking for ar... ar
> > >>> checking for rm... /bin/rm
> > >>> checking for rmdir... /bin/rmdir
> > >>> checking for openat... yes
> > >>> checking for reallocarray... yes
> > >>> checking for clock_gettime... yes
> > >>> checking linux/perf_event.h usability... yes
> > >>> checking linux/perf_event.h presence... yes
> > >>> checking for linux/perf_event.h... yes
> > >>> checking linux/hw_breakpoint.h usability... yes
> > >>> checking linux/hw_breakpoint.h presence... yes
> > >>> checking for linux/hw_breakpoint.h... yes
> > >>> checking for pkg-config... /usr/bin/pkg-config
> > >>> checking pkg-config is at least version 0.9.0... yes
> > >>> checking execinfo.h usability... yes
> > >>> checking execinfo.h presence... yes
> > >>> checking for execinfo.h... yes
> > >>> checking for backtrace... yes
> > >>> checking for backtrace_symbols_fd... yes
> > >>> checking for xmlto... /usr/bin/xmlto
> > >>> checking for mv... /bin/mv
> > >>> checking for a sed that does not truncate output... /bin/sed
> > >>> checking for asciidoc... /usr/bin/asciidoc
> > >>> checking for asciidoctor... no
> > >>> checking for EXT2FS... yes
> > >>> checking for COM_ERR... yes
> > >>> checking for REISERFS... yes
> > >>> checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
> > >>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
> > >>> checking linux/blkzoned.h usability... yes
> > >>> checking linux/blkzoned.h presence... yes
> > >>> checking for linux/blkzoned.h... yes
> > >>> checking for struct blk_zone.capacity... no
> > >>> checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
> > >>
> > >>> configure: error: linux/blkzoned.h does not provide blk_zone.capacity
> > >>
> > >>
> > >>>
> > >>> ---
> > >>>
> > >>> Info on the file in question (linux/blkzoned.h):
> > >>>
> > >>> $ dpkg -S /usr/include/linux/blkzoned.h
> > >>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
> > >>>
> > >>> $ dpkg -l linux-libc-dev
> > >>> Desired=Unknown/Install/Remove/Purge/Hold
> > >>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> > >>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> > >>> ||/ Name                 Version      Architecture Description
> > >>> +++-====================-============-============-====================================
> > >>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
> > >>> Headers for development
> > >>>
> > >>>
> > >>> So it appears that linux-libc-dev is way out-dated compared to my
> > >>> kernel.  I don't know how to update it, though... there doesn't appear
> > >>> to be a newer version available.
> > >>
> > >> You could disable the zoned.
> > >>
> > >>     ./configure --disable-zoned
> > >>
> > >>
> > >>
> > >>
> > >>

Re: Next steps in recovery?

[Robert Wyrick]

Re-running check now shows:

[1/7] checking root items                      (0:00:55 elapsed,
2649102 items checked)
[2/7] checking extents                         (0:02:13 elapsed,
1116141 items checked)
there is no free space entry for 18365358505984-18365358522368d, 130
items checked)
there is no free space entry for 18365358505984-18366416814080
cache appears valid but isn't 18365343072256
there is no free space entry for 19764429062144-19764429078528d, 348
items checked)
there is no free space entry for 19764429062144-19765502410752
cache appears valid but isn't 19764428668928
wanted bytes 49152, found 16384 for off 254016221675521 elapsed, 1534
items checked)
wanted bytes 1058373632, found 16384 for off 25401622167552
cache appears valid but isn't 25401606799360
there is no free space entry for 28659399229440-28659399245824d, 2636
items checked)
there is no free space entry for 28659399229440-28660413235200
cache appears valid but isn't 28659339493376
wanted offset 29154336178176, found 2915433616179200:33 elapsed, 2792
items checked)
wanted offset 29154336178176, found 29154336161792
cache appears valid but isn't 29154334474240
there is no free space entry for 30899331825664-30899331842048d, 3585
items checked)
there is no free space entry for 30899331825664-30900272234496
cache appears valid but isn't 30899198492672
there is no free space entry for 32134011568128-32134011584512d, 4474
items checked)
there is no free space entry for 32134011568128-32135075332096
cache appears valid but isn't 32134001590272
wanted offset 33148689629184, found 3314868961280000:59 elapsed, 4963
items checked)
wanted offset 33148689629184, found 33148689612800
cache appears valid but isn't 33148687613952
there is no free space entry for 34611225755648-34611225772032d, 6036
items checked)
there is no free space entry for 34611225755648-34612197720064
cache appears valid but isn't 34611123978240
there is no free space entry for 37374972723200-37374972739584d, 8051
items checked)
there is no free space entry for 37374972723200-37376042729472
cache appears valid but isn't 37374968987648
there is no free space entry for 37484494651392-37484494667776d, 8172
items checked)
there is no free space entry for 37484494651392-37485564395520
cache appears valid but isn't 37484490653696
wanted bytes 49152, found 32768 for off 377572293017606 elapsed, 8381
items checked)
wanted bytes 1065517056, found 32768 for off 37757229301760
cache appears valid but isn't 37757221076992
there is no free space entry for 38414356250624-38414356267008d, 9004
items checked)
there is no free space entry for 38414356250624-38415424815104
cache appears valid but isn't 38414351073280
there is no free space entry for 41509957402624-41509957419008d, 11792
items checked)
there is no free space entry for 41509957402624-41511022493696
cache appears valid but isn't 41509948751872
there is no free space entry for 42293815459840-42293815492608d, 12469
items checked)
there is no free space entry for 42293815459840-42294887579648
cache appears valid but isn't 42293813837824
[3/7] checking free space cache                (0:04:18 elapsed, 14910
items checked)
[4/7] checking fs roots                        (0:00:26 elapsed,
108894 items checked)
[5/7] checking csums (without verifying data)  (0:00:03 elapsed,
12350321 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 4
items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 15729059287040 bytes used, error(s) found
total csum bytes: 15313288548
total tree bytes: 18286706688
total fs tree bytes: 1791819776
total extent tree bytes: 229097472
btree space waste bytes: 1018811836
file data blocks allocated: 51587230765056
 referenced 15627926974464

On Tue, Sep 7, 2021 at 11:17 AM Robert Wyrick <rob@wyrick.org> wrote:
>
> Looks like I spoke too soon.
>
> I can now mount the FS readonly.
>
> I see this error in dmesg:
> [58995.896369] CPU: 10 PID: 83845 Comm: btrfs-transacti Tainted: P
>       OE     5.11.0-27-generic #29~20.04.1-Ubuntu
> [58995.896373] Hardware name: System manufacturer System Product
> Name/PRIME X370-PRO, BIOS 0515 03/30/2017
> [58995.896376] RIP: 0010:btrfs_run_delayed_refs+0x1af/0x200 [btrfs]
> [58995.896422] Code: 8b 55 50 f0 48 0f ba aa 48 0a 00 00 03 72 20 83
> f8 fb 74 3c 83 f8 e2 74 37 89 c6 48 c7 c7 50 7e 77 c0 89 45 d0 e8 96
> d4 4d d3 <0f> 0b 8b 45 d0 89 c1 ba 4c 08 00 00 4c 89 ef 89 45 d0 48 c7
> c6 c0
> [58995.896425] RSP: 0018:ffffb89a4a0dfdf8 EFLAGS: 00010282
> [58995.896428] RAX: 0000000000000000 RBX: ffffffffffffffff RCX: 0000000000000027
> [58995.896430] RDX: 0000000000000027 RSI: 00000000ffffdfff RDI: ffff960fdf098ac8
> [58995.896432] RBP: ffffb89a4a0dfe40 R08: ffff960fdf098ac0 R09: ffffb89a4a0dfbb8
> [58995.896434] R10: 0000000000000001 R11: 0000000000000001 R12: ffff96036a7d5378
> [58995.896435] R13: ffff960115028888 R14: ffff96036a7d5200 R15: 0000000000000000
> [58995.896437] FS:  0000000000000000(0000) GS:ffff960fdf080000(0000)
> knlGS:0000000000000000
> [58995.896439] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [58995.896441] CR2: 00005616936891e8 CR3: 000000010fe46000 CR4: 00000000003506e0
> [58995.896444] Call Trace:
> [58995.896450]  btrfs_commit_transaction+0x2c3/0xa80 [btrfs]
> [58995.896500]  ? start_transaction+0xd5/0x590 [btrfs]
> [58995.896549]  transaction_kthread+0x138/0x1b0 [btrfs]
> [58995.896596]  kthread+0x114/0x150
> [58995.896604]  ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
> [58995.896649]  ? kthread_park+0x90/0x90
> [58995.896653]  ret_from_fork+0x22/0x30
> [58995.896661] ---[ end trace c8ba04bdf2113cae ]---
> [58995.896664] BTRFS: error (device sdf) in
> btrfs_run_delayed_refs:2124: errno=-17 Object already exists
> [58995.896669] BTRFS info (device sdf): forced readonly
> [58995.896672] BTRFS warning (device sdf): Skipping commit of aborted
> transaction.
> [58995.896674] BTRFS: error (device sdf) in cleanup_transaction:1939:
> errno=-17 Object already exists
>
> Read-only is better than nothing, but what would be my next steps?
>
> On Tue, Sep 7, 2021 at 11:02 AM Robert Wyrick <rob@wyrick.org> wrote:
> >
> > Ran a repair:
> >
> > $ sudo ./btrfs check --repair -p /dev/sda  # I did NOT make install,
> > just ran from the compiled directory
> > enabling repair mode
> > WARNING:
> >
> > Do not use --repair unless you are advised to do so by a developer
> > or an experienced user, and then only after having accepted that no
> > fsck can successfully repair all types of filesystem corruption. Eg.
> > some software or hardware bugs can fatally damage a volume.
> > The operation will start in 10 seconds.
> > Use Ctrl-C to stop it.
> > 10 9 8 7 6 5 4 3 2 1
> > Starting repair.
> > Opening filesystem to check...
> > Checking filesystem on /dev/sda
> > UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
> > [1/7] checking root items                      (0:00:59 elapsed,
> > 2649102 items checked)
> > Fixed 0 roots.
> > Reset extent item (38179182174208) generation to 4057084elapsed,
> > 1116143 items checked)
> > No device size related problem found           (0:02:22 elapsed,
> > 1116143 items checked)
> > [2/7] checking extents                         (0:02:23 elapsed,
> > 1116143 items checked)
> > cache and super generation don't match, space cache will be invalidated
> > [3/7] checking free space cache                (0:00:00 elapsed)
> > Deleting bad dir index [8348950,96,3] root 259 (0:00:25 elapsed,
> > 106695 items checked)
> > repairing missing dir index item for inode 834922400:26 elapsed,
> > 108893 items checked)
> > [4/7] checking fs roots                        (0:01:04 elapsed,
> > 217787 items checked)
> > [5/7] checking csums (without verifying data)  (0:00:04 elapsed,
> > 12350321 items checked)
> > [6/7] checking root refs                       (0:00:00 elapsed, 4
> > items checked)
> > [7/7] checking quota groups skipped (not enabled on this FS)
> > found 15729059057664 bytes used, no error found
> > total csum bytes: 15313288548
> > total tree bytes: 18286739456
> > total fs tree bytes: 1791819776
> > total extent tree bytes: 229130240
> > btree space waste bytes: 1018844959
> > file data blocks allocated: 51587230502912
> >  referenced 15627926712320
> >
> > I can now mount the filesystem successfully!  Thank you for your help.
> >
> > I do have some additional questions if you don't mind...
> > I am already using RAID 1 to handle single disk outages.  I assume
> > things could have gone much worse and I could have lost the whole
> > filesystem.  Aside from backups (I know, I know), is there anything
> > else I can do to prevent such issues or make them easier to recover
> > from?  Could this problem have been avoided/detected earlier?  This
> > wasn't a disk failure and according to memtest86+, it wasn't due to
> > bad memory either....  I don't run scrubs very often.  Should I?  I
> > guess the more general question is:  What are the best practices for
> > maintaining a healthy btrfs file system?
> >
> > Thanks again!
> >
> > On Mon, Sep 6, 2021 at 10:53 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >
> > >
> > >
> > > On 2021/9/7 下午12:36, Robert Wyrick wrote:
> > > > What exactly would i be disabling?  I don't know what zoned does.
> > >
> > > The zoned device support.
> > >
> > > If you don't have any host-managed zoned device, there is no reason you
> > > would like to enable it.
> > >
> > > https://zonedstorage.io/introduction/
> > >
> > > Thanks,
> > > Qu
> > >
> > > >
> > > > On Mon, Sep 6, 2021, 9:07 PM Anand Jain <anand.jain@oracle.com> wrote:
> > > >>
> > > >> On 07/09/2021 10:36, Robert Wyrick wrote:
> > > >>> Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
> > > >>>
> > > >>> $ cat /etc/os-release
> > > >>> NAME="Linux Mint"
> > > >>> VERSION="20.2 (Uma)"
> > > >>> ID=linuxmint
> > > >>> ID_LIKE=ubuntu
> > > >>> PRETTY_NAME="Linux Mint 20.2"
> > > >>> VERSION_ID="20.2"
> > > >>> HOME_URL="https://www.linuxmint.com/"
> > > >>> SUPPORT_URL="https://forums.linuxmint.com/"
> > > >>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
> > > >>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
> > > >>> VERSION_CODENAME=uma
> > > >>> UBUNTU_CODENAME=focal
> > > >>>
> > > >>> $ uname -a
> > > >>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> > > >>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> > > >>>
> > > >>> $ ./configure
> > > >>> checking for gcc... gcc
> > > >>> checking whether the C compiler works... yes
> > > >>> checking for C compiler default output file name... a.out
> > > >>> checking for suffix of executables...
> > > >>> checking whether we are cross compiling... no
> > > >>> checking for suffix of object files... o
> > > >>> checking whether we are using the GNU C compiler... yes
> > > >>> checking whether gcc accepts -g... yes
> > > >>> checking for gcc option to accept ISO C89... none needed
> > > >>> checking how to run the C preprocessor... gcc -E
> > > >>> checking for grep that handles long lines and -e... /bin/grep
> > > >>> checking for egrep... /bin/grep -E
> > > >>> checking for ANSI C header files... yes
> > > >>> checking for sys/types.h... yes
> > > >>> checking for sys/stat.h... yes
> > > >>> checking for stdlib.h... yes
> > > >>> checking for string.h... yes
> > > >>> checking for memory.h... yes
> > > >>> checking for strings.h... yes
> > > >>> checking for inttypes.h... yes
> > > >>> checking for stdint.h... yes
> > > >>> checking for unistd.h... yes
> > > >>> checking minix/config.h usability... no
> > > >>> checking minix/config.h presence... no
> > > >>> checking for minix/config.h... no
> > > >>> checking whether it is safe to define __EXTENSIONS__... yes
> > > >>> checking for gcc... (cached) gcc
> > > >>> checking whether we are using the GNU C compiler... (cached) yes
> > > >>> checking whether gcc accepts -g... (cached) yes
> > > >>> checking for gcc option to accept ISO C89... (cached) none needed
> > > >>> checking whether C compiler accepts -std=gnu90... yes
> > > >>> checking build system type... x86_64-pc-linux-gnu
> > > >>> checking host system type... x86_64-pc-linux-gnu
> > > >>> checking for an ANSI C-conforming const... yes
> > > >>> checking for working volatile... yes
> > > >>> checking whether byte ordering is bigendian... no
> > > >>> checking for special C compiler options needed for large files... no
> > > >>> checking for _FILE_OFFSET_BITS value needed for large files... no
> > > >>> checking for a BSD-compatible install... /usr/bin/install -c
> > > >>> checking whether ln -s works... yes
> > > >>> checking for ar... ar
> > > >>> checking for rm... /bin/rm
> > > >>> checking for rmdir... /bin/rmdir
> > > >>> checking for openat... yes
> > > >>> checking for reallocarray... yes
> > > >>> checking for clock_gettime... yes
> > > >>> checking linux/perf_event.h usability... yes
> > > >>> checking linux/perf_event.h presence... yes
> > > >>> checking for linux/perf_event.h... yes
> > > >>> checking linux/hw_breakpoint.h usability... yes
> > > >>> checking linux/hw_breakpoint.h presence... yes
> > > >>> checking for linux/hw_breakpoint.h... yes
> > > >>> checking for pkg-config... /usr/bin/pkg-config
> > > >>> checking pkg-config is at least version 0.9.0... yes
> > > >>> checking execinfo.h usability... yes
> > > >>> checking execinfo.h presence... yes
> > > >>> checking for execinfo.h... yes
> > > >>> checking for backtrace... yes
> > > >>> checking for backtrace_symbols_fd... yes
> > > >>> checking for xmlto... /usr/bin/xmlto
> > > >>> checking for mv... /bin/mv
> > > >>> checking for a sed that does not truncate output... /bin/sed
> > > >>> checking for asciidoc... /usr/bin/asciidoc
> > > >>> checking for asciidoctor... no
> > > >>> checking for EXT2FS... yes
> > > >>> checking for COM_ERR... yes
> > > >>> checking for REISERFS... yes
> > > >>> checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
> > > >>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
> > > >>> checking linux/blkzoned.h usability... yes
> > > >>> checking linux/blkzoned.h presence... yes
> > > >>> checking for linux/blkzoned.h... yes
> > > >>> checking for struct blk_zone.capacity... no
> > > >>> checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
> > > >>
> > > >>> configure: error: linux/blkzoned.h does not provide blk_zone.capacity
> > > >>
> > > >>
> > > >>>
> > > >>> ---
> > > >>>
> > > >>> Info on the file in question (linux/blkzoned.h):
> > > >>>
> > > >>> $ dpkg -S /usr/include/linux/blkzoned.h
> > > >>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
> > > >>>
> > > >>> $ dpkg -l linux-libc-dev
> > > >>> Desired=Unknown/Install/Remove/Purge/Hold
> > > >>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> > > >>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> > > >>> ||/ Name                 Version      Architecture Description
> > > >>> +++-====================-============-============-====================================
> > > >>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
> > > >>> Headers for development
> > > >>>
> > > >>>
> > > >>> So it appears that linux-libc-dev is way out-dated compared to my
> > > >>> kernel.  I don't know how to update it, though... there doesn't appear
> > > >>> to be a newer version available.
> > > >>
> > > >> You could disable the zoned.
> > > >>
> > > >>     ./configure --disable-zoned
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>

Re: Next steps in recovery?

[Qu Wenruo]

On 2021/9/8 上午1:02, Robert Wyrick wrote:
> Ran a repair:
>
> $ sudo ./btrfs check --repair -p /dev/sda  # I did NOT make install,
> just ran from the compiled directory
> enabling repair mode
> WARNING:
>
> Do not use --repair unless you are advised to do so by a developer
> or an experienced user, and then only after having accepted that no
> fsck can successfully repair all types of filesystem corruption. Eg.
> some software or hardware bugs can fatally damage a volume.
> The operation will start in 10 seconds.
> Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting repair.
> Opening filesystem to check...
> Checking filesystem on /dev/sda
> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
> [1/7] checking root items                      (0:00:59 elapsed,
> 2649102 items checked)
> Fixed 0 roots.
> Reset extent item (38179182174208) generation to 4057084elapsed,
> 1116143 items checked)
> No device size related problem found           (0:02:22 elapsed,
> 1116143 items checked)
> [2/7] checking extents                         (0:02:23 elapsed,
> 1116143 items checked)
> cache and super generation don't match, space cache will be invalidated
> [3/7] checking free space cache                (0:00:00 elapsed)
> Deleting bad dir index [8348950,96,3] root 259 (0:00:25 elapsed,
> 106695 items checked)
> repairing missing dir index item for inode 834922400:26 elapsed,
> 108893 items checked)
> [4/7] checking fs roots                        (0:01:04 elapsed,
> 217787 items checked)
> [5/7] checking csums (without verifying data)  (0:00:04 elapsed,
> 12350321 items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 4
> items checked)
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 15729059057664 bytes used, no error found
> total csum bytes: 15313288548
> total tree bytes: 18286739456
> total fs tree bytes: 1791819776
> total extent tree bytes: 229130240
> btree space waste bytes: 1018844959
> file data blocks allocated: 51587230502912
>   referenced 15627926712320
>
> I can now mount the filesystem successfully!  Thank you for your help.
>
> I do have some additional questions if you don't mind...
> I am already using RAID 1 to handle single disk outages.

One thing to note is, RAID is not perfect, not even close to proper backup.

RAID is really only suitable to handle disk failures, nothing more than
that.

In a spectrum of backup, RAID is really just better than nothing.

In this particular case, all the corruption is from bitflips, thus all
copies are corrupted, no profile can save the day.

>  I assume
> things could have gone much worse and I could have lost the whole
> filesystem.

If you're using newer kernels all time, the kernel can detect the extent
generation problem before writing the corrupted data back to disk, thus
save the day.

>  Aside from backups (I know, I know), is there anything
> else I can do to prevent such issues or make them easier to recover
> from?

Newer kernel (v5.11 and newer) can prevent it.
Although when such rejection happens, it will not feel that comfort
though, as it would mostly result the fs to go RO.
But still way better than writing bad data onto disks.

>  Could this problem have been avoided/detected earlier?

Yes, newer kernel.

>  This
> wasn't a disk failure and according to memtest86+, it wasn't due to
> bad memory either....

I still don't believe, maybe you can try to run memtester (which is ran
in user space, and since we have kernel doing the page mapping, it may
expose a different workload on the memory controller than memtest86+)

Since the extent generation corruption is a super obvious bitflip.

>  I don't run scrubs very often.  Should I?

For newer kernels, the corruption can be rejected in first place, thus
the scrub is only going to detect problems already in the fs.

For older kernels, scrub won't detect the problem anyw.

So I guess you don't need that frequent scrub, but it's still recommended.
Maybe monthly?

>  I
> guess the more general question is:  What are the best practices for
> maintaining a healthy btrfs file system?

Well, healthy hardware, balanced kernel version between cutting edge and
stable.
Personally I'm more towards cutting edge thought.

Thanks,
Qu

>
> Thanks again!
>
> On Mon, Sep 6, 2021 at 10:53 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2021/9/7 下午12:36, Robert Wyrick wrote:
>>> What exactly would i be disabling?  I don't know what zoned does.
>>
>> The zoned device support.
>>
>> If you don't have any host-managed zoned device, there is no reason you
>> would like to enable it.
>>
>> https://zonedstorage.io/introduction/
>>
>> Thanks,
>> Qu
>>
>>>
>>> On Mon, Sep 6, 2021, 9:07 PM Anand Jain <anand.jain@oracle.com> wrote:
>>>>
>>>> On 07/09/2021 10:36, Robert Wyrick wrote:
>>>>> Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
>>>>>
>>>>> $ cat /etc/os-release
>>>>> NAME="Linux Mint"
>>>>> VERSION="20.2 (Uma)"
>>>>> ID=linuxmint
>>>>> ID_LIKE=ubuntu
>>>>> PRETTY_NAME="Linux Mint 20.2"
>>>>> VERSION_ID="20.2"
>>>>> HOME_URL="https://www.linuxmint.com/"
>>>>> SUPPORT_URL="https://forums.linuxmint.com/"
>>>>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
>>>>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
>>>>> VERSION_CODENAME=uma
>>>>> UBUNTU_CODENAME=focal
>>>>>
>>>>> $ uname -a
>>>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
>>>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>>>>
>>>>> $ ./configure
>>>>> checking for gcc... gcc
>>>>> checking whether the C compiler works... yes
>>>>> checking for C compiler default output file name... a.out
>>>>> checking for suffix of executables...
>>>>> checking whether we are cross compiling... no
>>>>> checking for suffix of object files... o
>>>>> checking whether we are using the GNU C compiler... yes
>>>>> checking whether gcc accepts -g... yes
>>>>> checking for gcc option to accept ISO C89... none needed
>>>>> checking how to run the C preprocessor... gcc -E
>>>>> checking for grep that handles long lines and -e... /bin/grep
>>>>> checking for egrep... /bin/grep -E
>>>>> checking for ANSI C header files... yes
>>>>> checking for sys/types.h... yes
>>>>> checking for sys/stat.h... yes
>>>>> checking for stdlib.h... yes
>>>>> checking for string.h... yes
>>>>> checking for memory.h... yes
>>>>> checking for strings.h... yes
>>>>> checking for inttypes.h... yes
>>>>> checking for stdint.h... yes
>>>>> checking for unistd.h... yes
>>>>> checking minix/config.h usability... no
>>>>> checking minix/config.h presence... no
>>>>> checking for minix/config.h... no
>>>>> checking whether it is safe to define __EXTENSIONS__... yes
>>>>> checking for gcc... (cached) gcc
>>>>> checking whether we are using the GNU C compiler... (cached) yes
>>>>> checking whether gcc accepts -g... (cached) yes
>>>>> checking for gcc option to accept ISO C89... (cached) none needed
>>>>> checking whether C compiler accepts -std=gnu90... yes
>>>>> checking build system type... x86_64-pc-linux-gnu
>>>>> checking host system type... x86_64-pc-linux-gnu
>>>>> checking for an ANSI C-conforming const... yes
>>>>> checking for working volatile... yes
>>>>> checking whether byte ordering is bigendian... no
>>>>> checking for special C compiler options needed for large files... no
>>>>> checking for _FILE_OFFSET_BITS value needed for large files... no
>>>>> checking for a BSD-compatible install... /usr/bin/install -c
>>>>> checking whether ln -s works... yes
>>>>> checking for ar... ar
>>>>> checking for rm... /bin/rm
>>>>> checking for rmdir... /bin/rmdir
>>>>> checking for openat... yes
>>>>> checking for reallocarray... yes
>>>>> checking for clock_gettime... yes
>>>>> checking linux/perf_event.h usability... yes
>>>>> checking linux/perf_event.h presence... yes
>>>>> checking for linux/perf_event.h... yes
>>>>> checking linux/hw_breakpoint.h usability... yes
>>>>> checking linux/hw_breakpoint.h presence... yes
>>>>> checking for linux/hw_breakpoint.h... yes
>>>>> checking for pkg-config... /usr/bin/pkg-config
>>>>> checking pkg-config is at least version 0.9.0... yes
>>>>> checking execinfo.h usability... yes
>>>>> checking execinfo.h presence... yes
>>>>> checking for execinfo.h... yes
>>>>> checking for backtrace... yes
>>>>> checking for backtrace_symbols_fd... yes
>>>>> checking for xmlto... /usr/bin/xmlto
>>>>> checking for mv... /bin/mv
>>>>> checking for a sed that does not truncate output... /bin/sed
>>>>> checking for asciidoc... /usr/bin/asciidoc
>>>>> checking for asciidoctor... no
>>>>> checking for EXT2FS... yes
>>>>> checking for COM_ERR... yes
>>>>> checking for REISERFS... yes
>>>>> checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
>>>>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
>>>>> checking linux/blkzoned.h usability... yes
>>>>> checking linux/blkzoned.h presence... yes
>>>>> checking for linux/blkzoned.h... yes
>>>>> checking for struct blk_zone.capacity... no
>>>>> checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
>>>>
>>>>> configure: error: linux/blkzoned.h does not provide blk_zone.capacity
>>>>
>>>>
>>>>>
>>>>> ---
>>>>>
>>>>> Info on the file in question (linux/blkzoned.h):
>>>>>
>>>>> $ dpkg -S /usr/include/linux/blkzoned.h
>>>>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
>>>>>
>>>>> $ dpkg -l linux-libc-dev
>>>>> Desired=Unknown/Install/Remove/Purge/Hold
>>>>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
>>>>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>>>>> ||/ Name                 Version      Architecture Description
>>>>> +++-====================-============-============-====================================
>>>>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
>>>>> Headers for development
>>>>>
>>>>>
>>>>> So it appears that linux-libc-dev is way out-dated compared to my
>>>>> kernel.  I don't know how to update it, though... there doesn't appear
>>>>> to be a newer version available.
>>>>
>>>> You could disable the zoned.
>>>>
>>>>      ./configure --disable-zoned
>>>>
>>>>
>>>>
>>>>
>>>>

Re: Next steps in recovery?

[Qu Wenruo]

On 2021/9/8 上午4:47, Robert Wyrick wrote:
> Re-running check now shows:
>
> [1/7] checking root items                      (0:00:55 elapsed,
> 2649102 items checked)
> [2/7] checking extents                         (0:02:13 elapsed,
> 1116141 items checked)
> there is no free space entry for 18365358505984-18365358522368d, 130
> items checked)
> there is no free space entry for 18365358505984-18366416814080
> cache appears valid but isn't 18365343072256
> there is no free space entry for 19764429062144-19764429078528d, 348
> items checked)
> there is no free space entry for 19764429062144-19765502410752
> cache appears valid but isn't 19764428668928
> wanted bytes 49152, found 16384 for off 254016221675521 elapsed, 1534
> items checked)
> wanted bytes 1058373632, found 16384 for off 25401622167552
> cache appears valid but isn't 25401606799360
> there is no free space entry for 28659399229440-28659399245824d, 2636
> items checked)
> there is no free space entry for 28659399229440-28660413235200
> cache appears valid but isn't 28659339493376
> wanted offset 29154336178176, found 2915433616179200:33 elapsed, 2792
> items checked)
> wanted offset 29154336178176, found 29154336161792
> cache appears valid but isn't 29154334474240
> there is no free space entry for 30899331825664-30899331842048d, 3585
> items checked)
> there is no free space entry for 30899331825664-30900272234496
> cache appears valid but isn't 30899198492672
> there is no free space entry for 32134011568128-32134011584512d, 4474
> items checked)
> there is no free space entry for 32134011568128-32135075332096
> cache appears valid but isn't 32134001590272
> wanted offset 33148689629184, found 3314868961280000:59 elapsed, 4963
> items checked)
> wanted offset 33148689629184, found 33148689612800
> cache appears valid but isn't 33148687613952
> there is no free space entry for 34611225755648-34611225772032d, 6036
> items checked)
> there is no free space entry for 34611225755648-34612197720064
> cache appears valid but isn't 34611123978240
> there is no free space entry for 37374972723200-37374972739584d, 8051
> items checked)
> there is no free space entry for 37374972723200-37376042729472
> cache appears valid but isn't 37374968987648
> there is no free space entry for 37484494651392-37484494667776d, 8172
> items checked)
> there is no free space entry for 37484494651392-37485564395520
> cache appears valid but isn't 37484490653696
> wanted bytes 49152, found 32768 for off 377572293017606 elapsed, 8381
> items checked)
> wanted bytes 1065517056, found 32768 for off 37757229301760
> cache appears valid but isn't 37757221076992
> there is no free space entry for 38414356250624-38414356267008d, 9004
> items checked)
> there is no free space entry for 38414356250624-38415424815104
> cache appears valid but isn't 38414351073280
> there is no free space entry for 41509957402624-41509957419008d, 11792
> items checked)
> there is no free space entry for 41509957402624-41511022493696
> cache appears valid but isn't 41509948751872
> there is no free space entry for 42293815459840-42293815492608d, 12469
> items checked)
> there is no free space entry for 42293815459840-42294887579648
> cache appears valid but isn't 42293813837824

All free space cache related problems.

You can just clear all v1 cache:

$ btrfs check --clear-space-cache v1 <dev>

Then it's also a good time to migrate to v2 cache, which is safer and
faster.

# mount <dev> -o space_cache=v2 <mnt>

Thanks,
Qu
> [3/7] checking free space cache                (0:04:18 elapsed, 14910
> items checked)
> [4/7] checking fs roots                        (0:00:26 elapsed,
> 108894 items checked)
> [5/7] checking csums (without verifying data)  (0:00:03 elapsed,
> 12350321 items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 4
> items checked)
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 15729059287040 bytes used, error(s) found
> total csum bytes: 15313288548
> total tree bytes: 18286706688
> total fs tree bytes: 1791819776
> total extent tree bytes: 229097472
> btree space waste bytes: 1018811836
> file data blocks allocated: 51587230765056
>   referenced 15627926974464
>
> On Tue, Sep 7, 2021 at 11:17 AM Robert Wyrick <rob@wyrick.org> wrote:
>>
>> Looks like I spoke too soon.
>>
>> I can now mount the FS readonly.
>>
>> I see this error in dmesg:
>> [58995.896369] CPU: 10 PID: 83845 Comm: btrfs-transacti Tainted: P
>>        OE     5.11.0-27-generic #29~20.04.1-Ubuntu
>> [58995.896373] Hardware name: System manufacturer System Product
>> Name/PRIME X370-PRO, BIOS 0515 03/30/2017
>> [58995.896376] RIP: 0010:btrfs_run_delayed_refs+0x1af/0x200 [btrfs]
>> [58995.896422] Code: 8b 55 50 f0 48 0f ba aa 48 0a 00 00 03 72 20 83
>> f8 fb 74 3c 83 f8 e2 74 37 89 c6 48 c7 c7 50 7e 77 c0 89 45 d0 e8 96
>> d4 4d d3 <0f> 0b 8b 45 d0 89 c1 ba 4c 08 00 00 4c 89 ef 89 45 d0 48 c7
>> c6 c0
>> [58995.896425] RSP: 0018:ffffb89a4a0dfdf8 EFLAGS: 00010282
>> [58995.896428] RAX: 0000000000000000 RBX: ffffffffffffffff RCX: 0000000000000027
>> [58995.896430] RDX: 0000000000000027 RSI: 00000000ffffdfff RDI: ffff960fdf098ac8
>> [58995.896432] RBP: ffffb89a4a0dfe40 R08: ffff960fdf098ac0 R09: ffffb89a4a0dfbb8
>> [58995.896434] R10: 0000000000000001 R11: 0000000000000001 R12: ffff96036a7d5378
>> [58995.896435] R13: ffff960115028888 R14: ffff96036a7d5200 R15: 0000000000000000
>> [58995.896437] FS:  0000000000000000(0000) GS:ffff960fdf080000(0000)
>> knlGS:0000000000000000
>> [58995.896439] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [58995.896441] CR2: 00005616936891e8 CR3: 000000010fe46000 CR4: 00000000003506e0
>> [58995.896444] Call Trace:
>> [58995.896450]  btrfs_commit_transaction+0x2c3/0xa80 [btrfs]
>> [58995.896500]  ? start_transaction+0xd5/0x590 [btrfs]
>> [58995.896549]  transaction_kthread+0x138/0x1b0 [btrfs]
>> [58995.896596]  kthread+0x114/0x150
>> [58995.896604]  ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
>> [58995.896649]  ? kthread_park+0x90/0x90
>> [58995.896653]  ret_from_fork+0x22/0x30
>> [58995.896661] ---[ end trace c8ba04bdf2113cae ]---
>> [58995.896664] BTRFS: error (device sdf) in
>> btrfs_run_delayed_refs:2124: errno=-17 Object already exists
>> [58995.896669] BTRFS info (device sdf): forced readonly
>> [58995.896672] BTRFS warning (device sdf): Skipping commit of aborted
>> transaction.
>> [58995.896674] BTRFS: error (device sdf) in cleanup_transaction:1939:
>> errno=-17 Object already exists
>>
>> Read-only is better than nothing, but what would be my next steps?
>>
>> On Tue, Sep 7, 2021 at 11:02 AM Robert Wyrick <rob@wyrick.org> wrote:
>>>
>>> Ran a repair:
>>>
>>> $ sudo ./btrfs check --repair -p /dev/sda  # I did NOT make install,
>>> just ran from the compiled directory
>>> enabling repair mode
>>> WARNING:
>>>
>>> Do not use --repair unless you are advised to do so by a developer
>>> or an experienced user, and then only after having accepted that no
>>> fsck can successfully repair all types of filesystem corruption. Eg.
>>> some software or hardware bugs can fatally damage a volume.
>>> The operation will start in 10 seconds.
>>> Use Ctrl-C to stop it.
>>> 10 9 8 7 6 5 4 3 2 1
>>> Starting repair.
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/sda
>>> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
>>> [1/7] checking root items                      (0:00:59 elapsed,
>>> 2649102 items checked)
>>> Fixed 0 roots.
>>> Reset extent item (38179182174208) generation to 4057084elapsed,
>>> 1116143 items checked)
>>> No device size related problem found           (0:02:22 elapsed,
>>> 1116143 items checked)
>>> [2/7] checking extents                         (0:02:23 elapsed,
>>> 1116143 items checked)
>>> cache and super generation don't match, space cache will be invalidated
>>> [3/7] checking free space cache                (0:00:00 elapsed)
>>> Deleting bad dir index [8348950,96,3] root 259 (0:00:25 elapsed,
>>> 106695 items checked)
>>> repairing missing dir index item for inode 834922400:26 elapsed,
>>> 108893 items checked)
>>> [4/7] checking fs roots                        (0:01:04 elapsed,
>>> 217787 items checked)
>>> [5/7] checking csums (without verifying data)  (0:00:04 elapsed,
>>> 12350321 items checked)
>>> [6/7] checking root refs                       (0:00:00 elapsed, 4
>>> items checked)
>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>> found 15729059057664 bytes used, no error found
>>> total csum bytes: 15313288548
>>> total tree bytes: 18286739456
>>> total fs tree bytes: 1791819776
>>> total extent tree bytes: 229130240
>>> btree space waste bytes: 1018844959
>>> file data blocks allocated: 51587230502912
>>>   referenced 15627926712320
>>>
>>> I can now mount the filesystem successfully!  Thank you for your help.
>>>
>>> I do have some additional questions if you don't mind...
>>> I am already using RAID 1 to handle single disk outages.  I assume
>>> things could have gone much worse and I could have lost the whole
>>> filesystem.  Aside from backups (I know, I know), is there anything
>>> else I can do to prevent such issues or make them easier to recover
>>> from?  Could this problem have been avoided/detected earlier?  This
>>> wasn't a disk failure and according to memtest86+, it wasn't due to
>>> bad memory either....  I don't run scrubs very often.  Should I?  I
>>> guess the more general question is:  What are the best practices for
>>> maintaining a healthy btrfs file system?
>>>
>>> Thanks again!
>>>
>>> On Mon, Sep 6, 2021 at 10:53 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2021/9/7 下午12:36, Robert Wyrick wrote:
>>>>> What exactly would i be disabling?  I don't know what zoned does.
>>>>
>>>> The zoned device support.
>>>>
>>>> If you don't have any host-managed zoned device, there is no reason you
>>>> would like to enable it.
>>>>
>>>> https://zonedstorage.io/introduction/
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>
>>>>> On Mon, Sep 6, 2021, 9:07 PM Anand Jain <anand.jain@oracle.com> wrote:
>>>>>>
>>>>>> On 07/09/2021 10:36, Robert Wyrick wrote:
>>>>>>> Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
>>>>>>>
>>>>>>> $ cat /etc/os-release
>>>>>>> NAME="Linux Mint"
>>>>>>> VERSION="20.2 (Uma)"
>>>>>>> ID=linuxmint
>>>>>>> ID_LIKE=ubuntu
>>>>>>> PRETTY_NAME="Linux Mint 20.2"
>>>>>>> VERSION_ID="20.2"
>>>>>>> HOME_URL="https://www.linuxmint.com/"
>>>>>>> SUPPORT_URL="https://forums.linuxmint.com/"
>>>>>>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
>>>>>>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
>>>>>>> VERSION_CODENAME=uma
>>>>>>> UBUNTU_CODENAME=focal
>>>>>>>
>>>>>>> $ uname -a
>>>>>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
>>>>>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>>>>>>
>>>>>>> $ ./configure
>>>>>>> checking for gcc... gcc
>>>>>>> checking whether the C compiler works... yes
>>>>>>> checking for C compiler default output file name... a.out
>>>>>>> checking for suffix of executables...
>>>>>>> checking whether we are cross compiling... no
>>>>>>> checking for suffix of object files... o
>>>>>>> checking whether we are using the GNU C compiler... yes
>>>>>>> checking whether gcc accepts -g... yes
>>>>>>> checking for gcc option to accept ISO C89... none needed
>>>>>>> checking how to run the C preprocessor... gcc -E
>>>>>>> checking for grep that handles long lines and -e... /bin/grep
>>>>>>> checking for egrep... /bin/grep -E
>>>>>>> checking for ANSI C header files... yes
>>>>>>> checking for sys/types.h... yes
>>>>>>> checking for sys/stat.h... yes
>>>>>>> checking for stdlib.h... yes
>>>>>>> checking for string.h... yes
>>>>>>> checking for memory.h... yes
>>>>>>> checking for strings.h... yes
>>>>>>> checking for inttypes.h... yes
>>>>>>> checking for stdint.h... yes
>>>>>>> checking for unistd.h... yes
>>>>>>> checking minix/config.h usability... no
>>>>>>> checking minix/config.h presence... no
>>>>>>> checking for minix/config.h... no
>>>>>>> checking whether it is safe to define __EXTENSIONS__... yes
>>>>>>> checking for gcc... (cached) gcc
>>>>>>> checking whether we are using the GNU C compiler... (cached) yes
>>>>>>> checking whether gcc accepts -g... (cached) yes
>>>>>>> checking for gcc option to accept ISO C89... (cached) none needed
>>>>>>> checking whether C compiler accepts -std=gnu90... yes
>>>>>>> checking build system type... x86_64-pc-linux-gnu
>>>>>>> checking host system type... x86_64-pc-linux-gnu
>>>>>>> checking for an ANSI C-conforming const... yes
>>>>>>> checking for working volatile... yes
>>>>>>> checking whether byte ordering is bigendian... no
>>>>>>> checking for special C compiler options needed for large files... no
>>>>>>> checking for _FILE_OFFSET_BITS value needed for large files... no
>>>>>>> checking for a BSD-compatible install... /usr/bin/install -c
>>>>>>> checking whether ln -s works... yes
>>>>>>> checking for ar... ar
>>>>>>> checking for rm... /bin/rm
>>>>>>> checking for rmdir... /bin/rmdir
>>>>>>> checking for openat... yes
>>>>>>> checking for reallocarray... yes
>>>>>>> checking for clock_gettime... yes
>>>>>>> checking linux/perf_event.h usability... yes
>>>>>>> checking linux/perf_event.h presence... yes
>>>>>>> checking for linux/perf_event.h... yes
>>>>>>> checking linux/hw_breakpoint.h usability... yes
>>>>>>> checking linux/hw_breakpoint.h presence... yes
>>>>>>> checking for linux/hw_breakpoint.h... yes
>>>>>>> checking for pkg-config... /usr/bin/pkg-config
>>>>>>> checking pkg-config is at least version 0.9.0... yes
>>>>>>> checking execinfo.h usability... yes
>>>>>>> checking execinfo.h presence... yes
>>>>>>> checking for execinfo.h... yes
>>>>>>> checking for backtrace... yes
>>>>>>> checking for backtrace_symbols_fd... yes
>>>>>>> checking for xmlto... /usr/bin/xmlto
>>>>>>> checking for mv... /bin/mv
>>>>>>> checking for a sed that does not truncate output... /bin/sed
>>>>>>> checking for asciidoc... /usr/bin/asciidoc
>>>>>>> checking for asciidoctor... no
>>>>>>> checking for EXT2FS... yes
>>>>>>> checking for COM_ERR... yes
>>>>>>> checking for REISERFS... yes
>>>>>>> checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
>>>>>>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
>>>>>>> checking linux/blkzoned.h usability... yes
>>>>>>> checking linux/blkzoned.h presence... yes
>>>>>>> checking for linux/blkzoned.h... yes
>>>>>>> checking for struct blk_zone.capacity... no
>>>>>>> checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
>>>>>>
>>>>>>> configure: error: linux/blkzoned.h does not provide blk_zone.capacity
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>>
>>>>>>> Info on the file in question (linux/blkzoned.h):
>>>>>>>
>>>>>>> $ dpkg -S /usr/include/linux/blkzoned.h
>>>>>>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
>>>>>>>
>>>>>>> $ dpkg -l linux-libc-dev
>>>>>>> Desired=Unknown/Install/Remove/Purge/Hold
>>>>>>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
>>>>>>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>>>>>>> ||/ Name                 Version      Architecture Description
>>>>>>> +++-====================-============-============-====================================
>>>>>>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
>>>>>>> Headers for development
>>>>>>>
>>>>>>>
>>>>>>> So it appears that linux-libc-dev is way out-dated compared to my
>>>>>>> kernel.  I don't know how to update it, though... there doesn't appear
>>>>>>> to be a newer version available.
>>>>>>
>>>>>> You could disable the zoned.
>>>>>>
>>>>>>      ./configure --disable-zoned
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>

Re: Next steps in recovery?

[Robert Wyrick]

Anything specific to address the dmesg error?

On Tue, Sep 7, 2021 at 5:17 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/8 上午4:47, Robert Wyrick wrote:
> > Re-running check now shows:
> >
> > [1/7] checking root items                      (0:00:55 elapsed,
> > 2649102 items checked)
> > [2/7] checking extents                         (0:02:13 elapsed,
> > 1116141 items checked)
> > there is no free space entry for 18365358505984-18365358522368d, 130
> > items checked)
> > there is no free space entry for 18365358505984-18366416814080
> > cache appears valid but isn't 18365343072256
> > there is no free space entry for 19764429062144-19764429078528d, 348
> > items checked)
> > there is no free space entry for 19764429062144-19765502410752
> > cache appears valid but isn't 19764428668928
> > wanted bytes 49152, found 16384 for off 254016221675521 elapsed, 1534
> > items checked)
> > wanted bytes 1058373632, found 16384 for off 25401622167552
> > cache appears valid but isn't 25401606799360
> > there is no free space entry for 28659399229440-28659399245824d, 2636
> > items checked)
> > there is no free space entry for 28659399229440-28660413235200
> > cache appears valid but isn't 28659339493376
> > wanted offset 29154336178176, found 2915433616179200:33 elapsed, 2792
> > items checked)
> > wanted offset 29154336178176, found 29154336161792
> > cache appears valid but isn't 29154334474240
> > there is no free space entry for 30899331825664-30899331842048d, 3585
> > items checked)
> > there is no free space entry for 30899331825664-30900272234496
> > cache appears valid but isn't 30899198492672
> > there is no free space entry for 32134011568128-32134011584512d, 4474
> > items checked)
> > there is no free space entry for 32134011568128-32135075332096
> > cache appears valid but isn't 32134001590272
> > wanted offset 33148689629184, found 3314868961280000:59 elapsed, 4963
> > items checked)
> > wanted offset 33148689629184, found 33148689612800
> > cache appears valid but isn't 33148687613952
> > there is no free space entry for 34611225755648-34611225772032d, 6036
> > items checked)
> > there is no free space entry for 34611225755648-34612197720064
> > cache appears valid but isn't 34611123978240
> > there is no free space entry for 37374972723200-37374972739584d, 8051
> > items checked)
> > there is no free space entry for 37374972723200-37376042729472
> > cache appears valid but isn't 37374968987648
> > there is no free space entry for 37484494651392-37484494667776d, 8172
> > items checked)
> > there is no free space entry for 37484494651392-37485564395520
> > cache appears valid but isn't 37484490653696
> > wanted bytes 49152, found 32768 for off 377572293017606 elapsed, 8381
> > items checked)
> > wanted bytes 1065517056, found 32768 for off 37757229301760
> > cache appears valid but isn't 37757221076992
> > there is no free space entry for 38414356250624-38414356267008d, 9004
> > items checked)
> > there is no free space entry for 38414356250624-38415424815104
> > cache appears valid but isn't 38414351073280
> > there is no free space entry for 41509957402624-41509957419008d, 11792
> > items checked)
> > there is no free space entry for 41509957402624-41511022493696
> > cache appears valid but isn't 41509948751872
> > there is no free space entry for 42293815459840-42293815492608d, 12469
> > items checked)
> > there is no free space entry for 42293815459840-42294887579648
> > cache appears valid but isn't 42293813837824
>
> All free space cache related problems.
>
> You can just clear all v1 cache:
>
> $ btrfs check --clear-space-cache v1 <dev>
>
> Then it's also a good time to migrate to v2 cache, which is safer and
> faster.
>
> # mount <dev> -o space_cache=v2 <mnt>
>
> Thanks,
> Qu
> > [3/7] checking free space cache                (0:04:18 elapsed, 14910
> > items checked)
> > [4/7] checking fs roots                        (0:00:26 elapsed,
> > 108894 items checked)
> > [5/7] checking csums (without verifying data)  (0:00:03 elapsed,
> > 12350321 items checked)
> > [6/7] checking root refs                       (0:00:00 elapsed, 4
> > items checked)
> > [7/7] checking quota groups skipped (not enabled on this FS)
> > found 15729059287040 bytes used, error(s) found
> > total csum bytes: 15313288548
> > total tree bytes: 18286706688
> > total fs tree bytes: 1791819776
> > total extent tree bytes: 229097472
> > btree space waste bytes: 1018811836
> > file data blocks allocated: 51587230765056
> >   referenced 15627926974464
> >
> > On Tue, Sep 7, 2021 at 11:17 AM Robert Wyrick <rob@wyrick.org> wrote:
> >>
> >> Looks like I spoke too soon.
> >>
> >> I can now mount the FS readonly.
> >>
> >> I see this error in dmesg:
> >> [58995.896369] CPU: 10 PID: 83845 Comm: btrfs-transacti Tainted: P
> >>        OE     5.11.0-27-generic #29~20.04.1-Ubuntu
> >> [58995.896373] Hardware name: System manufacturer System Product
> >> Name/PRIME X370-PRO, BIOS 0515 03/30/2017
> >> [58995.896376] RIP: 0010:btrfs_run_delayed_refs+0x1af/0x200 [btrfs]
> >> [58995.896422] Code: 8b 55 50 f0 48 0f ba aa 48 0a 00 00 03 72 20 83
> >> f8 fb 74 3c 83 f8 e2 74 37 89 c6 48 c7 c7 50 7e 77 c0 89 45 d0 e8 96
> >> d4 4d d3 <0f> 0b 8b 45 d0 89 c1 ba 4c 08 00 00 4c 89 ef 89 45 d0 48 c7
> >> c6 c0
> >> [58995.896425] RSP: 0018:ffffb89a4a0dfdf8 EFLAGS: 00010282
> >> [58995.896428] RAX: 0000000000000000 RBX: ffffffffffffffff RCX: 0000000000000027
> >> [58995.896430] RDX: 0000000000000027 RSI: 00000000ffffdfff RDI: ffff960fdf098ac8
> >> [58995.896432] RBP: ffffb89a4a0dfe40 R08: ffff960fdf098ac0 R09: ffffb89a4a0dfbb8
> >> [58995.896434] R10: 0000000000000001 R11: 0000000000000001 R12: ffff96036a7d5378
> >> [58995.896435] R13: ffff960115028888 R14: ffff96036a7d5200 R15: 0000000000000000
> >> [58995.896437] FS:  0000000000000000(0000) GS:ffff960fdf080000(0000)
> >> knlGS:0000000000000000
> >> [58995.896439] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >> [58995.896441] CR2: 00005616936891e8 CR3: 000000010fe46000 CR4: 00000000003506e0
> >> [58995.896444] Call Trace:
> >> [58995.896450]  btrfs_commit_transaction+0x2c3/0xa80 [btrfs]
> >> [58995.896500]  ? start_transaction+0xd5/0x590 [btrfs]
> >> [58995.896549]  transaction_kthread+0x138/0x1b0 [btrfs]
> >> [58995.896596]  kthread+0x114/0x150
> >> [58995.896604]  ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
> >> [58995.896649]  ? kthread_park+0x90/0x90
> >> [58995.896653]  ret_from_fork+0x22/0x30
> >> [58995.896661] ---[ end trace c8ba04bdf2113cae ]---
> >> [58995.896664] BTRFS: error (device sdf) in
> >> btrfs_run_delayed_refs:2124: errno=-17 Object already exists
> >> [58995.896669] BTRFS info (device sdf): forced readonly
> >> [58995.896672] BTRFS warning (device sdf): Skipping commit of aborted
> >> transaction.
> >> [58995.896674] BTRFS: error (device sdf) in cleanup_transaction:1939:
> >> errno=-17 Object already exists
> >>
> >> Read-only is better than nothing, but what would be my next steps?
> >>
> >> On Tue, Sep 7, 2021 at 11:02 AM Robert Wyrick <rob@wyrick.org> wrote:
> >>>
> >>> Ran a repair:
> >>>
> >>> $ sudo ./btrfs check --repair -p /dev/sda  # I did NOT make install,
> >>> just ran from the compiled directory
> >>> enabling repair mode
> >>> WARNING:
> >>>
> >>> Do not use --repair unless you are advised to do so by a developer
> >>> or an experienced user, and then only after having accepted that no
> >>> fsck can successfully repair all types of filesystem corruption. Eg.
> >>> some software or hardware bugs can fatally damage a volume.
> >>> The operation will start in 10 seconds.
> >>> Use Ctrl-C to stop it.
> >>> 10 9 8 7 6 5 4 3 2 1
> >>> Starting repair.
> >>> Opening filesystem to check...
> >>> Checking filesystem on /dev/sda
> >>> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
> >>> [1/7] checking root items                      (0:00:59 elapsed,
> >>> 2649102 items checked)
> >>> Fixed 0 roots.
> >>> Reset extent item (38179182174208) generation to 4057084elapsed,
> >>> 1116143 items checked)
> >>> No device size related problem found           (0:02:22 elapsed,
> >>> 1116143 items checked)
> >>> [2/7] checking extents                         (0:02:23 elapsed,
> >>> 1116143 items checked)
> >>> cache and super generation don't match, space cache will be invalidated
> >>> [3/7] checking free space cache                (0:00:00 elapsed)
> >>> Deleting bad dir index [8348950,96,3] root 259 (0:00:25 elapsed,
> >>> 106695 items checked)
> >>> repairing missing dir index item for inode 834922400:26 elapsed,
> >>> 108893 items checked)
> >>> [4/7] checking fs roots                        (0:01:04 elapsed,
> >>> 217787 items checked)
> >>> [5/7] checking csums (without verifying data)  (0:00:04 elapsed,
> >>> 12350321 items checked)
> >>> [6/7] checking root refs                       (0:00:00 elapsed, 4
> >>> items checked)
> >>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>> found 15729059057664 bytes used, no error found
> >>> total csum bytes: 15313288548
> >>> total tree bytes: 18286739456
> >>> total fs tree bytes: 1791819776
> >>> total extent tree bytes: 229130240
> >>> btree space waste bytes: 1018844959
> >>> file data blocks allocated: 51587230502912
> >>>   referenced 15627926712320
> >>>
> >>> I can now mount the filesystem successfully!  Thank you for your help.
> >>>
> >>> I do have some additional questions if you don't mind...
> >>> I am already using RAID 1 to handle single disk outages.  I assume
> >>> things could have gone much worse and I could have lost the whole
> >>> filesystem.  Aside from backups (I know, I know), is there anything
> >>> else I can do to prevent such issues or make them easier to recover
> >>> from?  Could this problem have been avoided/detected earlier?  This
> >>> wasn't a disk failure and according to memtest86+, it wasn't due to
> >>> bad memory either....  I don't run scrubs very often.  Should I?  I
> >>> guess the more general question is:  What are the best practices for
> >>> maintaining a healthy btrfs file system?
> >>>
> >>> Thanks again!
> >>>
> >>> On Mon, Sep 6, 2021 at 10:53 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2021/9/7 下午12:36, Robert Wyrick wrote:
> >>>>> What exactly would i be disabling?  I don't know what zoned does.
> >>>>
> >>>> The zoned device support.
> >>>>
> >>>> If you don't have any host-managed zoned device, there is no reason you
> >>>> would like to enable it.
> >>>>
> >>>> https://zonedstorage.io/introduction/
> >>>>
> >>>> Thanks,
> >>>> Qu
> >>>>
> >>>>>
> >>>>> On Mon, Sep 6, 2021, 9:07 PM Anand Jain <anand.jain@oracle.com> wrote:
> >>>>>>
> >>>>>> On 07/09/2021 10:36, Robert Wyrick wrote:
> >>>>>>> Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
> >>>>>>>
> >>>>>>> $ cat /etc/os-release
> >>>>>>> NAME="Linux Mint"
> >>>>>>> VERSION="20.2 (Uma)"
> >>>>>>> ID=linuxmint
> >>>>>>> ID_LIKE=ubuntu
> >>>>>>> PRETTY_NAME="Linux Mint 20.2"
> >>>>>>> VERSION_ID="20.2"
> >>>>>>> HOME_URL="https://www.linuxmint.com/"
> >>>>>>> SUPPORT_URL="https://forums.linuxmint.com/"
> >>>>>>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
> >>>>>>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
> >>>>>>> VERSION_CODENAME=uma
> >>>>>>> UBUNTU_CODENAME=focal
> >>>>>>>
> >>>>>>> $ uname -a
> >>>>>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
> >>>>>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> >>>>>>>
> >>>>>>> $ ./configure
> >>>>>>> checking for gcc... gcc
> >>>>>>> checking whether the C compiler works... yes
> >>>>>>> checking for C compiler default output file name... a.out
> >>>>>>> checking for suffix of executables...
> >>>>>>> checking whether we are cross compiling... no
> >>>>>>> checking for suffix of object files... o
> >>>>>>> checking whether we are using the GNU C compiler... yes
> >>>>>>> checking whether gcc accepts -g... yes
> >>>>>>> checking for gcc option to accept ISO C89... none needed
> >>>>>>> checking how to run the C preprocessor... gcc -E
> >>>>>>> checking for grep that handles long lines and -e... /bin/grep
> >>>>>>> checking for egrep... /bin/grep -E
> >>>>>>> checking for ANSI C header files... yes
> >>>>>>> checking for sys/types.h... yes
> >>>>>>> checking for sys/stat.h... yes
> >>>>>>> checking for stdlib.h... yes
> >>>>>>> checking for string.h... yes
> >>>>>>> checking for memory.h... yes
> >>>>>>> checking for strings.h... yes
> >>>>>>> checking for inttypes.h... yes
> >>>>>>> checking for stdint.h... yes
> >>>>>>> checking for unistd.h... yes
> >>>>>>> checking minix/config.h usability... no
> >>>>>>> checking minix/config.h presence... no
> >>>>>>> checking for minix/config.h... no
> >>>>>>> checking whether it is safe to define __EXTENSIONS__... yes
> >>>>>>> checking for gcc... (cached) gcc
> >>>>>>> checking whether we are using the GNU C compiler... (cached) yes
> >>>>>>> checking whether gcc accepts -g... (cached) yes
> >>>>>>> checking for gcc option to accept ISO C89... (cached) none needed
> >>>>>>> checking whether C compiler accepts -std=gnu90... yes
> >>>>>>> checking build system type... x86_64-pc-linux-gnu
> >>>>>>> checking host system type... x86_64-pc-linux-gnu
> >>>>>>> checking for an ANSI C-conforming const... yes
> >>>>>>> checking for working volatile... yes
> >>>>>>> checking whether byte ordering is bigendian... no
> >>>>>>> checking for special C compiler options needed for large files... no
> >>>>>>> checking for _FILE_OFFSET_BITS value needed for large files... no
> >>>>>>> checking for a BSD-compatible install... /usr/bin/install -c
> >>>>>>> checking whether ln -s works... yes
> >>>>>>> checking for ar... ar
> >>>>>>> checking for rm... /bin/rm
> >>>>>>> checking for rmdir... /bin/rmdir
> >>>>>>> checking for openat... yes
> >>>>>>> checking for reallocarray... yes
> >>>>>>> checking for clock_gettime... yes
> >>>>>>> checking linux/perf_event.h usability... yes
> >>>>>>> checking linux/perf_event.h presence... yes
> >>>>>>> checking for linux/perf_event.h... yes
> >>>>>>> checking linux/hw_breakpoint.h usability... yes
> >>>>>>> checking linux/hw_breakpoint.h presence... yes
> >>>>>>> checking for linux/hw_breakpoint.h... yes
> >>>>>>> checking for pkg-config... /usr/bin/pkg-config
> >>>>>>> checking pkg-config is at least version 0.9.0... yes
> >>>>>>> checking execinfo.h usability... yes
> >>>>>>> checking execinfo.h presence... yes
> >>>>>>> checking for execinfo.h... yes
> >>>>>>> checking for backtrace... yes
> >>>>>>> checking for backtrace_symbols_fd... yes
> >>>>>>> checking for xmlto... /usr/bin/xmlto
> >>>>>>> checking for mv... /bin/mv
> >>>>>>> checking for a sed that does not truncate output... /bin/sed
> >>>>>>> checking for asciidoc... /usr/bin/asciidoc
> >>>>>>> checking for asciidoctor... no
> >>>>>>> checking for EXT2FS... yes
> >>>>>>> checking for COM_ERR... yes
> >>>>>>> checking for REISERFS... yes
> >>>>>>> checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
> >>>>>>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
> >>>>>>> checking linux/blkzoned.h usability... yes
> >>>>>>> checking linux/blkzoned.h presence... yes
> >>>>>>> checking for linux/blkzoned.h... yes
> >>>>>>> checking for struct blk_zone.capacity... no
> >>>>>>> checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
> >>>>>>
> >>>>>>> configure: error: linux/blkzoned.h does not provide blk_zone.capacity
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> ---
> >>>>>>>
> >>>>>>> Info on the file in question (linux/blkzoned.h):
> >>>>>>>
> >>>>>>> $ dpkg -S /usr/include/linux/blkzoned.h
> >>>>>>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
> >>>>>>>
> >>>>>>> $ dpkg -l linux-libc-dev
> >>>>>>> Desired=Unknown/Install/Remove/Purge/Hold
> >>>>>>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> >>>>>>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> >>>>>>> ||/ Name                 Version      Architecture Description
> >>>>>>> +++-====================-============-============-====================================
> >>>>>>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
> >>>>>>> Headers for development
> >>>>>>>
> >>>>>>>
> >>>>>>> So it appears that linux-libc-dev is way out-dated compared to my
> >>>>>>> kernel.  I don't know how to update it, though... there doesn't appear
> >>>>>>> to be a newer version available.
> >>>>>>
> >>>>>> You could disable the zoned.
> >>>>>>
> >>>>>>      ./configure --disable-zoned
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>

Re: Next steps in recovery?

[Qu Wenruo]

On 2021/9/8 上午7:20, Robert Wyrick wrote:
> Anything specific to address the dmesg error?

The dmesg seems to be caused by the corrupted free space cache.

Thus as long as it doesn't cause long last problems (aka, committing bad 
data to disk), after clearing v1 space cache, it should be safe.

But to be extra safe, you'd better run btrfs check again after v1 space 
cache clearing to be sure.

Thanks,
Qu
> 
> On Tue, Sep 7, 2021 at 5:17 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2021/9/8 上午4:47, Robert Wyrick wrote:
>>> Re-running check now shows:
>>>
>>> [1/7] checking root items                      (0:00:55 elapsed,
>>> 2649102 items checked)
>>> [2/7] checking extents                         (0:02:13 elapsed,
>>> 1116141 items checked)
>>> there is no free space entry for 18365358505984-18365358522368d, 130
>>> items checked)
>>> there is no free space entry for 18365358505984-18366416814080
>>> cache appears valid but isn't 18365343072256
>>> there is no free space entry for 19764429062144-19764429078528d, 348
>>> items checked)
>>> there is no free space entry for 19764429062144-19765502410752
>>> cache appears valid but isn't 19764428668928
>>> wanted bytes 49152, found 16384 for off 254016221675521 elapsed, 1534
>>> items checked)
>>> wanted bytes 1058373632, found 16384 for off 25401622167552
>>> cache appears valid but isn't 25401606799360
>>> there is no free space entry for 28659399229440-28659399245824d, 2636
>>> items checked)
>>> there is no free space entry for 28659399229440-28660413235200
>>> cache appears valid but isn't 28659339493376
>>> wanted offset 29154336178176, found 2915433616179200:33 elapsed, 2792
>>> items checked)
>>> wanted offset 29154336178176, found 29154336161792
>>> cache appears valid but isn't 29154334474240
>>> there is no free space entry for 30899331825664-30899331842048d, 3585
>>> items checked)
>>> there is no free space entry for 30899331825664-30900272234496
>>> cache appears valid but isn't 30899198492672
>>> there is no free space entry for 32134011568128-32134011584512d, 4474
>>> items checked)
>>> there is no free space entry for 32134011568128-32135075332096
>>> cache appears valid but isn't 32134001590272
>>> wanted offset 33148689629184, found 3314868961280000:59 elapsed, 4963
>>> items checked)
>>> wanted offset 33148689629184, found 33148689612800
>>> cache appears valid but isn't 33148687613952
>>> there is no free space entry for 34611225755648-34611225772032d, 6036
>>> items checked)
>>> there is no free space entry for 34611225755648-34612197720064
>>> cache appears valid but isn't 34611123978240
>>> there is no free space entry for 37374972723200-37374972739584d, 8051
>>> items checked)
>>> there is no free space entry for 37374972723200-37376042729472
>>> cache appears valid but isn't 37374968987648
>>> there is no free space entry for 37484494651392-37484494667776d, 8172
>>> items checked)
>>> there is no free space entry for 37484494651392-37485564395520
>>> cache appears valid but isn't 37484490653696
>>> wanted bytes 49152, found 32768 for off 377572293017606 elapsed, 8381
>>> items checked)
>>> wanted bytes 1065517056, found 32768 for off 37757229301760
>>> cache appears valid but isn't 37757221076992
>>> there is no free space entry for 38414356250624-38414356267008d, 9004
>>> items checked)
>>> there is no free space entry for 38414356250624-38415424815104
>>> cache appears valid but isn't 38414351073280
>>> there is no free space entry for 41509957402624-41509957419008d, 11792
>>> items checked)
>>> there is no free space entry for 41509957402624-41511022493696
>>> cache appears valid but isn't 41509948751872
>>> there is no free space entry for 42293815459840-42293815492608d, 12469
>>> items checked)
>>> there is no free space entry for 42293815459840-42294887579648
>>> cache appears valid but isn't 42293813837824
>>
>> All free space cache related problems.
>>
>> You can just clear all v1 cache:
>>
>> $ btrfs check --clear-space-cache v1 <dev>
>>
>> Then it's also a good time to migrate to v2 cache, which is safer and
>> faster.
>>
>> # mount <dev> -o space_cache=v2 <mnt>
>>
>> Thanks,
>> Qu
>>> [3/7] checking free space cache                (0:04:18 elapsed, 14910
>>> items checked)
>>> [4/7] checking fs roots                        (0:00:26 elapsed,
>>> 108894 items checked)
>>> [5/7] checking csums (without verifying data)  (0:00:03 elapsed,
>>> 12350321 items checked)
>>> [6/7] checking root refs                       (0:00:00 elapsed, 4
>>> items checked)
>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>> found 15729059287040 bytes used, error(s) found
>>> total csum bytes: 15313288548
>>> total tree bytes: 18286706688
>>> total fs tree bytes: 1791819776
>>> total extent tree bytes: 229097472
>>> btree space waste bytes: 1018811836
>>> file data blocks allocated: 51587230765056
>>>    referenced 15627926974464
>>>
>>> On Tue, Sep 7, 2021 at 11:17 AM Robert Wyrick <rob@wyrick.org> wrote:
>>>>
>>>> Looks like I spoke too soon.
>>>>
>>>> I can now mount the FS readonly.
>>>>
>>>> I see this error in dmesg:
>>>> [58995.896369] CPU: 10 PID: 83845 Comm: btrfs-transacti Tainted: P
>>>>         OE     5.11.0-27-generic #29~20.04.1-Ubuntu
>>>> [58995.896373] Hardware name: System manufacturer System Product
>>>> Name/PRIME X370-PRO, BIOS 0515 03/30/2017
>>>> [58995.896376] RIP: 0010:btrfs_run_delayed_refs+0x1af/0x200 [btrfs]
>>>> [58995.896422] Code: 8b 55 50 f0 48 0f ba aa 48 0a 00 00 03 72 20 83
>>>> f8 fb 74 3c 83 f8 e2 74 37 89 c6 48 c7 c7 50 7e 77 c0 89 45 d0 e8 96
>>>> d4 4d d3 <0f> 0b 8b 45 d0 89 c1 ba 4c 08 00 00 4c 89 ef 89 45 d0 48 c7
>>>> c6 c0
>>>> [58995.896425] RSP: 0018:ffffb89a4a0dfdf8 EFLAGS: 00010282
>>>> [58995.896428] RAX: 0000000000000000 RBX: ffffffffffffffff RCX: 0000000000000027
>>>> [58995.896430] RDX: 0000000000000027 RSI: 00000000ffffdfff RDI: ffff960fdf098ac8
>>>> [58995.896432] RBP: ffffb89a4a0dfe40 R08: ffff960fdf098ac0 R09: ffffb89a4a0dfbb8
>>>> [58995.896434] R10: 0000000000000001 R11: 0000000000000001 R12: ffff96036a7d5378
>>>> [58995.896435] R13: ffff960115028888 R14: ffff96036a7d5200 R15: 0000000000000000
>>>> [58995.896437] FS:  0000000000000000(0000) GS:ffff960fdf080000(0000)
>>>> knlGS:0000000000000000
>>>> [58995.896439] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [58995.896441] CR2: 00005616936891e8 CR3: 000000010fe46000 CR4: 00000000003506e0
>>>> [58995.896444] Call Trace:
>>>> [58995.896450]  btrfs_commit_transaction+0x2c3/0xa80 [btrfs]
>>>> [58995.896500]  ? start_transaction+0xd5/0x590 [btrfs]
>>>> [58995.896549]  transaction_kthread+0x138/0x1b0 [btrfs]
>>>> [58995.896596]  kthread+0x114/0x150
>>>> [58995.896604]  ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
>>>> [58995.896649]  ? kthread_park+0x90/0x90
>>>> [58995.896653]  ret_from_fork+0x22/0x30
>>>> [58995.896661] ---[ end trace c8ba04bdf2113cae ]---
>>>> [58995.896664] BTRFS: error (device sdf) in
>>>> btrfs_run_delayed_refs:2124: errno=-17 Object already exists
>>>> [58995.896669] BTRFS info (device sdf): forced readonly
>>>> [58995.896672] BTRFS warning (device sdf): Skipping commit of aborted
>>>> transaction.
>>>> [58995.896674] BTRFS: error (device sdf) in cleanup_transaction:1939:
>>>> errno=-17 Object already exists
>>>>
>>>> Read-only is better than nothing, but what would be my next steps?
>>>>
>>>> On Tue, Sep 7, 2021 at 11:02 AM Robert Wyrick <rob@wyrick.org> wrote:
>>>>>
>>>>> Ran a repair:
>>>>>
>>>>> $ sudo ./btrfs check --repair -p /dev/sda  # I did NOT make install,
>>>>> just ran from the compiled directory
>>>>> enabling repair mode
>>>>> WARNING:
>>>>>
>>>>> Do not use --repair unless you are advised to do so by a developer
>>>>> or an experienced user, and then only after having accepted that no
>>>>> fsck can successfully repair all types of filesystem corruption. Eg.
>>>>> some software or hardware bugs can fatally damage a volume.
>>>>> The operation will start in 10 seconds.
>>>>> Use Ctrl-C to stop it.
>>>>> 10 9 8 7 6 5 4 3 2 1
>>>>> Starting repair.
>>>>> Opening filesystem to check...
>>>>> Checking filesystem on /dev/sda
>>>>> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
>>>>> [1/7] checking root items                      (0:00:59 elapsed,
>>>>> 2649102 items checked)
>>>>> Fixed 0 roots.
>>>>> Reset extent item (38179182174208) generation to 4057084elapsed,
>>>>> 1116143 items checked)
>>>>> No device size related problem found           (0:02:22 elapsed,
>>>>> 1116143 items checked)
>>>>> [2/7] checking extents                         (0:02:23 elapsed,
>>>>> 1116143 items checked)
>>>>> cache and super generation don't match, space cache will be invalidated
>>>>> [3/7] checking free space cache                (0:00:00 elapsed)
>>>>> Deleting bad dir index [8348950,96,3] root 259 (0:00:25 elapsed,
>>>>> 106695 items checked)
>>>>> repairing missing dir index item for inode 834922400:26 elapsed,
>>>>> 108893 items checked)
>>>>> [4/7] checking fs roots                        (0:01:04 elapsed,
>>>>> 217787 items checked)
>>>>> [5/7] checking csums (without verifying data)  (0:00:04 elapsed,
>>>>> 12350321 items checked)
>>>>> [6/7] checking root refs                       (0:00:00 elapsed, 4
>>>>> items checked)
>>>>> [7/7] checking quota groups skipped (not enabled on this FS)
>>>>> found 15729059057664 bytes used, no error found
>>>>> total csum bytes: 15313288548
>>>>> total tree bytes: 18286739456
>>>>> total fs tree bytes: 1791819776
>>>>> total extent tree bytes: 229130240
>>>>> btree space waste bytes: 1018844959
>>>>> file data blocks allocated: 51587230502912
>>>>>    referenced 15627926712320
>>>>>
>>>>> I can now mount the filesystem successfully!  Thank you for your help.
>>>>>
>>>>> I do have some additional questions if you don't mind...
>>>>> I am already using RAID 1 to handle single disk outages.  I assume
>>>>> things could have gone much worse and I could have lost the whole
>>>>> filesystem.  Aside from backups (I know, I know), is there anything
>>>>> else I can do to prevent such issues or make them easier to recover
>>>>> from?  Could this problem have been avoided/detected earlier?  This
>>>>> wasn't a disk failure and according to memtest86+, it wasn't due to
>>>>> bad memory either....  I don't run scrubs very often.  Should I?  I
>>>>> guess the more general question is:  What are the best practices for
>>>>> maintaining a healthy btrfs file system?
>>>>>
>>>>> Thanks again!
>>>>>
>>>>> On Mon, Sep 6, 2021 at 10:53 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2021/9/7 下午12:36, Robert Wyrick wrote:
>>>>>>> What exactly would i be disabling?  I don't know what zoned does.
>>>>>>
>>>>>> The zoned device support.
>>>>>>
>>>>>> If you don't have any host-managed zoned device, there is no reason you
>>>>>> would like to enable it.
>>>>>>
>>>>>> https://zonedstorage.io/introduction/
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 6, 2021, 9:07 PM Anand Jain <anand.jain@oracle.com> wrote:
>>>>>>>>
>>>>>>>> On 07/09/2021 10:36, Robert Wyrick wrote:
>>>>>>>>> Trying to build latest btrfs-progs.  I'm seeing errors in the configure script.
>>>>>>>>>
>>>>>>>>> $ cat /etc/os-release
>>>>>>>>> NAME="Linux Mint"
>>>>>>>>> VERSION="20.2 (Uma)"
>>>>>>>>> ID=linuxmint
>>>>>>>>> ID_LIKE=ubuntu
>>>>>>>>> PRETTY_NAME="Linux Mint 20.2"
>>>>>>>>> VERSION_ID="20.2"
>>>>>>>>> HOME_URL="https://www.linuxmint.com/"
>>>>>>>>> SUPPORT_URL="https://forums.linuxmint.com/"
>>>>>>>>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
>>>>>>>>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
>>>>>>>>> VERSION_CODENAME=uma
>>>>>>>>> UBUNTU_CODENAME=focal
>>>>>>>>>
>>>>>>>>> $ uname -a
>>>>>>>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11
>>>>>>>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>>>>>>>>
>>>>>>>>> $ ./configure
>>>>>>>>> checking for gcc... gcc
>>>>>>>>> checking whether the C compiler works... yes
>>>>>>>>> checking for C compiler default output file name... a.out
>>>>>>>>> checking for suffix of executables...
>>>>>>>>> checking whether we are cross compiling... no
>>>>>>>>> checking for suffix of object files... o
>>>>>>>>> checking whether we are using the GNU C compiler... yes
>>>>>>>>> checking whether gcc accepts -g... yes
>>>>>>>>> checking for gcc option to accept ISO C89... none needed
>>>>>>>>> checking how to run the C preprocessor... gcc -E
>>>>>>>>> checking for grep that handles long lines and -e... /bin/grep
>>>>>>>>> checking for egrep... /bin/grep -E
>>>>>>>>> checking for ANSI C header files... yes
>>>>>>>>> checking for sys/types.h... yes
>>>>>>>>> checking for sys/stat.h... yes
>>>>>>>>> checking for stdlib.h... yes
>>>>>>>>> checking for string.h... yes
>>>>>>>>> checking for memory.h... yes
>>>>>>>>> checking for strings.h... yes
>>>>>>>>> checking for inttypes.h... yes
>>>>>>>>> checking for stdint.h... yes
>>>>>>>>> checking for unistd.h... yes
>>>>>>>>> checking minix/config.h usability... no
>>>>>>>>> checking minix/config.h presence... no
>>>>>>>>> checking for minix/config.h... no
>>>>>>>>> checking whether it is safe to define __EXTENSIONS__... yes
>>>>>>>>> checking for gcc... (cached) gcc
>>>>>>>>> checking whether we are using the GNU C compiler... (cached) yes
>>>>>>>>> checking whether gcc accepts -g... (cached) yes
>>>>>>>>> checking for gcc option to accept ISO C89... (cached) none needed
>>>>>>>>> checking whether C compiler accepts -std=gnu90... yes
>>>>>>>>> checking build system type... x86_64-pc-linux-gnu
>>>>>>>>> checking host system type... x86_64-pc-linux-gnu
>>>>>>>>> checking for an ANSI C-conforming const... yes
>>>>>>>>> checking for working volatile... yes
>>>>>>>>> checking whether byte ordering is bigendian... no
>>>>>>>>> checking for special C compiler options needed for large files... no
>>>>>>>>> checking for _FILE_OFFSET_BITS value needed for large files... no
>>>>>>>>> checking for a BSD-compatible install... /usr/bin/install -c
>>>>>>>>> checking whether ln -s works... yes
>>>>>>>>> checking for ar... ar
>>>>>>>>> checking for rm... /bin/rm
>>>>>>>>> checking for rmdir... /bin/rmdir
>>>>>>>>> checking for openat... yes
>>>>>>>>> checking for reallocarray... yes
>>>>>>>>> checking for clock_gettime... yes
>>>>>>>>> checking linux/perf_event.h usability... yes
>>>>>>>>> checking linux/perf_event.h presence... yes
>>>>>>>>> checking for linux/perf_event.h... yes
>>>>>>>>> checking linux/hw_breakpoint.h usability... yes
>>>>>>>>> checking linux/hw_breakpoint.h presence... yes
>>>>>>>>> checking for linux/hw_breakpoint.h... yes
>>>>>>>>> checking for pkg-config... /usr/bin/pkg-config
>>>>>>>>> checking pkg-config is at least version 0.9.0... yes
>>>>>>>>> checking execinfo.h usability... yes
>>>>>>>>> checking execinfo.h presence... yes
>>>>>>>>> checking for execinfo.h... yes
>>>>>>>>> checking for backtrace... yes
>>>>>>>>> checking for backtrace_symbols_fd... yes
>>>>>>>>> checking for xmlto... /usr/bin/xmlto
>>>>>>>>> checking for mv... /bin/mv
>>>>>>>>> checking for a sed that does not truncate output... /bin/sed
>>>>>>>>> checking for asciidoc... /usr/bin/asciidoc
>>>>>>>>> checking for asciidoctor... no
>>>>>>>>> checking for EXT2FS... yes
>>>>>>>>> checking for COM_ERR... yes
>>>>>>>>> checking for REISERFS... yes
>>>>>>>>> checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes
>>>>>>>>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes
>>>>>>>>> checking linux/blkzoned.h usability... yes
>>>>>>>>> checking linux/blkzoned.h presence... yes
>>>>>>>>> checking for linux/blkzoned.h... yes
>>>>>>>>> checking for struct blk_zone.capacity... no
>>>>>>>>> checking for BLKGETZONESZ defined in linux/blkzoned.h... yes
>>>>>>>>
>>>>>>>>> configure: error: linux/blkzoned.h does not provide blk_zone.capacity
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> Info on the file in question (linux/blkzoned.h):
>>>>>>>>>
>>>>>>>>> $ dpkg -S /usr/include/linux/blkzoned.h
>>>>>>>>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
>>>>>>>>>
>>>>>>>>> $ dpkg -l linux-libc-dev
>>>>>>>>> Desired=Unknown/Install/Remove/Purge/Hold
>>>>>>>>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
>>>>>>>>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>>>>>>>>> ||/ Name                 Version      Architecture Description
>>>>>>>>> +++-====================-============-============-====================================
>>>>>>>>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux Kernel
>>>>>>>>> Headers for development
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So it appears that linux-libc-dev is way out-dated compared to my
>>>>>>>>> kernel.  I don't know how to update it, though... there doesn't appear
>>>>>>>>> to be a newer version available.
>>>>>>>>
>>>>>>>> You could disable the zoned.
>>>>>>>>
>>>>>>>>       ./configure --disable-zoned
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
> 

Re: Next steps in recovery?

[Su Yue]

On Wed 08 Sep 2021 at 07:15, Qu Wenruo <quwenruo.btrfs@gmx.com> 
wrote:

> On 2021/9/8 上午1:02, Robert Wyrick wrote:
>> Ran a repair:
>>
>> $ sudo ./btrfs check --repair -p /dev/sda  # I did NOT make 
>> install,
>> just ran from the compiled directory
>> enabling repair mode
>> WARNING:
>>
>> Do not use --repair unless you are advised to do so by a 
>> developer
>> or an experienced user, and then only after having accepted 
>> that no
>> fsck can successfully repair all types of filesystem 
>> corruption. Eg.
>> some software or hardware bugs can fatally damage a volume.
>> The operation will start in 10 seconds.
>> Use Ctrl-C to stop it.
>> 10 9 8 7 6 5 4 3 2 1
>> Starting repair.
>> Opening filesystem to check...
>> Checking filesystem on /dev/sda
>> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
>> [1/7] checking root items                      (0:00:59 
>> elapsed,
>> 2649102 items checked)
>> Fixed 0 roots.
>> Reset extent item (38179182174208) generation to 
>> 4057084elapsed,
>> 1116143 items checked)
>> No device size related problem found           (0:02:22 
>> elapsed,
>> 1116143 items checked)
>> [2/7] checking extents                         (0:02:23 
>> elapsed,
>> 1116143 items checked)
>> cache and super generation don't match, space cache will be 
>> invalidated
>> [3/7] checking free space cache                (0:00:00 
>> elapsed)
>> Deleting bad dir index [8348950,96,3] root 259 (0:00:25 
>> elapsed,
>> 106695 items checked)
>> repairing missing dir index item for inode 834922400:26 
>> elapsed,
>> 108893 items checked)
>> [4/7] checking fs roots                        (0:01:04 
>> elapsed,
>> 217787 items checked)
>> [5/7] checking csums (without verifying data)  (0:00:04 
>> elapsed,
>> 12350321 items checked)
>> [6/7] checking root refs                       (0:00:00 
>> elapsed, 4
>> items checked)
>> [7/7] checking quota groups skipped (not enabled on this FS)
>> found 15729059057664 bytes used, no error found
>> total csum bytes: 15313288548
>> total tree bytes: 18286739456
>> total fs tree bytes: 1791819776
>> total extent tree bytes: 229130240
>> btree space waste bytes: 1018844959
>> file data blocks allocated: 51587230502912
>>   referenced 15627926712320
>>
>> I can now mount the filesystem successfully!  Thank you for 
>> your help.
>>
>> I do have some additional questions if you don't mind...
>> I am already using RAID 1 to handle single disk outages.
>
> One thing to note is, RAID is not perfect, not even close to 
> proper backup.
>
> RAID is really only suitable to handle disk failures, nothing 
> more than
> that.
>
> In a spectrum of backup, RAID is really just better than 
> nothing.
>
> In this particular case, all the corruption is from bitflips, 
> thus all
> copies are corrupted, no profile can save the day.
>
>>  I assume
>> things could have gone much worse and I could have lost the 
>> whole
>> filesystem.
>
> If you're using newer kernels all time, the kernel can detect 
> the extent
> generation problem before writing the corrupted data back to 
> disk, thus
> save the day.
>
>>  Aside from backups (I know, I know), is there anything
>> else I can do to prevent such issues or make them easier to 
>> recover
>> from?
>
> Newer kernel (v5.11 and newer) can prevent it.
> Although when such rejection happens, it will not feel that 
> comfort
> though, as it would mostly result the fs to go RO.
> But still way better than writing bad data onto disks.
>
>>  Could this problem have been avoided/detected earlier?
>
> Yes, newer kernel.
>
>>  This
>> wasn't a disk failure and according to memtest86+, it wasn't 
>> due to
>> bad memory either....
>
> I still don't believe, maybe you can try to run memtester (which 
> is ran
> in user space, and since we have kernel doing the page mapping, 
> it may
> expose a different workload on the memory controller than 
> memtest86+)
>
And testmem5 using config anta777 may help. It's widely used to 
test
memory stability after overclocking but runs on evil Windows 
though.
It won't take too much time. Two cicles test of 64G memory 
consumes
about 4~6 hours.

--
Su

Subject
> Since the extent generation corruption is a super obvious 
> bitflip.
>
>>  I don't run scrubs very often.  Should I?
>
> For newer kernels, the corruption can be rejected in first 
> place, thus
> the scrub is only going to detect problems already in the fs.
>
> For older kernels, scrub won't detect the problem anyw.
>
> So I guess you don't need that frequent scrub, but it's still 
> recommended.
> Maybe monthly?
>
>>  I
>> guess the more general question is:  What are the best 
>> practices for
>> maintaining a healthy btrfs file system?
>
> Well, healthy hardware, balanced kernel version between cutting 
> edge and
> stable.
> Personally I'm more towards cutting edge thought.
>
> Thanks,
> Qu
>
>>
>> Thanks again!
>>
>> On Mon, Sep 6, 2021 at 10:53 PM Qu Wenruo 
>> <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2021/9/7 下午12:36, Robert Wyrick wrote:
>>>> What exactly would i be disabling?  I don't know what zoned 
>>>> does.
>>>
>>> The zoned device support.
>>>
>>> If you don't have any host-managed zoned device, there is no 
>>> reason you
>>> would like to enable it.
>>>
>>> https://zonedstorage.io/introduction/
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>> On Mon, Sep 6, 2021, 9:07 PM Anand Jain 
>>>> <anand.jain@oracle.com> wrote:
>>>>>
>>>>> On 07/09/2021 10:36, Robert Wyrick wrote:
>>>>>> Trying to build latest btrfs-progs.  I'm seeing errors in 
>>>>>> the configure script.
>>>>>>
>>>>>> $ cat /etc/os-release
>>>>>> NAME="Linux Mint"
>>>>>> VERSION="20.2 (Uma)"
>>>>>> ID=linuxmint
>>>>>> ID_LIKE=ubuntu
>>>>>> PRETTY_NAME="Linux Mint 20.2"
>>>>>> VERSION_ID="20.2"
>>>>>> HOME_URL="https://www.linuxmint.com/"
>>>>>> SUPPORT_URL="https://forums.linuxmint.com/"
>>>>>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
>>>>>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
>>>>>> VERSION_CODENAME=uma
>>>>>> UBUNTU_CODENAME=focal
>>>>>>
>>>>>> $ uname -a
>>>>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed 
>>>>>> Aug 11
>>>>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>>>>>
>>>>>> $ ./configure
>>>>>> checking for gcc... gcc
>>>>>> checking whether the C compiler works... yes
>>>>>> checking for C compiler default output file name... a.out
>>>>>> checking for suffix of executables...
>>>>>> checking whether we are cross compiling... no
>>>>>> checking for suffix of object files... o
>>>>>> checking whether we are using the GNU C compiler... yes
>>>>>> checking whether gcc accepts -g... yes
>>>>>> checking for gcc option to accept ISO C89... none needed
>>>>>> checking how to run the C preprocessor... gcc -E
>>>>>> checking for grep that handles long lines and -e... 
>>>>>> /bin/grep
>>>>>> checking for egrep... /bin/grep -E
>>>>>> checking for ANSI C header files... yes
>>>>>> checking for sys/types.h... yes
>>>>>> checking for sys/stat.h... yes
>>>>>> checking for stdlib.h... yes
>>>>>> checking for string.h... yes
>>>>>> checking for memory.h... yes
>>>>>> checking for strings.h... yes
>>>>>> checking for inttypes.h... yes
>>>>>> checking for stdint.h... yes
>>>>>> checking for unistd.h... yes
>>>>>> checking minix/config.h usability... no
>>>>>> checking minix/config.h presence... no
>>>>>> checking for minix/config.h... no
>>>>>> checking whether it is safe to define __EXTENSIONS__... yes
>>>>>> checking for gcc... (cached) gcc
>>>>>> checking whether we are using the GNU C compiler... 
>>>>>> (cached) yes
>>>>>> checking whether gcc accepts -g... (cached) yes
>>>>>> checking for gcc option to accept ISO C89... (cached) none 
>>>>>> needed
>>>>>> checking whether C compiler accepts -std=gnu90... yes
>>>>>> checking build system type... x86_64-pc-linux-gnu
>>>>>> checking host system type... x86_64-pc-linux-gnu
>>>>>> checking for an ANSI C-conforming const... yes
>>>>>> checking for working volatile... yes
>>>>>> checking whether byte ordering is bigendian... no
>>>>>> checking for special C compiler options needed for large 
>>>>>> files... no
>>>>>> checking for _FILE_OFFSET_BITS value needed for large 
>>>>>> files... no
>>>>>> checking for a BSD-compatible install... /usr/bin/install 
>>>>>> -c
>>>>>> checking whether ln -s works... yes
>>>>>> checking for ar... ar
>>>>>> checking for rm... /bin/rm
>>>>>> checking for rmdir... /bin/rmdir
>>>>>> checking for openat... yes
>>>>>> checking for reallocarray... yes
>>>>>> checking for clock_gettime... yes
>>>>>> checking linux/perf_event.h usability... yes
>>>>>> checking linux/perf_event.h presence... yes
>>>>>> checking for linux/perf_event.h... yes
>>>>>> checking linux/hw_breakpoint.h usability... yes
>>>>>> checking linux/hw_breakpoint.h presence... yes
>>>>>> checking for linux/hw_breakpoint.h... yes
>>>>>> checking for pkg-config... /usr/bin/pkg-config
>>>>>> checking pkg-config is at least version 0.9.0... yes
>>>>>> checking execinfo.h usability... yes
>>>>>> checking execinfo.h presence... yes
>>>>>> checking for execinfo.h... yes
>>>>>> checking for backtrace... yes
>>>>>> checking for backtrace_symbols_fd... yes
>>>>>> checking for xmlto... /usr/bin/xmlto
>>>>>> checking for mv... /bin/mv
>>>>>> checking for a sed that does not truncate output... 
>>>>>> /bin/sed
>>>>>> checking for asciidoc... /usr/bin/asciidoc
>>>>>> checking for asciidoctor... no
>>>>>> checking for EXT2FS... yes
>>>>>> checking for COM_ERR... yes
>>>>>> checking for REISERFS... yes
>>>>>> checking for FIEMAP_EXTENT_SHARED defined in 
>>>>>> linux/fiemap.h... yes
>>>>>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... 
>>>>>> yes
>>>>>> checking linux/blkzoned.h usability... yes
>>>>>> checking linux/blkzoned.h presence... yes
>>>>>> checking for linux/blkzoned.h... yes
>>>>>> checking for struct blk_zone.capacity... no
>>>>>> checking for BLKGETZONESZ defined in linux/blkzoned.h... 
>>>>>> yes
>>>>>
>>>>>> configure: error: linux/blkzoned.h does not provide 
>>>>>> blk_zone.capacity
>>>>>
>>>>>
>>>>>>
>>>>>> ---
>>>>>>
>>>>>> Info on the file in question (linux/blkzoned.h):
>>>>>>
>>>>>> $ dpkg -S /usr/include/linux/blkzoned.h
>>>>>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
>>>>>>
>>>>>> $ dpkg -l linux-libc-dev
>>>>>> Desired=Unknown/Install/Remove/Purge/Hold
>>>>>> | 
>>>>>> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
>>>>>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>>>>>> ||/ Name                 Version      Architecture 
>>>>>> Description
>>>>>> +++-====================-============-============-====================================
>>>>>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux 
>>>>>> Kernel
>>>>>> Headers for development
>>>>>>
>>>>>>
>>>>>> So it appears that linux-libc-dev is way out-dated compared 
>>>>>> to my
>>>>>> kernel.  I don't know how to update it, though... there 
>>>>>> doesn't appear
>>>>>> to be a newer version available.
>>>>>
>>>>> You could disable the zoned.
>>>>>
>>>>>      ./configure --disable-zoned
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>

Re: Next steps in recovery?

[Robert Wyrick]

I ran memtester for 2 complete passes.  No failures.  I only tested
51G of my 64G, though. :/  That's how much was "free".

On Tue, Sep 7, 2021 at 8:10 PM Su Yue <l@damenly.su> wrote:
>
>
> On Wed 08 Sep 2021 at 07:15, Qu Wenruo <quwenruo.btrfs@gmx.com>
> wrote:
>
> > On 2021/9/8 上午1:02, Robert Wyrick wrote:
> >> Ran a repair:
> >>
> >> $ sudo ./btrfs check --repair -p /dev/sda  # I did NOT make
> >> install,
> >> just ran from the compiled directory
> >> enabling repair mode
> >> WARNING:
> >>
> >> Do not use --repair unless you are advised to do so by a
> >> developer
> >> or an experienced user, and then only after having accepted
> >> that no
> >> fsck can successfully repair all types of filesystem
> >> corruption. Eg.
> >> some software or hardware bugs can fatally damage a volume.
> >> The operation will start in 10 seconds.
> >> Use Ctrl-C to stop it.
> >> 10 9 8 7 6 5 4 3 2 1
> >> Starting repair.
> >> Opening filesystem to check...
> >> Checking filesystem on /dev/sda
> >> UUID: 75f1f45c-552e-4ae2-a56f-46e44b6647cf
> >> [1/7] checking root items                      (0:00:59
> >> elapsed,
> >> 2649102 items checked)
> >> Fixed 0 roots.
> >> Reset extent item (38179182174208) generation to
> >> 4057084elapsed,
> >> 1116143 items checked)
> >> No device size related problem found           (0:02:22
> >> elapsed,
> >> 1116143 items checked)
> >> [2/7] checking extents                         (0:02:23
> >> elapsed,
> >> 1116143 items checked)
> >> cache and super generation don't match, space cache will be
> >> invalidated
> >> [3/7] checking free space cache                (0:00:00
> >> elapsed)
> >> Deleting bad dir index [8348950,96,3] root 259 (0:00:25
> >> elapsed,
> >> 106695 items checked)
> >> repairing missing dir index item for inode 834922400:26
> >> elapsed,
> >> 108893 items checked)
> >> [4/7] checking fs roots                        (0:01:04
> >> elapsed,
> >> 217787 items checked)
> >> [5/7] checking csums (without verifying data)  (0:00:04
> >> elapsed,
> >> 12350321 items checked)
> >> [6/7] checking root refs                       (0:00:00
> >> elapsed, 4
> >> items checked)
> >> [7/7] checking quota groups skipped (not enabled on this FS)
> >> found 15729059057664 bytes used, no error found
> >> total csum bytes: 15313288548
> >> total tree bytes: 18286739456
> >> total fs tree bytes: 1791819776
> >> total extent tree bytes: 229130240
> >> btree space waste bytes: 1018844959
> >> file data blocks allocated: 51587230502912
> >>   referenced 15627926712320
> >>
> >> I can now mount the filesystem successfully!  Thank you for
> >> your help.
> >>
> >> I do have some additional questions if you don't mind...
> >> I am already using RAID 1 to handle single disk outages.
> >
> > One thing to note is, RAID is not perfect, not even close to
> > proper backup.
> >
> > RAID is really only suitable to handle disk failures, nothing
> > more than
> > that.
> >
> > In a spectrum of backup, RAID is really just better than
> > nothing.
> >
> > In this particular case, all the corruption is from bitflips,
> > thus all
> > copies are corrupted, no profile can save the day.
> >
> >>  I assume
> >> things could have gone much worse and I could have lost the
> >> whole
> >> filesystem.
> >
> > If you're using newer kernels all time, the kernel can detect
> > the extent
> > generation problem before writing the corrupted data back to
> > disk, thus
> > save the day.
> >
> >>  Aside from backups (I know, I know), is there anything
> >> else I can do to prevent such issues or make them easier to
> >> recover
> >> from?
> >
> > Newer kernel (v5.11 and newer) can prevent it.
> > Although when such rejection happens, it will not feel that
> > comfort
> > though, as it would mostly result the fs to go RO.
> > But still way better than writing bad data onto disks.
> >
> >>  Could this problem have been avoided/detected earlier?
> >
> > Yes, newer kernel.
> >
> >>  This
> >> wasn't a disk failure and according to memtest86+, it wasn't
> >> due to
> >> bad memory either....
> >
> > I still don't believe, maybe you can try to run memtester (which
> > is ran
> > in user space, and since we have kernel doing the page mapping,
> > it may
> > expose a different workload on the memory controller than
> > memtest86+)
> >
> And testmem5 using config anta777 may help. It's widely used to
> test
> memory stability after overclocking but runs on evil Windows
> though.
> It won't take too much time. Two cicles test of 64G memory
> consumes
> about 4~6 hours.
>
> --
> Su
>
> Subject
> > Since the extent generation corruption is a super obvious
> > bitflip.
> >
> >>  I don't run scrubs very often.  Should I?
> >
> > For newer kernels, the corruption can be rejected in first
> > place, thus
> > the scrub is only going to detect problems already in the fs.
> >
> > For older kernels, scrub won't detect the problem anyw.
> >
> > So I guess you don't need that frequent scrub, but it's still
> > recommended.
> > Maybe monthly?
> >
> >>  I
> >> guess the more general question is:  What are the best
> >> practices for
> >> maintaining a healthy btrfs file system?
> >
> > Well, healthy hardware, balanced kernel version between cutting
> > edge and
> > stable.
> > Personally I'm more towards cutting edge thought.
> >
> > Thanks,
> > Qu
> >
> >>
> >> Thanks again!
> >>
> >> On Mon, Sep 6, 2021 at 10:53 PM Qu Wenruo
> >> <quwenruo.btrfs@gmx.com> wrote:
> >>>
> >>>
> >>>
> >>> On 2021/9/7 下午12:36, Robert Wyrick wrote:
> >>>> What exactly would i be disabling?  I don't know what zoned
> >>>> does.
> >>>
> >>> The zoned device support.
> >>>
> >>> If you don't have any host-managed zoned device, there is no
> >>> reason you
> >>> would like to enable it.
> >>>
> >>> https://zonedstorage.io/introduction/
> >>>
> >>> Thanks,
> >>> Qu
> >>>
> >>>>
> >>>> On Mon, Sep 6, 2021, 9:07 PM Anand Jain
> >>>> <anand.jain@oracle.com> wrote:
> >>>>>
> >>>>> On 07/09/2021 10:36, Robert Wyrick wrote:
> >>>>>> Trying to build latest btrfs-progs.  I'm seeing errors in
> >>>>>> the configure script.
> >>>>>>
> >>>>>> $ cat /etc/os-release
> >>>>>> NAME="Linux Mint"
> >>>>>> VERSION="20.2 (Uma)"
> >>>>>> ID=linuxmint
> >>>>>> ID_LIKE=ubuntu
> >>>>>> PRETTY_NAME="Linux Mint 20.2"
> >>>>>> VERSION_ID="20.2"
> >>>>>> HOME_URL="https://www.linuxmint.com/"
> >>>>>> SUPPORT_URL="https://forums.linuxmint.com/"
> >>>>>> BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
> >>>>>> PRIVACY_POLICY_URL="https://www.linuxmint.com/"
> >>>>>> VERSION_CODENAME=uma
> >>>>>> UBUNTU_CODENAME=focal
> >>>>>>
> >>>>>> $ uname -a
> >>>>>> Linux bigbox 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed
> >>>>>> Aug 11
> >>>>>> 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> >>>>>>
> >>>>>> $ ./configure
> >>>>>> checking for gcc... gcc
> >>>>>> checking whether the C compiler works... yes
> >>>>>> checking for C compiler default output file name... a.out
> >>>>>> checking for suffix of executables...
> >>>>>> checking whether we are cross compiling... no
> >>>>>> checking for suffix of object files... o
> >>>>>> checking whether we are using the GNU C compiler... yes
> >>>>>> checking whether gcc accepts -g... yes
> >>>>>> checking for gcc option to accept ISO C89... none needed
> >>>>>> checking how to run the C preprocessor... gcc -E
> >>>>>> checking for grep that handles long lines and -e...
> >>>>>> /bin/grep
> >>>>>> checking for egrep... /bin/grep -E
> >>>>>> checking for ANSI C header files... yes
> >>>>>> checking for sys/types.h... yes
> >>>>>> checking for sys/stat.h... yes
> >>>>>> checking for stdlib.h... yes
> >>>>>> checking for string.h... yes
> >>>>>> checking for memory.h... yes
> >>>>>> checking for strings.h... yes
> >>>>>> checking for inttypes.h... yes
> >>>>>> checking for stdint.h... yes
> >>>>>> checking for unistd.h... yes
> >>>>>> checking minix/config.h usability... no
> >>>>>> checking minix/config.h presence... no
> >>>>>> checking for minix/config.h... no
> >>>>>> checking whether it is safe to define __EXTENSIONS__... yes
> >>>>>> checking for gcc... (cached) gcc
> >>>>>> checking whether we are using the GNU C compiler...
> >>>>>> (cached) yes
> >>>>>> checking whether gcc accepts -g... (cached) yes
> >>>>>> checking for gcc option to accept ISO C89... (cached) none
> >>>>>> needed
> >>>>>> checking whether C compiler accepts -std=gnu90... yes
> >>>>>> checking build system type... x86_64-pc-linux-gnu
> >>>>>> checking host system type... x86_64-pc-linux-gnu
> >>>>>> checking for an ANSI C-conforming const... yes
> >>>>>> checking for working volatile... yes
> >>>>>> checking whether byte ordering is bigendian... no
> >>>>>> checking for special C compiler options needed for large
> >>>>>> files... no
> >>>>>> checking for _FILE_OFFSET_BITS value needed for large
> >>>>>> files... no
> >>>>>> checking for a BSD-compatible install... /usr/bin/install
> >>>>>> -c
> >>>>>> checking whether ln -s works... yes
> >>>>>> checking for ar... ar
> >>>>>> checking for rm... /bin/rm
> >>>>>> checking for rmdir... /bin/rmdir
> >>>>>> checking for openat... yes
> >>>>>> checking for reallocarray... yes
> >>>>>> checking for clock_gettime... yes
> >>>>>> checking linux/perf_event.h usability... yes
> >>>>>> checking linux/perf_event.h presence... yes
> >>>>>> checking for linux/perf_event.h... yes
> >>>>>> checking linux/hw_breakpoint.h usability... yes
> >>>>>> checking linux/hw_breakpoint.h presence... yes
> >>>>>> checking for linux/hw_breakpoint.h... yes
> >>>>>> checking for pkg-config... /usr/bin/pkg-config
> >>>>>> checking pkg-config is at least version 0.9.0... yes
> >>>>>> checking execinfo.h usability... yes
> >>>>>> checking execinfo.h presence... yes
> >>>>>> checking for execinfo.h... yes
> >>>>>> checking for backtrace... yes
> >>>>>> checking for backtrace_symbols_fd... yes
> >>>>>> checking for xmlto... /usr/bin/xmlto
> >>>>>> checking for mv... /bin/mv
> >>>>>> checking for a sed that does not truncate output...
> >>>>>> /bin/sed
> >>>>>> checking for asciidoc... /usr/bin/asciidoc
> >>>>>> checking for asciidoctor... no
> >>>>>> checking for EXT2FS... yes
> >>>>>> checking for COM_ERR... yes
> >>>>>> checking for REISERFS... yes
> >>>>>> checking for FIEMAP_EXTENT_SHARED defined in
> >>>>>> linux/fiemap.h... yes
> >>>>>> checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h...
> >>>>>> yes
> >>>>>> checking linux/blkzoned.h usability... yes
> >>>>>> checking linux/blkzoned.h presence... yes
> >>>>>> checking for linux/blkzoned.h... yes
> >>>>>> checking for struct blk_zone.capacity... no
> >>>>>> checking for BLKGETZONESZ defined in linux/blkzoned.h...
> >>>>>> yes
> >>>>>
> >>>>>> configure: error: linux/blkzoned.h does not provide
> >>>>>> blk_zone.capacity
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> ---
> >>>>>>
> >>>>>> Info on the file in question (linux/blkzoned.h):
> >>>>>>
> >>>>>> $ dpkg -S /usr/include/linux/blkzoned.h
> >>>>>> linux-libc-dev:amd64: /usr/include/linux/blkzoned.h
> >>>>>>
> >>>>>> $ dpkg -l linux-libc-dev
> >>>>>> Desired=Unknown/Install/Remove/Purge/Hold
> >>>>>> |
> >>>>>> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> >>>>>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> >>>>>> ||/ Name                 Version      Architecture
> >>>>>> Description
> >>>>>> +++-====================-============-============-====================================
> >>>>>> ii  linux-libc-dev:amd64 5.4.0-81.91  amd64        Linux
> >>>>>> Kernel
> >>>>>> Headers for development
> >>>>>>
> >>>>>>
> >>>>>> So it appears that linux-libc-dev is way out-dated compared
> >>>>>> to my
> >>>>>> kernel.  I don't know how to update it, though... there
> >>>>>> doesn't appear
> >>>>>> to be a newer version available.
> >>>>>
> >>>>> You could disable the zoned.
> >>>>>
> >>>>>      ./configure --disable-zoned
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>