trouble mounting btrfs filesystem....

trouble mounting btrfs filesystem....

[Scott E. Blomquist]

Hi All,

Early this morning there was a power glitch that affected our system.

The second enclosure went offline but the file system stayed up for a
bit before rebooting and recovering the 2 missing arrays sdb1 and
sdc1.

When mounting we get....

    Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS info (device sda1): has skinny extents
    Aug 12 14:54:52 localhost kernel: [ 8665.900321] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Aug 12 14:54:52 localhost kernel: [ 8665.985512] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Aug 12 14:54:52 localhost kernel: [ 8666.056845] BTRFS error (device sda1): failed to read block groups: -5
    Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device sda1): open_ctree failed

We are here...

    # uname -a
    Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

    # btrfs --version
    btrfs-progs v4.17.1
    
    # btrfs filesystem show
    Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
            Total devices 3 FS bytes used 75.05TiB
            devid    1 size 47.30TiB used 42.07TiB path /dev/sda1
            devid    2 size 21.83TiB used 16.61TiB path /dev/sdb1
            devid    3 size 21.83TiB used 16.61TiB path /dev/sdc1
    
Thanks for any help.

sb. Scott Blomquist

Re: trouble mounting btrfs filesystem....

[Scott E. Blomquist]

Hi All,

Is there any more info needed here?

I can restore from backup if needed but that will take a bit of time.

Checking around it looks like I could try...

    btrfs-zero-log /dev/sda1

Or maybe ..

   btrfsck --repair /dev/sda1

I am just not sure here and would prefer to do the right thing.

Any help would be much appreciated.

Thanks,

sb. Scott Blomquist


Scott E. Blomquist writes:
 > Hi All,
 > 
 > Early this morning there was a power glitch that affected our system.
 > 
 > The second enclosure went offline but the file system stayed up for a
 > bit before rebooting and recovering the 2 missing arrays sdb1 and
 > sdc1.
 > 
 > When mounting we get....
 > 
 >     Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS info (device sda1): has skinny extents
 >     Aug 12 14:54:52 localhost kernel: [ 8665.900321] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
 >     Aug 12 14:54:52 localhost kernel: [ 8665.985512] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
 >     Aug 12 14:54:52 localhost kernel: [ 8666.056845] BTRFS error (device sda1): failed to read block groups: -5
 >     Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device sda1): open_ctree failed
 > 
 > We are here...
 > 
 >     # uname -a
 >     Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
 > 
 >     # btrfs --version
 >     btrfs-progs v4.17.1
 >     
 >     # btrfs filesystem show
 >     Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
 >             Total devices 3 FS bytes used 75.05TiB
 >             devid    1 size 47.30TiB used 42.07TiB path /dev/sda1
 >             devid    2 size 21.83TiB used 16.61TiB path /dev/sdb1
 >             devid    3 size 21.83TiB used 16.61TiB path /dev/sdc1
 >     
 > Thanks for any help.
 > 
 > sb. Scott Blomquist

Re: trouble mounting btrfs filesystem....

[Dmitrii Tcvetkov]

> Scott E. Blomquist writes:
>  > Hi All,
>  > 
>  > Early this morning there was a power glitch that affected our
>  > system.
>  > 
>  > The second enclosure went offline but the file system stayed up
>  > for a bit before rebooting and recovering the 2 missing arrays
>  > sdb1 and sdc1.
>  > 
>  > When mounting we get....
>  > 
>  >     Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS info
>  > (device sda1): has skinny extents Aug 12 14:54:52 localhost
>  > kernel: [ 8665.900321] BTRFS error (device sda1): parent transid
>  > verify failed on 177443463479296 wanted 2159304 found 2159295 Aug
>  > 12 14:54:52 localhost kernel: [ 8665.985512] BTRFS error (device
>  > sda1): parent transid verify failed on 177443463479296 wanted
>  > 2159304 found 2159295 Aug 12 14:54:52 localhost kernel:
>  > [ 8666.056845] BTRFS error (device sda1): failed to read block
>  > groups: -5 Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS
>  > error (device sda1): open_ctree failed
>  > 
>  > We are here...
>  > 
>  >     # uname -a
>  >     Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00 EDT
>  > 2018 x86_64 x86_64 x86_64 GNU/Linux
>  > 
>  >     # btrfs --version
>  >     btrfs-progs v4.17.1
>  >     
>  >     # btrfs filesystem show
>  >     Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
>  >             Total devices 3 FS bytes used 75.05TiB
>  >             devid    1 size 47.30TiB used 42.07TiB path /dev/sda1
>  >             devid    2 size 21.83TiB used 16.61TiB path /dev/sdb1
>  >             devid    3 size 21.83TiB used 16.61TiB path /dev/sdc1
>  >     
>  > Thanks for any help.
>  > 
>  > sb. Scott Blomquist  
> Hi All,
> 
> Is there any more info needed here?
> 
> I can restore from backup if needed but that will take a bit of time.
> 
> Checking around it looks like I could try...
> 
>     btrfs-zero-log /dev/sda1
> 
> Or maybe ..
> 
>    btrfsck --repair /dev/sda1
> 
> I am just not sure here and would prefer to do the right thing.
> 
> Any help would be much appreciated.
> 
> Thanks,
> 
> sb. Scott Blomquist
> 
> 

I'm not a dev, just user.
btrfs-zero-log is for very specific case[1], not for transid errors.
Transid errors mean that some metadata writes are missing, if
they prevent you from mounting filesystem it's pretty much fatal. If
btrfs could recover metadata from good copy it'd have done that.

"wanted 2159304 found 2159295" means that some metadata is stale by 
9 commits. You could try to mount it with "ro,usebackuproot" mount
options as readonly mount is less strict. If that works you can try
"usebackuproot" without ro option. But 9 commits is probably too much
and there isn't enough data to rollback so far.

[1] https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log

Re: trouble mounting btrfs filesystem....

[Scott E. Blomquist]

Dmitrii Tcvetkov writes:
 > > Scott E. Blomquist writes:
 > >  > Hi All,
 > >  > 
 > >  > Early this morning there was a power glitch that affected our
 > >  > system.
 > >  > 
 > >  > The second enclosure went offline but the file system stayed up
 > >  > for a bit before rebooting and recovering the 2 missing arrays
 > >  > sdb1 and sdc1.
 > >  > 
 > >  > When mounting we get....
 > >  > 
 > >  >     Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS info
 > >  > (device sda1): has skinny extents Aug 12 14:54:52 localhost
 > >  > kernel: [ 8665.900321] BTRFS error (device sda1): parent transid
 > >  > verify failed on 177443463479296 wanted 2159304 found 2159295 Aug
 > >  > 12 14:54:52 localhost kernel: [ 8665.985512] BTRFS error (device
 > >  > sda1): parent transid verify failed on 177443463479296 wanted
 > >  > 2159304 found 2159295 Aug 12 14:54:52 localhost kernel:
 > >  > [ 8666.056845] BTRFS error (device sda1): failed to read block
 > >  > groups: -5 Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS
 > >  > error (device sda1): open_ctree failed
 > >  > 
 > >  > We are here...
 > >  > 
 > >  >     # uname -a
 > >  >     Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00 EDT
 > >  > 2018 x86_64 x86_64 x86_64 GNU/Linux
 > >  > 
 > >  >     # btrfs --version
 > >  >     btrfs-progs v4.17.1
 > >  >     
 > >  >     # btrfs filesystem show
 > >  >     Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
 > >  >             Total devices 3 FS bytes used 75.05TiB
 > >  >             devid    1 size 47.30TiB used 42.07TiB path /dev/sda1
 > >  >             devid    2 size 21.83TiB used 16.61TiB path /dev/sdb1
 > >  >             devid    3 size 21.83TiB used 16.61TiB path /dev/sdc1
 > >  >     
 > >  > Thanks for any help.
 > >  > 
 > >  > sb. Scott Blomquist  
 > > Hi All,
 > > 
 > > Is there any more info needed here?
 > > 
 > > I can restore from backup if needed but that will take a bit of time.
 > > 
 > > Checking around it looks like I could try...
 > > 
 > >     btrfs-zero-log /dev/sda1
 > > 
 > > Or maybe ..
 > > 
 > >    btrfsck --repair /dev/sda1
 > > 
 > > I am just not sure here and would prefer to do the right thing.
 > > 
 > > Any help would be much appreciated.
 > > 
 > > Thanks,
 > > 
 > > sb. Scott Blomquist
 > > 
 > > 
 > 
 > I'm not a dev, just user.
 > btrfs-zero-log is for very specific case[1], not for transid errors.
 > Transid errors mean that some metadata writes are missing, if
 > they prevent you from mounting filesystem it's pretty much fatal. If
 > btrfs could recover metadata from good copy it'd have done that.
 > 
 > "wanted 2159304 found 2159295" means that some metadata is stale by 
 > 9 commits. You could try to mount it with "ro,usebackuproot" mount
 > options as readonly mount is less strict. If that works you can try
 > "usebackuproot" without ro option. But 9 commits is probably too much
 > and there isn't enough data to rollback so far.
 > 
 > [1] https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log

Thank you.  So zero-log is not the right thing...

Unfortunately when mounting ro,usebackuproot I still get the same messages...

    Aug 14 09:08:15 localhost kernel: [160669.100314] BTRFS info (device sda1): trying to use backup root at mount time
    Aug 14 09:08:15 localhost kernel: [160669.100316] BTRFS info (device sda1): using free space tree
    Aug 14 09:08:15 localhost kernel: [160669.100318] BTRFS info (device sda1): has skinny extents
    Aug 14 09:10:24 localhost kernel: [160797.736704] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Aug 14 09:10:24 localhost kernel: [160797.815441] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Aug 14 09:10:24 localhost kernel: [160797.887708] BTRFS error (device sda1): failed to read block groups: -5
    Aug 14 09:10:24 localhost kernel: [160798.031183] BTRFS error (device sda1): open_ctree failed

it sounds like my only option maybe 'btrfs check --repair' and that
doesn't sound too hopeful.

Any other ideas?

Thanks,

sb. Scott Blomquist

Re: trouble mounting btrfs filesystem....

[Dmitrii Tcvetkov]

On Tue, 14 Aug 2018 09:31:56 -0400
"Scott E. Blomquist" <sb@techsquare.com> wrote:

> Dmitrii Tcvetkov writes:
>  > > Scott E. Blomquist writes:  
>  > >  > Hi All,
>  > >  > 
>  > >  > Early this morning there was a power glitch that affected our
>  > >  > system.
>  > >  > 
>  > >  > The second enclosure went offline but the file system stayed
>  > >  > up for a bit before rebooting and recovering the 2 missing
>  > >  > arrays sdb1 and sdc1.
>  > >  > 
>  > >  > When mounting we get....
>  > >  > 
>  > >  >     Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS
>  > >  > info (device sda1): has skinny extents Aug 12 14:54:52
>  > >  > localhost kernel: [ 8665.900321] BTRFS error (device sda1):
>  > >  > parent transid verify failed on 177443463479296 wanted
>  > >  > 2159304 found 2159295 Aug 12 14:54:52 localhost kernel:
>  > >  > [ 8665.985512] BTRFS error (device sda1): parent transid
>  > >  > verify failed on 177443463479296 wanted 2159304 found 2159295
>  > >  > Aug 12 14:54:52 localhost kernel: [ 8666.056845] BTRFS error
>  > >  > (device sda1): failed to read block groups: -5 Aug 12
>  > >  > 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device
>  > >  > sda1): open_ctree failed
>  > >  > 
>  > >  > We are here...
>  > >  > 
>  > >  >     # uname -a
>  > >  >     Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00
>  > >  > EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
>  > >  > 
>  > >  >     # btrfs --version
>  > >  >     btrfs-progs v4.17.1
>  > >  >     
>  > >  >     # btrfs filesystem show
>  > >  >     Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
>  > >  >             Total devices 3 FS bytes used 75.05TiB
>  > >  >             devid    1 size 47.30TiB used 42.07TiB
>  > >  > path /dev/sda1 devid    2 size 21.83TiB used 16.61TiB
>  > >  > path /dev/sdb1 devid    3 size 21.83TiB used 16.61TiB
>  > >  > path /dev/sdc1 
>  > >  > Thanks for any help.
>  > >  > 
>  > >  > sb. Scott Blomquist    
>  > > Hi All,
>  > > 
>  > > Is there any more info needed here?
>  > > 
>  > > I can restore from backup if needed but that will take a bit of
>  > > time.
>  > > 
>  > > Checking around it looks like I could try...
>  > > 
>  > >     btrfs-zero-log /dev/sda1
>  > > 
>  > > Or maybe ..
>  > > 
>  > >    btrfsck --repair /dev/sda1
>  > > 
>  > > I am just not sure here and would prefer to do the right thing.
>  > > 
>  > > Any help would be much appreciated.
>  > > 
>  > > Thanks,
>  > > 
>  > > sb. Scott Blomquist
>  > > 
>  > >   
>  > 
>  > I'm not a dev, just user.
>  > btrfs-zero-log is for very specific case[1], not for transid
>  > errors. Transid errors mean that some metadata writes are missing,
>  > if they prevent you from mounting filesystem it's pretty much
>  > fatal. If btrfs could recover metadata from good copy it'd have
>  > done that.
>  > 
>  > "wanted 2159304 found 2159295" means that some metadata is stale
>  > by 9 commits. You could try to mount it with "ro,usebackuproot"
>  > mount options as readonly mount is less strict. If that works you
>  > can try "usebackuproot" without ro option. But 9 commits is
>  > probably too much and there isn't enough data to rollback so far.
>  > 
>  > [1] https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log  
> 
> Thank you.  So zero-log is not the right thing...
> 
> Unfortunately when mounting ro,usebackuproot I still get the same
> messages...
> 
>     Aug 14 09:08:15 localhost kernel: [160669.100314] BTRFS info
> (device sda1): trying to use backup root at mount time Aug 14
> 09:08:15 localhost kernel: [160669.100316] BTRFS info (device sda1):
> using free space tree Aug 14 09:08:15 localhost kernel:
> [160669.100318] BTRFS info (device sda1): has skinny extents Aug 14
> 09:10:24 localhost kernel: [160797.736704] BTRFS error (device sda1):
> parent transid verify failed on 177443463479296 wanted 2159304 found
> 2159295 Aug 14 09:10:24 localhost kernel: [160797.815441] BTRFS error
> (device sda1): parent transid verify failed on 177443463479296 wanted
> 2159304 found 2159295 Aug 14 09:10:24 localhost kernel:
> [160797.887708] BTRFS error (device sda1): failed to read block
> groups: -5 Aug 14 09:10:24 localhost kernel: [160798.031183] BTRFS
> error (device sda1): open_ctree failed
> 
> it sounds like my only option maybe 'btrfs check --repair' and that
> doesn't sound too hopeful.
> 
> Any other ideas?
> 
> Thanks,
> 
> sb. Scott Blomquist

As far as I know btrfs check --repair doesn't fix transid errors.
Usually devs can tell whenever "btrfs check --repair" will repair
anything by looking at "btrfs check --readonly" output, but as your
filesystem is quite big this will take some time.

If usebackuproot doesn't help then filesystem is beyond repair and you
should try to refresh your backups with "btrfs restore" and restore from them[1].

[1] https://btrfs.wiki.kernel.org/index.php/FAQ#How_do_I_recover_from_a_.22parent_transid_verify_failed.22_error.3F

Re: trouble mounting btrfs filesystem....

[Scott E. Blomquist]

Dmitrii Tcvetkov writes:
 > On Tue, 14 Aug 2018 09:31:56 -0400
 > "Scott E. Blomquist" <sb@techsquare.com> wrote:
 > 
 > > Dmitrii Tcvetkov writes:
 > >  > > Scott E. Blomquist writes:  
 > >  > >  > Hi All,
 > >  > >  > 
 > >  > >  > Early this morning there was a power glitch that affected our
 > >  > >  > system.
 > >  > >  > 
 > >  > >  > The second enclosure went offline but the file system stayed
 > >  > >  > up for a bit before rebooting and recovering the 2 missing
 > >  > >  > arrays sdb1 and sdc1.
 > >  > >  > 
 > >  > >  > When mounting we get....
 > >  > >  > 
 > >  > >  >     Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS
 > >  > >  > info (device sda1): has skinny extents Aug 12 14:54:52
 > >  > >  > localhost kernel: [ 8665.900321] BTRFS error (device sda1):
 > >  > >  > parent transid verify failed on 177443463479296 wanted
 > >  > >  > 2159304 found 2159295 Aug 12 14:54:52 localhost kernel:
 > >  > >  > [ 8665.985512] BTRFS error (device sda1): parent transid
 > >  > >  > verify failed on 177443463479296 wanted 2159304 found 2159295
 > >  > >  > Aug 12 14:54:52 localhost kernel: [ 8666.056845] BTRFS error
 > >  > >  > (device sda1): failed to read block groups: -5 Aug 12
 > >  > >  > 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device
 > >  > >  > sda1): open_ctree failed
 > >  > >  > 
 > >  > >  > We are here...
 > >  > >  > 
 > >  > >  >     # uname -a
 > >  > >  >     Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00
 > >  > >  > EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
 > >  > >  > 
 > >  > >  >     # btrfs --version
 > >  > >  >     btrfs-progs v4.17.1
 > >  > >  >     
 > >  > >  >     # btrfs filesystem show
 > >  > >  >     Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
 > >  > >  >             Total devices 3 FS bytes used 75.05TiB
 > >  > >  >             devid    1 size 47.30TiB used 42.07TiB
 > >  > >  > path /dev/sda1 devid    2 size 21.83TiB used 16.61TiB
 > >  > >  > path /dev/sdb1 devid    3 size 21.83TiB used 16.61TiB
 > >  > >  > path /dev/sdc1 
 > >  > >  > Thanks for any help.
 > >  > >  > 
 > >  > >  > sb. Scott Blomquist    
 > >  > > Hi All,
 > >  > > 
 > >  > > Is there any more info needed here?
 > >  > > 
 > >  > > I can restore from backup if needed but that will take a bit of
 > >  > > time.
 > >  > > 
 > >  > > Checking around it looks like I could try...
 > >  > > 
 > >  > >     btrfs-zero-log /dev/sda1
 > >  > > 
 > >  > > Or maybe ..
 > >  > > 
 > >  > >    btrfsck --repair /dev/sda1
 > >  > > 
 > >  > > I am just not sure here and would prefer to do the right thing.
 > >  > > 
 > >  > > Any help would be much appreciated.
 > >  > > 
 > >  > > Thanks,
 > >  > > 
 > >  > > sb. Scott Blomquist
 > >  > > 
 > >  > >   
 > >  > 
 > >  > I'm not a dev, just user.
 > >  > btrfs-zero-log is for very specific case[1], not for transid
 > >  > errors. Transid errors mean that some metadata writes are missing,
 > >  > if they prevent you from mounting filesystem it's pretty much
 > >  > fatal. If btrfs could recover metadata from good copy it'd have
 > >  > done that.
 > >  > 
 > >  > "wanted 2159304 found 2159295" means that some metadata is stale
 > >  > by 9 commits. You could try to mount it with "ro,usebackuproot"
 > >  > mount options as readonly mount is less strict. If that works you
 > >  > can try "usebackuproot" without ro option. But 9 commits is
 > >  > probably too much and there isn't enough data to rollback so far.
 > >  > 
 > >  > [1] https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log  
 > > 
 > > Thank you.  So zero-log is not the right thing...
 > > 
 > > Unfortunately when mounting ro,usebackuproot I still get the same
 > > messages...
 > > 
 > >     Aug 14 09:08:15 localhost kernel: [160669.100314] BTRFS info
 > > (device sda1): trying to use backup root at mount time Aug 14
 > > 09:08:15 localhost kernel: [160669.100316] BTRFS info (device sda1):
 > > using free space tree Aug 14 09:08:15 localhost kernel:
 > > [160669.100318] BTRFS info (device sda1): has skinny extents Aug 14
 > > 09:10:24 localhost kernel: [160797.736704] BTRFS error (device sda1):
 > > parent transid verify failed on 177443463479296 wanted 2159304 found
 > > 2159295 Aug 14 09:10:24 localhost kernel: [160797.815441] BTRFS error
 > > (device sda1): parent transid verify failed on 177443463479296 wanted
 > > 2159304 found 2159295 Aug 14 09:10:24 localhost kernel:
 > > [160797.887708] BTRFS error (device sda1): failed to read block
 > > groups: -5 Aug 14 09:10:24 localhost kernel: [160798.031183] BTRFS
 > > error (device sda1): open_ctree failed
 > > 
 > > it sounds like my only option maybe 'btrfs check --repair' and that
 > > doesn't sound too hopeful.
 > > 
 > > Any other ideas?
 > > 
 > > Thanks,
 > > 
 > > sb. Scott Blomquist
 > 
 > As far as I know btrfs check --repair doesn't fix transid errors.
 > Usually devs can tell whenever "btrfs check --repair" will repair
 > anything by looking at "btrfs check --readonly" output, but as your
 > filesystem is quite big this will take some time.
 > 
 > If usebackuproot doesn't help then filesystem is beyond repair and you
 > should try to refresh your backups with "btrfs restore" and restore from them[1].
 > 
 > [1] https://btrfs.wiki.kernel.org/index.php/FAQ#How_do_I_recover_from_a_.22parent_transid_verify_failed.22_error.3F

Thanks.  Yup.  Looks bad...

    root@localhost:~# btrfs check --readonly /dev/sda1 
    Opening filesystem to check...
    parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
    Ignoring transid failure
    ERROR: child eb corrupted: parent bytenr=177443205808128 item=156 parent level=2 child level=0
    ERROR: cannot open file system

Looks like I will start the restore process unless there are any other
ideas.

Thanks again,

sb. Scott Blomquist

Re: trouble mounting btrfs filesystem....

[Hans van Kranenburg]

On 08/12/2018 09:19 PM, Scott E. Blomquist wrote:
> 
> Hi All,
> 
> Early this morning there was a power glitch that affected our system.
> 
> The second enclosure went offline but the file system stayed up for a
> bit before rebooting and recovering the 2 missing arrays sdb1 and
> sdc1.
> 
> When mounting we get....
> 
>     Aug 12 14:52:43 localhost kernel: [ 8536.649270] BTRFS info (device sda1): has skinny extents
>     Aug 12 14:54:52 localhost kernel: [ 8665.900321] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
>     Aug 12 14:54:52 localhost kernel: [ 8665.985512] BTRFS error (device sda1): parent transid verify failed on 177443463479296 wanted 2159304 found 2159295
>     Aug 12 14:54:52 localhost kernel: [ 8666.056845] BTRFS error (device sda1): failed to read block groups: -5
>     Aug 12 14:54:52 localhost kernel: [ 8666.254178] BTRFS error (device sda1): open_ctree failed
> 
> We are here...
> 
>     # uname -a
>     Linux localhost 4.17.14-custom #1 SMP Sun Aug 12 11:54:00 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
> 
>     # btrfs --version
>     btrfs-progs v4.17.1
>     
>     # btrfs filesystem show
>     Label: none  uuid: 8337c837-58cb-430a-a929-7f6d2f50bdbb
>             Total devices 3 FS bytes used 75.05TiB
>             devid    1 size 47.30TiB used 42.07TiB path /dev/sda1
>             devid    2 size 21.83TiB used 16.61TiB path /dev/sdb1
>             devid    3 size 21.83TiB used 16.61TiB path /dev/sdc1

What kind of devices are this? You say enclosure... is it a bunch of
disks doing its own RAID, with btrfs on top?

Do you have RAID1 metadata on top of that, or single?

At least if you go the mkfs route (I read the other replies) then also
find out what happened. If your storage is losing data in situations
like this while it told btrfs that the data was safe, you're running a
dangerous operation.

-- 
Hans van Kranenburg

Re: trouble mounting btrfs filesystem....

[Hans van Kranenburg]

On 08/14/2018 03:00 PM, Dmitrii Tcvetkov wrote:
>> Scott E. Blomquist writes:
>>  > Hi All,
>>  > 
>>  > [...]
> 
> I'm not a dev, just user.
> btrfs-zero-log is for very specific case[1], not for transid errors.
> Transid errors mean that some metadata writes are missing, if
> they prevent you from mounting filesystem it's pretty much fatal. If
> btrfs could recover metadata from good copy it'd have done that.
> 
> "wanted 2159304 found 2159295" means that some metadata is stale by 
> 9 commits. You could try to mount it with "ro,usebackuproot" mount
> options as readonly mount is less strict. If that works you can try
> "usebackuproot" without ro option. But 9 commits is probably too much
> and there isn't enough data to rollback so far.

Keep in mind that a successful mount with usebackuproot does not mean
you're looking at a consistent filesystem. After each transaction
commit, disk space that is no longer referenced is immediately freed for
reuse.

So, even if you can mount with usebackuproot, you have to hope that none
of the metadata blocks that were used back then have been overwritten
already, even the ones in distant corners of trees. A full check / scrub
/ etc would be needed to find out.

-- 
Hans van Kranenburg

Re: trouble mounting btrfs filesystem....

[Andrei Borzenkov]

14.08.2018 18:16, Hans van Kranenburg пишет:
> On 08/14/2018 03:00 PM, Dmitrii Tcvetkov wrote:
>>> Scott E. Blomquist writes:
>>>  > Hi All,
>>>  > 
>>>  > [...]
>>
>> I'm not a dev, just user.
>> btrfs-zero-log is for very specific case[1], not for transid errors.
>> Transid errors mean that some metadata writes are missing, if
>> they prevent you from mounting filesystem it's pretty much fatal. If
>> btrfs could recover metadata from good copy it'd have done that.
>>
>> "wanted 2159304 found 2159295" means that some metadata is stale by 
>> 9 commits. You could try to mount it with "ro,usebackuproot" mount
>> options as readonly mount is less strict. If that works you can try
>> "usebackuproot" without ro option. But 9 commits is probably too much
>> and there isn't enough data to rollback so far.
> 
> Keep in mind that a successful mount with usebackuproot does not mean
> you're looking at a consistent filesystem. After each transaction
> commit, disk space that is no longer referenced is immediately freed for
> reuse.
> 

With all respect - in this case btrfs should not even pretend it can do
"usebackuproot". What this option is for then?

> So, even if you can mount with usebackuproot, you have to hope that none
> of the metadata blocks that were used back then have been overwritten
> already, even the ones in distant corners of trees. A full check / scrub
> / etc would be needed to find out.
> 

Re: trouble mounting btrfs filesystem....

[Roman Mamedov]

On Tue, 14 Aug 2018 16:41:11 +0300
Dmitrii Tcvetkov <demfloro@demfloro.ru> wrote:

> If usebackuproot doesn't help then filesystem is beyond repair and you
> should try to refresh your backups with "btrfs restore" and restore from them[1].
> 
> [1] https://btrfs.wiki.kernel.org/index.php/FAQ#How_do_I_recover_from_a_.22parent_transid_verify_failed.22_error.3F

This is really the worst unfixed Btrfs issue today. This happens a lot on
unclean shutdowns or reboots, and the only advice is usually to "start over"
-- even if your FS is 40 TB and the discrepancy is just by half a dozen
transids. There needs to be a way in fsck to accept (likely minor!) FS damage
and forcefully fixup transids to what they should be -- or even nuke the
affected portions entirely.

-- 
With respect,
Roman

Re: trouble mounting btrfs filesystem....

[Hans van Kranenburg]

On 08/14/2018 07:09 PM, Andrei Borzenkov wrote:
> 14.08.2018 18:16, Hans van Kranenburg пишет:
>> On 08/14/2018 03:00 PM, Dmitrii Tcvetkov wrote:
>>>> Scott E. Blomquist writes:
>>>>  > Hi All,
>>>>  > 
>>>>  > [...]
>>>
>>> I'm not a dev, just user.
>>> btrfs-zero-log is for very specific case[1], not for transid errors.
>>> Transid errors mean that some metadata writes are missing, if
>>> they prevent you from mounting filesystem it's pretty much fatal. If
>>> btrfs could recover metadata from good copy it'd have done that.
>>>
>>> "wanted 2159304 found 2159295" means that some metadata is stale by 
>>> 9 commits. You could try to mount it with "ro,usebackuproot" mount
>>> options as readonly mount is less strict. If that works you can try
>>> "usebackuproot" without ro option. But 9 commits is probably too much
>>> and there isn't enough data to rollback so far.
>>
>> Keep in mind that a successful mount with usebackuproot does not mean
>> you're looking at a consistent filesystem. After each transaction
>> commit, disk space that is no longer referenced is immediately freed for
>> reuse.
>>
> 
> With all respect - in this case btrfs should not even pretend it can do
> "usebackuproot". What this option is for then?

I have no idea. As the Dutch say: "schiet mij maar in de kerstboom". The
commit that introduces it contains no explanation at all.

The only thing I can think of is when you're a developer and writing
buggy code that writes out corrupted tree root blocks, which makes the
filesystem crash in some way directly after a transaction ends while no
other new writes are done again yet, combined with getting fed up of
having to mkfs and put your testdata back all the time.

>> So, even if you can mount with usebackuproot, you have to hope that none
>> of the metadata blocks that were used back then have been overwritten
>> already, even the ones in distant corners of trees. A full check / scrub
>> / etc would be needed to find out.

-- 
Hans van Kranenburg