[linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Dennis Schridde]

[-- Attachment #1.1: Type: text/plain, Size: 1555 bytes --]

Hello!

I have an issue re-gaining access to my data.  The error messages are:
# lvchange -ay ernie/system
  Check of pool ernie/cache failed (status:1). Manual repair required!
# lvconvert --repair ernie/system
bad checksum in superblock
  Repair of cache metadata volume of cache ernie/system failed (status:1). 
Manual repair required!

How do I recover from this situation and repair the volume group?

For a full log, including the output of pvdisplay, vgdisplay and lvdisplay -a, 
please see attached log file.  If more information is necessary, please ask.

What led to this situation:
I am using one of the infamous AMD Ryzen 2400G with an AMD B350 chipset, which 
suffers from random lockups related to CPU C-states [1].  With the recent 
AGESA 1.0.0.2a firmware update and the introduction of the "Power Supply Idle 
Control = Typical Current Idle" setting, the system is stable, if it boots at 
all.  But it often takes several attempts to boot -- the failed attempts 
ending in weird firmware / EFI or CPU / idle related Linux kernel stack 
traces, which sometimes even require a hard reset, since soft-reboot (ctrl+alt
+del) sometimes has no effect, because init dies.  This was such a situation, 
where I had to reboot (ctrl+alt+del) and reset (hard) the system several 
times, at the end of which everything seemed fine, except that dracut was 
timing out when activating the disks.  Debugging the situation using a Fedora 
28 live system resulted in the attached log file.

--Dennis

[1]: https://bugzilla.kernel.org/show_bug.cgi?id=196683

[-- Attachment #1.2: ernie-lvm.log --]
[-- Type: text/x-log, Size: 5943 bytes --]

# pvdisplay
  WARNING: Not using lvmetad because a repair command was run.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
  --- Physical volume ---
  PV Name               /dev/sdb3
  VG Name               ernie
  PV Size               <3.64 TiB / not usable 2.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              953742
  Free PE               0
  Allocated PE          953742
  PV UUID               R4qBtd-llco-Y2dj-ymFq-KxIB-lJv5-ySnjoa
   
  --- Physical volume ---
  PV Name               /dev/nvme0n1
  VG Name               ernie
  PV Size               <232.89 GiB / not usable <3.18 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              59618
  Free PE               0
  Allocated PE          59618
  PV UUID               CFrtOF-iDtJ-FLjh-i1c7-6DEs-8K8g-K1Son5
   
# vgdisplay
  WARNING: Not using lvmetad because a repair command was run.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
  --- Volume group ---
  VG Name               ernie
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  131
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               <3.87 TiB
  PE Size               4.00 MiB
  Total PE              1013360
  Alloc PE / Size       1013360 / <3.87 TiB
  Free  PE / Size       0 / 0   
  VG UUID               zzTyFs-nGPU-ksKG-K37V-U2Pz-OBLs-djtFCH
   
# lvdisplay --all
  WARNING: Not using lvmetad because a repair command was run.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
  --- Logical volume ---
  LV Path                /dev/ernie/system
  LV Name                system
  VG Name                ernie
  LV UUID                BflUud-7t4Q-5j7W-TEg0-AJiL-gLLT-CH3l4Q
  LV Write Access        read/write
  LV Creation host, time archiso, 2018-03-06 16:13:22 -0500
  LV Cache pool name     cache
  LV Cache origin name   system_corig
  LV Status              NOT available
  LV Size                2.00 TiB
  Current LE             524288
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/ernie/data
  LV Name                data
  VG Name                ernie
  LV UUID                Ac62Oc-hkkD-tJNH-EVFl-zjd0-w5KW-VXe1vn
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-03-21 21:24:00 -0400
  LV Status              available
  # open                 1
  LV Size                <1.64 TiB
  Current LE             429454
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
   
  --- Logical volume ---
  Internal LV Name       cache
  VG Name                ernie
  LV UUID                d3w5lx-MYKd-hCmg-rOzn-22f9-QoUI-XzXjLt
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:13 -0400
  LV Pool metadata       cache_cmeta
  LV Pool data           cache_cdata
  LV Status              NOT available
  LV Size                <232.79 GiB
  Current LE             59594
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  Internal LV Name       lvol0_pmspare
  VG Name                ernie
  LV UUID                9XxlMS-GEjl-H56Q-Qvy8-n7vR-0jpp-la07Ka
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:13 -0400
  LV Status              NOT available
  LV Size                48.00 MiB
  Current LE             12
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  Internal LV Name       cache_cmeta
  VG Name                ernie
  LV UUID                oGkCUA-Do2q-zhEV-TMt7-TN8g-p25e-C0cpfG
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:13 -0400
  LV Status              NOT available
  LV Size                48.00 MiB
  Current LE             12
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  Internal LV Name       cache_cdata
  VG Name                ernie
  LV UUID                J6HA4c-1zbK-0yXX-gdEf-HwZi-2wDn-0GtotN
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:13 -0400
  LV Status              NOT available
  LV Size                <232.79 GiB
  Current LE             59594
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  Internal LV Name       system_corig
  VG Name                ernie
  LV UUID                3xrU9V-kzgH-7Ncr-AQoE-u4lv-iiki-BbZIom
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:44 -0400
  LV origin of Cache LV  system
  LV Status              NOT available
  LV Size                2.00 TiB
  Current LE             524288
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
# lvchange -ay /dev/ernie/system
  WARNING: Not using lvmetad because a repair command was run.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
  Check of pool ernie/cache failed (status:1). Manual repair required!
# lvconvert --repair /dev/ernie/system
  WARNING: Disabling lvmetad cache for repair command.
  WARNING: Not using lvmetad because of repair.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
bad checksum in superblock
  Repair of cache metadata volume of cache ernie/system failed (status:1). Manual repair required!

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 2258 bytes --]

On Friday, 11 May 2018 19:34:42 CEST Dennis Schridde wrote:
> Hello!
> 
> I have an issue re-gaining access to my data.  The error messages are:
> # lvchange -ay ernie/system
>   Check of pool ernie/cache failed (status:1). Manual repair required!
> # lvconvert --repair ernie/system
> bad checksum in superblock
>   Repair of cache metadata volume of cache ernie/system failed (status:1).
> Manual repair required!
> 
> How do I recover from this situation and repair the volume group?
> 
> For a full log, including the output of pvdisplay, vgdisplay and lvdisplay
> -a, please see attached log file.  If more information is necessary, please
> ask.
> 
> What led to this situation:
> I am using one of the infamous AMD Ryzen 2400G with an AMD B350 chipset,
> which suffers from random lockups related to CPU C-states [1].  With the
> recent AGESA 1.0.0.2a firmware update and the introduction of the "Power
> Supply Idle Control = Typical Current Idle" setting, the system is stable,
> if it boots at all.  But it often takes several attempts to boot -- the
> failed attempts ending in weird firmware / EFI or CPU / idle related Linux
> kernel stack traces, which sometimes even require a hard reset, since
> soft-reboot (ctrl+alt +del) sometimes has no effect, because init dies. 
> This was such a situation, where I had to reboot (ctrl+alt+del) and reset
> (hard) the system several times, at the end of which everything seemed
> fine, except that dracut was timing out when activating the disks. 
> Debugging the situation using a Fedora 28 live system resulted in the
> attached log file.
> 
> --Dennis
> 
> [1]: https://bugzilla.kernel.org/show_bug.cgi?id=196683

P.S. I already found older posts [2,3] describing a similar scenario.  At the 
time a recovery was impossible for the user, but it seems that the situation 
improved somewhat since then.  However, I am still stick, with `lvconvert --
repair` asking me to repair "manually" (whatever that means), `cache_dump --
repair` not being able to operate on non-active LVs and LVM refusing to 
activate the LV as long as it has not been repaired.

[2]: https://www.redhat.com/archives/linux-lvm/2016-December/msg00013.html
[3]: https://www.redhat.com/archives/linux-lvm/2015-August/msg00008.html

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

Hello!

In case the question comes up: Fedora 28 (the live system I am trying to use 
for recovery) is using Linux 4.16.3-301.fc28 and `lvm version`:
LVM version 2.02.177(2)
Library version 1.02.146
Driver version 4.37.0

The system I was originally using to break the cache was using Linux 4.16.7-
gentoo, and `LD_LIBRARY_PATH=$PWD/lib64 ./bin/lvm version` from its initrd 
reports (running on the Fedora 28 live system):
LVM version: 2.02.173(2)
Library version: 1.02.142
Driver version: 4.37.0

Could someone please give me a hint what is actually broken here?  Is it just 
the metadata?  If so, how can that break -- shouldn't that metadata never be 
modified after creating the LV?  And could it not be recovered from /etc/lvm/
archive?  What does "manual repair" mean in detail?  Is there some way to 
recover the cache?  Or is it at least possible to uncache the LV forcibly, to 
hopefully recover the data on the origin LV?  What is your recommendation to 
minimise data loss?

Best regards,
Dennis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Zdenek Kabelac]

Dne 15.5.2018 v 10:11 Dennis Schridde napsal(a):
> Hello!
> 
> In case the question comes up: Fedora 28 (the live system I am trying to use
> for recovery) is using Linux 4.16.3-301.fc28 and `lvm version`:
> LVM version 2.02.177(2)
> Library version 1.02.146
> Driver version 4.37.0
> 
> The system I was originally using to break the cache was using Linux 4.16.7-
> gentoo, and `LD_LIBRARY_PATH=$PWD/lib64 ./bin/lvm version` from its initrd
> reports (running on the Fedora 28 live system):
> LVM version: 2.02.173(2)
> Library version: 1.02.142
> Driver version: 4.37.0
> 
> Could someone please give me a hint what is actually broken here?  Is it just
> the metadata?  If so, how can that break -- shouldn't that metadata never be
> modified after creating the LV?  And could it not be recovered from /etc/lvm/
> archive?  What does "manual repair" mean in detail?  Is there some way to
> recover the cache?  Or is it at least possible to uncache the LV forcibly, to
> hopefully recover the data on the origin LV?  What is your recommendation to
> minimise data loss?


Hi


To get direct access to metadata - you could use your latest lvm2 2.02.177 
build (or even 'git master'). In these recent versions there is added support 
to activate directly these 'subLVs'.  So with latest lvm2 you can:

lvchange -ay  vg/lv_cmeta

and then you can capture content of this LV via 'dd' into file
and compress and attach 'xz' compressed to BZ.

(dd if=/dev/vg/lv_cmeta of=/tmp/file)

----

With older lvm2 you would need to 'swap-in'  temporary LV as metadata, which 
has been also documented several in the list and man pages for thin-pool.
But since you already have 2.02.177 - you should be able to activate _cmeta LV 
directly.

Regards


Zdenek

Re: [linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 522 bytes --]

On Tuesday, 15 May 2018 13:15:55 CEST Zdenek Kabelac wrote:
> Dne 15.5.2018 v 10:11 Dennis Schridde napsal(a):
> To get direct access to metadata - you could use your latest lvm2 2.02.177
> build (or even 'git master'). In these recent versions there is added
> support to activate directly these 'subLVs'.  So with latest lvm2 you can:
> 
> lvchange -ay  vg/lv_cmeta

I tried that (using 2.02.177), but it does not work:
# lvchange -ay ernie/cache_cmeta
  Operation not permitted on hidden LV ernie/cache_cmeta.

--Dennis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 1558 bytes --]

On Wednesday, 16 May 2018 20:31:15 CEST Dennis Schridde wrote:
> On Tuesday, 15 May 2018 13:15:55 CEST Zdenek Kabelac wrote:
> > Dne 15.5.2018 v 10:11 Dennis Schridde napsal(a):
> > To get direct access to metadata - you could use your latest lvm2 2.02.177
> > build (or even 'git master'). In these recent versions there is added
> > support to activate directly these 'subLVs'.  So with latest lvm2 you can:
> > 
> > lvchange -ay  vg/lv_cmeta
> 
> I tried that (using 2.02.177), but it does not work:
> # lvchange -ay ernie/cache_cmeta
>   Operation not permitted on hidden LV ernie/cache_cmeta.

Does anyone have a suggestion how I can activate this volume in order to 
extract the information Zdenek asked for?

For more context please see the start of this thread.  The gist is that my 
cached LV cannot be activated anymore, and `lvconvert --repair` reports:
bad checksum in superblock
  Repair of cache metadata volume of cache ernie/system failed (status:1). 
Manual repair required!

My most important questions are:
* What is broken?
  - What information does the superblock carry / what is its purpose?
  - Where is it located / which part of my disk was damaged?
  - What will be the consequence of it being irrecoverably lost?
* What does "manual repair" mean in detail?
  - Using a specific tool?
  - Flipping bits using a hex editor?
* Is there some way to recover the cache?  Or is it at least possible to 
uncache the LV forcibly, to hopefully recover the data on the origin LV?
* What is your recommendation to minimise data loss?

--Dennis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 2045 bytes --]

On Wednesday, 23 May 2018 21:39:47 CEST Dennis Schridde wrote:
> On Wednesday, 16 May 2018 20:31:15 CEST Dennis Schridde wrote:
> > On Tuesday, 15 May 2018 13:15:55 CEST Zdenek Kabelac wrote:
> > > Dne 15.5.2018 v 10:11 Dennis Schridde napsal(a):
> > > To get direct access to metadata - you could use your latest lvm2
> > > 2.02.177
> > > build (or even 'git master'). In these recent versions there is added
> > > support to activate directly these 'subLVs'.  So with latest lvm2 you
> > > can:
> > > 
> > > lvchange -ay  vg/lv_cmeta
> > 
> > I tried that (using 2.02.177), but it does not work:
> > # lvchange -ay ernie/cache_cmeta
> > 
> >   Operation not permitted on hidden LV ernie/cache_cmeta.
> 
> Does anyone have a suggestion how I can activate this volume in order to
> extract the information Zdenek asked for?
> 
> For more context please see the start of this thread.  The gist is that my
> cached LV cannot be activated anymore, and `lvconvert --repair` reports:
> bad checksum in superblock
>   Repair of cache metadata volume of cache ernie/system failed (status:1).
> Manual repair required!
> 
> My most important questions are:
> * What is broken?
>   - What information does the superblock carry / what is its purpose?
>   - Where is it located / which part of my disk was damaged?
>   - What will be the consequence of it being irrecoverably lost?
> * What does "manual repair" mean in detail?
>   - Using a specific tool?
>   - Flipping bits using a hex editor?
> * Is there some way to recover the cache?  Or is it at least possible to
> uncache the LV forcibly, to hopefully recover the data on the origin LV?
> * What is your recommendation to minimise data loss?

How would I create a backup of the affected LVs, so that dangerous commands 
would not destroy the data and get me into an unrecoverable state?  Is that 
possible without activating the volumes?  Is it possible to force-activate the 
volumes, without changing the bits on the disk -- just so that I can read from 
the LV and create the backup?

--Dennis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

Hello!

On Tuesday, 15 May 2018 13:15:55 CEST Zdenek Kabelac wrote:
> To get direct access to metadata - you could use your latest lvm2 2.02.177
> build (or even 'git master'). In these recent versions there is added
> support to activate directly these 'subLVs'.  So with latest lvm2 you can:
> 
> lvchange -ay  vg/lv_cmeta

This command was only successful in LVM 2.02.178-rc1, but failed in 2.02.177.

> and then you can capture content of this LV via 'dd' into file
> and compress and attach 'xz' compressed to BZ.

I created a report in bugzilla [4], but since the `cache_cmeta` LV appears to 
contain parts of my personal files, I cannot attach it.  However, I gained 
access to the metadata LV now and would be able to provide you access to 
certain specific parts of it, if requested.

My goal is still to gain access to as much of the data as possible.

--Dennis

[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1585670

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Marian Csontos]

On 06/04/2018 01:50 PM, Dennis Schridde wrote:
> Hello!
> 
> On Tuesday, 15 May 2018 13:15:55 CEST Zdenek Kabelac wrote:
>> To get direct access to metadata - you could use your latest lvm2 2.02.177
>> build (or even 'git master'). In these recent versions there is added
>> support to activate directly these 'subLVs'.  So with latest lvm2 you can:
>>
>> lvchange -ay  vg/lv_cmeta
> 
> This command was only successful in LVM 2.02.178-rc1, but failed in 2.02.177.
> 
>> and then you can capture content of this LV via 'dd' into file
>> and compress and attach 'xz' compressed to BZ.
> 
> I created a report in bugzilla [4], but since the `cache_cmeta` LV appears to
> contain parts of my personal files, I cannot attach it.  However, I gained

It definitely should not. _cdata is where fragments you your data are 
stored, and _cmeta contains only metadata (e.g. counters and references 
to cdata.)

If there really are fragments of data in _cmeta LV, something must have 
gone wrong elsewhere.

> access to the metadata LV now and would be able to provide you access to
> certain specific parts of it, if requested.
> 
> My goal is still to gain access to as much of the data as possible.
> 
> --Dennis
> 
> [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1585670
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] Check of pool ernie/cache failed (status:1). Manual repair required!

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

On Tuesday, 5 June 2018 10:14:16 CEST Marian Csontos wrote:
> On 06/04/2018 01:50 PM, Dennis Schridde wrote:
> > On Tuesday, 15 May 2018 13:15:55 CEST Zdenek Kabelac wrote:
> >> To get direct access to metadata - you could use your latest lvm2
> >> 2.02.177
> >> build (or even 'git master'). In these recent versions there is added
> >> support to activate directly these 'subLVs'.  So with latest lvm2 you
> >> can:
> >> 
> >> lvchange -ay  vg/lv_cmeta
> > 
> > This command was only successful in LVM 2.02.178-rc1, but failed in
> > 2.02.177.> 
> >> and then you can capture content of this LV via 'dd' into file
> >> and compress and attach 'xz' compressed to BZ.
> > 
> > I created a report in bugzilla [4], but since the `cache_cmeta` LV appears
> > to contain parts of my personal files, I cannot attach it.  However, I
> > gained
> It definitely should not. _cdata is where fragments you your data are
> stored, and _cmeta contains only metadata (e.g. counters and references
> to cdata.)
> 
> If there really are fragments of data in _cmeta LV, something must have
> gone wrong elsewhere.

Thanks for the insight.  I attached the first 4890783 bytes (i.e. those before 
the blocks of seemingly unstructured data, which I saw in `hexdump -C`) of 
that LV to the bug report [4].  Hopefully it contains enough information to 
figure out what is broken.

[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1585670

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]