[linux-lvm] LVM Thin - bad checksum in superblock - cannot perform operations on metadata

[linux-lvm] LVM Thin - bad checksum in superblock - cannot perform operations on metadata

[pavel olenev]

[-- Attachment #1.1: Type: text/plain, Size: 3016 bytes --]

Hello.
I am looking for advice on how to act in such a situation. Perhaps anyone
has gone through something similar?
We used lvm thin provisioning on one of our arrays. But after raid
controller failure we got problem with activation of all thin volumes:

# lvchange -a y data/thin
  Check of pool data/thin failed (status:1). Manual repair required!

I tried to use lvconvert --repair data/thin, but have this:

# lvconvert --repair data/thin
truncating metadata device to 4161600 4k blocks
bad checksum in superblock, wanted 1494954599
  Repair of thin metadata volume of thin pool data/thin failed (status:1).
Manual repair required!

I have seen recipes related to thin_dump but I cannot use them, since in
particular thin_dump works with a mounted device. Аnd I, due to a faulty
superblock, cannot achieve the presence of the LVM partitions in
/dev/mapper/ for working such a device

Are there any chances of a successful recovery? And what can be done in
such a situation?

Best regards,
Pavel

P.S. And some output:

# pvscan
  PV /dev/sdc1   VG data            lvm2 [<10.90 TiB / <4.87 TiB free]

# vgs
  VG   #PV #LV #SN Attr   VSize    VFree
  data   1  11   0 wz--n-  <10.90t <4.87t

# lvs -a
  LV              VG   Attr       LSize   Pool Origin Data%  Meta%  Move
Log Cpy%Sync Convert
  [lvol0_pmspare] data ewi-------  16.00g
  thin            data twi---tz--   6.00t
  [thin_tdata]    data Twi-------   6.00t
  [thin_tmeta]    data ewi-------  16.00g
  vm-105-disk-0   data Vwi---tz--  40.00g thin
  vm-105-disk-1   data Vwi---tz-- 500.00g thin
  vm-107-disk-1   data Vwi---tz--  40.00g thin
  vm-107-disk-2   data Vwi---tz--  40.00g thin
  vm-107-disk-3   data Vwi---tz-- 500.00g thin
  vm-123-disk-0   data Vwi---tz--  32.00g thin
  vm-148-disk-0   data Vwi---tz--   1.17t thin
  vm-161-disk-0   data Vwi---tz--  32.00g thin
  vm-166-disk-0   data Vwi---tz--  32.00g thin
  vm-168-disk-0   data Vwi---tz--  50.00g thin

# lvscan -a
File descriptor 7 (pipe:[224496]) leaked on lvscan invocation. Parent PID
24064: bash
  inactive          '/dev/data/thin' [6.00 TiB] inherit
  inactive          '/dev/data/vm-107-disk-1' [40.00 GiB] inherit
  inactive          '/dev/data/vm-107-disk-2' [40.00 GiB] inherit
  inactive          '/dev/data/vm-107-disk-3' [500.00 GiB] inherit
  inactive          '/dev/data/vm-105-disk-0' [40.00 GiB] inherit
  inactive          '/dev/data/vm-105-disk-1' [500.00 GiB] inherit
  inactive          '/dev/data/vm-148-disk-0' [1.17 TiB] inherit
  inactive          '/dev/data/vm-161-disk-0' [32.00 GiB] inherit
  inactive          '/dev/data/vm-166-disk-0' [32.00 GiB] inherit
  inactive          '/dev/data/vm-123-disk-0' [32.00 GiB] inherit
  inactive          '/dev/data/vm-168-disk-0' [50.00 GiB] inherit
  inactive          '/dev/data/lvol0_pmspare' [16.00 GiB] inherit
  inactive          '/dev/data/thin_tmeta' [16.00 GiB] inherit
  inactive          '/dev/data/thin_tdata' [6.00 TiB] inherit

[-- Attachment #1.2: Type: text/html, Size: 3445 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM Thin - bad checksum in superblock - cannot perform operations on metadata

[Ming-Hung Tsai]

Hi,

Could you try using thin_repair v0.9.0 please? The metadata could be
repaired if only the superblock was affected.

1. Check the pool configurations (transaction id, chunk size, pool size, etc.)

# lvs -o lv_name,transaction_id,chunk_size,size --units k

Taking the following output as an example, the tid is 13, and there
are 519936 data blocks (33275904/64).
```
  LV           TransId Chunk  LSize
  tp1               13 64.00k 33275904.00k
```

2. Create a blank LV for restoration. Here I use 16192 MB to keep
backward compatibility.

# lvcreate data1 --size 16192m --name tmeta_bak

3. Activate the metadata in read-only mode

# lvchange data1/thin_tmeta -ay -y

4. Repair the metadata, in which the tid, bs, and nr_blocks are values
from the first step.

# thin_repair -i /dev/mapper/data1-thin_meta -o
/dev/mapper/data1-tmeta_bak --transaction-id <tid> --data-block-size
<bs> --nr-data-blocks <nr_blocks>

5. Verify the repaired metadata

# thin_dump /dev/mapper/data1-tmeta_bak --skip-mappings
(10 devices should be listed if thin_repair runs successfully)

6. Swap the pool metadata with the repaired one

# lvchange -an data1/thin_meta
# lvconvert data1/thin --swapmetadata --poolmetadata data1/tmeta_bak

Step 2-6 above could be substituted by a single lvconvert command
below, but I would prefer doing these step-by-step.

# lvconvert --repair data1/thin --config
'global/thin_repair_options=["--transaction_id", "<tid>",
"--data-block-size", "<bs>", "--nr-data-blocks", "<nr_blocks>"]'


Ming-Hung Tsai

On Tue, Nov 23, 2021 at 3:33 PM pavel olenev <pavel.olenev@gmail.com> wrote:
>
> Hello.
> I am looking for advice on how to act in such a situation. Perhaps anyone has gone through something similar?
> We used lvm thin provisioning on one of our arrays. But after raid controller failure we got problem with activation of all thin volumes:
>
> # lvchange -a y data/thin
>   Check of pool data/thin failed (status:1). Manual repair required!
>
> I tried to use lvconvert --repair data/thin, but have this:
>
> # lvconvert --repair data/thin
> truncating metadata device to 4161600 4k blocks
> bad checksum in superblock, wanted 1494954599
>   Repair of thin metadata volume of thin pool data/thin failed (status:1). Manual repair required!

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/