[linux-lvm] thin: pool target too small

[linux-lvm] thin: pool target too small

[Duncan Townsend]

Hello!

I think the problem I'm having is a related problem to this thread:
https://www.redhat.com/archives/linux-lvm/2016-May/msg00092.html
continuation https://www.redhat.com/archives/linux-lvm/2016-June/msg00000.html
. In the previous thread, Zdenek Kabelac fixed the problem manually,
but there was no information about exactly what or how the problem was
fixed. I have also posted about this problem on the #lvm on freenode
and on Stack Exchange
(https://superuser.com/questions/1587224/lvm2-thin-pool-pool-target-too-small),
so my apologies to those of you who are seeing this again.

I had a problem with a runit script that caused my dmeventd to be
killed and restarted every 5 seconds. The script has been fixed, but
my lvm thin pool is still un-mountable. The following is an excerpt
from my system logs when the problem first appeared

device-mapper: thin: 253:10: reached low water mark for data device:
sending event.
lvm[1221]: WARNING: Sum of all thin volume sizes (2.81 TiB) exceeds
the size of thin pools and the size of whole volume group (1.86 TiB).
lvm[1221]: Size of logical volume
nellodee-nvme/nellodee-nvme-thin_tdata changed from 212.64 GiB (13609
extents) to <233.91 GiB (14970 extents).
device-mapper: thin: 253:10: growing the data device from 13609 to 14970 blocks
lvm[1221]: Logical volume nellodee-nvme/nellodee-nvme-thin_tdata
successfully resized.
lvm[1221]: dmeventd received break, scheduling exit.
lvm[1221]: dmeventd received break, scheduling exit.
lvm[1221]: WARNING: Thin pool
nellodee--nvme-nellodee--nvme--thin-tpool data is now 81.88% full.
<SNIP> (lots of repeats of "lvm[1221]: dmeventd received break,
scheduling exit.")
lvm[1221]: No longer monitoring thin pool
nellodee--nvme-nellodee--nvme--thin-tpool.
device-mapper: thin: 253:10: pool target (13609 blocks) too small:
expected 14970
device-mapper: table: 253:10: thin-pool: preresume failed, error = -22
lvm[1221]: dmeventd received break, scheduling exit.
(previous message repeats many times)

After this, the system became unresponsive, so I power cycled it. Upon
boot up, the following message was printed and I was dropped into an
emergency shell:

device-mapper: thin: 253:10: pool target (13609 blocks) too small:
expected 14970
device-mapper: table: 253:10: thin-pool: preresume failed, error = -22

I have tried using thin_repair, which reported success and didn't
solve the problem. I tried vgcfgrestore (using metadata backups going
back quite a ways), which also reported success and did not solve the
problem. I tried lvchange --repair. I tried lvextending the thin
volume, which reported "Cannot resize logical volume
nellodee-nvme/nellodee-nvme-thin with active component LV(s)". I tried
lvextending the underlying *_tdata LV, which reported "Can't resize
internal logical volume nellodee-nvme/nellodee-nvme-thin_tdata". I
have the LVM header, if that would be of interest to anyone.

I am at a loss here about how to proceed with fixing this problem. Is
there some flag I've missed or some tool I don't know about that I can
apply to fixing this problem? Thank you very much for your attention,
--Duncan Townsend

P.S. I cannot restore from backup. Ironically/amusingly, this happened
right in the middle of me bringing my new backup system online.

Re: [linux-lvm] thin: pool target too small

[Zdenek Kabelac]

Dne 21. 09. 20 v 1:48 Duncan Townsend napsal(a):
> Hello!
> 
> I think the problem I'm having is a related problem to this thread:
> https://www.redhat.com/archives/linux-lvm/2016-May/msg00092.html
> continuation https://www.redhat.com/archives/linux-lvm/2016-June/msg00000.html
> . In the previous thread, Zdenek Kabelac fixed the problem manually,
> but there was no information about exactly what or how the problem was
> fixed. I have also posted about this problem on the #lvm on freenode
> and on Stack Exchange
> (https://superuser.com/questions/1587224/lvm2-thin-pool-pool-target-too-small),
> so my apologies to those of you who are seeing this again.


Hi

At first it's worth to remain which version of  kernel, lvm2, thin-tools 
(d-m-p-d package on RHEL/Fedora-   aka  thin_check -V) is this.


> I had a problem with a runit script that caused my dmeventd to be
> killed and restarted every 5 seconds. The script has been fixed, but

Kill dmeventd is always BAD plan.
Either you do not want monitoring (set to 0 in lvm.conf) - or
leave it to it jobs - kill dmeventd in the middle of its work
isn't going to end well...)


> 
> device-mapper: thin: 253:10: reached low water mark for data device:
> sending event.
> lvm[1221]: WARNING: Sum of all thin volume sizes (2.81 TiB) exceeds
> the size of thin pools and the size of whole volume group (1.86 TiB).
> lvm[1221]: Size of logical volume
> nellodee-nvme/nellodee-nvme-thin_tdata changed from 212.64 GiB (13609
> extents) to <233.91 GiB (14970 extents).
> device-mapper: thin: 253:10: growing the data device from 13609 to 14970 blocks
> lvm[1221]: Logical volume nellodee-nvme/nellodee-nvme-thin_tdata
> successfully resized.

So here was successful resize -

> lvm[1221]: dmeventd received break, scheduling exit.
> lvm[1221]: dmeventd received break, scheduling exit. > lvm[1221]: WARNING: Thin pool
> nellodee--nvme-nellodee--nvme--thin-tpool data is now 81.88% full.
> <SNIP> (lots of repeats of "lvm[1221]: dmeventd received break,
> scheduling exit.")
> lvm[1221]: No longer monitoring thin pool
> nellodee--nvme-nellodee--nvme--thin-tpool.
> device-mapper: thin: 253:10: pool target (13609 blocks) too small:
> expected 14970

And now we can see the problem - the thin-pool was already upsized to bigger
size (13609 -> 14970 as seen above) - yet something has tried to activate 
thin-pool with smaller metadata volume.


> device-mapper: table: 253:10: thin-pool: preresume failed, error = -22

This is correct - it's preventing further damage of thin-pool to happen.

> lvm[1221]: dmeventd received break, scheduling exit.
> (previous message repeats many times)
> 
> After this, the system became unresponsive, so I power cycled it. Upon
> boot up, the following message was printed and I was dropped into an
> emergency shell:
> 
> device-mapper: thin: 253:10: pool target (13609 blocks) too small:
> expected 14970
> device-mapper: table: 253:10: thin-pool: preresume failed, error = -22


So the primary question is - how the LVM could have got 'smaller' metadata
back - have you played with  'vgcfgrestore' ?

So when you submit version of tools - also provide  /etc/lvm/archive
(eventually lvmdump archive)

> I have tried using thin_repair, which reported success and didn't
> solve the problem. I tried vgcfgrestore (using metadata backups going
> back quite a ways), which also reported success and did not solve the
> problem. I tried lvchange --repair. I tried lvextending the thin


'lvconvert --repair' can solve only very basic issues - it's not
able to resolve badly sized metadata device ATM.

For all other case you need to use manual repair steps.


> I am at a loss here about how to proceed with fixing this problem. Is
> there some flag I've missed or some tool I don't know about that I can
> apply to fixing this problem? Thank you very much for your attention,

I'd expect in your /etc/lvm/archive   (or in the 1st. 1MiB of your device 
header) there can be seen a history of changes of your lvm2 metadata and you 
should be able ot find when then _tmeta LV was matching your new metadata size
and maybe see when it's got previous size.

Without knowing more detail it's hard to give precise answer - but before you
will try to do some next steps of your recovery be sure you know what you
are doing - it's better to ask here the be sorry later.

Regards

Zdenek

Re: [linux-lvm] thin: pool target too small

[Duncan Townsend]

On Mon, Sep 21, 2020 at 5:23 AM Zdenek Kabelac <zkabelac@redhat.com> wrote:
>
> Dne 21. 09. 20 v 1:48 Duncan Townsend napsal(a):
> > Hello!
> >
> > I think the problem I'm having is a related problem to this thread:
> > https://www.redhat.com/archives/linux-lvm/2016-May/msg00092.html
> > continuation https://www.redhat.com/archives/linux-lvm/2016-June/msg00000.html
> > . In the previous thread, Zdenek Kabelac fixed the problem manually,
> > but there was no information about exactly what or how the problem was
> > fixed. I have also posted about this problem on the #lvm on freenode
> > and on Stack Exchange
> > (https://superuser.com/questions/1587224/lvm2-thin-pool-pool-target-too-small),
> > so my apologies to those of you who are seeing this again.
>
>
> Hi
>
> At first it's worth to remain which version of  kernel, lvm2, thin-tools
> (d-m-p-d package on RHEL/Fedora-   aka  thin_check -V) is this.

Ahh, thank you for the reminder. My apologies for not including this
in my original message. I use Void Linux on aarch64-musl:

# uname -a
Linux (none) 5.7.0_1 #1 SMP Thu Aug 6 20:19:56 UTC 2020 aarch64 GNU/Linux

# lvm version
  LVM version:     2.02.187(2) (2020-03-24)
  Library version: 1.02.170 (2020-03-24)
  Driver version:  4.42.0
  Configuration:   ./configure --prefix=/usr --sysconfdir=/etc
--sbindir=/usr/bin --bindir=/usr/bin --mandir=/usr/share/man
--infodir=/usr/share/info --localstatedir=/var --disable-selinux
--enable-readline --enable-pkgconfig --enable-fsadm --enable-applib
--enable-dmeventd --enable-cmdlib --enable-udev_sync
--enable-udev_rules --enable-lvmetad
--with-udevdir=/usr/lib/udev/rules.d --with-default-pid-dir=/run
--with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm
--with-default-locking-dir=/run/lock/lvm --enable-static_link
--host=x86_64-unknown-linux-musl --build=x86_64-unknown-linux-musl
--host=aarch64-linux-musl --with-sysroot=/usr/aarch64-linux-musl
--with-libtool-sysroot=/usr/aarch64-linux-musl

# thin_check -V
0.8.5

> > I had a problem with a runit script that caused my dmeventd to be
> > killed and restarted every 5 seconds. The script has been fixed, but
>
> Kill dmeventd is always BAD plan.
> Either you do not want monitoring (set to 0 in lvm.conf) - or
> leave it to it jobs - kill dmeventd in the middle of its work
> isn't going to end well...)

Thank you for reinforcing this. My runit script was fighting with
dracut in my initramfs. My runit script saw that there was a dmeventd
not under its control, and so tried to kill the one started by dracut.
I've gone and disabled the runit script and replaced it with a stub
that simply tried to kill the dracut-started dmeventd when it receives
a signal.

> > device-mapper: thin: 253:10: reached low water mark for data device:
> > sending event.
> > lvm[1221]: WARNING: Sum of all thin volume sizes (2.81 TiB) exceeds
> > the size of thin pools and the size of whole volume group (1.86 TiB).
> > lvm[1221]: Size of logical volume
> > nellodee-nvme/nellodee-nvme-thin_tdata changed from 212.64 GiB (13609
> > extents) to <233.91 GiB (14970 extents).
> > device-mapper: thin: 253:10: growing the data device from 13609 to 14970 blocks
> > lvm[1221]: Logical volume nellodee-nvme/nellodee-nvme-thin_tdata
> > successfully resized.
>
> So here was successful resize -
>
> > lvm[1221]: dmeventd received break, scheduling exit.
> > lvm[1221]: dmeventd received break, scheduling exit. > lvm[1221]: WARNING: Thin pool
> > nellodee--nvme-nellodee--nvme--thin-tpool data is now 81.88% full.
> > <SNIP> (lots of repeats of "lvm[1221]: dmeventd received break,
> > scheduling exit.")
> > lvm[1221]: No longer monitoring thin pool
> > nellodee--nvme-nellodee--nvme--thin-tpool.
> > device-mapper: thin: 253:10: pool target (13609 blocks) too small:
> > expected 14970
>
> And now we can see the problem - the thin-pool was already upsized to bigger
> size (13609 -> 14970 as seen above) - yet something has tried to activate
> thin-pool with smaller metadata volume.

I think what happened here is that the dmeventd started by dracut
finally exited, and then the dmeventd started by runit takes over.
Then the started-by-runit dmevent and tries to activate the thin-pool
which is in the process of being resized?

> > device-mapper: table: 253:10: thin-pool: preresume failed, error = -22
>
> This is correct - it's preventing further damage of thin-pool to happen.
>
> > lvm[1221]: dmeventd received break, scheduling exit.
> > (previous message repeats many times)
> >
> > After this, the system became unresponsive, so I power cycled it. Upon
> > boot up, the following message was printed and I was dropped into an
> > emergency shell:
> >
> > device-mapper: thin: 253:10: pool target (13609 blocks) too small:
> > expected 14970
> > device-mapper: table: 253:10: thin-pool: preresume failed, error = -22
>
>
> So the primary question is - how the LVM could have got 'smaller' metadata
> back - have you played with  'vgcfgrestore' ?
>
> So when you submit version of tools - also provide  /etc/lvm/archive
> (eventually lvmdump archive)

Yes, I have tried making significant use of vgcfgrestore. I make
extensive use of snapshots in my backup system, so my /etc/lvm/archive
has many entries. Restoring the one from just before the lvextend call
that triggered this mess has not fixed my problem.

> > I have tried using thin_repair, which reported success and didn't
> > solve the problem. I tried vgcfgrestore (using metadata backups going
> > back quite a ways), which also reported success and did not solve the
> > problem. I tried lvchange --repair. I tried lvextending the thin
>
>
> 'lvconvert --repair' can solve only very basic issues - it's not
> able to resolve badly sized metadata device ATM.
>
> For all other case you need to use manual repair steps.
>
>
> > I am at a loss here about how to proceed with fixing this problem. Is
> > there some flag I've missed or some tool I don't know about that I can
> > apply to fixing this problem? Thank you very much for your attention,
>
> I'd expect in your /etc/lvm/archive   (or in the 1st. 1MiB of your device
> header) there can be seen a history of changes of your lvm2 metadata and you
> should be able ot find when then _tmeta LV was matching your new metadata size
> and maybe see when it's got previous size.

I've replied privately with a tarball of my /etc/lvm/archive and the
lvm header. If I should send them to the broader list, I'll do that
too, but I want to be respectful of the size of what I drop in
people's inboxes.

> Without knowing more detail it's hard to give precise answer - but before you
> will try to do some next steps of your recovery be sure you know what you
> are doing - it's better to ask here the be sorry later.
>
> Regards
>
> Zdenek

Thank you so much for your help. I appreciate it very much!
--Duncan Townsend

Re: [linux-lvm] thin: pool target too small

[Zdenek Kabelac]

Dne 21. 09. 20 v 15:47 Duncan Townsend napsal(a):
> On Mon, Sep 21, 2020 at 5:23 AM Zdenek Kabelac <zkabelac@redhat.com> wrote:
>>
>> Dne 21. 09. 20 v 1:48 Duncan Townsend napsal(a):
>>> Hello!
> 
> Ahh, thank you for the reminder. My apologies for not including this
> in my original message. I use Void Linux on aarch64-musl:
> 
>>> I had a problem with a runit script that caused my dmeventd to be
>>> killed and restarted every 5 seconds. The script has been fixed, but
>>
>> Kill dmeventd is always BAD plan.
>> Either you do not want monitoring (set to 0 in lvm.conf) - or
>> leave it to it jobs - kill dmeventd in the middle of its work
>> isn't going to end well...)
> 
> Thank you for reinforcing this. My runit script was fighting with
> dracut in my initramfs. My runit script saw that there was a dmeventd
> not under its control, and so tried to kill the one started by dracut.
> I've gone and disabled the runit script and replaced it with a stub
> that simply tried to kill the dracut-started dmeventd when it receives
> a signal.

Ok - so from looking at the resulting 'mixture' of metadata you
have in your archive and what physically present on PV header are
and now noticing this is some 'not-so-standard' distro - I'm starting to 
suspect the reason of all these troubles.

Withing 'dracut' you shouldn't be firering dmeventd at all - monitoring
should be enabled  (by vgchange --monitor y) once you switch to your rootfs
so dmenventd is execute from your rootfs!

By letting 'dmeventd' running in your 'dracut world' - it lives in its
own environment and likely its very own locking dir.

That makes it very easy your dmeventd runs in parallel with your other lvm2 
commands - and since this way it's completely unprotected
(as file-locking is what it is) - as the resize is  operation for several
seconds it has happened your 'admins' command replaced whatever dmeventd was
doing.

So I think to prevent repeated occurrence of this problem - you'll need
to ensure your system-booting will follow the pattern from distros
like Fedora.

Regards

Zdenek

Re: [linux-lvm] thin: pool target too small

[Duncan Townsend]

[-- Attachment #1: Type: text/plain, Size: 3328 bytes --]

On Tue, Sep 22, 2020, 5:02 PM Zdenek Kabelac <zkabelac@redhat.com> wrote:

> Dne 21. 09. 20 v 15:47 Duncan Townsend napsal(a):
> > On Mon, Sep 21, 2020 at 5:23 AM Zdenek Kabelac <zkabelac@redhat.com>
> wrote:
> >>
> >> Dne 21. 09. 20 v 1:48 Duncan Townsend napsal(a):
> >>> Hello!
> >
> > Ahh, thank you for the reminder. My apologies for not including this
> > in my original message. I use Void Linux on aarch64-musl:
> >
> >>> I had a problem with a runit script that caused my dmeventd to be
> >>> killed and restarted every 5 seconds. The script has been fixed, but
> >>
> >> Kill dmeventd is always BAD plan.
> >> Either you do not want monitoring (set to 0 in lvm.conf) - or
> >> leave it to it jobs - kill dmeventd in the middle of its work
> >> isn't going to end well...)
> >
> > Thank you for reinforcing this. My runit script was fighting with
> > dracut in my initramfs. My runit script saw that there was a dmeventd
> > not under its control, and so tried to kill the one started by dracut.
> > I've gone and disabled the runit script and replaced it with a stub
> > that simply tried to kill the dracut-started dmeventd when it receives
> > a signal.
>
> Ok - so from looking at the resulting 'mixture' of metadata you
> have in your archive and what physically present on PV header are
> and now noticing this is some 'not-so-standard' distro - I'm starting to
> suspect the reason of all these troubles.
>

I'm running void linux, and I haven't messed with the initramfs at all from
the defaults. I'll report this to the distro maintainers. Thanks!

Withing 'dracut' you shouldn't be firering dmeventd at all - monitoring
> should be enabled  (by vgchange --monitor y) once you switch to your rootfs
> so dmenventd is execute from your rootfs!
>
> By letting 'dmeventd' running in your 'dracut world' - it lives in its
> own environment and likely its very own locking dir.
>
> That makes it very easy your dmeventd runs in parallel with your other
> lvm2
> commands - and since this way it's completely unprotected
> (as file-locking is what it is) - as the resize is  operation for several
> seconds it has happened your 'admins' command replaced whatever dmeventd
> was
> doing.
>

dmeventd does write its PID file into the correct directory in the
post-initramfs root, so whatever's happening is some weird hybrid. I'll
debug this further with my distro.

So I think to prevent repeated occurrence of this problem - you'll need
> to ensure your system-booting will follow the pattern from distros
> like Fedora.
>

I think for now, the easiest solution may be to try to stop dmeventd from
being started by dracut.

I have encountered a further problem in the process of restoring my thin
pool to a working state. After using vgcfgrestore to fix the mismatching
metadata using the file Zdenek kindly provided privately, when I try to
activate my thin LVs, I'm now getting the error message:

Thin pool <THIN POOL LONG NAME>-tpool transaction_id (MAJOR:MINOR)
transaction_id is XXX, while expected YYY.

where XXX == YYY + 2. Is this indicative of a deeper problem? I'm hesitant
to run any further operations on this volume without advice because I think
I'm in fairly uncharted territory now.

Thank you again for all your help!
--Duncan Townsend

P.S. This was typed on mobile. Please forgive any typos.

>

[-- Attachment #2: Type: text/html, Size: 5405 bytes --]

Re: [linux-lvm] thin: pool target too small

[Zdenek Kabelac]

Dne 23. 09. 20 v 20:13 Duncan Townsend napsal(a):
> On Tue, Sep 22, 2020, 5:02 PM Zdenek Kabelac <zkabelac@redhat.com 
> 
> dmeventd does write its PID file into the correct directory in the 
> post-initramfs root, so whatever's happening is some weird hybrid. I'll debug 
> this further with my distro.
> 
>     So I think to prevent repeated occurrence of this problem - you'll need
>     to ensure your system-booting will follow the pattern from distros
>     like Fedora.
> 
> 
> I think for now, the easiest solution may be to try to stop dmeventd from 
> being started by dracut.

Basically all you need to do for dracut (with reagards to dmeventd) is to 
setup inside dracut environemnt  'monitoring=0'  in /etc/lvm/lvm.conf there.
(so when it's copied system's lvm.conf there - replace with sed/awk...)

Also there is   'metadata_read_only=1' setting that can be useful for
dracut environment.

Dracut needs some bigger fixing on its own - but ATM we simply can't
provide set of features we would like to have.

> I have encountered a further problem in the process of restoring my thin pool 
> to a working state. After using vgcfgrestore to fix the mismatching metadata 
> using the file Zdenek kindly provided privately, when I try to activate my 
> thin LVs, I'm now getting the error message:
> 
> Thin pool <THIN POOL LONG NAME>-tpool transaction_id (MAJOR:MINOR) 
> transaction_id is XXX, while expected YYY.
Set the transaction_id to the right number in the ASCII lvm2 metadata file.

Zdenek

Re: [linux-lvm] thin: pool target too small

[Duncan Townsend]

On Wed, Sep 23, 2020 at 2:49 PM Zdenek Kabelac <zkabelac@redhat.com> wrote:
>
> Dne 23. 09. 20 v 20:13 Duncan Townsend napsal(a):
> > On Tue, Sep 22, 2020, 5:02 PM Zdenek Kabelac <zkabelac@redhat.com
> >
> > dmeventd does write its PID file into the correct directory in the
> > post-initramfs root, so whatever's happening is some weird hybrid. I'll debug
> > this further with my distro.
> >
> >     So I think to prevent repeated occurrence of this problem - you'll need
> >     to ensure your system-booting will follow the pattern from distros
> >     like Fedora.
> >
> >
> > I think for now, the easiest solution may be to try to stop dmeventd from
> > being started by dracut.
>
> Basically all you need to do for dracut (with reagards to dmeventd) is to
> setup inside dracut environemnt  'monitoring=0'  in /etc/lvm/lvm.conf there.
> (so when it's copied system's lvm.conf there - replace with sed/awk...)
>
> Also there is   'metadata_read_only=1' setting that can be useful for
> dracut environment.

Thanks for the advice!

> Dracut needs some bigger fixing on its own - but ATM we simply can't
> provide set of features we would like to have.
>
> > I have encountered a further problem in the process of restoring my thin pool
> > to a working state. After using vgcfgrestore to fix the mismatching metadata
> > using the file Zdenek kindly provided privately, when I try to activate my
> > thin LVs, I'm now getting the error message:
> >
> > Thin pool <THIN POOL LONG NAME>-tpool transaction_id (MAJOR:MINOR)
> > transaction_id is XXX, while expected YYY.
> Set the transaction_id to the right number in the ASCII lvm2 metadata file.

I apologize, but I am back with a related, similar problem. After
editing the metadata file and replacing the transaction number, my
system became serviceable again. After making absolutely sure that
dmeventd was running correctly, my next order of business was to
finish backing up before any other tragedy happens. Unfortunately,
taking a snapshot as part of the backup process has once again brought
my system to its knees. The first error message I saw was:

  WARNING: Sum of all thin volume sizes (XXX TiB) exceeds the size of
thin pool <VG>/<THIN POOL LV> and the size of whole volume group (YYY
TiB).
  device-mapper: message ioctl on  (MAJOR:MINOR) failed: File exists
  Failed to process thin pool message "create_snap 11 4".
  Failed to suspend thin snapshot origin <VG>/<THIN LV>.
  Internal error: Writing metadata in critical section.
  Releasing activation in critical section.
  libdevmapper exiting with 1 device(s) still suspended.

There were further error messages as further snapshots were attempted,
but I was unable to capture them as my system went down. Upon reboot,
the "transaction_id" message that I referred to in my previous message
was repeated (but with increased transaction IDs).

I will reply privately with my lvm metadata archive and with my
header. My profuse thanks, again, for assisting me getting my system
back up and running.
--Duncan Townsend

Re: [linux-lvm] thin: pool target too small

[Zdenek Kabelac]

Dne 23. 09. 20 v 21:54 Duncan Townsend napsal(a):
> On Wed, Sep 23, 2020 at 2:49 PM Zdenek Kabelac <zkabelac@redhat.com> wrote:
>>
>> Dne 23. 09. 20 v 20:13 Duncan Townsend napsal(a):
>>> On Tue, Sep 22, 2020, 5:02 PM Zdenek Kabelac <zkabelac@redhat.com
>>> I have encountered a further problem in the process of restoring my thin pool
>>> to a working state. After using vgcfgrestore to fix the mismatching metadata
>>> using the file Zdenek kindly provided privately, when I try to activate my
>>> thin LVs, I'm now getting the error message:
>>>
>>> Thin pool <THIN POOL LONG NAME>-tpool transaction_id (MAJOR:MINOR)
>>> transaction_id is XXX, while expected YYY.
>> Set the transaction_id to the right number in the ASCII lvm2 metadata file.
> 
> I apologize, but I am back with a related, similar problem. After
> editing the metadata file and replacing the transaction number, my
> system became serviceable again. After making absolutely sure that
> dmeventd was running correctly, my next order of business was to
> finish backing up before any other tragedy happens. Unfortunately,
> taking a snapshot as part of the backup process has once again brought
> my system to its knees. The first error message I saw was:

Hi

And now you've hit an interesting bug inside lvm2 code - I've opened new BZ

https://bugzilla.redhat.com/show_bug.cgi?id=1882483

This actually explains few so far not well understood problems I've
seen before without good explanation how to hit them.

>    WARNING: Sum of all thin volume sizes (XXX TiB) exceeds the size of
> thin pool <VG>/<THIN POOL LV> and the size of whole volume group (YYY
> TiB).
>    device-mapper: message ioctl on  (MAJOR:MINOR) failed: File exists
>    Failed to process thin pool message "create_snap 11 4".
>    Failed to suspend thin snapshot origin <VG>/<THIN LV>.
>    Internal error: Writing metadata in critical section.
>    Releasing activation in critical section.
>    libdevmapper exiting with 1 device(s) still suspended.

So I've now quite simple reproducer for unhanded error case.
It's basically exposing mismatch between kernel (_tmeta) and lvm2
metadata content.  And lvm2 can handle this discovery better
than what you see now,

> There were further error messages as further snapshots were attempted,
> but I was unable to capture them as my system went down. Upon reboot,
> the "transaction_id" message that I referred to in my previous message
> was repeated (but with increased transaction IDs).

For better fix it would need to be better understood what has happened
in parallel while 'lvm' inside dmeventd was resizing pool data.

It looks like the 'other' lvm managed to create another snapshot
(and thus the DeviceID appeared to already exists - while it should not
according to lvm2 metadata before it hit problem with mismatch of
transaction_id.

> I will reply privately with my lvm metadata archive and with my
> header. My profuse thanks, again, for assisting me getting my system
> back up and running.

So the valid fix would be to take 'thin_dump' of kernel metadata
(aka content of _tmeta device)
Then check what you have in lvm2 metadata and likely you will
find some device in kernel - for which you don't have match
in lvm2 metadata -  these devices would need to be copied
from your other sequence of lvm2 metadata.

Other maybe more simple way could be to just remove devices
from xml thin_dump and thin_restore those metadata that should should
now match lvm2.

The last issue is then to match 'transaction_id' with the number
stored in kernel metadata.

So not sure which way you want to go and how important those
snapshot (that could be dropped) are ?

Zdenek

Re: [linux-lvm] thin: pool target too small

[Duncan Townsend]

On Thu, Sep 24, 2020 at 1:54 PM Zdenek Kabelac <zkabelac@redhat.com> wrote:
> Dne 23. 09. 20 v 21:54 Duncan Townsend napsal(a):
> > On Wed, Sep 23, 2020 at 2:49 PM Zdenek Kabelac <zkabelac@redhat.com> wrote:
> >> Dne 23. 09. 20 v 20:13 Duncan Townsend napsal(a):
> > I apologize, but I am back with a related, similar problem. After
> > editing the metadata file and replacing the transaction number, my
> > system became serviceable again. After making absolutely sure that
> > dmeventd was running correctly, my next order of business was to
> > finish backing up before any other tragedy happens. Unfortunately,
> > taking a snapshot as part of the backup process has once again brought
> > my system to its knees. The first error message I saw was:
>
> Hi
>
> And now you've hit an interesting bug inside lvm2 code - I've opened new BZ
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1882483
>
> This actually explains few so far not well understood problems I've
> seen before without good explanation how to hit them.

Ahh! Exciting! Thanks for opening the bugzilla. Is there any
additional information I could supply to make this easier?

> >    WARNING: Sum of all thin volume sizes (XXX TiB) exceeds the size of
> > thin pool <VG>/<THIN POOL LV> and the size of whole volume group (YYY
> > TiB).
> >    device-mapper: message ioctl on  (MAJOR:MINOR) failed: File exists
> >    Failed to process thin pool message "create_snap 11 4".
> >    Failed to suspend thin snapshot origin <VG>/<THIN LV>.
> >    Internal error: Writing metadata in critical section.
> >    Releasing activation in critical section.
> >    libdevmapper exiting with 1 device(s) still suspended.
>
> So I've now quite simple reproducer for unhanded error case.
> It's basically exposing mismatch between kernel (_tmeta) and lvm2
> metadata content.  And lvm2 can handle this discovery better
> than what you see now,
>
> > There were further error messages as further snapshots were attempted,
> > but I was unable to capture them as my system went down. Upon reboot,
> > the "transaction_id" message that I referred to in my previous message
> > was repeated (but with increased transaction IDs).
>
> For better fix it would need to be better understood what has happened
> in parallel while 'lvm' inside dmeventd was resizing pool data.

To the best of my knowledge, no other LVM operations were in flight at
the time. The script that I use issues LVM commands strictly
sequentially and there are no other daemons that make use of LVM
(except dmeventd and lvmetad, of course). Is there something else I
should be looking out for? Is there another log that I can examine?

> It looks like the 'other' lvm managed to create another snapshot
> (and thus the DeviceID appeared to already exists - while it should not
> according to lvm2 metadata before it hit problem with mismatch of
> transaction_id.
>
> > I will reply privately with my lvm metadata archive and with my
> > header. My profuse thanks, again, for assisting me getting my system
> > back up and running.
>
> So the valid fix would be to take 'thin_dump' of kernel metadata
> (aka content of _tmeta device)
> Then check what you have in lvm2 metadata and likely you will
> find some device in kernel - for which you don't have match
> in lvm2 metadata -  these devices would need to be copied
> from your other sequence of lvm2 metadata.

I'll reply privately with the compressed output of thin_dump.

> Other maybe more simple way could be to just remove devices
> from xml thin_dump and thin_restore those metadata that should should
> now match lvm2.
>
> The last issue is then to match 'transaction_id' with the number
> stored in kernel metadata.
>
> So not sure which way you want to go and how important those
> snapshot (that could be dropped) are ?

Would it be reasonable to use vgcfgrestore again on the
manually-repaired metadata I used before? I'm not entirely sure what
to look for while editing the XML from thin_dump, and I would very
much like to avoid causing further damage to my system. (Also, FWIW,
thin_dump appears to segfault when run with musl-libc instead of
glibc. I had to use a glibc recovery image to get this dump. I'm
following up with the void distro maintainers about this.)

I plan to "just" create a new thin pool and copy all the thin LVs over
to the new pool. I'm hoping that that will let me wash my hands of
this matter and restore my system to a stable state. The snapshots
that I create are purely read-only so that the slow-moving backup
process can get an atomic view of the filesystem in use. They contain
no important information that isn't also captured in the origin LV
itself.

Thank you!
--Duncan Townsend

Re: [linux-lvm] thin: pool target too small

[Duncan Townsend]

[-- Attachment #1: Type: text/plain, Size: 5349 bytes --]

On Sat, Sep 26, 2020, 8:30 AM Duncan Townsend <duncancmt@gmail.com> wrote:

> On Thu, Sep 24, 2020 at 1:54 PM Zdenek Kabelac <zkabelac@redhat.com>
> wrote:
> > Dne 23. 09. 20 v 21:54 Duncan Townsend napsal(a):
> > > On Wed, Sep 23, 2020 at 2:49 PM Zdenek Kabelac <zkabelac@redhat.com>
> wrote:
> > >> Dne 23. 09. 20 v 20:13 Duncan Townsend napsal(a):
> > > I apologize, but I am back with a related, similar problem. After
> > > editing the metadata file and replacing the transaction number, my
> > > system became serviceable again. After making absolutely sure that
> > > dmeventd was running correctly, my next order of business was to
> > > finish backing up before any other tragedy happens. Unfortunately,
> > > taking a snapshot as part of the backup process has once again brought
> > > my system to its knees. The first error message I saw was:
> >
> > Hi
> >
> > And now you've hit an interesting bug inside lvm2 code - I've opened new
> BZ
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1882483
> >
> > This actually explains few so far not well understood problems I've
> > seen before without good explanation how to hit them.
>
> Ahh! Exciting! Thanks for opening the bugzilla. Is there any
> additional information I could supply to make this easier?
>
> > >    WARNING: Sum of all thin volume sizes (XXX TiB) exceeds the size of
> > > thin pool <VG>/<THIN POOL LV> and the size of whole volume group (YYY
> > > TiB).
> > >    device-mapper: message ioctl on  (MAJOR:MINOR) failed: File exists
> > >    Failed to process thin pool message "create_snap 11 4".
> > >    Failed to suspend thin snapshot origin <VG>/<THIN LV>.
> > >    Internal error: Writing metadata in critical section.
> > >    Releasing activation in critical section.
> > >    libdevmapper exiting with 1 device(s) still suspended.
> >
> > So I've now quite simple reproducer for unhanded error case.
> > It's basically exposing mismatch between kernel (_tmeta) and lvm2
> > metadata content.  And lvm2 can handle this discovery better
> > than what you see now,
> >
> > > There were further error messages as further snapshots were attempted,
> > > but I was unable to capture them as my system went down. Upon reboot,
> > > the "transaction_id" message that I referred to in my previous message
> > > was repeated (but with increased transaction IDs).
> >
> > For better fix it would need to be better understood what has happened
> > in parallel while 'lvm' inside dmeventd was resizing pool data.
>
> To the best of my knowledge, no other LVM operations were in flight at
> the time. The script that I use issues LVM commands strictly
> sequentially and there are no other daemons that make use of LVM
> (except dmeventd and lvmetad, of course). Is there something else I
> should be looking out for? Is there another log that I can examine?
>
> > It looks like the 'other' lvm managed to create another snapshot
> > (and thus the DeviceID appeared to already exists - while it should not
> > according to lvm2 metadata before it hit problem with mismatch of
> > transaction_id.
> >
> > > I will reply privately with my lvm metadata archive and with my
> > > header. My profuse thanks, again, for assisting me getting my system
> > > back up and running.
> >
> > So the valid fix would be to take 'thin_dump' of kernel metadata
> > (aka content of _tmeta device)
> > Then check what you have in lvm2 metadata and likely you will
> > find some device in kernel - for which you don't have match
> > in lvm2 metadata -  these devices would need to be copied
> > from your other sequence of lvm2 metadata.
>
> I'll reply privately with the compressed output of thin_dump.
>
> > Other maybe more simple way could be to just remove devices
> > from xml thin_dump and thin_restore those metadata that should should
> > now match lvm2.
> >
> > The last issue is then to match 'transaction_id' with the number
> > stored in kernel metadata.
> >
> > So not sure which way you want to go and how important those
> > snapshot (that could be dropped) are ?
>
> Would it be reasonable to use vgcfgrestore again on the
> manually-repaired metadata I used before? I'm not entirely sure what
> to look for while editing the XML from thin_dump, and I would very
> much like to avoid causing further damage to my system. (Also, FWIW,
> thin_dump appears to segfault when run with musl-libc instead of
> glibc. I had to use a glibc recovery image to get this dump. I'm
> following up with the void distro maintainers about this.)
>
> I plan to "just" create a new thin pool and copy all the thin LVs over
> to the new pool. I'm hoping that that will let me wash my hands of
> this matter and restore my system to a stable state. The snapshots
> that I create are purely read-only so that the slow-moving backup
> process can get an atomic view of the filesystem in use. They contain
> no important information that isn't also captured in the origin LV
> itself.
>

Hello again!

Could someone advise whether the above strategy (restoring my metadata with
vgcfgrestore, making a new thin pool, and copying the contents of my thin
LVs over at the filesystem level) is likely to cause data loss? Is it
likely to solve my ongoing problem with this frustrating thin pool?

Thank you for your advice!
--Duncan Townsend

P.S. This was written on mobile. Please forgive my typos.

>

[-- Attachment #2: Type: text/html, Size: 6967 bytes --]

Re: [linux-lvm] thin: pool target too small

[Zdenek Kabelac]

Dne 29. 09. 20 v 16:33 Duncan Townsend napsal(a):
> On Sat, Sep 26, 2020, 8:30 AM Duncan Townsend <duncancmt@gmail.com 
> <mailto:duncancmt@gmail.com>> wrote:
> 
>>      > > There were further error messages as further snapshots were attempted,
>      > > but I was unable to capture them as my system went down. Upon reboot,
>      > > the "transaction_id" message that I referred to in my previous message
>      > > was repeated (but with increased transaction IDs).
>      >
>      > For better fix it would need to be better understood what has happened
>      > in parallel while 'lvm' inside dmeventd was resizing pool data.
> 

So the lvm2 has been fixed upstream to report more educative messages to
the user - although it still does require some experience in managing
thin-pool kernel metadata and lvm2 metadata.

>     To the best of my knowledge, no other LVM operations were in flight at
>     the time. The script that I use issues LVM commands strictly

In your case - dmeventd did 'unlocked' resize - while other command
was taking a snapshot - and it happened the sequence with 'snapshot' has
won - so until the reload of thin-pool - lvm2 has not spotted difference.
(which is simply a bad race cause due to badly working locking on your system)


>     Would it be reasonable to use vgcfgrestore again on the
>     manually-repaired metadata I used before? I'm not entirely sure what

You will need to vgcfgrestore - but I think you've misused my passed recoverd 
piece, where I've specifically asked to only replace specific segments of 
resized thin-pool within your latest VG metadata - since those likely have
all the proper mappings to thin LVs.

While you have taken the metadata from 'resize' moment - you've lost all
the thinLV lvm2 metadata for later created one.

I'll try to make one for you.

>     to look for while editing the XML from thin_dump, and I would very
>     much like to avoid causing further damage to my system. (Also, FWIW,
>     thin_dump appears to segfault when run with musl-libc instead of

Well - lvm2 is glibc oriented project - so users of those 'esoteric'
distribution need to be expert on its own.

If you can provide coredump or even better patch for crash - we might
replace the code with something better usable - but there is zero testing
with anything else then glibc...


Zdenek

Re: [linux-lvm] thin: pool target too small

[Duncan Townsend]

[-- Attachment #1: Type: text/plain, Size: 4055 bytes --]

On Tue, Sep 29, 2020, 10:54 AM Zdenek Kabelac <zkabelac@redhat.com> wrote:

> Dne 29. 09. 20 v 16:33 Duncan Townsend napsal(a):
> > On Sat, Sep 26, 2020, 8:30 AM Duncan Townsend <duncancmt@gmail.com
> > <mailto:duncancmt@gmail.com>> wrote:
> >
> >>      > > There were further error messages as further snapshots were
> attempted,
> >      > > but I was unable to capture them as my system went down. Upon
> reboot,
> >      > > the "transaction_id" message that I referred to in my previous
> message
> >      > > was repeated (but with increased transaction IDs).
> >      >
> >      > For better fix it would need to be better understood what has
> happened
> >      > in parallel while 'lvm' inside dmeventd was resizing pool data.
> >
>
> So the lvm2 has been fixed upstream to report more educative messages to
> the user - although it still does require some experience in managing
> thin-pool kernel metadata and lvm2 metadata.
>

That's good news! However, I believe I lack the requisite experience. Is
there some documentation that I ought to read as a starting point? Or is it
best to just read the source?

>     To the best of my knowledge, no other LVM operations were in flight at
> >     the time. The script that I use issues LVM commands strictly
>
> In your case - dmeventd did 'unlocked' resize - while other command
> was taking a snapshot - and it happened the sequence with 'snapshot' has
> won - so until the reload of thin-pool - lvm2 has not spotted difference.
> (which is simply a bad race cause due to badly working locking on your
> system)
>

After reading more about lvm locking, it looks like the original issue
might have been that the locking directory lives on a lv instead of on a
non-lvm-managed block device. (Although, the locking directory is on a
different vg on a different pv from the one that had the error.)

Is there a way to make dmeventd (or any other lvm program) abort if this
locking fails? Should I switch to using a clustered locking daemon (even
though I have only the single, non-virtualized host)?

>     Would it be reasonable to use vgcfgrestore again on the
> >     manually-repaired metadata I used before? I'm not entirely sure what
>
> You will need to vgcfgrestore - but I think you've misused my passed
> recoverd
> piece, where I've specifically asked to only replace specific segments of
> resized thin-pool within your latest VG metadata - since those likely have
> all the proper mappings to thin LVs.
>

All I did was use vgcfgrestore to apply the metadata file attached to your
previous private email. I had to edit the transaction number, as I noted
previously. That was a single line change. Was that the wrong thing to do?
I lack the experience with lvm/thin metadata, so I am flying a bit blind
here. I apologize if I've made things worse.

While you have taken the metadata from 'resize' moment - you've lost all
> the thinLV lvm2 metadata for later created one.
>
> I'll try to make one for you.
>

Thank you very much. I am extremely grateful that you've helped me so much
in repairing my system.

>     to look for while editing the XML from thin_dump, and I would very
> >     much like to avoid causing further damage to my system. (Also, FWIW,
> >     thin_dump appears to segfault when run with musl-libc instead of
>
> Well - lvm2 is glibc oriented project - so users of those 'esoteric'
> distribution need to be expert on its own.
>
> If you can provide coredump or even better patch for crash - we might
> replace the code with something better usable - but there is zero testing
> with anything else then glibc...
>

Noted. I believe I'll be switching to glibc because there are a number of
other packages that are broken for this distro.

If you have an interest, this is the issue I've opened with my distro about
the crash: https://github.com/void-linux/void-packages/issues/25125 . I
despair that this will receive much attention, given that not even gdb
works properly.

Thanks again!
--Duncan Townsend

P.S. This was written on mobile. Please forgive my typos.

[-- Attachment #2: Type: text/html, Size: 6124 bytes --]

Re: [linux-lvm] thin: pool target too small

[Duncan Townsend]

[-- Attachment #1: Type: text/plain, Size: 4497 bytes --]

On Wed, Sep 30, 2020, 1:00 PM Duncan Townsend <duncancmt@gmail.com> wrote:

> On Tue, Sep 29, 2020, 10:54 AM Zdenek Kabelac <zkabelac@redhat.com> wrote:
>
>> Dne 29. 09. 20 v 16:33 Duncan Townsend napsal(a):
>> > On Sat, Sep 26, 2020, 8:30 AM Duncan Townsend <duncancmt@gmail.com
>> > <mailto:duncancmt@gmail.com>> wrote:
>> >
>> >>      > > There were further error messages as further snapshots were
>> attempted,
>> >      > > but I was unable to capture them as my system went down. Upon
>> reboot,
>> >      > > the "transaction_id" message that I referred to in my previous
>> message
>> >      > > was repeated (but with increased transaction IDs).
>> >      >
>> >      > For better fix it would need to be better understood what has
>> happened
>> >      > in parallel while 'lvm' inside dmeventd was resizing pool data.
>> >
>>
>> So the lvm2 has been fixed upstream to report more educative messages to
>> the user - although it still does require some experience in managing
>> thin-pool kernel metadata and lvm2 metadata.
>>
>
> That's good news! However, I believe I lack the requisite experience. Is
> there some documentation that I ought to read as a starting point? Or is it
> best to just read the source?
>
> >     To the best of my knowledge, no other LVM operations were in flight
>> at
>> >     the time. The script that I use issues LVM commands strictly
>>
>> In your case - dmeventd did 'unlocked' resize - while other command
>> was taking a snapshot - and it happened the sequence with 'snapshot' has
>> won - so until the reload of thin-pool - lvm2 has not spotted difference.
>> (which is simply a bad race cause due to badly working locking on your
>> system)
>>
>
> After reading more about lvm locking, it looks like the original issue
> might have been that the locking directory lives on a lv instead of on a
> non-lvm-managed block device. (Although, the locking directory is on a
> different vg on a different pv from the one that had the error.)
>
> Is there a way to make dmeventd (or any other lvm program) abort if this
> locking fails? Should I switch to using a clustered locking daemon (even
> though I have only the single, non-virtualized host)?
>
> >     Would it be reasonable to use vgcfgrestore again on the
>> >     manually-repaired metadata I used before? I'm not entirely sure what
>>
>> You will need to vgcfgrestore - but I think you've misused my passed
>> recoverd
>> piece, where I've specifically asked to only replace specific segments of
>> resized thin-pool within your latest VG metadata - since those likely have
>> all the proper mappings to thin LVs.
>>
>
> All I did was use vgcfgrestore to apply the metadata file attached to your
> previous private email. I had to edit the transaction number, as I noted
> previously. That was a single line change. Was that the wrong thing to do?
> I lack the experience with lvm/thin metadata, so I am flying a bit blind
> here. I apologize if I've made things worse.
>
> While you have taken the metadata from 'resize' moment - you've lost all
>> the thinLV lvm2 metadata for later created one.
>>
>> I'll try to make one for you.
>>
>
> Thank you very much. I am extremely grateful that you've helped me so much
> in repairing my system.
>
> >     to look for while editing the XML from thin_dump, and I would very
>> >     much like to avoid causing further damage to my system. (Also, FWIW,
>> >     thin_dump appears to segfault when run with musl-libc instead of
>>
>> Well - lvm2 is glibc oriented project - so users of those 'esoteric'
>> distribution need to be expert on its own.
>>
>> If you can provide coredump or even better patch for crash - we might
>> replace the code with something better usable - but there is zero testing
>> with anything else then glibc...
>>
>
> Noted. I believe I'll be switching to glibc because there are a number of
> other packages that are broken for this distro.
>
> If you have an interest, this is the issue I've opened with my distro
> about the crash: https://github.com/void-linux/void-packages/issues/25125
> . I despair that this will receive much attention, given that not even gdb
> works properly.
>

Hello! Could somebody advise whether restoring the VG metadata is likely to
cause this system's condition to worsen? At this point, all I want is to do
is get the data off this drive and then start over with something more
stable.

Thanks for the help!
--Duncan Townsend

P.S. This was written on mobile. Please forgive my typos.

>

[-- Attachment #2: Type: text/html, Size: 7129 bytes --]

Re: [linux-lvm] thin: pool target too small

[Duncan Townsend]

On Fri, 2 Oct 2020, Duncan Townsend wrote:

> On Wed, Sep 30, 2020, 1:00 PM Duncan Townsend <duncancmt@gmail.com> wrote:
>
>> On Tue, Sep 29, 2020, 10:54 AM Zdenek Kabelac <zkabelac@redhat.com> wrote:
>>
>>> Dne 29. 09. 20 v 16:33 Duncan Townsend napsal(a):
>>>
>>> So the lvm2 has been fixed upstream to report more educative messages to
>>> the user - although it still does require some experience in managing
>>> thin-pool kernel metadata and lvm2 metadata.
>>>
>>
>> That's good news! However, I believe I lack the requisite experience. Is
>> there some documentation that I ought to read as a starting point? Or is it
>> best to just read the source?
>>
>>> In your case - dmeventd did 'unlocked' resize - while other command
>>> was taking a snapshot - and it happened the sequence with 'snapshot' has
>>> won - so until the reload of thin-pool - lvm2 has not spotted difference.
>>> (which is simply a bad race cause due to badly working locking on your
>>> system)
>>>
>>
>> After reading more about lvm locking, it looks like the original issue
>> might have been that the locking directory lives on a lv instead of on a
>> non-lvm-managed block device. (Although, the locking directory is on a
>> different vg on a different pv from the one that had the error.)
>>
>> Is there a way to make dmeventd (or any other lvm program) abort if this
>> locking fails? Should I switch to using a clustered locking daemon (even
>> though I have only the single, non-virtualized host)?
>>
>>> You will need to vgcfgrestore - but I think you've misused my passed
>>> recoverd
>>> piece, where I've specifically asked to only replace specific segments of
>>> resized thin-pool within your latest VG metadata - since those likely have
>>> all the proper mappings to thin LVs.
>>>
>>
>> All I did was use vgcfgrestore to apply the metadata file attached to your
>> previous private email. I had to edit the transaction number, as I noted
>> previously. That was a single line change. Was that the wrong thing to do?
>> I lack the experience with lvm/thin metadata, so I am flying a bit blind
>> here. I apologize if I've made things worse.
>>
>> While you have taken the metadata from 'resize' moment - you've lost all
>>> the thinLV lvm2 metadata for later created one.
>>>
>>> I'll try to make one for you.
>>>
>>
>> Thank you very much. I am extremely grateful that you've helped me so much
>> in repairing my system.
>>
>>> Well - lvm2 is glibc oriented project - so users of those 'esoteric'
>>> distribution need to be expert on its own.
>>>
>>> If you can provide coredump or even better patch for crash - we might
>>> replace the code with something better usable - but there is zero testing
>>> with anything else then glibc...
>>>
>>
>> Noted. I believe I'll be switching to glibc because there are a number of
>> other packages that are broken for this distro.
>>
>> If you have an interest, this is the issue I've opened with my distro
>> about the crash: https://github.com/void-linux/void-packages/issues/25125
>> . I despair that this will receive much attention, given that not even gdb
>> works properly.
>>
>
> Hello! Could somebody advise whether restoring the VG metadata is likely to
> cause this system's condition to worsen? At this point, all I want is to do
> is get the data off this drive and then start over with something more
> stable.

I believe I have repaired my system to the point of usability. I ended up 
editing *JUST* the transaction number in the metadata backup from before 
the snapshot creation that brought the system down. Restoring that VG 
metadata backup enabled me to activate my thin pool and LVs. After booting 
the system and inspecting further, I discovered that the locking directory 
for dmeventd was *NOT* mounted on a thin LV as a stated earlier, but was 
instead mounted on a tmpfs. As Zdenek hinted that the root cause of my 
original problem may have been a failure of local, file-based locking, my 
conclusion is that I may have gotten unlucky with the timing of the 
delivery of a SIGTERM from my broken runit script. I'm happy to be shown 
to be wrong about that conclusion, but it's the best I can draw given the 
information available to me.

The follow-on problem appears to have been a metadata mismatch on the 
transaction ID, which was effectively repaired by the aforementioned 2nd 
metadata transaction ID editing and restoration.

I have now been able to boot my system and complete several full 
iterations of my backup script using thin LV snapshotting. There were no 
problems. FWIW, I have been using this backup script as a cronjob for the 
last 8 years with minimal modification. I would be surprised to discover 
that there's something fundamental about the script that is erroneous.

Thanks again to Zdenek Kabelac for all the help. If there's any additional 
information I can provide to stop this from happening to anyone else in 
the future, I'm happy to assist. Otherwise, I'm going to consider this 
problem solved.

--Duncan Townsend