Re: [linux-lvm] [lvmlockd] recovery lvmlockd after kill_vg

[David Teigland <teigland@redhat.com>]

On Fri, Sep 28, 2018 at 11:14:35AM +0800, Damon Wang wrote:

> as the manual says, it should deactivate volumes and drop lockspace as
> quick as possible,

A year ago we discussed a more automated solution for forcing a VG offline
when its sanlock lockspace was shut down:

https://www.redhat.com/archives/lvm-devel/2017-September/msg00011.html

The idea was to forcibly shut down LVs (using dmsetup wipe_table) in the
VG when the kill_vg happened, then to automatically do the 'lvmlockctl
--drop' when the LVs were safely shut down.  There were some loose ends
around integrating this solution that I never sorted out, so it's remained
on my todo list.

> TTY can get a message, but it's not a good way to listen or monitor, so I
> run vgck periodically and parse its stdout and stderr, once "sanlock lease
> storage failure" or
> something unusual happens, an alert will be triggered and I'll do some
> check(I hope all this process can be automatically).

If you are specifically interested in detecting this lease timeout
condition, there are some sanlock commands that can give you this info.
You can also detect ahead of time that a VG's lockspace is getting close
the threshold.  I can get back to you with more specific fields to look
at, but for now take a look at 'sanlock client renewal' and some of the
internal details that are printed by 'sanlock client status -D'.

Dave