[linux-lvm] Why do lvcreate with clvmd insist on VG being available on all nodes?

[linux-lvm] Why do lvcreate with clvmd insist on VG being available on all nodes?

[Jacek Konieczny]

Hello,

I am building a system where I use clustered LVM on a DRBD to provide
shared block devices in a cluster. And I don't quite like and quite not
understand some behaviour.

Currently I have two nodes in the cluster, running: Corosync, DLM,
clvmd, DRBD, Pacemaker and my service.

Everything works fine when both nodes are up. When I put one to standby
with 'crm node node1 standby' (which, among others, stops the DRBD) the
other note is not fully functional.

If I leave DLM and CLVMD running on the inactive node, then:

lvchange -aey shared_vg/vol_name
lvchange -aen shared_vg/vol_name

work properly, as I would expect (make the volume available/unavailable
on the node). But an attempt to create a new volume:

lvcreate -n new_volume -L 1M shared_vg

fails with:

Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL

Indeed, the VG is not available at the standby node at that moment. But,
as it is not available there, I see no point in locking it there.



Is there some real, important reason to block lvcreate in such case?




I have also tried stopping (cleanly) dlm_controld and clvmd on the standby node,
hoping LVM will then work like in a single-node cluster, but then even
volume activation fails with:

  cluster request failed: Host is down

…until I restart clvmd on the active host with 'clvmd -S'.

When clvmd is stopped on the inactive node and 'clvmd -S' has been run
on the active node, then both 'lvchange' and 'lvcreate' work as
expected, but that doesn't look like a graceful switch-over. And another
'clvmd -S' stopped clvmd all together (this seems like a bug to me)

And one more thing bothers me… my system would be very scalable to many
nodes, where only two share active storage (when using DRBD). But this
won't work if LVM would refuse some operations when any VG is not
available on all nodes.

Greets,
        Jacek

Re: [linux-lvm] Why do lvcreate with clvmd insist on VG being available on all nodes?

[Zdenek Kabelac]

Dne 14.11.2012 16:16, Jacek Konieczny napsal(a):
> Hello,
>
> I am building a system where I use clustered LVM on a DRBD to provide
> shared block devices in a cluster. And I don't quite like and quite not
> understand some behaviour.
>
> Currently I have two nodes in the cluster, running: Corosync, DLM,
> clvmd, DRBD, Pacemaker and my service.
>
> Everything works fine when both nodes are up. When I put one to standby
> with 'crm node node1 standby' (which, among others, stops the DRBD) the
> other note is not fully functional.
>
> If I leave DLM and CLVMD running on the inactive node, then:
>
> lvchange -aey shared_vg/vol_name
> lvchange -aen shared_vg/vol_name
>
> work properly, as I would expect (make the volume available/unavailable
> on the node). But an attempt to create a new volume:
>
> lvcreate -n new_volume -L 1M shared_vg
>
> fails with:
>
> Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
>


Haven't really tried to understand what are you trying to achieve,
but if you want to have node being activated only on one cluster node,
you may easily use    lvcreate -aey  option.

If you are using default clustered operation - it's not surprising,
the operation is refused if other nodes are not responding.


> Indeed, the VG is not available at the standby node at that moment. But,
> as it is not available there, I see no point in locking it there.

Well - you would need to write your own type of locking with support
of 'standby'  currently clvmd doesn't work with such state (and it's not
quite clear to me how it actually should even work).
So far either node is in cluster or is fenced.

>
> Is there some real, important reason to block lvcreate in such case?

As long as you would use exclusive activation for lvcreate it should work.
(or maybe 'local' - just test - but since you are trying to use unsupported
operational mode - you need to take responsibility for the results)


>
> When clvmd is stopped on the inactive node and 'clvmd -S' has been run
> on the active node, then both 'lvchange' and 'lvcreate' work as
> expected, but that doesn't look like a graceful switch-over. And another
> 'clvmd -S' stopped clvmd all together (this seems like a bug to me)
>
> And one more thing bothers me… my system would be very scalable to many
> nodes, where only two share active storage (when using DRBD). But this
> won't work if LVM would refuse some operations when any VG is not
> available on all nodes.

Obviously using clustered VG in non-clustered environment isn't smart plan.
What you could do - is to   disable clustering support on VG

vgchange -cn <vg> --config 'global {locking_type = 0}'


Note - you may always go around any locking problem with the above config 
option - just do not report then problems with broken disk content and badly
activated volumes on cluster nodes.

Zdenek

Re: [linux-lvm] Why do lvcreate with clvmd insist on VG being available on all nodes?

[Jacek Konieczny]

On Thu, Nov 15, 2012 at 10:09:35AM +0100, Zdenek Kabelac wrote:
> > work properly, as I would expect (make the volume available/unavailable
> > on the node). But an attempt to create a new volume:
> >
> > lvcreate -n new_volume -L 1M shared_vg
> >
> > fails with:
> >
> > Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
> >
> 
> 
> Haven't really tried to understand what are you trying to achieve,
> but if you want to have node being activated only on one cluster node,
> you may easily use    lvcreate -aey  option.
> 
> If you are using default clustered operation - it's not surprising,
> the operation is refused if other nodes are not responding.

Hmmm didn't though about the initial activation. In fact, I don't need
that volume activated as soon as it is created. You are right, I should
try 'lvcreate -aey' or 'lvcreate -an'.

My stupid mistake, indeed.

'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.

Though, LVM have some problems with tracking the exclusive activations
later…

> > Indeed, the VG is not available at the standby node at that moment. But,
> > as it is not available there, I see no point in locking it there.
> 
> Well - you would need to write your own type of locking with support
> of 'standby'  currently clvmd doesn't work with such state (and it's not
> quite clear to me how it actually should even work).
> So far either node is in cluster or is fenced.

I will try to make this work somehow… I don't think a node in standby
(or when one of the VGs is not available there) should be quite
different that one fenced.

> > When clvmd is stopped on the inactive node and 'clvmd -S' has been run
> > on the active node, then both 'lvchange' and 'lvcreate' work as
> > expected, but that doesn't look like a graceful switch-over. And another
> > 'clvmd -S' stopped clvmd all together (this seems like a bug to me)
> >
> > And one more thing bothers me… my system would be very scalable to many
> > nodes, where only two share active storage (when using DRBD). But this
> > won't work if LVM would refuse some operations when any VG is not
> > available on all nodes.
> 
> Obviously using clustered VG in non-clustered environment isn't smart plan.
> What you could do - is to   disable clustering support on VG

Clusters do not have to be symmetrical. Cluster when different nodes
have a bit different set of resources available are still clusters.

I do need clustering locking – in case a volume group is available on
a few nodes it must not be possible to more than one node use any of the
logical volumes there.

> Note - you may always go around any locking problem with the above config 
> option - just do not report then problems with broken disk content and badly
> activated volumes on cluster nodes.

I am aware of the risks. That is why I do use clvmd. I don't quite
understand some of the clvmd behaviour and I wrote here to know if there
are some other risks taken into account by clvmd that I am not aware of.

It seem I still need some work and learning to make it work properly. 

Greets,
        Jacek

Re: [linux-lvm] Why do lvcreate with clvmd insist on VG being available on all nodes?

[Zdenek Kabelac]

Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
> On Thu, Nov 15, 2012 at 10:09:35AM +0100, Zdenek Kabelac wrote:
>>> work properly, as I would expect (make the volume available/unavailable
>>> on the node). But an attempt to create a new volume:
>>>
>>> lvcreate -n new_volume -L 1M shared_vg
>>>
>>> fails with:
>>>
>>> Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
>>>
>>
>>
>> Haven't really tried to understand what are you trying to achieve,
>> but if you want to have node being activated only on one cluster node,
>> you may easily use    lvcreate -aey  option.
>>
>> If you are using default clustered operation - it's not surprising,
>> the operation is refused if other nodes are not responding.
>
> Hmmm didn't though about the initial activation. In fact, I don't need
> that volume activated as soon as it is created. You are right, I should
> try 'lvcreate -aey' or 'lvcreate -an'.
>
> My stupid mistake, indeed.
>
> 'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.
>
> Though, LVM have some problems with tracking the exclusive activations
> later…

If you know about any such bug - just open rhbz with full description of such 
erroneous case.



>>> Indeed, the VG is not available at the standby node at that moment. But,
>>> as it is not available there, I see no point in locking it there.
>>
>> Well - you would need to write your own type of locking with support
>> of 'standby'  currently clvmd doesn't work with such state (and it's not
>> quite clear to me how it actually should even work).
>> So far either node is in cluster or is fenced.
>
> I will try to make this work somehow… I don't think a node in standby
> (or when one of the VGs is not available there) should be quite
> different that one fenced.
>
>>> When clvmd is stopped on the inactive node and 'clvmd -S' has been run
>>> on the active node, then both 'lvchange' and 'lvcreate' work as
>>> expected, but that doesn't look like a graceful switch-over. And another
>>> 'clvmd -S' stopped clvmd all together (this seems like a bug to me)
>>>
>>> And one more thing bothers me… my system would be very scalable to many
>>> nodes, where only two share active storage (when using DRBD). But this
>>> won't work if LVM would refuse some operations when any VG is not
>>> available on all nodes.
>>
>> Obviously using clustered VG in non-clustered environment isn't smart plan.
>> What you could do - is to   disable clustering support on VG
>
> Clusters do not have to be symmetrical. Cluster when different nodes
> have a bit different set of resources available are still clusters.

You want to support different scheme - thus you need to probably write your 
own clvmd-like daemon to cover all new cases you bring in with non-symmetrical 
cases.

> I do need clustering locking – in case a volume group is available on
> a few nodes it must not be possible to more than one node use any of the
> logical volumes there.

clvmd typical use case is  'vg' used on couple cluster nodes.

While you are probably trying to use  N:M mapping of vg and clustered nodes.

>
>> Note - you may always go around any locking problem with the above config
>> option - just do not report then problems with broken disk content and badly
>> activated volumes on cluster nodes.
>
> I am aware of the risks. That is why I do use clvmd. I don't quite
> understand some of the clvmd behaviour and I wrote here to know if there
> are some other risks taken into account by clvmd that I am not aware of.
>
> It seem I still need some work and learning to make it work properly.

You could surely create something to support your specific use case,
however clvmd needs to have some predictable behaviour in many error-case 
scenarios - and this is when you would need to start to think hard, how to 
resolve them in your non-symmetrical case.

Zdenek

Re: [linux-lvm] Why do lvcreate with clvmd insist on VG being available on all nodes?

[Jacek Konieczny]

On Thu, Nov 15, 2012 at 12:01:10PM +0100, Zdenek Kabelac wrote:
> Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
> > On Thu, Nov 15, 2012 at 10:09:35AM +0100, Zdenek Kabelac wrote:
> >>> work properly, as I would expect (make the volume available/unavailable
> >>> on the node). But an attempt to create a new volume:
> >>>
> >>> lvcreate -n new_volume -L 1M shared_vg
> >>>
> >>> fails with:
> >>>
> >>> Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
> >>>

> >> Haven't really tried to understand what are you trying to achieve,
> >> but if you want to have node being activated only on one cluster node,
> >> you may easily use    lvcreate -aey  option.
> >>
> > My stupid mistake, indeed.
> >
> > 'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.


> > Though, LVM have some problems with tracking the exclusive activations
> > later…
> 
> If you know about any such bug - just open rhbz with full description of such 
> erroneous case.

It was just another mistake of mine. LVM properly tracks the exclusive
locks – the volumes were being deactivated by something else.

> > Clusters do not have to be symmetrical. Cluster when different nodes
> > have a bit different set of resources available are still clusters.
> 
> You want to support different scheme - thus you need to probably write your 
> own clvmd-like daemon to cover all new cases you bring in with non-symmetrical 
> cases.

I think this will not be needed.

> clvmd typical use case is  'vg' used on couple cluster nodes.

I see.
 
> While you are probably trying to use  N:M mapping of vg and clustered nodes.

Exactly. But it seems it should not be a problem in my case.


After knowing my mistake I can see LVM already provides the
On Thu, Nov 15, 2012 at 12:01:10PM +0100, Zdenek Kabelac wrote:
> Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
> > On Thu, Nov 15, 2012 at 10:09:35AM +0100, Zdenek Kabelac wrote:
> >>> work properly, as I would expect (make the volume available/unavailable
> >>> on the node). But an attempt to create a new volume:
> >>>
> >>> lvcreate -n new_volume -L 1M shared_vg
> >>>
> >>> fails with:
> >>>
> >>> Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
> >>>

> >> Haven't really tried to understand what are you trying to achieve,
> >> but if you want to have node being activated only on one cluster node,
> >> you may easily use    lvcreate -aey  option.
> >>
> > My stupid mistake, indeed.
> >
> > 'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.


> > Though, LVM have some problems with tracking the exclusive activations
> > later…
> 
> If you know about any such bug - just open rhbz with full description of such 
> erroneous case.

It was just another mistake of mine. LVM properly tracks the exclusive
locks – the volumes were being deactivated by something else.

> > Clusters do not have to be symmetrical. Cluster when different nodes
> > have a bit different set of resources available are still clusters.
> 
> You want to support different scheme - thus you need to probably write your 
> own clvmd-like daemon to cover all new cases you bring in with non-symmetrical 
> cases.

I think this will not be needed.

> clvmd typical use case is  'vg' used on couple cluster nodes.

I see.
 
> While you are probably trying to use  N:M mapping of vg and clustered nodes.

Exactly. But it seems it should not be a problem in my case.

After knowing my mistake I can see LVM already provides the
functionality I need.

To summarize:

- The default LV activation mode is '-ay', which means, for clustered
  volume groups, that the volume is to be active on every node in the
  cluster.

– The activation is not always explicit. 'lvcreate' uses '-ay' when no
  other '-a' option is given.

– The '-ay' activation won't work if any node in the cluster cannot
  access the volume group (e.g. when the DRBD device holding it is
  Secondary or not configured at that node).

– However, my use case doesn't need more than one node using any of the
  volumes at any time. In fact, it is very important that only a single
  machine uses each LV at a time.

– For this scenario '-aey' should be always used.

– Explicitly using '-aey' with 'lvcreate' fixes the 'Volume group for
  uuid not found' for me.

– Other tests made showed that the locking works as expected when
  a volume group becomes available and unavailable on various cluster
  nodes. A node having access to a VG can create and/or activate LVs
  there in exclusive node and all other nodes will comply with that
  lock whenever they gain access to this VG.

So, it seems that clvmd is not that bound to the 'symmetrical cluster'
scenario, provided no more than one node needs to access a volume at
a time.

Does this make sense?

Greets,
        Jacek

Re: [linux-lvm] Why do lvcreate with clvmd insist on VG being available on all nodes?

[Zdenek Kabelac]

Dne 15.11.2012 14:30, Jacek Konieczny napsal(a):
> On Thu, Nov 15, 2012 at 12:01:10PM +0100, Zdenek Kabelac wrote:
>> Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
>>> On Thu, Nov 15, 2012 at 10:09:35AM +0100, Zdenek Kabelac wrote:
>>>>> work properly, as I would expect (make the volume available/unavailable
>>>>> on the node). But an attempt to create a new volume:
>>>>>
>>>>> lvcreate -n new_volume -L 1M shared_vg
>>>>>
>>>>> fails with:
>>>>>
>>>>> Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
>>>>>
>
>>>> Haven't really tried to understand what are you trying to achieve,
>>>> but if you want to have node being activated only on one cluster node,
>>>> you may easily use    lvcreate -aey  option.
>>>>
>>> My stupid mistake, indeed.
>>>
>>> 'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.
>
>
>>> Though, LVM have some problems with tracking the exclusive activations
>>> later…
>>
>> If you know about any such bug - just open rhbz with full description of such
>> erroneous case.
>
> It was just another mistake of mine. LVM properly tracks the exclusive
> locks – the volumes were being deactivated by something else.
>
>>> Clusters do not have to be symmetrical. Cluster when different nodes
>>> have a bit different set of resources available are still clusters.
>>
>> You want to support different scheme - thus you need to probably write your
>> own clvmd-like daemon to cover all new cases you bring in with non-symmetrical
>> cases.
>
> I think this will not be needed.
>
>> clvmd typical use case is  'vg' used on couple cluster nodes.
>
> I see.
>
>> While you are probably trying to use  N:M mapping of vg and clustered nodes.
>
> Exactly. But it seems it should not be a problem in my case.
>
>
> After knowing my mistake I can see LVM already provides the
> On Thu, Nov 15, 2012 at 12:01:10PM +0100, Zdenek Kabelac wrote:
>> Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
>>> On Thu, Nov 15, 2012 at 10:09:35AM +0100, Zdenek Kabelac wrote:
>>>>> work properly, as I would expect (make the volume available/unavailable
>>>>> on the node). But an attempt to create a new volume:
>>>>>
>>>>> lvcreate -n new_volume -L 1M shared_vg
>>>>>
>>>>> fails with:
>>>>>
>>>>> Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
>>>>>
>
>>>> Haven't really tried to understand what are you trying to achieve,
>>>> but if you want to have node being activated only on one cluster node,
>>>> you may easily use    lvcreate -aey  option.
>>>>
>>> My stupid mistake, indeed.
>>>
>>> 'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.
>
>
>>> Though, LVM have some problems with tracking the exclusive activations
>>> later…
>>
>> If you know about any such bug - just open rhbz with full description of such
>> erroneous case.
>
> It was just another mistake of mine. LVM properly tracks the exclusive
> locks – the volumes were being deactivated by something else.
>
>>> Clusters do not have to be symmetrical. Cluster when different nodes
>>> have a bit different set of resources available are still clusters.
>>
>> You want to support different scheme - thus you need to probably write your
>> own clvmd-like daemon to cover all new cases you bring in with non-symmetrical
>> cases.
>
> I think this will not be needed.
>
>> clvmd typical use case is  'vg' used on couple cluster nodes.
>
> I see.
>
>> While you are probably trying to use  N:M mapping of vg and clustered nodes.
>
> Exactly. But it seems it should not be a problem in my case.
>
> After knowing my mistake I can see LVM already provides the
> functionality I need.
>
> To summarize:
>
> - The default LV activation mode is '-ay', which means, for clustered
>    volume groups, that the volume is to be active on every node in the
>    cluster.
>
> – The activation is not always explicit. 'lvcreate' uses '-ay' when no
>    other '-a' option is given.
>
> – The '-ay' activation won't work if any node in the cluster cannot
>    access the volume group (e.g. when the DRBD device holding it is
>    Secondary or not configured at that node).
>
> – However, my use case doesn't need more than one node using any of the
>    volumes at any time. In fact, it is very important that only a single
>    machine uses each LV at a time.
>
> – For this scenario '-aey' should be always used.
>
> – Explicitly using '-aey' with 'lvcreate' fixes the 'Volume group for
>    uuid not found' for me.
>
> – Other tests made showed that the locking works as expected when
>    a volume group becomes available and unavailable on various cluster
>    nodes. A node having access to a VG can create and/or activate LVs
>    there in exclusive node and all other nodes will comply with that
>    lock whenever they gain access to this VG.
>
> So, it seems that clvmd is not that bound to the 'symmetrical cluster'
> scenario, provided no more than one node needs to access a volume at
> a time.
>
> Does this make sense?
>

I'm still somewhat confused with your term  'stand-by' cluster node.
But it looks like -aey mostly fixes your problems.

Zdenek