* [linux-lvm] The benefits of lvmlockd over clvmd? @ 2018-01-09 3:15 Eric Ren 2018-01-09 16:06 ` David Teigland 0 siblings, 1 reply; 8+ messages in thread From: Eric Ren @ 2018-01-09 3:15 UTC (permalink / raw) To: David Teigland, LVM general discussion and development Hi David, Regarding the question of the subject, I can think of three main benefits of lvmlockd over clvmd: - lvmlockd supports two cluster locking plugins: dlm and sanlock. sanlock plugin can supports up to ~2000 nodes that benefits LVM usage in big virtulizaton/storage cluster, while dlm plugin fits HA clsuter. - lvmlockd has better design than clvmd. clvmd is command-line level based locking system, which means the whole LVM software will get hang if any LVM command gets dead-locking issue. However, lvmlockd is *resources* based cluster locking. The resources to protect is VG and LV so that the deadlock issue will be isolated inside the resource and operations on other VG/LV can still proceed. - lvmlockd can work with lvmetad. But, I may be wrong in some points. Could you please help correct me and complete the benefit list? Thanks! Eric ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] The benefits of lvmlockd over clvmd? 2018-01-09 3:15 [linux-lvm] The benefits of lvmlockd over clvmd? Eric Ren @ 2018-01-09 16:06 ` David Teigland 2018-01-10 7:11 ` Eric Ren 0 siblings, 1 reply; 8+ messages in thread From: David Teigland @ 2018-01-09 16:06 UTC (permalink / raw) To: Eric Ren; +Cc: LVM general discussion and development On Tue, Jan 09, 2018 at 11:15:24AM +0800, Eric Ren wrote: > Hi David, > > Regarding the question of the subject, I can think of three main benefits of > lvmlockd over clvmd: > > - lvmlockd supports two cluster locking plugins: dlm and sanlock. sanlock > plugin can supports up to ~2000 nodes > that benefits LVM usage in big virtulizaton/storage cluster, True, although it's never been tried anywhere near that many. The main point hiding behind the big number is that hosts are pretty much unaware of each other, so adding more doesn't have any affect, and when something happens to one, others are unaffected because they are unaware. > while dlm plugin fits HA clsuter. > > - lvmlockd has better design than clvmd. clvmd is command-line level based > locking system, which means the > �whole LVM software will get hang if any LVM command gets dead-locking > issue. However, lvmlockd is *resources* based > cluster locking. The resources to protect is VG and LV so that the deadlock > issue will be isolated inside the resource and > operations on other VG/LV can still proceed. > > - lvmlockd can work with lvmetad. > > But, I may be wrong in some points. Could you please help correct me and > complete the benefit list? To me the biggest benefit is the design and internal implementation, which I admit don't make for great marketing. The design in general follows the idea described above, in which hosts fundamentally operate unaware of others and one host never has any effect on another. That's diametrically opposite to the original clvm "single system image" design in which everything that happens is in theory meant to be happening everywhere. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] The benefits of lvmlockd over clvmd? 2018-01-09 16:06 ` David Teigland @ 2018-01-10 7:11 ` Eric Ren 2018-01-10 9:36 ` Zdenek Kabelac 2018-01-10 16:45 ` David Teigland 0 siblings, 2 replies; 8+ messages in thread From: Eric Ren @ 2018-01-10 7:11 UTC (permalink / raw) To: David Teigland; +Cc: LVM general discussion and development Hi David, Thanks for your explanations! On 01/10/2018 12:06 AM, David Teigland wrote: > On Tue, Jan 09, 2018 at 11:15:24AM +0800, Eric Ren wrote: >> Hi David, >> >> Regarding the question of the subject, I can think of three main benefits of >> lvmlockd over clvmd: >> >> - lvmlockd supports two cluster locking plugins: dlm and sanlock. sanlock >> plugin can supports up to ~2000 nodes >> that benefits LVM usage in big virtulizaton/storage cluster, > True, although it's never been tried anywhere near that many. The main > point hiding behind the big number is that hosts are pretty much unaware > of each other, so adding more doesn't have any affect, and when something > happens to one, others are unaffected because they are unaware. The comments above is only talking about lvmlockd with sanlock, and it's because the different protocols/algorithms used by them: sanlock with Paxos, dlm with corosync, right? > >> while dlm plugin fits HA clsuter. >> >> - lvmlockd has better design than clvmd. clvmd is command-line level based >> locking system, which means the >> Â whole LVM software will get hang if any LVM command gets dead-locking >> issue. However, lvmlockd is *resources* based >> cluster locking. The resources to protect is VG and LV so that the deadlock >> issue will be isolated inside the resource and >> operations on other VG/LV can still proceed. Is this point roughly true? >> >> - lvmlockd can work with lvmetad. >> >> But, I may be wrong in some points. Could you please help correct me and >> complete the benefit list? > To me the biggest benefit is the design and internal implementation, which > I admit don't make for great marketing. The design in general follows the > idea described above, in which hosts fundamentally operate unaware of Sorry, "the idea described above" by me? > others and one host never has any effect on another. That's diametrically For example, with clvmd the command "lvchange -ay VG/LV" will try to activate the LV on every hosts, but with lvmlockd, we need to perform "lvchange -asy" on each host :) > opposite to the original clvm "single system image" design in which > everything that happens is in theory meant to be happening everywhere. Got it. Thanks again. Regards, Eric ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] The benefits of lvmlockd over clvmd? 2018-01-10 7:11 ` Eric Ren @ 2018-01-10 9:36 ` Zdenek Kabelac 2018-01-10 14:42 ` Eric Ren 2018-01-10 16:45 ` David Teigland 1 sibling, 1 reply; 8+ messages in thread From: Zdenek Kabelac @ 2018-01-10 9:36 UTC (permalink / raw) To: LVM general discussion and development, Eric Ren Dne 10.1.2018 v 08:11 Eric Ren napsal(a): > Hi David, > > Thanks for your explanations! > > On 01/10/2018 12:06 AM, David Teigland wrote: >> On Tue, Jan 09, 2018 at 11:15:24AM +0800, Eric Ren wrote: >>> Hi David, >>> >>> Regarding the question of the subject, I can think of three main benefits of >>> lvmlockd over clvmd: >>> >>> - lvmlockd supports two cluster locking plugins: dlm and sanlock. sanlock >>> plugin can supports up to ~2000 nodes >>> that benefits LVM usage in big virtulizaton/storage cluster, >> True, although it's never been tried anywhere near that many. The main >> point hiding behind the big number is that hosts are pretty much unaware >> of each other, so adding more doesn't have any affect, and when something >> happens to one, others are unaffected because they are unaware. > > The comments above is only talking about lvmlockd with sanlock, and it's > because the different protocols/algorithms used by them: sanlock with Paxos, > dlm with corosync, right? > >> >>> while dlm plugin fits HA clsuter. >>> >>> - lvmlockd has better design than clvmd. clvmd is command-line level based >>> locking system, which means the >>> whole LVM software will get hang if any LVM command gets dead-locking >>> issue. However, lvmlockd is *resources* based >>> cluster locking. The resources to protect is VG and LV so that the deadlock >>> issue will be isolated inside the resource and >>> operations on other VG/LV can still proceed. > > Is this point roughly true? > >>> >>> - lvmlockd can work with lvmetad. >>> >>> But, I may be wrong in some points. Could you please help correct me and >>> complete the benefit list? >> To me the biggest benefit is the design and internal implementation, which >> I admit don't make for great marketing. The design in general follows the >> idea described above, in which hosts fundamentally operate unaware of > > Sorry, "the idea described above" by me? > >> others and one host never has any effect on another. That's diametrically > > For example, with clvmd the command "lvchange -ay VG/LV" will try to activate > the LV > on every hosts, but with lvmlockd, we need to perform "lvchange -asy" on each > host :) > There are couple fuzzy sentences - so lets try to make them more clear. Default mode for 'clvmd' is to 'share' resource everywhere - which clearly comes from original 'gfs' requirement and 'linear/striped' volume that can be easily activated on many nodes. However over the time - different use-cases got more priority so basically every new dm target (except mirror) does NOT support shared storage (maybe raid will one day...). So targets like snapshot, thin, cache, raid do require 'so called' exclusive activation. So here comes the difference - lvmlockd in its default goes with 'exclusive/local' activation and shared (old clvmd default) needs to be requested. Another difference is - 'clvmd' world is 'automating' activation around the whole cluster (so from node A it's possible to activate volume on node B without ANY other command then 'lvchange). With 'lvmlockd' mechanism - this was 'dropped' and it's users responsibility to initiate i.e. ssh command with activation on another node(s) and resolve error handling. There are various pros&cons over each solution - both needs setups and while 'clvmd' world is 'set & done' lvmlockd world scripting needs to be born in some way. Also ATM 'lvmetad' can't be used even with lvmlockd - simply because we are not (yet) capable to handle 'udev' around the cluster (and it's not clear we ever will). On the positive side - we are working hard to enhance 'scanning' speed - so in majority of use-cases there is no real performance gain with lvmetad usage anyway. Regards Zdenek ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] The benefits of lvmlockd over clvmd? 2018-01-10 9:36 ` Zdenek Kabelac @ 2018-01-10 14:42 ` Eric Ren 2018-01-10 15:35 ` Zdenek Kabelac 0 siblings, 1 reply; 8+ messages in thread From: Eric Ren @ 2018-01-10 14:42 UTC (permalink / raw) To: LVM general discussion and development, Zdenek Kabelac Zdenek, Thanks for helping make this more clear to me :) > > There are couple fuzzy sentences - so lets try to make them more clear. > > Default mode for 'clvmd' is to 'share' resource everywhere - which > clearly comes from original 'gfs' requirement and 'linear/striped' > volume that can be easily activated on many nodes. > > However over the time - different use-cases got more priority so > basically every new dm target (except mirror) does NOT support shared > storage (maybe raid will one day...). So targets like snapshot, > thin, cache, raid do require 'so called' exclusive activation. Good to know the history about clvmd :) > > So here comes the difference - lvmlockd in its default goes with > 'exclusive/local' activation and shared (old clvmd default) needs to > be requested. > > Another difference is - 'clvmd' world is 'automating' activation > around the whole cluster (so from node A it's possible to activate > volume on node B without ANY other command then 'lvchange). > > With 'lvmlockd' mechanism - this was 'dropped' and it's users > responsibility to initiate i.e. ssh command with activation on another > node(s) and resolve error handling. > > There are various pros&cons over each solution - both needs setups and > while 'clvmd' world is 'set & done' lvmlockd world scripting needs > to be born in some way. True. > Also ATM 'lvmetad' can't be used even with lvmlockd - simply because > we are not (yet) capable to handle 'udev' around the cluster (and it's > not clear we ever will). This sentence surprises me much. According to manpage of lvmlockd, it seems clear that lvmlockd can work with lvmetad now. IIRC, it's not the first time you mentioned about "cluster udev". It gives me a impression that the currect udev system is not 100% reliable for shared disks in cluster, no matter if we use lvmetad or not, right? If so, could you please give an example scenario where lvmetad may not work well with lvmlockd? > > On the positive side - we are working hard to enhance 'scanning' speed > - so in majority of use-cases there is no real performance gain with > lvmetad usage anyway. Great! Thanks. Regards, Eric ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] The benefits of lvmlockd over clvmd? 2018-01-10 14:42 ` Eric Ren @ 2018-01-10 15:35 ` Zdenek Kabelac 2018-01-10 17:25 ` David Teigland 0 siblings, 1 reply; 8+ messages in thread From: Zdenek Kabelac @ 2018-01-10 15:35 UTC (permalink / raw) To: Eric Ren, LVM general discussion and development Dne 10.1.2018 v 15:42 Eric Ren napsal(a): > Zdenek, > > Thanks for helping make this more clear to me :) > >> >> There are couple fuzzy sentences - so lets try to make them more clear. >> >> Default mode for 'clvmd' is to 'share' resource everywhere - which clearly >> comes from original 'gfs' requirement and 'linear/striped' volume that can >> be easily activated on many nodes. >> >> However over the time - different use-cases got more priority so basically >> every new dm target (except mirror) does NOT support shared storage (maybe >> raid will one day...). So targets like snapshot, thin, cache, raid do >> require 'so called' exclusive activation. > > Good to know the history about clvmd :) > >> >> So here comes the difference - lvmlockd in its default goes with >> 'exclusive/local' activation and shared (old clvmd default) needs to be >> requested. >> >> Another difference is - 'clvmd' world is 'automating' activation around >> the whole cluster (so from node A it's possible to activate volume on node B >> without ANY other command then 'lvchange). >> >> With 'lvmlockd' mechanism - this was 'dropped' and it's users responsibility >> to initiate i.e. ssh command with activation on another node(s) and resolve >> error handling. >> >> There are various pros&cons over each solution - both needs setups and while >> 'clvmd' world is 'set & done' lvmlockd world scripting needs to be born in >> some way. > > True. > >> Also ATM 'lvmetad' can't be used even with lvmlockd - simply because we are >> not (yet) capable to handle 'udev' around the cluster (and it's not clear we >> ever will). > > This sentence surprises me much. According to manpage of lvmlockd, it seems > clear that lvmlockd can work with lvmetad now. > IIRC, it's not the first time you mentioned about "cluster udev". It gives me > a impression that the currect udev system is not > 100% reliable for shared disks in cluster, no matter if we use lvmetad or not, > right? If so, could you please give an example > scenario where lvmetad may not work well with lvmlockd? > Hi The world of udevd/systemd is complicated monster - which has no notation for handling bad/duplicate/.... devices and so on. Current design of lvmetad is not sufficient to live in ocean of bugs in this category - so as said - ATM it's highly recommend to keep lvmetad off in clusters. Regards Zdenek ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] The benefits of lvmlockd over clvmd? 2018-01-10 15:35 ` Zdenek Kabelac @ 2018-01-10 17:25 ` David Teigland 0 siblings, 0 replies; 8+ messages in thread From: David Teigland @ 2018-01-10 17:25 UTC (permalink / raw) To: zkabelac, linux-lvm; +Cc: Eric Ren On Wed, Jan 10, 2018 at 04:35:56PM +0100, Zdenek Kabelac wrote: > > > Also ATM� 'lvmetad' can't be used even with lvmlockd - simply > > > because we are not (yet) capable to handle 'udev' around the cluster > > > (and it's not clear we ever will). > > > > This sentence surprises me much. According to manpage of lvmlockd, it It surprises me too, since I developed lvmlockd and lvmetad features precisely to make it work. > > seems clear that lvmlockd can work with lvmetad now. > > IIRC, it's not the first time you mentioned about "cluster udev". It > > gives me a impression that the currect udev system is not > > 100% reliable for shared disks in cluster, no matter if we use lvmetad > > or not, right? If so, could you please give an example > > scenario where lvmetad may not work well with lvmlockd? > > The world of udevd/systemd is complicated monster - which has no notation for > handling bad/duplicate/.... devices and so on. > > Current design of lvmetad is not sufficient to live in ocean of bugs in this > category - so as said - ATM it's highly recommend to keep lvmetad off in > clusters. There are indeed plenty of problems with lvmetad, which is why I've been trying to get us to get rid of lvmetad, and improved disk scanning to be so much more efficient: https://sourceware.org/git/?p=lvm2.git;a=shortlog;h=refs/heads/dev-dct-new-scan-32 However, you have not pointed out any specific problems unique to lvmlockd with lvmetad. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] The benefits of lvmlockd over clvmd? 2018-01-10 7:11 ` Eric Ren 2018-01-10 9:36 ` Zdenek Kabelac @ 2018-01-10 16:45 ` David Teigland 1 sibling, 0 replies; 8+ messages in thread From: David Teigland @ 2018-01-10 16:45 UTC (permalink / raw) To: Eric Ren; +Cc: LVM general discussion and development On Wed, Jan 10, 2018 at 03:11:24PM +0800, Eric Ren wrote: > > True, although it's never been tried anywhere near that many. The main > > point hiding behind the big number is that hosts are pretty much unaware > > of each other, so adding more doesn't have any affect, and when something > > happens to one, others are unaffected because they are unaware. > > The comments above is only talking about lvmlockd with sanlock, and it's > because the different protocols/algorithms used by them: sanlock with Paxos, > dlm with corosync, right? right > > > - lvmlockd has better design than clvmd. clvmd is command-line level > > > based locking system, which means the �whole LVM software will get > > > hang if any LVM command gets dead-locking issue. However, lvmlockd > > > is *resources* based cluster locking. The resources to protect is VG > > > and LV so that the deadlock issue will be isolated inside the > > > resource and operations on other VG/LV can still proceed. > > Is this point roughly true? It's vague enough to pass I suppose, but I don't think comparing them in terms of locking is very helpful, because clvmd is not really a locking system, it's a parallel execution system. Keep in mind we're generalizing and largely discussing opinion here, so there are bound to be some valid disagreements about this. Using lvmlockd, lvm just acquires and releases simple locks around making changes. There's no notion of "nodes", no concept of remote/local. Either a command gets a lock or it doesn't, not much else can happen or go wrong, it's pretty trivial. clvmd is vastly different: "all nodes" and remote/local nodes is the central controlling idea. Locks are not the main thing, the main thing is running all commands in parallel everywhere. There are endless ways things can go wrong when trying to run commands in parallel on all nodes. What's more, there's no point to it, since nothing actually needs to run in parallel everywhere. > > > But, I may be wrong in some points. Could you please help correct me and > > > complete the benefit list? > > To me the biggest benefit is the design and internal implementation, which > > I admit don't make for great marketing. The design in general follows the > > idea described above, in which hosts fundamentally operate unaware of > > Sorry, "the idea described above" by me? No, the idea I described about sanlock not being affected by other nodes. Dave ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-01-10 17:25 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-01-09 3:15 [linux-lvm] The benefits of lvmlockd over clvmd? Eric Ren 2018-01-09 16:06 ` David Teigland 2018-01-10 7:11 ` Eric Ren 2018-01-10 9:36 ` Zdenek Kabelac 2018-01-10 14:42 ` Eric Ren 2018-01-10 15:35 ` Zdenek Kabelac 2018-01-10 17:25 ` David Teigland 2018-01-10 16:45 ` David Teigland
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).