[Cluster-devel] DLM connection channel switch take too long time (> 5mins)

[Cluster-devel] DLM connection channel switch take too long time (> 5mins)

[Gang He]

Hello list and David Teigland,

I got a problem under a two rings cluster, the problem can be reproduced with the below steps.
1) setup a two rings cluster with two nodes.
e.g. 
clvm1(nodeid 172204569)  addr_list eth0 10.67.162.25 eth1 192.168.152.240
clvm2(nodeid 172204570)  addr_list eth0 10.67.162.26 eth1 192.168.152.103

2) the whole cluster works well, then I put eth0 down on node clvm2, and restart pacemaker service on that node.
ifconfig eth0 down
rcpacemaker restart

3) the whole cluster still work well (that means corosync is very smooth to switch to the other ring).
Then, I can mount ocfs2 file system on node clvm2 quickly with the command 
mount /dev/sda /mnt/ocfs2 

4) Next, I do the same mount on node clvm1, the mount command will be hanged for about 5 mins, and finally the mount command is done.
But, if we setup a ocfs2 file system resource in pacemaker,
the pacemaker resource agent will consider ocfs2 file system resource startup failure before this command returns,
the pacemaker will fence node clvm1. 
This problem is impacting our customer's estimate, since they think the two rings can be switched smoothly.

According to this problem, I can see the mount command is hanged with the below back trace,
clvm1:/ # cat /proc/6688/stack
[<ffffffffa04b8f2d>] new_lockspace+0x92d/0xa70 [dlm]
[<ffffffffa04b92d9>] dlm_new_lockspace+0x69/0x160 [dlm]
[<ffffffffa04db758>] user_cluster_connect+0xc8/0x350 [ocfs2_stack_user]
[<ffffffffa0483872>] ocfs2_cluster_connect+0x192/0x240 [ocfs2_stackglue]
[<ffffffffa0577efc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2]
[<ffffffffa05c2983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2]
[<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0
[<ffffffff8120ea1a>] mount_fs+0x3a/0x170
[<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110
[<ffffffff8122b123>] do_mount+0x213/0xcd0
[<ffffffff8122bed5>] SyS_mount+0x85/0xd0
[<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
[<ffffffffffffffff>] 0xffffffffffffffff

The root cause is in sctp_connect_to_sock() function in lowcomms.c,
1075
1076         log_print("connecting to %d", con->nodeid);
1077
1078         /* Turn off Nagle's algorithm */
1079         kernel_setsockopt(sock, SOL_TCP, TCP_NODELAY, (char *)&one,
1080                           sizeof(one));
1081
1082         result = sock->ops->connect(sock, (struct sockaddr *)&daddr, addr_len,
1083                                    O_NONBLOCK);  <<= here, this invoking will cost > 5 mins before return ETIMEDOUT(-110).
1084         printk(KERN_ERR "sctp_connect_to_sock connect: %d\n", result);
1085
1086         if (result == -EINPROGRESS)
1087                 result = 0;
1088         if (result == 0)
1089                 goto out;

Then, I want to know if this problem was found/fixed before? 
it looks DLM can not switch the second ring very quickly, this will impact the above application (e.g. CLVM, ocfs2) to create a new lock space before it's startup.

Thanks
Gang

[Cluster-devel] [ClusterLabs] DLM connection channel switch take too long time (> 5mins)

[FeldHost™ Admin]

Hello Gang He,

which type of corosync rrp_mode you use? Passive or Active? Did you try test both? Also, what kernel version you use? I see some SCTP fixes in latest kernels.

> On 8 Mar 2018, at 08:52, Gang He <ghe@suse.com> wrote:
> 
> Hello list and David Teigland,
> 
> I got a problem under a two rings cluster, the problem can be reproduced with the below steps.
> 1) setup a two rings cluster with two nodes.
> e.g. 
> clvm1(nodeid 172204569)  addr_list eth0 10.67.162.25 eth1 192.168.152.240
> clvm2(nodeid 172204570)  addr_list eth0 10.67.162.26 eth1 192.168.152.103
> 
> 2) the whole cluster works well, then I put eth0 down on node clvm2, and restart pacemaker service on that node.
> ifconfig eth0 down
> rcpacemaker restart
> 
> 3) the whole cluster still work well (that means corosync is very smooth to switch to the other ring).
> Then, I can mount ocfs2 file system on node clvm2 quickly with the command 
> mount /dev/sda /mnt/ocfs2 
> 
> 4) Next, I do the same mount on node clvm1, the mount command will be hanged for about 5 mins, and finally the mount command is done.
> But, if we setup a ocfs2 file system resource in pacemaker,
> the pacemaker resource agent will consider ocfs2 file system resource startup failure before this command returns,
> the pacemaker will fence node clvm1. 
> This problem is impacting our customer's estimate, since they think the two rings can be switched smoothly.
> 
> According to this problem, I can see the mount command is hanged with the below back trace,
> clvm1:/ # cat /proc/6688/stack
> [<ffffffffa04b8f2d>] new_lockspace+0x92d/0xa70 [dlm]
> [<ffffffffa04b92d9>] dlm_new_lockspace+0x69/0x160 [dlm]
> [<ffffffffa04db758>] user_cluster_connect+0xc8/0x350 [ocfs2_stack_user]
> [<ffffffffa0483872>] ocfs2_cluster_connect+0x192/0x240 [ocfs2_stackglue]
> [<ffffffffa0577efc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2]
> [<ffffffffa05c2983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2]
> [<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0
> [<ffffffff8120ea1a>] mount_fs+0x3a/0x170
> [<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110
> [<ffffffff8122b123>] do_mount+0x213/0xcd0
> [<ffffffff8122bed5>] SyS_mount+0x85/0xd0
> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
> [<ffffffffffffffff>] 0xffffffffffffffff
> 
> The root cause is in sctp_connect_to_sock() function in lowcomms.c,
> 1075
> 1076         log_print("connecting to %d", con->nodeid);
> 1077
> 1078         /* Turn off Nagle's algorithm */
> 1079         kernel_setsockopt(sock, SOL_TCP, TCP_NODELAY, (char *)&one,
> 1080                           sizeof(one));
> 1081
> 1082         result = sock->ops->connect(sock, (struct sockaddr *)&daddr, addr_len,
> 1083                                    O_NONBLOCK);  <<= here, this invoking will cost > 5 mins before return ETIMEDOUT(-110).
> 1084         printk(KERN_ERR "sctp_connect_to_sock connect: %d\n", result);
> 1085
> 1086         if (result == -EINPROGRESS)
> 1087                 result = 0;
> 1088         if (result == 0)
> 1089                 goto out;
> 
> Then, I want to know if this problem was found/fixed before? 
> it looks DLM can not switch the second ring very quickly, this will impact the above application (e.g. CLVM, ocfs2) to create a new lock space before it's startup.
> 
> Thanks
> Gang
> 
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

[Cluster-devel] [ClusterLabs] DLM connection channel switch take too long time (> 5mins)

[Gang He]

Hi Feldhost,


>>> 
> Hello Gang He,
> 
> which type of corosync rrp_mode you use? Passive or Active? 
clvm1:/etc/corosync # cat corosync.conf  | grep rrp_mode
        rrp_mode:       passive

Did you try test both?
No, only this mode. 
Also, what kernel version you use? I see some SCTP fixes in latest kernels.
clvm1:/etc/corosync # uname -r
4.4.114-94.11-default
It looks that sock->ops->connect() function is blocked for too long time before return, under broken network situation. 
In normal network, sock->ops->connect() function returns very quickly.

Thanks
Gang

> 
>> On 8 Mar 2018, at 08:52, Gang He <ghe@suse.com> wrote:
>> 
>> Hello list and David Teigland,
>> 
>> I got a problem under a two rings cluster, the problem can be reproduced 
> with the below steps.
>> 1) setup a two rings cluster with two nodes.
>> e.g. 
>> clvm1(nodeid 172204569)  addr_list eth0 10.67.162.25 eth1 192.168.152.240
>> clvm2(nodeid 172204570)  addr_list eth0 10.67.162.26 eth1 192.168.152.103
>> 
>> 2) the whole cluster works well, then I put eth0 down on node clvm2, and 
> restart pacemaker service on that node.
>> ifconfig eth0 down
>> rcpacemaker restart
>> 
>> 3) the whole cluster still work well (that means corosync is very smooth to 
> switch to the other ring).
>> Then, I can mount ocfs2 file system on node clvm2 quickly with the command 
>> mount /dev/sda /mnt/ocfs2 
>> 
>> 4) Next, I do the same mount on node clvm1, the mount command will be hanged 
> for about 5 mins, and finally the mount command is done.
>> But, if we setup a ocfs2 file system resource in pacemaker,
>> the pacemaker resource agent will consider ocfs2 file system resource 
> startup failure before this command returns,
>> the pacemaker will fence node clvm1. 
>> This problem is impacting our customer's estimate, since they think the two 
> rings can be switched smoothly.
>> 
>> According to this problem, I can see the mount command is hanged with the 
> below back trace,
>> clvm1:/ # cat /proc/6688/stack
>> [<ffffffffa04b8f2d>] new_lockspace+0x92d/0xa70 [dlm]
>> [<ffffffffa04b92d9>] dlm_new_lockspace+0x69/0x160 [dlm]
>> [<ffffffffa04db758>] user_cluster_connect+0xc8/0x350 [ocfs2_stack_user]
>> [<ffffffffa0483872>] ocfs2_cluster_connect+0x192/0x240 [ocfs2_stackglue]
>> [<ffffffffa0577efc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2]
>> [<ffffffffa05c2983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2]
>> [<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0
>> [<ffffffff8120ea1a>] mount_fs+0x3a/0x170
>> [<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110
>> [<ffffffff8122b123>] do_mount+0x213/0xcd0
>> [<ffffffff8122bed5>] SyS_mount+0x85/0xd0
>> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
>> [<ffffffffffffffff>] 0xffffffffffffffff
>> 
>> The root cause is in sctp_connect_to_sock() function in lowcomms.c,
>> 1075
>> 1076         log_print("connecting to %d", con->nodeid);
>> 1077
>> 1078         /* Turn off Nagle's algorithm */
>> 1079         kernel_setsockopt(sock, SOL_TCP, TCP_NODELAY, (char *)&one,
>> 1080                           sizeof(one));
>> 1081
>> 1082         result = sock->ops->connect(sock, (struct sockaddr *)&daddr, 
> addr_len,
>> 1083                                    O_NONBLOCK);  <<= here, this invoking 
> will cost > 5 mins before return ETIMEDOUT(-110).
>> 1084         printk(KERN_ERR "sctp_connect_to_sock connect: %d\n", result);
>> 1085
>> 1086         if (result == -EINPROGRESS)
>> 1087                 result = 0;
>> 1088         if (result == 0)
>> 1089                 goto out;
>> 
>> Then, I want to know if this problem was found/fixed before? 
>> it looks DLM can not switch the second ring very quickly, this will impact 
> the above application (e.g. CLVM, ocfs2) to create a new lock space before 
> it's startup.
>> 
>> Thanks
>> Gang
>> 
>> 
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org 
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> Project Home: http://www.clusterlabs.org 
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>> Bugs: http://bugs.clusterlabs.org

[Cluster-devel] [ClusterLabs] DLM connection channel switch take too long time (> 5mins)

[FeldHost™ Admin]

Hi, so try to use active mode.

https://www.suse.com/documentation/sle_ha/book_sleha/data/sec_ha_installation_terms.html

That fixes I saw in 4.14.*

> On 8 Mar 2018, at 09:12, Gang He <ghe@suse.com> wrote:
> 
> Hi Feldhost,
> 
> 
>>>> 
>> Hello Gang He,
>> 
>> which type of corosync rrp_mode you use? Passive or Active? 
> clvm1:/etc/corosync # cat corosync.conf  | grep rrp_mode
>        rrp_mode:       passive
> 
> Did you try test both?
> No, only this mode. 
> Also, what kernel version you use? I see some SCTP fixes in latest kernels.
> clvm1:/etc/corosync # uname -r
> 4.4.114-94.11-default
> It looks that sock->ops->connect() function is blocked for too long time before return, under broken network situation. 
> In normal network, sock->ops->connect() function returns very quickly.
> 
> Thanks
> Gang
> 
>> 
>>> On 8 Mar 2018, at 08:52, Gang He <ghe@suse.com> wrote:
>>> 
>>> Hello list and David Teigland,
>>> 
>>> I got a problem under a two rings cluster, the problem can be reproduced 
>> with the below steps.
>>> 1) setup a two rings cluster with two nodes.
>>> e.g. 
>>> clvm1(nodeid 172204569)  addr_list eth0 10.67.162.25 eth1 192.168.152.240
>>> clvm2(nodeid 172204570)  addr_list eth0 10.67.162.26 eth1 192.168.152.103
>>> 
>>> 2) the whole cluster works well, then I put eth0 down on node clvm2, and 
>> restart pacemaker service on that node.
>>> ifconfig eth0 down
>>> rcpacemaker restart
>>> 
>>> 3) the whole cluster still work well (that means corosync is very smooth to 
>> switch to the other ring).
>>> Then, I can mount ocfs2 file system on node clvm2 quickly with the command 
>>> mount /dev/sda /mnt/ocfs2 
>>> 
>>> 4) Next, I do the same mount on node clvm1, the mount command will be hanged 
>> for about 5 mins, and finally the mount command is done.
>>> But, if we setup a ocfs2 file system resource in pacemaker,
>>> the pacemaker resource agent will consider ocfs2 file system resource 
>> startup failure before this command returns,
>>> the pacemaker will fence node clvm1. 
>>> This problem is impacting our customer's estimate, since they think the two 
>> rings can be switched smoothly.
>>> 
>>> According to this problem, I can see the mount command is hanged with the 
>> below back trace,
>>> clvm1:/ # cat /proc/6688/stack
>>> [<ffffffffa04b8f2d>] new_lockspace+0x92d/0xa70 [dlm]
>>> [<ffffffffa04b92d9>] dlm_new_lockspace+0x69/0x160 [dlm]
>>> [<ffffffffa04db758>] user_cluster_connect+0xc8/0x350 [ocfs2_stack_user]
>>> [<ffffffffa0483872>] ocfs2_cluster_connect+0x192/0x240 [ocfs2_stackglue]
>>> [<ffffffffa0577efc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2]
>>> [<ffffffffa05c2983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2]
>>> [<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0
>>> [<ffffffff8120ea1a>] mount_fs+0x3a/0x170
>>> [<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110
>>> [<ffffffff8122b123>] do_mount+0x213/0xcd0
>>> [<ffffffff8122bed5>] SyS_mount+0x85/0xd0
>>> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>> 
>>> The root cause is in sctp_connect_to_sock() function in lowcomms.c,
>>> 1075
>>> 1076         log_print("connecting to %d", con->nodeid);
>>> 1077
>>> 1078         /* Turn off Nagle's algorithm */
>>> 1079         kernel_setsockopt(sock, SOL_TCP, TCP_NODELAY, (char *)&one,
>>> 1080                           sizeof(one));
>>> 1081
>>> 1082         result = sock->ops->connect(sock, (struct sockaddr *)&daddr, 
>> addr_len,
>>> 1083                                    O_NONBLOCK);  <<= here, this invoking 
>> will cost > 5 mins before return ETIMEDOUT(-110).
>>> 1084         printk(KERN_ERR "sctp_connect_to_sock connect: %d\n", result);
>>> 1085
>>> 1086         if (result == -EINPROGRESS)
>>> 1087                 result = 0;
>>> 1088         if (result == 0)
>>> 1089                 goto out;
>>> 
>>> Then, I want to know if this problem was found/fixed before? 
>>> it looks DLM can not switch the second ring very quickly, this will impact 
>> the above application (e.g. CLVM, ocfs2) to create a new lock space before 
>> it's startup.
>>> 
>>> Thanks
>>> Gang
>>> 
>>> 
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org 
>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>> 
>>> Project Home: http://www.clusterlabs.org 
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>>> Bugs: http://bugs.clusterlabs.org

[Cluster-devel] [ClusterLabs] DLM connection channel switch take too long time (> 5mins)

[Gang He]

Hi Feldhost,

I use active rrp_mode in corosync.conf and reboot the cluster to let the configuration effective.
But, the about 5 mins hang in new_lockspace() function is still here.

Thanks
Gang
 

>>> 
> Hi, so try to use active mode.
> 
> https://www.suse.com/documentation/sle_ha/book_sleha/data/sec_ha_installatio 
> n_terms.html
> 
> That fixes I saw in 4.14.*
> 
>> On 8 Mar 2018, at 09:12, Gang He <ghe@suse.com> wrote:
>> 
>> Hi Feldhost,
>> 
>> 
>>>>> 
>>> Hello Gang He,
>>> 
>>> which type of corosync rrp_mode you use? Passive or Active? 
>> clvm1:/etc/corosync # cat corosync.conf  | grep rrp_mode
>>        rrp_mode:       passive
>> 
>> Did you try test both?
>> No, only this mode. 
>> Also, what kernel version you use? I see some SCTP fixes in latest kernels.
>> clvm1:/etc/corosync # uname -r
>> 4.4.114-94.11-default
>> It looks that sock->ops->connect() function is blocked for too long time before 
> return, under broken network situation. 
>> In normal network, sock->ops->connect() function returns very quickly.
>> 
>> Thanks
>> Gang
>> 
>>> 
>>>> On 8 Mar 2018, at 08:52, Gang He <ghe@suse.com> wrote:
>>>> 
>>>> Hello list and David Teigland,
>>>> 
>>>> I got a problem under a two rings cluster, the problem can be reproduced 
>>> with the below steps.
>>>> 1) setup a two rings cluster with two nodes.
>>>> e.g. 
>>>> clvm1(nodeid 172204569)  addr_list eth0 10.67.162.25 eth1 192.168.152.240
>>>> clvm2(nodeid 172204570)  addr_list eth0 10.67.162.26 eth1 192.168.152.103
>>>> 
>>>> 2) the whole cluster works well, then I put eth0 down on node clvm2, and 
>>> restart pacemaker service on that node.
>>>> ifconfig eth0 down
>>>> rcpacemaker restart
>>>> 
>>>> 3) the whole cluster still work well (that means corosync is very smooth to 
>>> switch to the other ring).
>>>> Then, I can mount ocfs2 file system on node clvm2 quickly with the command 
>>>> mount /dev/sda /mnt/ocfs2 
>>>> 
>>>> 4) Next, I do the same mount on node clvm1, the mount command will be hanged 
> 
>>> for about 5 mins, and finally the mount command is done.
>>>> But, if we setup a ocfs2 file system resource in pacemaker,
>>>> the pacemaker resource agent will consider ocfs2 file system resource 
>>> startup failure before this command returns,
>>>> the pacemaker will fence node clvm1. 
>>>> This problem is impacting our customer's estimate, since they think the two 
>>> rings can be switched smoothly.
>>>> 
>>>> According to this problem, I can see the mount command is hanged with the 
>>> below back trace,
>>>> clvm1:/ # cat /proc/6688/stack
>>>> [<ffffffffa04b8f2d>] new_lockspace+0x92d/0xa70 [dlm]
>>>> [<ffffffffa04b92d9>] dlm_new_lockspace+0x69/0x160 [dlm]
>>>> [<ffffffffa04db758>] user_cluster_connect+0xc8/0x350 [ocfs2_stack_user]
>>>> [<ffffffffa0483872>] ocfs2_cluster_connect+0x192/0x240 [ocfs2_stackglue]
>>>> [<ffffffffa0577efc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2]
>>>> [<ffffffffa05c2983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2]
>>>> [<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0
>>>> [<ffffffff8120ea1a>] mount_fs+0x3a/0x170
>>>> [<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110
>>>> [<ffffffff8122b123>] do_mount+0x213/0xcd0
>>>> [<ffffffff8122bed5>] SyS_mount+0x85/0xd0
>>>> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
>>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>> 
>>>> The root cause is in sctp_connect_to_sock() function in lowcomms.c,
>>>> 1075
>>>> 1076         log_print("connecting to %d", con->nodeid);
>>>> 1077
>>>> 1078         /* Turn off Nagle's algorithm */
>>>> 1079         kernel_setsockopt(sock, SOL_TCP, TCP_NODELAY, (char *)&one,
>>>> 1080                           sizeof(one));
>>>> 1081
>>>> 1082         result = sock->ops->connect(sock, (struct sockaddr *)&daddr, 
>>> addr_len,
>>>> 1083                                    O_NONBLOCK);  <<= here, this invoking 
>>> will cost > 5 mins before return ETIMEDOUT(-110).
>>>> 1084         printk(KERN_ERR "sctp_connect_to_sock connect: %d\n", result);
>>>> 1085
>>>> 1086         if (result == -EINPROGRESS)
>>>> 1087                 result = 0;
>>>> 1088         if (result == 0)
>>>> 1089                 goto out;
>>>> 
>>>> Then, I want to know if this problem was found/fixed before? 
>>>> it looks DLM can not switch the second ring very quickly, this will impact 
>>> the above application (e.g. CLVM, ocfs2) to create a new lock space before 
>>> it's startup.
>>>> 
>>>> Thanks
>>>> Gang
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Users mailing list: Users at clusterlabs.org 
>>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>> 
>>>> Project Home: http://www.clusterlabs.org 
>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>>>> Bugs: http://bugs.clusterlabs.org

[Cluster-devel] [ClusterLabs] DLM connection channel switch take too long time (> 5mins)

[David Teigland]

> I use active rrp_mode in corosync.conf and reboot the cluster to let the configuration effective.
> But, the about 5 mins hang in new_lockspace() function is still here.

The last time I tested connection failures with sctp was several years
ago, but I recall seeing similar problems.  I had hoped that some of the
sctp changes might have helped, but perhaps they didn't.
Dave

[Cluster-devel] [ClusterLabs] DLM connection channel switch take too long time (> 5mins)

[Gang He]

Hello David,

If sctp implementation did not fix this problem, there is any workaround for a two-rings cluster?
Could we use TCP protocol in DLM under a two-rings cluster to by-pass connection channel switch issue?

Thanks
Gang 


>>> 
>>  I use active rrp_mode in corosync.conf and reboot the cluster to let the 
> configuration effective.
>> But, the about 5 mins hang in new_lockspace() function is still here.
> 
> The last time I tested connection failures with sctp was several years
> ago, but I recall seeing similar problems.  I had hoped that some of the
> sctp changes might have helped, but perhaps they didn't.
> Dave

[Cluster-devel] [ClusterLabs] DLM connection channel switch take too long time (> 5mins)

[Digimer]

On 2018-03-08 12:10 PM, David Teigland wrote:
>> I use active rrp_mode in corosync.conf and reboot the cluster to let the configuration effective.
>> But, the about 5 mins hang in new_lockspace() function is still here.
> 
> The last time I tested connection failures with sctp was several years
> ago, but I recall seeing similar problems.  I had hoped that some of the
> sctp changes might have helped, but perhaps they didn't.
> Dave

To add to this; We found serious issues with DLM over sctp/rrp. Our
solution was to remove RRP and reply on active/passive (mode=1) bonding.
I do not believe you can make anything using DLM reliable on RRP in
either active or passive mode.

-- 
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein?s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould

[Cluster-devel] [ClusterLabs] DLM connection channel switch take too long time (> 5mins)

[Gang He]

Hello Digimer,



>>> 
> On 2018-03-08 12:10 PM, David Teigland wrote:
>>> I use active rrp_mode in corosync.conf and reboot the cluster to let the 
> configuration effective.
>>> But, the about 5 mins hang in new_lockspace() function is still here.
>> 
>> The last time I tested connection failures with sctp was several years
>> ago, but I recall seeing similar problems.  I had hoped that some of the
>> sctp changes might have helped, but perhaps they didn't.
>> Dave
> 
> To add to this; We found serious issues with DLM over sctp/rrp. Our
> solution was to remove RRP and reply on active/passive (mode=1) bonding.
> I do not believe you can make anything using DLM reliable on RRP in
> either active or passive mode.
Do you have the detailed steps to describe this workaround? 
My means is, how to remove RRP? and reply on active/passive (mode=1) bonding?
From the code, we have to use sctp protocol in DLM on a two-rings cluster.

Thanks
Gang

> 
> -- 
> Digimer
> Papers and Projects: https://alteeve.com/w/ 
> "I am, somehow, less interested in the weight and convolutions of
> Einstein?s brain than in the near certainty that people of equal talent
> have lived and died in cotton fields and sweatshops." - Stephen Jay Gould

[Cluster-devel] [ClusterLabs] DLM connection channel switch take too long time (> 5mins)

[Digimer]

On 2018-03-09 01:32 AM, Gang He wrote:
> Hello Digimer,
> 
> 
> 
>>>>
>> On 2018-03-08 12:10 PM, David Teigland wrote:
>>>> I use active rrp_mode in corosync.conf and reboot the cluster to let the 
>> configuration effective.
>>>> But, the about 5 mins hang in new_lockspace() function is still here.
>>>
>>> The last time I tested connection failures with sctp was several years
>>> ago, but I recall seeing similar problems.  I had hoped that some of the
>>> sctp changes might have helped, but perhaps they didn't.
>>> Dave
>>
>> To add to this; We found serious issues with DLM over sctp/rrp. Our
>> solution was to remove RRP and reply on active/passive (mode=1) bonding.
>> I do not believe you can make anything using DLM reliable on RRP in
>> either active or passive mode.
> Do you have the detailed steps to describe this workaround? 
> My means is, how to remove RRP? and reply on active/passive (mode=1) bonding?
> From the code, we have to use sctp protocol in DLM on a two-rings cluster.
> 
> Thanks
> Gang

I'm using RHEL 6, so for me, disabling rrp was simply removing the rrp
attribute and the <altname> child elements. As for bonding, here's how I
did it;

https://www.alteeve.com/w/AN!Cluster_Tutorial_2#Configuring_our_Bridge.2C_Bonds_and_Interfaces

-- 
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein?s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould