[Cluster-devel] [RFC PATCH dlm 00/10] dlm_controld config settings can be

[Cluster-devel] [RFC PATCH dlm 00/10] dlm_controld config settings can be

[Heming Zhao]

This new feature gives dlm ability to change config settings dynamically.
It's very useful for debugging, and also useful for adjusting HA 
behaviours while running. This feature gives more power function to
advanced users. And HA Resource-Agent (RA) could also benefit from this
feature.

The new man page dlm_tool.8 show some example of this feature.

For easy managing command running list, there is a patch to add "-A" on
"dlm run_(check|cancel)". We can get rid of the boring uuid input when
checking lots of running commands result/status.

Heming Zhao (10):
  dlm_tool man: add command "joinleave", add "USAGE" section
  dlm_tool: add run_(check|cancel) all feature
  dlm_tool man: add dynamic setting and examples
  dlm_controld: put MAX_LINE in header file
  dlm_controld: add dynamic setting items in "struct dlm_option"
  dlm_controld: change dlm_options[] to shared memory type
  dlm_controld: make few APIs public
  dlm_controld: support "dlm_tool dump_config" to show dynamic setting
  dlm_controld: add new API set_opt_online()
  dlm_controld: enable "dlm_tool run|run_start" dynamic setting feature

 dlm_controld/action.c      |   5 ++
 dlm_controld/config.c      | 132 +++++++++++++++++++++++++++++++++--
 dlm_controld/dlm_daemon.h  |  15 +++-
 dlm_controld/helper.c      |  41 ++++++++---
 dlm_controld/logging.c     |  18 +++--
 dlm_controld/main.c        |  86 ++++++++++++++---------
 dlm_controld/node_config.c |   2 -
 dlm_tool/dlm_tool.8        | 139 +++++++++++++++++++++++++++++++++++++
 dlm_tool/main.c            |  80 +++++++++++++++------
 9 files changed, 442 insertions(+), 76 deletions(-)

-- 
2.32.0

[Cluster-devel] [PATCH 01/10] dlm_tool man: add command "joinleave", add "USAGE" section

[Heming Zhao]

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_tool/dlm_tool.8 | 79 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)

diff --git a/dlm_tool/dlm_tool.8 b/dlm_tool/dlm_tool.8
index bbc11c261b7e..056c4af0213c 100644
--- a/dlm_tool/dlm_tool.8
+++ b/dlm_tool/dlm_tool.8
@@ -48,6 +48,10 @@ dlm_tool \- a utility for the dlm and dlm_controld daemon
 .br
 	Leave a lockspace.
 
+.BI joinleave " name"
+.br
+	Do two actions: first Join then Leave a lockspace.
+
 .BI lockdebug " name"
 .br
 	Complete display of locks from the lockspace.
@@ -112,6 +116,81 @@ Print help, then exit
 .B \-V
 Print program version information, then exit
 
+.SH USAGE
+
+dlm_tool ls [-n] [ls_name]
+
+-
+
+dlm_tool status [-v]
+
+-
+
+dlm_tool dump
+
+-
+
+dlm_tool dump_config
+
+-
+
+dlm_tool fence_ack \fIls-name\fP
+
+-
+
+dlm_tool log_plock
+
+-
+
+dlm_tool plocks \fIls-name\fP
+
+-
+
+dlm_tool join [-e 0|1] [-f 0|1] \fIls-name\fP
+
+-
+
+dlm_tool leave \fIls-name\fP
+
+-
+
+dlm_tool joinleave [-e 0|1] [-f 0|1] \fIls-name\fP
+
+-
+
+dlm_tool lockdebug [-s] [-v] [-w] \fIls-name\fP
+
+-
+
+dlm_tool lockdump [-M] \fIls-name\fP
+
+-
+
+dlm_tool run|run_start [-n] \fIcommand\fP
+
+* current run & run_start support \fIcommand\fP:
+.RS 4
+"lvm lvchange --refresh"
+.br
+"lvm lvs"
+.RE
+
+-
+
+dlm_tool run_check|run_cancel [-i "sec"] \fIuuid\fP
+
+-
+
+dlm_tool run_list
+
+-
+
+dlm_tool -h
+
+-
+
+dlm_tool -V
+
 .SH SEE ALSO
 .BR dlm_controld (8),
 .BR dlm.conf (5)
-- 
2.32.0

[Cluster-devel] [PATCH 02/10] dlm_tool: add run_(check|cancel) all feature

[Heming Zhao]

Current run_check and run_cancel require uuid for executing.
If users want to check/cancel all commands, they must input every
command uuid, which is a boring action.
This commit add a new option "-A", which makes one time travel
all the command without uuid.

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_tool/dlm_tool.8 |  5 ++-
 dlm_tool/main.c     | 80 ++++++++++++++++++++++++++++++++-------------
 2 files changed, 62 insertions(+), 23 deletions(-)

diff --git a/dlm_tool/dlm_tool.8 b/dlm_tool/dlm_tool.8
index 056c4af0213c..086596c67dbf 100644
--- a/dlm_tool/dlm_tool.8
+++ b/dlm_tool/dlm_tool.8
@@ -110,6 +110,9 @@ Wide lockdebug output
 .B \-M
 Include MSTCPY locks in lockdump output
 
+.B \-A
+Include all uuid in run_check and run_cancel
+
 .B \-h
 Print help, then exit
 
@@ -177,7 +180,7 @@ dlm_tool run|run_start [-n] \fIcommand\fP
 
 -
 
-dlm_tool run_check|run_cancel [-i "sec"] \fIuuid\fP
+dlm_tool run_check|run_cancel [-A] [-i "sec"] \fIuuid\fP
 
 -
 
diff --git a/dlm_tool/main.c b/dlm_tool/main.c
index 53d9336d59c6..37f381f6d980 100644
--- a/dlm_tool/main.c
+++ b/dlm_tool/main.c
@@ -53,6 +53,7 @@ static char *lsname;
 static int operation;
 static int opt_ind;
 static int ls_all_nodes = 0;
+static int opt_all_uuid = 0;
 static int opt_excl = 0;
 static int opt_fs = 0;
 static int dump_mstcpy = 0;
@@ -215,7 +216,7 @@ static void print_usage(void)
 	printf("\n");
 }
 
-#define OPTION_STRING "MhVnm:e:f:vwsi:"
+#define OPTION_STRING "MhVAnm:e:f:vwsi:"
 
 static void decode_arguments(int argc, char **argv)
 {
@@ -254,6 +255,10 @@ static void decode_arguments(int argc, char **argv)
 			ls_all_nodes = 1;
 			break;
 
+		case 'A':
+			opt_all_uuid = 1;
+			break;
+
 		case 's':
 			summarize = 1;
 			break;
@@ -456,8 +461,10 @@ static void decode_arguments(int argc, char **argv)
 		fprintf(stderr, "command required\n");
 		exit(EXIT_FAILURE);
 	} else if (need_uuid) {
-		fprintf(stderr, "uuid required\n");
-		exit(EXIT_FAILURE);
+		if (!opt_all_uuid) {
+			fprintf(stderr, "uuid required\n");
+			exit(EXIT_FAILURE);
+		}
 	} else {
 		fprintf(stderr, "missing arg\n");
 		exit(EXIT_FAILURE);
@@ -1486,14 +1493,42 @@ static void do_log_plock(void)
 	printf("\n");
 }
 
+static int check_or_cancel(int do_cancel)
+{
+	uint32_t flags = 0;
+	uint32_t check_status = 0;
+	int rv = 0;
+
+	if (do_cancel)
+		flags = DLMC_FLAG_RUN_CHECK_CANCEL;
+	else
+		flags = DLMC_FLAG_RUN_CHECK_CLEAR;
+
+	rv = dlmc_run_check(run_uuid, strlen(run_uuid), wait_sec, flags,
+			&check_status);
+
+	printf("check_status: ");
+
+	if (check_status & DLMC_RUN_STATUS_WAITING)
+		printf("waiting ");
+	if (check_status & DLMC_RUN_STATUS_DONE)
+		printf("done ");
+	if (check_status & DLMC_RUN_STATUS_FAILED)
+		printf("failed ");
+	printf("\n");
+	return rv;
+}
+
 static int do_run(int op)
 {
 	int do_start = (op == OP_RUN) || (op == OP_RUN_START);
 	int do_check = (op == OP_RUN) || (op == OP_RUN_CHECK);
 	int do_cancel = (op == OP_RUN_CANCEL);
 	uint32_t flags = 0;
-	uint32_t check_status = 0;
-	int rv = 0;
+	int rv = 0, pos = 0;
+	char buf[DLMC_DUMP_SIZE];
+	char *p;
+
 
 	if (do_start) {
 		/* FIXME: use proper option to specify */
@@ -1508,23 +1543,24 @@ static int do_run(int op)
 	}
 
 	if (do_check || do_cancel) {
-		if (do_cancel)
-			flags = DLMC_FLAG_RUN_CHECK_CANCEL;
-		else
-			flags = DLMC_FLAG_RUN_CHECK_CLEAR;
-
-		rv = dlmc_run_check(run_uuid, strlen(run_uuid), wait_sec, flags,
-				    &check_status);
-
-		printf("check_status: ");
-
-		if (check_status & DLMC_RUN_STATUS_WAITING)
-			printf("waiting ");
-		if (check_status & DLMC_RUN_STATUS_DONE)
-			printf("done ");
-		if (check_status & DLMC_RUN_STATUS_FAILED)
-			printf("failed ");
-		printf("\n");
+		if (opt_all_uuid) {
+			memset(buf, 0, sizeof(buf));
+			dlmc_dump_run(buf);
+			buf[DLMC_DUMP_SIZE-1] = '\0';
+			while (1) {
+				p = strstr(buf + pos, "run_uuid ");
+				if (!p)
+					break;
+				pos = p - buf;
+				sscanf(buf + pos, "run_uuid %s start_nodeid", run_uuid);
+				pos += 10 + strlen(run_uuid); /* 10: "run_uuid" + 2 spaces */
+				printf("uuid: %s\n", run_uuid);
+				rv += check_or_cancel(do_cancel);
+			}
+			return rv;
+		} else {
+			rv = check_or_cancel(do_cancel);
+		}
 	}
 
 	return rv;
-- 
2.32.0

[Cluster-devel] [PATCH 03/10] dlm_tool man: add dynamic setting and examples

[Heming Zhao]

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_tool/dlm_tool.8 | 57 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

diff --git a/dlm_tool/dlm_tool.8 b/dlm_tool/dlm_tool.8
index 086596c67dbf..cdfe948815bf 100644
--- a/dlm_tool/dlm_tool.8
+++ b/dlm_tool/dlm_tool.8
@@ -176,6 +176,8 @@ dlm_tool run|run_start [-n] \fIcommand\fP
 "lvm lvchange --refresh"
 .br
 "lvm lvs"
+.br
+"opt dlm_controld-config-item=x"  (see below \fBEXAMPLES\fP section)
 .RE
 
 -
@@ -194,6 +196,61 @@ dlm_tool -h
 
 dlm_tool -V
 
+.SH EXAMPLES
+
+command \fIdlm_tool run "opt xxx"\fP can do the dynamic config job. It makes possible to debug, or to do flexible operations on the fly.
+
+If the line of "dlm_tool dump_config" output leading with "(*)", the START symbol, it means this item supports dynamic config. i.e.
+
+.RS 4
+.EX
+# dlm_tool dump_config | grep log_debug
+(*-) log_debug=1
+.EE
+
+The "log_debug" supports dynamic setting.
+.br
+The '-' (minus) means dynamic config doesn't take effect on this item.
+.br
+If "(*-)" becomes "(*+)", it means this item is dynamically changed on the fly.
+.RE
+
+
+Below show options: restore, restore_all
+
+.RS 4
+.EX
+# dlm_tool dump_config | grep log_debug
+(*-) log_debug=1
+# dlm_tool run "opt log_debug=0"
+# dlm_tool dump_config | grep log_debug
+(*+) log_debug=0
+
+# dlm_tool run "opt log_debug=1"
+# dlm_tool dump_config | grep log_debug
+(*+) log_debug=1
+
+# dlm_tool run "opt log_debug=restore"
+# dlm_tool dump_config | grep log_debug
+(*-) log_debug=1
+
+# dlm_tool dump_config | grep _debug
+(*-) daemon_debug=0
+(*-) log_debug=1
+(*-) plock_debug=0
+# dlm_tool run "opt daemon_debug=1 log_debug=0"
+# dlm_tool dump_config | grep _debug
+(*+) daemon_debug=1
+(*+) log_debug=0
+(*-) plock_debug=0
+# dlm_tool run "opt restore_all"
+# dlm_tool dump_config | grep _debug
+(*-) daemon_debug=0
+(*-) log_debug=1
+(*-) plock_debug=0
+.EE
+.RE
+
 .SH SEE ALSO
 .BR dlm_controld (8),
 .BR dlm.conf (5)
-- 
2.32.0

[Cluster-devel] [PATCH 04/10] dlm_controld: put MAX_LINE in header file

[Heming Zhao]

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_controld/config.c      | 4 ----
 dlm_controld/dlm_daemon.h  | 3 +++
 dlm_controld/node_config.c | 2 --
 3 files changed, 3 insertions(+), 6 deletions(-)

diff --git a/dlm_controld/config.c b/dlm_controld/config.c
index 947480dabe85..d77f3826bfe6 100644
--- a/dlm_controld/config.c
+++ b/dlm_controld/config.c
@@ -21,10 +21,6 @@ master node=2 weight=1
 
 #endif
 
-/* The max line length in dlm.conf */
-
-#define MAX_LINE 256
-
 int get_weight(struct lockspace *ls, int nodeid)
 {
 	int i;
diff --git a/dlm_controld/dlm_daemon.h b/dlm_controld/dlm_daemon.h
index 436fc9109aa6..eae17a28c749 100644
--- a/dlm_controld/dlm_daemon.h
+++ b/dlm_controld/dlm_daemon.h
@@ -220,6 +220,9 @@ void log_level(char *name_in, uint32_t level_in, const char *fmt, ...);
 #define log_dlock(ls, fmt, args...) log_level((ls)->name, LOG_PLOCK|LOG_DEBUG, fmt, ##args)
 #define log_elock(ls, fmt, args...) log_level((ls)->name, LOG_PLOCK|LOG_ERR, fmt, ##args)
 
+/* The max line length in dlm.conf */
+#define MAX_LINE 256
+
 /* dlm_header types */
 enum {
 	DLM_MSG_PROTOCOL = 1,
diff --git a/dlm_controld/node_config.c b/dlm_controld/node_config.c
index fe794be72f22..0889451848b9 100644
--- a/dlm_controld/node_config.c
+++ b/dlm_controld/node_config.c
@@ -8,8 +8,6 @@
 
 #include "dlm_daemon.h"
 
-#define MAX_LINE 4096
-
 static struct node_config nc[MAX_NODES];
 
 static const struct node_config nc_default = {
-- 
2.32.0

[Cluster-devel] [PATCH 05/10] dlm_controld: add dynamic setting items in "struct dlm_option"

[Heming Zhao]

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_controld/dlm_daemon.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/dlm_controld/dlm_daemon.h b/dlm_controld/dlm_daemon.h
index eae17a28c749..76f5f4d082a5 100644
--- a/dlm_controld/dlm_daemon.h
+++ b/dlm_controld/dlm_daemon.h
@@ -132,6 +132,7 @@ struct dlm_option {
 	const char *name;
 	char letter;
 	int req_arg;
+	int dynamic;
 	const char *desc;
 
 	int use_int;
@@ -151,6 +152,11 @@ struct dlm_option {
 	int file_int;
 	char *file_str;
 	unsigned int file_uint;
+
+	int dynamic_set;
+	int dynamic_int;
+	char *dynamic_str;
+	unsigned int dynamic_uint;
 };
 
 EXTERN struct dlm_option dlm_options[dlm_options_max];
-- 
2.32.0

[Cluster-devel] [PATCH 06/10] dlm_controld: change dlm_options[] to shared memory type

[Heming Zhao]

This commit makes dlm_options[] could be modified after fork().
The patch is foundation to dynamic setting feature.

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_controld/dlm_daemon.h |  2 +-
 dlm_controld/main.c       | 14 +++++++++++++-
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/dlm_controld/dlm_daemon.h b/dlm_controld/dlm_daemon.h
index 76f5f4d082a5..e20e98e78707 100644
--- a/dlm_controld/dlm_daemon.h
+++ b/dlm_controld/dlm_daemon.h
@@ -159,7 +159,7 @@ struct dlm_option {
 	unsigned int dynamic_uint;
 };
 
-EXTERN struct dlm_option dlm_options[dlm_options_max];
+EXTERN struct dlm_option *dlm_options;
 #define opt(x) dlm_options[x].use_int
 #define opts(x) dlm_options[x].use_str
 #define optu(x) dlm_options[x].use_uint
diff --git a/dlm_controld/main.c b/dlm_controld/main.c
index 504cafa12ec6..2a4f33b1b455 100644
--- a/dlm_controld/main.c
+++ b/dlm_controld/main.c
@@ -14,6 +14,7 @@
 #include <linux/genetlink.h>
 #include <linux/dlm_netlink.h>
 #include <uuid/uuid.h>
+#include <sys/mman.h>
 
 #ifdef USE_SD_NOTIFY
 #include <systemd/sd-daemon.h>
@@ -2104,11 +2105,21 @@ int main(int argc, char **argv)
 	struct sigaction act;
 	int fd, rv;
 
+	/* share the cfg memory for helper process */
+	dlm_options = mmap(NULL, sizeof(struct dlm_option)*dlm_options_max,
+						PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANON, -1, 0);
+	if (dlm_options == MAP_FAILED) {
+		perror("mmap failed");
+		return 1;
+	}
+
 	/*
-	 * config priority: cli, config file, default
+	 * config priority: dynamic, cli, config file, default
 	 * - explicit cli setting will override default,
 	 * - explicit file setting will override default
 	 * - explicit file setting will not override explicit cli setting
+	 * - explicit dynamic setting will override using value,
+	 *   "dlm_tool dump_config" will show which option is using dynamic setting.
 	 */
 
 	set_opt_defaults();
-- 
2.32.0

[Cluster-devel] [PATCH 07/10] dlm_controld: make few APIs public

[Heming Zhao]

This commit makes possible for helper process accessing config info

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_controld/action.c     |  5 +++++
 dlm_controld/dlm_daemon.h |  3 +++
 dlm_controld/logging.c    | 18 +++++++++++++-----
 dlm_controld/main.c       |  2 +-
 4 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/dlm_controld/action.c b/dlm_controld/action.c
index 0eff27997e1c..baddaf81cb4f 100644
--- a/dlm_controld/action.c
+++ b/dlm_controld/action.c
@@ -766,6 +766,11 @@ static int set_configfs_cluster(const char *name, char *str, int num)
 	return 0;
 }
 
+int set_configfs_opt(const char *name, char *str, int num)
+{
+	return set_configfs_cluster(name, str, num);
+}
+
 #define NET_RMEM_DEFAULT 4194304
 #define NET_RMEM_MAX 4194304
 
diff --git a/dlm_controld/dlm_daemon.h b/dlm_controld/dlm_daemon.h
index e20e98e78707..904bb0b50df9 100644
--- a/dlm_controld/dlm_daemon.h
+++ b/dlm_controld/dlm_daemon.h
@@ -392,6 +392,7 @@ int setup_configfs_members(void);
 int check_uncontrolled_lockspaces(void);
 int setup_misc_devices(void);
 int path_exists(const char *path);
+int set_configfs_opt(const char *name, char *str, int num);
 
 /* config.c */
 void set_opt_file(int update);
@@ -480,6 +481,7 @@ struct lockspace *find_ls_id(uint32_t id);
 const char *dlm_mode_str(int mode);
 void cluster_dead(int ci);
 struct dlm_option *get_dlm_option(char *name);
+int get_ind_name(char *s);
 struct run *find_run(char *uuid_str);
 void clear_run(struct run *run);
 void send_helper_run_request(struct run_request *req);
@@ -531,6 +533,7 @@ void init_logging(void);
 void close_logging(void);
 void copy_log_dump(char *buf, int *len);
 void copy_log_dump_plock(char *buf, int *len);
+void set_logfile_priority(void);
 
 /* crc.c */
 uint32_t cpgname_to_crc(const char *data, int len);
diff --git a/dlm_controld/logging.c b/dlm_controld/logging.c
index d48b8aebc237..2c57138ce766 100644
--- a/dlm_controld/logging.c
+++ b/dlm_controld/logging.c
@@ -14,6 +14,18 @@ static int logfile_priority;
 static char logfile[PATH_MAX];
 static FILE *logfile_fp;
 
+/* logfile_priority is the only one of these options that
+   can be controlled from command line, environment variable
+   and dynamic setting.
+ */
+void set_logfile_priority(void)
+{
+	if (opt(debug_logfile_ind))
+		logfile_priority = LOG_DEBUG;
+	else
+		logfile_priority = DEFAULT_LOGFILE_PRIORITY;
+}
+
 void init_logging(void)
 {
 	mode_t old_umask;
@@ -24,11 +36,7 @@ void init_logging(void)
 	logfile_priority = DEFAULT_LOGFILE_PRIORITY;
 	strcpy(logfile, DEFAULT_LOGFILE);
 
-	/* logfile_priority is the only one of these options that
-	   can be controlled from command line or environment variable */
-
-	if (opt(debug_logfile_ind))
-		logfile_priority = LOG_DEBUG;
+	set_logfile_priority();
 
 	if (logfile[0]) {
 		old_umask = umask(0077);
diff --git a/dlm_controld/main.c b/dlm_controld/main.c
index 2a4f33b1b455..91187f2d2402 100644
--- a/dlm_controld/main.c
+++ b/dlm_controld/main.c
@@ -1902,7 +1902,7 @@ static void set_opt_defaults(void)
 			"Print program version information, then exit");
 }
 
-static int get_ind_name(char *s)
+int get_ind_name(char *s)
 {
 	char name[PATH_MAX];
 	char *p = s;
-- 
2.32.0

[Cluster-devel] [PATCH 08/10] dlm_controld: support "dlm_tool dump_config" to show dynamic setting

[Heming Zhao]

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_controld/main.c | 70 +++++++++++++++++++++++++--------------------
 1 file changed, 39 insertions(+), 31 deletions(-)

diff --git a/dlm_controld/main.c b/dlm_controld/main.c
index 91187f2d2402..c4ee3540f508 100644
--- a/dlm_controld/main.c
+++ b/dlm_controld/main.c
@@ -920,17 +920,24 @@ static void copy_options(char *buf, int *len)
 {
 	struct dlm_option *o;
 	char tmp[256];
-	int i, ret, pos = 0;
+	int i, ret, pos = 0, p=5;
 
 	for (i = 0; i < dlm_options_max; i++) {
 		o = &dlm_options[i];
 
 		memset(tmp, 0, sizeof(tmp));
 
+		if (o->dynamic)
+			snprintf(tmp, p+1, "(%c%c) ", '*', o->dynamic_set ? '+' : '-');
+		else
+			memset(tmp, ' ', p);
+
 		if (o->req_arg == req_arg_str)
-			snprintf(tmp, 255, "%s=%s\n", o->name, o->use_str);
+			snprintf(tmp+p, 250, "%s=%s\n", o->name, o->use_str);
+		else if (o->req_arg == req_arg_uint)
+			snprintf(tmp+p, 250, "%s=%u\n", o->name, o->use_uint);
 		else
-			snprintf(tmp, 255, "%s=%d\n", o->name, o->use_int);
+			snprintf(tmp+p, 250, "%s=%d\n", o->name, o->use_int);
 
 		if (pos + strlen(tmp) >= LOG_DUMP_SIZE)
 			break;
@@ -1750,12 +1757,13 @@ static void print_usage(void)
 
 static void set_opt_default(int ind, const char *name, char letter, int arg_type,
 			    int default_int, const char *default_str,
-			    unsigned int default_uint, const char *desc)
+			    unsigned int default_uint, int dynamic, const char *desc)
 {
 	dlm_options[ind].name = name;
 	dlm_options[ind].letter = letter;
 	dlm_options[ind].req_arg = arg_type;
 	dlm_options[ind].desc = desc;
+	dlm_options[ind].dynamic = dynamic;
 	dlm_options[ind].default_int = default_int;
 	dlm_options[ind].default_str = default_str;
 	dlm_options[ind].default_uint = default_uint;
@@ -1768,137 +1776,137 @@ static void set_opt_defaults(void)
 {
 	set_opt_default(daemon_debug_ind,
 			"daemon_debug", 'D', no_arg,
-			0, NULL, 0,
+			0, NULL, 0, 1,
 			"enable debugging to stderr and don't fork");
 
 	set_opt_default(foreground_ind,
 			"foreground", '\0', no_arg,
-			0, NULL, 0,
+			0, NULL, 0, 0,
 			"don't fork");
 
 	set_opt_default(log_debug_ind,
 			"log_debug", 'K', no_arg,
-			0, NULL, 0,
+			0, NULL, 0, 1,
 			"enable kernel dlm debugging messages");
 
 	set_opt_default(timewarn_ind,
 			"timewarn", '\0', req_arg_int,
-			0, NULL, 0,
+			0, NULL, 0, 0,
 			""); /* do not advertise */
 
 	set_opt_default(protocol_ind,
 			"protocol", 'r', req_arg_str,
-			-1, "detect", 0,
+			-1, "detect", 0, 0,
 			"dlm kernel lowcomms protocol: tcp, sctp, detect");
 
 	set_opt_default(port_ind,
 			"port", 'R', req_arg_uint,
-			-1, NULL, 21064,
+			-1, NULL, 21064, 0,
 			"dlm kernel lowcomms protocol port");
 
 	set_opt_default(bind_all_ind,
 			"bind_all", '\0', req_arg_int,
-			0, NULL, 0,
+			0, NULL, 0, 0,
 			""); /* do not advertise */
 
 	set_opt_default(mark_ind,
 			"mark", '\0', req_arg_uint,
-			0, NULL, 0,
+			0, NULL, 0, 0,
 			"set mark value for DLM if not explicit by nodeid specified");
 
 	set_opt_default(debug_logfile_ind,
 			"debug_logfile", 'L', no_arg,
-			0, NULL, 0,
+			0, NULL, 0, 1,
 			"write debugging to log file");
 
 	set_opt_default(enable_fscontrol_ind,
 			"enable_fscontrol", '\0', req_arg_bool,
-			0, NULL, 0,
+			0, NULL, 0, 0,
 			""); /* do not advertise */
 
 	set_opt_default(enable_plock_ind,
 			"enable_plock", 'p', req_arg_bool,
-			1, NULL, 0,
+			1, NULL, 0, 0,
 			"enable/disable posix lock support for cluster fs");
 
 	set_opt_default(plock_debug_ind,
 			"plock_debug", 'P', no_arg,
-			0, NULL, 0,
+			0, NULL, 0, 1,
 			"enable plock debugging");
 
 	set_opt_default(plock_rate_limit_ind,
 			"plock_rate_limit", 'l', req_arg_int,
-			0, NULL, 0,
+			0, NULL, 0, 1,
 			"limit rate of plock operations (0 for none)");
 
 	set_opt_default(plock_ownership_ind,
 			"plock_ownership", 'o', req_arg_bool,
-			0, NULL, 0,
+			0, NULL, 0, 0,
 			"enable/disable plock ownership");
 
 	set_opt_default(drop_resources_time_ind,
 			"drop_resources_time", 't', req_arg_int,
-			10000, NULL, 0,
+			10000, NULL, 0, 1,
 			"plock ownership drop resources time (milliseconds)");
 
 	set_opt_default(drop_resources_count_ind,
 			"drop_resources_count", 'c', req_arg_int,
-			10, NULL, 0,
+			10, NULL, 0, 1,
 			"plock ownership drop resources count");
 
 	set_opt_default(drop_resources_age_ind,
 			"drop_resources_age", 'a', req_arg_int,
-			10000, NULL, 0,
+			10000, NULL, 0, 1,
 			"plock ownership drop resources age (milliseconds)");
 
 	set_opt_default(post_join_delay_ind,
 			"post_join_delay", 'j', req_arg_int,
-			30, NULL, 0,
+			30, NULL, 0, 1,
 			"seconds to delay fencing after cluster join");
 
 	set_opt_default(enable_fencing_ind,
 			"enable_fencing", 'f', req_arg_bool,
-			1, NULL, 0,
+			1, NULL, 0, 0,
 			"enable/disable fencing");
 
 	set_opt_default(enable_concurrent_fencing_ind,
 			"enable_concurrent_fencing", '\0', req_arg_bool,
-			0, NULL, 0,
+			0, NULL, 0, 0,
 			"enable/disable concurrent fencing");
 
 	set_opt_default(enable_startup_fencing_ind,
 			"enable_startup_fencing", 's', req_arg_bool,
-			1, NULL, 0,
+			1, NULL, 0, 0,
 			"enable/disable startup fencing");
 
 	set_opt_default(repeat_failed_fencing_ind,
 			"repeat_failed_fencing", '\0', req_arg_bool,
-			1, NULL, 0,
+			1, NULL, 0, 1,
 			"enable/disable retrying after fencing fails");
 
 	set_opt_default(enable_quorum_fencing_ind,
 			"enable_quorum_fencing", 'q', req_arg_bool,
-			1, NULL, 0,
+			1, NULL, 0, 1,
 			"enable/disable quorum requirement for fencing");
 
 	set_opt_default(enable_quorum_lockspace_ind,
 			"enable_quorum_lockspace", '\0', req_arg_bool,
-			1, NULL, 0,
+			1, NULL, 0, 1,
 			"enable/disable quorum requirement for lockspace operations");
 
 	set_opt_default(enable_helper_ind,
 			"enable_helper", '\0', req_arg_bool,
-			1, NULL, 0,
+			1, NULL, 0, 0,
 			"enable/disable helper process for running commands");
 
 	set_opt_default(help_ind,
 			"help", 'h', no_arg,
-			-1, NULL, 0,
+			-1, NULL, 0, 0,
 			"print this help, then exit");
 
 	set_opt_default(version_ind,
 			"version", 'V', no_arg,
-			-1, NULL, 0,
+			-1, NULL, 0, 0,
 			"Print program version information, then exit");
 }
 
-- 
2.32.0

[Cluster-devel] [PATCH 09/10] dlm_controld: add new API set_opt_online()

[Heming Zhao]

This API has ability to change dlm_controld settings on the fly.

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_controld/config.c     | 128 ++++++++++++++++++++++++++++++++++++++
 dlm_controld/dlm_daemon.h |   1 +
 2 files changed, 129 insertions(+)

diff --git a/dlm_controld/config.c b/dlm_controld/config.c
index d77f3826bfe6..9e799c8c04d2 100644
--- a/dlm_controld/config.c
+++ b/dlm_controld/config.c
@@ -292,3 +292,129 @@ void set_opt_file(int update)
 	fclose(file);
 }
 
+static void reset_opt_value(struct dlm_option *o)
+{
+	/* config priority: cli, config file, default */
+
+	if (o->cli_set) {
+		o->use_int = o->cli_int;
+		o->use_uint = o->cli_uint;
+		o->use_str = o->cli_str;
+		return;
+	}
+
+	if (o->file_set) {
+		o->use_int = o->file_int;
+		o->use_uint = o->file_uint;
+		o->use_str = o->file_str;
+		return;
+	}
+
+	o->use_int = o->default_int;
+	o->use_uint = o->default_uint;
+	o->use_str = (char *)o->default_str;
+
+	return;
+}
+
+/*
+ * do the clean/restore job:
+ * - clean up dlm_options[].dynamic_xx
+ * - using high priority config to set use_xx items.
+ */
+static void reset_dynamic(struct dlm_option *o)
+{
+	o->dynamic_set = 0;
+	o->dynamic_int = 0;
+	if (o->dynamic_str){
+		free(o->dynamic_str);
+		o->dynamic_str = NULL;
+	}
+	o->dynamic_uint = 0;
+	reset_opt_value(o);
+
+	set_configfs_opt("log_debug", NULL, opt(log_debug_ind));
+	set_logfile_priority();
+
+	return;
+}
+
+void set_opt_online(int argc, char **argv)
+{
+	int i = -1, val = 0, ind;
+	unsigned int uval = 0;
+	struct dlm_option *o;
+	char str[MAX_LINE];
+
+	if (!strcmp(argv[0], "restore_all")) {
+		for (i = 0; i < dlm_options_max; i++)
+			reset_dynamic(&dlm_options[i]);
+
+		return;
+	}
+
+	while (++i < argc) {
+		ind = get_ind_name(argv[i]);
+		if (ind < 0)
+			continue;
+		o = &dlm_options[ind];
+		if (!o || !o->dynamic)
+			continue;
+
+		get_val_str(argv[i], str);
+		if (!strcmp(str, "restore")) {
+			reset_dynamic(o);
+			continue;
+		}
+
+		o->dynamic_set++;
+
+		if (!o->req_arg || o->req_arg == req_arg_int) {
+			get_val_int(argv[i], &val);
+			if (!o->req_arg)
+				val = val ? 1 : 0;
+
+			o->dynamic_int = val;
+
+			log_debug("config dynamic %s = %d previous use %d",
+				  o->name, o->dynamic_int, o->use_int);
+			o->use_int = o->dynamic_int;
+
+		} else if (o->req_arg == req_arg_uint) {
+			get_val_uint(argv[i], &uval);
+			o->dynamic_uint = uval;
+
+			log_debug("config dynamic %s = %u previous use %u",
+				  o->name, o->dynamic_uint, o->use_uint);
+			o->use_uint = o->dynamic_uint;
+
+		} else if (o->req_arg == req_arg_bool) {
+			get_val_int(argv[i], &val);
+			o->dynamic_int = val ? 1 : 0;
+
+			log_debug("config dynamic %s = %d previous use %d",
+				  o->name, o->dynamic_int, o->use_int);
+			o->use_int = o->dynamic_int;
+
+		} else if (o->req_arg == req_arg_str) {
+			memset(str, 0, sizeof(str));
+			get_val_str(argv[i], str);
+
+			o->dynamic_str = strdup(str);
+
+			log_debug("config dynamic %s = %s previous use %s",
+				  o->name, o->dynamic_str, o->use_str);
+			o->use_str = o->dynamic_str;
+		}
+
+		if (ind == log_debug_ind) {
+			set_configfs_opt("log_debug", NULL, opt(log_debug_ind));
+		} else if (ind == debug_logfile_ind) {
+			set_logfile_priority();
+		}
+
+	}
+
+	return;
+}
+
diff --git a/dlm_controld/dlm_daemon.h b/dlm_controld/dlm_daemon.h
index 904bb0b50df9..8cc9f666872e 100644
--- a/dlm_controld/dlm_daemon.h
+++ b/dlm_controld/dlm_daemon.h
@@ -398,6 +398,7 @@ int set_configfs_opt(const char *name, char *str, int num);
 void set_opt_file(int update);
 int get_weight(struct lockspace *ls, int nodeid);
 void setup_lockspace_config(struct lockspace *ls);
+void set_opt_online(int argc, char **argv);
 
 /* cpg.c */
 void process_lockspace_changes(void);
-- 
2.32.0

[Cluster-devel] [PATCH 10/10] dlm_controld: enable "dlm_tool run|run_start" dynamic setting feature

[Heming Zhao]

This commit gives dlm_tool the ability to change dlm_controld settings
on the fly.

Signed-off-by: Heming Zhao <heming.zhao@suse.com>
---
 dlm_controld/helper.c | 41 ++++++++++++++++++++++++++++++++---------
 1 file changed, 32 insertions(+), 9 deletions(-)

diff --git a/dlm_controld/helper.c b/dlm_controld/helper.c
index a20965b76195..dd03cb1c973f 100644
--- a/dlm_controld/helper.c
+++ b/dlm_controld/helper.c
@@ -48,6 +48,7 @@ do { \
 
 #define CMD_ID_LVCHANGE_REFRESH 1
 #define CMD_ID_LVS 2
+#define CMD_ID_OPT 3
 
 static int _get_cmd_id(char **av, int av_count)
 {
@@ -64,6 +65,11 @@ static int _get_cmd_id(char **av, int av_count)
 		return CMD_ID_LVS;
 	}
 
+	if ((av_count >= 2) &&
+	    (!strcmp(av[0], "option") || !strcmp(av[0], "opt"))) {
+		return CMD_ID_OPT;
+	}
+
 	return 0;
 }
 
@@ -133,19 +139,19 @@ static void _clear_running_cmd(struct running *running)
 
 /* runs in child process that was forked by helper */
 
-static void exec_command(char *cmd_str, int out_fd)
+static int exec_command(char *cmd_str, int out_fd)
 {
 	char cmd_buf[16];
 	char arg[ONE_ARG_LEN];
 	char *av[MAX_AV_COUNT + 1]; /* +1 for NULL */
 	int av_count = 0;
-	int i, rv, arg_len, cmd_len, cmd_id;
+	int i, rv = -1, arg_len, cmd_len, cmd_id;
 
 	for (i = 0; i < MAX_AV_COUNT + 1; i++)
 		av[i] = NULL;
 
 	if (!cmd_str[0])
-		return;
+		return rv;
 
 	/* this should already be done, but make sure */
 	cmd_str[RUN_COMMAND_LEN - 1] = '\0';
@@ -214,11 +220,26 @@ static void exec_command(char *cmd_str, int out_fd)
 		log_helper("write cmd_buf from child error %d", rv);
 	close(out_fd);
 
-	/* if we return before exec, the child does exit(1) (failure) */
-	if (!cmd_id)
-		return;
+	switch (cmd_id) {
+	case 0:
+		/* if we return from here, the child does exit(1) (failure) */
+		rv = -1;
+		break;
+	case 1:
+	case 2:
+		rv = 0;
+		execvp(av[0], av);
+		break; /* useless */
+	case 3:
+		set_opt_online(av_count-1, av+1);
+		rv = 0;
+		break;
+	default:
+		rv = -1;
+		break;
+	}
 
-	execvp(av[0], av);
+	return rv;
 }
 
 static int read_request(int fd, struct run_request *req)
@@ -352,8 +373,10 @@ int run_helper(int in_fd, int out_fd, int log_stderr)
 				pid = fork();
 				if (!pid) {
 					close(cmd_pipe[0]);
-					exec_command(req.command, cmd_pipe[1]);
-					exit(1);
+					if (exec_command(req.command, cmd_pipe[1]) < 0)
+						exit(1);
+					else
+						exit(0);
 				}
 
 				close(cmd_pipe[1]);
-- 
2.32.0

[Cluster-devel] [RFC PATCH dlm 00/10] dlm_controld config settings can be

[David Teigland]

On Sun, Sep 19, 2021 at 02:43:12PM +0800, Heming Zhao wrote:
> This new feature gives dlm ability to change config settings dynamically.

Hi Heming,

Letting dlm_controld reload certain settings from dlm.conf makes sense,
but I'd like something more basic.  Let the user edit dlm.conf, then
run dlm_tool reload_config to tell dlm_controld to reread dlm.conf, and
apply new settings.  This doesn't involve remote nodes, it's all local.
If the user wants, they can make changes on other nodes the same way.  

(We don't need to touch the "run" extension, which I'd like to remove
some day since it doesn't really belong in the dlm.)

Dave

[Cluster-devel] [RFC PATCH dlm 00/10] dlm_controld config settings can be

[heming.zhao]

On 9/21/21 1:57 AM, David Teigland wrote:
> On Sun, Sep 19, 2021 at 02:43:12PM +0800, Heming Zhao wrote:
>> This new feature gives dlm ability to change config settings dynamically.
> 
> Hi Heming,
> 
> Letting dlm_controld reload certain settings from dlm.conf makes sense,
> but I'd like something more basic.  Let the user edit dlm.conf, then
> run dlm_tool reload_config to tell dlm_controld to reread dlm.conf, and
> apply new settings.  This doesn't involve remote nodes, it's all local.
> If the user wants, they can make changes on other nodes the same way.
> 
> (We don't need to touch the "run" extension, which I'd like to remove
> some day since it doesn't really belong in the dlm.)
> 

Hello Dave,

Thank you for your comment.

One of the reasons for my patch based on run command is user could send
command to all nodes by only one-line command.
Dynamic settings (my patch) requires user the least time/work to change
dlm_controld behaviors. If not, it needs to separately edit dlm.conf &
run reload cmd on every local node.

But I am ok with the reload_config idea, it's more basic.
We could give dlm_controld a chance to change behavior on the fly.
If needed, I could file a new patch for feature "reload_config", can I do it?

Heming

[Cluster-devel] [RFC PATCH dlm 00/10] dlm_controld config settings can be

[David Teigland]

On Tue, Sep 21, 2021 at 02:38:45PM +0800, heming.zhao at suse.com wrote:
> But I am ok with the reload_config idea, it's more basic.
> We could give dlm_controld a chance to change behavior on the fly.
> If needed, I could file a new patch for feature "reload_config", can I do it?

Yes, I'd welcome a patch adding a dlm_tool reload_config option to reread
dlm.conf and apply new settings from it.
Thanks,
Dave

[Cluster-devel] [RFC PATCH dlm 00/10] dlm_controld config settings can be

[heming.zhao]

On 9/21/21 9:54 PM, David Teigland wrote:
> On Tue, Sep 21, 2021 at 02:38:45PM +0800, heming.zhao at suse.com wrote:
>> But I am ok with the reload_config idea, it's more basic.
>> We could give dlm_controld a chance to change behavior on the fly.
>> If needed, I could file a new patch for feature "reload_config", can I do it?
> 
> Yes, I'd welcome a patch adding a dlm_tool reload_config option to reread
> dlm.conf and apply new settings from it.
> Thanks,
> Dave
> 

Before sending the new feature patch, I want to have more discussion.

In my opinion, changing and reloading dlm.conf is a ***heavy*** way to update
dlm_controld settings. This manner is well suited for the change of a setting
still valid to next turn startup dlm_controld.
There are some use cases which require a ***light*** way to update settings.
It just needs updating settings in memory and validating in a short time
(or until one dlm_controld running turn).

Examples:

1. debug

debugging the code temporarily with no intention to change dlm.conf.

2. to get some log

temporarily enable dlm log for checking status.

3. temporarily change dlm_controld behaviour

(there is a production use case.)
a. In 3+ nodes HA cluster env, dlm_controld startup with default settings.
b. after stopping 2 of 3 nodes in cluster. (by cmd "crm cluster stop")
c. the only left node is lacking quorum and quorate become "NO". (by cmd "corosync-quorumtool")
d. on the last active node, run "crm cluster stop" to stop this node.
    the result is the node will be hung and will be fenced for a short while by trigging timeout.

The hanging in <d> can be avoid by temporarily changing settings:
enable_quorum_fencing and enable_quorum_lockspace from default value 1 to 0.
and the changing does not need lasting for node restart.

So,
If there is no chance to add dynamic updating setting by run command.
Is it a good idea to add a parameter "-I", like "dlm_tool -I reload_config".
"-I" means directly change without reading from dlm.conf.
When users want to restore the original setting, run "dlm_tool reload_config"
(without "-I") to override with predefined/default values.

Thanks
Heming

[Cluster-devel] [RFC PATCH dlm 00/10] dlm_controld config settings can be

[David Teigland]

On Wed, Sep 22, 2021 at 05:32:49PM +0800, heming.zhao at suse.com wrote:
> If there is no chance to add dynamic updating setting by run command.
> Is it a good idea to add a parameter "-I", like "dlm_tool -I reload_config".
> "-I" means directly change without reading from dlm.conf.
> When users want to restore the original setting, run "dlm_tool reload_config"
> (without "-I") to override with predefined/default values.

How about "dlm_tool set_config x=y" to make a temporary change in
dlm_controld, and "dlm_tool reload_config" to reread dlm.conf.
Dave

[Cluster-devel] [RFC PATCH dlm 00/10] dlm_controld config settings can be

[heming.zhao]

On 9/22/21 9:46 PM, David Teigland wrote:
> On Wed, Sep 22, 2021 at 05:32:49PM +0800, heming.zhao at suse.com wrote:
>> If there is no chance to add dynamic updating setting by run command.
>> Is it a good idea to add a parameter "-I", like "dlm_tool -I reload_config".
>> "-I" means directly change without reading from dlm.conf.
>> When users want to restore the original setting, run "dlm_tool reload_config"
>> (without "-I") to override with predefined/default values.
> 
> How about "dlm_tool set_config x=y" to make a temporary change in
> dlm_controld, and "dlm_tool reload_config" to reread dlm.conf.
> Dave
> 

Yes, "set_config x=y" is more better.
I just found I missed part of parameter of "-I reload_config". The whole
cmd should be "-I reload_config x=y". But add a new command is more clear
and better.

Heming