[PATCH] Improve mirror DSO's failure logging

[PATCH] Improve mirror DSO's failure logging

[Malahal Naineni]

The mirror target has the following device states. The mirror DSO
(daemons/dmeventd/plugins/mirror/dmeventd_mirror.c) doesn't know any of these
states. This patchs adds these states to the DSO for better error reporting.

	A => Alive - No failures
	D => Dead - A write failure occurred leaving mirror out-of-sync
	S => Sync - A sychronization failure occurred, mirror out-of-sync
	R => Read - A read failure occurred, mirror data unaffected

diff -r fff61ad560ad -r c107c082a3a5 daemons/dmeventd/plugins/mirror/dmeventd_mirror.c
--- a/daemons/dmeventd/plugins/mirror/dmeventd_mirror.c	Thu Oct 22 18:32:27 2009 -0700
+++ b/daemons/dmeventd/plugins/mirror/dmeventd_mirror.c	Fri Nov 20 15:33:30 2009 -0800
@@ -28,9 +28,13 @@
 #include <syslog.h> /* FIXME Replace syslog with multilog */
 /* FIXME Missing openlog? */
 
-#define ME_IGNORE    0
-#define ME_INSYNC    1
-#define ME_FAILURE   2
+#define ME_IGNORE    			0
+#define ME_INSYNC    			1
+#define ME_SYNC_FAILURE   		2
+#define ME_LOG_FAILURE 			3
+#define ME_READ_FAILURE   		4
+#define ME_PRIMARY_WRITE_FAILURE	5
+#define ME_SECONDARY_WRITE_FAILURE	6
 
 /*
  * register_device() is called first and performs initialisation.
@@ -53,7 +57,7 @@ static pthread_mutex_t _event_mutex = PT
 
 static int _get_mirror_event(char *params)
 {
-	int i, r = ME_INSYNC;
+	int i, r;
 	char **args = NULL;
 	char *dev_status_str;
 	char *log_status_str;
@@ -89,22 +93,40 @@ static int _get_mirror_event(char *param
 	sync_str = args[num_devs];
 
 	/* Check for bad mirror devices */
-	for (i = 0; i < num_devs; i++)
-		if (dev_status_str[i] == 'D') {
+	r = -1;
+	for (i = 0; i < num_devs && r == -1; i++) {
+		switch (dev_status_str[i]) {
+		case 'D':
 			syslog(LOG_ERR, "Mirror device, %s, has failed.\n", args[i]);
-			r = ME_FAILURE;
+			if (i == 0)
+				r = ME_PRIMARY_WRITE_FAILURE;
+			else 
+				r = ME_SECONDARY_WRITE_FAILURE;
+			break;
+		case 'S':
+			syslog(LOG_ERR, "Mirror synchronization failed. "
+					"device, %s, failed.\n", args[i]);
+			r = ME_SYNC_FAILURE;
+			break;
+		case 'R':
+			syslog(LOG_ERR, "Mirror device, %s, read failed.\n",
+					args[i]);
+			r = ME_READ_FAILURE;
+			break;
 		}
+	}
 
 	/* Check for bad disk log device */
 	if (log_argc > 1 && log_status_str[0] == 'D') {
 		syslog(LOG_ERR, "Log device, %s, has failed.\n",
 		       args[2 + num_devs + log_argc]);
-		r = ME_FAILURE;
+		r = ME_LOG_FAILURE;
 	}
 
-	if (r == ME_FAILURE)
+	if (r != -1)	/* if a failure occurred */
 		goto out;
 
+	r = ME_INSYNC;	/* assume INSYNC event */
 	p = strstr(sync_str, "/");
 	if (p) {
 		p[0] = '\0';
@@ -210,7 +232,9 @@ void process_event(struct dm_task *dmt,
 			*/
 			syslog(LOG_NOTICE, "%s is now in-sync\n", device);
 			break;
-		case ME_FAILURE:
+		case ME_LOG_FAILURE:
+		case ME_PRIMARY_WRITE_FAILURE:
+		case ME_SECONDARY_WRITE_FAILURE:
 			syslog(LOG_ERR, "Device failure in %s\n", device);
 			if (_remove_failed_devices(device))
 				/* FIXME Why are all the error return codes unused? Get rid of them? */
@@ -222,6 +246,8 @@ void process_event(struct dm_task *dmt,
 					device);
 			*/
 			break;
+		case ME_READ_FAILURE: /* we just ignore for now */
+		case ME_SYNC_FAILURE:
 		case ME_IGNORE:
 			break;
 		default:

[PATCH] Improve mirror DSO's failure logging

[Takahiro Yasui]

On 11/20/09 18:36, Malahal Naineni wrote:
> The mirror target has the following device states. The mirror DSO
> (daemons/dmeventd/plugins/mirror/dmeventd_mirror.c) doesn't know any of these
> states. This patchs adds these states to the DSO for better error reporting.
> 
> 	A => Alive - No failures
> 	D => Dead - A write failure occurred leaving mirror out-of-sync
> 	S => Sync - A sychronization failure occurred, mirror out-of-sync
> 	R => Read - A read failure occurred, mirror data unaffected

I think that the idea to improve an error reporting is very good.

>  static int _get_mirror_event(char *params)
>  {
> -	int i, r = ME_INSYNC;
> +	int i, r;
>  	char **args = NULL;
>  	char *dev_status_str;
>  	char *log_status_str;
> @@ -89,22 +93,40 @@ static int _get_mirror_event(char *param
>  	sync_str = args[num_devs];
>  
>  	/* Check for bad mirror devices */
> -	for (i = 0; i < num_devs; i++)
> -		if (dev_status_str[i] == 'D') {
> +	r = -1;
> +	for (i = 0; i < num_devs && r == -1; i++) {
> +		switch (dev_status_str[i]) {
> +		case 'D':
>  			syslog(LOG_ERR, "Mirror device, %s, has failed.\n", args[i]);
> -			r = ME_FAILURE;
> +			if (i == 0)
> +				r = ME_PRIMARY_WRITE_FAILURE;
> +			else 
> +				r = ME_SECONDARY_WRITE_FAILURE;
> +			break;
> +		case 'S':
> +			syslog(LOG_ERR, "Mirror synchronization failed. "
> +					"device, %s, failed.\n", args[i]);
> +			r = ME_SYNC_FAILURE;
> +			break;
> +		case 'R':
> +			syslog(LOG_ERR, "Mirror device, %s, read failed.\n",
> +					args[i]);
> +			r = ME_READ_FAILURE;
> +			break;
>  		}
> +	}

I'm afraid that only an error status of the device with the smallest index
of the dev_status_str[] array will be reported. For example, if the status
is "ARD," "r" will have a value, ME_READ_FAILURE. Then, a write failure
event (ME_PRIMARY_WRITE_FAILURE and ME_SECONDARY_WRITE_FAILURE) won't be
detected in process_event(). I think that "WRITE_FAILURE" events should
have priority.

Thanks,
Taka

[PATCH] Improve mirror DSO's failure logging

[Jonathan Brassow]

On Nov 23, 2009, at 4:00 PM, Takahiro Yasui wrote:

> I'm afraid that only an error status of the device with the smallest  
> index
> of the dev_status_str[] array will be reported. For example, if the  
> status
> is "ARD," "r" will have a value, ME_READ_FAILURE. Then, a write  
> failure
> event (ME_PRIMARY_WRITE_FAILURE and ME_SECONDARY_WRITE_FAILURE)  
> won't be
> detected in process_event(). I think that "WRITE_FAILURE" events  
> should
> have priority.
>
> Thanks,
> Taka

Good catch Taka, I agree.  Otherwise, I welcome this patch.

  brassow

[PATCH] Improve mirror DSO's failure logging

[malahal]

Takahiro Yasui [tyasui at redhat.com] wrote:
> I'm afraid that only an error status of the device with the smallest index
> of the dev_status_str[] array will be reported. For example, if the status
> is "ARD," "r" will have a value, ME_READ_FAILURE. Then, a write failure
> event (ME_PRIMARY_WRITE_FAILURE and ME_SECONDARY_WRITE_FAILURE) won't be
> detected in process_event(). I think that "WRITE_FAILURE" events should
> have priority.

Thank you, Takahiro. Just posted a patch fixing the above issue. I used
them as flags to fix it. Let me know what you think.

See
http://permalink.gmane.org/gmane.linux.kernel.device-mapper.devel/10675
for details.

Thanks, Malahal.

[PATCH] Improve mirror DSO's failure logging

[Takahiro Yasui]

On 11/30/09 05:32, malahal at us.ibm.com wrote:
> Takahiro Yasui [tyasui at redhat.com] wrote:
>> I'm afraid that only an error status of the device with the smallest index
>> of the dev_status_str[] array will be reported. For example, if the status
>> is "ARD," "r" will have a value, ME_READ_FAILURE. Then, a write failure
>> event (ME_PRIMARY_WRITE_FAILURE and ME_SECONDARY_WRITE_FAILURE) won't be
>> detected in process_event(). I think that "WRITE_FAILURE" events should
>> have priority.
> 
> Thank you, Takahiro. Just posted a patch fixing the above issue. I used
> them as flags to fix it. Let me know what you think.
> 
> See
> http://permalink.gmane.org/gmane.linux.kernel.device-mapper.devel/10675
> for details.

I'm afraid that you need to refresh this patch set on top of the recent
lvm2 source. The patch similar to your patch #1 has been merged recently.

Please see this link.
http://permalink.gmane.org/gmane.linux.lvm.devel/2312

Thanks,
Taka