[PATCH v2] md-cluster: fix safemode_delay value when converting to clustered bitmap

[PATCH v2] md-cluster: fix safemode_delay value when converting to clustered bitmap

[Zhao Heming]

When array convert to clustered bitmap, the safe_mode_delay doesn't clean and vice versa.
the /sys/block/mdX/md/safe_mode_delay keep original value after changing bitmap type.
in safe_delay_store(), the code forbids setting mddev->safemode_delay when array is clustered.
So in cluster-md env, the expected safemode_delay value should be 0.

reproduction steps:
```
node1 # mdadm --zero-superblock /dev/sd{b,c,d}
node1 # mdadm -C /dev/md0 -b internal -e 1.2 -n 2 -l mirror /dev/sdb /dev/sdc
node1 # cat /sys/block/md0/md/safe_mode_delay
0.204
node1 # mdadm -G /dev/md0 -b none
node1 # mdadm --grow /dev/md0 --bitmap=clustered
node1 # cat /sys/block/md0/md/safe_mode_delay
0.204  <== doesn't change, should ZERO for cluster-md

node1 # mdadm --zero-superblock /dev/sd{b,c,d}
node1 # mdadm -C /dev/md0 -b clustered -e 1.2 -n 2 -l mirror /dev/sdb /dev/sdc
node1 # cat /sys/block/md0/md/safe_mode_delay
0.000
node1 # mdadm -G /dev/md0 -b none
node1 # cat /sys/block/md0/md/safe_mode_delay
0.000  <== doesn't change, should default value
```

Neil said md_setup_cluster/md_cluster_stop are good places to fix.
After investigation, md_setup_cluster() is a good place for setting,
but md_cluster_stop are not pair for restoring.
see below flow:
(user space)
mdadm -C /dev/md0 -b clustered -e 1.2 -n 2 -l mirror /dev/sda /dev/sdb
mdadm --grow /dev/md0 -b none
(kernel space)
SET_ARRAY_INFO
 update_array_info
  + mddev->bitmap_info.nodes = 0;
  + md_cluster_ops->leave(mddev)
  + md_bitmap_destroy
     md_bitmap_free //won't trigger md_cluster_stop() because above set 0.

Signed-off-by: Zhao Heming <heming.zhao@suse.com>
---
v2:
- change setting path from location_store to md_setup_cluster
- add restoring path

---
 drivers/md/md.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/md/md.c b/drivers/md/md.c
index f567f53..f082f5c 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -101,6 +101,8 @@ static int remove_and_add_spares(struct mddev *mddev,
  * count by 2 for every hour elapsed between read errors.
  */
 #define MD_DEFAULT_MAX_CORRECTED_READ_ERRORS 20
+/* Default safemode delay: 200 msec */
+#define DEFAULT_SAFEMODE_DELAY ((200 * HZ)/1000 +1) 
 /*
  * Current RAID-1,4,5 parallel reconstruction 'guaranteed speed limit'
  * is 1000 KB/sec, so the extra system load does not show up that much.
@@ -5982,7 +5984,7 @@ int md_run(struct mddev *mddev)
 	if (mddev_is_clustered(mddev))
 		mddev->safemode_delay = 0;
 	else
-		mddev->safemode_delay = (200 * HZ)/1000 +1; /* 200 msec delay */
+		mddev->safemode_delay = DEFAULT_SAFEMODE_DELAY;
 	mddev->in_sync = 1;
 	smp_wmb();
 	spin_lock(&mddev->lock);
@@ -7361,6 +7363,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
 
 				mddev->bitmap_info.nodes = 0;
 				md_cluster_ops->leave(mddev);
+				mddev->safemode_delay = DEFAULT_SAFEMODE_DELAY;
 			}
 			mddev_suspend(mddev);
 			md_bitmap_destroy(mddev);
@@ -8366,6 +8369,7 @@ int md_setup_cluster(struct mddev *mddev, int nodes)
 	}
 	spin_unlock(&pers_lock);
 
+	mddev->safemode_delay = 0;
 	return md_cluster_ops->join(mddev, nodes);
 }
 
-- 
1.8.3.1

Re: [PATCH v2] md-cluster: fix safemode_delay value when converting to clustered bitmap

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3571 bytes --]

On Thu, Jul 16 2020, Zhao Heming wrote:

> When array convert to clustered bitmap, the safe_mode_delay doesn't clean and vice versa.
> the /sys/block/mdX/md/safe_mode_delay keep original value after changing bitmap type.
> in safe_delay_store(), the code forbids setting mddev->safemode_delay when array is clustered.
> So in cluster-md env, the expected safemode_delay value should be 0.
>
> reproduction steps:
> ```
> node1 # mdadm --zero-superblock /dev/sd{b,c,d}
> node1 # mdadm -C /dev/md0 -b internal -e 1.2 -n 2 -l mirror /dev/sdb /dev/sdc
> node1 # cat /sys/block/md0/md/safe_mode_delay
> 0.204
> node1 # mdadm -G /dev/md0 -b none
> node1 # mdadm --grow /dev/md0 --bitmap=clustered
> node1 # cat /sys/block/md0/md/safe_mode_delay
> 0.204  <== doesn't change, should ZERO for cluster-md
>
> node1 # mdadm --zero-superblock /dev/sd{b,c,d}
> node1 # mdadm -C /dev/md0 -b clustered -e 1.2 -n 2 -l mirror /dev/sdb /dev/sdc
> node1 # cat /sys/block/md0/md/safe_mode_delay
> 0.000
> node1 # mdadm -G /dev/md0 -b none
> node1 # cat /sys/block/md0/md/safe_mode_delay
> 0.000  <== doesn't change, should default value
> ```
>
> Neil said md_setup_cluster/md_cluster_stop are good places to fix.
> After investigation, md_setup_cluster() is a good place for setting,
> but md_cluster_stop are not pair for restoring.
> see below flow:
> (user space)
> mdadm -C /dev/md0 -b clustered -e 1.2 -n 2 -l mirror /dev/sda /dev/sdb
> mdadm --grow /dev/md0 -b none
> (kernel space)
> SET_ARRAY_INFO
>  update_array_info
>   + mddev->bitmap_info.nodes = 0;
>   + md_cluster_ops->leave(mddev)
>   + md_bitmap_destroy
>      md_bitmap_free //won't trigger md_cluster_stop() because above set 0.
>
> Signed-off-by: Zhao Heming <heming.zhao@suse.com>
> ---
> v2:
> - change setting path from location_store to md_setup_cluster
> - add restoring path
>
> ---
>  drivers/md/md.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index f567f53..f082f5c 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -101,6 +101,8 @@ static int remove_and_add_spares(struct mddev *mddev,
>   * count by 2 for every hour elapsed between read errors.
>   */
>  #define MD_DEFAULT_MAX_CORRECTED_READ_ERRORS 20
> +/* Default safemode delay: 200 msec */
> +#define DEFAULT_SAFEMODE_DELAY ((200 * HZ)/1000 +1) 
>  /*
>   * Current RAID-1,4,5 parallel reconstruction 'guaranteed speed limit'
>   * is 1000 KB/sec, so the extra system load does not show up that much.
> @@ -5982,7 +5984,7 @@ int md_run(struct mddev *mddev)
>  	if (mddev_is_clustered(mddev))
>  		mddev->safemode_delay = 0;
>  	else
> -		mddev->safemode_delay = (200 * HZ)/1000 +1; /* 200 msec delay */
> +		mddev->safemode_delay = DEFAULT_SAFEMODE_DELAY;
>  	mddev->in_sync = 1;
>  	smp_wmb();
>  	spin_lock(&mddev->lock);
> @@ -7361,6 +7363,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
>  
>  				mddev->bitmap_info.nodes = 0;
>  				md_cluster_ops->leave(mddev);
> +				mddev->safemode_delay = DEFAULT_SAFEMODE_DELAY;
>  			}
>  			mddev_suspend(mddev);
>  			md_bitmap_destroy(mddev);
> @@ -8366,6 +8369,7 @@ int md_setup_cluster(struct mddev *mddev, int nodes)
>  	}
>  	spin_unlock(&pers_lock);
>  
> +	mddev->safemode_delay = 0;
>  	return md_cluster_ops->join(mddev, nodes);

->join can fail.
I'd rather you checked the error there, and only clear safemode_delay if
the return value is zero.

NeilBrown


>  }
>  
> -- 
> 1.8.3.1

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [PATCH v2] md-cluster: fix safemode_delay value when converting to clustered bitmap

[heming.zhao]

On 7/17/20 7:36 AM, NeilBrown wrote:
> On Thu, Jul 16 2020, Zhao Heming wrote:
> 
>> When array convert to clustered bitmap, the safe_mode_delay doesn't clean and vice versa.
>> the /sys/block/mdX/md/safe_mode_delay keep original value after changing bitmap type.
... ...
>>   			}
>>   			mddev_suspend(mddev);
>>   			md_bitmap_destroy(mddev);
>> @@ -8366,6 +8369,7 @@ int md_setup_cluster(struct mddev *mddev, int nodes)
>>   	}
>>   	spin_unlock(&pers_lock);
>>   
>> +	mddev->safemode_delay = 0;
>>   	return md_cluster_ops->join(mddev, nodes);
> 
> ->join can fail.
> I'd rather you checked the error there, and only clear safemode_delay if
> the return value is zero.
> 
> NeilBrown
> 
accept, I was aware of this issue after I sent the patch.
the md_cluster_stop() looks a good place to do the clear job.
md_bitmap_read_sb will call md_cluster_stop when md_setup_cluster return error.
I will send patch v3 after I finish test.

btw,
with investigation safemode_delay bug, I found there is another bug with cluster-md:
md_cluster module can't rmmod in some condition.

below test using original SUSE leap15.2 kernel (5.3.18-xx)
```
node1 # mdadm -C /dev/md0 -b clustered -e 1.2 -n 2 -l mirror /dev/sda /dev/sdb
mdadm: array /dev/md0 started.
node1 # lsmod | egrep  "raid|md_"
md_cluster             28672  1
raid1                  53248  1
md_mod                176128  2 raid1,md_cluster
dlm                   212992  14 md_cluster
node1 # mdadm -S --scan
node1 # lsmod | egrep  "raid|md_"
md_cluster             28672  0
raid1                  53248  0
md_mod                176128  2 raid1,md_cluster
dlm                   212992  9 md_cluster       <=== looks cluster-md holds sth, but md_cluster can rmmod now.

node1 # mdadm --zero-superblock /dev/sd{a,b}
node1 # mdadm -C /dev/md0 -b clustered -e 1.2 -n 2 -l mirror /dev/sda /dev/sdb
node1 # mdadm -G /dev/md0 -b none
node1 # lsmod | egrep  "raid|md_"
md_cluster             28672  1
raid1                  53248  1
md_mod                176128  2 raid1,md_cluster
dlm                   212992  9 md_cluster
node1 # mdadm -S --scan
node1 # lsmod | egrep  "raid|md_"
md_cluster             28672  1   <=== something still live
raid1                  53248  0
md_mod                176128  2 raid1,md_cluster
dlm                   212992  9 md_cluster
```

I plan to fix this rmmod bug after I fixing mdadm show active issue.
> 
>>   }
>>   
>> -- 
>> 1.8.3.1