[PATCH 0/6] OLCE raid5/0 (all arrays in container)

[PATCH 0/6] OLCE raid5/0 (all arrays in container)

[Adam Kwolek]

The following series adds possibility to reshape all (raid0/5) arrays in container for expansion feature.
The main problem resolved in this series is how initialize second (and next) raid0 reshapes for container reshape
executed as set of array reshapes.

For i.e. second raid0 array mdmon should initialize metadata to let know to mdadm that there is something more to reshape.
But problem is what to do when mdmon is not loaded? (for first raid0 array this is not a problem, reshape_super() in mdadm updates metadata).

this is resolved by allowing mdadm to know what was reshaped so far. This allows to reshape arrays that are not reshaped yet,
without update from mdmon.

BR
Adam 

---

Adam Kwolek (6):
      FIX: sync_completed == 0 causes reshape cancellation in metadata
      FIX: mdadm hangs during reshape
      FIX: mdadm throws coredump on exit in error case
      FIX: reshape raid0 on second array
      FIX: continue raid0 reshape
      imsm: Update metadata for second array


 Grow.c        |   95 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 monitor.c     |    4 ++
 super-intel.c |   35 +++++++++++++++++++--
 3 files changed, 120 insertions(+), 14 deletions(-)

-- 
Signature

[PATCH 1/6] imsm: Update metadata for second array

[Adam Kwolek]

When second array reshape is about to start external metadata should be updated
by mdmon in imsm_set_array_state().
for this purposes imsm_progress_container_reshape() is reused.

Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
---

 super-intel.c |   35 ++++++++++++++++++++++++++++++++---
 1 files changed, 32 insertions(+), 3 deletions(-)

diff --git a/super-intel.c b/super-intel.c
index cb2b51b..4cbb070 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -4863,7 +4863,9 @@ static void handle_missing(struct intel_super *super, struct imsm_dev *dev)
 
 static void imsm_set_disk(struct active_array *a, int n, int state);
 
-static void imsm_progress_container_reshape(struct intel_super *super)
+static void imsm_progress_container_reshape(struct intel_super *super,
+					    struct imsm_dev *dev_in,
+					    int disk_count)
 {
 	/* if no device has a migr_state, but some device has a
 	 * different number of members than the previous device, start
@@ -4871,7 +4873,7 @@ static void imsm_progress_container_reshape(struct intel_super *super)
 	 * previous.
 	 */
 	struct imsm_super *mpb = super->anchor;
-	int prev_disks = -1;
+	int prev_disks = disk_count;
 	int i;
 
 	for (i = 0; i < mpb->num_raid_devs; i++) {
@@ -4884,6 +4886,10 @@ static void imsm_progress_container_reshape(struct intel_super *super)
 		if (dev->vol.migr_state)
 			return;
 
+		if (((dev_in != NULL) && disk_count > 0) &&
+		     (dev_in != dev))
+			continue;
+
 		if (prev_disks == -1)
 			prev_disks = map->num_members;
 		if (prev_disks == map->num_members)
@@ -4999,7 +5005,9 @@ static int imsm_set_array_state(struct active_array *a, int consistent)
 						       * array size
 						       */
 				super->updates_pending++;
-				imsm_progress_container_reshape(super);
+				imsm_progress_container_reshape(super,
+								NULL,
+								-1);
 			}				
 		}
 	}
@@ -5072,6 +5080,27 @@ static int imsm_set_array_state(struct active_array *a, int consistent)
 
 		for (mdi = a->info.devs; mdi; mdi = mdi->next)
 			imsm_set_disk(a, mdi->disk.raid_disk, mdi->curr_state);
+	} else if (dev->vol.migr_state == 0) {
+		/* check here if this is container action
+		 * and next metadata should be prepared
+		*/
+		struct imsm_map *map = get_imsm_map(dev, 0);
+		struct dl *dl = NULL;
+		int disks_count = 0;
+
+		/* check if this array should be reshaped
+		 */
+		for (dl = super->disks; dl; dl = dl->next)
+			if (dl->index >= 0)
+				disks_count++;
+		if (disks_count > map->num_members) {
+			/* manage changes in volume
+			 */
+			imsm_progress_container_reshape(super,
+							dev,
+							disks_count);
+			super->updates_pending++;
+		}
 	}
 
 	return consistent;

[PATCH 2/6] FIX: continue raid0 reshape

[Adam Kwolek]

In case when second array is raid0 array, mdmon will not start reshape
for it because this array is not monitored.
To resolve this situation, all reshaped devices are collected in to list.
If nothing is found for reshape (no arrays with reshape_active flag),
already reshaped list is searched and checked if everything
is reshaped. If we find that raid0 array is not reshaped we are continuing
reshape for this device.

Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
---

 Grow.c |   30 ++++++++++++++++++++++++++++++
 1 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/Grow.c b/Grow.c
index a765308..fadaf2d 100644
--- a/Grow.c
+++ b/Grow.c
@@ -2056,6 +2056,7 @@ int reshape_container(char *container, int cfd, char *devname,
 		      int quiet)
 {
 	struct mdinfo *cc = NULL;
+	struct mdstat_ent *mdstat_collection = NULL;
 
 	/* component_size is not meaningful for a container,
 	 * so pass '-1' meaning 'no change'
@@ -2113,6 +2114,31 @@ int reshape_container(char *container, int cfd, char *devname,
 				continue;
 			break;
 		}
+		/* check if reshape decision should be taken here
+		 */
+		if (!content) {
+			for (content = cc; content ; content = content->next) {
+				struct mdstat_ent *mdstat_check =
+							      mdstat_collection;
+				char *subarray;
+
+				subarray = strchr(content->text_version + 1,
+						  '/') + 1;
+				mdstat = mdstat_by_subdev(subarray,
+						  devname2devnum(container));
+				while (mdstat_check &&
+				       mdstat &&
+					(mdstat->devnum !=
+					 mdstat_check->devnum)) {
+					mdstat_check = mdstat_check->next;
+				}
+				if (mdstat_check) {
+					free_mdstat(mdstat);
+					continue;
+				}
+				break;
+			}
+		}
 		if (!content)
 			break;
 
@@ -2131,11 +2157,15 @@ int reshape_container(char *container, int cfd, char *devname,
 				   content, force,
 				   backup_file, quiet, 1);
 		close(fd);
+		mdstat->next = mdstat_collection;
+		mdstat_collection = mdstat;
 		if (rv)
 			break;
 	}
 	unfreeze(st);
+	free_mdstat(mdstat_collection);
 	sysfs_free(cc);
+
 	exit(0);
 }
 

[PATCH 3/6] FIX: reshape raid0 on second array

[Adam Kwolek]

When reshape_array() is called for raid0 array (second and next),
we know that mdmon will update metadata. To do this after takeover
and mdmon start(optional) metadata is reread to be sure that
we should reshape this array and what reshape conditions we should check.

If info was read internally we should release it before exit.

Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
---

 Grow.c |   57 ++++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 files changed, 50 insertions(+), 7 deletions(-)

diff --git a/Grow.c b/Grow.c
index fadaf2d..7cc6bae 100644
--- a/Grow.c
+++ b/Grow.c
@@ -1240,7 +1240,7 @@ char *analyse_change(struct mdinfo *info, struct reshape *re)
 	return NULL;
 }
 
-static int reshape_array(char *container, int fd, char *devname,
+static int reshape_array(char *container, int cfd, int fd, char *devname,
 			 struct supertype *st, struct mdinfo *info,
 			 int force, char *backup_file, int quiet, int forked);
 static int reshape_container(char *container, int cfd, char *devname,
@@ -1560,8 +1560,8 @@ int Grow_reshape(char *devname, int fd, int quiet, char *backup_file,
 			goto release;
 		}
 		sync_metadata(st);
-		rv = reshape_array(container, fd, devname, st, &info, force,
-				   backup_file, quiet, 0);
+		rv = reshape_array(container, cfd, fd, devname,
+				   st, &info, force, backup_file, quiet, 0);
 		frozen = 0;
 	}
 release:
@@ -1570,7 +1570,7 @@ release:
 	return rv;
 }
 
-static int reshape_array(char *container, int fd, char *devname,
+static int reshape_array(char *container, int cfd, int fd, char *devname,
 			 struct supertype *st, struct mdinfo *info,
 			 int force,
 			 char *backup_file, int quiet, int forked)
@@ -1595,6 +1595,8 @@ static int reshape_array(char *container, int fd, char *devname,
 	unsigned long long array_size;
 	int done;
 	struct mdinfo *sra;
+	int info_reloaded = 0;
+
 
 	msg = analyse_change(info, &reshape);
 	if (msg) {
@@ -1648,7 +1650,43 @@ static int reshape_array(char *container, int fd, char *devname,
 	if (reshape.level > 0 && st->ss->external &&
 	    !mdmon_running(st->container_dev)) {
 		start_mdmon(st->container_dev);
-		ping_monitor(container);
+	}
+	ping_monitor(container);
+
+	/* reload metadat as it is possible to change made by monitor
+	 */
+	if ((cfd >= 0) &&
+	   (info->array.level == 0)) {
+		char *subarray = strchr(info->text_version+1, '/')+1;
+
+		st->ss->load_container(st, cfd, NULL);
+		info = st->ss->container_content(st, subarray);
+		sysfs_init(info, fd, st->devnum);
+		info_reloaded = 1;
+		msg = analyse_change(info, &reshape);
+		if (msg) {
+			fprintf(stderr, Name ": %s\n", msg);
+			return 1;
+		}
+	}
+
+	if (ioctl(fd, GET_ARRAY_INFO, &array) != 0) {
+		dprintf("Canot get array information.\n");
+		return 1;
+	}
+	spares_needed = max(reshape.before.data_disks,
+			    reshape.after.data_disks)
+		+ reshape.parity - array.raid_disks;
+
+	if (!force && spares_needed < info->array.spare_disks) {
+		fprintf(stderr,
+			Name ": Need %d spare%s to avoid degraded array,"
+				" and only have %d.\n"
+				"       Use --force to over-ride this check.\n",
+			spares_needed,
+			spares_needed == 1 ? "" : "s",
+			info->array.spare_disks);
+		return 1;
 	}
 
 	/* ->reshape_super might have chosen some spares from the
@@ -2033,6 +2071,8 @@ static int reshape_array(char *container, int fd, char *devname,
 		}
 	}
 out:
+	if (info_reloaded)
+		sysfs_free(info);
 	if (forked)
 		return 0;
 	exit(0);
@@ -2045,6 +2085,8 @@ release:
 	}
 	if (!forked)
 		unfreeze(st);
+	if (info_reloaded)
+		sysfs_free(info);
 	return rv;
 }
 
@@ -2153,7 +2195,7 @@ int reshape_container(char *container, int cfd, char *devname,
 
 		sysfs_init(content, fd, mdstat->devnum);
 
-		rv = reshape_array(container, fd, adev, st,
+		rv = reshape_array(container, cfd, fd, adev, st,
 				   content, force,
 				   backup_file, quiet, 1);
 		close(fd);
@@ -3178,7 +3220,8 @@ int Grow_continue(int mdfd, struct supertype *st, struct mdinfo *info,
 	int err = sysfs_set_str(info, NULL, "array_state", "readonly");
 	if (err)
 		return err;
-	return reshape_array(NULL, mdfd, "array", st, info, 1, backup_file, 0, 0);
+	return reshape_array(NULL, -1, mdfd, "array", st,
+			     info, 1, backup_file, 0, 0);
 }
 
 

[PATCH 4/6] FIX: mdadm throws coredump on exit in error case

[Adam Kwolek]

When mdadm falls in "reduce size" error on takeovered array it jumps
to release and tries execute backward takeover. This time sra pointer is not initialized
and coredump is generated.

Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
---

 Grow.c |    3 +--
 1 files changed, 1 insertions(+), 2 deletions(-)

diff --git a/Grow.c b/Grow.c
index 7cc6bae..f1a6218 100644
--- a/Grow.c
+++ b/Grow.c
@@ -1594,7 +1594,7 @@ static int reshape_array(char *container, int cfd, int fd, char *devname,
 	unsigned long cache;
 	unsigned long long array_size;
 	int done;
-	struct mdinfo *sra;
+	struct mdinfo *sra = NULL;
 	int info_reloaded = 0;
 
 
@@ -1802,7 +1802,6 @@ static int reshape_array(char *container, int cfd, int fd, char *devname,
 	sra = sysfs_read(fd, 0,
 			 GET_COMPONENT|GET_DEVS|GET_OFFSET|GET_STATE|GET_CHUNK|
 			 GET_CACHE);
-
 	if (!sra) {
 		fprintf(stderr, Name ": %s: Cannot get array details from sysfs\n",
 			devname);

[PATCH 5/6] FIX: mdadm hangs during reshape

[Adam Kwolek]

During reshape when reshape is finished in md, progress_reshape() hangs
on select(). Timeout is introduced to allow for reshape conditions check.

Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
---

 Grow.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/Grow.c b/Grow.c
index f1a6218..763287b 100644
--- a/Grow.c
+++ b/Grow.c
@@ -2459,6 +2459,7 @@ int progress_reshape(struct mdinfo *info, struct reshape *reshape,
 		 * waiting forever on a dead array
 		 */
 		char action[20];
+		struct timeval timeout;
 		fd_set rfds;
 		if (sysfs_get_str(info, NULL, "sync_action",
 				  action, 20) <= 0 ||
@@ -2466,7 +2467,9 @@ int progress_reshape(struct mdinfo *info, struct reshape *reshape,
 			break;
 		FD_ZERO(&rfds);
 		FD_SET(fd, &rfds);
-		select(fd+1, NULL, NULL, &rfds, NULL);
+		timeout.tv_sec = 1;
+		timeout.tv_usec = 0;
+		select(fd+1, NULL, NULL, &rfds, &timeout);
 		if (sysfs_fd_get_ll(fd, &completed) < 0) {
 			close(fd);
 			return -1;

[PATCH 6/6] FIX: sync_completed == 0 causes reshape cancellation in metadata

[Adam Kwolek]

md signals reshape completion (whole area or parts) by setting sync_completed to 0.
This causes in set_array_state() to rollback metadata changes (super-intel.c:4977.
To avoid this do not allow for set last_checkpoint to 0 if reshape is finished.

This was also root cause of my previous fix for finalization reshape that I agreed earlier is not necessary,

Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
---

 monitor.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/monitor.c b/monitor.c
index 1107d47..b7287d8 100644
--- a/monitor.c
+++ b/monitor.c
@@ -363,8 +363,10 @@ static int read_and_act(struct active_array *a)
 		/* Reshape has progressed or completed so we need to
 		 * update the array state - and possibly the array size
 		 */
-		a->last_checkpoint = sync_completed;
+		if (sync_completed != 0)
+			a->last_checkpoint = sync_completed;
 		a->container->ss->set_array_state(a, a->curr_state <= clean);
+		a->last_checkpoint = sync_completed;
 	}
 
 	if (sync_completed > a->last_checkpoint)

Re: [PATCH 0/6] OLCE raid5/0 (all arrays in container)

[NeilBrown]

On Thu, 13 Jan 2011 15:50:02 +0100 Adam Kwolek <adam.kwolek@intel.com> wrote:

> The following series adds possibility to reshape all (raid0/5) arrays in container for expansion feature.
> The main problem resolved in this series is how initialize second (and next) raid0 reshapes for container reshape
> executed as set of array reshapes.
> 
> For i.e. second raid0 array mdmon should initialize metadata to let know to mdadm that there is something more to reshape.
> But problem is what to do when mdmon is not loaded? (for first raid0 array this is not a problem, reshape_super() in mdadm updates metadata).
> 
> this is resolved by allowing mdadm to know what was reshaped so far. This allows to reshape arrays that are not reshaped yet,
> without update from mdmon.

There certainly are some complexities with reshaping a container with 2 RAID0
arrays in it.  However the approach you are taking seems too complicated.

This is how I think the particular case of 2 RAID0 array would work.  It
should generalise neatly to all other combinations.

1/ mdadm decides to proceeds with the reshape, updates the metadata (because
   mdmon isn't running), converts the array to RAID4, starts mdmon and then
   lets the reshape for the first array proceed.

2/ when the reshape completes, mdmon will notice and will call
   imsm_progress_container_reshape which will update the metadata
   for the second array so that reshape appears to be active.

3/ mdadm will convert the raid4 back to raid0 and so mdmon will exit.

4/ mdadm calls container_content and finds that the second array needs to
   be reshaped.  It converts the array to raid4 and starts mdmon.
   Then they oversee the reshape of the second array

5/ when the reshape completes, mdadm converts the array back to RAID0 and
   mdmon exits.

All done.


This should avoid almost all special-casing.

NeilBrown


> 
> BR
> Adam 
> 
> ---
> 
> Adam Kwolek (6):
>       FIX: sync_completed == 0 causes reshape cancellation in metadata
>       FIX: mdadm hangs during reshape
>       FIX: mdadm throws coredump on exit in error case
>       FIX: reshape raid0 on second array
>       FIX: continue raid0 reshape
>       imsm: Update metadata for second array
> 
> 
>  Grow.c        |   95 +++++++++++++++++++++++++++++++++++++++++++++++++++------
>  monitor.c     |    4 ++
>  super-intel.c |   35 +++++++++++++++++++--
>  3 files changed, 120 insertions(+), 14 deletions(-)
> 

Re: [PATCH 4/6] FIX: mdadm throws coredump on exit in error case

[NeilBrown]

On Thu, 13 Jan 2011 15:50:34 +0100 Adam Kwolek <adam.kwolek@intel.com> wrote:

> When mdadm falls in "reduce size" error on takeovered array it jumps
> to release and tries execute backward takeover. This time sra pointer is not initialized
> and coredump is generated.
> 
> Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
> ---
> 
>  Grow.c |    3 +--
>  1 files changed, 1 insertions(+), 2 deletions(-)
> 
> diff --git a/Grow.c b/Grow.c
> index 7cc6bae..f1a6218 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -1594,7 +1594,7 @@ static int reshape_array(char *container, int cfd, int fd, char *devname,
>  	unsigned long cache;
>  	unsigned long long array_size;
>  	int done;
> -	struct mdinfo *sra;
> +	struct mdinfo *sra = NULL;
>  	int info_reloaded = 0;
>  
>  
> @@ -1802,7 +1802,6 @@ static int reshape_array(char *container, int cfd, int fd, char *devname,
>  	sra = sysfs_read(fd, 0,
>  			 GET_COMPONENT|GET_DEVS|GET_OFFSET|GET_STATE|GET_CHUNK|
>  			 GET_CACHE);
> -
>  	if (!sra) {
>  		fprintf(stderr, Name ": %s: Cannot get array details from sysfs\n",
>  			devname);
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Applied, thanks.

NeilBrown

Re: [PATCH 5/6] FIX: mdadm hangs during reshape

[NeilBrown]

On Thu, 13 Jan 2011 15:50:42 +0100 Adam Kwolek <adam.kwolek@intel.com> wrote:

> During reshape when reshape is finished in md, progress_reshape() hangs
> on select(). Timeout is introduced to allow for reshape conditions check.
> 
> Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
> ---
> 
>  Grow.c |    5 ++++-
>  1 files changed, 4 insertions(+), 1 deletions(-)
> 
> diff --git a/Grow.c b/Grow.c
> index f1a6218..763287b 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -2459,6 +2459,7 @@ int progress_reshape(struct mdinfo *info, struct reshape *reshape,
>  		 * waiting forever on a dead array
>  		 */
>  		char action[20];
> +		struct timeval timeout;
>  		fd_set rfds;
>  		if (sysfs_get_str(info, NULL, "sync_action",
>  				  action, 20) <= 0 ||
> @@ -2466,7 +2467,9 @@ int progress_reshape(struct mdinfo *info, struct reshape *reshape,
>  			break;
>  		FD_ZERO(&rfds);
>  		FD_SET(fd, &rfds);
> -		select(fd+1, NULL, NULL, &rfds, NULL);
> +		timeout.tv_sec = 1;
> +		timeout.tv_usec = 0;
> +		select(fd+1, NULL, NULL, &rfds, &timeout);
>  		if (sysfs_fd_get_ll(fd, &completed) < 0) {
>  			close(fd);
>  			return -1;
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Putting a timeout in here is definitely the wrong thing to do.

If you can reproduce the hang, please report the values of 
  completed, max_progress, wait_point

at that time, and the details of the array (level, devices, size, layout,
chunk size, etc).

NeilBrown

Re: [PATCH 6/6] FIX: sync_completed == 0 causes reshape cancellation in metadata

[NeilBrown]

On Thu, 13 Jan 2011 15:50:50 +0100 Adam Kwolek <adam.kwolek@intel.com> wrote:

> md signals reshape completion (whole area or parts) by setting sync_completed to 0.
> This causes in set_array_state() to rollback metadata changes (super-intel.c:4977.
> To avoid this do not allow for set last_checkpoint to 0 if reshape is finished.
> 
> This was also root cause of my previous fix for finalization reshape that I agreed earlier is not necessary,
> 
> Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
> ---
> 
>  monitor.c |    4 +++-
>  1 files changed, 3 insertions(+), 1 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index 1107d47..b7287d8 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -363,8 +363,10 @@ static int read_and_act(struct active_array *a)
>  		/* Reshape has progressed or completed so we need to
>  		 * update the array state - and possibly the array size
>  		 */
> -		a->last_checkpoint = sync_completed;
> +		if (sync_completed != 0)
> +			a->last_checkpoint = sync_completed;
>  		a->container->ss->set_array_state(a, a->curr_state <= clean);
> +		a->last_checkpoint = sync_completed;
>  	}
>  
>  	if (sync_completed > a->last_checkpoint)
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Applied, thanks.!!!

NeilBrown

RE: [PATCH 5/6] FIX: mdadm hangs during reshape

[Kwolek, Adam]

> -----Original Message-----
> From: NeilBrown [mailto:neilb@suse.de]
> Sent: Monday, January 17, 2011 2:37 AM
> To: Kwolek, Adam
> Cc: linux-raid@vger.kernel.org; Williams, Dan J; Ciechanowski, Ed;
> Neubauer, Wojciech
> Subject: Re: [PATCH 5/6] FIX: mdadm hangs during reshape
> 
> On Thu, 13 Jan 2011 15:50:42 +0100 Adam Kwolek <adam.kwolek@intel.com>
> wrote:
> 
> > During reshape when reshape is finished in md, progress_reshape()
> hangs
> > on select(). Timeout is introduced to allow for reshape conditions
> check.
> >
> > Signed-off-by: Adam Kwolek <adam.kwolek@intel.com>
> > ---
> >
> >  Grow.c |    5 ++++-
> >  1 files changed, 4 insertions(+), 1 deletions(-)
> >
> > diff --git a/Grow.c b/Grow.c
> > index f1a6218..763287b 100644
> > --- a/Grow.c
> > +++ b/Grow.c
> > @@ -2459,6 +2459,7 @@ int progress_reshape(struct mdinfo *info,
> struct reshape *reshape,
> >  		 * waiting forever on a dead array
> >  		 */
> >  		char action[20];
> > +		struct timeval timeout;
> >  		fd_set rfds;
> >  		if (sysfs_get_str(info, NULL, "sync_action",
> >  				  action, 20) <= 0 ||
> > @@ -2466,7 +2467,9 @@ int progress_reshape(struct mdinfo *info,
> struct reshape *reshape,
> >  			break;
> >  		FD_ZERO(&rfds);
> >  		FD_SET(fd, &rfds);
> > -		select(fd+1, NULL, NULL, &rfds, NULL);
> > +		timeout.tv_sec = 1;
> > +		timeout.tv_usec = 0;
> > +		select(fd+1, NULL, NULL, &rfds, &timeout);
> >  		if (sysfs_fd_get_ll(fd, &completed) < 0) {
> >  			close(fd);
> >  			return -1;
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Putting a timeout in here is definitely the wrong thing to do.
> 
> If you can reproduce the hang, please report the values of
>   completed, max_progress, wait_point
>



array: 
  Level: 0
  Chunk: 64
  Size: 100485
  Disks in array: 3
  Spare disks: 1
  Layout: n/a

Configuration:
  mdadm -C /dev/md/container -amd -e -n3 /dev/sda /dev/sdb /dev/sdc
  mdadm -C /dev/md/raid0_vol -amd -l 0 --chunk 64 --size 100485 -n3  /dev/sda /dev/sdb /dev/sdc -R
  mdadm --add /dev/md/container /dev/sdd

OLCE command:
 mdadm --grow /dev/md/container --raid-devices 4 --backup-file=/backup_file.bak


mdadm stops at:
   completed = 0 (possiblet before select() check for this should be added)
   max_progress == wait_point = 200960


I hope this helps

BR
Adam



> at that time, and the details of the array (level, devices, size,
> layout,
> chunk size, etc).
> 
> NeilBrown

RE: [PATCH 0/6] OLCE raid5/0 (all arrays in container)

[Kwolek, Adam]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of NeilBrown
> Sent: Monday, January 17, 2011 2:31 AM
> To: Kwolek, Adam
> Cc: linux-raid@vger.kernel.org; Williams, Dan J; Ciechanowski, Ed;
> Neubauer, Wojciech
> Subject: Re: [PATCH 0/6] OLCE raid5/0 (all arrays in container)
> 
> On Thu, 13 Jan 2011 15:50:02 +0100 Adam Kwolek <adam.kwolek@intel.com>
> wrote:
> 
> > The following series adds possibility to reshape all (raid0/5) arrays
> in container for expansion feature.
> > The main problem resolved in this series is how initialize second
> (and next) raid0 reshapes for container reshape
> > executed as set of array reshapes.
> >
> > For i.e. second raid0 array mdmon should initialize metadata to let
> know to mdadm that there is something more to reshape.
> > But problem is what to do when mdmon is not loaded? (for first raid0
> array this is not a problem, reshape_super() in mdadm updates
> metadata).
> >
> > this is resolved by allowing mdadm to know what was reshaped so far.
> This allows to reshape arrays that are not reshaped yet,
> > without update from mdmon.
> 
> There certainly are some complexities with reshaping a container with 2
> RAID0
> arrays in it.  However the approach you are taking seems too
> complicated.
> 
> This is how I think the particular case of 2 RAID0 array would work.
> It
> should generalise neatly to all other combinations.
> 
> 1/ mdadm decides to proceeds with the reshape, updates the metadata
> (because
>    mdmon isn't running), converts the array to RAID4, starts mdmon and
> then
>    lets the reshape for the first array proceed.
> 
> 2/ when the reshape completes, mdmon will notice and will call
>    imsm_progress_container_reshape which will update the metadata
>    for the second array so that reshape appears to be active.

For i.e. second raid5 it is ok.
This will never happened when second array is raid0, it is not monitored yet as arrays metadata are updated/monitored one by one.
This means that that mdmon cannot tell (via metadata) to mdadm that next raid0 array needs to be reshaped.
This information is used in container_content() before second takeover (in reshape_array()) is executed.

> 
> 3/ mdadm will convert the raid4 back to raid0 and so mdmon will exit.
> 
> 4/ mdadm calls container_content and finds that the second array needs
> to
>    be reshaped.  It converts the array to raid4 and starts mdmon.
>    Then they oversee the reshape of the second array

Metadata for second array is not updates by mdmon as we want "now" start mdmon for it.

> 
> 5/ when the reshape completes, mdadm converts the array back to RAID0
> and
>    mdmon exits.
> 
> All done.


All looks ok, except "next/2nd" raid0 array is reshaped. At "previous/1st" reshape finish, it is not monitored yet, so mdmon can't update metadata,
before mdadm enters reshape_array() to execute takeover and then mdmon is started.
Decision to enter reshape_array() is made based on metadata (not updated).

This is a reason why I'm using additional list for reshape decision. If mdmon doesn't make update for raid0 (not monitored)
Mdadm tries to run mdmon for it (in reshape_array()) and waits for metadata update and decision about reshape.

Such implementation allows to follow directions for single reshape_super() call for container operation.

BR
Adam


> 
> 
> This should avoid almost all special-casing.
> 
> NeilBrown
> 
> 
> >
> > BR
> > Adam
> >
> > ---
> >
> > Adam Kwolek (6):
> >       FIX: sync_completed == 0 causes reshape cancellation in
> metadata
> >       FIX: mdadm hangs during reshape
> >       FIX: mdadm throws coredump on exit in error case
> >       FIX: reshape raid0 on second array
> >       FIX: continue raid0 reshape
> >       imsm: Update metadata for second array
> >
> >
> >  Grow.c        |   95
> +++++++++++++++++++++++++++++++++++++++++++++++++++------
> >  monitor.c     |    4 ++
> >  super-intel.c |   35 +++++++++++++++++++--
> >  3 files changed, 120 insertions(+), 14 deletions(-)
> >
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/6] OLCE raid5/0 (all arrays in container)

[NeilBrown]

On Mon, 17 Jan 2011 11:52:09 +0000 "Kwolek, Adam" <adam.kwolek@intel.com>
wrote:

> 
> 
> > -----Original Message-----
> > From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> > owner@vger.kernel.org] On Behalf Of NeilBrown
> > Sent: Monday, January 17, 2011 2:31 AM
> > To: Kwolek, Adam
> > Cc: linux-raid@vger.kernel.org; Williams, Dan J; Ciechanowski, Ed;
> > Neubauer, Wojciech
> > Subject: Re: [PATCH 0/6] OLCE raid5/0 (all arrays in container)
> > 
> > On Thu, 13 Jan 2011 15:50:02 +0100 Adam Kwolek <adam.kwolek@intel.com>
> > wrote:
> > 
> > > The following series adds possibility to reshape all (raid0/5) arrays
> > in container for expansion feature.
> > > The main problem resolved in this series is how initialize second
> > (and next) raid0 reshapes for container reshape
> > > executed as set of array reshapes.
> > >
> > > For i.e. second raid0 array mdmon should initialize metadata to let
> > know to mdadm that there is something more to reshape.
> > > But problem is what to do when mdmon is not loaded? (for first raid0
> > array this is not a problem, reshape_super() in mdadm updates
> > metadata).
> > >
> > > this is resolved by allowing mdadm to know what was reshaped so far.
> > This allows to reshape arrays that are not reshaped yet,
> > > without update from mdmon.
> > 
> > There certainly are some complexities with reshaping a container with 2
> > RAID0
> > arrays in it.  However the approach you are taking seems too
> > complicated.
> > 
> > This is how I think the particular case of 2 RAID0 array would work.
> > It
> > should generalise neatly to all other combinations.
> > 
> > 1/ mdadm decides to proceeds with the reshape, updates the metadata
> > (because
> >    mdmon isn't running), converts the array to RAID4, starts mdmon and
> > then
> >    lets the reshape for the first array proceed.
> > 
> > 2/ when the reshape completes, mdmon will notice and will call
> >    imsm_progress_container_reshape which will update the metadata
> >    for the second array so that reshape appears to be active.
> 
> For i.e. second raid5 it is ok.
> This will never happened when second array is raid0, it is not monitored yet as arrays metadata are updated/monitored one by one.
> This means that that mdmon cannot tell (via metadata) to mdadm that next raid0 array needs to be reshaped.
> This information is used in container_content() before second takeover (in reshape_array()) is executed.

When the reshape of the first array completes, imsm_set_array_state will call
imsm_progress_container_reshape which will cause the second array to be
marked as migrating.  container_content will be able to see this and
reshape_array will be called.  It should all work.

> 
> > 
> > 3/ mdadm will convert the raid4 back to raid0 and so mdmon will exit.
> > 
> > 4/ mdadm calls container_content and finds that the second array needs
> > to
> >    be reshaped.  It converts the array to raid4 and starts mdmon.
> >    Then they oversee the reshape of the second array
> 
> Metadata for second array is not updates by mdmon as we want "now" start mdmon for it.
> 

Yes it will, it is updated at the same time that the previous array finished
the reshape.

NeilBrown


> > 
> > 5/ when the reshape completes, mdadm converts the array back to RAID0
> > and
> >    mdmon exits.
> > 
> > All done.
> 
> 
> All looks ok, except "next/2nd" raid0 array is reshaped. At "previous/1st" reshape finish, it is not monitored yet, so mdmon can't update metadata,
> before mdadm enters reshape_array() to execute takeover and then mdmon is started.
> Decision to enter reshape_array() is made based on metadata (not updated).
> 
> This is a reason why I'm using additional list for reshape decision. If mdmon doesn't make update for raid0 (not monitored)
> Mdadm tries to run mdmon for it (in reshape_array()) and waits for metadata update and decision about reshape.
> 
> Such implementation allows to follow directions for single reshape_super() call for container operation.
> 
> BR
> Adam
> 
> 
> > 
> > 
> > This should avoid almost all special-casing.
> > 
> > NeilBrown
> > 
> > 
> > >
> > > BR
> > > Adam
> > >
> > > ---
> > >
> > > Adam Kwolek (6):
> > >       FIX: sync_completed == 0 causes reshape cancellation in
> > metadata
> > >       FIX: mdadm hangs during reshape
> > >       FIX: mdadm throws coredump on exit in error case
> > >       FIX: reshape raid0 on second array
> > >       FIX: continue raid0 reshape
> > >       imsm: Update metadata for second array
> > >
> > >
> > >  Grow.c        |   95
> > +++++++++++++++++++++++++++++++++++++++++++++++++++------
> > >  monitor.c     |    4 ++
> > >  super-intel.c |   35 +++++++++++++++++++--
> > >  3 files changed, 120 insertions(+), 14 deletions(-)
> > >
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> > in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html