[PATCH 1/5] libdm-deptree-return-failure-activate-and-preload-children

[PATCH 1/5] libdm-deptree-return-failure-activate-and-preload-children

[Mike Snitzer]

Return error immediately to dm_tree_preload_children() and
dm_tree_activate_children() callers.

Otherwise resume_lv and its variants can fail silently.

Catching these failures is especially important now that dm targets like
crypt and snapshot-merge can fail in .preresume

Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
 libdm/libdm-deptree.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/libdm/libdm-deptree.c b/libdm/libdm-deptree.c
index 1465fd3..bb10fe5 100644
--- a/libdm/libdm-deptree.c
+++ b/libdm/libdm-deptree.c
@@ -1158,7 +1158,8 @@ int dm_tree_activate_children(struct dm_tree_node *dnode,
 			continue;
 
 		if (dm_tree_node_num_children(child, 0))
-			dm_tree_activate_children(child, uuid_prefix, uuid_prefix_len);
+			if (!dm_tree_activate_children(child, uuid_prefix, uuid_prefix_len))
+				return_0;
 	}
 
 	handle = NULL;
@@ -1204,7 +1205,7 @@ int dm_tree_activate_children(struct dm_tree_node *dnode,
 				log_error("Unable to resume %s (%" PRIu32
 					  ":%" PRIu32 ")", child->name, child->info.major,
 					  child->info.minor);
-				continue;
+				return 0;
 			}
 
 			/* Update cached info */
@@ -1621,7 +1622,8 @@ int dm_tree_preload_children(struct dm_tree_node *dnode,
 			continue;
 
 		if (dm_tree_node_num_children(child, 0))
-			dm_tree_preload_children(child, uuid_prefix, uuid_prefix_len);
+			if (!dm_tree_preload_children(child, uuid_prefix, uuid_prefix_len))
+				return_0;
 
 		/* FIXME Cope if name exists with no uuid? */
 		if (!child->info.exists) {
@@ -1655,7 +1657,7 @@ int dm_tree_preload_children(struct dm_tree_node *dnode,
 			log_error("Unable to resume %s (%" PRIu32
 				  ":%" PRIu32 ")", child->name, child->info.major,
 				  child->info.minor);
-			continue;
+			return 0;
 		}
 
 		/* Update cached info */
-- 
1.6.5.2

[PATCH 2/5] libdm-deptree-return-failure-suspend-children

[Mike Snitzer]

Return error immediately to dm_tree_suspend_children() callers.

Otherwise suspend_lv and its variants can fail silently.

Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
 libdm/libdm-deptree.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/libdm/libdm-deptree.c b/libdm/libdm-deptree.c
index bb10fe5..6f5355d 100644
--- a/libdm/libdm-deptree.c
+++ b/libdm/libdm-deptree.c
@@ -1109,7 +1109,7 @@ int dm_tree_suspend_children(struct dm_tree_node *dnode,
 			log_error("Unable to suspend %s (%" PRIu32
 				  ":%" PRIu32 ")", name, info.major,
 				  info.minor);
-			continue;
+			return 0;
 		}
 
 		/* Update cached info */
@@ -1130,7 +1130,8 @@ int dm_tree_suspend_children(struct dm_tree_node *dnode,
 			continue;
 
 		if (dm_tree_node_num_children(child, 0))
-			dm_tree_suspend_children(child, uuid_prefix, uuid_prefix_len);
+			if (!dm_tree_suspend_children(child, uuid_prefix, uuid_prefix_len))
+				return_0;
 	}
 
 	return 1;
-- 
1.6.5.2

[PATCH 3/5] libdm-deptree-return-failure-deactivate-children

[Mike Snitzer]

Return error immediately to dm_tree_deactivate_children() callers.

Otherwise deactivate_lv can fail silently.

Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
 libdm/libdm-deptree.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/libdm/libdm-deptree.c b/libdm/libdm-deptree.c
index 6f5355d..91f7177 100644
--- a/libdm/libdm-deptree.c
+++ b/libdm/libdm-deptree.c
@@ -1043,11 +1043,12 @@ int dm_tree_deactivate_children(struct dm_tree_node *dnode,
 			log_error("Unable to deactivate %s (%" PRIu32
 				  ":%" PRIu32 ")", name, info.major,
 				  info.minor);
-			continue;
+			return 0;
 		}
 
 		if (dm_tree_node_num_children(child, 0))
-			dm_tree_deactivate_children(child, uuid_prefix, uuid_prefix_len);
+			if (!dm_tree_deactivate_children(child, uuid_prefix, uuid_prefix_len))
+				return_0;
 	}
 
 	return 1;
-- 
1.6.5.2

[PATCH 4/5] lvm-stack-on-resume-and-suspend-failures

[Mike Snitzer]

Add missing 'stack;' for all suspend_lv and resume_lv callers.

Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
 lib/locking/locking.c   |    6 ++++--
 lib/metadata/lv_manip.c |    3 ++-
 lib/metadata/mirror.c   |    5 +++--
 tools/lvchange.c        |    6 ++++--
 tools/lvconvert.c       |    3 ++-
 tools/lvresize.c        |    3 ++-
 tools/pvmove.c          |   21 ++++++++++++++-------
 tools/toollib.c         |   13 ++++++++++++-
 8 files changed, 43 insertions(+), 17 deletions(-)

diff --git a/lib/locking/locking.c b/lib/locking/locking.c
index 9d86433..de890b2 100644
--- a/lib/locking/locking.c
+++ b/lib/locking/locking.c
@@ -439,7 +439,8 @@ int resume_lvs(struct cmd_context *cmd, struct dm_list *lvs)
 	struct lv_list *lvl;
 
 	dm_list_iterate_items(lvl, lvs)
-		resume_lv(cmd, lvl->lv);
+		if (!resume_lv(cmd, lvl->lv))
+			stack;
 
 	return 1;
 }
@@ -455,7 +456,8 @@ int suspend_lvs(struct cmd_context *cmd, struct dm_list *lvs)
 			log_error("Failed to suspend %s", lvl->lv->name);
 			dm_list_uniterate(lvh, lvs, &lvl->list) {
 				lvl = dm_list_item(lvh, struct lv_list);
-				resume_lv(cmd, lvl->lv);
+				if (!resume_lv(cmd, lvl->lv))
+					stack;
 			}
 
 			return 0;
diff --git a/lib/metadata/lv_manip.c b/lib/metadata/lv_manip.c
index 5041fd8..260a9ee 100644
--- a/lib/metadata/lv_manip.c
+++ b/lib/metadata/lv_manip.c
@@ -1822,7 +1822,8 @@ int lv_rename(struct cmd_context *cmd, struct logical_volume *lv,
 	if (!(r = vg_commit(vg)))
 		stack;
 
-	resume_lvs(cmd, &lvs_changed);
+	if (!resume_lvs(cmd, &lvs_changed))
+		stack;
 out:
 	backup(vg);
 	return r;
diff --git a/lib/metadata/mirror.c b/lib/metadata/mirror.c
index 6936e8c..d53a219 100644
--- a/lib/metadata/mirror.c
+++ b/lib/metadata/mirror.c
@@ -596,8 +596,9 @@ static int _remove_mirror_images(struct logical_volume *lv,
 	}
 
 	if (!vg_commit(mirrored_seg->lv->vg)) {
-		resume_lv(mirrored_seg->lv->vg->cmd, mirrored_seg->lv);
-		return 0;
+		if (!resume_lv(mirrored_seg->lv->vg->cmd, mirrored_seg->lv))
+			stack;
+		return_0;
 	}
 
 	log_very_verbose("Updating \"%s\" in kernel", mirrored_seg->lv->name);
diff --git a/tools/lvchange.c b/tools/lvchange.c
index ab4dc33..ce94101 100644
--- a/tools/lvchange.c
+++ b/tools/lvchange.c
@@ -64,7 +64,8 @@ static int lvchange_permission(struct cmd_context *cmd,
 	}
 
 	if (!vg_commit(lv->vg)) {
-		resume_lv(cmd, lv);
+		if (!resume_lv(cmd, lv))
+			stack;
 		goto_out;
 	}
 
@@ -385,7 +386,8 @@ static int lvchange_readahead(struct cmd_context *cmd,
 	}
 
 	if (!vg_commit(lv->vg)) {
-		resume_lv(cmd, lv);
+		if (!resume_lv(cmd, lv))
+			stack;
 		goto_out;
 	}
 
diff --git a/tools/lvconvert.c b/tools/lvconvert.c
index 0821c30..d1b86bf 100644
--- a/tools/lvconvert.c
+++ b/tools/lvconvert.c
@@ -790,7 +790,8 @@ static int _lvconvert_mirrors(struct cmd_context *cmd, struct logical_volume *lv
 	}
 
 	if (!vg_commit(lv->vg)) {
-		resume_lv(cmd, lv);
+		if (!resume_lv(cmd, lv))
+			stack;
 		goto_out;
 	}
 
diff --git a/tools/lvresize.c b/tools/lvresize.c
index 3b3a2eb..761ed27 100644
--- a/tools/lvresize.c
+++ b/tools/lvresize.c
@@ -638,7 +638,8 @@ static int _lvresize(struct cmd_context *cmd, struct volume_group *vg,
 
 	if (!vg_commit(vg)) {
 		stack;
-		resume_lv(cmd, lock_lv);
+		if (!resume_lv(cmd, lock_lv))
+			stack;
 		backup(vg);
 		return ECMD_FAILED;
 	}
diff --git a/tools/pvmove.c b/tools/pvmove.c
index ac09d6a..a56e377 100644
--- a/tools/pvmove.c
+++ b/tools/pvmove.c
@@ -292,7 +292,8 @@ static int _update_metadata(struct cmd_context *cmd, struct volume_group *vg,
 	/* Suspend mirrors on subsequent calls */
 	if (!first_time) {
 		if (!suspend_lv(cmd, lv_mirr)) {
-			resume_lvs(cmd, lvs_changed);
+			if (!resume_lvs(cmd, lvs_changed))
+				stack;
 			vg_revert(vg);
 			goto_out;
 		}
@@ -302,8 +303,10 @@ static int _update_metadata(struct cmd_context *cmd, struct volume_group *vg,
 	if (!vg_commit(vg)) {
 		log_error("ABORTING: Volume group metadata update failed.");
 		if (!first_time)
-			resume_lv(cmd, lv_mirr);
-		resume_lvs(cmd, lvs_changed);
+			if (!resume_lv(cmd, lv_mirr))
+				stack;
+		if (!resume_lvs(cmd, lvs_changed))
+			stack;
 		goto out;
 	}
 
@@ -327,7 +330,8 @@ static int _update_metadata(struct cmd_context *cmd, struct volume_group *vg,
 	} else if (!resume_lv(cmd, lv_mirr)) {
 		log_error("Unable to reactivate logical volume \"%s\"",
 			  lv_mirr->name);
-		resume_lvs(cmd, lvs_changed);
+		if (!resume_lvs(cmd, lvs_changed))
+			stack;
 		goto out;
 	}
 
@@ -501,8 +505,10 @@ static int _finish_pvmove(struct cmd_context *cmd, struct volume_group *vg,
 		log_error("ABORTING: Failed to write new data locations "
 			  "to disk.");
 		vg_revert(vg);
-		resume_lv(cmd, lv_mirr);
-		resume_lvs(cmd, lvs_changed);
+		if (!resume_lv(cmd, lv_mirr))
+			stack;
+		if (!resume_lvs(cmd, lvs_changed))
+			stack;
 		return 0;
 	}
 
@@ -514,7 +520,8 @@ static int _finish_pvmove(struct cmd_context *cmd, struct volume_group *vg,
 	}
 
 	/* Unsuspend LVs */
-	resume_lvs(cmd, lvs_changed);
+	if (!resume_lvs(cmd, lvs_changed))
+		stack;
 
 	/* Deactivate mirror LV */
 	if (!deactivate_lv(cmd, lv_mirr)) {
diff --git a/tools/toollib.c b/tools/toollib.c
index 39a6d69..ada1302 100644
--- a/tools/toollib.c
+++ b/tools/toollib.c
@@ -1252,7 +1252,18 @@ int vgcreate_params_set_from_args(struct cmd_context *cmd,
 
 int lv_refresh(struct cmd_context *cmd, struct logical_volume *lv)
 {
-	return suspend_lv(cmd, lv) && resume_lv(cmd, lv);
+	int r = 0;
+
+	r = suspend_lv(cmd, lv);
+	if (!r)
+		goto_out;
+
+	r = resume_lv(cmd, lv);
+	if (!r)
+		goto_out;
+
+out:
+	return r;
 }
 
 int vg_refresh_visible(struct cmd_context *cmd, struct volume_group *vg)
-- 
1.6.5.2

[PATCH 5/5] lvm-stack-on-activate-and-deactivate-failures

[Mike Snitzer]

Add missing 'stack;' for all activate_lv and deactivate_lv callers.

Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
 lib/activate/activate.c |    3 ++-
 lib/locking/locking.c   |    3 ++-
 tools/pvmove.c          |   11 +++++++++--
 tools/vgchange.c        |   20 +++++++++++++++-----
 4 files changed, 28 insertions(+), 9 deletions(-)

diff --git a/lib/activate/activate.c b/lib/activate/activate.c
index b3a70db..7256dab 100644
--- a/lib/activate/activate.c
+++ b/lib/activate/activate.c
@@ -710,7 +710,8 @@ int lv_is_active(struct logical_volume *lv)
 	 * FIXME: check status to not deactivate already activate device
 	 */
 	if (activate_lv_excl(lv->vg->cmd, lv)) {
-		deactivate_lv(lv->vg->cmd, lv);
+		if (!deactivate_lv(lv->vg->cmd, lv))
+			stack;
 		return 0;
 	}
 
diff --git a/lib/locking/locking.c b/lib/locking/locking.c
index de890b2..8b89b23 100644
--- a/lib/locking/locking.c
+++ b/lib/locking/locking.c
@@ -483,7 +483,8 @@ int activate_lvs(struct cmd_context *cmd, struct dm_list *lvs, unsigned exclusiv
 			log_error("Failed to activate %s", lvl->lv->name);
 			dm_list_uniterate(lvh, lvs, &lvl->list) {
 				lvl = dm_list_item(lvh, struct lv_list);
-				activate_lv(cmd, lvl->lv);
+				if (!activate_lv(cmd, lvl->lv))
+					stack;
 			}
 			return 0;
 		}
diff --git a/tools/pvmove.c b/tools/pvmove.c
index a56e377..0819746 100644
--- a/tools/pvmove.c
+++ b/tools/pvmove.c
@@ -261,10 +261,17 @@ static struct logical_volume *_set_up_pvmove_lv(struct cmd_context *cmd,
 static int _activate_lv(struct cmd_context *cmd, struct logical_volume *lv_mirr,
 			unsigned exclusive)
 {
+	int r = 0;
+
 	if (exclusive)
-		return activate_lv_excl(cmd, lv_mirr);
+		r = activate_lv_excl(cmd, lv_mirr);
+	else
+		r = activate_lv(cmd, lv_mirr);
 
-	return activate_lv(cmd, lv_mirr);
+	if (!r)
+		stack;
+
+	return r;
 }
 
 static int _finish_pvmove(struct cmd_context *cmd, struct volume_group *vg,
diff --git a/tools/vgchange.c b/tools/vgchange.c
index 80ac6e7..c1aa5cf 100644
--- a/tools/vgchange.c
+++ b/tools/vgchange.c
@@ -79,19 +79,29 @@ static int _activate_lvs_in_vg(struct cmd_context *cmd,
 			continue;
 
 		if (activate == CHANGE_AN) {
-			if (!deactivate_lv(cmd, lv))
+			if (!deactivate_lv(cmd, lv)) {
+				stack;
 				continue;
+			}
 		} else if (activate == CHANGE_ALN) {
-			if (!deactivate_lv_local(cmd, lv))
+			if (!deactivate_lv_local(cmd, lv)) {
+				stack;
 				continue;
+			}
 		} else if (lv_is_origin(lv) || (activate == CHANGE_AE)) {
-			if (!activate_lv_excl(cmd, lv))
+			if (!activate_lv_excl(cmd, lv)) {
+				stack;
 				continue;
+			}
 		} else if (activate == CHANGE_ALY) {
-			if (!activate_lv_local(cmd, lv))
+			if (!activate_lv_local(cmd, lv)) {
+				stack;
 				continue;
-		} else if (!activate_lv(cmd, lv))
+			}
+		} else if (!activate_lv(cmd, lv)) {
+			stack;
 			continue;
+		}
 
 		if (activate != CHANGE_AN && activate != CHANGE_ALN &&
 		    (lv->status & (PVMOVE|CONVERTING)))
-- 
1.6.5.2

[PATCH 1/5] libdm-deptree-return-failure-activate-and-preload-children

[Zdenek Kabelac]

Dne 20.11.2009 22:29, Mike Snitzer napsal(a):
> Return error immediately to dm_tree_preload_children() and
> dm_tree_activate_children() callers.
> 
> Otherwise resume_lv and its variants can fail silently.
> 
> Catching these failures is especially important now that dm targets like
> crypt and snapshot-merge can fail in .preresume
> 

Similar case applies for replicator - and I'm using similar code in my own
patch set.

So it's definitely needed.

Ack


> Signed-off-by: Mike Snitzer <snitzer@redhat.com>
> ---
>  libdm/libdm-deptree.c |   10 ++++++----
>  1 files changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/libdm/libdm-deptree.c b/libdm/libdm-deptree.c
> index 1465fd3..bb10fe5 100644
> --- a/libdm/libdm-deptree.c
> +++ b/libdm/libdm-deptree.c
> @@ -1158,7 +1158,8 @@ int dm_tree_activate_children(struct dm_tree_node *dnode,
>  			continue;
>  
>  		if (dm_tree_node_num_children(child, 0))
> -			dm_tree_activate_children(child, uuid_prefix, uuid_prefix_len);
> +			if (!dm_tree_activate_children(child, uuid_prefix, uuid_prefix_len))
> +				return_0;
>  	}
>  
>  	handle = NULL;
> @@ -1204,7 +1205,7 @@ int dm_tree_activate_children(struct dm_tree_node *dnode,
>  				log_error("Unable to resume %s (%" PRIu32
>  					  ":%" PRIu32 ")", child->name, child->info.major,

[PATCH 2/5] libdm-deptree-return-failure-suspend-children

[Zdenek Kabelac]

Dne 20.11.2009 22:29, Mike Snitzer napsal(a):
> Return error immediately to dm_tree_suspend_children() callers.
> 
> Otherwise suspend_lv and its variants can fail silently.
> 
> Signed-off-by: Mike Snitzer <snitzer@redhat.com>
> ---
>  libdm/libdm-deptree.c |    5 +++--
>  1 files changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/libdm/libdm-deptree.c b/libdm/libdm-deptree.c
> index bb10fe5..6f5355d 100644
> --- a/libdm/libdm-deptree.c
> +++ b/libdm/libdm-deptree.c
> @@ -1109,7 +1109,7 @@ int dm_tree_suspend_children(struct dm_tree_node *dnode,
>  			log_error("Unable to suspend %s (%" PRIu32
>  				  ":%" PRIu32 ")", name, info.major,
>  				  info.minor);
> -			continue;
> +			return 0;
>  		}


This is a bit hard to tell what's the right way here - either to continue
suspending  'the rest of the monster' or to stop with the first failure.

In case of the replicator - when suspend of one head would fail - I would
probably prefer to see the code to continue suspending remaing heads.


Zdenek

>  
>  		/* Update cached info */
> @@ -1130,7 +1130,8 @@ int dm_tree_suspend_children(struct dm_tree_node *dnode,
>  			continue;
>  
>  		if (dm_tree_node_num_children(child, 0))
> -			dm_tree_suspend_children(child, uuid_prefix, uuid_prefix_len);
> +			if (!dm_tree_suspend_children(child, uuid_prefix, uuid_prefix_len))
> +				return_0;
>  	}
>  
>  	return 1;

[PATCH 4/5] lvm-stack-on-resume-and-suspend-failures

[Zdenek Kabelac]

Dne 20.11.2009 22:29, Mike Snitzer napsal(a):
> Add missing 'stack;' for all suspend_lv and resume_lv callers.
> 

How about using some macro like:

#define test_stack(a...) \
	do { \
		if (!a) \
			stack; \
	} while(0);

Well the name is randomly picked - but it could make code more compact and
readable ?

Zdenek


> Signed-off-by: Mike Snitzer <snitzer@redhat.com>
> ---
>  lib/locking/locking.c   |    6 ++++--
>  lib/metadata/lv_manip.c |    3 ++-
>  lib/metadata/mirror.c   |    5 +++--
>  tools/lvchange.c        |    6 ++++--
>  tools/lvconvert.c       |    3 ++-
>  tools/lvresize.c        |    3 ++-
>  tools/pvmove.c          |   21 ++++++++++++++-------
>  tools/toollib.c         |   13 ++++++++++++-
>  8 files changed, 43 insertions(+), 17 deletions(-)
> 

Re: [PATCH 2/5] libdm-deptree-return-failure-suspend-children

[Mike Snitzer]

On Fri, Nov 20 2009 at  5:57pm -0500,
Zdenek Kabelac <zkabelac@redhat.com> wrote:

> Dne 20.11.2009 22:29, Mike Snitzer napsal(a):
> > Return error immediately to dm_tree_suspend_children() callers.
> > 
> > Otherwise suspend_lv and its variants can fail silently.
> > 
> > Signed-off-by: Mike Snitzer <snitzer@redhat.com>
> > ---
> >  libdm/libdm-deptree.c |    5 +++--
> >  1 files changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/libdm/libdm-deptree.c b/libdm/libdm-deptree.c
> > index bb10fe5..6f5355d 100644
> > --- a/libdm/libdm-deptree.c
> > +++ b/libdm/libdm-deptree.c
> > @@ -1109,7 +1109,7 @@ int dm_tree_suspend_children(struct dm_tree_node *dnode,
> >  			log_error("Unable to suspend %s (%" PRIu32
> >  				  ":%" PRIu32 ")", name, info.major,
> >  				  info.minor);
> > -			continue;
> > +			return 0;
> >  		}
> 
> 
> This is a bit hard to tell what's the right way here - either to continue
> suspending  'the rest of the monster' or to stop with the first failure.
> 
> In case of the replicator - when suspend of one head would fail - I would
> probably prefer to see the code to continue suspending remaing heads.

OK, the other possibility that Alasdair and I discussed is that we could
return failure once all nodes at the same level have been processed.

Would that work for replicator?  Are all heads at the same level in the tree?

Mike

Re: [PATCH 2/5] libdm-deptree-return-failure-suspend-children

[Zdenek Kabelac]

Dne 21.11.2009 03:26, Mike Snitzer napsal(a):
> On Fri, Nov 20 2009 at  5:57pm -0500,
> Zdenek Kabelac <zkabelac@redhat.com> wrote:
> 
>> Dne 20.11.2009 22:29, Mike Snitzer napsal(a):
>>> Return error immediately to dm_tree_suspend_children() callers.
>>>
>>> Otherwise suspend_lv and its variants can fail silently.
>>>
>>> Signed-off-by: Mike Snitzer <snitzer@redhat.com>
>>> ---
>>>  libdm/libdm-deptree.c |    5 +++--
>>>  1 files changed, 3 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/libdm/libdm-deptree.c b/libdm/libdm-deptree.c
>>> index bb10fe5..6f5355d 100644
>>> --- a/libdm/libdm-deptree.c
>>> +++ b/libdm/libdm-deptree.c
>>> @@ -1109,7 +1109,7 @@ int dm_tree_suspend_children(struct dm_tree_node *dnode,
>>>  			log_error("Unable to suspend %s (%" PRIu32
>>>  				  ":%" PRIu32 ")", name, info.major,
>>>  				  info.minor);
>>> -			continue;
>>> +			return 0;
>>>  		}
>>
>>
>> This is a bit hard to tell what's the right way here - either to continue
>> suspending  'the rest of the monster' or to stop with the first failure.
>>
>> In case of the replicator - when suspend of one head would fail - I would
>> probably prefer to see the code to continue suspending remaing heads.
> 
> OK, the other possibility that Alasdair and I discussed is that we could
> return failure once all nodes at the same level have been processed.
> 
> Would that work for replicator?  Are all heads at the same level in the tree?

Yes - that would probably work well for replicator (though the error in the
suspend as rather hypothetical in this case anyway)

But - still - is it the best thing to stop suspending elements from tree - if
one path fails?

Zdenek