Improving dm-mirror as a final year project

Improving dm-mirror as a final year project

[Miklos Vajna]

[-- Attachment #1.1: Type: text/plain, Size: 1713 bytes --]

Hi,

I got the possibility to work on dm-mirror as a final year project at
ULX, the Hungarian distributor of Red Hat.

Get my feet wet, I created two small patches:

1) dm-mirror: allow setting ios count to 0

Always read from the default_mirror in that case.

2) dm-mirror: allow setting the default mirror

These can help in case one data leg of a mirror is a remote (iSCSI) one,
so the default RR aproach is not optimal for reading. (One may set the
ios count to 0, set the default mirror to the local one, and that will
cause a read speedup.)

I do not yet have the right to send those patches (I do this in
university time, so the copyright is not mine), but I hope to be able to
do so - to get them reviewed.

So the final year project's aim is to improve "the fault tolerance and
performance of dm-mirror". We (I and my mentors) have two ideas in that
area, not counting the above patches:

1) Make the currently hardwired RR read approach modular, that would
allow implementing for example a weighted RR algorithm. (In case one
disk is two times faster than the other one, etc.)

2) From our experiments, it seems that in case the dm-mirror looses one
of its legs and there is a write to the mirror, it gets converted to a
linear volume. It would be nice (not sure how easy) to use the mirror
log to mark the dirty blocks, so that the volume would not be converted
to a linear one: once the other leg is back, it could be updated based
on the mirror log.

The question: what do you (who have much more experience with dm-mirror
than me) think - are these reasonable goals? If not, what would you
improve/change/add/remove to the above list?

Thanks,

Miklos

PS: I'm not subscribed, please keep me in CC.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Improving dm-mirror as a final year project

[nishant mungse]

[-- Attachment #1.1: Type: text/plain, Size: 2446 bytes --]

On Tue, Jan 25, 2011 at 10:53 AM, Miklos Vajna <vmiklos@frugalware.org>wrote:

> Hi,
>
> I got the possibility to work on dm-mirror as a final year project at
> ULX, the Hungarian distributor of Red Hat.
>
> Get my feet wet, I created two small patches:
>
> 1) dm-mirror: allow setting ios count to 0
>
> Always read from the default_mirror in that case.
>
> 2) dm-mirror: allow setting the default mirror
>
> These can help in case one data leg of a mirror is a remote (iSCSI) one,
> so the default RR aproach is not optimal for reading. (One may set the
> ios count to 0, set the default mirror to the local one, and that will
> cause a read speedup.)
>
> I do not yet have the right to send those patches (I do this in
> university time, so the copyright is not mine), but I hope to be able to
> do so - to get them reviewed.
>
> So the final year project's aim is to improve "the fault tolerance and
> performance of dm-mirror". We (I and my mentors) have two ideas in that
> area, not counting the above patches:
>
> 1) Make the currently hardwired RR read approach modular, that would
> allow implementing for example a weighted RR algorithm. (In case one
> disk is two times faster than the other one, etc.)
>
> 2) From our experiments, it seems that in case the dm-mirror looses one
> of its legs and there is a write to the mirror, it gets converted to a
> linear volume. It would be nice (not sure how easy) to use the mirror
> log to mark the dirty blocks, so that the volume would not be converted
> to a linear one: once the other leg is back, it could be updated based
> on the mirror log.
>
> The question: what do you (who have much more experience with dm-mirror
> than me) think - are these reasonable goals? If not, what would you
> improve/change/add/remove to the above list?
>
> Thanks,
>
> Miklos
>
> PS: I'm not subscribed, please keep me in CC.
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>


hello all,
There is one doubt regarding the dm-raid1. As far as i know, in dm-raid1 the
data is written parallelly  on all the mirrors of mirrorset and if any of
the mirror fails to write the data then dm-mirror adds this mirror to fail
list by increasing the error count in "fail mirror" function in dm-raid1.
Actually my doubt is where this error count is decremented? i.e after kcpyd
or before and where exactly this error count is decremented?


Regards,
Nishant.

[-- Attachment #1.2: Type: text/html, Size: 3091 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Improving dm-mirror as a final year project

[Malahal Naineni]

nishant mungse [nishantmungse@gmail.com] wrote:
> 
>    hello all,
>    There is one doubt regarding the dm-raid1. As far as i know, in dm-raid1
>    the data is written parallelly� on all the mirrors of mirrorset and if any
>    of the mirror fails to write the data then dm-mirror adds this mirror to
>    fail list by increasing the error count in "fail mirror" function in
>    dm-raid1.
>    Actually my doubt is where this error count is decremented? i.e after
>    kcpyd or before and where exactly this error count is decremented?

There is no actual 'fail list' of mirror legs. The error_count is never
decremented. The 'error_count' is only used to tell if the mirror leg
has encountered errors. The only way to clear that counter is
re-loading the table!

See the comment:

        /*
         * error_count is used for nothing more than a
         * simple way to tell if a device has encountered
         * errors.
         */
        atomic_inc(&m->error_count);

Hope that helps,
Malahal.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: Improving dm-mirror as a final year project

[nishant mungse]

[-- Attachment #1.1: Type: text/plain, Size: 1615 bytes --]

On Wed, Jan 26, 2011 at 3:04 PM, Malahal Naineni <malahal@us.ibm.com> wrote:

> nishant mungse [nishantmungse@gmail.com] wrote:
> >
> >    hello all,
> >    There is one doubt regarding the dm-raid1. As far as i know, in
> dm-raid1
> >    the data is written parallelly� on all the mirrors of mirrorset and if
> any
> >    of the mirror fails to write the data then dm-mirror adds this mirror
> to
> >    fail list by increasing the error count in "fail mirror" function in
> >    dm-raid1.
> >    Actually my doubt is where this error count is decremented? i.e after
> >    kcpyd or before and where exactly this error count is decremented?
>
> There is no actual 'fail list' of mirror legs. The error_count is never
> decremented. The 'error_count' is only used to tell if the mirror leg
> has encountered errors. The only way to clear that counter is
> re-loading the table!
>
> See the comment:
>
>        /*
>         * error_count is used for nothing more than a
>         * simple way to tell if a device has encountered
>         * errors.
>         */
>        atomic_inc(&m->error_count);
>
> Hope that helps,
> Malahal.
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

Hi Malahal

Thanks for reply. I am not able to understand what  this
"INIT_WORK(&ms->kmirrord_work, do_mirror)" function does. It  is written in
mirror_ctr of dm-raid1. What i think is whenever a bio comes dm_mirror will
be called if that is case then why this is written in mirror_ctr. Please
answer this.

Regards,
Nishant.

[-- Attachment #1.2: Type: text/html, Size: 2359 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Improving dm-mirror as a final year project

[Jonathan Brassow]

On Jan 25, 2011, at 9:53 AM, Miklos Vajna wrote:

> Hi,
>
> I got the possibility to work on dm-mirror as a final year project at
> ULX, the Hungarian distributor of Red Hat.
>
> Get my feet wet, I created two small patches:
>
> 1) dm-mirror: allow setting ios count to 0
>
> Always read from the default_mirror in that case.
>
> 2) dm-mirror: allow setting the default mirror
>
> These can help in case one data leg of a mirror is a remote (iSCSI)  
> one,
> so the default RR aproach is not optimal for reading. (One may set the
> ios count to 0, set the default mirror to the local one, and that will
> cause a read speedup.)
>
> I do not yet have the right to send those patches (I do this in
> university time, so the copyright is not mine), but I hope to be  
> able to
> do so - to get them reviewed.
>
> So the final year project's aim is to improve "the fault tolerance and
> performance of dm-mirror". We (I and my mentors) have two ideas in  
> that
> area, not counting the above patches:
>
> 1) Make the currently hardwired RR read approach modular, that would
> allow implementing for example a weighted RR algorithm. (In case one
> disk is two times faster than the other one, etc.)
>
> 2) From our experiments, it seems that in case the dm-mirror looses  
> one
> of its legs and there is a write to the mirror, it gets converted to a
> linear volume. It would be nice (not sure how easy) to use the mirror
> log to mark the dirty blocks, so that the volume would not be  
> converted
> to a linear one: once the other leg is back, it could be updated based
> on the mirror log.
>
> The question: what do you (who have much more experience with dm- 
> mirror
> than me) think - are these reasonable goals? If not, what would you
> improve/change/add/remove to the above list?

Would you consider working on the recently added dm-raid.c?  It is an  
attempt to access the MD personalities through device-mapper.  As  
such, RAID456 (the initial drop) will be available - in addition to  
RAID1.  There is much to be done in this area - large projects as well  
as small - encompassing performance issues, metadata layout, testing,  
etc...  There is also a lot of attention being paid to this area.

I think device-mapper mirroring will be used for a while, but it will  
likely become deprecated.

  brassow

Re: Improving dm-mirror as a final year project

[Miklos Vajna]

Hi Jonathan,

On Tue, Feb 01, 2011 at 10:12:36AM -0600, Jonathan Brassow <jbrassow@redhat.com> wrote:
> Would you consider working on the recently added dm-raid.c?  It is an  
> attempt to access the MD personalities through device-mapper.  As  
> such, RAID456 (the initial drop) will be available - in addition to  
> RAID1.  There is much to be done in this area - large projects as well  
> as small - encompassing performance issues, metadata layout, testing,  
> etc...  There is also a lot of attention being paid to this area.
> 
> I think device-mapper mirroring will be used for a while, but it will  
> likely become deprecated.

Sure, that sounds interesting. The first part of my MSc thesis should be
done till 2011.05.13 (14 weeks), I have about 2 workdays / week for the
project. Of course the actual development time is a bit less, since I
have to produce paper from this (about 30 pages for the first part).

(The second part should be done in Q3-Q4, when I'll have a bit more
dedicated time.)

Do you have a project idea for such a timeframe?

My mentors are asked: is dm-raid supported in a RHEL version?

As far as I see it was added in commit
9d09e663d5502c46f2d9481c04c1087e1c2da698 after 2.6.37, so it's not part
of RHEL (I created my dm-mirror patches on RHEL5) - and it's a pro for
them if I can do my development based on a RHEL kernel, what they use.

So - given such a timeframe, is it better to develop dm-mirror, which is
fairly stable or is it possible to develop dm-raid, even if it's quite
new?

Thanks,

Miklos

Re: Improving dm-mirror as a final year project

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

On Tue, Feb 01, 2011 at 10:12:36AM -0600, Jonathan Brassow <jbrassow@redhat.com> wrote:
> > I do not yet have the right to send those patches (I do this in
> > university time, so the copyright is not mine), but I hope to be  
> > able to
> > do so - to get them reviewed.

Hi,

I'm attaching the two patches. I tested these on RHEL5, but I don't
think there are major changes in newer versions, either.

If that's the only problem, I can test it on latest linux-2.6.git.

Any feedback is welcome. :)

Thanks,

Miklos

[-- Attachment #2: 0001-dm-mirror-allow-setting-ios-count-to-0.patch --]
[-- Type: text/plain, Size: 1473 bytes --]

From 2a5aaeef1caed0019abdce92fdb6e6a4413677f8 Mon Sep 17 00:00:00 2001
From: Miklos Vajna <vmiklos@ulx.hu>
Date: Sat, 25 Sep 2010 14:56:28 +0200
Subject: [PATCH] dm-mirror: allow setting ios count to 0

Always read from the default_mirror in that case. This can help in case
the default_mirror is way faster than the other data leg.

Signed-off-by: Miklos Vajna <vmiklos@ulx.hu>
---
 drivers/md/dm-raid1.c |   15 +++++++++++++--
 1 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c
index 23c1d65..318899e 100644
--- a/drivers/md/dm-raid1.c
+++ b/drivers/md/dm-raid1.c
@@ -849,6 +849,15 @@ static struct mirror *choose_mirror(struct mirror_set *ms)
 	 * the first we tried, so we know when we're done.
 	 */
 	ret = start_mirror = ms->read_mirror;
+
+	/*
+	 * If MIN_READS is zero, then always use the default one.
+	 */
+	if (!atomic_read(&ms->rr_ios_set)) {
+		ret = ms->default_mirror;
+		goto use_mirror;
+	}
+
 	do {
 		if (likely(!atomic_read(&ret->error_count) &&
 			   !atomic_dec_and_test(&ms->rr_ios)))
@@ -1848,8 +1857,10 @@ static int mirror_message(struct dm_target *ti, unsigned argc, char **argv)
 
 	if (sscanf(argv[3], "%u", &rr_ios_set) != 1 ||
 	    rr_ios_set < 2) {
-		DMERR("Round robin read ios have to be > 1");
-		return -EINVAL;
+		if (rr_ios_set != 0) {
+			DMERR("Round robin read ios have to be > 1 or 0");
+			return -EINVAL;
+		}
 	}
 
 	md = dm_table_get_md(ti->table);
-- 
1.5.5.6


[-- Attachment #3: 0002-dm-mirror-allow-setting-the-default-mirror.patch --]
[-- Type: text/plain, Size: 2712 bytes --]

From 02c399b2171dec66d7c014caa1efbc58c0ba8ee1 Mon Sep 17 00:00:00 2001
From: Miklos Vajna <vmiklos@ulx.hu>
Date: Sat, 25 Sep 2010 17:16:11 +0200
Subject: [PATCH] dm-mirror: allow setting the default mirror

This can help in case an other data leg is faster than the default one
and the ios count is set to zero.

Signed-off-by: Miklos Vajna <vmiklos@ulx.hu>
---
 drivers/md/dm-raid1.c |   54 +++++++++++++++++++++++++++++++-----------------
 1 files changed, 35 insertions(+), 19 deletions(-)

diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c
index 318899e..b746b0c 100644
--- a/drivers/md/dm-raid1.c
+++ b/drivers/md/dm-raid1.c
@@ -1845,31 +1845,47 @@ static void mirror_resume(struct dm_target *ti)
 /* Set round robin ios via message. */
 static int mirror_message(struct dm_target *ti, unsigned argc, char **argv)
 {
-	unsigned rr_ios_set;
 	struct mirror_set *ms = ti->private;
 	struct mapped_device *md;
 
-	if (argc != 4 ||
-	    strncmp(argv[0], "io_balance", strlen(argv[0])) ||
-	    strncmp(argv[1], "round_robin", strlen(argv[1])) ||
-	    strncmp(argv[2], "ios", strlen(argv[2])))
-		return -EINVAL;
-
-	if (sscanf(argv[3], "%u", &rr_ios_set) != 1 ||
-	    rr_ios_set < 2) {
-		if (rr_ios_set != 0) {
-			DMERR("Round robin read ios have to be > 1 or 0");
-			return -EINVAL;
+	if (argc == 4 &&
+	    !strncmp(argv[0], "io_balance", strlen(argv[0])) &&
+	    !strncmp(argv[1], "round_robin", strlen(argv[1])) &&
+	    !strncmp(argv[2], "ios", strlen(argv[2]))) {
+		unsigned rr_ios_set;
+		if (sscanf(argv[3], "%u", &rr_ios_set) != 1 ||
+				rr_ios_set < 2) {
+			if (rr_ios_set != 0) {
+				DMERR("Round robin read ios have to be > 1 or 0");
+				return -EINVAL;
+			}
 		}
+
+		md = dm_table_get_md(ti->table);
+		DMINFO("Setting round robin read ios for \"%s\" to %u",
+				dm_device_name(md), rr_ios_set);
+		dm_put(md);
+		atomic_set(&ms->rr_ios_set, rr_ios_set);
+		atomic_set(&ms->rr_ios, rr_ios_set);
+		return 0;
 	}
 
-	md = dm_table_get_md(ti->table);
-	DMINFO("Setting round robin read ios for \"%s\" to %u",
-	        dm_device_name(md), rr_ios_set);
-	dm_put(md);
-	atomic_set(&ms->rr_ios_set, rr_ios_set);
-	atomic_set(&ms->rr_ios, rr_ios_set);
-	return 0;
+	if (argc == 3 &&
+		!strncmp(argv[0], "io_balance", strlen(argv[0])) &&
+		!strncmp(argv[1], "default", strlen(argv[1]))) {
+		unsigned int m;
+		for (m = 0; m < ms->nr_mirrors; m++)
+			if (!strncmp(argv[2], ms->mirror[m].dev->name, strlen(argv[2]))) {
+				ms->default_mirror = &ms->mirror[m];
+				md = dm_table_get_md(ti->table);
+				DMINFO("Setting default device for \"%s\" to \"%s\"",
+						dm_device_name(md), argv[2]);
+				dm_put(md);
+				return 0;
+			}
+	}
+
+	return -EINVAL;
 }
 
 /*
-- 
1.5.5.6


[-- Attachment #4: Type: text/plain, Size: 0 bytes --]

Re: Improving dm-mirror as a final year project

[Jonathan Brassow]

On Feb 14, 2011, at 11:33 AM, Miklos Vajna wrote:

> On Tue, Feb 01, 2011 at 10:12:36AM -0600, Jonathan Brassow <jbrassow@redhat.com 
> > wrote:
>>> I do not yet have the right to send those patches (I do this in
>>> university time, so the copyright is not mine), but I hope to be
>>> able to
>>> do so - to get them reviewed.
>
> Hi,
>
> I'm attaching the two patches. I tested these on RHEL5, but I don't
> think there are major changes in newer versions, either.
>
> If that's the only problem, I can test it on latest linux-2.6.git.
>
> Any feedback is welcome. :)

Thanks for the patches.  I've seen the first one before (slightly  
different) - I'll discuss it with others whether to include in rhel5.   
There is no read-balancing in rhel6/upstream.

The second patch addresses which device should be primary.  This can  
be done when creating the mirror.  I'm not sure how much benefit there  
is to doing this additional step.  Most people will access dm mirrors  
through LVM - not through the dm message interface.  If it makes sense  
upstream - and you can argue for it - though, I would consider it.

Wether changes are going into rhel5 or rhel6, we still like it when  
they go upstream first.  We generally don't like feature inversion.

If you have any interest in dm-raid, these are some of the things that  
need to be done:
1) Definition of new MD superblock:  Some of this is started, and I've  
got a working version, but I'm sure there are pieces missing related  
to offsets that must be tracked for RAID type conversion, etc.
2) Bitmap work:  The bitmap keeps track of which areas of the array  
are being written.  Right now, I take all the bitmap code "as-is".   
There are a number of things in this area to be improved.  Firstly, we  
don't necessarily need all the fields in the bitmap superblock -  
perhaps this could be streamlined and added to the new MD superblock.   
Secondly, things are way too slow.  I get a 10x slowdown when using a  
bitmap with RAID1 through device-mapper.  This could be due to  the  
region-size chosen, the bitmap being at a stupid offset, or something  
else.  This problem could be solved by trial-and-error or through  
profiling and reason... seems like a great small project.
3) Conversion code:  New device-mapper targets (very simple small  
ones) must be written to engage the MD RAID conversion code (like when  
you change RAID4 to RAID5, for example)
4) Failure testing
5) LVM code: to handle creation of RAID devices
6) dmeventd code: to handle device failures

This is an incomplete list, but does have a variety of complexity and  
duration.

  brassow

Re: Improving dm-mirror as a final year project

[Miklos Vajna]

On Mon, Feb 14, 2011 at 03:31:00PM -0600, Jonathan Brassow <jbrassow@redhat.com> wrote:
> Thanks for the patches.  I've seen the first one before (slightly  
> different) - I'll discuss it with others whether to include in rhel5.   
> There is no read-balancing in rhel6/upstream.

Oh, do I read the code correctly that rhel6/upstream always reads from
the first mirror and switches only in case there is a read failure?

> The second patch addresses which device should be primary.  This can  
> be done when creating the mirror.  I'm not sure how much benefit there  
> is to doing this additional step.  Most people will access dm mirrors  
> through LVM - not through the dm message interface.  If it makes sense  
> upstream - and you can argue for it - though, I would consider it.

Is there such a messaging interface for lvm as well? I choosed this way
as in this case I did not have to alter the metadata.

One useful use case I can imagine is when both data legs of the mirror
are provided by iscsi and the administrator does not realise what is the
faster leg and her bad decision is found out only after there is some
data on the mirror.

My patch allows one to just set the first mirror in that case, without
saving data, recreating the mirror and restoring data. (Unless I missed
some other neat trick on how to do so.)

> Wether changes are going into rhel5 or rhel6, we still like it when  
> they go upstream first.  We generally don't like feature inversion.

Sure - I was not aware at all that the round robin part of the code is
RHEL5-specific.

> If you have any interest in dm-raid, these are some of the things that  
> need to be done:

Thanks for the list - I must admit that some of the points are Chinese
to me; I'm not that familiar with the codebase, just with the basic LVM
commands anc concepts.

> 1) Definition of new MD superblock:  Some of this is started, and I've  
> got a working version, but I'm sure there are pieces missing related  
> to offsets that must be tracked for RAID type conversion, etc.
> 2) Bitmap work:  The bitmap keeps track of which areas of the array  
> are being written.  Right now, I take all the bitmap code "as-is".   
> There are a number of things in this area to be improved.  Firstly, we  
> don't necessarily need all the fields in the bitmap superblock -  
> perhaps this could be streamlined and added to the new MD superblock.   
> Secondly, things are way too slow.  I get a 10x slowdown when using a  
> bitmap with RAID1 through device-mapper.  This could be due to  the  
> region-size chosen, the bitmap being at a stupid offset, or something  
> else.  This problem could be solved by trial-and-error or through  
> profiling and reason... seems like a great small project.
> 3) Conversion code:  New device-mapper targets (very simple small  
> ones) must be written to engage the MD RAID conversion code (like when  
> you change RAID4 to RAID5, for example)
> 4) Failure testing
> 5) LVM code: to handle creation of RAID devices
> 6) dmeventd code: to handle device failures

Before choosing from this list:

I first have to evaluate the current status of dm-raid so that we can
decide with my mentors if the topic of my thesis should be dm-mirror or
dm-raid (ie. if dm-raid is mature enough that I can write a thesis about
it). Where is the newest version of dm-raid.c? I saw the upstream kernel
has a single commit from this January, but I guess the rawhide / rhel
kernel contained this earlier - maybe there is a newer version than
upstream somewhere?

Also, is there any documentation on dm-raid? Google found
http://www.linux-archive.org/device-mapper-development/454656-dm-raid-wrapper-target-md-raid456.html
but maybe there is now a better way to create raid4 than using
gime_raid.pl?

And a last question: is support for raid1 a planned feature? I think
that would be interesting as well. (If dm-raid is going to replace
dm-mirror in the long run.)

Thanks!

Re: Improving dm-mirror as a final year project

[Jonathan Brassow]

[-- Attachment #1: Type: text/plain, Size: 5075 bytes --]


On Feb 15, 2011, at 6:52 AM, Miklos Vajna wrote:

> On Mon, Feb 14, 2011 at 03:31:00PM -0600, Jonathan Brassow <jbrassow@redhat.com 
> > wrote:
>> Thanks for the patches.  I've seen the first one before (slightly
>> different) - I'll discuss it with others whether to include in rhel5.
>> There is no read-balancing in rhel6/upstream.
>
> Oh, do I read the code correctly that rhel6/upstream always reads from
> the first mirror and switches only in case there is a read failure?

yes

>
>> The second patch addresses which device should be primary.  This can
>> be done when creating the mirror.  I'm not sure how much benefit  
>> there
>> is to doing this additional step.  Most people will access dm mirrors
>> through LVM - not through the dm message interface.  If it makes  
>> sense
>> upstream - and you can argue for it - though, I would consider it.
>
> Is there such a messaging interface for lvm as well? I choosed this  
> way
> as in this case I did not have to alter the metadata.

There is no msg interface via LVM (although LVM could use the message  
interface for some things.)

> One useful use case I can imagine is when both data legs of the mirror
> are provided by iscsi and the administrator does not realise what is  
> the
> faster leg and her bad decision is found out only after there is some
> data on the mirror.

Perhaps, but if you don't encode this in the LVM metadata, you will  
have to perform the action every time you reboot.  Instead, you could  
reorder the devices in userspace and reload the table.

> My patch allows one to just set the first mirror in that case, without
> saving data, recreating the mirror and restoring data. (Unless I  
> missed
> some other neat trick on how to do so.)
>
>> Wether changes are going into rhel5 or rhel6, we still like it when
>> they go upstream first.  We generally don't like feature inversion.
>
> Sure - I was not aware at all that the round robin part of the code is
> RHEL5-specific.
>
>> If you have any interest in dm-raid, these are some of the things  
>> that
>> need to be done:
>
> Thanks for the list - I must admit that some of the points are Chinese
> to me; I'm not that familiar with the codebase, just with the basic  
> LVM
> commands anc concepts.
>
>> 1) Definition of new MD superblock:  Some of this is started, and  
>> I've
>> got a working version, but I'm sure there are pieces missing related
>> to offsets that must be tracked for RAID type conversion, etc.
>> 2) Bitmap work:  The bitmap keeps track of which areas of the array
>> are being written.  Right now, I take all the bitmap code "as-is".
>> There are a number of things in this area to be improved.  Firstly,  
>> we
>> don't necessarily need all the fields in the bitmap superblock -
>> perhaps this could be streamlined and added to the new MD superblock.
>> Secondly, things are way too slow.  I get a 10x slowdown when using a
>> bitmap with RAID1 through device-mapper.  This could be due to  the
>> region-size chosen, the bitmap being at a stupid offset, or something
>> else.  This problem could be solved by trial-and-error or through
>> profiling and reason... seems like a great small project.
>> 3) Conversion code:  New device-mapper targets (very simple small
>> ones) must be written to engage the MD RAID conversion code (like  
>> when
>> you change RAID4 to RAID5, for example)
>> 4) Failure testing
>> 5) LVM code: to handle creation of RAID devices
>> 6) dmeventd code: to handle device failures
>
> Before choosing from this list:
>
> I first have to evaluate the current status of dm-raid so that we can
> decide with my mentors if the topic of my thesis should be dm-mirror  
> or
> dm-raid (ie. if dm-raid is mature enough that I can write a thesis  
> about
> it). Where is the newest version of dm-raid.c? I saw the upstream  
> kernel
> has a single commit from this January, but I guess the rawhide / rhel
> kernel contained this earlier - maybe there is a newer version than
> upstream somewhere?

The basic component that covers RAID456 is available upstream, as you  
saw.  I have an additional set of ~12 (reasonably small) patches that  
add RAID1 and superblock/bitmap support.  These patches are not yet  
upstream nor are they in any RHEL product.

> Also, is there any documentation on dm-raid? Google found
> http://www.linux-archive.org/device-mapper-development/454656-dm-raid-wrapper-target-md-raid456.html
> but maybe there is now a better way to create raid4 than using
> gime_raid.pl?

Yes, I have a script called 'gime_raid.pl' that creates the device- 
mapper tables for dm-raid.  Eventually, this will be pushed into LVM,  
but it was much easier (for testing purposes) to start with a perl  
script.

> And a last question: is support for raid1 a planned feature? I think
> that would be interesting as well. (If dm-raid is going to replace
> dm-mirror in the long run.)

Yes, RAID1 is planned; and works to a large extent.

  brassow

For convenience, I've attached the patches I'm working on (quilt  
directory) and the latest gime_raid.pl script.


[-- Attachment #2: dm-raid-patches.tgz --]
[-- Type: application/octet-stream, Size: 14456 bytes --]

[-- Attachment #3: Type: text/plain, Size: 4 bytes --]






[-- Attachment #4: Type: text/plain, Size: 0 bytes --]

Re: Improving dm-mirror as a final year project

[Miklos Vajna]

On Tue, Feb 15, 2011 at 04:13:15PM -0600, Jonathan Brassow <jbrassow@redhat.com> wrote:
> > Oh, do I read the code correctly that rhel6/upstream always reads from
> > the first mirror and switches only in case there is a read failure?
> 
> yes

Hm, was the was reason for dropping the round robin feature? I thought
that round robin causes better performance, given that IO reads are done
in an async way. Did I miss something or just nobody ported the patch to
RHEL6/upstream?

> Perhaps, but if you don't encode this in the LVM metadata, you will  
> have to perform the action every time you reboot.  Instead, you could  
> reorder the devices in userspace and reload the table.

I was not aware such a reorder is possible. That patch then does not
make too much sense, agreed. ;) (Do you have a pointer to some
documentation on how that reorder can be done? I can't find anything
about reorder in the lvconvert/dmsetup manpage.

> The basic component that covers RAID456 is available upstream, as you  
> saw.  I have an additional set of ~12 (reasonably small) patches that  
> add RAID1 and superblock/bitmap support.  These patches are not yet  
> upstream nor are they in any RHEL product.

Then what is the recommended platform to hack dm-raid? I have RHEL6 at
the moment. Is it OK to try to cherry-pick the single commit from
upstream + apply your patches or is it better to install rawhide where
the kernel is already 2.6.38rc5 (as far as I see) and only apply your
patches there?

> Yes, I have a script called 'gime_raid.pl' that creates the device- 
> mapper tables for dm-raid.  Eventually, this will be pushed into LVM,  
> but it was much easier (for testing purposes) to start with a perl  
> script.

Sure. :)

> For convenience, I've attached the patches I'm working on (quilt  
> directory) and the latest gime_raid.pl script.

Thanks!

Re: Improving dm-mirror as a final year project

[Miklos Vajna]

On Wed, Feb 16, 2011 at 06:12:19PM +0100, Miklos Vajna <vmiklos@ulx.hu> wrote:
> > The basic component that covers RAID456 is available upstream, as you  
> > saw.  I have an additional set of ~12 (reasonably small) patches that  
> > add RAID1 and superblock/bitmap support.  These patches are not yet  
> > upstream nor are they in any RHEL product.
> 
> Then what is the recommended platform to hack dm-raid? I have RHEL6 at
> the moment. Is it OK to try to cherry-pick the single commit from
> upstream + apply your patches or is it better to install rawhide where
> the kernel is already 2.6.38rc5 (as far as I see) and only apply your
> patches there?
>
> (...) 
> 
> > For convenience, I've attached the patches I'm working on (quilt  
> > directory) and the latest gime_raid.pl script.

I tried these on Fedora 15 with mixed results.

I prepared a kernel source tree:

----
$ wget http://download.fedora.redhat.com/pub/fedora/linux/development/15/source/SRPMS/kernel-2.6.38-0.rc4.git0.2.fc15.src.rpm
$ rpm -Uvh kernel-2.6.38-0.rc4.git0.2.fc15.src.rpm 2>&1 | grep -v mockb
$ cd ~/rpmbuild/SPECS
# yum install xmlto asciidoc elfutils-devel 'perl(ExtUtils::Embed)' # build-depends for kernel
$ rpmbuild -bp --target=`uname -m` kernel.spec 2>&1 | tee prep.log
----

Then continue the build manually:

----
$ cd ~/rpmbuild/BUILD/kernel-2.6.37.fc15/linux-2.6.37.i686
$ perl -p -i -e "s/^EXTRAVERSION.*/EXTRAVERSION = -0.rc4.git7.1.fc15.i686.PAE/" Makefile

$ cp /usr/src/kernels/2.6.38-0.rc4.git7.1.fc15.i686.PAE/Module.symvers .

$ make prepare
$ make modules_prepare
$ make M=drivers/md
# cp drivers/md/*.ko /lib/modules/2.6.38-0.rc4.git7.1.fc15.i686.PAE/extra/
# depmod -a
----

Apply the patches:

----
$ git quiltimport --author "Jonathan Brassow <jbrassow@redhat.com>" --patches ~/dm-raid-patches/
----

Build, install and reload the modules, then try it:

----
# ~vmiklos/dm-raid-patches/gime_raid.pl raid4 /dev/sd[bcdef]1
RAID type    : raid4
Block devices: /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
Device size  : 770868BB
# ls /dev/mapper/
control  raid  vg_diploma1f-lv_root  vg_diploma1f-lv_swap
----

No hanging! (It did before I applied your patches.)

However almost anything else fails. Here is what I tried:

(I reverted the virtual machine to a snapshot before each attempt.)

Creating a raid1:

----
# ~vmiklos/dm-raid-patches/gime_raid.pl raid1 /dev/sd[bc]1
RAID type    : raid1
Block devices: /dev/sdb1 /dev/sdc1
Device size  : 192717BB

Message from syslogd@diploma1-f at Feb 17 11:19:57 ...
 kernel:[ 1395.143044] Oops: 0000 [#1] SMP

(...)

sh: line 1:  4209 Done                    echo 0 192717 raid raid1 0 2 - /dev/sdb1 - /dev/sdc1
      4210 Killed                  | dmsetup create raid
Failed to create "raid":
  0 192717 raid raid1 0 2 - /dev/sdb1 - /dev/sdc1
----

dmesg:

----
[ 1269.709043] bio: create slab <bio-1> at 1
[ 1269.733414] md/raid1:mdX: not clean -- starting background reconstruction
[ 1269.734568] md/raid1:mdX: active with 2 out of 2 mirrors
[ 1269.752416] BUG: unable to handle kernel NULL pointer dereference at 000001f4
[ 1269.753039] IP: [<c070c3d0>] md_integrity_register+0x29/0xdc
[ 1269.753039] *pdpt = 000000000c77c001 *pde = 000000000f5be067 *pte = 0000000000000000 
[ 1269.753039] Oops: 0000 [#1] SMP 
[ 1269.753039] last sysfs file: /sys/module/raid1/initstate
[ 1269.753039] Modules linked in: dm_raid raid1 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer ppdev vmw_balloon snd pcnet32 mii soundcore snd_page_alloc i2c_piix4 parport_pc i2c_core parport ipv6 mptspi mptscsih mptbase scsi_transport_spi [last unloaded: raid456]
[ 1269.753039] 
[ 1269.753039] Pid: 4227, comm: dmsetup Not tainted 2.6.38-0.rc4.git7.1.fc15.i686.PAE #1 440BX Desktop Reference Platform/VMware Virtual Platform
[ 1269.753039] EIP: 0060:[<c070c3d0>] EFLAGS: 00010246 CPU: 0
[ 1269.753039] EIP is at md_integrity_register+0x29/0xdc
[ 1269.753039] EAX: 00000000 EBX: cf8dd398 ECX: 00000000 EDX: 00000000
[ 1269.753039] ESI: cf8dd00c EDI: 00000000 EBP: cc79dd4c ESP: cc79dd34
[ 1269.753039]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 1269.753039] Process dmsetup (pid: 4227, ti=cc79c000 task=cf8e25b0 task.ti=cc79c000)
[ 1269.753039] Stack:
[ 1269.753039]  cc780f68 cc780f68 cf8dd01c cf8dd00c cc780f68 0000000c cc79dd74 d10b7d46
[ 1269.753039]  d10ba952 d10ba775 00000002 00000002 000000e0 cf8dd01c cf8dd00c cf8dd01c
[ 1269.753039]  cc79ddfc c070f8cf 00000001 c098c29a 00000001 00000021 c040b453 cf8dd394
[ 1269.753039] Call Trace:
[ 1269.753039]  [<d10b7d46>] run+0x24c/0x256 [raid1]
[ 1269.753039]  [<c070f8cf>] md_run+0x4f0/0x7ab
[ 1269.753039]  [<c040b453>] ? do_softirq+0x8c/0x92
[ 1269.753039]  [<c0420d64>] ? smp_apic_timer_interrupt+0x6b/0x78
[ 1269.753039]  [<c0435ad9>] ? __might_sleep+0x29/0xe4
[ 1269.753039]  [<c042f5a4>] ? should_resched+0xd/0x27
[ 1269.753039]  [<c07e603e>] ? _cond_resched+0xd/0x21
[ 1269.753039]  [<d10f5ff7>] raid_ctr+0x8b4/0x8d4 [dm_raid]
[ 1269.753039]  [<c07177f5>] dm_table_add_target+0x165/0x202
[ 1269.753039]  [<c04d4f7f>] ? vfree+0x25/0x27
[ 1269.753039]  [<c071715b>] ? alloc_targets+0x8c/0xb1
[ 1269.753039]  [<c071a05c>] table_load+0x233/0x242
[ 1269.753039]  [<c0719aa8>] dm_ctl_ioctl+0x1af/0x1ed
[ 1269.753039]  [<c043474f>] ? pick_next_task_fair+0x85/0x8d
[ 1269.753039]  [<c0719e29>] ? table_load+0x0/0x242
[ 1269.753039]  [<c07198f9>] ? dm_ctl_ioctl+0x0/0x1ed
[ 1269.753039]  [<c04f9c4f>] do_vfs_ioctl+0x451/0x482
[ 1269.753039]  [<c0491ca0>] ? __call_rcu+0xdb/0xe1
[ 1269.753039]  [<c0501713>] ? mntput_no_expire+0x28/0xbd
[ 1269.753039]  [<c05017c6>] ? mntput+0x1e/0x20
[ 1269.753039]  [<c04f4f4f>] ? path_put+0x1a/0x1d
[ 1269.753039]  [<c04f9cc7>] sys_ioctl+0x47/0x60
[ 1269.753039]  [<c040969f>] sysenter_do_call+0x12/0x28
[ 1269.753039] Code: 5d c3 55 89 e5 57 56 53 83 ec 0c 3e 8d 74 26 00 89 c6 8b 5e 10 8d 40 10 89 45 f0 31 c0 3b 5d f0 0f 84 b0 00 00 00 8b 56 30 31 ff <83> ba f4 01 00 00 00 0f 85 9e 00 00 00 eb 38 8b 43 6c a8 02 75 
[ 1269.753039] EIP: [<c070c3d0>] md_integrity_register+0x29/0xdc SS:ESP 0068:cc79dd34
[ 1269.753039] CR2: 00000000000001f4
[ 1269.898378] ---[ end trace 49f34abab1d4a1b8 ]---
----

Creating and then deleting a raid4:

----
root@diploma1-f:~# ~vmiklos/dm-raid-patches/gime_raid.pl raid4 /dev/sd[bcdef]1
RAID type    : raid4
Block devices: /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
Device size  : 770868BB

root@diploma1-f:~# dmsetup remove raid
Killed
----

dmesg:

----
[ 1289.594670] bio: create slab <bio-1> at 1
[ 1289.617603] md/raid:mdX: not clean -- starting background reconstruction
[ 1289.623386] md/raid:mdX: device sdf1 operational as raid disk 4
[ 1289.634279] md/raid:mdX: device sde1 operational as raid disk 3
[ 1289.634889] md/raid:mdX: device sdd1 operational as raid disk 2
[ 1289.635479] md/raid:mdX: device sdc1 operational as raid disk 1
[ 1289.636071] md/raid:mdX: device sdb1 operational as raid disk 0
[ 1289.661945] md/raid:mdX: allocated 5265kB
[ 1289.663995] md/raid:mdX: raid level 5 active with 5 out of 5 devices, algorithm 4
[ 1289.665539] RAID conf printout:
[ 1289.665630]  --- level:5 rd:5 wd:5
[ 1289.665742]  disk 0, o:1, dev:sdb1
[ 1289.665759]  disk 1, o:1, dev:sdc1
[ 1289.665765]  disk 2, o:1, dev:sdd1
[ 1289.665770]  disk 3, o:1, dev:sde1
[ 1289.665776]  disk 4, o:1, dev:sdf1
[ 1289.685823] md: resync of RAID array mdX
[ 1289.686486] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 1289.687117] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 1289.701594] md: using 128k window, over a total of 96256 blocks.
[ 1289.892256] attempt to access beyond end of device
[ 1289.893552] sdc1: rw=0, want=193160, limit=192717
[ 1289.895278] attempt to access beyond end of device
[ 1289.895768] sdf1: rw=0, want=193160, limit=192717
[ 1289.896795] attempt to access beyond end of device
[ 1289.897354] sde1: rw=0, want=193160, limit=192717
[ 1289.897991] attempt to access beyond end of device
[ 1289.898565] sdd1: rw=0, want=193160, limit=192717
[ 1289.899524] attempt to access beyond end of device
[ 1289.900051] sdb1: rw=0, want=193160, limit=192717
[ 1289.901523] md/raid:mdX: Disk failure on sdf1, disabling device.
[ 1289.901528] md/raid:mdX: Operation continuing on 4 devices.
[ 1289.909029] md/raid:mdX: Disk failure on sde1, disabling device.
[ 1289.909033] md/raid:mdX: Operation continuing on 3 devices.
[ 1289.917651] md/raid:mdX: Disk failure on sdd1, disabling device.
[ 1289.917656] md/raid:mdX: Operation continuing on 2 devices.
[ 1289.918748] md/raid:mdX: Disk failure on sdc1, disabling device.
[ 1289.918752] md/raid:mdX: Operation continuing on 1 devices.
[ 1289.933445] md/raid:mdX: Disk failure on sdb1, disabling device.
[ 1289.933449] md/raid:mdX: Operation continuing on 0 devices.
[ 1289.936280] Buffer I/O error on device dm-2, logical block 192673
[ 1289.937184] Buffer I/O error on device dm-2, logical block 192672
[ 1289.954961] Buffer I/O error on device dm-2, logical block 192673
[ 1289.955619] Buffer I/O error on device dm-2, logical block 192672
[ 1289.956650] Buffer I/O error on device dm-2, logical block 192712
[ 1289.960572] Buffer I/O error on device dm-2, logical block 192713
[ 1289.961338] Buffer I/O error on device dm-2, logical block 192712
[ 1289.962174] Buffer I/O error on device dm-2, logical block 192713
[ 1289.963016] Buffer I/O error on device dm-2, logical block 0
[ 1289.977325] Buffer I/O error on device dm-2, logical block 1
[ 1290.812860] md: mdX: resync done.
[ 1290.813983] md: checkpointing resync of mdX.
[ 1290.822997] md: resync of RAID array mdX
[ 1290.831231] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 1290.832595] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 1290.833594] md: using 128k window, over a total of 96256 blocks.
[ 1290.834244] md: resuming resync of mdX from checkpoint.
[ 1290.834877] md: mdX: resync done.
[ 1290.850001] RAID conf printout:
[ 1290.850062]  --- level:5 rd:5 wd:0
[ 1290.850070]  disk 0, o:0, dev:sdb1
[ 1290.850076]  disk 1, o:0, dev:sdc1
[ 1290.850081]  disk 2, o:0, dev:sdd1
[ 1290.850086]  disk 3, o:0, dev:sde1
[ 1290.850091]  disk 4, o:0, dev:sdf1
[ 1293.215726] BUG: sleeping function called from invalid context at arch/x86/mm/fault.c:1081
[ 1293.216017] in_atomic(): 0, irqs_disabled(): 1, pid: 4250, name: dmsetup
[ 1293.216017] Pid: 4250, comm: dmsetup Not tainted 2.6.38-0.rc4.git7.1.fc15.i686.PAE #1
[ 1293.216017] Call Trace:
[ 1293.216017]  [<c0435b8d>] ? __might_sleep+0xdd/0xe4
[ 1293.216017]  [<c07ea191>] ? do_page_fault+0x179/0x30c
[ 1293.216017]  [<c0465112>] ? tick_dev_program_event+0x2f/0x137
[ 1293.216017]  [<c04e0b72>] ? kmem_cache_free+0x67/0x94
[ 1293.216017]  [<c0535dfd>] ? release_sysfs_dirent+0x79/0x8c
[ 1293.216017]  [<c07ea018>] ? do_page_fault+0x0/0x30c
[ 1293.216017]  [<c07e7d97>] ? error_code+0x67/0x6c
[ 1293.216017]  [<c07e726b>] ? _raw_spin_lock_irqsave+0x15/0x27
[ 1293.216017]  [<c05c7d92>] ? __disk_unblock_events+0x23/0x9e
[ 1293.216017]  [<c042f5a4>] ? should_resched+0xd/0x27
[ 1293.216017]  [<c05c91bc>] ? disk_unblock_events+0x1b/0x1d
[ 1293.216017]  [<c0511052>] ? blkdev_put+0xbb/0xe7
[ 1293.216017]  [<c0716f35>] ? close_dev+0x30/0x3a
[ 1293.216017]  [<c0716f61>] ? dm_put_device+0x22/0x33
[ 1293.216017]  [<d10f56e7>] ? context_free+0x58/0x73 [dm_raid]
[ 1293.216017]  [<d10f5740>] ? raid_dtr+0x3e/0x41 [dm_raid]
[ 1293.216017]  [<c0717550>] ? dm_table_destroy+0x5b/0xcf
[ 1293.216017]  [<c0469a28>] ? arch_local_irq_save+0x12/0x17
[ 1293.216017]  [<c0715ae3>] ? __dm_destroy+0xfa/0x1c2
[ 1293.216017]  [<c0716412>] ? dm_destroy+0x12/0x14
[ 1293.216017]  [<c071a146>] ? dev_remove+0xdb/0xe5
[ 1293.216017]  [<c05dae00>] ? copy_to_user+0x12/0x4b
[ 1293.216017]  [<c0719aa8>] ? dm_ctl_ioctl+0x1af/0x1ed
[ 1293.216017]  [<c058fae7>] ? newary+0x10a/0x11c
[ 1293.216017]  [<c071a06b>] ? dev_remove+0x0/0xe5
[ 1293.216017]  [<c07198f9>] ? dm_ctl_ioctl+0x0/0x1ed
[ 1293.216017]  [<c04f9c4f>] ? do_vfs_ioctl+0x451/0x482
[ 1293.216017]  [<c0491ca0>] ? __call_rcu+0xdb/0xe1
[ 1293.216017]  [<c0501713>] ? mntput_no_expire+0x28/0xbd
[ 1293.216017]  [<c05017c6>] ? mntput+0x1e/0x20
[ 1293.216017]  [<c04f4f4f>] ? path_put+0x1a/0x1d
[ 1293.216017]  [<c04f9cc7>] ? sys_ioctl+0x47/0x60
[ 1293.216017]  [<c040969f>] ? sysenter_do_call+0x12/0x28
[ 1293.216017] BUG: unable to handle kernel NULL pointer dereference at 0000080c
[ 1293.216017] IP: [<c07e726b>] _raw_spin_lock_irqsave+0x15/0x27
[ 1293.216017] *pdpt = 000000000f7cb001 *pde = 000000000c661067 *pte = 0000000000000000 
[ 1293.216017] Oops: 0002 [#1] SMP 
[ 1293.216017] last sysfs file: /sys/devices/virtual/block/dm-2/range
[ 1293.216017] Modules linked in: dm_raid raid1 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer ppdev vmw_balloon snd pcnet32 mii soundcore snd_page_alloc i2c_piix4 parport_pc i2c_core parport ipv6 mptspi mptscsih mptbase scsi_transport_spi [last unloaded: raid456]
[ 1293.216017] 
[ 1293.216017] Pid: 4250, comm: dmsetup Not tainted 2.6.38-0.rc4.git7.1.fc15.i686.PAE #1 440BX Desktop Reference Platform/VMware Virtual Platform
[ 1293.216017] EIP: 0060:[<c07e726b>] EFLAGS: 00010082 CPU: 0
[ 1293.216017] EIP is at _raw_spin_lock_irqsave+0x15/0x27
[ 1293.216017] EAX: 00000282 EBX: 0000080c ECX: 00000083 EDX: 00000100
[ 1293.216017] ESI: cf98be00 EDI: 00000001 EBP: cf5b9dfc ESP: cf5b9df8
[ 1293.216017]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 1293.216017] Process dmsetup (pid: 4250, ti=cf5b8000 task=cfa77110 task.ti=cf5b8000)
[ 1293.216017] Stack:
[ 1293.216017]  00000800 cf5b9e18 c05c7d92 c042f5a4 0000080c cbc31400 cbc31410 00000083
[ 1293.216017]  cf5b9e20 c05c91bc cf5b9e34 c0511052 cda488c0 cccc5000 00000000 cf5b9e40
[ 1293.216017]  c0716f35 cda488c0 cf5b9e4c c0716f61 cccc5000 cf5b9e60 d10f56e7 cccc5000
[ 1293.216017] Call Trace:
[ 1293.216017]  [<c05c7d92>] __disk_unblock_events+0x23/0x9e
[ 1293.216017]  [<c042f5a4>] ? should_resched+0xd/0x27
[ 1293.216017]  [<c05c91bc>] disk_unblock_events+0x1b/0x1d
[ 1293.216017]  [<c0511052>] blkdev_put+0xbb/0xe7
[ 1293.216017]  [<c0716f35>] close_dev+0x30/0x3a
[ 1293.216017]  [<c0716f61>] dm_put_device+0x22/0x33
[ 1293.216017]  [<d10f56e7>] context_free+0x58/0x73 [dm_raid]
[ 1293.216017]  [<d10f5740>] raid_dtr+0x3e/0x41 [dm_raid]
[ 1293.216017]  [<c0717550>] dm_table_destroy+0x5b/0xcf
[ 1293.216017]  [<c0469a28>] ? arch_local_irq_save+0x12/0x17
[ 1293.216017]  [<c0715ae3>] __dm_destroy+0xfa/0x1c2
[ 1293.216017]  [<c0716412>] dm_destroy+0x12/0x14
[ 1293.216017]  [<c071a146>] dev_remove+0xdb/0xe5
[ 1293.216017]  [<c05dae00>] ? copy_to_user+0x12/0x4b
[ 1293.216017]  [<c0719aa8>] dm_ctl_ioctl+0x1af/0x1ed
[ 1293.216017]  [<c058fae7>] ? newary+0x10a/0x11c
[ 1293.216017]  [<c071a06b>] ? dev_remove+0x0/0xe5
[ 1293.216017]  [<c07198f9>] ? dm_ctl_ioctl+0x0/0x1ed
[ 1293.216017]  [<c04f9c4f>] do_vfs_ioctl+0x451/0x482
[ 1293.216017]  [<c0491ca0>] ? __call_rcu+0xdb/0xe1
[ 1293.216017]  [<c0501713>] ? mntput_no_expire+0x28/0xbd
[ 1293.216017]  [<c05017c6>] ? mntput+0x1e/0x20
[ 1293.216017]  [<c04f4f4f>] ? path_put+0x1a/0x1d
[ 1293.216017]  [<c04f9cc7>] sys_ioctl+0x47/0x60
[ 1293.216017]  [<c040969f>] sysenter_do_call+0x12/0x28
[ 1293.216017] Code: 0f 95 c0 0f b6 c0 c3 55 89 e5 3e 8d 74 26 00 e8 06 28 c8 ff 5d c3 55 89 e5 53 3e 8d 74 26 00 89 c3 e8 b0 27 c8 ff ba 00 01 00 00 <3e> 66 0f c1 13 38 f2 74 06 f3 90 8a 13 eb f6 5b 5d c3 55 89 e5 
[ 1293.216017] EIP: [<c07e726b>] _raw_spin_lock_irqsave+0x15/0x27 SS:ESP 0068:cf5b9df8
[ 1293.216017] CR2: 000000000000080c
[ 1293.216017] ---[ end trace 49f34abab1d4a1b8 ]---
----

Creating a raid4 and creating a filesystem on it:

----
root@diploma1-f:~# ~vmiklos/dm-raid-patches/gime_raid.pl raid4 /dev/sd[bcdef]1
RAID type    : raid4
Block devices: /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
Device size  : 770868BB

root@diploma1-f:~# mkfs.ext4 /dev/mapper/raid 
mke2fs 1.41.14 (22-Dec-2010)
----

and it hangs here.

dmesg:

----
[ 1270.954599] bio: create slab <bio-1> at 1
[ 1270.963550] md/raid:mdX: not clean -- starting background reconstruction
[ 1270.987463] md/raid:mdX: device sdf1 operational as raid disk 4
[ 1270.989895] md/raid:mdX: device sde1 operational as raid disk 3
[ 1270.990513] md/raid:mdX: device sdd1 operational as raid disk 2
[ 1270.991101] md/raid:mdX: device sdc1 operational as raid disk 1
[ 1270.992075] md/raid:mdX: device sdb1 operational as raid disk 0
[ 1271.028680] md/raid:mdX: allocated 5265kB
[ 1271.047135] md/raid:mdX: raid level 5 active with 5 out of 5 devices, algorithm 4
[ 1271.048513] RAID conf printout:
[ 1271.048604]  --- level:5 rd:5 wd:5
[ 1271.048749]  disk 0, o:1, dev:sdb1
[ 1271.048768]  disk 1, o:1, dev:sdc1
[ 1271.048773]  disk 2, o:1, dev:sdd1
[ 1271.048779]  disk 3, o:1, dev:sde1
[ 1271.048784]  disk 4, o:1, dev:sdf1
[ 1271.079401] md: resync of RAID array mdX
[ 1271.080482] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 1271.081111] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 1271.082639] md: using 128k window, over a total of 96256 blocks.
[ 1271.261775] attempt to access beyond end of device
[ 1271.262542] sdc1: rw=0, want=193160, limit=192717
[ 1271.263924] attempt to access beyond end of device
[ 1271.264467] sdf1: rw=0, want=193160, limit=192717
[ 1271.265172] attempt to access beyond end of device
[ 1271.265747] sde1: rw=0, want=193160, limit=192717
[ 1271.266436] attempt to access beyond end of device
[ 1271.266946] sdd1: rw=0, want=193160, limit=192717
[ 1271.267872] attempt to access beyond end of device
[ 1271.281551] sdb1: rw=0, want=193160, limit=192717
[ 1271.282992] md/raid:mdX: Disk failure on sdf1, disabling device.
[ 1271.282996] md/raid:mdX: Operation continuing on 4 devices.
[ 1271.284807] md/raid:mdX: Disk failure on sde1, disabling device.
[ 1271.284811] md/raid:mdX: Operation continuing on 3 devices.
[ 1271.285962] md/raid:mdX: Disk failure on sdd1, disabling device.
[ 1271.285966] md/raid:mdX: Operation continuing on 2 devices.
[ 1271.300746] md/raid:mdX: Disk failure on sdc1, disabling device.
[ 1271.300750] md/raid:mdX: Operation continuing on 1 devices.
[ 1271.302077] md/raid:mdX: Disk failure on sdb1, disabling device.
[ 1271.302081] md/raid:mdX: Operation continuing on 0 devices.
[ 1271.322206] Buffer I/O error on device dm-2, logical block 192673
[ 1271.323102] Buffer I/O error on device dm-2, logical block 192672
[ 1271.326465] Buffer I/O error on device dm-2, logical block 192672
[ 1271.338964] Buffer I/O error on device dm-2, logical block 192673
[ 1271.339820] Buffer I/O error on device dm-2, logical block 192712
[ 1271.340476] Buffer I/O error on device dm-2, logical block 192713
[ 1271.341118] Buffer I/O error on device dm-2, logical block 192712
[ 1271.341700] Buffer I/O error on device dm-2, logical block 192713
[ 1271.356296] Buffer I/O error on device dm-2, logical block 0
[ 1271.356867] Buffer I/O error on device dm-2, logical block 1
[ 1272.188477] md: mdX: resync done.
[ 1272.189634] md: checkpointing resync of mdX.
[ 1272.192828] md: resync of RAID array mdX
[ 1272.193363] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 1272.194090] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 1272.195083] md: using 128k window, over a total of 96256 blocks.
[ 1272.195723] md: resuming resync of mdX from checkpoint.
[ 1272.209475] md: mdX: resync done.
[ 1272.210697] RAID conf printout:
[ 1272.210709]  --- level:5 rd:5 wd:0
[ 1272.210716]  disk 0, o:0, dev:sdb1
[ 1272.210721]  disk 1, o:0, dev:sdc1
[ 1272.210726]  disk 2, o:0, dev:sdd1
[ 1272.210731]  disk 3, o:0, dev:sde1
[ 1272.210736]  disk 4, o:0, dev:sdf1
----

Are these expected or the above works for you?

Thanks.

Re: Improving dm-mirror as a final year project

[Jonathan Brassow]

On Feb 17, 2011, at 4:49 AM, Miklos Vajna wrote:

> On Wed, Feb 16, 2011 at 06:12:19PM +0100, Miklos Vajna  
> <vmiklos@ulx.hu> wrote:
>>> The basic component that covers RAID456 is available upstream, as  
>>> you
>>> saw.  I have an additional set of ~12 (reasonably small) patches  
>>> that
>>> add RAID1 and superblock/bitmap support.  These patches are not yet
>>> upstream nor are they in any RHEL product.
>>
>> Then what is the recommended platform to hack dm-raid? I have RHEL6  
>> at
>> the moment. Is it OK to try to cherry-pick the single commit from
>> upstream + apply your patches or is it better to install rawhide  
>> where
>> the kernel is already 2.6.38rc5 (as far as I see) and only apply your
>> patches there?

I tend to use RHEL6 + the latest upstream kernel.  I would /not/  
cherry-pick the upstream dm-raid patch and then add my next set - you  
might miss some other valuable MD patches (there have been a couple).

>>
>>> For convenience, I've attached the patches I'm working on (quilt
>>> directory) and the latest gime_raid.pl script.
>
> I tried these on Fedora 15 with mixed results.
>
> I prepared a kernel source tree:
>
> ----
> $ wget http://download.fedora.redhat.com/pub/fedora/linux/development/15/source/SRPMS/kernel-2.6.38-0.rc4.git0.2.fc15.src.rpm

Please use upstream (kernel.org) kernel.

Contact me with any trouble.

  brassow

Improving dm-mirror as a final year project

[Miklos Vajna]

Hi,

I got the possibility to work on dm-mirror as a final year project at
ULX, the Hungarian distributor of Red Hat.

Get my feet wet, I created two small patches:

1) dm-mirror: allow setting ios count to 0

Always read from the default_mirror in that case.

2) dm-mirror: allow setting the default mirror

These can help in case one data leg of a mirror is a remote (iSCSI) one,
so the default RR aproach is not optimal for reading. (One may set the
ios count to 0, set the default mirror to the local one, and that will
cause a read speedup.)

I do not yet have the right to send those patches (I do this in
university time, so the copyright is not mine), but I hope to be able to
do so - to get them reviewed.

So the final year project's aim is to improve "the fault tolerance and
performance of dm-mirror". We (I and my mentors) have two ideas in that
area, not counting the above patches:

1) Make the currently hardwired RR read approach modular, that would
allow implementing for example a weighted RR algorithm. (In case one
disk is two times faster than the other one, etc.)

2) From our experiments, it seems that in case the dm-mirror looses one
of its legs and there is a write to the mirror, it gets converted to a
linear volume. It would be nice (not sure how easy) to use the mirror
log to mark the dirty blocks, so that the volume would not be converted
to a linear one: once the other leg is back, it could be updated based
on the mirror log.

The question: what do you (who have much more experience with dm-mirror
than me) think - are these reasonable goals? If not, what would you
improve/change/add/remove to the above list?

Thanks,

Miklos

PS: Sorry if this is a duplicate, first I did not subscribe to the list.