[PATCH] MD: Quickly return errors if too many devices have failed.

[PATCH] MD: Quickly return errors if too many devices have failed.

[Jonathan Brassow]

Neil,

I've noticed that when too many devices fail in a RAID arrary that
addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
Mar 12 11:52:53 bp-01 kernel: sector=800 i=3           (null)           (null)  
         (null)           (null) 1
Mar 12 11:52:53 bp-01 kernel: ------------[ cut here ]------------
Mar 12 11:52:53 bp-01 kernel: WARNING: at drivers/md/raid5.c:354 init_stripe+0x2d4/0x370 [raid456]()
Mar 12 11:52:53 bp-01 kernel: Hardware name: PowerEdge R415
Mar 12 11:52:53 bp-01 kernel: Modules linked in: raid456 md_mod async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx sunrpc ipv6 dcdbas freq_table mperf kvm_amd kvm crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw fam15h_power k10temp amd64_edac_mod edac_core edac_mce_amd i2c_piix4 i2c_core bnx2 sg ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul pata_acpi ata_generic pata_atiixp ahci libahci mptsas mptscsih mptbase scsi_transport_sas bfa scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
Mar 12 11:52:53 bp-01 kernel: Pid: 8604, comm: dd Not tainted 3.8.0 #8
Mar 12 11:52:53 bp-01 kernel: Call Trace:
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81056dcf>] warn_slowpath_common+0x7f/0xc0
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81056e2a>] warn_slowpath_null+0x1a/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffffa040f854>] init_stripe+0x2d4/0x370 [raid456]
Mar 12 11:52:53 bp-01 kernel: [<ffffffffa040fc45>] get_active_stripe+0x355/0x3f0 [raid456]
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8107c050>] ? wake_up_bit+0x40/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffffa0413c66>] make_request+0x1a6/0x430 [raid456]
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8107c050>] ? wake_up_bit+0x40/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffffa03f0123>] md_make_request+0xd3/0x200 [md_mod]
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81122cd5>] ? mempool_alloc_slab+0x15/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81122e40>] ? mempool_alloc+0x60/0x170
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8124a8ba>] generic_make_request+0xca/0x100
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8124a969>] submit_bio+0x79/0x160
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b4825>] ? bio_alloc_bioset+0x65/0x120
Mar 12 11:52:53 bp-01 kernel: [<ffffffff815458b2>] ? _raw_spin_unlock_irqrestore+0x12/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811af318>] submit_bh+0x128/0x200
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b1dd0>] __block_write_full_page+0x1e0/0x330
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b06b0>] ? lock_buffer+0x30/0x30
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b5920>] ? I_BDEV+0x10/0x10
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b5920>] ? I_BDEV+0x10/0x10
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b2055>] block_write_full_page+0x15/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b6988>] blkdev_writepage+0x18/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112a5f7>] __writepage+0x17/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112b875>] write_cache_pages+0x245/0x520
Mar 12 11:52:53 bp-01 kernel: [<ffffffff810844c9>] ? __wake_up_common+0x59/0x90
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112a5e0>] ? set_page_dirty+0x60/0x60
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112bba1>] generic_writepages+0x51/0x80
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811bd8bf>] ? send_to_group+0x13f/0x200
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112bbf0>] do_writepages+0x20/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811210f1>] __filemap_fdatawrite_range+0x51/0x60
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112133f>] filemap_fdatawrite+0x1f/0x30
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81121385>] filemap_write_and_wait+0x35/0x60
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b6ca1>] __sync_blockdev+0x21/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b6cd3>] sync_blockdev+0x13/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b6ed9>] __blkdev_put+0x69/0x1d0
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b7096>] blkdev_put+0x56/0x140
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b71a4>] blkdev_close+0x24/0x30
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8117f7bb>] __fput+0xbb/0x260
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8117f9ce>] ____fput+0xe/0x10
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81077e5f>] task_work_run+0x8f/0xf0
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81014a54>] do_notify_resume+0x84/0x90
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8154e992>] int_signal+0x12/0x17
Mar 12 11:52:53 bp-01 kernel: ---[ end trace d71de83816e8d215 ]---

Are other people seeing this, or is this an artifact of the way I am killing
devices ('echo offline > /sys/block/$dev/device/state')?

I would prefer to get immediate errors if nothing can be done to satisfy the
request and I've been thinking of something like the attached patch.  The
patch below is incomplete.  It does not take into account any reshaping that
is going on, nor does it try to figure out if a mirror set in RAID10 has died;
but I hope it gets the basic idea across.

Is this a good way to handle this situation, or am I missing something?

 brassow

<no comments: POC>

Signed-off-by: Jonathan Brassow <jbrassow@redhat.com>


Index: linux-upstream/drivers/md/raid1.c
===================================================================
--- linux-upstream.orig/drivers/md/raid1.c
+++ linux-upstream/drivers/md/raid1.c
@@ -210,6 +210,17 @@ static void put_buf(struct r1bio *r1_bio
 	lower_barrier(conf);
 }
 
+static int has_failed(struct r1conf *conf) {
+	struct mddev *mddev = conf->mddev;
+	struct md_rdev *rdev;
+
+	rdev_for_each(rdev, mddev)
+		if (likely(!test_bit(Faulty, &rdev->flags)))
+			return 0;
+
+	return 1;
+}
+
 static void reschedule_retry(struct r1bio *r1_bio)
 {
 	unsigned long flags;
@@ -1007,6 +1018,11 @@ static void make_request(struct mddev *m
 	int sectors_handled;
 	int max_sectors;
 
+	if (has_failed(conf)) {
+		bio_endio(bio, -EIO);
+		return;
+	}
+
 	/*
 	 * Register the new request and wait if the reconstruction
 	 * thread has put up a bar for new requests.
Index: linux-upstream/drivers/md/raid10.c
===================================================================
--- linux-upstream.orig/drivers/md/raid10.c
+++ linux-upstream/drivers/md/raid10.c
@@ -270,6 +270,18 @@ static void put_buf(struct r10bio *r10_b
 	lower_barrier(conf);
 }
 
+static int has_failed(struct r10conf *conf) {
+	struct mddev *mddev = conf->mddev;
+	struct md_rdev *rdev;
+
+	rdev_for_each(rdev, mddev)
+		if (likely(!test_bit(Faulty, &rdev->flags)))
+			return 0;
+
+	return 1;
+
+}
+
 static void reschedule_retry(struct r10bio *r10_bio)
 {
 	unsigned long flags;
@@ -1159,6 +1171,11 @@ static void make_request(struct mddev *m
 	int max_sectors;
 	int sectors;
 
+	if (has_failed(conf)) {
+		bio_endio(bio, -EIO);
+		return;
+	}
+
 	if (unlikely(bio->bi_rw & REQ_FLUSH)) {
 		md_flush_request(mddev, bio);
 		return;
Index: linux-upstream/drivers/md/raid5.c
===================================================================
--- linux-upstream.orig/drivers/md/raid5.c
+++ linux-upstream/drivers/md/raid5.c
@@ -4241,6 +4241,11 @@ static void make_request(struct mddev *m
 	const int rw = bio_data_dir(bi);
 	int remaining;
 
+	if (has_failed(conf)) {
+		bio_endio(bi, -EIO);
+		return;
+	}
+
 	if (unlikely(bi->bi_rw & REQ_FLUSH)) {
 		md_flush_request(mddev, bi);
 		return;

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2158 bytes --]

On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <jbrassow@redhat.com>
wrote:

> Neil,
> 
> I've noticed that when too many devices fail in a RAID arrary that
> addtional I/O will hang, yielding an endless supply of:
> Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
> Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
> Mar 12 11:52:53 bp-01 kernel: sector=800 i=3           (null)           (null)  
>          (null)           (null) 1

This is the third report in as many weeks that mentions that WARN_ON.
The first two where quite different causes.
I think this one is the same as the first one, which means it would be fixed
by  
      md/raid5: schedule_construction should abort if nothing to do.

which is commit 29d90fa2adbdd9f in linux-next.

> Mar 12 11:52:53 bp-01 kernel: ------------[ cut here ]------------
> Mar 12 11:52:53 bp-01 kernel: WARNING: at drivers/md/raid5.c:354 init_stripe+0x2d4/0x370 [raid456]()

> 
> Are other people seeing this, or is this an artifact of the way I am killing
> devices ('echo offline > /sys/block/$dev/device/state')?

That is a perfectly good way to kill a device.

> 
> I would prefer to get immediate errors if nothing can be done to satisfy the
> request and I've been thinking of something like the attached patch.  The
> patch below is incomplete.  It does not take into account any reshaping that
> is going on, nor does it try to figure out if a mirror set in RAID10 has died;
> but I hope it gets the basic idea across.
> 
> Is this a good way to handle this situation, or am I missing something?

I think we do get immediate errors (once all bugs are fixed).
Your patch does extra work for every request which is only of value if the
array has failed - and it really doesn't make sense to optimise for a failed
array.
The current approach is to just try to satisfy a request and once we find
that we need to do something that is impossible - return an error at that
point.  I think that is best.

Can you try the commit I identified and see if it makes the problem go away?

Thanks,
NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[Brassow Jonathan]

On Mar 17, 2013, at 6:49 PM, NeilBrown wrote:

>> 
>> 
>> I would prefer to get immediate errors if nothing can be done to satisfy the
>> request and I've been thinking of something like the attached patch.  The
>> patch below is incomplete.  It does not take into account any reshaping that
>> is going on, nor does it try to figure out if a mirror set in RAID10 has died;
>> but I hope it gets the basic idea across.
>> 
>> Is this a good way to handle this situation, or am I missing something?
> 
> I think we do get immediate errors (once all bugs are fixed).
> Your patch does extra work for every request which is only of value if the
> array has failed - and it really doesn't make sense to optimise for a failed
> array.
> The current approach is to just try to satisfy a request and once we find
> that we need to do something that is impossible - return an error at that
> point.  I think that is best.
> 
> Can you try the commit I identified and see if it makes the problem go away?

Yes, that sounds better.  I will try with the commit you mentioned.

thanks,
 brassow

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[Roy Sigurd Karlsbakk]

> > Are other people seeing this, or is this an artifact of the way I am
> > killing devices ('echo offline > /sys/block/$dev/device/state')?
> 
> That is a perfectly good way to kill a device.

Anyone that knows what may have replaced this in 3.8, or if this isn't available for virtio devices? I have a vm on 3.8 (ubuntu raring alpha/beta/something), and I find no such file there. Works great on the host OS, though (ubuntu precise, linux 3.2).

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[Brassow Jonathan]

On Mar 18, 2013, at 12:31 PM, Roy Sigurd Karlsbakk wrote:

>>> Are other people seeing this, or is this an artifact of the way I am
>>> killing devices ('echo offline > /sys/block/$dev/device/state')?
>> 
>> That is a perfectly good way to kill a device.
> 
> Anyone that knows what may have replaced this in 3.8, or if this isn't available for virtio devices? I have a vm on 3.8 (ubuntu raring alpha/beta/something), and I find no such file there. Works great on the host OS, though (ubuntu precise, linux 3.2).

I'm using this specifically on 3.8+.  I'm not using VMs at the moment, but in the past it has been available there - not for partitions and loopback devices though.

 brassow

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[Brassow Jonathan]

On Mar 17, 2013, at 6:49 PM, NeilBrown wrote:

> On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <jbrassow@redhat.com>
> wrote:
> 
>> Neil,
>> 
>> I've noticed that when too many devices fail in a RAID arrary that
>> addtional I/O will hang, yielding an endless supply of:
>> Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
>> Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
>> Mar 12 11:52:53 bp-01 kernel: sector=800 i=3           (null)           (null)  
>>         (null)           (null) 1
> 
> This is the third report in as many weeks that mentions that WARN_ON.
> The first two where quite different causes.
> I think this one is the same as the first one, which means it would be fixed
> by  
>      md/raid5: schedule_construction should abort if nothing to do.
> 
> which is commit 29d90fa2adbdd9f in linux-next.

Sorry, I don't see this commit in linux-next:
(the "for-next" branch of) git://github.com/neilbrown/linux.git
or git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git

Where should I be looking?

I did grab a patch from an earlier discussion where you mentioned a similar commit ID.  It didn't solve the problem, but it did prevent an endless progression of the same error messages.  I only saw one instance of the above after the patch.

I'm fairly certain that the hang was affecting more than just RAID5 though.  It also happened with raid1/10.  I'll go back with 3.9.0-rc3 and make sure that's true until I can figure out which 'linux-next' commit you are talking about.

thanks,
 brassow

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]

On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <jbrassow@redhat.com>
wrote:

> 
> On Mar 17, 2013, at 6:49 PM, NeilBrown wrote:
> 
> > On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <jbrassow@redhat.com>
> > wrote:
> > 
> >> Neil,
> >> 
> >> I've noticed that when too many devices fail in a RAID arrary that
> >> addtional I/O will hang, yielding an endless supply of:
> >> Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
> >> Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
> >> Mar 12 11:52:53 bp-01 kernel: sector=800 i=3           (null)           (null)  
> >>         (null)           (null) 1
> > 
> > This is the third report in as many weeks that mentions that WARN_ON.
> > The first two where quite different causes.
> > I think this one is the same as the first one, which means it would be fixed
> > by  
> >      md/raid5: schedule_construction should abort if nothing to do.
> > 
> > which is commit 29d90fa2adbdd9f in linux-next.
> 
> Sorry, I don't see this commit in linux-next:
> (the "for-next" branch of) git://github.com/neilbrown/linux.git
> or git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> 
> Where should I be looking?

Sorry, I probably messed up.
I meant this commit:
 http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8ca


> 
> I did grab a patch from an earlier discussion where you mentioned a similar commit ID.  It didn't solve the problem, but it did prevent an endless progression of the same error messages.  I only saw one instance of the above after the patch.
> 
> I'm fairly certain that the hang was affecting more than just RAID5 though.  It also happened with raid1/10.  I'll go back with 3.9.0-rc3 and make sure that's true until I can figure out which 'linux-next' commit you are talking about.

If you could get something concrete I'd love to hear about it, thanks.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[Brassow Jonathan]

On Mar 19, 2013, at 9:46 PM, NeilBrown wrote:

> On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <jbrassow@redhat.com>
> wrote:
> 
>> 
>> On Mar 17, 2013, at 6:49 PM, NeilBrown wrote:
>> 
>>> On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <jbrassow@redhat.com>
>>> wrote:
>>> 
>>>> Neil,
>>>> 
>>>> I've noticed that when too many devices fail in a RAID arrary that
>>>> addtional I/O will hang, yielding an endless supply of:
>>>> Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
>>>> Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
>>>> Mar 12 11:52:53 bp-01 kernel: sector=800 i=3           (null)           (null)  
>>>>        (null)           (null) 1
>>> 
>>> This is the third report in as many weeks that mentions that WARN_ON.
>>> The first two where quite different causes.
>>> I think this one is the same as the first one, which means it would be fixed
>>> by  
>>>     md/raid5: schedule_construction should abort if nothing to do.
>>> 
>>> which is commit 29d90fa2adbdd9f in linux-next.
>> 
>> Sorry, I don't see this commit in linux-next:
>> (the "for-next" branch of) git://github.com/neilbrown/linux.git
>> or git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
>> 
>> Where should I be looking?
> 
> Sorry, I probably messed up.
> I meant this commit:
> http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8ca

Yes, I found this patch in 'for-next'.  I tested 3.9.0-rc3 with and without this patch.  The good news is that my issue with RAID5 appears to be fixed with this patch.  To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size).  Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write.  After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write.  That's good.  Will this patch make it back to 3.[78]?

However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min.  Hung task messages with call traces and many many errors are the result.  This is bad.  I haven't figured out why these are so slow yet.

On a different topic, I've noticed the following commits in 'for-next':
  90584fc MD: Prevent sysfs operations on uninitialized kobjects
  e3620a3 MD RAID5: Avoid accessing gendisk or queue structs when not available
but these are not in 3.9.0-rc3.  They should make their way into 3.9.0 as well as 3.8.0.  (They apply cleanly to the 3.8 kernel, but I hadn't bothered to notify 'stable' - only mention the regression was introduced in 3.8-rc1.)

Thanks,
 brassow

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3704 bytes --]

On Wed, 20 Mar 2013 15:56:03 -0500 Brassow Jonathan <jbrassow@redhat.com>
wrote:

> 
> On Mar 19, 2013, at 9:46 PM, NeilBrown wrote:
> 
> > On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <jbrassow@redhat.com>
> > wrote:
> > 
> >> 
> >> On Mar 17, 2013, at 6:49 PM, NeilBrown wrote:
> >> 
> >>> On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <jbrassow@redhat.com>
> >>> wrote:
> >>> 
> >>>> Neil,
> >>>> 
> >>>> I've noticed that when too many devices fail in a RAID arrary that
> >>>> addtional I/O will hang, yielding an endless supply of:
> >>>> Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
> >>>> Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
> >>>> Mar 12 11:52:53 bp-01 kernel: sector=800 i=3           (null)           (null)  
> >>>>        (null)           (null) 1
> >>> 
> >>> This is the third report in as many weeks that mentions that WARN_ON.
> >>> The first two where quite different causes.
> >>> I think this one is the same as the first one, which means it would be fixed
> >>> by  
> >>>     md/raid5: schedule_construction should abort if nothing to do.
> >>> 
> >>> which is commit 29d90fa2adbdd9f in linux-next.
> >> 
> >> Sorry, I don't see this commit in linux-next:
> >> (the "for-next" branch of) git://github.com/neilbrown/linux.git
> >> or git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> >> 
> >> Where should I be looking?
> > 
> > Sorry, I probably messed up.
> > I meant this commit:
> > http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8ca
> 
> Yes, I found this patch in 'for-next'.  I tested 3.9.0-rc3 with and without this patch.  The good news is that my issue with RAID5 appears to be fixed with this patch.  To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size).  Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write.  After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write.  That's good.  Will this patch make it back to 3.[78]?
> 
> However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min.  Hung task messages with call traces and many many errors are the result.  This is bad.  I haven't figured out why these are so slow yet.

What happens if you take RAID out of the picture?
i.e. write to a single device, then "kill" that device, then try issuing a
40M write request to it.

If that takes 2.5 minutes to resolve, then I think it is correct for RAID1 to
also take 2.5 minutes to resolve. 
If it resolves much more quickly than it does with RAID1, then that is a
problem we should certainly address.


> 
> On a different topic, I've noticed the following commits in 'for-next':
>   90584fc MD: Prevent sysfs operations on uninitialized kobjects
>   e3620a3 MD RAID5: Avoid accessing gendisk or queue structs when not available
> but these are not in 3.9.0-rc3.  They should make their way into 3.9.0 as well as 3.8.0.  (They apply cleanly to the 3.8 kernel, but I hadn't bothered to notify 'stable' - only mention the regression was introduced in 3.8-rc1.)

They are due to be sent to Linus today.
The second one is  tagged for -stable and will go to 3.8.x (I think 3.7.x is
closed).
The first isn't - my quick examination suggested that the current code is
safe but not ideal.  If you think it is appropriate for -stable  (i.e. fixes
a real bug that could hurt users) let me know why and I'll make sure it goes
to -stable.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[Brassow Jonathan]

On Mar 20, 2013, at 6:04 PM, NeilBrown wrote:

> On Wed, 20 Mar 2013 15:56:03 -0500 Brassow Jonathan <jbrassow@redhat.com>
> wrote:
> 
>> 
>> On Mar 19, 2013, at 9:46 PM, NeilBrown wrote:
>> 
>>> On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <jbrassow@redhat.com>
>>> wrote:
>>> 
>>>> 
>>>> On Mar 17, 2013, at 6:49 PM, NeilBrown wrote:
>>>> 
>>>>> On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <jbrassow@redhat.com>
>>>>> wrote:
>>>>> 
>>>>>> Neil,
>>>>>> 
>>>>>> I've noticed that when too many devices fail in a RAID arrary that
>>>>>> addtional I/O will hang, yielding an endless supply of:
>>>>>> Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
>>>>>> Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
>>>>>> Mar 12 11:52:53 bp-01 kernel: sector=800 i=3           (null)           (null)  
>>>>>>       (null)           (null) 1
>>>>> 
>>>>> This is the third report in as many weeks that mentions that WARN_ON.
>>>>> The first two where quite different causes.
>>>>> I think this one is the same as the first one, which means it would be fixed
>>>>> by  
>>>>>    md/raid5: schedule_construction should abort if nothing to do.
>>>>> 
>>>>> which is commit 29d90fa2adbdd9f in linux-next.
>>>> 
>>>> Sorry, I don't see this commit in linux-next:
>>>> (the "for-next" branch of) git://github.com/neilbrown/linux.git
>>>> or git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
>>>> 
>>>> Where should I be looking?
>>> 
>>> Sorry, I probably messed up.
>>> I meant this commit:
>>> http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8ca
>> 
>> Yes, I found this patch in 'for-next'.  I tested 3.9.0-rc3 with and without this patch.  The good news is that my issue with RAID5 appears to be fixed with this patch.  To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size).  Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write.  After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write.  That's good.  Will this patch make it back to 3.[78]?
>> 
>> However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min.  Hung task messages with call traces and many many errors are the result.  This is bad.  I haven't figured out why these are so slow yet.
> 
> What happens if you take RAID out of the picture?
> i.e. write to a single device, then "kill" that device, then try issuing a
> 40M write request to it.
> 
> If that takes 2.5 minutes to resolve, then I think it is correct for RAID1 to
> also take 2.5 minutes to resolve. 
> If it resolves much more quickly than it does with RAID1, then that is a
> problem we should certainly address.

The test is a little different because once you offline a device, you can't open it.  So, I had to start I/O and then kill the device.  I still get 158MB/s - 3 orders of magnitude faster than RAID1.  Besides, if RAID10 takes 9+ minutes to complete, we'd still have something to fix.  I have also tested this with an "error" device and it also returns in sub-second time.

 brassow

[root@bp-01 ~]# off.sh sda
Turning off sda
[root@bp-01 ~]# dd if=/dev/zero of=/dev/sda1 bs=4M count=10
dd: opening `/dev/sda1': No such device or address
[root@bp-01 ~]# on.sh sda
Turning on sda
[root@bp-01 ~]# dd if=/dev/zero of=/dev/sda bs=4M count=1000 &
[1] 5203
[root@bp-01 ~]# off.sh sda
Turning off sda
[root@bp-01 ~]# 1000+0 records in
1000+0 records out
4194304000 bytes (4.2 GB) copied, 26.5564 s, 158 MB/s

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[Brassow Jonathan]

On Mar 20, 2013, at 6:04 PM, NeilBrown wrote:

>> 
>> On a different topic, I've noticed the following commits in 'for-next':
>>  90584fc MD: Prevent sysfs operations on uninitialized kobjects
>>  e3620a3 MD RAID5: Avoid accessing gendisk or queue structs when not available
>> but these are not in 3.9.0-rc3.  They should make their way into 3.9.0 as well as 3.8.0.  (They apply cleanly to the 3.8 kernel, but I hadn't bothered to notify 'stable' - only mention the regression was introduced in 3.8-rc1.)
> 
> They are due to be sent to Linus today.
> The second one is  tagged for -stable and will go to 3.8.x (I think 3.7.x is
> closed).
> The first isn't - my quick examination suggested that the current code is
> safe but not ideal.  If you think it is appropriate for -stable  (i.e. fixes
> a real bug that could hurt users) let me know why and I'll make sure it goes
> to -stable.

Yes, the second is the important one.  We don't need the first backported to stable.

Thanks!
 brassow

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 4519 bytes --]

On Thu, 21 Mar 2013 08:58:54 -0500 Brassow Jonathan <jbrassow@redhat.com>
wrote:

> 
> On Mar 20, 2013, at 6:04 PM, NeilBrown wrote:
> 
> > On Wed, 20 Mar 2013 15:56:03 -0500 Brassow Jonathan <jbrassow@redhat.com>
> > wrote:
> > 
> >> 
> >> On Mar 19, 2013, at 9:46 PM, NeilBrown wrote:
> >> 
> >>> On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <jbrassow@redhat.com>
> >>> wrote:
> >>> 
> >>>> 
> >>>> On Mar 17, 2013, at 6:49 PM, NeilBrown wrote:
> >>>> 
> >>>>> On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <jbrassow@redhat.com>
> >>>>> wrote:
> >>>>> 
> >>>>>> Neil,
> >>>>>> 
> >>>>>> I've noticed that when too many devices fail in a RAID arrary that
> >>>>>> addtional I/O will hang, yielding an endless supply of:
> >>>>>> Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
> >>>>>> Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
> >>>>>> Mar 12 11:52:53 bp-01 kernel: sector=800 i=3           (null)           (null)  
> >>>>>>       (null)           (null) 1
> >>>>> 
> >>>>> This is the third report in as many weeks that mentions that WARN_ON.
> >>>>> The first two where quite different causes.
> >>>>> I think this one is the same as the first one, which means it would be fixed
> >>>>> by  
> >>>>>    md/raid5: schedule_construction should abort if nothing to do.
> >>>>> 
> >>>>> which is commit 29d90fa2adbdd9f in linux-next.
> >>>> 
> >>>> Sorry, I don't see this commit in linux-next:
> >>>> (the "for-next" branch of) git://github.com/neilbrown/linux.git
> >>>> or git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> >>>> 
> >>>> Where should I be looking?
> >>> 
> >>> Sorry, I probably messed up.
> >>> I meant this commit:
> >>> http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8ca
> >> 
> >> Yes, I found this patch in 'for-next'.  I tested 3.9.0-rc3 with and without this patch.  The good news is that my issue with RAID5 appears to be fixed with this patch.  To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size).  Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write.  After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write.  That's good.  Will this patch make it back to 3.[78]?
> >> 
> >> However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min.  Hung task messages with call traces and many many errors are the result.  This is bad.  I haven't figured out why these are so slow yet.
> > 
> > What happens if you take RAID out of the picture?
> > i.e. write to a single device, then "kill" that device, then try issuing a
> > 40M write request to it.
> > 
> > If that takes 2.5 minutes to resolve, then I think it is correct for RAID1 to
> > also take 2.5 minutes to resolve. 
> > If it resolves much more quickly than it does with RAID1, then that is a
> > problem we should certainly address.
> 
> The test is a little different because once you offline a device, you can't open it.  So, I had to start I/O and then kill the device.  I still get 158MB/s - 3 orders of magnitude faster than RAID1.  Besides, if RAID10 takes 9+ minutes to complete, we'd still have something to fix.  I have also tested this with an "error" device and it also returns in sub-second time.
> 
>  brassow
> 
> [root@bp-01 ~]# off.sh sda
> Turning off sda
> [root@bp-01 ~]# dd if=/dev/zero of=/dev/sda1 bs=4M count=10
> dd: opening `/dev/sda1': No such device or address
> [root@bp-01 ~]# on.sh sda
> Turning on sda
> [root@bp-01 ~]# dd if=/dev/zero of=/dev/sda bs=4M count=1000 &
> [1] 5203
> [root@bp-01 ~]# off.sh sda
> Turning off sda
> [root@bp-01 ~]# 1000+0 records in
> 1000+0 records out
> 4194304000 bytes (4.2 GB) copied, 26.5564 s, 158 MB/s
> 

Maybe if you could show me some/all of the error messages that you get during
these long delays it might help.  Also the error messages you (presumably)
got from the kernel from the above plain-disk test.

It should quickly fail all but one copy of the data, then try writing to that
copy exactly the same way that it would write to a plain disk.

For RAID10 large writes have to be chopped up for striping, so the extra
requests which all have to fail could be the reason for the extra delay with
RAID10.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH] MD: Quickly return errors if too many devices have failed.

[Brassow Jonathan]

On Mar 27, 2013, at 7:13 PM, NeilBrown wrote:

> On Thu, 21 Mar 2013 08:58:54 -0500 Brassow Jonathan <jbrassow@redhat.com>
> wrote:
> 
>> 
>> On Mar 20, 2013, at 6:04 PM, NeilBrown wrote:
>> 
>>> On Wed, 20 Mar 2013 15:56:03 -0500 Brassow Jonathan <jbrassow@redhat.com>
>>> wrote:
>>> 
>>>> 
>>>> On Mar 19, 2013, at 9:46 PM, NeilBrown wrote:
>>>> 
>>>>> On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <jbrassow@redhat.com>
>>>>> wrote:
>>>>> 
>>>>>> 
>>>>>> On Mar 17, 2013, at 6:49 PM, NeilBrown wrote:
>>>>>> 
>>>>>>> On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <jbrassow@redhat.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Neil,
>>>>>>>> 
>>>>>>>> I've noticed that when too many devices fail in a RAID arrary that
>>>>>>>> addtional I/O will hang, yielding an endless supply of:
>>>>>>>> Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
>>>>>>>> Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
>>>>>>>> Mar 12 11:52:53 bp-01 kernel: sector=800 i=3           (null)           (null)  
>>>>>>>>      (null)           (null) 1
>>>>>>> 
>>>>>>> This is the third report in as many weeks that mentions that WARN_ON.
>>>>>>> The first two where quite different causes.
>>>>>>> I think this one is the same as the first one, which means it would be fixed
>>>>>>> by  
>>>>>>>   md/raid5: schedule_construction should abort if nothing to do.
>>>>>>> 
>>>>>>> which is commit 29d90fa2adbdd9f in linux-next.
>>>>>> 
>>>>>> Sorry, I don't see this commit in linux-next:
>>>>>> (the "for-next" branch of) git://github.com/neilbrown/linux.git
>>>>>> or git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
>>>>>> 
>>>>>> Where should I be looking?
>>>>> 
>>>>> Sorry, I probably messed up.
>>>>> I meant this commit:
>>>>> http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8ca
>>>> 
>>>> Yes, I found this patch in 'for-next'.  I tested 3.9.0-rc3 with and without this patch.  The good news is that my issue with RAID5 appears to be fixed with this patch.  To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size).  Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write.  After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write.  That's good.  Will this patch make it back to 3.[78]?
>>>> 
>>>> However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min.  Hung task messages with call traces and many many errors are the result.  This is bad.  I haven't figured out why these are so slow yet.
>>> 
>>> What happens if you take RAID out of the picture?
>>> i.e. write to a single device, then "kill" that device, then try issuing a
>>> 40M write request to it.
>>> 
>>> If that takes 2.5 minutes to resolve, then I think it is correct for RAID1 to
>>> also take 2.5 minutes to resolve. 
>>> If it resolves much more quickly than it does with RAID1, then that is a
>>> problem we should certainly address.
>> 
>> The test is a little different because once you offline a device, you can't open it.  So, I had to start I/O and then kill the device.  I still get 158MB/s - 3 orders of magnitude faster than RAID1.  Besides, if RAID10 takes 9+ minutes to complete, we'd still have something to fix.  I have also tested this with an "error" device and it also returns in sub-second time.
>> 
>> brassow
>> 
>> [root@bp-01 ~]# off.sh sda
>> Turning off sda
>> [root@bp-01 ~]# dd if=/dev/zero of=/dev/sda1 bs=4M count=10
>> dd: opening `/dev/sda1': No such device or address
>> [root@bp-01 ~]# on.sh sda
>> Turning on sda
>> [root@bp-01 ~]# dd if=/dev/zero of=/dev/sda bs=4M count=1000 &
>> [1] 5203
>> [root@bp-01 ~]# off.sh sda
>> Turning off sda
>> [root@bp-01 ~]# 1000+0 records in
>> 1000+0 records out
>> 4194304000 bytes (4.2 GB) copied, 26.5564 s, 158 MB/s
>> 
> 
> Maybe if you could show me some/all of the error messages that you get during
> these long delays it might help.  Also the error messages you (presumably)
> got from the kernel from the above plain-disk test.
> 
> It should quickly fail all but one copy of the data, then try writing to that
> copy exactly the same way that it would write to a plain disk.
> 
> For RAID10 large writes have to be chopped up for striping, so the extra
> requests which all have to fail could be the reason for the extra delay with
> RAID10.

Found the problem.  dm-raid does not support badblock tracking.  dm-raid was not setting badblocks.shift = -1.  Thus, RAID1/10 were trying to do narrow_write_error().  I'll have a patch for that shortly.

 brassow