feature re-quest for "re-write"

feature re-quest for "re-write"

[Mikael Abrahamsson]

Hi,

we have "check", "repair", "replacement" and other operations on raid 
volumes.

I am not a programmer, but I was wondering how much work it would require 
to take current code and implement "rewrite", basically re-writing every 
block in the md raid level. Since "repair" and "check" doesn't seem to 
properly detect a few errors, wouldn't it make sense to try least 
existance / easiest implementation route to just re-write all data on the 
entire array? If reads fail, re-calculate from parity, if reads work, just 
write again.

The goal of this new mode would be to eradicate pending sectors by 
re-writing everything on the drive.

If this doesn't seem like a sensible approach, what would be a sensible 
approach to avoid having pending sectors keep being "pending" even after 
"check" and "repair"?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: feature re-quest for "re-write"

[Brad Campbell]

On 22/02/14 02:09, Mikael Abrahamsson wrote:
>

> If this doesn't seem like a sensible approach, what would be a sensible
> approach to avoid having pending sectors keep being "pending" even after
> "check" and "repair"?
>

The only reason I've ever seen this personally was when the pending 
sectors were on non-data parts of the drive, like some of the space 
around the superblock. Have you verified that these issues are really on 
sectors in the data area? SMART should tell you the LBA of the first 
error in a read test.

Brad

Re: feature re-quest for "re-write"

[Eyal Lebedinsky]

In my case (see earlier thread "raid check does not..." the pending sector is early
in the device, in sector 261696 of a 4TB component (whole space in one partition of
each component). So yes, inside the data area.

I still have it reported in my daily logwatch, any idea what to try?

   Currently unreadable (pending) sectors detected:
	/dev/sdi [SAT] - 48 Time(s)
	1 unreadable sectors detected

Eyal

On 02/24/14 12:30, Brad Campbell wrote:
> On 22/02/14 02:09, Mikael Abrahamsson wrote:
>>
>
>> If this doesn't seem like a sensible approach, what would be a sensible
>> approach to avoid having pending sectors keep being "pending" even after
>> "check" and "repair"?
>>
>
> The only reason I've ever seen this personally was when the pending sectors were on non-data parts of the drive, like some of the space around the superblock. Have you verified that these issues are really on sectors in the data area? SMART should tell you the LBA of the first error in a read test.
>
> Brad

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: feature re-quest for "re-write"

[Brad Campbell]

On 24/02/14 09:46, Eyal Lebedinsky wrote:
> In my case (see earlier thread "raid check does not..." the pending
> sector is early
> in the device, in sector 261696 of a 4TB component (whole space in one
> partition of
> each component). So yes, inside the data area.
>
> I still have it reported in my daily logwatch, any idea what to try?
>

Yes, can you run a dd of the md device from well before to well after 
the theoretical position of the error?

If the dd passes cleanly, it indicates the bad block is a parity block 
rather than a data block. That hopefully will help narrow down the scope 
of the search.

Brad

Re: feature re-quest for "re-write"

[Eyal Lebedinsky]

I know that the i/o error is in /dev/sdi sector 261696 (consistent kernel and smart reports)
- /dev/sdi1 starts 2048 sectors later
- /dev/md127 is a 7 devs raid6 so there is 5 times as much data in the array until we hit
   the bad sector

# dd if=/dev/sdi1 of=/dev/null skip=$((1*(261696-2048))) count=1
dd: error reading '/dev/sdi1': Input/output error
0+0 records in
0+0 records out

The error is in one sector but 8 sectors will be read (a 4k buffer) so to get a clean read:

# dd if=/dev/sdi1 of=/dev/null skip=$((1*(261696-2048)+8)) count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 5.5852e-05 s, 9.2 MB/s

# echo 3 >'/proc/sys/vm/drop_caches'
# dd if=/dev/md127 of=/dev/null skip=$((5*(261696-2048))) count=5
5+0 records in
5+0 records out
2560 bytes (2.6 kB) copied, 0.00380436 s, 673 kB/s

Now reading *much* more than necessary (first 10GB of the array):

# echo 3 >'/proc/sys/vm/drop_caches'
# dd if=/dev/md127 of=/dev/null count=$((20*1024*1024))
20971520+0 records in
20971520+0 records out
10737418240 bytes (11 GB) copied, 13.5717 s, 791 MB/s

Note that I do not expect to get an error because reading the array will not read the P/Q checksums
(it assumes good data and avoids the calculations overhead of verifying P/Q).

BTW, due to the use of a buffer layer I could have done the whole test using 4k blocks rather than
sectors, but it makes no difference in this case.

Eyal

On 02/24/14 13:11, Brad Campbell wrote:
> On 24/02/14 09:46, Eyal Lebedinsky wrote:
>> In my case (see earlier thread "raid check does not..." the pending
>> sector is early
>> in the device, in sector 261696 of a 4TB component (whole space in one
>> partition of
>> each component). So yes, inside the data area.
>>
>> I still have it reported in my daily logwatch, any idea what to try?
>>
>
> Yes, can you run a dd of the md device from well before to well after the theoretical position of the error?
>
> If the dd passes cleanly, it indicates the bad block is a parity block rather than a data block. That hopefully will help narrow down the scope of the search.
>
> Brad

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: feature re-quest for "re-write"

[Wilson Jonathan]

> # echo 3 >'/proc/sys/vm/drop_caches'
> # dd if=/dev/md127 of=/dev/null count=$((20*1024*1024))
> 20971520+0 records in
> 20971520+0 records out
> 10737418240 bytes (11 GB) copied, 13.5717 s, 791 MB/s
> 
> Note that I do not expect to get an error because reading the array will not read the P/Q checksums
> (it assumes good data and avoids the calculations overhead of verifying P/Q).
> 
> BTW, due to the use of a buffer layer I could have done the whole test using 4k blocks rather than
> sectors, but it makes no difference in this case.
> 
> Eyal
> 

I wonder, could you not use dd to perform a "refresh"?

dd if=/dev/md127 of=/dev/md127 count=... bs=....

As that would force a re-calc and write of P & Q and data.

That said, its just a "could you/would it" suggestion not a "do it" due
to the inherent dangers of dd.

Re: feature re-quest for "re-write"

[Eyal Lebedinsky]

My main interest is to understand why 'check' does not actually check.
I already know how to fix the problem, by writing to the location I
can force the pending reallocation to happen, but then I will not have
the test case anymore.

The OP asks for a specific solution, but I think that the 'check' action
should already correctly rewrite failed (i/o error) sectors. It does not
always know which sector to rewrite when it finds a raid6 mismatch
without an i/o error (with raid5 it never knows).

Eyal

On 02/25/14 01:14, Wilson Jonathan wrote:
>
>> # echo 3 >'/proc/sys/vm/drop_caches'
>> # dd if=/dev/md127 of=/dev/null count=$((20*1024*1024))
>> 20971520+0 records in
>> 20971520+0 records out
>> 10737418240 bytes (11 GB) copied, 13.5717 s, 791 MB/s
>>
>> Note that I do not expect to get an error because reading the array will not read the P/Q checksums
>> (it assumes good data and avoids the calculations overhead of verifying P/Q).
>>
>> BTW, due to the use of a buffer layer I could have done the whole test using 4k blocks rather than
>> sectors, but it makes no difference in this case.
>>
>> Eyal
>>
>
> I wonder, could you not use dd to perform a "refresh"?
>
> dd if=/dev/md127 of=/dev/md127 count=... bs=....
>
> As that would force a re-calc and write of P & Q and data.
>
> That said, its just a "could you/would it" suggestion not a "do it" due
> to the inherent dangers of dd.

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: feature re-quest for "re-write"

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2444 bytes --]

On Tue, 25 Feb 2014 07:39:14 +1100 Eyal Lebedinsky <eyal@eyal.emu.id.au>
wrote:

> My main interest is to understand why 'check' does not actually check.
> I already know how to fix the problem, by writing to the location I
> can force the pending reallocation to happen, but then I will not have
> the test case anymore.
> 
> The OP asks for a specific solution, but I think that the 'check' action
> should already correctly rewrite failed (i/o error) sectors. It does not
> always know which sector to rewrite when it finds a raid6 mismatch
> without an i/o error (with raid5 it never knows).
> 

I cannot reproduce the problem.  In my testing a read error is fixed by
'check'.  For you it clearly isn't.  I wonder what is different.

During normal 'check' or 'repair' etc the read requests are allowed to be
combined by the io scheduler so when we get a read error, it could be one
error for a megabyte of more of the address space.
So the first thing raid5.c does is arrange to read all the blocks again but
to prohibit the merging of requests.  This time any read error will be for a
single 4K block.

Once we have that reliable read error the data is constructed from the other
blocks and the new block is written out.

This suggests that when there is a read error you should see e.g.

[  714.808494] end_request: I/O error, dev sds, sector 8141872

then shortly after that another similar error, possibly with a slightly
different sector number (at most a few thousand sectors later).

Then something like

md/raid:md0: read error corrected (8 sectors at 8141872 on sds)


However in the log Mikael Abrahamsson posted on 16 Jan 2014 
(Subject: Re: read errors not corrected when doing check on RAID6)

we only see that first 'end_request' message.  No second one and no "read
error corrected".

This seems to suggest that the second read succeeded, which is odd (to say
the least).

In your log posted 21 Feb 2014
(Subject: raid 'check' does not provoke expected i/o error)
there aren't even any read errors during 'check'.
The drive sometimes reports a read error and something doesn't?
Does reading the drive with 'dd' already report an error, and with 'check'
never report an error?



So I'm a bit stumped.  It looks like md is doing the right thing, but maybe
the drive is getting confused.
Are all the people who report this using the same sort of drive??

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: feature re-quest for "re-write"

[Eyal Lebedinsky]

My case is consistent.

Reading /dev/sdi1 provokes the end_request error. This is 100% reproducible.
Reading /dev/md127 runs clean (no error message).
Doing a 'check' completes clean (no error message).(*)
smartctl shows one pending sector.

Some details listed below.

Eyal

(*) I run a check action by setting sync_min/sync_max/sync_action
to cover the bad sector. However, just to be sure, I allowed an overnight
full check which also ran clean. The bad sector is still pending.

This is how I run the short tests:

# parted -l
Model: ATA WDC WD4001FAEX-0 (scsi)
Disk /dev/sdi: 4001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
  1      1049kB  4001GB  4001GB

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid6 sdf1[7] sdd1[1] sdc1[0] sdg1[4] sdh1[5] sde1[2] sdi1[6]
       19534425600 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]
       bitmap: 0/30 pages [0KB], 65536KB chunk

# cat /sys/block/md127/md/chunk_size
524288

# sys="/sys/block/md127/md"
# echo       '0' >$sys/sync_min	# check first
# echo '1000384' >$sys/sync_max	#   1m sectors, 0.5GB
# echo 'check'   >$sys/sync_action

Examining /proc/mdstat every second:
16:30:46       [>....................]  check =  0.0% (131360/3906885120) finish=495.6min speed=131360K/sec
16:30:47       [>....................]  check =  0.0% (273696/3906885120) finish=237.8min speed=273696K/sec
16:30:48       [>....................]  check =  0.0% (416032/3906885120) finish=312.9min speed=208016K/sec
16:30:49       [>....................]  check =  0.0% (561952/3906885120) finish=347.5min speed=187317K/sec
16:30:50       [>....................]  check =  0.0% (707872/3906885120) finish=367.8min speed=176968K/sec
16:30:51       [>....................]  check =  0.0% (844072/3906885120) finish=385.6min speed=168814K/sec
16:30:52       [>....................]  check =  0.0% (991016/3906885120) finish=394.1min speed=165169K/sec
16:30:53       [>....................]  check =  0.0% (1136936/3906885120) finish=400.7min speed=162419K/sec
16:30:54       [>....................]  check =  0.0% (1283368/3906885120) finish=405.7min speed=160421K/sec
16:30:55       [>....................]  check =  0.0% (1427752/3906885120) finish=410.3min speed=158639K/sec
16:30:56       [>....................]  check =  0.0% (1544492/3906885120) finish=463.5min speed=140408K/sec
16:30:57       [>....................]  check =  0.0% (1726304/3906885120) finish=452.4min speed=143858K/sec
16:30:58       [>....................]  check =  0.0% (1866592/3906885120) finish=453.2min speed=143584K/sec
16:30:59       [>....................]  check =  0.0% (2012000/3906885120) finish=452.8min speed=143714K/sec
16:31:00       [>....................]  check =  0.0% (2154336/3906885120) finish=453.1min speed=143622K/sec
16:31:01       [>....................]  check =  0.0% (2226336/3906885120) finish=467.6min speed=139146K/sec
16:31:02       [>....................]  check =  0.0% (2401636/3906885120) finish=460.6min speed=141272K/sec
16:31:03       [>....................]  check =  0.0% (2549592/3906885120) finish=459.4min speed=141644K/sec
16:31:04       [>....................]  check =  0.0% (2690864/3906885120) finish=459.4min speed=141625K/sec
16:31:05       [>....................]  check =  0.0% (2834776/3906885120) finish=459.0min speed=141738K/sec
16:31:06       [>....................]  check =  0.0% (2928880/3906885120) finish=466.5min speed=139470K/sec
16:31:07       [>....................]  check =  0.0% (3029760/3906885120) finish=472.4min speed=137716K/sec
16:31:08       [>....................]  check =  0.0% (3111680/3906885120) finish=480.9min speed=135290K/sec
16:31:09       [>....................]  check =  0.0% (3258624/3906885120) finish=479.1min speed=135776K/sec
16:31:10       [>....................]  check =  0.0% (3401472/3906885120) finish=478.1min speed=136058K/sec
16:31:11       [>....................]  check =  0.0% (3544832/3906885120) finish=477.1min speed=136339K/sec
16:31:12       [>....................]  check =  0.0% (3657476/3906885120) finish=480.2min speed=135462K/sec
16:31:13       [>....................]  check =  0.0% (3797764/3906885120) finish=479.6min speed=135634K/sec
16:31:14       [>....................]  check =  0.1% (3941636/3906885120) finish=478.5min speed=135918K/sec
16:31:15       [>....................]  check =  0.1% (4076292/3906885120) finish=478.7min speed=135876K/sec
16:31:16       [>....................]  check =  0.1% (4221188/3906885120) finish=477.6min speed=136167K/sec
16:31:17       [>....................]  check =  0.1% (4325252/3906885120) finish=481.2min speed=135164K/sec
16:31:18       [>....................]  check =  0.1% (4497992/3906885120) finish=477.1min speed=136302K/sec
16:31:19       [>....................]  check =  0.1% (4644936/3906885120) finish=477.1min speed=136300K/sec
16:31:20       [>....................]  check =  0.1% (4779088/3906885120) finish=477.3min speed=136233K/sec
16:31:21       [>....................]  check =  0.1% (4914888/3906885120) finish=477.4min speed=136220K/sec
16:31:22       [>....................]  check =  0.1% (4990720/3906885120) finish=487.6min speed=133366K/sec
16:31:23       [>....................]  check =  0.1% (4999896/3906885120) finish=502.2min speed=129485K/sec

# cat /sys/block/md127/md/mismatch_cnt
0

# echo 'idle'   >$sys/sync_action

# dmesg|tail
[ 4134.750324] md: data-check of RAID array md127
[ 4134.756992] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 4134.764956] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[ 4134.776816] md: using 128k window, over a total of 3906885120k.
[ 4174.065003] md: md_do_sync() got signal ... exiting

On 02/25/14 14:16, NeilBrown wrote:
> On Tue, 25 Feb 2014 07:39:14 +1100 Eyal Lebedinsky <eyal@eyal.emu.id.au>
> wrote:
>
>> My main interest is to understand why 'check' does not actually check.
>> I already know how to fix the problem, by writing to the location I
>> can force the pending reallocation to happen, but then I will not have
>> the test case anymore.
>>
>> The OP asks for a specific solution, but I think that the 'check' action
>> should already correctly rewrite failed (i/o error) sectors. It does not
>> always know which sector to rewrite when it finds a raid6 mismatch
>> without an i/o error (with raid5 it never knows).
>>
>
> I cannot reproduce the problem.  In my testing a read error is fixed by
> 'check'.  For you it clearly isn't.  I wonder what is different.
>
> During normal 'check' or 'repair' etc the read requests are allowed to be
> combined by the io scheduler so when we get a read error, it could be one
> error for a megabyte of more of the address space.
> So the first thing raid5.c does is arrange to read all the blocks again but
> to prohibit the merging of requests.  This time any read error will be for a
> single 4K block.
>
> Once we have that reliable read error the data is constructed from the other
> blocks and the new block is written out.
>
> This suggests that when there is a read error you should see e.g.
>
> [  714.808494] end_request: I/O error, dev sds, sector 8141872
>
> then shortly after that another similar error, possibly with a slightly
> different sector number (at most a few thousand sectors later).
>
> Then something like
>
> md/raid:md0: read error corrected (8 sectors at 8141872 on sds)
>
>
> However in the log Mikael Abrahamsson posted on 16 Jan 2014
> (Subject: Re: read errors not corrected when doing check on RAID6)
>
> we only see that first 'end_request' message.  No second one and no "read
> error corrected".
>
> This seems to suggest that the second read succeeded, which is odd (to say
> the least).
>
> In your log posted 21 Feb 2014
> (Subject: raid 'check' does not provoke expected i/o error)
> there aren't even any read errors during 'check'.
> The drive sometimes reports a read error and something doesn't?
> Does reading the drive with 'dd' already report an error, and with 'check'
> never report an error?
>
>
>
> So I'm a bit stumped.  It looks like md is doing the right thing, but maybe
> the drive is getting confused.
> Are all the people who report this using the same sort of drive??
>
> NeilBrown
>

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: feature re-quest for "re-write"

[Stan Hoeppner]

On 2/24/2014 11:58 PM, Eyal Lebedinsky wrote:
...
> (*) I run a check action by setting sync_min/sync_max/sync_action
> to cover the bad sector. However, just to be sure, I allowed an overnight
> full check which also ran clean. The bad sector is still pending.

What is the expected behavior when the drive's spare sector pool has
been exhausted, and thus the sector cannot be remapped by the drive
firmware?

Unless md now keeps a spare sector pool of its own and remaps bad
sectors, the only way to fix this situation is to replace the drive.
And if indeed drive spare pool exhaustion is the cause of the sector not
being remapped, the drive needs to be replaced.

-- 
Stan

Re: feature re-quest for "re-write"

[Eyal Lebedinsky]

The disk (actually the whole array) is relatively new and so far has no reallocated sectors.

Eyal

On 02/25/14 18:05, Stan Hoeppner wrote:
> On 2/24/2014 11:58 PM, Eyal Lebedinsky wrote:
> ...
>> (*) I run a check action by setting sync_min/sync_max/sync_action
>> to cover the bad sector. However, just to be sure, I allowed an overnight
>> full check which also ran clean. The bad sector is still pending.
>
> What is the expected behavior when the drive's spare sector pool has
> been exhausted, and thus the sector cannot be remapped by the drive
> firmware?
>
> Unless md now keeps a spare sector pool of its own and remaps bad
> sectors, the only way to fix this situation is to replace the drive.
> And if indeed drive spare pool exhaustion is the cause of the sector not
> being remapped, the drive needs to be replaced.

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: feature re-quest for "re-write"

[Eyal Lebedinsky]

BTW, Is there a monitoring tool to trace all i/o to a device? I could then
log activity to /dev/sd[c-i]1 during a (short) 'check' and see if all sectors
are really read. Or does md have a debug facility for this?

Eyal

On 02/25/14 14:16, NeilBrown wrote:
> On Tue, 25 Feb 2014 07:39:14 +1100 Eyal Lebedinsky <eyal@eyal.emu.id.au>
> wrote:
>
>> My main interest is to understand why 'check' does not actually check.
>> I already know how to fix the problem, by writing to the location I
>> can force the pending reallocation to happen, but then I will not have
>> the test case anymore.
>>
>> The OP asks for a specific solution, but I think that the 'check' action
>> should already correctly rewrite failed (i/o error) sectors. It does not
>> always know which sector to rewrite when it finds a raid6 mismatch
>> without an i/o error (with raid5 it never knows).
>>
>
> I cannot reproduce the problem.  In my testing a read error is fixed by
> 'check'.  For you it clearly isn't.  I wonder what is different.
>
> During normal 'check' or 'repair' etc the read requests are allowed to be
> combined by the io scheduler so when we get a read error, it could be one
> error for a megabyte of more of the address space.
> So the first thing raid5.c does is arrange to read all the blocks again but
> to prohibit the merging of requests.  This time any read error will be for a
> single 4K block.
>
> Once we have that reliable read error the data is constructed from the other
> blocks and the new block is written out.
>
> This suggests that when there is a read error you should see e.g.
>
> [  714.808494] end_request: I/O error, dev sds, sector 8141872
>
> then shortly after that another similar error, possibly with a slightly
> different sector number (at most a few thousand sectors later).
>
> Then something like
>
> md/raid:md0: read error corrected (8 sectors at 8141872 on sds)
>
>
> However in the log Mikael Abrahamsson posted on 16 Jan 2014
> (Subject: Re: read errors not corrected when doing check on RAID6)
>
> we only see that first 'end_request' message.  No second one and no "read
> error corrected".
>
> This seems to suggest that the second read succeeded, which is odd (to say
> the least).
>
> In your log posted 21 Feb 2014
> (Subject: raid 'check' does not provoke expected i/o error)
> there aren't even any read errors during 'check'.
> The drive sometimes reports a read error and something doesn't?
> Does reading the drive with 'dd' already report an error, and with 'check'
> never report an error?
>
>
>
> So I'm a bit stumped.  It looks like md is doing the right thing, but maybe
> the drive is getting confused.
> Are all the people who report this using the same sort of drive??
>
> NeilBrown
>

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: feature re-quest for "re-write"

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3332 bytes --]

On Tue, 25 Feb 2014 18:58:16 +1100 Eyal Lebedinsky <eyal@eyal.emu.id.au>
wrote:

> BTW, Is there a monitoring tool to trace all i/o to a device? I could then
> log activity to /dev/sd[c-i]1 during a (short) 'check' and see if all sectors
> are really read. Or does md have a debug facility for this?

blktrace will collect a trace, blkparse will print it out for you.
You need to trace the 'whole' device.

So something like

  blktrace /dev/sd[c-i]
  # run the test
  ctrl-C
  blkparse sd[c-i]*

blktrace creates several files, I think one for each device on each CPU.


NeilBrown

> 
> Eyal
> 
> On 02/25/14 14:16, NeilBrown wrote:
> > On Tue, 25 Feb 2014 07:39:14 +1100 Eyal Lebedinsky <eyal@eyal.emu.id.au>
> > wrote:
> >
> >> My main interest is to understand why 'check' does not actually check.
> >> I already know how to fix the problem, by writing to the location I
> >> can force the pending reallocation to happen, but then I will not have
> >> the test case anymore.
> >>
> >> The OP asks for a specific solution, but I think that the 'check' action
> >> should already correctly rewrite failed (i/o error) sectors. It does not
> >> always know which sector to rewrite when it finds a raid6 mismatch
> >> without an i/o error (with raid5 it never knows).
> >>
> >
> > I cannot reproduce the problem.  In my testing a read error is fixed by
> > 'check'.  For you it clearly isn't.  I wonder what is different.
> >
> > During normal 'check' or 'repair' etc the read requests are allowed to be
> > combined by the io scheduler so when we get a read error, it could be one
> > error for a megabyte of more of the address space.
> > So the first thing raid5.c does is arrange to read all the blocks again but
> > to prohibit the merging of requests.  This time any read error will be for a
> > single 4K block.
> >
> > Once we have that reliable read error the data is constructed from the other
> > blocks and the new block is written out.
> >
> > This suggests that when there is a read error you should see e.g.
> >
> > [  714.808494] end_request: I/O error, dev sds, sector 8141872
> >
> > then shortly after that another similar error, possibly with a slightly
> > different sector number (at most a few thousand sectors later).
> >
> > Then something like
> >
> > md/raid:md0: read error corrected (8 sectors at 8141872 on sds)
> >
> >
> > However in the log Mikael Abrahamsson posted on 16 Jan 2014
> > (Subject: Re: read errors not corrected when doing check on RAID6)
> >
> > we only see that first 'end_request' message.  No second one and no "read
> > error corrected".
> >
> > This seems to suggest that the second read succeeded, which is odd (to say
> > the least).
> >
> > In your log posted 21 Feb 2014
> > (Subject: raid 'check' does not provoke expected i/o error)
> > there aren't even any read errors during 'check'.
> > The drive sometimes reports a read error and something doesn't?
> > Does reading the drive with 'dd' already report an error, and with 'check'
> > never report an error?
> >
> >
> >
> > So I'm a bit stumped.  It looks like md is doing the right thing, but maybe
> > the drive is getting confused.
> > Are all the people who report this using the same sort of drive??
> >
> > NeilBrown
> >
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: feature re-quest for "re-write"

[Eyal Lebedinsky]

This is helpful Neil.

I am running blktrace/blkparse and trying to understand what it is telling
me.

If I got it right then I see that doing a check of md127 (from the start)
starts reading with this entry

8,129  6      327     0.992307218 20259  D   R 264200 + 504 [md127_resync]

which means that the real data starts rather further into the stripes.
Actually, further than the bad block: sector 259648 of sdi1 is before the
first read operation. Though I am not even sure that the blkparse 264200
is sectors and now 1KB blocks or 4KB blocks.

Following is some speculation.

Does md127 store a header before it starts striping the data? May this
be why it rarely actually needs to read parts of this header?
(I thought that superblocks and what not are stored at the far end).

If so, then the content of this sector is not part of the redundant data and may
not be trivial to recover. Then again, I expect important data is recorded more
than once.

If this is the case then the calculation to correlate the bad sector to the fs
block (which I need to do whenever I find a bad sector in order to investigate
my data loss) is more complicated than I assumed.

Final thought: if this sector is in an important header, when it *does* need
to be read (and fail), how bad a reaction should I expect?

Eyal

On 02/25/14 19:35, NeilBrown wrote:
> On Tue, 25 Feb 2014 18:58:16 +1100 Eyal Lebedinsky <eyal@eyal.emu.id.au>
> wrote:
>
>> BTW, Is there a monitoring tool to trace all i/o to a device? I could then
>> log activity to /dev/sd[c-i]1 during a (short) 'check' and see if all sectors
>> are really read. Or does md have a debug facility for this?
>
> blktrace will collect a trace, blkparse will print it out for you.
> You need to trace the 'whole' device.
>
> So something like
>
>    blktrace /dev/sd[c-i]
>    # run the test
>    ctrl-C
>    blkparse sd[c-i]*
>
> blktrace creates several files, I think one for each device on each CPU.
>
>
> NeilBrown
>
>>
>> Eyal
>>
>> On 02/25/14 14:16, NeilBrown wrote:
>>> On Tue, 25 Feb 2014 07:39:14 +1100 Eyal Lebedinsky <eyal@eyal.emu.id.au>
>>> wrote:
>>>
>>>> My main interest is to understand why 'check' does not actually check.
>>>> I already know how to fix the problem, by writing to the location I
>>>> can force the pending reallocation to happen, but then I will not have
>>>> the test case anymore.
>>>>
>>>> The OP asks for a specific solution, but I think that the 'check' action
>>>> should already correctly rewrite failed (i/o error) sectors. It does not
>>>> always know which sector to rewrite when it finds a raid6 mismatch
>>>> without an i/o error (with raid5 it never knows).
>>>>
>>>
>>> I cannot reproduce the problem.  In my testing a read error is fixed by
>>> 'check'.  For you it clearly isn't.  I wonder what is different.
>>>
>>> During normal 'check' or 'repair' etc the read requests are allowed to be
>>> combined by the io scheduler so when we get a read error, it could be one
>>> error for a megabyte of more of the address space.
>>> So the first thing raid5.c does is arrange to read all the blocks again but
>>> to prohibit the merging of requests.  This time any read error will be for a
>>> single 4K block.
>>>
>>> Once we have that reliable read error the data is constructed from the other
>>> blocks and the new block is written out.
>>>
>>> This suggests that when there is a read error you should see e.g.
>>>
>>> [  714.808494] end_request: I/O error, dev sds, sector 8141872
>>>
>>> then shortly after that another similar error, possibly with a slightly
>>> different sector number (at most a few thousand sectors later).
>>>
>>> Then something like
>>>
>>> md/raid:md0: read error corrected (8 sectors at 8141872 on sds)
>>>
>>>
>>> However in the log Mikael Abrahamsson posted on 16 Jan 2014
>>> (Subject: Re: read errors not corrected when doing check on RAID6)
>>>
>>> we only see that first 'end_request' message.  No second one and no "read
>>> error corrected".
>>>
>>> This seems to suggest that the second read succeeded, which is odd (to say
>>> the least).
>>>
>>> In your log posted 21 Feb 2014
>>> (Subject: raid 'check' does not provoke expected i/o error)
>>> there aren't even any read errors during 'check'.
>>> The drive sometimes reports a read error and something doesn't?
>>> Does reading the drive with 'dd' already report an error, and with 'check'
>>> never report an error?
>>>
>>>
>>>
>>> So I'm a bit stumped.  It looks like md is doing the right thing, but maybe
>>> the drive is getting confused.
>>> Are all the people who report this using the same sort of drive??
>>>
>>> NeilBrown
>>>
>>
>

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: feature re-quest for "re-write"

[Mikael Abrahamsson]

On Tue, 25 Feb 2014, Eyal Lebedinsky wrote:

> Final thought: if this sector is in an important header, when it *does* 
> need to be read (and fail), how bad a reaction should I expect?

I have two thoughts here:

Check data offset when doing mdadm -E. There you will see how much unused 
data is allocated between the superblock and start of the actual array 
data contents. This might be where your pending block is.

Regarding re-write. I have had happen to me that one drive that had bad 
blocks that "check" didn't find errors on, when I rebooted that drive had 
read errors on the superblock, was not assembled into the array, and 
instead md started rebuilding to a spare since the array was degraded. So 
my wonder is, when issuing "check" or "repair", does md actually check if 
the superblocks are readable? If not, perhaps it should? Should it check 
the contents of the superblocks are consistent with the data that the 
kernel has in its data structures?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: feature re-quest for "re-write"

[Eyal Lebedinsky]

Yes, this matches what I saw. From "mdadm -E /dev/sdi1":

  Avail Dev Size : 7813771264 (3725.90 GiB 4000.65 GB)
      Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
   Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors

Is the layout of this 128KB area documented? I expect basic superblock data, log, bitmap,
what else? Are there copies elsewhere on the disk that can be used? I wonder what the
bad sector 259648 (close to the end of the header) covers.

Can it be rebuilt from the other members of the array (when stopped one can expect the
log and bitmap to be clearable)? Don't know.

Maybe build a new array with the --assume-clean option that will rewrite the header but
leave the data alone? Doco says "not  recommended".

Or just give up: fail and remove the disk, clear the superblock then add it and go through
a full resync. This way feels safer as I do not touch the other members.

cheers
	Eyal

On 02/25/14 22:28, Mikael Abrahamsson wrote:
> On Tue, 25 Feb 2014, Eyal Lebedinsky wrote:
>
>> Final thought: if this sector is in an important header, when it *does* need to be read (and fail), how bad a reaction should I expect?
>
> I have two thoughts here:
>
> Check data offset when doing mdadm -E. There you will see how much unused data is allocated between the superblock and start of the actual array data contents. This might be where your pending block is.
>
> Regarding re-write. I have had happen to me that one drive that had bad blocks that "check" didn't find errors on, when I rebooted that drive had read errors on the superblock, was not assembled into the array, and instead md started rebuilding to a spare since the array was degraded. So my wonder is, when issuing "check" or "repair", does md actually check if the superblocks are readable? If not, perhaps it should? Should it check the contents of the superblocks are consistent with the data that the kernel has in its data structures?

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: feature re-quest for "re-write"

[Mikael Abrahamsson]

On Tue, 25 Feb 2014, Eyal Lebedinsky wrote:

> Or just give up: fail and remove the disk, clear the superblock then add 
> it and go through a full resync. This way feels safer as I do not touch 
> the other members.

I am not sure this will work either. I would expect that the empty data in 
"data offset" is never touched even when doing rebuild. Perhaps also 
something that should be done?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: feature re-quest for "re-write"

[Eyal Lebedinsky]

I expect that zeroing (and then recreating the header) will surely write the whole area.
Or I can play really safe and write to the bad sector myself to force the reallocation
while the disk is out.

Eyal


On 02/25/14 23:17, Mikael Abrahamsson wrote:
> On Tue, 25 Feb 2014, Eyal Lebedinsky wrote:
>
>> Or just give up: fail and remove the disk, clear the superblock then add it and go through a full resync. This way feels safer as I do not touch the other members.
>
> I am not sure this will work either. I would expect that the empty data in "data offset" is never touched even when doing rebuild. Perhaps also something that should be done?
>

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au)

Re: feature re-quest for "re-write"

[Mikael Abrahamsson]

On Mon, 24 Feb 2014, Brad Campbell wrote:

> The only reason I've ever seen this personally was when the pending 
> sectors were on non-data parts of the drive, like some of the space 
> around the superblock. Have you verified that these issues are really on 
> sectors in the data area? SMART should tell you the LBA of the first 
> error in a read test.

I even received UNC errors in the log when doing "repair" but the sector 
still wasn't re-written. So yes, they were on data part.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: feature re-quest for "re-write"

[Brad Campbell]

On 22/02/14 02:09, Mikael Abrahamsson wrote:
>
> Hi,
>
> we have "check", "repair", "replacement" and other operations on raid
> volumes.
>
> I am not a programmer, but I was wondering how much work it would
> require to take current code and implement "rewrite", basically
> re-writing every block in the md raid level. Since "repair" and "check"
> doesn't seem to properly detect a few errors, wouldn't it make sense to
> try least existance / easiest implementation route to just re-write all
> data on the entire array? If reads fail, re-calculate from parity, if
> reads work, just write again.

Now, this is after 3 minutes of looking at raid5.c, so if I've missed 
something obvious please feel free to yell at me. I'm not much of a 
programmer. Having said that -

Can someone check my understanding of this bit of code?

static void handle_parity_checks6(struct r5conf *conf, struct 
stripe_head *sh,
                                   struct stripe_head_state *s,
                                   int disks)
<....>

         switch (sh->check_state) {
         case check_state_idle:
                 /* start a new check operation if there are < 2 failures */
                 if (s->failed == s->q_failed) {
                         /* The only possible failed device holds Q, so it
                          * makes sense to check P (If anything else 
were failed,
                          * we would have used P to recreate it).
                          */
                         sh->check_state = check_state_run;
                 }
                 if (!s->q_failed && s->failed < 2) {
                         /* Q is not failed, and we didn't use it to 
generate
                          * anything, so it makes sense to check it
                          */
                         if (sh->check_state == check_state_run)
                                 sh->check_state = check_state_run_pq;
                         else
                                 sh->check_state = check_state_run_q;
                 }


So we get passed a stripe. If it's not being checked we :

- If Q has failed we initiate check_state_run (which checks only P)

- If we have less than 2 failed drives (lets say we have none), if we 
are already checking P (check_state_run) we upgrade that to 
check_state_run_pq (and therefore check both).

However

- If we were check_state_idle, beacuse we had 0 failed drives, then we 
only mark check_state_run_q and therefore skip checking P ??

Regards,
Brad

Re: feature re-quest for "re-write"

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3222 bytes --]

On Mon, 24 Feb 2014 10:24:36 +0800 Brad Campbell <lists2009@fnarfbargle.com>
wrote:

> On 22/02/14 02:09, Mikael Abrahamsson wrote:
> >
> > Hi,
> >
> > we have "check", "repair", "replacement" and other operations on raid
> > volumes.
> >
> > I am not a programmer, but I was wondering how much work it would
> > require to take current code and implement "rewrite", basically
> > re-writing every block in the md raid level. Since "repair" and "check"
> > doesn't seem to properly detect a few errors, wouldn't it make sense to
> > try least existance / easiest implementation route to just re-write all
> > data on the entire array? If reads fail, re-calculate from parity, if
> > reads work, just write again.
> 
> Now, this is after 3 minutes of looking at raid5.c, so if I've missed 
> something obvious please feel free to yell at me. I'm not much of a 
> programmer. Having said that -
> 
> Can someone check my understanding of this bit of code?
> 
> static void handle_parity_checks6(struct r5conf *conf, struct 
> stripe_head *sh,
>                                    struct stripe_head_state *s,
>                                    int disks)
> <....>
> 
>          switch (sh->check_state) {
>          case check_state_idle:
>                  /* start a new check operation if there are < 2 failures */
>                  if (s->failed == s->q_failed) {
>                          /* The only possible failed device holds Q, so it
>                           * makes sense to check P (If anything else 
> were failed,
>                           * we would have used P to recreate it).
>                           */
>                          sh->check_state = check_state_run;
>                  }
>                  if (!s->q_failed && s->failed < 2) {
>                          /* Q is not failed, and we didn't use it to 
> generate
>                           * anything, so it makes sense to check it
>                           */
>                          if (sh->check_state == check_state_run)
>                                  sh->check_state = check_state_run_pq;
>                          else
>                                  sh->check_state = check_state_run_q;
>                  }
> 
>
> So we get passed a stripe. If it's not being checked we :
> 
> - If Q has failed we initiate check_state_run (which checks only P)
> 
> - If we have less than 2 failed drives (lets say we have none), if we 
> are already checking P (check_state_run) we upgrade that to 
> check_state_run_pq (and therefore check both).
> 
> However
> 
> - If we were check_state_idle, beacuse we had 0 failed drives, then we 
> only mark check_state_run_q and therefore skip checking P ??

This code is obviously too subtle.

If 0 drives have failed, then 's->failed' is 0 (it is the count of failed
drives), and  's->q_failed' is also 0 (it is a boolean flag, and q clearly
hasn't failed as nothing has).
So the first 'if' branch will be followed (as "0 == 0") and check_state set to
check_state_run.
Then as q_failed is still 0 and failed < 2, check_state gets set to
check_state_run_pq.

So it does check both p and q.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: feature re-quest for "re-write"

[Brad Campbell]

On 25/02/14 10:10, NeilBrown wrote:

> This code is obviously too subtle.

Not at all, it's my understanding that is under-developed. I was just 
looking for something obvious to explain the behaviour others have been 
reporting where a check won't trigger a re-write of a pending sector if 
the sector is a p or q rather than data.

> If 0 drives have failed, then 's->failed' is 0 (it is the count of failed
> drives), and  's->q_failed' is also 0 (it is a boolean flag, and q clearly
> hasn't failed as nothing has).
> So the first 'if' branch will be followed (as "0 == 0") and check_state set to
> check_state_run.
> Then as q_failed is still 0 and failed < 2, check_state gets set to
> check_state_run_pq.
>

Got it, thanks for taking the time to set me straight.

Regards,
Brad