Random MD Raid resyncs?

Random MD Raid resyncs?

[Jesse Wheeler]

All:

Thanks for the invaluable resource that is this mailing list.  I've
learned quite a bit over the space of this month.

I have the following configuration and question:

-- SNIP --

==

Linux version 2.6.18-164.2.1.el5 (mockbuild@builder10.centos.org) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Wed Sep 30 12:52:46
EDT 2009.  CentOS 5.4

==

Personalities : [raid10] [raid6] [raid5] [raid4]
md1 : active raid5 sda2[0] sdb1[1] sdc1[2] sdd1[3] sde1[4] sdf1[5]
     393230080 blocks level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU]

md2 : active raid5 sda4[0] sdb4[1] sdc3[2] sdd3[3] sde3[4] sdf4[5]
     424034560 blocks level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU]
     [===>.................]  resync = 17.4% (14793792/84806912)
finish=19.4min speed=60024K/sec

md0 : active raid10 sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1] sda3[0]
     235938048 blocks 256K chunks 2 near-copies [6/6] [UUUUUU]

==

[root@fatboy log]# mdadm --detail /dev/md0
/dev/md0:
       Version : 0.90
 Creation Time : Sun Nov  1 11:24:00 2009
    Raid Level : raid10
    Array Size : 235938048 (225.01 GiB 241.60 GB)
 Used Dev Size : 78646016 (75.00 GiB 80.53 GB)
  Raid Devices : 6
 Total Devices : 6
Preferred Minor : 0
   Persistence : Superblock is persistent

   Update Time : Mon Nov  2 09:46:48 2009
         State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
 Spare Devices : 0

        Layout : near=2
    Chunk Size : 256K

          UUID : 0d994fe1:d4b4be70:3fe15b01:0e4bd6cb
        Events : 0.4

   Number   Major   Minor   RaidDevice State
      0       8        3        0      active sync   /dev/sda3
      1       8       18        1      active sync   /dev/sdb2
      2       8       34        2      active sync   /dev/sdc2
      3       8       50        3      active sync   /dev/sdd2
      4       8       66        4      active sync   /dev/sde2
      5       8       82        5      active sync   /dev/sdf2

==

[root@fatboy log]# mdadm --detail /dev/md1
/dev/md1:
       Version : 0.90
 Creation Time : Sun Nov  1 11:28:46 2009
    Raid Level : raid5
    Array Size : 393230080 (375.01 GiB 402.67 GB)
 Used Dev Size : 78646016 (75.00 GiB 80.53 GB)
  Raid Devices : 6
 Total Devices : 6
Preferred Minor : 1
   Persistence : Superblock is persistent

   Update Time : Mon Nov  2 09:39:51 2009
         State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
 Spare Devices : 0

        Layout : left-symmetric
    Chunk Size : 256K

          UUID : f537dfcb:fe7bd1a9:68316ddd:10b55eed
        Events : 0.14

   Number   Major   Minor   RaidDevice State
      0       8        2        0      active sync   /dev/sda2
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1

==

[root@fatboy log]# mdadm --detail /dev/md2
/dev/md2:
       Version : 0.90
 Creation Time : Sun Nov  1 11:25:03 2009
    Raid Level : raid5
    Array Size : 424034560 (404.39 GiB 434.21 GB)
 Used Dev Size : 84806912 (80.88 GiB 86.84 GB)
  Raid Devices : 6
 Total Devices : 6
Preferred Minor : 2
   Persistence : Superblock is persistent

   Update Time : Mon Nov  2 08:10:46 2009
         State : clean, resyncing
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
 Spare Devices : 0

        Layout : left-symmetric
    Chunk Size : 256K

 Rebuild Status : 33% complete

          UUID : 0030789c:58c839d6:6529cdec:6347f0f3
        Events : 0.64

   Number   Major   Minor   RaidDevice State
      0       8        4        0      active sync   /dev/sda4
      1       8       20        1      active sync   /dev/sdb4
      2       8       35        2      active sync   /dev/sdc3
      3       8       51        3      active sync   /dev/sdd3
      4       8       67        4      active sync   /dev/sde3
      5       8       84        5      active sync   /dev/sdf4

==

-- END SNIP --

My question (please forgive the ignorance if it is showing -- I'm used
to FreeBSD and Solaris) is this..

I am noticing that my MD0 (Raid10), MD1 (Raid5), MD2 (Raid 5) arrays
"seem" to be resyncing at somewhat random intervals.  Mind you, I have
also noticed that the kernel has put a 'governor' on the resync
process as to limit performance issues.  Is this normal behavior? I've
checked CRON thinking that the resync might be a scheduled occurance,
as well as my /var/log/messages.  The only indication in the SYSLOG
are events like:

-- SNIP --

Nov  2 09:15:42 fatboy dnsmasq[3648]: exiting on receipt of SIGTERM
Nov  2 09:15:43 fatboy dnsmasq[6922]: started, version 2.45 cachesize 150
Nov  2 09:15:43 fatboy dnsmasq[6922]: compile time options: IPv6
GNU-getopt no-ISC-leasefile no-DBus no-I18N TFTP
Nov  2 09:15:43 fatboy dnsmasq[6922]: using local addresses only for
domain devotioconsulting.net
Nov  2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.220.220#53
Nov  2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.222.222#53
Nov  2 09:15:43 fatboy dnsmasq[6922]: read /etc/hosts - 14 addresses
Nov  2 09:21:04 fatboy kernel: md: syncing RAID array md1
Nov  2 09:21:04 fatboy kernel: md: minimum _guaranteed_ reconstruction
speed: 1000 KB/sec/disc.
Nov  2 09:21:04 fatboy kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for reconstruction.
Nov  2 09:21:04 fatboy kernel: md: using 128k window, over a total of
78646016 blocks.
Nov  2 09:21:04 fatboy kernel: md: delaying resync of md2 until md1
has finished resync (they share one or more physical units)
Nov  2 09:39:51 fatboy kernel: md: md1: sync done.

-- END SNIP

.. which do not indicate any outward hardware reason (i.e. LMSensors,
my UPS Monitoring daemon, etc) are not picking up any events to
trigger the resync process.

Is this normal?

I would think it somewhat odd to have an array just 'resync' randomly.

Please let me know if I am missing something.

--
Jesse W. Wheeler
Member-Owner
Devotio Consulting, L.L.C.
--



-- 
Jesse W. Wheeler
mailto: jwwstpete@gmail.com
Charlotte, North Carolina - U.S.A.

==
"I'm a white male, age 18 to 49. Everyone listens to me, no matter how
dumb my suggestions are." --Homer Simpson

"I really don't understand how bipartisanship is ever going to work
when one of the parties is insane."-- John Cole

"I defy the tyranny of precedent. I cannot afford the luxury of a
closed mind. I go for anything new that might improve the past." --
Clara Barton

"Any dictator would admire the uniformity and obedience of the U.S.
media." -- Noam Chomsky

"We are called to speak for the weak, for the voiceless, for victims
of our nation and for those it calls enemy...". -- Martin Luther King,
"Beyond Vietnam"

"Be yourself; everyone else is already taken." - Oscar Wilde

* My Other Car is a Health Insurance Payment
* My Car Has Better Insurance Than I Do
* My Death Panel is an HMO
* Underinsured Baby on Board
* WWJD:   Who Would Jesus Deny? (Healthcare Reform Now!)

"Quitbull with Lipstick: She's a gift that keeps on giving; an
Everlasting Gobstopper of fail." -- SilentBrook on DKos talking about
Palin
==
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Random MD Raid resyncs?

[NeilBrown]

On Tue, November 3, 2009 1:57 am, Jesse Wheeler wrote:
> All:
>
> Thanks for the invaluable resource that is this mailing list.  I've
> learned quite a bit over the space of this month.
>
> I have the following configuration and question:
>
.....
>
> Nov  2 09:15:42 fatboy dnsmasq[3648]: exiting on receipt of SIGTERM
> Nov  2 09:15:43 fatboy dnsmasq[6922]: started, version 2.45 cachesize 150
> Nov  2 09:15:43 fatboy dnsmasq[6922]: compile time options: IPv6
> GNU-getopt no-ISC-leasefile no-DBus no-I18N TFTP
> Nov  2 09:15:43 fatboy dnsmasq[6922]: using local addresses only for
> domain devotioconsulting.net
> Nov  2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.220.220#53
> Nov  2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.222.222#53
> Nov  2 09:15:43 fatboy dnsmasq[6922]: read /etc/hosts - 14 addresses
> Nov  2 09:21:04 fatboy kernel: md: syncing RAID array md1

The only possible explanation for this that I can think
of is the the array was started in 'auto-read-only' mode in which
it will not write to the devices at all until a write request comes in
through the top level device.  Once that write request arrives, md accepts
that the array really is meant to be used and starts any resync etc that
might be needed.

How is /dev/md1 used?  Is it possible that nothing tried to
write to it until this moment?

If it was something started by a cron job, it would not say
"resync =" but rather 'check =' or 'repair ='.

NeilBrown


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Random MD Raid resyncs?

[Jesse Wheeler]

Hi Neil:

Thanks for the help.

AFAIK, any of the RAID devices are not mounted 'auto-read-only' by the
kernel.  I'll double check /etc/fstab, /var/log/messages, and
/var/log/dmesg when I get back to the site.  Unless there was a change
by the upstream vendor, I wouldn't mount a production filesystem in
that manner.  Of course, I've been surprised before!

I'll double check and report back.  Thanks again for the suggestion!

On Mon, Nov 2, 2009 at 5:23 PM, NeilBrown <neilb@suse.de> wrote:
> On Tue, November 3, 2009 1:57 am, Jesse Wheeler wrote:
>> All:
>>
>> Thanks for the invaluable resource that is this mailing list.  I've
>> learned quite a bit over the space of this month.
>>
>> I have the following configuration and question:
>>
> .....
>>
>> Nov  2 09:15:42 fatboy dnsmasq[3648]: exiting on receipt of SIGTERM
>> Nov  2 09:15:43 fatboy dnsmasq[6922]: started, version 2.45 cachesize 150
>> Nov  2 09:15:43 fatboy dnsmasq[6922]: compile time options: IPv6
>> GNU-getopt no-ISC-leasefile no-DBus no-I18N TFTP
>> Nov  2 09:15:43 fatboy dnsmasq[6922]: using local addresses only for
>> domain devotioconsulting.net
>> Nov  2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.220.220#53
>> Nov  2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.222.222#53
>> Nov  2 09:15:43 fatboy dnsmasq[6922]: read /etc/hosts - 14 addresses
>> Nov  2 09:21:04 fatboy kernel: md: syncing RAID array md1
>
> The only possible explanation for this that I can think
> of is the the array was started in 'auto-read-only' mode in which
> it will not write to the devices at all until a write request comes in
> through the top level device.  Once that write request arrives, md accepts
> that the array really is meant to be used and starts any resync etc that
> might be needed.
>
> How is /dev/md1 used?  Is it possible that nothing tried to
> write to it until this moment?
>
> If it was something started by a cron job, it would not say
> "resync =" but rather 'check =' or 'repair ='.
>
> NeilBrown
>
>
>



-- 
Jesse W. Wheeler
mailto: jwwstpete@gmail.com
Charlotte, North Carolina - U.S.A.

==
"I'm a white male, age 18 to 49. Everyone listens to me, no matter how
dumb my suggestions are." --Homer Simpson

"I really don't understand how bipartisanship is ever going to work
when one of the parties is insane."-- John Cole

"I defy the tyranny of precedent. I cannot afford the luxury of a
closed mind. I go for anything new that might improve the past." --
Clara Barton

"Any dictator would admire the uniformity and obedience of the U.S.
media." -- Noam Chomsky

"We are called to speak for the weak, for the voiceless, for victims
of our nation and for those it calls enemy...". -- Martin Luther King,
"Beyond Vietnam"

"Be yourself; everyone else is already taken." - Oscar Wilde

* My Other Car is a Health Insurance Payment
* My Car Has Better Insurance Than I Do
* My Death Panel is an HMO
* Underinsured Baby on Board
* WWJD:   Who Would Jesus Deny? (Healthcare Reform Now!)

"Quitbull with Lipstick: She's a gift that keeps on giving; an
Everlasting Gobstopper of fail." -- SilentBrook on DKos talking about
Palin
==
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Random MD Raid resyncs?

[Gabor Gombas]

On Mon, Nov 02, 2009 at 06:40:59PM -0500, Jesse Wheeler wrote:

> AFAIK, any of the RAID devices are not mounted 'auto-read-only' by the
> kernel.  I'll double check /etc/fstab, /var/log/messages, and
> /var/log/dmesg when I get back to the site.  Unless there was a change
> by the upstream vendor, I wouldn't mount a production filesystem in
> that manner.  Of course, I've been surprised before!

auto-read-only is an internal state of the RAID array, it has nothing to
do with file systems. So you probably won't find anything in the
locations you mention above; auto-read-only is only displayed in
/proc/mdstat. Look at the mdadm man page and read the sections "Auto
Assembly" and "INCREMENTAL MODE".

Gabor

-- 
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------

Re: Random MD Raid resyncs?

[Jesse Wheeler]

Hi Gabor:

> auto-read-only is an internal state of the RAID array, it has nothing to
> do with file systems. So you probably won't find anything in the
> locations you mention above; auto-read-only is only displayed in
> /proc/mdstat. Look at the mdadm man page and read the sections "Auto
> Assembly" and "INCREMENTAL MODE".

Interesting.  I just got done reading that section in the manpage; if
anything, I didn't know mdadm would process an array in that fashion.
However, I don't see this behavior listed in my /proc/mdstat output.
If it's there, I'm missing it:

-- SNIP --

# cat /proc/mdstat
Personalities : [raid10] [raid6] [raid5] [raid4]
md1 : active raid5 sda2[0] sdb1[1] sdc1[2] sdd1[3] sde1[4] sdf1[5]
      393230080 blocks level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU]

md2 : active raid5 sda4[0] sdb4[1] sdc3[2] sdd3[3] sde3[4] sdf4[5]
      424034560 blocks level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU]

md0 : active raid10 sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1] sda3[0]
      235938048 blocks 256K chunks 2 near-copies [6/6] [UUUUUU]

unused devices: <none>

-- END SNIP --

On Tue, Nov 3, 2009 at 5:05 AM, Gabor Gombas <gombasg@sztaki.hu> wrote:
> On Mon, Nov 02, 2009 at 06:40:59PM -0500, Jesse Wheeler wrote:
>
>> AFAIK, any of the RAID devices are not mounted 'auto-read-only' by the
>> kernel.  I'll double check /etc/fstab, /var/log/messages, and
>> /var/log/dmesg when I get back to the site.  Unless there was a change
>> by the upstream vendor, I wouldn't mount a production filesystem in
>> that manner.  Of course, I've been surprised before!
>
> auto-read-only is an internal state of the RAID array, it has nothing to
> do with file systems. So you probably won't find anything in the
> locations you mention above; auto-read-only is only displayed in
> /proc/mdstat. Look at the mdadm man page and read the sections "Auto
> Assembly" and "INCREMENTAL MODE".
>
> Gabor
>
> --
>     ---------------------------------------------------------
>     MTA SZTAKI Computer and Automation Research Institute
>                Hungarian Academy of Sciences
>     ---------------------------------------------------------
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Random MD Raid resyncs?

[Gabor Gombas]

On Tue, Nov 03, 2009 at 09:44:50AM -0500, Jesse Wheeler wrote:

> Interesting.  I just got done reading that section in the manpage; if
> anything, I didn't know mdadm would process an array in that fashion.
> However, I don't see this behavior listed in my /proc/mdstat output.
> If it's there, I'm missing it:

auto-read-only goes away with the first write request. If you have a
file system on the RAID device, just mounting it R/W causes the superblock
to be updated and that in turn ends auto-read-only mode. Unless you have
an unused RAID device, it's not that easy to catch the moment when the
array is still in auto-read-only mode.

Gabor

-- 
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------

Re: Random MD Raid resyncs?

[Jesse Wheeler]

Understood.  I've actually done some googling since my last e-mail
reading about your suggestion.  It's quite fascinating to me.  I'll
keep an eye on this machine for the time being and stick with some
multi-level backup until my confidence level goes back up.

Thanks everyone for the insight.

On Tue, Nov 3, 2009 at 10:45 AM, Gabor Gombas <gombasg@sztaki.hu> wrote:
> On Tue, Nov 03, 2009 at 09:44:50AM -0500, Jesse Wheeler wrote:
>
>> Interesting.  I just got done reading that section in the manpage; if
>> anything, I didn't know mdadm would process an array in that fashion.
>> However, I don't see this behavior listed in my /proc/mdstat output.
>> If it's there, I'm missing it:
>
> auto-read-only goes away with the first write request. If you have a
> file system on the RAID device, just mounting it R/W causes the superblock
> to be updated and that in turn ends auto-read-only mode. Unless you have
> an unused RAID device, it's not that easy to catch the moment when the
> array is still in auto-read-only mode.
>
> Gabor
>
> --
>     ---------------------------------------------------------
>     MTA SZTAKI Computer and Automation Research Institute
>                Hungarian Academy of Sciences
>     ---------------------------------------------------------
>



-- 
Jesse W. Wheeler
mailto: jwwstpete@gmail.com
Charlotte, North Carolina - U.S.A.

==
"I'm a white male, age 18 to 49. Everyone listens to me, no matter how
dumb my suggestions are." --Homer Simpson

"I really don't understand how bipartisanship is ever going to work
when one of the parties is insane."-- John Cole

"I defy the tyranny of precedent. I cannot afford the luxury of a
closed mind. I go for anything new that might improve the past." --
Clara Barton

"Any dictator would admire the uniformity and obedience of the U.S.
media." -- Noam Chomsky

"We are called to speak for the weak, for the voiceless, for victims
of our nation and for those it calls enemy...". -- Martin Luther King,
"Beyond Vietnam"

"Be yourself; everyone else is already taken." - Oscar Wilde

* My Other Car is a Health Insurance Payment
* My Car Has Better Insurance Than I Do
* My Death Panel is an HMO
* Underinsured Baby on Board
* WWJD:   Who Would Jesus Deny? (Healthcare Reform Now!)

"Quitbull with Lipstick: She's a gift that keeps on giving; an
Everlasting Gobstopper of fail." -- SilentBrook on DKos talking about
Palin
==
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html