* Random MD Raid resyncs? [not found] <f1ef83020911020655kf8935c5icf9aea31edb03c57@mail.gmail.com> @ 2009-11-02 14:57 ` Jesse Wheeler 2009-11-02 22:23 ` NeilBrown 0 siblings, 1 reply; 7+ messages in thread From: Jesse Wheeler @ 2009-11-02 14:57 UTC (permalink / raw) To: linux-raid All: Thanks for the invaluable resource that is this mailing list. I've learned quite a bit over the space of this month. I have the following configuration and question: -- SNIP -- == Linux version 2.6.18-164.2.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Wed Sep 30 12:52:46 EDT 2009. CentOS 5.4 == Personalities : [raid10] [raid6] [raid5] [raid4] md1 : active raid5 sda2[0] sdb1[1] sdc1[2] sdd1[3] sde1[4] sdf1[5] 393230080 blocks level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU] md2 : active raid5 sda4[0] sdb4[1] sdc3[2] sdd3[3] sde3[4] sdf4[5] 424034560 blocks level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU] [===>.................] resync = 17.4% (14793792/84806912) finish=19.4min speed=60024K/sec md0 : active raid10 sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1] sda3[0] 235938048 blocks 256K chunks 2 near-copies [6/6] [UUUUUU] == [root@fatboy log]# mdadm --detail /dev/md0 /dev/md0: Version : 0.90 Creation Time : Sun Nov 1 11:24:00 2009 Raid Level : raid10 Array Size : 235938048 (225.01 GiB 241.60 GB) Used Dev Size : 78646016 (75.00 GiB 80.53 GB) Raid Devices : 6 Total Devices : 6 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Mon Nov 2 09:46:48 2009 State : clean Active Devices : 6 Working Devices : 6 Failed Devices : 0 Spare Devices : 0 Layout : near=2 Chunk Size : 256K UUID : 0d994fe1:d4b4be70:3fe15b01:0e4bd6cb Events : 0.4 Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 1 8 18 1 active sync /dev/sdb2 2 8 34 2 active sync /dev/sdc2 3 8 50 3 active sync /dev/sdd2 4 8 66 4 active sync /dev/sde2 5 8 82 5 active sync /dev/sdf2 == [root@fatboy log]# mdadm --detail /dev/md1 /dev/md1: Version : 0.90 Creation Time : Sun Nov 1 11:28:46 2009 Raid Level : raid5 Array Size : 393230080 (375.01 GiB 402.67 GB) Used Dev Size : 78646016 (75.00 GiB 80.53 GB) Raid Devices : 6 Total Devices : 6 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Mon Nov 2 09:39:51 2009 State : clean Active Devices : 6 Working Devices : 6 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 256K UUID : f537dfcb:fe7bd1a9:68316ddd:10b55eed Events : 0.14 Number Major Minor RaidDevice State 0 8 2 0 active sync /dev/sda2 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1 4 8 65 4 active sync /dev/sde1 5 8 81 5 active sync /dev/sdf1 == [root@fatboy log]# mdadm --detail /dev/md2 /dev/md2: Version : 0.90 Creation Time : Sun Nov 1 11:25:03 2009 Raid Level : raid5 Array Size : 424034560 (404.39 GiB 434.21 GB) Used Dev Size : 84806912 (80.88 GiB 86.84 GB) Raid Devices : 6 Total Devices : 6 Preferred Minor : 2 Persistence : Superblock is persistent Update Time : Mon Nov 2 08:10:46 2009 State : clean, resyncing Active Devices : 6 Working Devices : 6 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 256K Rebuild Status : 33% complete UUID : 0030789c:58c839d6:6529cdec:6347f0f3 Events : 0.64 Number Major Minor RaidDevice State 0 8 4 0 active sync /dev/sda4 1 8 20 1 active sync /dev/sdb4 2 8 35 2 active sync /dev/sdc3 3 8 51 3 active sync /dev/sdd3 4 8 67 4 active sync /dev/sde3 5 8 84 5 active sync /dev/sdf4 == -- END SNIP -- My question (please forgive the ignorance if it is showing -- I'm used to FreeBSD and Solaris) is this.. I am noticing that my MD0 (Raid10), MD1 (Raid5), MD2 (Raid 5) arrays "seem" to be resyncing at somewhat random intervals. Mind you, I have also noticed that the kernel has put a 'governor' on the resync process as to limit performance issues. Is this normal behavior? I've checked CRON thinking that the resync might be a scheduled occurance, as well as my /var/log/messages. The only indication in the SYSLOG are events like: -- SNIP -- Nov 2 09:15:42 fatboy dnsmasq[3648]: exiting on receipt of SIGTERM Nov 2 09:15:43 fatboy dnsmasq[6922]: started, version 2.45 cachesize 150 Nov 2 09:15:43 fatboy dnsmasq[6922]: compile time options: IPv6 GNU-getopt no-ISC-leasefile no-DBus no-I18N TFTP Nov 2 09:15:43 fatboy dnsmasq[6922]: using local addresses only for domain devotioconsulting.net Nov 2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.220.220#53 Nov 2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.222.222#53 Nov 2 09:15:43 fatboy dnsmasq[6922]: read /etc/hosts - 14 addresses Nov 2 09:21:04 fatboy kernel: md: syncing RAID array md1 Nov 2 09:21:04 fatboy kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. Nov 2 09:21:04 fatboy kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction. Nov 2 09:21:04 fatboy kernel: md: using 128k window, over a total of 78646016 blocks. Nov 2 09:21:04 fatboy kernel: md: delaying resync of md2 until md1 has finished resync (they share one or more physical units) Nov 2 09:39:51 fatboy kernel: md: md1: sync done. -- END SNIP .. which do not indicate any outward hardware reason (i.e. LMSensors, my UPS Monitoring daemon, etc) are not picking up any events to trigger the resync process. Is this normal? I would think it somewhat odd to have an array just 'resync' randomly. Please let me know if I am missing something. -- Jesse W. Wheeler Member-Owner Devotio Consulting, L.L.C. -- -- Jesse W. Wheeler mailto: jwwstpete@gmail.com Charlotte, North Carolina - U.S.A. == "I'm a white male, age 18 to 49. Everyone listens to me, no matter how dumb my suggestions are." --Homer Simpson "I really don't understand how bipartisanship is ever going to work when one of the parties is insane."-- John Cole "I defy the tyranny of precedent. I cannot afford the luxury of a closed mind. I go for anything new that might improve the past." -- Clara Barton "Any dictator would admire the uniformity and obedience of the U.S. media." -- Noam Chomsky "We are called to speak for the weak, for the voiceless, for victims of our nation and for those it calls enemy...". -- Martin Luther King, "Beyond Vietnam" "Be yourself; everyone else is already taken." - Oscar Wilde * My Other Car is a Health Insurance Payment * My Car Has Better Insurance Than I Do * My Death Panel is an HMO * Underinsured Baby on Board * WWJD: Who Would Jesus Deny? (Healthcare Reform Now!) "Quitbull with Lipstick: She's a gift that keeps on giving; an Everlasting Gobstopper of fail." -- SilentBrook on DKos talking about Palin == -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random MD Raid resyncs? 2009-11-02 14:57 ` Random MD Raid resyncs? Jesse Wheeler @ 2009-11-02 22:23 ` NeilBrown 2009-11-02 23:40 ` Jesse Wheeler 0 siblings, 1 reply; 7+ messages in thread From: NeilBrown @ 2009-11-02 22:23 UTC (permalink / raw) To: Jesse Wheeler; +Cc: linux-raid On Tue, November 3, 2009 1:57 am, Jesse Wheeler wrote: > All: > > Thanks for the invaluable resource that is this mailing list. I've > learned quite a bit over the space of this month. > > I have the following configuration and question: > ..... > > Nov 2 09:15:42 fatboy dnsmasq[3648]: exiting on receipt of SIGTERM > Nov 2 09:15:43 fatboy dnsmasq[6922]: started, version 2.45 cachesize 150 > Nov 2 09:15:43 fatboy dnsmasq[6922]: compile time options: IPv6 > GNU-getopt no-ISC-leasefile no-DBus no-I18N TFTP > Nov 2 09:15:43 fatboy dnsmasq[6922]: using local addresses only for > domain devotioconsulting.net > Nov 2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.220.220#53 > Nov 2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.222.222#53 > Nov 2 09:15:43 fatboy dnsmasq[6922]: read /etc/hosts - 14 addresses > Nov 2 09:21:04 fatboy kernel: md: syncing RAID array md1 The only possible explanation for this that I can think of is the the array was started in 'auto-read-only' mode in which it will not write to the devices at all until a write request comes in through the top level device. Once that write request arrives, md accepts that the array really is meant to be used and starts any resync etc that might be needed. How is /dev/md1 used? Is it possible that nothing tried to write to it until this moment? If it was something started by a cron job, it would not say "resync =" but rather 'check =' or 'repair ='. NeilBrown -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random MD Raid resyncs? 2009-11-02 22:23 ` NeilBrown @ 2009-11-02 23:40 ` Jesse Wheeler 2009-11-03 10:05 ` Gabor Gombas 0 siblings, 1 reply; 7+ messages in thread From: Jesse Wheeler @ 2009-11-02 23:40 UTC (permalink / raw) To: NeilBrown, linux-raid Hi Neil: Thanks for the help. AFAIK, any of the RAID devices are not mounted 'auto-read-only' by the kernel. I'll double check /etc/fstab, /var/log/messages, and /var/log/dmesg when I get back to the site. Unless there was a change by the upstream vendor, I wouldn't mount a production filesystem in that manner. Of course, I've been surprised before! I'll double check and report back. Thanks again for the suggestion! On Mon, Nov 2, 2009 at 5:23 PM, NeilBrown <neilb@suse.de> wrote: > On Tue, November 3, 2009 1:57 am, Jesse Wheeler wrote: >> All: >> >> Thanks for the invaluable resource that is this mailing list. I've >> learned quite a bit over the space of this month. >> >> I have the following configuration and question: >> > ..... >> >> Nov 2 09:15:42 fatboy dnsmasq[3648]: exiting on receipt of SIGTERM >> Nov 2 09:15:43 fatboy dnsmasq[6922]: started, version 2.45 cachesize 150 >> Nov 2 09:15:43 fatboy dnsmasq[6922]: compile time options: IPv6 >> GNU-getopt no-ISC-leasefile no-DBus no-I18N TFTP >> Nov 2 09:15:43 fatboy dnsmasq[6922]: using local addresses only for >> domain devotioconsulting.net >> Nov 2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.220.220#53 >> Nov 2 09:15:43 fatboy dnsmasq[6922]: using nameserver 208.67.222.222#53 >> Nov 2 09:15:43 fatboy dnsmasq[6922]: read /etc/hosts - 14 addresses >> Nov 2 09:21:04 fatboy kernel: md: syncing RAID array md1 > > The only possible explanation for this that I can think > of is the the array was started in 'auto-read-only' mode in which > it will not write to the devices at all until a write request comes in > through the top level device. Once that write request arrives, md accepts > that the array really is meant to be used and starts any resync etc that > might be needed. > > How is /dev/md1 used? Is it possible that nothing tried to > write to it until this moment? > > If it was something started by a cron job, it would not say > "resync =" but rather 'check =' or 'repair ='. > > NeilBrown > > > -- Jesse W. Wheeler mailto: jwwstpete@gmail.com Charlotte, North Carolina - U.S.A. == "I'm a white male, age 18 to 49. Everyone listens to me, no matter how dumb my suggestions are." --Homer Simpson "I really don't understand how bipartisanship is ever going to work when one of the parties is insane."-- John Cole "I defy the tyranny of precedent. I cannot afford the luxury of a closed mind. I go for anything new that might improve the past." -- Clara Barton "Any dictator would admire the uniformity and obedience of the U.S. media." -- Noam Chomsky "We are called to speak for the weak, for the voiceless, for victims of our nation and for those it calls enemy...". -- Martin Luther King, "Beyond Vietnam" "Be yourself; everyone else is already taken." - Oscar Wilde * My Other Car is a Health Insurance Payment * My Car Has Better Insurance Than I Do * My Death Panel is an HMO * Underinsured Baby on Board * WWJD: Who Would Jesus Deny? (Healthcare Reform Now!) "Quitbull with Lipstick: She's a gift that keeps on giving; an Everlasting Gobstopper of fail." -- SilentBrook on DKos talking about Palin == -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random MD Raid resyncs? 2009-11-02 23:40 ` Jesse Wheeler @ 2009-11-03 10:05 ` Gabor Gombas 2009-11-03 14:44 ` Jesse Wheeler 0 siblings, 1 reply; 7+ messages in thread From: Gabor Gombas @ 2009-11-03 10:05 UTC (permalink / raw) To: Jesse Wheeler; +Cc: NeilBrown, linux-raid On Mon, Nov 02, 2009 at 06:40:59PM -0500, Jesse Wheeler wrote: > AFAIK, any of the RAID devices are not mounted 'auto-read-only' by the > kernel. I'll double check /etc/fstab, /var/log/messages, and > /var/log/dmesg when I get back to the site. Unless there was a change > by the upstream vendor, I wouldn't mount a production filesystem in > that manner. Of course, I've been surprised before! auto-read-only is an internal state of the RAID array, it has nothing to do with file systems. So you probably won't find anything in the locations you mention above; auto-read-only is only displayed in /proc/mdstat. Look at the mdadm man page and read the sections "Auto Assembly" and "INCREMENTAL MODE". Gabor -- --------------------------------------------------------- MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences --------------------------------------------------------- ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random MD Raid resyncs? 2009-11-03 10:05 ` Gabor Gombas @ 2009-11-03 14:44 ` Jesse Wheeler 2009-11-03 15:45 ` Gabor Gombas 0 siblings, 1 reply; 7+ messages in thread From: Jesse Wheeler @ 2009-11-03 14:44 UTC (permalink / raw) To: Gabor Gombas; +Cc: NeilBrown, linux-raid Hi Gabor: > auto-read-only is an internal state of the RAID array, it has nothing to > do with file systems. So you probably won't find anything in the > locations you mention above; auto-read-only is only displayed in > /proc/mdstat. Look at the mdadm man page and read the sections "Auto > Assembly" and "INCREMENTAL MODE". Interesting. I just got done reading that section in the manpage; if anything, I didn't know mdadm would process an array in that fashion. However, I don't see this behavior listed in my /proc/mdstat output. If it's there, I'm missing it: -- SNIP -- # cat /proc/mdstat Personalities : [raid10] [raid6] [raid5] [raid4] md1 : active raid5 sda2[0] sdb1[1] sdc1[2] sdd1[3] sde1[4] sdf1[5] 393230080 blocks level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU] md2 : active raid5 sda4[0] sdb4[1] sdc3[2] sdd3[3] sde3[4] sdf4[5] 424034560 blocks level 5, 256k chunk, algorithm 2 [6/6] [UUUUUU] md0 : active raid10 sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1] sda3[0] 235938048 blocks 256K chunks 2 near-copies [6/6] [UUUUUU] unused devices: <none> -- END SNIP -- On Tue, Nov 3, 2009 at 5:05 AM, Gabor Gombas <gombasg@sztaki.hu> wrote: > On Mon, Nov 02, 2009 at 06:40:59PM -0500, Jesse Wheeler wrote: > >> AFAIK, any of the RAID devices are not mounted 'auto-read-only' by the >> kernel. I'll double check /etc/fstab, /var/log/messages, and >> /var/log/dmesg when I get back to the site. Unless there was a change >> by the upstream vendor, I wouldn't mount a production filesystem in >> that manner. Of course, I've been surprised before! > > auto-read-only is an internal state of the RAID array, it has nothing to > do with file systems. So you probably won't find anything in the > locations you mention above; auto-read-only is only displayed in > /proc/mdstat. Look at the mdadm man page and read the sections "Auto > Assembly" and "INCREMENTAL MODE". > > Gabor > > -- > --------------------------------------------------------- > MTA SZTAKI Computer and Automation Research Institute > Hungarian Academy of Sciences > --------------------------------------------------------- > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random MD Raid resyncs? 2009-11-03 14:44 ` Jesse Wheeler @ 2009-11-03 15:45 ` Gabor Gombas 2009-11-03 18:25 ` Jesse Wheeler 0 siblings, 1 reply; 7+ messages in thread From: Gabor Gombas @ 2009-11-03 15:45 UTC (permalink / raw) To: Jesse Wheeler; +Cc: NeilBrown, linux-raid On Tue, Nov 03, 2009 at 09:44:50AM -0500, Jesse Wheeler wrote: > Interesting. I just got done reading that section in the manpage; if > anything, I didn't know mdadm would process an array in that fashion. > However, I don't see this behavior listed in my /proc/mdstat output. > If it's there, I'm missing it: auto-read-only goes away with the first write request. If you have a file system on the RAID device, just mounting it R/W causes the superblock to be updated and that in turn ends auto-read-only mode. Unless you have an unused RAID device, it's not that easy to catch the moment when the array is still in auto-read-only mode. Gabor -- --------------------------------------------------------- MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences --------------------------------------------------------- ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random MD Raid resyncs? 2009-11-03 15:45 ` Gabor Gombas @ 2009-11-03 18:25 ` Jesse Wheeler 0 siblings, 0 replies; 7+ messages in thread From: Jesse Wheeler @ 2009-11-03 18:25 UTC (permalink / raw) To: Gabor Gombas; +Cc: NeilBrown, linux-raid Understood. I've actually done some googling since my last e-mail reading about your suggestion. It's quite fascinating to me. I'll keep an eye on this machine for the time being and stick with some multi-level backup until my confidence level goes back up. Thanks everyone for the insight. On Tue, Nov 3, 2009 at 10:45 AM, Gabor Gombas <gombasg@sztaki.hu> wrote: > On Tue, Nov 03, 2009 at 09:44:50AM -0500, Jesse Wheeler wrote: > >> Interesting. I just got done reading that section in the manpage; if >> anything, I didn't know mdadm would process an array in that fashion. >> However, I don't see this behavior listed in my /proc/mdstat output. >> If it's there, I'm missing it: > > auto-read-only goes away with the first write request. If you have a > file system on the RAID device, just mounting it R/W causes the superblock > to be updated and that in turn ends auto-read-only mode. Unless you have > an unused RAID device, it's not that easy to catch the moment when the > array is still in auto-read-only mode. > > Gabor > > -- > --------------------------------------------------------- > MTA SZTAKI Computer and Automation Research Institute > Hungarian Academy of Sciences > --------------------------------------------------------- > -- Jesse W. Wheeler mailto: jwwstpete@gmail.com Charlotte, North Carolina - U.S.A. == "I'm a white male, age 18 to 49. Everyone listens to me, no matter how dumb my suggestions are." --Homer Simpson "I really don't understand how bipartisanship is ever going to work when one of the parties is insane."-- John Cole "I defy the tyranny of precedent. I cannot afford the luxury of a closed mind. I go for anything new that might improve the past." -- Clara Barton "Any dictator would admire the uniformity and obedience of the U.S. media." -- Noam Chomsky "We are called to speak for the weak, for the voiceless, for victims of our nation and for those it calls enemy...". -- Martin Luther King, "Beyond Vietnam" "Be yourself; everyone else is already taken." - Oscar Wilde * My Other Car is a Health Insurance Payment * My Car Has Better Insurance Than I Do * My Death Panel is an HMO * Underinsured Baby on Board * WWJD: Who Would Jesus Deny? (Healthcare Reform Now!) "Quitbull with Lipstick: She's a gift that keeps on giving; an Everlasting Gobstopper of fail." -- SilentBrook on DKos talking about Palin == -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2009-11-03 18:25 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <f1ef83020911020655kf8935c5icf9aea31edb03c57@mail.gmail.com> 2009-11-02 14:57 ` Random MD Raid resyncs? Jesse Wheeler 2009-11-02 22:23 ` NeilBrown 2009-11-02 23:40 ` Jesse Wheeler 2009-11-03 10:05 ` Gabor Gombas 2009-11-03 14:44 ` Jesse Wheeler 2009-11-03 15:45 ` Gabor Gombas 2009-11-03 18:25 ` Jesse Wheeler
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.