All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Justin Stephenson" <justin@evensteveninc.com>
To: linux-raid@vger.kernel.org
Subject: RAID 6 crashes system when being accessed
Date: Fri, 04 Jul 2014 14:11:55 +0000	[thread overview]
Message-ID: <emb6d3f02d-046d-4466-87c3-f86175b9b08a@littlez> (raw)

Hello,

I am experiencing some issues with my md raid. It is crashing my system 
when accessed with any "verve". The reboot initiates a resync of the 
raid. I have gone through the crash/reboot/resynced a number of times 
now and the crash happens within minutes of mounting the raid.

Here are some details:

- It is a raid 6 with 7 3TB devices.
- Formatted as EXT4
- mdadm v3.2.6 - 25th October 2012
- centos 6.5 kernel 2.6.32-431.3.1.el6.x86_64
- It has been running flawlessly for the previous 6 months.
- I have a cron script running that resyncs monthly.
- When the raid is unmounted, the system runs fine. (I have an 
additional "dumb" hardware raid 1 for dailies attached to an ESATA port. 
This runs perfectly).
- I am in the process of re-syncing the raid 6 again right now.
- I have run an fsck on the raid volume after it was fully synced and 
everything came up clean.

- there have been lots of power outages the last while with the hot 
summer in Toronto. My UPS shuts the system down for me, though I think I 
can correlate the issues with the power outages.

Below are the printouts from /proc/mdstat and mdadm --examine

I am not very experienced with mdadm and am a newbie with linux. Any 
assistance you might be able to offer would be appreciated.

Best,

- Justin

-----------------------------
/proc/mdstat

Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdg1[4] sdh1[5] sdf1[3] sdc1[0] sde1[2] sdd1[1] 
sdb1[6]
       14650666880 blocks super 1.2 level 6, 128k chunk, algorithm 2 
[7/7] [UUUUUUU]
       [>....................] resync = 4.7% (138332896/2930133376) 
finish=353.0min speed=131774K/sec
------------------------------
mdadm --examine

[root@BigBlue Desktop]# mdadm --examine /dev/sd[b-h]1
/dev/sdb1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 0849c677:64e4772e:8892d80b:47e0097a
            Name : BigBlue:0 (local to host BigBlue)
   Creation Time : Wed Jan 15 22:46:32 2014
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860267024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650666880 (13971.97 GiB 15002.28 GB)
   Used Dev Size : 5860266752 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
           State : active
     Device UUID : 7f7ace48:01a4b6f6:bf182ae3:6acddef9

     Update Time : Fri Jul 4 09:26:41 2014
        Checksum : a1aab985 - correct
          Events : 10631

          Layout : left-symmetric
      Chunk Size : 128K

    Device Role : Active device 6
    Array State : AAAAAAA ('A' == active, '.' == missing)
/dev/sdc1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 0849c677:64e4772e:8892d80b:47e0097a
            Name : BigBlue:0 (local to host BigBlue)
   Creation Time : Wed Jan 15 22:46:32 2014
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860267024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650666880 (13971.97 GiB 15002.28 GB)
   Used Dev Size : 5860266752 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
           State : active
     Device UUID : 8a4d11bd:fe387740:1edc1151:b0ebba43

     Update Time : Fri Jul 4 09:26:41 2014
        Checksum : 1772032b - correct
          Events : 10631

          Layout : left-symmetric
      Chunk Size : 128K

    Device Role : Active device 0
    Array State : AAAAAAA ('A' == active, '.' == missing)
/dev/sdd1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 0849c677:64e4772e:8892d80b:47e0097a
            Name : BigBlue:0 (local to host BigBlue)
   Creation Time : Wed Jan 15 22:46:32 2014
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860267024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650666880 (13971.97 GiB 15002.28 GB)
   Used Dev Size : 5860266752 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
           State : active
     Device UUID : aa158e32:f4449a9b:73e95000:22852278

     Update Time : Fri Jul 4 09:26:41 2014
        Checksum : cbb87e08 - correct
          Events : 10631

          Layout : left-symmetric
      Chunk Size : 128K

    Device Role : Active device 1
    Array State : AAAAAAA ('A' == active, '.' == missing)
/dev/sde1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 0849c677:64e4772e:8892d80b:47e0097a
            Name : BigBlue:0 (local to host BigBlue)
   Creation Time : Wed Jan 15 22:46:32 2014
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860267024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650666880 (13971.97 GiB 15002.28 GB)
   Used Dev Size : 5860266752 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
           State : active
     Device UUID : 7bee5d42:cb66c602:162e88f1:e140447c

     Update Time : Fri Jul 4 09:26:41 2014
        Checksum : 380d7914 - correct
          Events : 10631

          Layout : left-symmetric
      Chunk Size : 128K

    Device Role : Active device 2
    Array State : AAAAAAA ('A' == active, '.' == missing)
/dev/sdf1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 0849c677:64e4772e:8892d80b:47e0097a
            Name : BigBlue:0 (local to host BigBlue)
   Creation Time : Wed Jan 15 22:46:32 2014
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860267024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650666880 (13971.97 GiB 15002.28 GB)
   Used Dev Size : 5860266752 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
           State : active
     Device UUID : b4a7b35c:6e377711:4b291874:edb2c4d0

     Update Time : Fri Jul 4 09:26:41 2014
        Checksum : 38247032 - correct
          Events : 10631

          Layout : left-symmetric
      Chunk Size : 128K

    Device Role : Active device 3
    Array State : AAAAAAA ('A' == active, '.' == missing)
/dev/sdg1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 0849c677:64e4772e:8892d80b:47e0097a
            Name : BigBlue:0 (local to host BigBlue)
   Creation Time : Wed Jan 15 22:46:32 2014
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860267024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650666880 (13971.97 GiB 15002.28 GB)
   Used Dev Size : 5860266752 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
           State : active
     Device UUID : 7c0bba5a:6b205302:d026a78d:91f5ebdd

     Update Time : Fri Jul 4 09:26:41 2014
        Checksum : 4dbcfd29 - correct
          Events : 10631

          Layout : left-symmetric
      Chunk Size : 128K

    Device Role : Active device 4
    Array State : AAAAAAA ('A' == active, '.' == missing)
/dev/sdh1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 0849c677:64e4772e:8892d80b:47e0097a
            Name : BigBlue:0 (local to host BigBlue)
   Creation Time : Wed Jan 15 22:46:32 2014
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860267024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650666880 (13971.97 GiB 15002.28 GB)
   Used Dev Size : 5860266752 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
           State : active
     Device UUID : a750a146:7ee7383e:2929d803:2bee9dc1

     Update Time : Fri Jul 4 09:26:41 2014
        Checksum : cf6d0452 - correct
          Events : 10631

          Layout : left-symmetric
      Chunk Size : 128K

    Device Role : Active device 5
    Array State : AAAAAAA ('A' == active, '.' == missing)





             reply	other threads:[~2014-07-04 14:11 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-07-04 14:11 Justin Stephenson [this message]
2014-07-04 19:34 ` RAID 6 crashes system when being accessed Stan Hoeppner
2014-07-05  1:08   ` Re[2]: " Justin Stephenson
2014-07-05  4:17     ` Roger Heflin
2014-07-05 19:22       ` Re[4]: " Justin Stephenson
2014-07-05 20:42         ` Roger Heflin
2014-07-07  0:54           ` Re[6]: " Justin Stephenson
2014-07-07  1:56             ` Roger Heflin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=emb6d3f02d-046d-4466-87c3-f86175b9b08a@littlez \
    --to=justin@evensteveninc.com \
    --cc=linux-raid@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.