Re: Reassembling Raid5 in degraded state

Re: Reassembling Raid5 in degraded state

[Christian Deufel]

Hey Wol

 > My plan now would be to run mdadm --assemble --force /dev/md3 with 3
 > disk, to get the Raid going in a degraded state.

 >Yup, this would almost certainly work. I would recommend overlays and
 >running a fsck just to check it's all okay before actually doing it on
 >the actual disks. The event counts say to me that you'll probably lose
 >little to nothing.

So as I was trying to reassemble my Raid it crashed again. But this time 
sdc vanished.

I get the following output:

[root@dirvish ~]# mdadm --stop /dev/md3
mdadm: stopped /dev/md3
[root@dirvish ~]# mdadm --assemble --force /dev/md3 /dev/sdc1 /dev/sdd1 
/dev/sdf            1
mdadm: forcing event count in /dev/sdc1(2) from 5995154 upto 5995162
mdadm: clearing FAULTY flag for device 0 in /dev/md3 for /dev/sdc1
mdadm: failed to add /dev/sdc1 to /dev/md3: Invalid argument
mdadm: /dev/md3 assembled from 2 drives - not enough to start the array.

and when I checked with fdisk sdc wasn't there anymore.

[root@dirvish ~]# fdisk -l

Disk /dev/sda: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   fd  Linux raid 
autodetect
/dev/sda2              14         535     4192965   fd  Linux raid 
autodetect
/dev/sda3             536       30401   239898645   fd  Linux raid 
autodetect

Disk /dev/sdb: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104391   fd  Linux raid 
autodetect
/dev/sdb2              14         535     4192965   fd  Linux raid 
autodetect
/dev/sdb3             536       30401   239898645   fd  Linux raid 
autodetect

Disk /dev/sdd: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot      Start         End      Blocks   Id  System
/dev/sdd1   *           1      243201  1953512001   fd  Linux raid 
autodetect

Disk /dev/sde: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot      Start         End      Blocks   Id  System
/dev/sde1   *           1      243201  1953512001   fd  Linux raid 
autodetect

Disk /dev/sdf: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1      243201  1953512001   83  Linux

...

After a reboot of the system sdc was back again but vanished again after 
I retried the assembly.

Would the assembly also work with my 4th HDD, sde1 although in the mdadm 
--eaxmine it is labeld as spare?

Greetings

Christian

Re: Reassembling Raid5 in degraded state

[Phil Turmel]

On 1/14/20 6:28 AM, Christian Deufel wrote:
> Hey Wol
> 
>  > My plan now would be to run mdadm --assemble --force /dev/md3 with 3
>  > disk, to get the Raid going in a degraded state.
> 
>  >Yup, this would almost certainly work. I would recommend overlays and
>  >running a fsck just to check it's all okay before actually doing it on
>  >the actual disks. The event counts say to me that you'll probably lose
>  >little to nothing.
> 
> So as I was trying to reassemble my Raid it crashed again. But this time 
> sdc vanished.

I've experienced the problem of various disks disappearing when you 
attempt to use them.  It has always been some kind of power supply 
problem in my cases.

Consider relocating these drives to another box to continue your 
recovery efforts.

Phil

Re: Reassembling Raid5 in degraded state

[Wol]

On 13/01/2020 17:31, Song Liu wrote:
>> Song, Neil,
>>
>> Just a guess as to what went wrong, but the array event count does not
>> match the disk counts. Iirc this is one of the events that cause an
> Which mismatch do you mean?
> 
>> assemble to stop. Is it possible that a crash at the wrong moment could
>> interrupt an update and trigger this problem?
> It looks like sdc1 failed first. Then sdd1 and sdf1 got events for sdc1 failed.
> Based on super block on sdd1 and sdf1, we already have two failed drive,
> so assemble stopped.
> 
> Does this answer the question?
> 
Sorry yes it does. My fault for mis-reading the logs the OP sent.

Cheers.,
Wol

Re: Reassembling Raid5 in degraded state

[Song Liu]

On Mon, Jan 13, 2020 at 7:04 AM Wols Lists <antlists@youngman.org.uk> wrote:
>
> On 13/01/20 09:41, Christian Deufel wrote:
> > My plan now would be to run mdadm --assemble --force /dev/md3 with 3
> > disk, to get the Raid going in a degraded state.
>
> Yup, this would almost certainly work. I would recommend overlays and
> running a fsck just to check it's all okay before actually doing it on
> the actual disks. The event counts say to me that you'll probably lose
> little to nothing.
>
> > Does anyone have any experience in doing so and can recommend which 3
> > disks I should use. I would use sdc1 sdd1 and sdf1, since sdd1 and sdf1
> > are displayed as active sync in every examine and sdc1 as it is also
> > displayed as active sync.
>
> Those three disks would be perfect.
>
> > Do you think that by doing it this way I have a chance to get my Data
> > back or do you have any other suggestion as to get the Data back and the
> > Raid running again?
>
> You shouldn't have any trouble, I hope. Take a look at
>
> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>
> and take note of the comment about using the latest mdadm - what version
> is yours (mdadm --version)? That might assemble the array no problem at all.
>
> Song, Neil,
>
> Just a guess as to what went wrong, but the array event count does not
> match the disk counts. Iirc this is one of the events that cause an

Which mismatch do you mean?

> assemble to stop. Is it possible that a crash at the wrong moment could
> interrupt an update and trigger this problem?

It looks like sdc1 failed first. Then sdd1 and sdf1 got events for sdc1 failed.
Based on super block on sdd1 and sdf1, we already have two failed drive,
so assemble stopped.

Does this answer the question?

Thanks,
Song

Re: Reassembling Raid5 in degraded state

[Wols Lists]

On 13/01/20 09:41, Christian Deufel wrote:
> My plan now would be to run mdadm --assemble --force /dev/md3 with 3
> disk, to get the Raid going in a degraded state.

Yup, this would almost certainly work. I would recommend overlays and
running a fsck just to check it's all okay before actually doing it on
the actual disks. The event counts say to me that you'll probably lose
little to nothing.

> Does anyone have any experience in doing so and can recommend which 3
> disks I should use. I would use sdc1 sdd1 and sdf1, since sdd1 and sdf1
> are displayed as active sync in every examine and sdc1 as it is also
> displayed as active sync.

Those three disks would be perfect.

> Do you think that by doing it this way I have a chance to get my Data
> back or do you have any other suggestion as to get the Data back and the
> Raid running again?

You shouldn't have any trouble, I hope. Take a look at

https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn

and take note of the comment about using the latest mdadm - what version
is yours (mdadm --version)? That might assemble the array no problem at all.

Song, Neil,

Just a guess as to what went wrong, but the array event count does not
match the disk counts. Iirc this is one of the events that cause an
assemble to stop. Is it possible that a crash at the wrong moment could
interrupt an update and trigger this problem?

Cheers,
Wol

Reassembling Raid5 in degraded state

[Christian Deufel]

Hi

I'm having a problem with a 4 disk Raid 5 MD-Array.
After a crash it didn't reassemble correctly, by that I mean it crashed 
during the reassemble, and cat /proc/mdstat reads as follows:

[root@dirvish ~]# cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
...
md3 : inactive sdd1[0] sde1[5] sdf1[3] sdc1[2]
       7814047744 blocks
...

mdadm --examine for the disks in the Raid reads as follows:

[root@dirvish ~]# mdadm -E /dev/sdc1
/dev/sdc1:
           Magic : a92b4efc
         Version : 0.90.00
            UUID : 84b70068:58203635:60d8aaf0:b60ee018
   Creation Time : Mon Feb 28 18:46:58 2011
      Raid Level : raid5
   Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
      Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
    Raid Devices : 4
   Total Devices : 4
Preferred Minor : 3

     Update Time : Wed Jan  8 12:26:35 2020
           State : clean
  Active Devices : 3
Working Devices : 4
  Failed Devices : 1
   Spare Devices : 1
        Checksum : bd93dda1 - correct
          Events : 5995154

          Layout : left-symmetric
      Chunk Size : 64K

       Number   Major   Minor   RaidDevice State
this     2       8       33        2      active sync   /dev/sdc1

    0     0       8       49        0      active sync   /dev/sdd1
    1     1       0        0        1      faulty removed
    2     2       8       33        2      active sync   /dev/sdc1
    3     3       8       81        3      active sync   /dev/sdf1
    4     4       8       65        4      spare   /dev/sde1
[root@dirvish ~]# mdadm -E /dev/sdd1
/dev/sdd1:
           Magic : a92b4efc
         Version : 0.90.00
            UUID : 84b70068:58203635:60d8aaf0:b60ee018
   Creation Time : Mon Feb 28 18:46:58 2011
      Raid Level : raid5
   Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
      Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
    Raid Devices : 4
   Total Devices : 3
Preferred Minor : 3

     Update Time : Wed Jan  8 13:30:15 2020
           State : clean
  Active Devices : 2
Working Devices : 2
  Failed Devices : 2
   Spare Devices : 0
        Checksum : bd93ec60 - correct
          Events : 5995162

          Layout : left-symmetric
      Chunk Size : 64K

       Number   Major   Minor   RaidDevice State
this     0       8       49        0      active sync   /dev/sdd1

    0     0       8       49        0      active sync   /dev/sdd1
    1     1       0        0        1      faulty removed
    2     2       0        0        2      faulty removed
    3     3       8       81        3      active sync   /dev/sdf1
[root@dirvish ~]# mdadm -E /dev/sde1
/dev/sde1:
           Magic : a92b4efc
         Version : 0.90.00
            UUID : 84b70068:58203635:60d8aaf0:b60ee018
   Creation Time : Mon Feb 28 18:46:58 2011
      Raid Level : raid5
   Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
      Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
    Raid Devices : 4
   Total Devices : 3
Preferred Minor : 3

     Update Time : Wed Jan  8 13:30:15 2020
           State : clean
  Active Devices : 2
Working Devices : 2
  Failed Devices : 2
   Spare Devices : 0
        Checksum : bd93ecbd - correct
          Events : 5995162

          Layout : left-symmetric
      Chunk Size : 64K

       Number   Major   Minor   RaidDevice State
this     5       8       65       -1      spare   /dev/sde1

    0     0       8       49        0      active sync   /dev/sdd1
    1     1       0        0        1      faulty removed
    2     2       0        0        2      faulty removed
    3     3       8       81        3      active sync   /dev/sdf1
[root@dirvish ~]# mdadm -E /dev/sdf1
/dev/sdf1:
           Magic : a92b4efc
         Version : 0.90.00
            UUID : 84b70068:58203635:60d8aaf0:b60ee018
   Creation Time : Mon Feb 28 18:46:58 2011
      Raid Level : raid5
   Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
      Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
    Raid Devices : 4
   Total Devices : 3
Preferred Minor : 3

     Update Time : Wed Jan  8 13:30:15 2020
           State : clean
  Active Devices : 2
Working Devices : 2
  Failed Devices : 2
   Spare Devices : 0
        Checksum : bd93ec86 - correct
          Events : 5995162

          Layout : left-symmetric
      Chunk Size : 64K

       Number   Major   Minor   RaidDevice State
this     3       8       81        3      active sync   /dev/sdf1

    0     0       8       49        0      active sync   /dev/sdd1
    1     1       0        0        1      faulty removed
    2     2       0        0        2      faulty removed
    3     3       8       81        3      active sync   /dev/sdf1

My plan now would be to run mdadm --assemble --force /dev/md3 with 3 
disk, to get the Raid going in a degraded state.
Does anyone have any experience in doing so and can recommend which 3 
disks I should use. I would use sdc1 sdd1 and sdf1, since sdd1 and sdf1 
are displayed as active sync in every examine and sdc1 as it is also 
displayed as active sync.
Do you think that by doing it this way I have a chance to get my Data 
back or do you have any other suggestion as to get the Data back and the 
Raid running again?

Greetings
Christian

  -------------------------------------------------------------------
| Christian Deufel
|
|
| inView Groupware und Datentechnik GmbH
| Emmy-Noether-Str.2
| 79110 Freiburg
|
| https://www.inView.de
| christian.deufel@inView.de
| Tel. 0761 - 45 75 48-22
| Fax. 0761 - 45 75 48-99
|
| Amtsgericht Freiburg HRB-6769
| Ust-ID DE 219531868
| Geschäftsführer: Caspar Fromelt, Frank Kopp
  -------------------------------------------------------------------