Hello,

So, I have a bad situation.  I run a raid5 array with 3 drives, I noticed one had fallen out of the array.  I need to setup better monitoring, it turns out this was quite some time ago (back in Nov!).  This leaves two drives and puts me into a bit of a scare.  So I decide to move the important data off to a different array.  I created a dir in /mnt made a new LVM partition formatted it as EXT4 and started syncing stuff over...  Kinda forgot to actually mount it though so all that data was syncing right back into the bad array (woops!).  Server died this morning, I assume the extra stress may have done something with the drives, or perhaps it filled up root and panicked.  In either case I could not boot.  Setup a fresh OS on my other raid array and got some tools installed and now I am working on trying to assemble the bad raid array enough to pull out my data.  My data is contained within an LVM within the raid array.

I have attached the examine and mdadm of the drives, as you can see ddd has a really old update time.  This drive was having lots of i/o errors.  So I want to use sde and sdf to assemble a read-only array, assemble the LVM, mount then copy my important data off.  They are 185 events different so I assume there will be some slight data corruption.  But I am hoping its mostly fine and likely part of my bad rsync.

So unfortunately I don't know what mdadm version was used to make this array or the OS version as that's all stuck on the dead array.  Here is what I am running on my new install I am using to try and recover the data:
Linux 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux
mdadm - v4.1 - 2018-10-01



The initial -D looked like this:

```
root@kglhost-1:~# mdadm -D /dev/md1
/dev/md1:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 3
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 3

              Name : blah:1
              UUID : fba7c062:e352fa39:fdc09bf9:e21c4617
            Events : 18094545

    Number   Major   Minor   RaidDevice

       -       8       82        -        /dev/sdf2
       -       8       66        -        /dev/sde2
       -       8       50        -        /dev/sdd2
```

I tried to run the array but that failed:

```
# mdadm -o -R  /dev/md1
mdadm: failed to start array /dev/md/1: Input/output error
```

In dmesg it says

```
[Tue Feb  2 21:14:42 2021] md: kicking non-fresh sdd2 from array!
[Tue Feb  2 21:14:42 2021] md: kicking non-fresh sdf2 from array!
[Tue Feb  2 21:14:42 2021] md/raid:md1: device sde2 operational as raid disk 1
[Tue Feb  2 21:14:42 2021] md/raid:md1: not enough operational devices (2/3 failed)
[Tue Feb  2 21:14:42 2021] md/raid:md1: failed to run raid set.
[Tue Feb  2 21:14:42 2021] md: pers->run() failed ...
```

That made the array look like so:

```
# mdadm -D /dev/md1
/dev/md1:
           Version : 1.2
     Creation Time : Thu Jul 30 21:34:20 2015
        Raid Level : raid5
     Used Dev Size : 489615360 (466.93 GiB 501.37 GB)
      Raid Devices : 3
     Total Devices : 1
       Persistence : Superblock is persistent

       Update Time : Tue Feb  2 13:55:02 2021
             State : active, FAILED, Not Started
    Active Devices : 1
   Working Devices : 1
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : unknown

              Name : blah:1
              UUID : fba7c062:e352fa39:fdc09bf9:e21c4617
            Events : 18094730

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       -       0        0        1      removed
       -       0        0        2      removed

       -       8       66        1      sync   /dev/sde2
```

I was hoping that assume-clean might be helpful, but seems I can't assemble with that option

```
# mdadm --assemble --assume-clean -o  /dev/md1 /dev/sde /dev/sdf
mdadm: :option --assume-clean not valid in assemble mode
```

So I tried a more normal assemble but it does not have enough drives to start the array:

```
# mdadm --assemble -o  /dev/md1 /dev/sde2 /dev/sdf2
mdadm: /dev/md1 assembled from 1 drive - not enough to start the array.
```


mdstat looked like this awhile ago:

```
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : inactive sdf2[2](S) sde2[1](S)
      979230720 blocks super 1.2

md2 : active raid5 sdb1[1] sda1[0] sdc1[2]
      3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 1/15 pages [4KB], 65536KB chunk

md0 : active raid1 sde1[4]
      242624 blocks super 1.0 [3/1] [_U_]

unused devices: <none>
```

Now it looks like so
```
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : inactive sdf2[2](S) sde2[1](S)
      979230720 blocks super 1.2

md2 : active raid5 sdb1[1] sda1[0] sdc1[2]
      3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

md0 : active raid1 sde1[4]
      242624 blocks super 1.0 [3/1] [_U_]

unused devices: <none>
```


I am really concerned about --force...  and https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn does nothing to alleviate those fears.

Anyone have suggestions on what to do next?
Thanks!