Can't mount Old RHEL 6 Raid with new install of CentOS 7, now can't mount with original RHEL 6

Can't mount Old RHEL 6 Raid with new install of CentOS 7, now can't mount with original RHEL 6

[John Dawson]

I have a machine which had a drive with RHEL 6.X installed with a raid 
device setup on separate disk(s).  Installed a new hard disk in machine 
and installed CentOS 7.  CentOS 7 wouldn't mount the raid.  Put the old 
drive back in and now RHEL 6.X won't mount the raid.  Is the raid 
permanently hosed?  Can I get the data on it back? How? Thx.

JD

Re: Can't mount Old RHEL 6 Raid with new install of CentOS 7, now can't mount with original RHEL 6

[Chris Murphy]

On Thu, Aug 11, 2016 at 6:19 AM, John Dawson <linux@celticblues.com> wrote:
> I have a machine which had a drive with RHEL 6.X installed with a raid
> device setup on separate disk(s).  Installed a new hard disk in machine and
> installed CentOS 7.  CentOS 7 wouldn't mount the raid.  Put the old drive
> back in and now RHEL 6.X won't mount the raid.  Is the raid permanently
> hosed?  Can I get the data on it back? How? Thx.

RHEL comes with a support contract so you should contact Red Hat about
that part.

Also, not anywhere near enough information has been provided, almost
like you think what you're experiencing is a widely known problem with
a known solution. But it isn't. So you should provide mdadm -E
information for each member block device, whether or not the array
assembles manually, if not what error do you get in user and kernel
space, and what command you're using to mount the array that you say
fails, and what the error message is.

Also include mdadm version on both systems because few people will
have any idea what mdadm version is on the particular installation of
RHEL and CentOS you're using, as these things aren't standardized at
all across distros.

-- 
Chris Murphy

Re: Can't mount Old RHEL 6 Raid with new install of CentOS 7, now can't mount with original RHEL 6

[John Dawson]

Sorry.. my bad... I am not using RHEL... I am using CentOS...

     mdadm --version = mdadm - v3.2.3 - 23rd December 2011


When I try to assemble the raid I get the following errors:

      sudo mdadm --assemble /dev/md127 /dev/sdb1 /dev/sdc1

I get the following:

      mdadm: /dev/md127 has been started with 2 drives.

When I try to mount

      sudo mount /dev/md127 /proj

I get the following:

      mount: you must specify the filesystem type

When I specify the filesystem type,

      sudo mount -t ext4 /dev/md127 /proj/

I get the following:

      mount: wrong fs type, bad option, bad superblock on /dev/md127, 
missing codepage or helper program, or other error In some cases useful 
info is found in syslog - try       dmesg | tail  or so


===============================================
dmesg | tail
===============================================
md: md127 stopped.
md: bind<sdb1>
md: bind<sdc1>
bio: create slab <bio-1> at 1
md/raid0:md127: md_size is 3907039232 sectors.
md: RAID0 configuration for md127 - 1 zone
md: zone0=[sdc1/sdb1]
       zone-offset=         0KB, device-offset=         0KB, 
size=1953519616KB
EXT4-fs (md127): VFS: Can't find ext4 filesystem

===============================================
"sudo mdadm -D /dev/md127" results
===============================================

/dev/md127:
         Version : 1.2
   Creation Time : Fri Aug  5 16:46:10 2016
      Raid Level : raid0
      Array Size : 1953519616 (1863.02 GiB 2000.40 GB)
    Raid Devices : 2
   Total Devices : 2
     Persistence : Superblock is persistent

     Update Time : Fri Aug  5 16:46:10 2016
           State : clean
  Active Devices : 2
Working Devices : 2
  Failed Devices : 0
   Spare Devices : 0

      Chunk Size : 512K

            Name : mymachine:0  (local to host mymachine)
            UUID : 8217dfb5:a97a15df:94a85926:3fea6697
          Events : 0

     Number   Major   Minor   RaidDevice State
        0       8       33        0      active sync   /dev/sdc1
        1       8       17        1      active sync   /dev/sdb1



===============================================
"sudo mdadm --examine /dev/sdb1" results
===============================================

/dev/sdb1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 8217dfb5:a97a15df:94a85926:3fea6697
            Name : mymachine:0  (local to host mymachine)
   Creation Time : Fri Aug  5 16:46:10 2016
      Raid Level : raid0
    Raid Devices : 2

  Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
     Data Offset : 2048 sectors
    Super Offset : 8 sectors
           State : clean
     Device UUID : d9bab0d7:793e5168:4457d25b:24614a41

     Update Time : Fri Aug  5 16:46:10 2016
        Checksum : 744c405e - correct
          Events : 0

      Chunk Size : 512K

    Device Role : Active device 1
    Array State : AA ('A' == active, '.' == missing)


===============================================
"sudo mdadm --examine /dev/sdc1" results
===============================================

/dev/sdc1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 8217dfb5:a97a15df:94a85926:3fea6697
            Name : mymachine:0  (local to host mymachine)
   Creation Time : Fri Aug  5 16:46:10 2016
      Raid Level : raid0
    Raid Devices : 2

  Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
     Data Offset : 2048 sectors
    Super Offset : 8 sectors
           State : clean
     Device UUID : a0dfb805:18718fed:6985075a:12fbb196

     Update Time : Fri Aug  5 16:46:10 2016
        Checksum : 7b2f5fd6 - correct
          Events : 0

      Chunk Size : 512K

    Device Role : Active device 0
    Array State : AA ('A' == active, '.' == missing)




On 2016-08-11 20:38, Chris Murphy wrote:
> On Thu, Aug 11, 2016 at 6:19 AM, John Dawson <linux@celticblues.com> 
> wrote:
>> I have a machine which had a drive with RHEL 6.X installed with a raid
>> device setup on separate disk(s). Installed a new hard disk in machine 
>> and
>> installed CentOS 7. CentOS 7 wouldn't mount the raid. Put the old 
>> drive
>> back in and now RHEL 6.X won't mount the raid. Is the raid permanently
>> hosed? Can I get the data on it back? How? Thx.
> 
> RHEL comes with a support contract so you should contact Red Hat about
> that part.
> 
> Also, not anywhere near enough information has been provided, almost
> like you think what you're experiencing is a widely known problem with
> a known solution. But it isn't. So you should provide mdadm -E
> information for each member block device, whether or not the array
> assembles manually, if not what error do you get in user and kernel
> space, and what command you're using to mount the array that you say
> fails, and what the error message is.
> 
> Also include mdadm version on both systems because few people will
> have any idea what mdadm version is on the particular installation of
> RHEL and CentOS you're using, as these things aren't standardized at
> all across distros.

Re: Can't mount Old RHEL 6 Raid with new install of CentOS 7, now can't mount with original RHEL 6

[Chris Murphy]

On Tue, Aug 16, 2016 at 11:35 AM, John Dawson <linux@celticblues.com> wrote:

> ===============================================
> "sudo mdadm -D /dev/md127" results
> ===============================================
>
> /dev/md127:
>         Version : 1.2
>   Creation Time : Fri Aug  5 16:46:10 2016

Did you really create this array on 10 days ago with CentOS 6 and then
upgrade to CentOS 7 and it didn't work? Or did you happen to try to
fix the problem by doing mdadm --create ?





-- 
Chris Murphy

Re: Can't mount Old RHEL 6 Raid with new install of CentOS 7, now can't mount with original RHEL 6

[Chris Murphy]

On Tue, Aug 16, 2016 at 12:33 PM,  <linux@celticblues.com> wrote:
> The array was originally created well over a year ago, perhaps 2 or 3 years
> ago. It was working great. When the machine was updated to CentOS 7, the
> raid assembled manually fine and was mountable only once. After rebooting
> and trying to mount in /etc/fstab it could no longer be assembled and
> mounted...

How do you explain your mdadm -E and -D output, which shows the
creation time 10 days ago?


> ===============================================
> "sudo mdadm -D /dev/md127" results
> ===============================================
>
> /dev/md127:
> Version : 1.2
> Creation Time : Fri Aug 5 16:46:10 2016


Further the -E output shows Events: 0 which means the array you're
showing us has never been used. The only explanation I can think of is
you used mdadm --create, but something significant is missing from
your explanation because there is no possible way a normal mount or
assemble changes the Creation Time of the array, or the Event count.
But --create will do that.


-- 
Chris Murphy

Re: Can't mount Old RHEL 6 Raid with new install of CentOS 7, now can't mount with original RHEL 6

[Chris Murphy]

On Tue, Aug 16, 2016 at 12:59 PM,  <linux@celticblues.com> wrote:
> I tried to assemble the raid array again. If that doesn't explain it, then I
> don't know.


Why would trying to assemble the raid change anything? The array is
assembling correctly per the metadata on those two drives. That
there's no ext4 signature on that array, the date of creation of the
array is 10 days ago, all suggests that the original array used
metadata version other than 1.2, putting the ext4 superblock in the
location where 10 days ago mdadm --create obliterated that ext4
superblock with an mdadm 1.2 superblock.

There are a lot of "fix me" guides out there that suggest doing mdadm
--create and it's 99% of the time really horrifically bad advice that
causes data loss that looks awfully lot like what's going on here.


>The two drives that make up the array haven't been
> touched/modified.

Obviously that's not true.

The only possible chance to recover this is if you have an exact
sequence of events modifying the drive, so that they can maybe be
reversed. Without that, any change will do more damage and decrease
the chance of recovery. But seeing as this is a raid0 I suspect you
have a complete backup of the drive anyway so you're probably better
off just starting over with it.



-- 
Chris Murphy