Can't start array and Negative "Used Dev Size"

Can't start array and Negative "Used Dev Size"

[Simon Matthews]

Problem 1: "Used Dev Size"
====================
Note: the system is a Gentoo box, so perhaps I have missed a kernel
configuration option or use flag to deal with large hard drives.

A week or two ago, I resized a raid1 array using 2x3TB drives. I went
through the usual routine: failed one drive, installed and partitioned
(with gdisk) the new 3TB drive, added it to the array, waited for it
to sync, then did the same for the other drive. Finally, I grew the
array to max size and resized the filesystem to its maximum size.
However, after a reboot, I got many errors such as:
EXT3-fs error (device md5): ext3_get_inode_loc: unable to read inode
block - inode=150568961, block=301137922

I tracked this down to the array being the wrong size (too small), so
I unmounted the filesystem grew the array (again) to its max size and
remounted. It seems to be working now, however, it is still syncing:
md5 : active raid1 sdd2[0] sdc2[1]
      2773437376 blocks [2/2] [UU]
      [=======>.............]  resync = 38.2% (1060384320/2773437376)
finish=357.9min speed=79766K/sec

Investigating further, both sdc2 and sdd2 show a negative "Used Dev Size":
mdadm --examine /dev/sdc2
/dev/sdc2:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 5e21499a:f5562ae2:3b3bf1a1:6e290ac2
  Creation Time : Tue May 15 16:33:14 2007
     Raid Level : raid1
  Used Dev Size : -1521529920 (2644.96 GiB 2840.00 GB)      <<<<<<< WTF???
     Array Size : 2773437376 (2644.96 GiB 2840.00 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 5

    Update Time : Tue Jun 28 21:01:14 2011
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : dfcdddaf - correct
         Events : 2222657


      Number   Major   Minor   RaidDevice State
this     1       8       34        1      active sync   /dev/sdc2

   0     0       8       50        0      active sync   /dev/sdd2
   1     1       8       34        1      active sync   /dev/sdc2

--detail shows a negative dev size also:
mdadm --detail /dev/md5
/dev/md5:
        Version : 0.90
  Creation Time : Tue May 15 16:33:14 2007
     Raid Level : raid1
     Array Size : 2773437376 (2644.96 GiB 2840.00 GB)
  Used Dev Size : -1
  <<<<<< WTF?
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 5
    Persistence : Superblock is persistent

    Update Time : Tue Jun 28 21:01:14 2011
          State : active, resyncing
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

 Rebuild Status : 38% complete

           UUID : 5e21499a:f5562ae2:3b3bf1a1:6e290ac2
         Events : 0.2222657

    Number   Major   Minor   RaidDevice State
       0       8       50        0      active sync   /dev/sdd2
       1       8       34        1      active sync   /dev/sdc2

Since, I obviously don't want the array to shrink again and this looks
dangerous, I would appreciate advice on how to handle this problem.

Problem 2: Can't start array
====================
Whatever I do, I can't start md4:
mdadm /dev/md4 --assemble
mdadm: /dev/md4 is already in use.

/proc/mdadm:
md4 : inactive sdc1[0](S)
      58591232 blocks super 1.2

 mdadm --detail /dev/md4
mdadm: md device /dev/md4 does not appear to be active.

# mdadm --examine /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6b67311b:9732e436:07da8ce8:61e8af9c
           Name : server2:4  (local to host server2)
  Creation Time : Fri Jun 10 20:41:23 2011
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 117182464 (55.88 GiB 60.00 GB)
     Array Size : 117182320 (55.88 GiB 60.00 GB)
  Used Dev Size : 117182320 (55.88 GiB 60.00 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : f8d1f97e:b15f2e09:a7d55392:b193991a

    Update Time : Tue Jun 28 19:20:08 2011
       Checksum : f6fb6a5 - correct
         Events : 53


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)

 # mdadm --examine /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6b67311b:9732e436:07da8ce8:61e8af9c
           Name : server2:4  (local to host server2)
  Creation Time : Fri Jun 10 20:41:23 2011
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 117182464 (55.88 GiB 60.00 GB)
     Array Size : 117182320 (55.88 GiB 60.00 GB)
  Used Dev Size : 117182320 (55.88 GiB 60.00 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 44d1af39:96641daa:ee077d7b:d244ef54

    Update Time : Tue Jun 28 19:20:08 2011
       Checksum : 8e939e3f - correct
         Events : 53


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)


Thanks!
Simon

Re: Can't start array and Negative "Used Dev Size"

[NeilBrown]

On Tue, 28 Jun 2011 21:29:37 -0700 Simon Matthews
<simon.d.matthews@gmail.com> wrote:

> Problem 1: "Used Dev Size"
> ====================
> Note: the system is a Gentoo box, so perhaps I have missed a kernel
> configuration option or use flag to deal with large hard drives.
> 
> A week or two ago, I resized a raid1 array using 2x3TB drives. I went

Oopps.  That array is using 0.90 metadata which can only handle up to 2TB
devices.  The 'resize' code should catch that you are asking the impossible,
but it doesn't it seems.

You need to simply recreate the array as 1.0.
i.e.
 mdadm -S /dev/md5
 mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean

Then all should be happiness.
> 
> Problem 2: Can't start array
> ====================
> Whatever I do, I can't start md4:
> mdadm /dev/md4 --assemble
> mdadm: /dev/md4 is already in use.
> 
> /proc/mdadm:
> md4 : inactive sdc1[0](S)
>       58591232 blocks super 1.2

What do you get if you:

  mdadm -S /dev/md4
  mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
??

NeilBrown

Re: Can't start array and Negative "Used Dev Size"

[Simon Matthews]

Neil,



On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown <neilb@suse.de> wrote:
>  mdadm -S /dev/md5
>  mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean

Will I lose data if I do this? Should I use metadata 1.2 ?

>
> Then all should be happiness.
>>
>
>  mdadm -S /dev/md4
>  mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose

That solved it. The array started.

Thanks!

Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Can't start array and Negative "Used Dev Size"

[NeilBrown]

On Tue, 28 Jun 2011 22:24:41 -0700 Simon Matthews
<simon.d.matthews@gmail.com> wrote:

> Neil,
> 
> 
> 
> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown <neilb@suse.de> wrote:
> >  mdadm -S /dev/md5
> >  mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean
> 
> Will I lose data if I do this? Should I use metadata 1.2 ?

If you use 1.2 you will lose data.  If you use 1.0 you will not.

With 0.90 and 1.0 the data starts at the start of each device. so 1.0 will
see the same data as 0.90 would.

With 1.2 there is some metadata first and the start starts later, so if you
use that the data will appear in the wrong place.

NeilBrown

> 
> >
> > Then all should be happiness.
> >>
> >
> >  mdadm -S /dev/md4
> >  mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
> 
> That solved it. The array started.
> 
> Thanks!
> 
> Simon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Can't start array and Negative "Used Dev Size"

[Simon Matthews]

Neil,



On Tue, Jun 28, 2011 at 10:37 PM, NeilBrown <neilb@suse.de> wrote:
>> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown <neilb@suse.de> wrote:
>> >  mdadm -S /dev/md5
>> >  mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean
>>

Am I correct in thinking that this should be a quick operation?

Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Can't start array and Negative "Used Dev Size"

[NeilBrown]

On Tue, 28 Jun 2011 22:59:43 -0700 Simon Matthews
<simon.d.matthews@gmail.com> wrote:

> Neil,
> 
> 
> 
> On Tue, Jun 28, 2011 at 10:37 PM, NeilBrown <neilb@suse.de> wrote:
> >> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown <neilb@suse.de> wrote:
> >> >  mdadm -S /dev/md5
> >> >  mdadm -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean
> >>
> 
> Am I correct in thinking that this should be a quick operation?
> 

Yes.  Virtually instantaneous.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Can't start array and Negative "Used Dev Size"

[Simon Matthews]

Neil,



On Tue, Jun 28, 2011 at 10:24 PM, Simon Matthews
<simon.d.matthews@gmail.com> wrote:
> Neil,
>
>
>
> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown <neilb@suse.de> wrote:

>>  mdadm -S /dev/md4
>>  mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
>
> That solved it. The array started.

Do you have any idea why the array did not start when the system
booted? I also have an md6 on the same hard drives that was created at
the same time as md4, but md6 started on the boot.

Simon


>
> Thanks!
>
> Simon
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Can't start array and Negative "Used Dev Size"

[NeilBrown]

On Wed, 29 Jun 2011 08:45:33 -0700 Simon Matthews
<simon.d.matthews@gmail.com> wrote:

> Neil,
> 
> 
> 
> On Tue, Jun 28, 2011 at 10:24 PM, Simon Matthews
> <simon.d.matthews@gmail.com> wrote:
> > Neil,
> >
> >
> >
> > On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown <neilb@suse.de> wrote:
> 
> >>  mdadm -S /dev/md4
> >>  mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
> >
> > That solved it. The array started.
> 
> Do you have any idea why the array did not start when the system
> booted? I also have an md6 on the same hard drives that was created at
> the same time as md4, but md6 started on the boot.
> 

Not really ... I would need to see logs to be at all confident.

Based on the very limit info I have my best guess is that something -
probably udev - ran
   mdadm --incremental /dev/sdc1

but didn't run
   mdadm --incremental /dev/sdd1

I cannot imagine why it would do that though.

This would have the effect of leaving sdc1 as a member of md4, but md4 still
being inactive.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Can't start array and Negative "Used Dev Size"

[Simon Matthews]

Neil,

On Wed, Jun 29, 2011 at 5:25 PM, NeilBrown <neilb@suse.de> wrote:
> On Wed, 29 Jun 2011 08:45:33 -0700 Simon Matthews
> <simon.d.matthews@gmail.com> wrote:
>
>> Neil,
>>
>>
>>
>> On Tue, Jun 28, 2011 at 10:24 PM, Simon Matthews
>> <simon.d.matthews@gmail.com> wrote:
>> > Neil,
>> >
>> >
>> >
>> > On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown <neilb@suse.de> wrote:
>>
>> >>  mdadm -S /dev/md4
>> >>  mdadm -A /dev/md4 /dev/sdc1 /dev/sdd1 --verbose
>> >
>> > That solved it. The array started.
>>
>> Do you have any idea why the array did not start when the system
>> booted? I also have an md6 on the same hard drives that was created at
>> the same time as md4, but md6 started on the boot.
>>
>
> Not really ... I would need to see logs to be at all confident.
>
> Based on the very limit info I have my best guess is that something -
> probably udev - ran
>   mdadm --incremental /dev/sdc1
>
> but didn't run
>   mdadm --incremental /dev/sdd1
>
> I cannot imagine why it would do that though.
>
> This would have the effect of leaving sdc1 as a member of md4, but md4 still
> being inactive.
>

The system seems to take a long time to start one of the hard drives,
with many messages about doing resets. I am going to swap out the
mobile rack that the drive is installed in (it is limiting it to
1.5Gbps, instead of connecting at 3Gbps).

It still seems odd, because the other arrays that use partitions on
that disk start up.

Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Can't start array and Negative "Used Dev Size"

[Simon Matthews]

Neil,

On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 28 Jun 2011 21:29:37 -0700 Simon Matthews
> <simon.d.matthews@gmail.com> wrote:
>
>> Problem 1: "Used Dev Size"
>> ====================
>> Note: the system is a Gentoo box, so perhaps I have missed a kernel
>> configuration option or use flag to deal with large hard drives.
>>
>> A week or two ago, I resized a raid1 array using 2x3TB drives. I went
>
> Oopps.  That array is using 0.90 metadata which can only handle up to 2TB
> devices.  The 'resize' code should catch that you are asking the impossible,
> but it doesn't it seems.
>
> You need to simply recreate the array as 1.0.
> i.e.
>  mdadm -S /dev/md5
>  mdadm  -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean

Before I do this (tomorrow), do I need to add the partitions to the command:

mdadm  -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean /dev/sdd2 /dev/sdc2

Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Can't start array and Negative "Used Dev Size"

[Simon Matthews]

Neil,

On Fri, Jul 1, 2011 at 9:41 PM, Simon Matthews
<simon.d.matthews@gmail.com> wrote:
> Neil,
>
> On Tue, Jun 28, 2011 at 10:18 PM, NeilBrown <neilb@suse.de> wrote:
>> On Tue, 28 Jun 2011 21:29:37 -0700 Simon Matthews
>> <simon.d.matthews@gmail.com> wrote:
>>
>>> Problem 1: "Used Dev Size"
>>> ====================
>>> Note: the system is a Gentoo box, so perhaps I have missed a kernel
>>> configuration option or use flag to deal with large hard drives.
>>>
>>> A week or two ago, I resized a raid1 array using 2x3TB drives. I went
>>
>> Oopps.  That array is using 0.90 metadata which can only handle up to 2TB
>> devices.  The 'resize' code should catch that you are asking the impossible,
>> but it doesn't it seems.
>>
>> You need to simply recreate the array as 1.0.
>> i.e.
>>  mdadm -S /dev/md5
>>  mdadm  -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean
>
> Before I do this (tomorrow), do I need to add the partitions to the command:
>
> mdadm  -C /dev/md5 --metadata 1.0 -l1 -n2 --assume-clean /dev/sdd2 /dev/sdc2

I went ahead and did this. Everything looks good -- I think.

Why do the array sizes from --examine on my metadata 1.0 and metadata
1.2 arrays appear to be twice the size of the array:
 # mdadm --examine /dev/sde2
/dev/sde2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8f16e81f:3324004c:8d020c9b:a981e2ae
           Name : server2:7  (local to host server2)
  Creation Time : Wed Jun 29 10:39:32 2011
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 2925064957 (1394.78 GiB 1497.63 GB)          <<<
     Array Size : 2925064684 (1394.78 GiB 1497.63 GB)           <<<
how is 2925064684 equal to 1394.78 GiB?
  Used Dev Size : 2925064684 (1394.78 GiB 1497.63 GB)        <<<
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : c78ff04c:98c9ea48:77db4b85:46ac6dc1

    Update Time : Fri Jul  1 23:13:02 2011
       Checksum : e446cf2c - correct
         Events : 14


   Device Role : Active device 0
   Array State : A. ('A' == active, '.' == missing)




Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Can't start array and Negative "Used Dev Size"

[Luca Berra]

On Fri, Jul 01, 2011 at 11:19:31PM -0700, Simon Matthews wrote:
> Avail Dev Size : 2925064957 (1394.78 GiB 1497.63 GB)          <<<
>     Array Size : 2925064684 (1394.78 GiB 1497.63 GB)           <<<
>how is 2925064684 equal to 1394.78 GiB?
>  Used Dev Size : 2925064684 (1394.78 GiB 1497.63 GB)        <<<
>    Data Offset : 2048 sectors
>   Super Offset : 8 sectors

the numbers are in 512 bytes sectors
L.

-- 
Luca Berra -- bluca@comedia.it