Wrong array size detected after reboot

Wrong array size detected after reboot

[Markus Irle]

Hi guys,

After a reboot mdadm detects a much too small size for a raid5 array.
The array is part of a lvm volume group, which fails because the
device mapper notices that the physical volume is too small for the
volume group. At the moment I can't access any data.

What I did:

Array consisted of 3x 1.5TB disks.
After a disk failure a replaced the disks one by one with 3TB disks.
Resized array, volume group and partition to the new size according to
these instructions: https://raid.wiki.kernel.org/index.php/Growing

Everything went fine the array's up and running for a couple of months.
I believe this is the first reboot since.

I'm at a complete loss as what to do now. Any help is greatly appreciated.

Here's the kern.log (I don't remember the misaligned warning, when I
replaced the disks):
Sep  7 19:09:43 fandango kernel: [    3.259374] raid5: device sde1
operational as raid disk 2
Sep  7 19:09:43 fandango kernel: [    3.259378] raid5: device sdd1
operational as raid disk 1
Sep  7 19:09:43 fandango kernel: [    3.259380] raid5: device sdb1
operational as raid disk 0
Sep  7 19:09:43 fandango kernel: [    3.259727] raid5: allocated 3228kB for md2
Sep  7 19:09:43 fandango kernel: [    3.259764] raid5: raid level 5
set md2 active with 3 out of 3 devices, algorithm 2
Sep  7 19:09:43 fandango kernel: [    3.259766] RAID5 conf printout:
Sep  7 19:09:43 fandango kernel: [    3.259768]  --- rd:3 wd:3
Sep  7 19:09:43 fandango kernel: [    3.259770]  disk 0, o:1, dev:sdb1
Sep  7 19:09:43 fandango kernel: [    3.259771]  disk 1, o:1, dev:sdd1
Sep  7 19:09:43 fandango kernel: [    3.259773]  disk 2, o:1, dev:sde1
Sep  7 19:09:43 fandango kernel: [    3.259781] md2: Warning: Device
sde1 is misaligned
Sep  7 19:09:43 fandango kernel: [    3.259783] md2: Warning: Device
sdd1 is misaligned
Sep  7 19:09:43 fandango kernel: [    3.259785] md2: Warning: Device
sdb1 is misaligned
Sep  7 19:09:43 fandango kernel: [    3.259806] md2: detected capacity
change from 0 to 1603139141632
Sep  7 19:09:43 fandango kernel: [    3.261439]  md2: unknown partition table
Sep  7 19:09:43 fandango kernel: [    3.409802] device-mapper: table:
252:2: md2 too small for target: start=384, len=11721064448,
dev_size=3131131136

mdadm:
/dev/md2:
        Version : 00.90
  Creation Time : Wed Apr 29 23:31:24 2009
     Raid Level : raid5
     Array Size : 1565565568 (1493.04 GiB 1603.14 GB)
  Used Dev Size : 782782784 (746.52 GiB 801.57 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Fri Sep  7 21:30:40 2012
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : bbc76518:54235173:67388cd8:6b8a4141 (local to host fandango)
         Events : 0.16232920

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       49        1      active sync   /dev/sdd1
       2       8       65        2      active sync   /dev/sde1


The size is much too small. It should be around 6TB not 1.5TB.
Interestingly enough, the Array Size / Used Dev Size ratio seems to be correct.

Many Thanks!

Regards,
Markus

Re: Wrong array size detected after reboot

[Phil Turmel]

Hi Markus,

On 09/07/2012 04:54 PM, Markus Irle wrote:
> Hi guys,
> 
> After a reboot mdadm detects a much too small size for a raid5 array.
> The array is part of a lvm volume group, which fails because the
> device mapper notices that the physical volume is too small for the
> volume group. At the moment I can't access any data.

As soon as I read this far I thought "version 0.90?".

> What I did:
> 
> Array consisted of 3x 1.5TB disks.
> After a disk failure a replaced the disks one by one with 3TB disks.
> Resized array, volume group and partition to the new size according to
> these instructions: https://raid.wiki.kernel.org/index.php/Growing
> 
> Everything went fine the array's up and running for a couple of months.
> I believe this is the first reboot since.
> 
> I'm at a complete loss as what to do now. Any help is greatly appreciated.
> 
> Here's the kern.log (I don't remember the misaligned warning, when I
> replaced the disks):
> Sep  7 19:09:43 fandango kernel: [    3.259374] raid5: device sde1
> operational as raid disk 2
> Sep  7 19:09:43 fandango kernel: [    3.259378] raid5: device sdd1
> operational as raid disk 1
> Sep  7 19:09:43 fandango kernel: [    3.259380] raid5: device sdb1
> operational as raid disk 0
> Sep  7 19:09:43 fandango kernel: [    3.259727] raid5: allocated 3228kB for md2
> Sep  7 19:09:43 fandango kernel: [    3.259764] raid5: raid level 5
> set md2 active with 3 out of 3 devices, algorithm 2
> Sep  7 19:09:43 fandango kernel: [    3.259766] RAID5 conf printout:
> Sep  7 19:09:43 fandango kernel: [    3.259768]  --- rd:3 wd:3
> Sep  7 19:09:43 fandango kernel: [    3.259770]  disk 0, o:1, dev:sdb1
> Sep  7 19:09:43 fandango kernel: [    3.259771]  disk 1, o:1, dev:sdd1
> Sep  7 19:09:43 fandango kernel: [    3.259773]  disk 2, o:1, dev:sde1
> Sep  7 19:09:43 fandango kernel: [    3.259781] md2: Warning: Device
> sde1 is misaligned
> Sep  7 19:09:43 fandango kernel: [    3.259783] md2: Warning: Device
> sdd1 is misaligned
> Sep  7 19:09:43 fandango kernel: [    3.259785] md2: Warning: Device
> sdb1 is misaligned
> Sep  7 19:09:43 fandango kernel: [    3.259806] md2: detected capacity
> change from 0 to 1603139141632
> Sep  7 19:09:43 fandango kernel: [    3.261439]  md2: unknown partition table
> Sep  7 19:09:43 fandango kernel: [    3.409802] device-mapper: table:
> 252:2: md2 too small for target: start=384, len=11721064448,
> dev_size=3131131136
> 
> mdadm:
> /dev/md2:
>         Version : 00.90

Uh-huh.

>   Creation Time : Wed Apr 29 23:31:24 2009
>      Raid Level : raid5
>      Array Size : 1565565568 (1493.04 GiB 1603.14 GB)
>   Used Dev Size : 782782784 (746.52 GiB 801.57 GB)
>    Raid Devices : 3
>   Total Devices : 3
> Preferred Minor : 2
>     Persistence : Superblock is persistent
> 
>     Update Time : Fri Sep  7 21:30:40 2012
>           State : clean
>  Active Devices : 3
> Working Devices : 3
>  Failed Devices : 0
>   Spare Devices : 0
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>            UUID : bbc76518:54235173:67388cd8:6b8a4141 (local to host fandango)
>          Events : 0.16232920
> 
>     Number   Major   Minor   RaidDevice State
>        0       8       17        0      active sync   /dev/sdb1
>        1       8       49        1      active sync   /dev/sdd1
>        2       8       65        2      active sync   /dev/sde1
> 
> 
> The size is much too small. It should be around 6TB not 1.5TB.
> Interestingly enough, the Array Size / Used Dev Size ratio seems to be correct.

Until recently, due to a long-standing bug, devices larger than 2T were
not usable with v0.90 meta-data.  I don't remember precisely when that
bug was fixed, but I believe it was in the past year.  There are also
potential identification problems with v0.90 when used on the last
partition of a device, but you have a misalignment warning that would
prevent that.

You should see that the used dev size is very close to 2TiB less than
the actual size of your devices.

First, try the latest stable mainline kernel that you can.  That should
let you back up your data.

Second, reconstruct your array with a recent version of fdisk or parted
that won't put your first partition at sector 63.  Use metadata version
1.0 for bootable mirrored partitions, or version 1.2 otherwise (the
modern default).

HTH,

Phil

Re: Wrong array size detected after reboot

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 5027 bytes --]

On Sat, 08 Sep 2012 22:52:18 -0400 Phil Turmel <philip@turmel.org> wrote:

> Hi Markus,
> 
> On 09/07/2012 04:54 PM, Markus Irle wrote:
> > Hi guys,
> > 
> > After a reboot mdadm detects a much too small size for a raid5 array.
> > The array is part of a lvm volume group, which fails because the
> > device mapper notices that the physical volume is too small for the
> > volume group. At the moment I can't access any data.
> 
> As soon as I read this far I thought "version 0.90?".
> 
> > What I did:
> > 
> > Array consisted of 3x 1.5TB disks.
> > After a disk failure a replaced the disks one by one with 3TB disks.
> > Resized array, volume group and partition to the new size according to
> > these instructions: https://raid.wiki.kernel.org/index.php/Growing
> > 
> > Everything went fine the array's up and running for a couple of months.
> > I believe this is the first reboot since.
> > 
> > I'm at a complete loss as what to do now. Any help is greatly appreciated.
> > 
> > Here's the kern.log (I don't remember the misaligned warning, when I
> > replaced the disks):
> > Sep  7 19:09:43 fandango kernel: [    3.259374] raid5: device sde1
> > operational as raid disk 2
> > Sep  7 19:09:43 fandango kernel: [    3.259378] raid5: device sdd1
> > operational as raid disk 1
> > Sep  7 19:09:43 fandango kernel: [    3.259380] raid5: device sdb1
> > operational as raid disk 0
> > Sep  7 19:09:43 fandango kernel: [    3.259727] raid5: allocated 3228kB for md2
> > Sep  7 19:09:43 fandango kernel: [    3.259764] raid5: raid level 5
> > set md2 active with 3 out of 3 devices, algorithm 2
> > Sep  7 19:09:43 fandango kernel: [    3.259766] RAID5 conf printout:
> > Sep  7 19:09:43 fandango kernel: [    3.259768]  --- rd:3 wd:3
> > Sep  7 19:09:43 fandango kernel: [    3.259770]  disk 0, o:1, dev:sdb1
> > Sep  7 19:09:43 fandango kernel: [    3.259771]  disk 1, o:1, dev:sdd1
> > Sep  7 19:09:43 fandango kernel: [    3.259773]  disk 2, o:1, dev:sde1
> > Sep  7 19:09:43 fandango kernel: [    3.259781] md2: Warning: Device
> > sde1 is misaligned
> > Sep  7 19:09:43 fandango kernel: [    3.259783] md2: Warning: Device
> > sdd1 is misaligned
> > Sep  7 19:09:43 fandango kernel: [    3.259785] md2: Warning: Device
> > sdb1 is misaligned
> > Sep  7 19:09:43 fandango kernel: [    3.259806] md2: detected capacity
> > change from 0 to 1603139141632
> > Sep  7 19:09:43 fandango kernel: [    3.261439]  md2: unknown partition table
> > Sep  7 19:09:43 fandango kernel: [    3.409802] device-mapper: table:
> > 252:2: md2 too small for target: start=384, len=11721064448,
> > dev_size=3131131136
> > 
> > mdadm:
> > /dev/md2:
> >         Version : 00.90
> 
> Uh-huh.
> 
> >   Creation Time : Wed Apr 29 23:31:24 2009
> >      Raid Level : raid5
> >      Array Size : 1565565568 (1493.04 GiB 1603.14 GB)
> >   Used Dev Size : 782782784 (746.52 GiB 801.57 GB)
> >    Raid Devices : 3
> >   Total Devices : 3
> > Preferred Minor : 2
> >     Persistence : Superblock is persistent
> > 
> >     Update Time : Fri Sep  7 21:30:40 2012
> >           State : clean
> >  Active Devices : 3
> > Working Devices : 3
> >  Failed Devices : 0
> >   Spare Devices : 0
> > 
> >          Layout : left-symmetric
> >      Chunk Size : 64K
> > 
> >            UUID : bbc76518:54235173:67388cd8:6b8a4141 (local to host fandango)
> >          Events : 0.16232920
> > 
> >     Number   Major   Minor   RaidDevice State
> >        0       8       17        0      active sync   /dev/sdb1
> >        1       8       49        1      active sync   /dev/sdd1
> >        2       8       65        2      active sync   /dev/sde1
> > 
> > 
> > The size is much too small. It should be around 6TB not 1.5TB.
> > Interestingly enough, the Array Size / Used Dev Size ratio seems to be correct.
> 
> Until recently, due to a long-standing bug, devices larger than 2T were
> not usable with v0.90 meta-data.  I don't remember precisely when that
> bug was fixed, but I believe it was in the past year.  There are also
> potential identification problems with v0.90 when used on the last
> partition of a device, but you have a misalignment warning that would
> prevent that.

The bug was fixed in 3.1-rc6, (commit 27a7b260f71439c40546)

NeilBrown

> 
> You should see that the used dev size is very close to 2TiB less than
> the actual size of your devices.
> 
> First, try the latest stable mainline kernel that you can.  That should
> let you back up your data.
> 
> Second, reconstruct your array with a recent version of fdisk or parted
> that won't put your first partition at sector 63.  Use metadata version
> 1.0 for bootable mirrored partitions, or version 1.2 otherwise (the
> modern default).
> 
> HTH,
> 
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Wrong array size detected after reboot

[Markus Irle]

Hi guys,

Thanks for your help.

On Mon, Sep 10, 2012 at 1:13 AM, NeilBrown <neilb@suse.de> wrote:
> On Sat, 08 Sep 2012 22:52:18 -0400 Phil Turmel <philip@turmel.org> wrote:

**snip**

>> Until recently, due to a long-standing bug, devices larger than 2T were
>> not usable with v0.90 meta-data.  I don't remember precisely when that
>> bug was fixed, but I believe it was in the past year.  There are also
>> potential identification problems with v0.90 when used on the last
>> partition of a device, but you have a misalignment warning that would
>> prevent that.
>
> The bug was fixed in 3.1-rc6, (commit 27a7b260f71439c40546)

I'm running 3.2 (3.2.0-31-generic, latest in current Ubuntu release 12.04) now.

>> You should see that the used dev size is very close to 2TiB less than
>> the actual size of your devices.
>>
>> First, try the latest stable mainline kernel that you can.  That should
>> let you back up your data.

Unfortunately nothing's changed.

The array still thinks it's small:
[    1.693557]  --- level:5 rd:3 wd:3
[    1.693559]  disk 0, o:1, dev:sdb1
[    1.693561]  disk 1, o:1, dev:sdd1
[    1.693562]  disk 2, o:1, dev:sde1
[    1.693593] md2: detected capacity change from 0 to 1603139141632

and the vg doesn't start:
[    1.778499] device-mapper: table: 252:0: md2 too small for target:
start=384, len=11721064448, dev_size=3131131136

Cheers,
Markus

Re: Wrong array size detected after reboot

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1970 bytes --]

On Sun, 16 Sep 2012 01:32:06 +0200 Markus Irle <tha.bear@gmail.com> wrote:

> Hi guys,
> 
> Thanks for your help.
> 
> On Mon, Sep 10, 2012 at 1:13 AM, NeilBrown <neilb@suse.de> wrote:
> > On Sat, 08 Sep 2012 22:52:18 -0400 Phil Turmel <philip@turmel.org> wrote:
> 
> **snip**
> 
> >> Until recently, due to a long-standing bug, devices larger than 2T were
> >> not usable with v0.90 meta-data.  I don't remember precisely when that
> >> bug was fixed, but I believe it was in the past year.  There are also
> >> potential identification problems with v0.90 when used on the last
> >> partition of a device, but you have a misalignment warning that would
> >> prevent that.
> >
> > The bug was fixed in 3.1-rc6, (commit 27a7b260f71439c40546)
> 
> I'm running 3.2 (3.2.0-31-generic, latest in current Ubuntu release 12.04) now.

Sorry, I got that wrong.
That commit is the one that introduced the bug.

It's fixed by 667a5313ecd7308d which will be in 3.6, and is being
back-ported to most -stable kernels, thought it doesn't seem to have arrived
in any yet.

Maybe you can ask Ubuntu to provide a kernel containing that commit?
Or compile your own?

Or find a kernel older than 3.2...

NeilBrown

> 
> >> You should see that the used dev size is very close to 2TiB less than
> >> the actual size of your devices.
> >>
> >> First, try the latest stable mainline kernel that you can.  That should
> >> let you back up your data.
> 
> Unfortunately nothing's changed.
> 
> The array still thinks it's small:
> [    1.693557]  --- level:5 rd:3 wd:3
> [    1.693559]  disk 0, o:1, dev:sdb1
> [    1.693561]  disk 1, o:1, dev:sdd1
> [    1.693562]  disk 2, o:1, dev:sde1
> [    1.693593] md2: detected capacity change from 0 to 1603139141632
> 
> and the vg doesn't start:
> [    1.778499] device-mapper: table: 252:0: md2 too small for target:
> start=384, len=11721064448, dev_size=3131131136
> 
> Cheers,
> Markus


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Wrong array size detected after reboot

[Markus Irle]

Hi Neal,

On Wed, Sep 19, 2012 at 3:42 AM, NeilBrown <neilb@suse.de> wrote:
> On Sun, 16 Sep 2012 01:32:06 +0200 Markus Irle <tha.bear@gmail.com> wrote:
>
>> I'm running 3.2 (3.2.0-31-generic, latest in current Ubuntu release 12.04) now.
>
> Sorry, I got that wrong.
> That commit is the one that introduced the bug.
>
> It's fixed by 667a5313ecd7308d which will be in 3.6, and is being
> back-ported to most -stable kernels, thought it doesn't seem to have arrived
> in any yet.
>
> Maybe you can ask Ubuntu to provide a kernel containing that commit?
> Or compile your own?
>
> Or find a kernel older than 3.2...

Ok, I'm a bit confused.

I was running 2.6.38 when I replaced the drives and grew the array.
Apparently growing went fine, because I was using the array for a
couple of weeks. But it didn't write the correct metadata (0.90), thus
reassembling after boot fails (or rather is the wrong size).

I tried booting with 3.0.0, 3.2.0 and 3.5.0 with exactly the same results.

I feel that no matter what kernel, as long as the metadata's wrong, I
can't assemble it correctly, correct?

Cheers,
Marks.

Re: Wrong array size detected after reboot

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1731 bytes --]

On Wed, 19 Sep 2012 20:34:11 +0200 Markus Irle <tha.bear@gmail.com> wrote:

> Hi Neal,
> 
> On Wed, Sep 19, 2012 at 3:42 AM, NeilBrown <neilb@suse.de> wrote:
> > On Sun, 16 Sep 2012 01:32:06 +0200 Markus Irle <tha.bear@gmail.com> wrote:
> >
> >> I'm running 3.2 (3.2.0-31-generic, latest in current Ubuntu release 12.04) now.
> >
> > Sorry, I got that wrong.
> > That commit is the one that introduced the bug.
> >
> > It's fixed by 667a5313ecd7308d which will be in 3.6, and is being
> > back-ported to most -stable kernels, thought it doesn't seem to have arrived
> > in any yet.
> >
> > Maybe you can ask Ubuntu to provide a kernel containing that commit?
> > Or compile your own?
> >
> > Or find a kernel older than 3.2...
> 
> Ok, I'm a bit confused.

Me too it would seem - sorry.  I was thinking of a RAID0 problem with similar
symptoms.

> 
> I was running 2.6.38 when I replaced the drives and grew the array.
> Apparently growing went fine, because I was using the array for a
> couple of weeks. But it didn't write the correct metadata (0.90), thus
> reassembling after boot fails (or rather is the wrong size).
> 
> I tried booting with 3.0.0, 3.2.0 and 3.5.0 with exactly the same results.
> 
> I feel that no matter what kernel, as long as the metadata's wrong, I
> can't assemble it correctly, correct?

Yes, the metadata has lots the most-significant-bit of the 'size'.

So it should be sufficient to:
  mdadm -G /dev/md2 --size 2930266432

where the size is calculated as the current "used device size" plus 2^31.
You could probably just do "--size max"

This will cause a resync of the 2G of data which is probably unnecessary, but
is awkward to avoid.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Wrong array size detected after reboot

[Markus Irle]

Hi Neil,

On Thu, Sep 20, 2012 at 1:11 AM, NeilBrown <neilb@suse.de> wrote:
>
> Yes, the metadata has lots the most-significant-bit of the 'size'.
>
> So it should be sufficient to:
>   mdadm -G /dev/md2 --size 2930266432
>
> where the size is calculated as the current "used device size" plus 2^31.
> You could probably just do "--size max"
>
> This will cause a resync of the 2G of data which is probably unnecessary, but
> is awkward to avoid.

Thanks, will try.

Resyncing won't overwrite the existing data, will it?
Sorry for being overly cautious...

Cheers,
Markus

Re: Wrong array size detected after reboot

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

On Thu, 20 Sep 2012 11:53:04 +0200 Markus Irle <tha.bear@gmail.com> wrote:

> Hi Neil,
> 
> On Thu, Sep 20, 2012 at 1:11 AM, NeilBrown <neilb@suse.de> wrote:
> >
> > Yes, the metadata has lots the most-significant-bit of the 'size'.
> >
> > So it should be sufficient to:
> >   mdadm -G /dev/md2 --size 2930266432
> >
> > where the size is calculated as the current "used device size" plus 2^31.
> > You could probably just do "--size max"
> >
> > This will cause a resync of the 2G of data which is probably unnecessary, but
> > is awkward to avoid.
> 
> Thanks, will try.
> 
> Resyncing won't overwrite the existing data, will it?

No, it won't.  It'll just check all the parity blocks are correct.

> Sorry for being overly cautious...

Not something to apologise for ....

NeilBrown

> 
> Cheers,
> Markus


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Wrong array size detected after reboot

[Markus Irle]

Hi Neil,

For the record:

It worked!

I grew to "--size max", sync kicked in, but I'm able to mount the
volumes and the data's all there.

Thanks for the help guys!

Cheers,
Markus


On Thu, Sep 20, 2012 at 1:39 PM, NeilBrown <neilb@suse.de> wrote:
> On Thu, 20 Sep 2012 11:53:04 +0200 Markus Irle <tha.bear@gmail.com> wrote:
>
>> Hi Neil,
>>
>> On Thu, Sep 20, 2012 at 1:11 AM, NeilBrown <neilb@suse.de> wrote:
>> >
>> > Yes, the metadata has lots the most-significant-bit of the 'size'.
>> >
>> > So it should be sufficient to:
>> >   mdadm -G /dev/md2 --size 2930266432
>> >
>> > where the size is calculated as the current "used device size" plus 2^31.
>> > You could probably just do "--size max"
>> >
>> > This will cause a resync of the 2G of data which is probably unnecessary, but
>> > is awkward to avoid.
>>
>> Thanks, will try.
>>
>> Resyncing won't overwrite the existing data, will it?
>
> No, it won't.  It'll just check all the parity blocks are correct.
>
>> Sorry for being overly cautious...
>
> Not something to apologise for ....
>
> NeilBrown
>
>>
>> Cheers,
>> Markus
>