RAID0 size over 2 TB

RAID0 size over 2 TB

[azurIt]

Hi,

i'm having problems with growing my RAID0 array over 2 TB. I installed two 4 TB drives, created 4 TB partitions but array cannot be resized over 2 TB. Superblock versoin is 0.90 and kernel version is 3.2.53 - according to linux RAID wiki, it should support 4 TB component/array size:
https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format

Some more info:

# mdadm --detail /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Tue Sep 28 16:54:04 2010
     Raid Level : raid1
     Array Size : 2147483520 (2048.00 GiB 2199.02 GB)
  Used Dev Size : 2147483520 (2048.00 GiB 2199.02 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Sat Sep  6 23:04:44 2014
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 9a363004:ad04abfe:2e31e1c3:caf66df4
         Events : 0.77066

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1







# fdisk /dev/sdd

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.


WARNING: The size of this disk is 4.0 TB (4000787030016 bytes).
DOS partition table format can not be used on drives for volumes
larger than (2199023255040 bytes) for 512-byte sectors. Use parted(1) and GUID 
partition table format (GPT).


Command (m for help): p

Disk /dev/sdd: 4000.8 GB, 4000787030016 bytes
256 heads, 63 sectors/track, 484501 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1  4294967295  2147483647+  ee  GPT








# fdisk /dev/sdc

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.


WARNING: The size of this disk is 4.0 TB (4000787030016 bytes).
DOS partition table format can not be used on drives for volumes
larger than (2199023255040 bytes) for 512-byte sectors. Use parted(1) and GUID 
partition table format (GPT).


Command (m for help): p

Disk /dev/sdc: 4000.8 GB, 4000787030016 bytes
256 heads, 63 sectors/track, 484501 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1  4294967295  2147483647+  ee  GPT






# mdadm --grow /dev/md2 --size=max       
mdadm: component size of /dev/md2 unchanged at 2147483520K







# mdadm --grow /dev/md2 --size=3221225280
mdadm: Cannot set device size for /dev/md2: No space left on device






Any hints? Thank you.


azur

Re: RAID0 size over 2 TB - RAID1

[azurIt]

>Hi,
>
>i'm having problems with growing my RAID0 array over 2 TB. I installed two 4 TB drives, created 4 TB partitions but array cannot be resized over 2 TB. Superblock versoin is 0.90 and kernel version is 3.2.53 - according to linux RAID wiki, it should support 4 TB component/array size:
>https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format




I'm sorry for a mistake, it's RAID1 of course.

azur

Re: RAID0 size over 2 TB

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1653 bytes --]

On Sat, 06 Sep 2014 23:09:28 +0200 "azurIt" <azurit@pobox.sk> wrote:

> Hi,
> 
> i'm having problems with growing my RAID0 array over 2 TB. I installed two 4 TB drives, created 4 TB partitions but array cannot be resized over 2 TB. Superblock versoin is 0.90 and kernel version is 3.2.53 - according to linux RAID wiki, it should support 4 TB component/array size:
> https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format
> 
> Some more info:
> 
> # mdadm --detail /dev/md2
> /dev/md2:
>         Version : 0.90
>   Creation Time : Tue Sep 28 16:54:04 2010
>      Raid Level : raid1
>      Array Size : 2147483520 (2048.00 GiB 2199.02 GB)
>   Used Dev Size : 2147483520 (2048.00 GiB 2199.02 GB)
>    Raid Devices : 2
>   Total Devices : 2
> Preferred Minor : 2
>     Persistence : Superblock is persistent
> 
>     Update Time : Sat Sep  6 23:04:44 2014
>           State : clean 
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 0
> 
>            UUID : 9a363004:ad04abfe:2e31e1c3:caf66df4
>          Events : 0.77066
> 
>     Number   Major   Minor   RaidDevice State
>        0       8       33        0      active sync   /dev/sdc1
>        1       8       49        1      active sync   /dev/sdd1
> 
> 
> 
> 
> 
> 
> 
> # fdisk /dev/sdd
> 
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.

Why are you using "fdisk" even though "fdisk" is telling you not to use it?

What does
  cat /proc/partitions

say about the sizes of the partitions?

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID0 size over 2 TB

[azurIt]

>
> CC: linux-raid@vger.kernel.org
>On Sat, 06 Sep 2014 23:09:28 +0200 "azurIt" <azurit@pobox.sk> wrote:
>
>> Hi,
>> 
>> i'm having problems with growing my RAID0 array over 2 TB. I installed two 4 TB drives, created 4 TB partitions but array cannot be resized over 2 TB. Superblock versoin is 0.90 and kernel version is 3.2.53 - according to linux RAID wiki, it should support 4 TB component/array size:
>> https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format
>> 
>> Some more info:
>> 
>> # mdadm --detail /dev/md2
>> /dev/md2:
>>         Version : 0.90
>>   Creation Time : Tue Sep 28 16:54:04 2010
>>      Raid Level : raid1
>>      Array Size : 2147483520 (2048.00 GiB 2199.02 GB)
>>   Used Dev Size : 2147483520 (2048.00 GiB 2199.02 GB)
>>    Raid Devices : 2
>>   Total Devices : 2
>> Preferred Minor : 2
>>     Persistence : Superblock is persistent
>> 
>>     Update Time : Sat Sep  6 23:04:44 2014
>>           State : clean 
>>  Active Devices : 2
>> Working Devices : 2
>>  Failed Devices : 0
>>   Spare Devices : 0
>> 
>>            UUID : 9a363004:ad04abfe:2e31e1c3:caf66df4
>>          Events : 0.77066
>> 
>>     Number   Major   Minor   RaidDevice State
>>        0       8       33        0      active sync   /dev/sdc1
>>        1       8       49        1      active sync   /dev/sdd1
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> # fdisk /dev/sdd
>> 
>> WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.
>
>Why are you using "fdisk" even though "fdisk" is telling you not to use it?
>
>What does
>  cat /proc/partitions
>
>say about the sizes of the partitions?
>
>NeilBrown
>



I, of course, didn't use fdisk to create partitions - they were created using cgdisk, but it's graphical tool so i wasn't able to send output from it, so i get output from fdisk to prove, that partitions are large enough.


# cat /proc/partitions
major minor  #blocks  name

   9        2 2147483520 md2
   8       32 3907018584 sdc
   8       33 2147483647 sdc1
   8       48 3907018584 sdd
   8       49 2147483647 sdd1

Re: RAID0 size over 2 TB

[Mikael Abrahamsson]

On Mon, 8 Sep 2014, azurIt wrote:

> I, of course, didn't use fdisk to create partitions - they were created using cgdisk, but it's graphical tool so i wasn't able to send output from it, so i get output from fdisk to prove, that partitions are large enough.
>
> # cat /proc/partitions
> major minor  #blocks  name
>
>   8       33 2147483647 sdc1
>   8       49 2147483647 sdd1

.... they're not.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: RAID0 size over 2 TB

[azurIt]

>
> CC: linux-raid@vger.kernel.org
>On Mon, 8 Sep 2014, azurIt wrote:
>
>> I, of course, didn't use fdisk to create partitions - they were created using cgdisk, but it's graphical tool so i wasn't able to send output from it, so i get output from fdisk to prove, that partitions are large enough.
>>
>> # cat /proc/partitions
>> major minor  #blocks  name
>>
>>   8       33 2147483647 sdc1
>>   8       49 2147483647 sdd1
>
>.... they're not.
>
>-- 
>Mikael Abrahamsson    email: swmike@swm.pp.se




so why cgdisk and also fdisk sees them as 4TB?

Re: RAID0 size over 2 TB

[Chris Murphy]

On Sep 8, 2014, at 1:24 AM, azurIt <azurit@pobox.sk> wrote:

>> 
>> CC: linux-raid@vger.kernel.org
>> On Mon, 8 Sep 2014, azurIt wrote:
>> 
>>> I, of course, didn't use fdisk to create partitions - they were created using cgdisk, but it's graphical tool so i wasn't able to send output from it, so i get output from fdisk to prove, that partitions are large enough.
>>> 
>>> # cat /proc/partitions
>>> major minor  #blocks  name
>>> 
>>>  8       33 2147483647 sdc1
>>>  8       49 2147483647 sdd1
>> 
>> .... they're not.
>> 
>> -- 
>> Mikael Abrahamsson    email: swmike@swm.pp.se
> 
> 
> 
> 
> so why cgdisk and also fdisk sees them as 4TB?

Newer versions of fdisk do support GPT. Yours isn't new enough. Since you have cgdisk, you should use:

gdisk -l /dev/sdc

or

parted -s /dev/sdc u s p


Chris Murphy

Re: RAID0 size over 2 TB

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2813 bytes --]

On Mon, 08 Sep 2014 09:18:11 +0200 "azurIt" <azurit@pobox.sk> wrote:

> >
> > CC: linux-raid@vger.kernel.org
> >On Sat, 06 Sep 2014 23:09:28 +0200 "azurIt" <azurit@pobox.sk> wrote:
> >
> >> Hi,
> >> 
> >> i'm having problems with growing my RAID0 array over 2 TB. I installed two 4 TB drives, created 4 TB partitions but array cannot be resized over 2 TB. Superblock versoin is 0.90 and kernel version is 3.2.53 - according to linux RAID wiki, it should support 4 TB component/array size:
> >> https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format
> >> 
> >> Some more info:
> >> 
> >> # mdadm --detail /dev/md2
> >> /dev/md2:
> >>         Version : 0.90
> >>   Creation Time : Tue Sep 28 16:54:04 2010
> >>      Raid Level : raid1
> >>      Array Size : 2147483520 (2048.00 GiB 2199.02 GB)
> >>   Used Dev Size : 2147483520 (2048.00 GiB 2199.02 GB)
> >>    Raid Devices : 2
> >>   Total Devices : 2
> >> Preferred Minor : 2
> >>     Persistence : Superblock is persistent
> >> 
> >>     Update Time : Sat Sep  6 23:04:44 2014
> >>           State : clean 
> >>  Active Devices : 2
> >> Working Devices : 2
> >>  Failed Devices : 0
> >>   Spare Devices : 0
> >> 
> >>            UUID : 9a363004:ad04abfe:2e31e1c3:caf66df4
> >>          Events : 0.77066
> >> 
> >>     Number   Major   Minor   RaidDevice State
> >>        0       8       33        0      active sync   /dev/sdc1
> >>        1       8       49        1      active sync   /dev/sdd1
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> # fdisk /dev/sdd
> >> 
> >> WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.
> >
> >Why are you using "fdisk" even though "fdisk" is telling you not to use it?
> >
> >What does
> >  cat /proc/partitions
> >
> >say about the sizes of the partitions?
> >
> >NeilBrown
> >
> 
> 
> 
> I, of course, didn't use fdisk to create partitions - they were created using cgdisk, but it's graphical tool so i wasn't able to send output from it, so i get output from fdisk to prove, that partitions are large enough.

Yet fdisk *didn't* prove that the partitions are large enough - it said the
partitions were:

/dev/sdd1               1  4294967295  2147483647+  ee  GPT
                                       ^^^^^^^^^^^

2 Gig.

And /proc/partitions tells me:

> 
> 
> # cat /proc/partitions
> major minor  #blocks  name
> 
>    9        2 2147483520 md2
>    8       32 3907018584 sdc
>    8       33 2147483647 sdc1
>    8       48 3907018584 sdd
>    8       49 2147483647 sdd1

that the partitions are 2Gig.  You need to sort the partition size out.  I
cannot help you.  I have no experience with GPT.  Maybe a reboot?

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID0 size over 2 TB

[azurIt]

> CC: linux-raid@vger.kernel.org
>On Mon, 08 Sep 2014 09:18:11 +0200 "azurIt" <azurit@pobox.sk> wrote:
>
>> >
>> > CC: linux-raid@vger.kernel.org
>> >On Sat, 06 Sep 2014 23:09:28 +0200 "azurIt" <azurit@pobox.sk> wrote:
>> >
>> >> Hi,
>> >> 
>> >> i'm having problems with growing my RAID0 array over 2 TB. I installed two 4 TB drives, created 4 TB partitions but array cannot be resized over 2 TB. Superblock versoin is 0.90 and kernel version is 3.2.53 - according to linux RAID wiki, it should support 4 TB component/array size:
>> >> https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format
>> >> 
>> >> Some more info:
>> >> 
>> >> # mdadm --detail /dev/md2
>> >> /dev/md2:
>> >>         Version : 0.90
>> >>   Creation Time : Tue Sep 28 16:54:04 2010
>> >>      Raid Level : raid1
>> >>      Array Size : 2147483520 (2048.00 GiB 2199.02 GB)
>> >>   Used Dev Size : 2147483520 (2048.00 GiB 2199.02 GB)
>> >>    Raid Devices : 2
>> >>   Total Devices : 2
>> >> Preferred Minor : 2
>> >>     Persistence : Superblock is persistent
>> >> 
>> >>     Update Time : Sat Sep  6 23:04:44 2014
>> >>           State : clean 
>> >>  Active Devices : 2
>> >> Working Devices : 2
>> >>  Failed Devices : 0
>> >>   Spare Devices : 0
>> >> 
>> >>            UUID : 9a363004:ad04abfe:2e31e1c3:caf66df4
>> >>          Events : 0.77066
>> >> 
>> >>     Number   Major   Minor   RaidDevice State
>> >>        0       8       33        0      active sync   /dev/sdc1
>> >>        1       8       49        1      active sync   /dev/sdd1
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> # fdisk /dev/sdd
>> >> 
>> >> WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.
>> >
>> >Why are you using "fdisk" even though "fdisk" is telling you not to use it?
>> >
>> >What does
>> >  cat /proc/partitions
>> >
>> >say about the sizes of the partitions?
>> >
>> >NeilBrown
>> >
>> 
>> 
>> 
>> I, of course, didn't use fdisk to create partitions - they were created using cgdisk, but it's graphical tool so i wasn't able to send output from it, so i get output from fdisk to prove, that partitions are large enough.
>
>Yet fdisk *didn't* prove that the partitions are large enough - it said the
>partitions were:
>
>/dev/sdd1               1  4294967295  2147483647+  ee  GPT
>                                       ^^^^^^^^^^^
>
>2 Gig.
>
>And /proc/partitions tells me:
>
>> 
>> 
>> # cat /proc/partitions
>> major minor  #blocks  name
>> 
>>    9        2 2147483520 md2
>>    8       32 3907018584 sdc
>>    8       33 2147483647 sdc1
>>    8       48 3907018584 sdd
>>    8       49 2147483647 sdd1
>
>that the partitions are 2Gig.  You need to sort the partition size out.  I
>cannot help you.  I have no experience with GPT.  Maybe a reboot?
>
>NeilBrown
>




This is really strange, here is what cgdisk is reporting:
http://imgur.com/U8VnAdM

Re: RAID0 size over 2 TB

[Chris Murphy]

On Sep 8, 2014, at 1:33 AM, azurIt <azurit@pobox.sk> wrote:
> 
> This is really strange, here is what cgdisk is reporting:
> http://imgur.com/U8VnAdM

Does your kernel have CONFIG_EFI_PARTITION=y ?

Old versions of fdisk report the protective MBR, it does not show you GPT partition map. It's normal for the protective MBR to max out out 2TB because MBR encodes 32-bit 512 byte sectors on 512e devices. Your kernel appears to think the disk is 2TB, which suggests to me it's looking only at the MBR, meaning it doesn't have GPT support turned on.


Chris Murphy

Re: RAID0 size over 2 TB

[Chris Murphy]

On Sep 8, 2014, at 12:27 PM, Chris Murphy <lists@colorremedies.com> wrote:

> Your kernel appears to think the disk is 2TB,

					^partition


Chris Murphy

Re: RAID0 size over 2 TB

[azurIt]

______________________________________________________________
> Od: Chris Murphy <lists@colorremedies.com>
> Komu: azurIt <azurit@pobox.sk>
> Dátum: 08.09.2014 20:27
> Predmet: Re: RAID0 size over 2 TB
>
> CC: linux-raid@vger.kernel.org
>On Sep 8, 2014, at 1:33 AM, azurIt <azurit@pobox.sk> wrote:
>> 
>> This is really strange, here is what cgdisk is reporting:
>> http://imgur.com/U8VnAdM
>
>Does your kernel have CONFIG_EFI_PARTITION=y ?
>
>Old versions of fdisk report the protective MBR, it does not show you GPT partition map. It's normal for the protective MBR to max out out 2TB because MBR encodes 32-bit 512 byte sectors on 512e devices. Your kernel appears to think the disk is 2TB, which suggests to me it's looking only at the MBR, meaning it doesn't have GPT support turned on.
>
>
>Chris Murphy--
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html



Ok so, problem is resolved. This is what happened:
 - i replaced 2x 2 TB drives with 2x 4 TB drives WITHOUT rebooting
 - i created 4 TB partition on every drive
 - kernel still thinks that partitions are 2 TB, probably because of some kind of partition table cache
 - when i added these partitions to RAID array, kernel didn't know they are 4 TB and, even, lightly corrupted them (cgdisk was reporting some kind of issues after i added partitions into RAID array, but everything was working ok)

The solution was to call command 'partprobe' (which is part of 'parted' package) after partitioning but BEFORE adding partitions into RAID array.

Thanks to everyone involved!

azur
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html