3TB disk hassles

3TB disk hassles

[Neil Conway]

Howdy...

After much banging of heads on walls, I am throwing in the towel and
asking the experts ;-) ... To cut a long story short:

Is it possible to make a 3TB disk work properly in Linux?

Our "disk" is 12x300GB in RAID5 (with 1 hot-spare) on a 3ware 9500-S12,
so it's actually 2.7TiB ish.  It's also /dev/sda - i.e., the one and
only disk in the system.

Problems are arising due to the 32-bit-ness of normal partition tables.
 I can use parted to make a 2.7TB partition (sda4), and
/proc/partitions looks fine until a reboot, whereupon the top bits are
lost and the big partition looks like a 700GB partition instead of a
2.7TB one; this is a bad thing ;-)

I've had my hopes raised by GPT, but after more reading it appears this
doesn't work on vanilla x86 PCs.

Tips gratefully received.

Neil

PS: not on-list; I'll be reading the real-time archivers, but CCs of
any replies would be appreciated.


	
	
		
___________________________________________________________ 
ALL-NEW Yahoo! Messenger - all new features - even more fun! http://uk.messenger.yahoo.com

Re: 3TB disk hassles

[Michelle Konzack]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Am 2004-12-16 14:52:29, schrieb Neil Conway:
> Howdy...

> Is it possible to make a 3TB disk work properly in Linux?

Ask 3Ware for it

> Our "disk" is 12x300GB in RAID5 (with 1 hot-spare) on a 3ware 9500-S12,
> so it's actually 2.7TiB ish.  It's also /dev/sda - i.e., the one and
> only disk in the system.

I the discription of my 12-Channel Controller (not a 95xx) here
is written, that you can have only 8 HDD's in a Raid-5 Array...

This is a limitation by the Hardware and I had to split the Array
in 2 x 6 HDD's.

> Tips gratefully received.
> 
> Neil

Greetings
Michelle

-- 
Linux-User #280138 with the Linux Counter, http://counter.li.org/ 
Michelle Konzack   Apt. 917                  ICQ #328449886
                   50, rue de Soultz         MSM LinuxMichi
0033/3/88452356    67100 Strasbourg/France   IRC #Debian (irc.icq.com)

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 3TB disk hassles

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> Howdy...
>
> After much banging of heads on walls, I am throwing in the towel and
> asking the experts ;-) ... To cut a long story short:
>
> Is it possible to make a 3TB disk work properly in Linux?
>
> Our "disk" is 12x300GB in RAID5 (with 1 hot-spare) on a 3ware 9500-S12,
> so it's actually 2.7TiB ish.  It's also /dev/sda - i.e., the one and
> only disk in the system.
>
> Problems are arising due to the 32-bit-ness of normal partition tables.
>  I can use parted to make a 2.7TB partition (sda4), and
> /proc/partitions looks fine until a reboot, whereupon the top bits are
> lost and the big partition looks like a 700GB partition instead of a
> 2.7TB one; this is a bad thing ;-)
>
> I've had my hopes raised by GPT, but after more reading it appears this
> doesn't work on vanilla x86 PCs.
>
> Tips gratefully received.
>
> Neil
>
> PS: not on-list; I'll be reading the real-time archivers, but CCs of
> any replies would be appreciated.

Are you sure your card supports creating a single volume in excess of 2TB? 
Some cards have such a limit, although you can create many 2TB volumes on the 
same card.

Mark.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBwaueBn4EFUVUIO0RAhtLAKCK6ZcujlH1LGQ4SMssXlhoiifRqACgykSR
uXt8wfwTM2N6nxPBOFNaGtc=
=F7JI
-----END PGP SIGNATURE-----

Re: 3TB disk hassles

[Hans Kristian Rosbach]

> Are you sure your card supports creating a single volume in excess of 2TB? 
> Some cards have such a limit, although you can create many 2TB volumes on the 
> same card.

http://www.3ware.com/products/serial_ata9000.asp
States: "Single array capacity scales to over 3 terabytes per controller
(64-bit LBA support)"

-HK

Re: 3TB disk hassles

[Michelle Konzack]

[-- Attachment #1: Type: text/plain, Size: 615 bytes --]

Am 2004-12-16 15:37:02, schrieb Mark Watts:

> Are you sure your card supports creating a single volume in excess of
> 2TB? 
> Some cards have such a limit, although you can create many 2TB volumes
> on the 
> same card.

You can have 4 TByte on one 12-Channel Card,
but in two Arrays of 6 HDD's   :-)

> Mark.

Greetings
Michelle

-- 
Linux-User #280138 with the Linux Counter, http://counter.li.org/ 
Michelle Konzack   Apt. 917                  ICQ #328449886
                   50, rue de Soultz         MSM LinuxMichi
0033/3/88452356    67100 Strasbourg/France   IRC #Debian (irc.icq.com)

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 3TB disk hassles

[Alan Cox]

On Iau, 2004-12-16 at 16:03, Jan Engelhardt wrote:
> >You can have 4 TByte on one 12-Channel Card,
> >but in two Arrays of 6 HDD's   :-)
> 
> Maybe some LVM trickery can aggregate ungrowable hardware raids together to a 
> single block device.

LVM does not mix with volumes > 2Tb in my experience. I don't know if
anyone has fixed it yet but I'd advise caution.

Remember you don't need a partition table. You can just leave the volume
unpartitioned. You can also use other partition formats providing you
don't need the BIOS boot gunk to boot off that volume. 

Alan

Re: 3TB disk hassles

[Jan Engelhardt]

>> Are you sure your card supports creating a single volume in excess of
>> 2TB? 
>> Some cards have such a limit, although you can create many 2TB volumes
>> on the 
>> same card.
>
>You can have 4 TByte on one 12-Channel Card,
>but in two Arrays of 6 HDD's   :-)

Maybe some LVM trickery can aggregate ungrowable hardware raids together to a 
single block device.



Jan Engelhardt
-- 
ENOSPC

Re: 3TB disk hassles

[Neil Conway]

Hi all - thanks for the replies.

 --- Hans Kristian Rosbach <hk@isphuset.no> wrote: 
> > Are you sure your card supports creating a single volume in excess
> of 2TB? 
> > Some cards have such a limit, although you can create many 2TB
> volumes on the 
> > same card.
> 
> http://www.3ware.com/products/serial_ata9000.asp
> States: "Single array capacity scales to over 3 terabytes per
> controller
> (64-bit LBA support)"

Yes indeed, the 3ware unit is more than happy to make a 2.7TB (TiB)
array with all 11 disks (12th disk is hot spare).  This can be
partitioned fine, mke2fs'd to ext3 fine (maybe xfs would be better
though!), filled with lots of data and fsck'ed with no problems at all.
 (We've used more than one config, but right now it's running FC3 with
2.6.9-1.681_FC3.)

The problems only occur after a reboot, at which point Linux still sees
the disk as a 2.7TB disk, but the partition table read from the disk
says that sda4 isn't a 2.7TB partition any more, but a mere (!) 700GB.

Catting /proc/partitions right now gives:
[root@fuslsb ~]# cat /proc/partitions
major minor  #blocks  name

   8     0 2929582080 sda
   8     1     136521 sda1
   8     2   20972857 sda2
   8     3    4200997 sda3
   8     4 2904270862 sda4
[root@fuslsb ~]#

This is fine - but only because I re-ran parted after the boot. 
Indeed, parted itself shows that things aren't really right, presumably
because the partition table is a "normal" one and doesn't have space
for more than 2^32 * 512 bytes (i.e. 2^41 bytes, or 2TB binary).

[root@fuslsb ~]# parted /dev/sda print
Disk geometry for /dev/sda: 0.000-2860920.000 megabytes
Disk label type: msdos
Minor    Start       End     Type      Filesystem  Flags
1          0.031    133.352  primary   ext3        boot
2        133.352  20614.658  primary   ext3
3      20614.658  24717.194  primary   linux-swap
4      24717.195 763767.208  primary   ext3
Information: Don't forget to update /etc/fstab, if necessary.

I guess this means parted is sucking the partition table (incorrect of
course) from the MBR, and this is what the kernel sees on reboot.

fdisk gives similar output:
[root@fuslsb ~]# fdisk -l /dev/sda

Disk /dev/sda: 2999.8 GB, 2999892049920 bytes
255 heads, 63 sectors/track, 364716 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          17      136521   83  Linux
/dev/sda2              18        2628    20972857+  83  Linux
/dev/sda3            2629        3151     4200997+  82  Linux swap
/dev/sda4            3152       97367   756787214+  83  Linux

My understanding (evolving as I learn more about it) of this is simply
that normal partition tables as stored in the MBR cannot handle disks
bigger than 2TB, since they assume 512-byte sectors and only have a
32-bit field to indicate partition start and end points.

However, browsing the net, it seems like shedloads of people are using
>2TB disks out there, so what's the magic bullet?

Right now, the only scheme I have vaguely concocted in my head that
will make this work for us is to add another disk to become the boot
disk - this is actually a major PITA cos there's really no physical
space in the chassis - and then use a GPT/EFI (whatever?) partition
table on the big disk.  This means that BIOS won't understand the big
disk at all but that'll be OK since it isn't trying to boot from it.

Anyone with a magic tip for how to do this is welcome to tell us all
;-))

thanks
Neil





		
___________________________________________________________ 
Win a castle for NYE with your mates and Yahoo! Messenger 
http://uk.messenger.yahoo.com

Re: 3TB disk hassles

[Michelle Konzack]

[-- Attachment #1: Type: text/plain, Size: 505 bytes --]

Hello Jan, 

Am 2004-12-16 17:03:55, schrieb Jan Engelhardt:

> Maybe some LVM trickery can aggregate ungrowable hardware raids together to a 
> single block device.

Yes, thats right.

> Jan Engelhardt

Greetings
Michelle

-- 
Linux-User #280138 with the Linux Counter, http://counter.li.org/ 
Michelle Konzack   Apt. 917                  ICQ #328449886
                   50, rue de Soultz         MSM LinuxMichi
0033/3/88452356    67100 Strasbourg/France   IRC #Debian (irc.icq.com)

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 3TB disk hassles

[Tomas Carnecky]

Neil Conway wrote:
  > Right now, the only scheme I have vaguely concocted in my head that
> will make this work for us is to add another disk to become the boot
> disk - this is actually a major PITA cos there's really no physical
> space in the chassis - and then use a GPT/EFI (whatever?) partition
> table on the big disk.  This means that BIOS won't understand the big
> disk at all but that'll be OK since it isn't trying to boot from it.
> 
> Anyone with a magic tip for how to do this is welcome to tell us all
> ;-))
> 

I had a GUID partition table (GPT) on my system (x86, normal 
mainboard/BIOS etc) and it worked fine. I didn't need a separate boot 
disk. I used grub as the boot loader. I think if you enable GPT in the 
kernel you should be able to boot stright from the big disk.

tom

Re: 3TB disk hassles

[Neil Conway]

Hi Tom...

 --- Tomas Carnecky <tom@dbservice.com> wrote: 
> I had a GUID partition table (GPT) on my system (x86, normal 
> mainboard/BIOS etc) and it worked fine. I didn't need a separate boot
> disk. I used grub as the boot loader. I think if you enable GPT in
> the 
> kernel you should be able to boot stright from the big disk.

Wow, that's unexpected but encouraging news.  What distro?  Did it
allow you to go GPT right from the off, or did you have to migrate from
an MSDOS ptbl to a GPT one after installation?

Thanks for the tip.
Neil



	
	
		
___________________________________________________________ 
ALL-NEW Yahoo! Messenger - all new features - even more fun! http://uk.messenger.yahoo.com

Re: 3TB disk hassles

[Tomas Carnecky]

Tomas Carnecky wrote:
> Neil Conway wrote:
>  > Right now, the only scheme I have vaguely concocted in my head that
> 
>> will make this work for us is to add another disk to become the boot
>> disk - this is actually a major PITA cos there's really no physical
>> space in the chassis - and then use a GPT/EFI (whatever?) partition
>> table on the big disk.  This means that BIOS won't understand the big
>> disk at all but that'll be OK since it isn't trying to boot from it.
>>
>> Anyone with a magic tip for how to do this is welcome to tell us all
>> ;-))
>>
> 
> I had a GUID partition table (GPT) on my system (x86, normal 
> mainboard/BIOS etc) and it worked fine. I didn't need a separate boot 
> disk. I used grub as the boot loader. I think if you enable GPT in the 
> kernel you should be able to boot stright from the big disk.
> 

However, the disk array was only 360GB and the boot partition was the
first one on the disk, maybe 60MB big (in the first 1024 sectors).

tom

Re: 3TB disk hassles

[Tomas Carnecky]

Neil Conway wrote:
> Hi Tom...
> 
>  --- Tomas Carnecky <tom@dbservice.com> wrote: 
> 
>>I had a GUID partition table (GPT) on my system (x86, normal 
>>mainboard/BIOS etc) and it worked fine. I didn't need a separate boot
>>disk. I used grub as the boot loader. I think if you enable GPT in
>>the 
>>kernel you should be able to boot stright from the big disk.
> 
> 
> Wow, that's unexpected but encouraging news.  What distro?  Did it
> allow you to go GPT right from the off, or did you have to migrate from
> an MSDOS ptbl to a GPT one after installation?
> 

It was gentoo, and I even think I installed it right onto the GPT disk, 
so no migration. But I'm not sure. You just have to look that your 
kernel supports GPT. I don't know if the kernel from the gentoo livecd 
supports GPT.

Also have a look here how to create GPT partitions:
http://www.google.ch/search?q=site%3Ausefulthings.org.uk+gpt
I think I did it like it's shown there, mklabel, mkpart and mount them.
I don't think I migrated from MSDOS to GPT, because I don't even know 
how it'is possible if you have only one disk with the system on it.

While looking for gentoo GPT support, I found this:
http://www.ussg.iu.edu/hypermail/linux/kernel/0411.1/0624.html
Looks like CONFIG_EFI_PARTITION is enabled by default now on the newer 
kernels.

tom

Re: 3TB disk hassles

[Adam Heath]

On Thu, 16 Dec 2004, Neil Conway wrote:

> Howdy...
>
> After much banging of heads on walls, I am throwing in the towel and
> asking the experts ;-) ... To cut a long story short:
>
> Is it possible to make a 3TB disk work properly in Linux?
>
> Our "disk" is 12x300GB in RAID5 (with 1 hot-spare) on a 3ware 9500-S12,
> so it's actually 2.7TiB ish.  It's also /dev/sda - i.e., the one and
> only disk in the system.
>
> Problems are arising due to the 32-bit-ness of normal partition tables.
>  I can use parted to make a 2.7TB partition (sda4), and
> /proc/partitions looks fine until a reboot, whereupon the top bits are
> lost and the big partition looks like a 700GB partition instead of a
> 2.7TB one; this is a bad thing ;-)
>
> I've had my hopes raised by GPT, but after more reading it appears this
> doesn't work on vanilla x86 PCs.
>
> Tips gratefully received.

Maybe use LVM on the raw device?

Re: 3TB disk hassles

[Pedro Venda (SYSADM)]

Neil Conway wrote:
> Howdy...
> 
> After much banging of heads on walls, I am throwing in the towel and
> asking the experts ;-) ... To cut a long story short:
> 
> Is it possible to make a 3TB disk work properly in Linux?
> 
> Our "disk" is 12x300GB in RAID5 (with 1 hot-spare) on a 3ware 9500-S12,
> so it's actually 2.7TiB ish.  It's also /dev/sda - i.e., the one and
> only disk in the system.

not meaning to criticise... but isn't it a good idea to have a separate raid1 
volume to boot the system?

is there a good reason why one should mix system & storage?

I think that would solve your problem.

regards,
pedro venda.

-- 

Pedro João Lopes Venda
email: pjvenda@rnl.ist.utl.pt
http://maxwell.rnl.ist.utl.pt

Equipa de Administração de Sistemas
Rede das Novas Licenciaturas (RNL)
Instituto Superior Técnico
http://www.rnl.ist.utl.pt
http://mega.ist.utl.pt

Re: 3TB disk hassles

[Andries Brouwer]

On Thu, Dec 16, 2004 at 04:00:36PM +0000, Alan Cox wrote:

> Remember you don't need a partition table. You can just leave the volume
> unpartitioned. You can also use other partition formats providing you
> don't need the BIOS boot gunk to boot off that volume. 

Yes, indeed.

One can use a standard DOS-type partition table, and pick a new type -
I reserved 88 for this purpose today - where type 88 indicates a
plaintext partition table found elsewhere on the disk.
Where is elsewhere? In the starting sector of the type 88 partition
(that can have length 1).
This allows one to have the initial part of the disk (at most 2 TB)
partitioned in old-fashioned manner.

The plaintext partition table is just a table with lines
	<start> <size>
that one can edit with emacs or vi.

There is magic to recognize it, namely the line
	"# Plaintext partition table"
and magic to indicate the end of the table, namely "# end".

That is all. If anybody wants it I can send the trivial code.
(Am using it now, but unfortunately I do not have 3 TB disks.)

Comments welcome.

Andries

Re: 3TB disk hassles

[H. Peter Anvin]

Followup to:  <20041218001254.GA8886@pclin040.win.tue.nl>
By author:    Andries Brouwer <aebr@win.tue.nl>
In newsgroup: linux.dev.kernel
> 
> Yes, indeed.
> 
> One can use a standard DOS-type partition table, and pick a new type -
> I reserved 88 for this purpose today - where type 88 indicates a
> plaintext partition table found elsewhere on the disk.
> Where is elsewhere? In the starting sector of the type 88 partition
> (that can have length 1).
> This allows one to have the initial part of the disk (at most 2 TB)
> partitioned in old-fashioned manner.
> 
> The plaintext partition table is just a table with lines
> 	<start> <size>
> that one can edit with emacs or vi.
> 
> There is magic to recognize it, namely the line
> 	"# Plaintext partition table"
> and magic to indicate the end of the table, namely "# end".
> 
> That is all. If anybody wants it I can send the trivial code.
> (Am using it now, but unfortunately I do not have 3 TB disks.)
> 

First, what's wrong with the GUID partition table format?  Let's stick
to standards as long as they work; especially for things that
potentially affect multiple operating systems.

Second, several problems with this.  Sector 0 is the boot sector, so
using it is a really bad choice.  (I'd reserve several sector for
master boot code.)  In fact, rather than having a separate partition,
why don't we just specify that a sector starting with "# Plaintext partition
table" has to start within the first 64 sectors of the disk (it's
common for DOS partition tables to have the first partition start at
offset 63.)

Third, it ought to be possible to put more information than this,
e.g. for raid detect.

	-hpa

Re: 3TB disk hassles

[Andries Brouwer]

On Sat, Dec 18, 2004 at 03:08:42AM +0000, H. Peter Anvin wrote:
> Followup to:  <20041218001254.GA8886@pclin040.win.tue.nl>
> By author:    Andries Brouwer <aebr@win.tue.nl>
> In newsgroup: linux.dev.kernel
> > 
> > Yes, indeed.
> > 
> > One can use a standard DOS-type partition table, and pick a new type -
> > I reserved 88 for this purpose today - where type 88 indicates a
> > plaintext partition table found elsewhere on the disk.
> > Where is elsewhere? In the starting sector of the type 88 partition
> > (that can have length 1).
> > This allows one to have the initial part of the disk (at most 2 TB)
> > partitioned in old-fashioned manner.
> > 
> > The plaintext partition table is just a table with lines
> > 	<start> <size>
> > that one can edit with emacs or vi.
> > 
> > There is magic to recognize it, namely the line
> > 	"# Plaintext partition table"
> > and magic to indicate the end of the table, namely "# end".
> > 
> > That is all. If anybody wants it I can send the trivial code.
> > (Am using it now, but unfortunately I do not have 3 TB disks.)
> 
> First, what's wrong with the GUID partition table format?  Let's stick
> to standards as long as they work; especially for things that
> potentially affect multiple operating systems.
> 
> Second, several problems with this.  Sector 0 is the boot sector, so
> using it is a really bad choice.  (I'd reserve several sector for
> master boot code.)  In fact, rather than having a separate partition,
> why don't we just specify that a sector starting with "# Plaintext partition
> table" has to start within the first 64 sectors of the disk (it's
> common for DOS partition tables to have the first partition start at
> offset 63.)
> 
> Third, it ought to be possible to put more information than this,
> e.g. for raid detect.

Concerning third, I allow for labels and comments (after #), so anybody
can add any type of recognizable comments or pragmas. The header line
and closing line are examples of comments that already have significance.

Concerning second, I think you misunderstood something. Don't know why
you think I am using MBR - certainly not to put this plaintext table.
In fact MBR has a traditional DOS-type partition table that allows for
a boot setup in traditional ways in case utilities are used that only
understand the old ways. But type 88 describes a (very short) partition
that contains the plaintext partition table. Life is easy:
	# emacs partition_table
	# cat partition_table > /dev/sda2
and reboot or
	# blockdev --rereadpt /dev/sda

Concerning one, it is a somewhat complicated format that takes over
your disk, rather inconvenient. It seems to me that one needs a good
reason (like a BIOS that understands the format and is able to boot
from it) to choose it.

Andries

Re: 3TB disk hassles

[H. Peter Anvin]

Andries Brouwer wrote:
> 
> Concerning one, it is a somewhat complicated format that takes over
> your disk, rather inconvenient. It seems to me that one needs a good
> reason (like a BIOS that understands the format and is able to boot
> from it) to choose it.
> 

Not really; it's actually a very simple table.

	-hpa

Re: 3TB disk hassles

[Jeff V. Merkey]

H. Peter Anvin wrote:

> Andries Brouwer wrote:
>
>>
>> Concerning one, it is a somewhat complicated format that takes over
>> your disk, rather inconvenient. It seems to me that one needs a good
>> reason (like a BIOS that understands the format and is able to boot
>> from it) to choose it.
>>
>
> Not really; it's actually a very simple table.
>

It would be nice to know when this is going to make it in for my Linux 
projects.
I am running a 3Ware 9500 series with 3.1 TB disks.  I am able to use 
all the
storage at present with dsfs.  dsfs can support volumes up to 281 TB at 
present
but linux readir() can get into some problems when directories get 
really large.

I am not seeing problems with files that are 1.5 TB in size.  Have not 
tried to
create a 3TB file yet, but in theory, the VFS looks to support it.  I am 
getting around the
partition problem by basically ignoring the table extents (fdisk is 
broken with these large
partitions and wraps back to 700GB) if I have only created a single 
partition, I just query
the drive geometry and take the remaining space on the device and I 
ignore the partition
table.  It works fine.  If I detect more than one of my partitions I 
revert back to the actual
partition dimensions. 

For Jens edification, I am using the BIO subsystem with this and I am 
seeing no problems
reading and writing these huge drives, so I think Linux 2.6.9 and 2.6.10 
will support this
well, and appears to.  I will be testing a combined striped array at 
around 20TB with multiple
controllers and FC/AL and will update if any problems are encountered. 

Other than the partition problem, the base kernel seems to support these 
huge sizes with
64 bit LBA addressing very well.

Jeff

Re: 3TB disk hassles

[Jeff V. Merkey]

Jeff V. Merkey wrote:

> H. Peter Anvin wrote:
>
>> Andries Brouwer wrote:
>>
>>>
>>> Concerning one, it is a somewhat complicated format that takes over
>>> your disk, rather inconvenient. It seems to me that one needs a good
>>> reason (like a BIOS that understands the format and is able to boot
>>> from it) to choose it.
>>>
>>
>> Not really; it's actually a very simple table.
>>
>
> It would be nice to know when this is going to make it in for my Linux 
> projects.
> I am running a 3Ware 9500 series with 3.1 TB disks.  I am able to use 
> all the
> storage at present with dsfs.  dsfs can support volumes up to 281 TB 
> at present
> but linux readir() can get into some problems when directories get 
> really large.
>
> I am not seeing problems with files that are 1.5 TB in size.  Have not 
> tried to
> create a 3TB file yet, but in theory, the VFS looks to support it.  I 
> am getting around the
> partition problem by basically ignoring the table extents (fdisk is 
> broken with these large
> partitions and wraps back to 700GB) if I have only created a single 
> partition, I just query
> the drive geometry and take the remaining space on the device and I 
> ignore the partition
> table.  It works fine.  If I detect more than one of my partitions I 
> revert back to the actual
> partition dimensions.
> For Jens edification, I am using the BIO subsystem with this and I am 
> seeing no problems
> reading and writing these huge drives, so I think Linux 2.6.9 and 
> 2.6.10 will support this
> well, and appears to.  I will be testing a combined striped array at 
> around 20TB with multiple
> controllers and FC/AL and will update if any problems are encountered.
> Other than the partition problem, the base kernel seems to support 
> these huge sizes with
> 64 bit LBA addressing very well.
>
> Jeff
>

One other item I noticed is that the compiler for X86 has some problems 
doing math for a 64 bit target
variable, so when you are using Large LBA and doing something like:

sector_t lba = part.start_lba + (block * (block_size /  sector_size));

you need to cast the variables if they are defined as 32 bit numbers 
because the compiler is too stupid
to realize you are adding the cumlative result into a 64-bit value, and 
it will wrap the offset as a 32 bit number.

i.e. 

sector_t lba = part.start_lba + (sector_t)((sector_t)block * 
((sector_t)block_size /  (sector_t)sector_size));

This works but if you leave off the type casting on any of the variables 
the number reverts to a 32 bit value
and wraps when you are calculating a 64 bit lba address. 

Jeff

Re: 3TB disk hassles

[linux-os]

On Mon, 3 Jan 2005, Jeff V. Merkey wrote:

> Jeff V. Merkey wrote:
>
>> H. Peter Anvin wrote:
>> 
>>> Andries Brouwer wrote:
>>> 
>>>> 
>>>> Concerning one, it is a somewhat complicated format that takes over
>>>> your disk, rather inconvenient. It seems to me that one needs a good
>>>> reason (like a BIOS that understands the format and is able to boot
>>>> from it) to choose it.
>>>> 
>>> 
>>> Not really; it's actually a very simple table.
>>> 
>> 
>> It would be nice to know when this is going to make it in for my Linux 
>> projects.
>> I am running a 3Ware 9500 series with 3.1 TB disks.  I am able to use all 
>> the
>> storage at present with dsfs.  dsfs can support volumes up to 281 TB at 
>> present
>> but linux readir() can get into some problems when directories get really 
>> large.
>> 
>> I am not seeing problems with files that are 1.5 TB in size.  Have not 
>> tried to
>> create a 3TB file yet, but in theory, the VFS looks to support it.  I am 
>> getting around the
>> partition problem by basically ignoring the table extents (fdisk is broken 
>> with these large
>> partitions and wraps back to 700GB) if I have only created a single 
>> partition, I just query
>> the drive geometry and take the remaining space on the device and I ignore 
>> the partition
>> table.  It works fine.  If I detect more than one of my partitions I revert 
>> back to the actual
>> partition dimensions.
>> For Jens edification, I am using the BIO subsystem with this and I am 
>> seeing no problems
>> reading and writing these huge drives, so I think Linux 2.6.9 and 2.6.10 
>> will support this
>> well, and appears to.  I will be testing a combined striped array at around 
>> 20TB with multiple
>> controllers and FC/AL and will update if any problems are encountered.
>> Other than the partition problem, the base kernel seems to support these 
>> huge sizes with
>> 64 bit LBA addressing very well.
>> 
>> Jeff
>> 
>
> One other item I noticed is that the compiler for X86 has some problems doing 
> math for a 64 bit target
> variable, so when you are using Large LBA and doing something like:
>
> sector_t lba = part.start_lba + (block * (block_size /  sector_size));


I think the writer forgot about promotion rules so it's not
a compiler problem.

>
> you need to cast the variables if they are defined as 32 bit numbers because 
> the compiler is too stupid
> to realize you are adding the cumlative result into a 64-bit value, and it 
> will wrap the offset as a 32 bit number.
>

If block_size = uint32_t, and sector_size = uint32_t, then
block_size / sector_size is uint32_t, nothing more. Now, we have
a uint32_t * another uint32_t which is uint32_t with a possible
wrap, still correct. Then we have uint32_t + uint32_t which will
be promoted to sector_t (uint64_t).

If you need to promote variables ahead of the final assignment, the
writer needs to do this, not the compiler.


> i.e. 
> sector_t lba = part.start_lba + (sector_t)((sector_t)block * 
> ((sector_t)block_size /  (sector_t)sector_size));
>
> This works but if you leave off the type casting on any of the variables the 
> number reverts to a 32 bit value
> and wraps when you are calculating a 64 bit lba address. 
> Jeff
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: 3TB disk hassles

[Neil Conway]

Howdy...  Apologies for the somewhat tardy reply; I've been
concentrating on getting the hardware to play nice recently and not
worrying so much about the software.

--- Tomas Carnecky <tom@dbservice.com> wrote:
> It was gentoo, and I even think I installed it right onto the GPT
> disk, 
> so no migration. But I'm not sure. You just have to look that your 
> kernel supports GPT. I don't know if the kernel from the gentoo
> livecd 
> supports GPT.
> 
> Also have a look here how to create GPT partitions:
> http://www.google.ch/search?q=site%3Ausefulthings.org.uk+gpt
> I think I did it like it's shown there, mklabel, mkpart and mount
> them.
> I don't think I migrated from MSDOS to GPT, because I don't even know
> how it'is possible if you have only one disk with the system on it.

Bizarre...  I will give this a try on a spare system as soon as I can. 
I thought sure I had read somewhere that typical x86 PC BIOSes just
didn't understand the GPT ptbl, and thus couldn't boot from a GPT'ed
disk.

thanks,
Neil

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 3TB disk hassles

[Neil Conway]

Howdy...

--- "Pedro Venda (SYSADM)" <pjvenda@rnl.ist.utl.pt> wrote:
> Neil Conway wrote:
> > Howdy...
> > After much banging of heads on walls, I am throwing in the towel
> and
> > asking the experts ;-) ... To cut a long story short:
> > Is it possible to make a 3TB disk work properly in Linux?
> > Our "disk" is 12x300GB in RAID5 (with 1 hot-spare) on a 3ware
> 9500-S12,
> > so it's actually 2.7TiB ish.  It's also /dev/sda - i.e., the one
> and
> > only disk in the system.
> 
> not meaning to criticise... but isn't it a good idea to have a
> separate raid1 volume to boot the system?

Well, yes, and we would if we could.  Sadly, the chassis we got from
our vendor only has space for the 12 hot-swap disks, and we need the
capacity too badly to lose 2 slots for a boot volume.  If only we could
take a sliver of each of the 12 disks to make a tiny RAID5 boot
volume...

Regards,
Neil


		
__________________________________ 
Do you Yahoo!? 
Yahoo! Mail - Easier than ever with enhanced search. Learn more.
http://info.mail.yahoo.com/mail_250

Re: 3TB disk hassles

[Jan Lindheim]

On Thu, 16 Dec 2004 14:52:29 +0000, Neil Conway wrote:

> Howdy...
> 
> After much banging of heads on walls, I am throwing in the towel and
> asking the experts ;-) ... To cut a long story short:
> 
> Is it possible to make a 3TB disk work properly in Linux?
> 
> Our "disk" is 12x300GB in RAID5 (with 1 hot-spare) on a 3ware 9500-S12,
> so it's actually 2.7TiB ish.  It's also /dev/sda - i.e., the one and
> only disk in the system.
> 
> Problems are arising due to the 32-bit-ness of normal partition tables.
>  I can use parted to make a 2.7TB partition (sda4), and
> /proc/partitions looks fine until a reboot, whereupon the top bits are
> lost and the big partition looks like a 700GB partition instead of a
> 2.7TB one; this is a bad thing ;-)
> 
> I've had my hopes raised by GPT, but after more reading it appears this
> doesn't work on vanilla x86 PCs.
> 
> Tips gratefully received.
> 
> Neil
> 
> PS: not on-list; I'll be reading the real-time archivers, but CCs of
> any replies would be appreciated.
> 
> 
> 	
> 	
> 		
> ___________________________________________________________ 
> ALL-NEW Yahoo! Messenger - all new features - even more fun! http://uk.messenger.yahoo.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Neil,
I just set up a new file server a couple of weeks ago, which
holds two RAIDs, making up 4.1 TB each.  I'm not aware of any
distributions that will do the partitioning for you.  I was
able to get the system configured by booting Knoppix and using
parted for the partitioning.  Fedora refused to install on this
system after checking the GPT partition table.  The latest
Mandrake Linux warned me about the partitions, but at least
it accepted going on with the install anyway. lilo is used as
the bootloader and has no problem existing on the MBR of the RAID.
My system is a dual 3.2 GHz Xeon system with 8GB RAM, two
3Ware 9500S RAID controllers, 12 400GB disks pr. RAID controller.
Here's a df from the system:

[root@asaphome ~]# df
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             7.9G  1.1G  6.8G  14% /
/dev/sda3              14G  3.4G   11G  25% /usr
/dev/sda4             4.0T  232G  3.8T   6% /home
/dev/sdb1             4.1T   24G  4.0T   1% /archive

Good Luck!

Regards Jan Lindheim

Re: 3TB disk hassles

[H. Peter Anvin]

Followup to:  <20050206105958.42872.qmail@web26501.mail.ukl.yahoo.com>
By author:    Neil Conway <nconway_kernel@yahoo.co.uk>
In newsgroup: linux.dev.kernel
> 
> Since writing the above, I've been searching for more info.  I
> downloaded four different versions of grub (GNU Grub Legacy, GNU Grub2,
> gentoo and Fedora Core 3).  NONE of these showed any evidence of GPT
> support (I was in a hurry, so I searched for strings EFI, GUID, GPT,
> TB).
> 
> Mucho confused puppy here.
> 
> I fail to see how grub can work on a GPT boot device if it can't parse
> the partition table.  I conclude that I'm still missing something. 
> Perhaps a layer before grub is supposed to parse the GPT instead?  If
> so, isn't that getting us straight back to a GPT-aware BIOS?
> 
> Tell me if this logic is broken: even if a special boot sector is used,
> which IS GPT-aware (though fitting that into the boot sector would be a
> challenge ;-)), once grub loads, it's still going to have to figure out
> how to find the root(hdX,Y) partition from which to load the kernel
> image.  This surely means it has to have either a GPT-parser
> internally, or rely on a pre-parsed list.  No?
> 
> Perhaps one of the other several distros (that I didn't check) has a
> GPT-aware grub.  But Tomas Carnecky said early in this thread that
> gentoo had allowed him to set up a GPT-booting system on x86.  I guess
> it's possible that a cheat was used - maybe an old-style partition
> table in the MBR was used to define the first (boot) partition, but
> surely that's forbidden by the whole EFI spec anyway?
> 

No, it's encouraged.

> 
> Andries Brouwer kindly wrote a patch which I haven't had time to test
> yet (see earlier in thread).  While it would be nice to find a way
> around the problem which didn't require deviations from vanilla
> distros, I think Andries' patch is looking like the only sane fix right
> now.
> 

Note that Andries' patch does *EXACTLY* the same thing as the GPT/EFI
spec does (by using an old-style partition table for the first 2 TB.)

It should be pretty easy to add native support for this in EXTLINUX;
the big problem is supporting true access > 2 TB, which I currently
don't have any way to test.

I'll put that on my todo list.

	-hpa

Re: 3TB disk hassles

[Bodo Eggert]

On Sun, 6 Feb 2005, Neil Conway wrote:

> Since writing the above, I've been searching for more info.  I
> downloaded four different versions of grub (GNU Grub Legacy, GNU Grub2,
> gentoo and Fedora Core 3).  NONE of these showed any evidence of GPT
> support (I was in a hurry, so I searched for strings EFI, GUID, GPT,
> TB).

I'd use lilo in that case. AFAI understood it can start from any device
provided the BIOS can access the boot files. (May require a 5MB /boot 
partition if the disk is larger than the BIOS can access)

HTH

> I fail to see how grub can work on a GPT boot device if it can't parse
> the partition table.  I conclude that I'm still missing something. 
> Perhaps a layer before grub is supposed to parse the GPT instead?  If
> so, isn't that getting us straight back to a GPT-aware BIOS?

If grub parses the partition table, it will do that without any BIOS
support (except maybe for reading the raw data). So even a GPT-aware BIOS
should not change a thing.

-- 
Funny quotes:
23. If at first you don't succeed, destroy all evidence that you tried.

Re: 3TB disk hassles

[Neil Conway]

Argh...

--- Neil Conway <nconway_kernel@yahoo.co.uk> wrote:
> Hi...
> 
> --- Bodo Eggert <7eggert@gmx.de> wrote:
> > No common x86 BIOS can understand any partition table. Booting is
> > done by
> > loading the first sector of the boot device and executing it. The
> > common
> 
> D'oh!!  Red-face here.  Can't believe my brainlessness.
> Thanks for putting me straight - that explains a lot.  Now to try it
> ;-)

Ah, if only it was that simple.

Since writing the above, I've been searching for more info.  I
downloaded four different versions of grub (GNU Grub Legacy, GNU Grub2,
gentoo and Fedora Core 3).  NONE of these showed any evidence of GPT
support (I was in a hurry, so I searched for strings EFI, GUID, GPT,
TB).

Mucho confused puppy here.

I fail to see how grub can work on a GPT boot device if it can't parse
the partition table.  I conclude that I'm still missing something. 
Perhaps a layer before grub is supposed to parse the GPT instead?  If
so, isn't that getting us straight back to a GPT-aware BIOS?

Tell me if this logic is broken: even if a special boot sector is used,
which IS GPT-aware (though fitting that into the boot sector would be a
challenge ;-)), once grub loads, it's still going to have to figure out
how to find the root(hdX,Y) partition from which to load the kernel
image.  This surely means it has to have either a GPT-parser
internally, or rely on a pre-parsed list.  No?

Perhaps one of the other several distros (that I didn't check) has a
GPT-aware grub.  But Tomas Carnecky said early in this thread that
gentoo had allowed him to set up a GPT-booting system on x86.  I guess
it's possible that a cheat was used - maybe an old-style partition
table in the MBR was used to define the first (boot) partition, but
surely that's forbidden by the whole EFI spec anyway?

Andries Brouwer kindly wrote a patch which I haven't had time to test
yet (see earlier in thread).  While it would be nice to find a way
around the problem which didn't require deviations from vanilla
distros, I think Andries' patch is looking like the only sane fix right
now.

Anyone with a definitive answer to the question "can I use GPT on a
vanilla x86 mobo", do speak up :-)

Regards,
Neil
PS: I really didn't think that >2TiB disks were quite so far out on the
bleeding edge :-/



		
__________________________________ 
Do you Yahoo!? 
Meet the all-new My Yahoo! - Try it today! 
http://my.yahoo.com 
 

Re: 3TB disk hassles

[Neil Conway]

Hi...

--- Bodo Eggert <7eggert@gmx.de> wrote:
> No common x86 BIOS can understand any partition table. Booting is
> done by
> loading the first sector of the boot device and executing it. The
> common

D'oh!!  Red-face here.  Can't believe my brainlessness.

Thanks for putting me straight - that explains a lot.  Now to try it
;-)

Neil
PS: I should go back to sleep now, clearly not awake for the last
month.



		
__________________________________ 
Do you Yahoo!? 
Yahoo! Mail - now with 250MB free storage. Learn more.
http://info.mail.yahoo.com/mail_250

Re: 3TB disk hassles

[Bodo Eggert]

Neil Conway <nconway_kernel@yahoo.co.uk> wrote:

> I thought sure I had read somewhere that typical x86 PC BIOSes just
> didn't understand the GPT ptbl, and thus couldn't boot from a GPT'ed
> disk.

No common x86 BIOS can understand any partition table. Booting is done by
loading the first sector of the boot device and executing it. The common
MBR code will move it's code to a unused location, load the first sector
of the active partition and execute it. If you replace the MBR code with
the GPT MBR code, it should understand GPTs. If you replace it with lilo
or grub, one of these tools will run. If you replace it with a boot virus,
it will (usurally) install it's hooks and chain-load the original boot
sector, which in turn does it's work.

Re: 3TB disk hassles

[Rico Tudor]

Neil,

I have a ~2TB ext2 FS on 3Ware RAID5, and simply dispense with
partitioning.  If you use LILO and it complains, arrange for /boot/*
and your kernel image to have low i-numbers: this will keep the blocks
numbers in a tradition range.  Done!