* mdadm raid6->LVM->ext4 setup
@ 2010-11-16 18:08 Zoltan Szecsei
2010-11-16 18:16 ` Roman Mamedov
2010-11-16 22:17 ` Luca Berra
0 siblings, 2 replies; 6+ messages in thread
From: Zoltan Szecsei @ 2010-11-16 18:08 UTC (permalink / raw)
To: linux-raid
Hi,
with the kind help from some of you on this list, I am finally
formatting the filesystem.
I'd be grateful if any of you would like to comment on my chosen
chunk/stripe/stride/ etc values, as from tomorrow I will be loading TBs
of 400MB images onto this system, so would prefer to be coerced into
making changes before then.
Thanks & regards,
Zoltan
*** Setup:
8 * 5900RPM Seagte 2TB drives on SiI3132 PCIx x1 card.
Linux gs0 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 19:52:42 UTC 2010
x86_64 GNU/Linux
mdadm - v3.1.4 - 31st August 2010
LVM version: 2.02.54(1) (2009-10-26)
Library version: 1.02.39 (2009-10-26)
Driver version: 4.15.0
mke2fs 1.41.11 (14-Mar-2010)
*** Commands used:
mdadm --create /dev/md0 --metadata=1.2 --auto=md --assume-clean
--bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6
--raid-devices=8 /dev/sd[abcdefgh]
mdadm --detail --scan >> /etc/mdadm.conf
mdadm --assemble /dev/md0
pvcreate --metadatasize 512 /dev/md0
vgcreate geo_vg0 /dev/md0
lvcreate -n geo_lvm0 -l 100%FREE geo_vg0
mkfs -t ext4 -m 1-b 4096 -E stride=128,stripe-width=768
/dev/geo_vg0/geo_lvm0
*** Results:
root@gs0:/home/geograph# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Tue Nov 16 17:28:37 2010
Raid Level : raid6
Array Size : 11721077760 (11178.09 GiB 12002.38 GB)
Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Nov 16 19:44:10 2010
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : gs0:0 (local to host gs0)
UUID : c35e7642:7d78c385:61a26db1:89a5ace5
Events : 3
Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
1 8 16 1 active sync /dev/sdb
2 8 32 2 active sync /dev/sdc
3 8 48 3 active sync /dev/sdd
4 8 64 4 active sync /dev/sde
5 8 80 5 active sync /dev/sdf
6 8 96 6 active sync /dev/sdg
7 8 112 7 active sync /dev/sdh
root@gs0:/home/geograph#
*******************************************************************
root@gs0:/home/geograph# pvdisplay
--- Physical volume ---
PV Name /dev/md0
VG Name geo_vg0
PV Size 10.92 TiB / not usable 0
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 2861463
Free PE 0
Allocated PE 2861463
PV UUID oE4Itq-rAwe-RQzK-JTnn-XAVk-vObv-ORx5HO
root@gs0:/home/geograph# vgdisplay
--- Volume group ---
VG Name geo_vg0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 10.92 TiB
PE Size 4.00 MiB
Total PE 2861463
Alloc PE / Size 2861463 / 10.92 TiB
Free PE / Size 0 / 0
VG UUID 0S3VyN-aMu4-U60o-hz3z-R6RL-hGUd-nJHCTg
root@gs0:/home/geograph# lvdisplay
--- Logical volume ---
LV Name /dev/geo_vg0/geo_lvm0
VG Name geo_vg0
LV UUID dnhY9R-mGO3-oRg6-0EP7-i6kt-2dIW-R9028r
LV Write Access read/write
LV Status available
# open 1
LV Size 10.92 TiB
Current LE 2861463
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 12288
Block device 252:0
root@gs0:/home/geograph#
*********************************************************
root@gs0:/home/geograph# echo `date` > zz_mkfs ; mkfs -t ext4 -m 1 -b
4096 -E stride=128,stripe-width=768 /dev/geo_vg0/geo_lvm0 ; echo `date`
>> zz_mkfs
mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=768 blocks
732536832 inodes, 2930138112 blocks
29301381 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
89421 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616,
78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
2560000000
[still running]
--
===========================================
Zoltan Szecsei PrGISc [PGP0031]
Geograph (Pty) Ltd.
P.O. Box 7, Muizenberg 7950, South Africa.
65 Main Road, Muizenberg 7945
Western Cape, South Africa.
34° 6'16.35"S 18°28'5.62"E
Tel: +27-21-7884897 Mobile: +27-83-6004028
Fax: +27-86-6115323 www.geograph.co.za
===========================================
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1153 / Virus Database: 424/3260 - Release Date: 11/16/10
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm raid6->LVM->ext4 setup
2010-11-16 18:08 mdadm raid6->LVM->ext4 setup Zoltan Szecsei
@ 2010-11-16 18:16 ` Roman Mamedov
2010-11-16 18:23 ` Zoltan Szecsei
2010-11-16 22:17 ` Luca Berra
1 sibling, 1 reply; 6+ messages in thread
From: Roman Mamedov @ 2010-11-16 18:16 UTC (permalink / raw)
To: Zoltan Szecsei; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 332 bytes --]
On Tue, 16 Nov 2010 20:08:44 +0200
Zoltan Szecsei <zoltans@geograph.co.za> wrote:
> 8 * 5900RPM Seagte 2TB drives on SiI3132 PCIx x1 card.
Excuse me, but how is that? This is a 2-port controller, so do you use port
multipliers, or 4 cards? (The latter would work an order of magnitude faster.)
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm raid6->LVM->ext4 setup
2010-11-16 18:16 ` Roman Mamedov
@ 2010-11-16 18:23 ` Zoltan Szecsei
0 siblings, 0 replies; 6+ messages in thread
From: Zoltan Szecsei @ 2010-11-16 18:23 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-raid
On 2010-11-16 20:16, Roman Mamedov wrote:
> On Tue, 16 Nov 2010 20:08:44 +0200
> Zoltan Szecsei<zoltans@geograph.co.za> wrote:
>
>
>> 8 * 5900RPM Seagte 2TB drives on SiI3132 PCIx x1 card.
>>
> Excuse me, but how is that? This is a 2-port controller, so do you use port
> multipliers, or 4 cards? (The latter would work an order of magnitude faster.)
>
>
The chassis has 2 * 4 port multipliers
http://www.chyangfun.com/pro01_2_3.asp
Regards,
Zoltan
--
===========================================
Zoltan Szecsei PrGISc [PGP0031]
Geograph (Pty) Ltd.
P.O. Box 7, Muizenberg 7950, South Africa.
65 Main Road, Muizenberg 7945
Western Cape, South Africa.
34° 6'16.35"S 18°28'5.62"E
Tel: +27-21-7884897 Mobile: +27-83-6004028
Fax: +27-86-6115323 www.geograph.co.za
===========================================
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1153 / Virus Database: 424/3260 - Release Date: 11/16/10
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm raid6->LVM->ext4 setup
2010-11-16 18:08 mdadm raid6->LVM->ext4 setup Zoltan Szecsei
2010-11-16 18:16 ` Roman Mamedov
@ 2010-11-16 22:17 ` Luca Berra
2010-11-16 23:39 ` Michal Soltys
1 sibling, 1 reply; 6+ messages in thread
From: Luca Berra @ 2010-11-16 22:17 UTC (permalink / raw)
To: linux-raid
On Tue, Nov 16, 2010 at 08:08:44PM +0200, Zoltan Szecsei wrote:
> *** Setup:
> 8 * 5900RPM Seagte 2TB drives on SiI3132 PCIx x1 card.
> Linux gs0 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 19:52:42 UTC 2010
> x86_64 GNU/Linux
> mdadm - v3.1.4 - 31st August 2010
> LVM version: 2.02.54(1) (2009-10-26)
> Library version: 1.02.39 (2009-10-26)
> Driver version: 4.15.0
> mke2fs 1.41.11 (14-Mar-2010)
>
> *** Commands used:
> mdadm --create /dev/md0 --metadata=1.2 --auto=md --assume-clean
> --bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6
> --raid-devices=8 /dev/sd[abcdefgh]
> mdadm --detail --scan >> /etc/mdadm.conf
> mdadm --assemble /dev/md0
as i said this is not useful
> pvcreate --metadatasize 512 /dev/md0
you should use --dataalignment, not --metadatasize, also you should
align to a full stripe
> vgcreate geo_vg0 /dev/md0
> lvcreate -n geo_lvm0 -l 100%FREE geo_vg0
> mkfs -t ext4 -m 1-b 4096 -E stride=128,stripe-width=768
> /dev/geo_vg0/geo_lvm0
--
Luca Berra -- bluca@comedia.it
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm raid6->LVM->ext4 setup
2010-11-16 22:17 ` Luca Berra
@ 2010-11-16 23:39 ` Michal Soltys
2010-11-17 7:49 ` Zoltan Szecsei
0 siblings, 1 reply; 6+ messages in thread
From: Michal Soltys @ 2010-11-16 23:39 UTC (permalink / raw)
To: linux-raid; +Cc: zoltans
On 10-11-16 23:17, Luca Berra wrote:
> On Tue, Nov 16, 2010 at 08:08:44PM +0200, Zoltan Szecsei wrote:
>>
>> *** Commands used:
>> mdadm --create /dev/md0 --metadata=1.2 --auto=md --assume-clean
>> --bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6
>> --raid-devices=8 /dev/sd[abcdefgh]
>> mdadm --detail --scan >> /etc/mdadm.conf
>> mdadm --assemble /dev/md0
> as i said this is not useful
>> pvcreate --metadatasize 512 /dev/md0
> you should use --dataalignment, not --metadatasize, also you should
Also with reference to metadatasize - keep in mind the value is rounded
up to the /next/ 64k boundary (or it was a while ago, perhaps it's
changed now). So in your case, 512 would really result with 576.
With reference to --dataalignment - there's also --dataalignoffset
option (although not really important in your setup).
You might consider adjusting
/sys/block/<yourmddevice>/md/stripe_cache_size to higher values when you
have everything set. It tends to give quite a performance boost.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm raid6->LVM->ext4 setup
2010-11-16 23:39 ` Michal Soltys
@ 2010-11-17 7:49 ` Zoltan Szecsei
0 siblings, 0 replies; 6+ messages in thread
From: Zoltan Szecsei @ 2010-11-17 7:49 UTC (permalink / raw)
To: linux-raid
On 2010-11-17 01:39, Michal Soltys wrote:
> On 10-11-16 23:17, Luca Berra wrote:
>> On Tue, Nov 16, 2010 at 08:08:44PM +0200, Zoltan Szecsei wrote:
>>>
>>> *** Commands used:
>>> mdadm --create /dev/md0 --metadata=1.2 --auto=md --assume-clean
>>> --bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6
>>> --raid-devices=8 /dev/sd[abcdefgh]
>>> mdadm --detail --scan >> /etc/mdadm.conf
>>> mdadm --assemble /dev/md0
>> as i said this is not useful
Accepted, but I also understood "not harmfull". I tried it out of
curiosity, it took a nano-second to complete, so didn't try to "undo it".
>>> pvcreate --metadatasize 512 /dev/md0
>> you should use --dataalignment, not --metadatasize, also you should
I checked out man pvcreate & lvm.conf had all the stuff below in it, so
I assumed it would sort itself out.
md_component_detection = 1
md_chunk_alignment = 1
data_alignment_detection = 1
data_alignment_offset_detection = 1
>
> Also with reference to metadatasize - keep in mind the value is
> rounded up to the /next/ 64k boundary (or it was a while ago, perhaps
> it's changed now). So in your case, 512 would really result with 576.
>
> With reference to --dataalignment - there's also --dataalignoffset
> option (although not really important in your setup).
>
> You might consider adjusting
> /sys/block/<yourmddevice>/md/stripe_cache_size to higher values when
> you have everything set. It tends to give quite a performance boost.
I had trouble getting my head around all the calculations, mainly
because I do not know if my sector size of 512 is falsely reported or
not (so I just accepted it as 512).
I am loading the array as we speak, with 200MB to 400MB tiff files. In
the past 11 hours approx 995GB was loaded, so that calculates to a write
speed of 25MB/s
I have no problem stopping & deleting this if you feel that this
represents problems in my setup, but then please can someone do the
alignment calculations & command syntax for me.
Regards to all,
Zoltan
--
===========================================
Zoltan Szecsei PrGISc [PGP0031]
Geograph (Pty) Ltd.
P.O. Box 7, Muizenberg 7950, South Africa.
65 Main Road, Muizenberg 7945
Western Cape, South Africa.
34° 6'16.35"S 18°28'5.62"E
Tel: +27-21-7884897 Mobile: +27-83-6004028
Fax: +27-86-6115323 www.geograph.co.za
===========================================
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1153 / Virus Database: 424/3261 - Release Date: 11/16/10
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-11-17 7:49 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-11-16 18:08 mdadm raid6->LVM->ext4 setup Zoltan Szecsei
2010-11-16 18:16 ` Roman Mamedov
2010-11-16 18:23 ` Zoltan Szecsei
2010-11-16 22:17 ` Luca Berra
2010-11-16 23:39 ` Michal Soltys
2010-11-17 7:49 ` Zoltan Szecsei
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.