RAID5 created by 8 disks works with xfs

RAID5 created by 8 disks works with xfs

[daobang wang]

Hi ALL,

    How to adjust the xfs and raid parameters to improve the total
performance when RAID5 created by 8 disks works with xfs, and i writed
a test program, which started 100 threads to write big files, 500MB
per file, and delete it after writing finish. Thank you very much.

Best Wishes,
Daobang Wang.

Re: RAID5 created by 8 disks works with xfs

[Mathias Burén]

On 31 March 2012 02:22, daobang wang <wangdb1981@gmail.com> wrote:
> Hi ALL,
>
>    How to adjust the xfs and raid parameters to improve the total
> performance when RAID5 created by 8 disks works with xfs, and i writed
> a test program, which started 100 threads to write big files, 500MB
> per file, and delete it after writing finish. Thank you very much.
>
> Best Wishes,
> Daobang Wang.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi,

See http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E
. Also see http://hep.kbfi.ee/index.php/IT/KernelTuning . For example,
RAID5 with 8 harddrives and 64K stripe size:

mkfs.xfs -d su=64k,sw=7 -l version=2,su=64k /dev/md0

Consider mounting the filesystem with logbufs=8,logbsize=256k

Kind regards,
Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 3/31/2012 2:59 AM, Mathias Burén wrote:
> On 31 March 2012 02:22, daobang wang <wangdb1981@gmail.com> wrote:
>> Hi ALL,
>>
>>    How to adjust the xfs and raid parameters to improve the total
>> performance when RAID5 created by 8 disks works with xfs, and i writed
>> a test program, which started 100 threads to write big files, 500MB
>> per file, and delete it after writing finish. Thank you very much.
>>
>> Best Wishes,
>> Daobang Wang.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Hi,
> 
> See http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E
> . Also see http://hep.kbfi.ee/index.php/IT/KernelTuning . For example,
> RAID5 with 8 harddrives and 64K stripe size:
> 
> mkfs.xfs -d su=64k,sw=7 -l version=2,su=64k /dev/md0

This is unnecessary.  mkfs.xfs creates w/stripe alignment automatically
when the target device is an md device.

> Consider mounting the filesystem with logbufs=8,logbsize=256k

This is unnecessary for two reasons:

1.  These are the default values in recent kernels
2.  His workload is the opposite of "metadata heavy"
    logbufs and logbsize exist for metadata operations
    to the journal, they are in memory journal write buffers

The OP's stated workload is 100 streaming writes of 500MB files.  This
is not anything close to a sane, real world workload.  Writing 100 x
500MB files in parallel to 7 spindles is an exercise in stupidity, and
especially to a RAID5 array with only 7 spindles.  The OP is pushing
those drives to their seek limit of about 150 head seeks/sec without
actually writing much data, and *that* is what is ruining his
performance.  What *should* be a streaming write workload of large files
has been turned into a massively random IO pattern due mostly to the
unrealistic write thread count, and partly to disk striping and the way
XFS allocation groups are created on a striped array.

Assuming these are 2TB drives, to get much closer to ideal write
performance, and make this more of a streaming workload, what the OP
should be doing is writing no more than 8 files in parallel to at least
8 different directories with XFS sitting on an md linear array of 4 md
RAID1 devices, assuming he needs protection from drive failure *and*
parallel write performance:

$ mdadm -C /dev/md0 -l 1 -n 2 /dev/sd[ab]
$ mdadm -C /dev/md1 -l 1 -n 2 /dev/sd[cd]
$ mdadm -C /dev/md2 -l 1 -n 2 /dev/sd[ef]
$ mdadm -C /dev/md3 -l 1 -n 2 /dev/sd[gh]
$ mdadm -C /dev/md4 -l linear -n 4 /dev/md[0-3]
$ mkfs.xfs -d agcount=8 /dev/md4

and mount with the inode64 option in fstab so we get the inode64
allocator, which spreads the metadata across all of the AGs instead of
stuffing in all in the first AG and yields other benefits.

This setup eliminates striping, tons of head seeks, and gets much closer
to pure streaming write performance.  Writing 8 files in parallel to 8
directories will cause XFS to put each file in a different allocation
group.  Since we created 8 AGs, this means we'll have 2 files being
written to each disk in parallel.  This reduces time wasted in head seek
latency by an order of magnitude and will dramatically increase disk
throughput in MB/s compared to the 100 files in parallel workload, which
again is simply stupid to do on this limited disk hardware.

This 100 file parallel write workload needs about 6 times as many
spindles to be realistic, configured as a linear array of 24 RAID1
devices and formatted with 48 AGs.  This would give you ~4 write streams
per drive, 2 per AG, or somewhere around 50% to 66% of the per drive
performance compared to the 8 drive 8 thread scenario I recommended above.

Final note:  It is simply not possible to optimize XFS nor mdraid to get
you any better performance when writing 100 x 500MB files in parallel.
The lack of sufficient spindles is the problem.

-- 
Stan


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Thanks to Mathias and stan, Here is the detail of the configuration.

1. RAID5 with 8 2TB ST32000644NS disks , i can extend to 16 disks.
the RAID5 created with Chunk Size of 64K and left-symmetric

2. Volume Group on RAID5 with full capacity

3. Logical Volume on the Volume Group with full capacity

4. XFS filesystem created on the Logical Volume with option "-f -i
size=512", and mount option is "-t xfs -o
defaults,usrquota,grpquota,noatime,nodiratime,nobarrier,delaylog,logbsize=262144",

5. The real application is 200 D1(2Mb/s) video streams write 500MB
files on the XFS.

This is the pressure testing, just verify the reliability of the
system, we will not use it in real envrionment, 100 video streams
writen is our goal. is there any clue for optimize the application?

Thank you very much.

Best Regards,
Daobang Wang.

On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 3/31/2012 2:59 AM, Mathias Burén wrote:
>> On 31 March 2012 02:22, daobang wang <wangdb1981@gmail.com> wrote:
>>> Hi ALL,
>>>
>>>    How to adjust the xfs and raid parameters to improve the total
>>> performance when RAID5 created by 8 disks works with xfs, and i writed
>>> a test program, which started 100 threads to write big files, 500MB
>>> per file, and delete it after writing finish. Thank you very much.
>>>
>>> Best Wishes,
>>> Daobang Wang.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> Hi,
>>
>> See
>> http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E
>> . Also see http://hep.kbfi.ee/index.php/IT/KernelTuning . For example,
>> RAID5 with 8 harddrives and 64K stripe size:
>>
>> mkfs.xfs -d su=64k,sw=7 -l version=2,su=64k /dev/md0
>
> This is unnecessary.  mkfs.xfs creates w/stripe alignment automatically
> when the target device is an md device.
>
>> Consider mounting the filesystem with logbufs=8,logbsize=256k
>
> This is unnecessary for two reasons:
>
> 1.  These are the default values in recent kernels
> 2.  His workload is the opposite of "metadata heavy"
>     logbufs and logbsize exist for metadata operations
>     to the journal, they are in memory journal write buffers
>
> The OP's stated workload is 100 streaming writes of 500MB files.  This
> is not anything close to a sane, real world workload.  Writing 100 x
> 500MB files in parallel to 7 spindles is an exercise in stupidity, and
> especially to a RAID5 array with only 7 spindles.  The OP is pushing
> those drives to their seek limit of about 150 head seeks/sec without
> actually writing much data, and *that* is what is ruining his
> performance.  What *should* be a streaming write workload of large files
> has been turned into a massively random IO pattern due mostly to the
> unrealistic write thread count, and partly to disk striping and the way
> XFS allocation groups are created on a striped array.
>
> Assuming these are 2TB drives, to get much closer to ideal write
> performance, and make this more of a streaming workload, what the OP
> should be doing is writing no more than 8 files in parallel to at least
> 8 different directories with XFS sitting on an md linear array of 4 md
> RAID1 devices, assuming he needs protection from drive failure *and*
> parallel write performance:
>
> $ mdadm -C /dev/md0 -l 1 -n 2 /dev/sd[ab]
> $ mdadm -C /dev/md1 -l 1 -n 2 /dev/sd[cd]
> $ mdadm -C /dev/md2 -l 1 -n 2 /dev/sd[ef]
> $ mdadm -C /dev/md3 -l 1 -n 2 /dev/sd[gh]
> $ mdadm -C /dev/md4 -l linear -n 4 /dev/md[0-3]
> $ mkfs.xfs -d agcount=8 /dev/md4
>
> and mount with the inode64 option in fstab so we get the inode64
> allocator, which spreads the metadata across all of the AGs instead of
> stuffing in all in the first AG and yields other benefits.
>
> This setup eliminates striping, tons of head seeks, and gets much closer
> to pure streaming write performance.  Writing 8 files in parallel to 8
> directories will cause XFS to put each file in a different allocation
> group.  Since we created 8 AGs, this means we'll have 2 files being
> written to each disk in parallel.  This reduces time wasted in head seek
> latency by an order of magnitude and will dramatically increase disk
> throughput in MB/s compared to the 100 files in parallel workload, which
> again is simply stupid to do on this limited disk hardware.
>
> This 100 file parallel write workload needs about 6 times as many
> spindles to be realistic, configured as a linear array of 24 RAID1
> devices and formatted with 48 AGs.  This would give you ~4 write streams
> per drive, 2 per AG, or somewhere around 50% to 66% of the per drive
> performance compared to the 8 drive 8 thread scenario I recommended above.
>
> Final note:  It is simply not possible to optimize XFS nor mdraid to get
> you any better performance when writing 100 x 500MB files in parallel.
> The lack of sufficient spindles is the problem.
>
> --
> Stan
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

There is another problem occurred, it seems that the file system was
damaged when the pressure is very high, it reported input/output error
when i typed ls or other command, and I tried to repair it with
xfs_repair /dev/vg00/lv0000, the xfs_repir alloc memory failed, we
have 4GB memory on the machine, and the logical volume was a little
more than 15TB, Could it be repair successfully if we have enough
memory?

Thank you very much!

Best Wishes,
Daobang Wang.

On 4/1/12, daobang wang <wangdb1981@gmail.com> wrote:
> Thanks to Mathias and stan, Here is the detail of the configuration.
>
> 1. RAID5 with 8 2TB ST32000644NS disks , i can extend to 16 disks.
> the RAID5 created with Chunk Size of 64K and left-symmetric
>
> 2. Volume Group on RAID5 with full capacity
>
> 3. Logical Volume on the Volume Group with full capacity
>
> 4. XFS filesystem created on the Logical Volume with option "-f -i
> size=512", and mount option is "-t xfs -o
> defaults,usrquota,grpquota,noatime,nodiratime,nobarrier,delaylog,logbsize=262144",
>
> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
> files on the XFS.
>
> This is the pressure testing, just verify the reliability of the
> system, we will not use it in real envrionment, 100 video streams
> writen is our goal. is there any clue for optimize the application?
>
> Thank you very much.
>
> Best Regards,
> Daobang Wang.
>
> On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>> On 3/31/2012 2:59 AM, Mathias Burén wrote:
>>> On 31 March 2012 02:22, daobang wang <wangdb1981@gmail.com> wrote:
>>>> Hi ALL,
>>>>
>>>>    How to adjust the xfs and raid parameters to improve the total
>>>> performance when RAID5 created by 8 disks works with xfs, and i writed
>>>> a test program, which started 100 threads to write big files, 500MB
>>>> per file, and delete it after writing finish. Thank you very much.
>>>>
>>>> Best Wishes,
>>>> Daobang Wang.
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid"
>>>> in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> Hi,
>>>
>>> See
>>> http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E
>>> . Also see http://hep.kbfi.ee/index.php/IT/KernelTuning . For example,
>>> RAID5 with 8 harddrives and 64K stripe size:
>>>
>>> mkfs.xfs -d su=64k,sw=7 -l version=2,su=64k /dev/md0
>>
>> This is unnecessary.  mkfs.xfs creates w/stripe alignment automatically
>> when the target device is an md device.
>>
>>> Consider mounting the filesystem with logbufs=8,logbsize=256k
>>
>> This is unnecessary for two reasons:
>>
>> 1.  These are the default values in recent kernels
>> 2.  His workload is the opposite of "metadata heavy"
>>     logbufs and logbsize exist for metadata operations
>>     to the journal, they are in memory journal write buffers
>>
>> The OP's stated workload is 100 streaming writes of 500MB files.  This
>> is not anything close to a sane, real world workload.  Writing 100 x
>> 500MB files in parallel to 7 spindles is an exercise in stupidity, and
>> especially to a RAID5 array with only 7 spindles.  The OP is pushing
>> those drives to their seek limit of about 150 head seeks/sec without
>> actually writing much data, and *that* is what is ruining his
>> performance.  What *should* be a streaming write workload of large files
>> has been turned into a massively random IO pattern due mostly to the
>> unrealistic write thread count, and partly to disk striping and the way
>> XFS allocation groups are created on a striped array.
>>
>> Assuming these are 2TB drives, to get much closer to ideal write
>> performance, and make this more of a streaming workload, what the OP
>> should be doing is writing no more than 8 files in parallel to at least
>> 8 different directories with XFS sitting on an md linear array of 4 md
>> RAID1 devices, assuming he needs protection from drive failure *and*
>> parallel write performance:
>>
>> $ mdadm -C /dev/md0 -l 1 -n 2 /dev/sd[ab]
>> $ mdadm -C /dev/md1 -l 1 -n 2 /dev/sd[cd]
>> $ mdadm -C /dev/md2 -l 1 -n 2 /dev/sd[ef]
>> $ mdadm -C /dev/md3 -l 1 -n 2 /dev/sd[gh]
>> $ mdadm -C /dev/md4 -l linear -n 4 /dev/md[0-3]
>> $ mkfs.xfs -d agcount=8 /dev/md4
>>
>> and mount with the inode64 option in fstab so we get the inode64
>> allocator, which spreads the metadata across all of the AGs instead of
>> stuffing in all in the first AG and yields other benefits.
>>
>> This setup eliminates striping, tons of head seeks, and gets much closer
>> to pure streaming write performance.  Writing 8 files in parallel to 8
>> directories will cause XFS to put each file in a different allocation
>> group.  Since we created 8 AGs, this means we'll have 2 files being
>> written to each disk in parallel.  This reduces time wasted in head seek
>> latency by an order of magnitude and will dramatically increase disk
>> throughput in MB/s compared to the 100 files in parallel workload, which
>> again is simply stupid to do on this limited disk hardware.
>>
>> This 100 file parallel write workload needs about 6 times as many
>> spindles to be realistic, configured as a linear array of 24 RAID1
>> devices and formatted with 48 AGs.  This would give you ~4 write streams
>> per drive, 2 per AG, or somewhere around 50% to 66% of the per drive
>> performance compared to the 8 drive 8 thread scenario I recommended
>> above.
>>
>> Final note:  It is simply not possible to optimize XFS nor mdraid to get
>> you any better performance when writing 100 x 500MB files in parallel.
>> The lack of sufficient spindles is the problem.
>>
>> --
>> Stan
>>
>>
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 3/31/2012 8:16 PM, daobang wang wrote:
> Thanks to Mathias and stan, Here is the detail of the configuration.
> 
> 1. RAID5 with 8 2TB ST32000644NS disks , i can extend to 16 disks.
> the RAID5 created with Chunk Size of 64K and left-symmetric
> 
> 2. Volume Group on RAID5 with full capacity
> 
> 3. Logical Volume on the Volume Group with full capacity

LVM will create unnecessary overhead for this workload.  Do not use it.
 Directly format the md device with XFS and proper alignment.  Again,
mkfs.xfs will do this automatically.

> 4. XFS filesystem created on the Logical Volume with option "-f -i
> size=512", and mount option is "-t xfs -o
> defaults,usrquota,grpquota,noatime,nodiratime,nobarrier,delaylog,logbsize=262144",

What kernel version are you using?  logbsize=262144 has been the default
for quite some time now.

NEVER disable barriers unless you have a quality BBWC RAID controller
and a good working UPS.  Are these 8 disks connected to a BBWC RAID
card?  Have you verified the write back cache is working properly?

> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
> files on the XFS.

This is a 50 MB/s raw stream rate with 200 writers to 200 target files.
 It is very likely that neither 8 nor 16 disks in RAID5 will be able to
sync this rate due to excessive head seeking, as I previously mentioned.
 Having LVM layered between XFS and mdraid will make this situation even
worse.

> This is the pressure testing, just verify the reliability of the
> system, we will not use it in real envrionment, 100 video streams
> writen is our goal. 

So now you have an aggregate 25 MB/s random write workload with 100
writers.  This is still going to likely be too much for an 8 or 16 disk
RAID5 array for the same reason as 200 threads--too many disk seeks.

> is there any clue for optimize the application?

This is the Linux-raid list.  We can help you optimize your RAID, and
since many of us use XFS, we can help you there as well.  I've never
seen your application so it is impossible to suggest how to optimize it.
 As far as optimizing your 16 disks with mdraid and XFS, I've already
given you pointers on how you need to configure your storage for optimal
performance with this workload.  However, experience tells me you simply
don't have enough spindles to sync all these parallel writers if you
also need redundancy.

The fact that you're disabling barriers likely in lieu of BBWC seems to
indicate you're not terribly worried about losing all your data in the
event of a drive failure.  If this is the case, 16 x 7.2k spindles with
6 parallel writers per spindle at .25 MB/s each might work.  Simply
create an md linear array with 16 disks, no LVM, and format it with

$ mkfs.xfs -d agcount=48 /dev/md0

and configure you application to write each video stream file to a
different directory.  This will allow for 96 total streams, 6 being
written to each drive concurrently, for 6*.25 MB/s = 1.5MB/s per drive.

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

Flaky mouse button cause a premature send.  Sorry for the duplicate.

On 3/31/2012 8:16 PM, daobang wang wrote:
> Thanks to Mathias and stan, Here is the detail of the configuration.
> 
> 1. RAID5 with 8 2TB ST32000644NS disks , i can extend to 16 disks.
> the RAID5 created with Chunk Size of 64K and left-symmetric
> 
> 2. Volume Group on RAID5 with full capacity
> 
> 3. Logical Volume on the Volume Group with full capacity

LVM will create unnecessary overhead for this workload.  Do not use it.
 Directly format the md device with XFS and proper alignment.  Again,
mkfs.xfs will do this automatically.

> 4. XFS filesystem created on the Logical Volume with option "-f -i
> size=512", and mount option is "-t xfs -o
> defaults,usrquota,grpquota,noatime,nodiratime,nobarrier,delaylog,logbsize=262144",

What kernel version are you using?  logbsize=262144 has been the default
for quite some time now.

NEVER disable barriers unless you have a quality BBWC RAID controller
and a good working UPS.  Are these 8 disks connected to a BBWC RAID
card?  Have you verified the write back cache is working properly?

> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
> files on the XFS.

This is a 50 MB/s raw stream rate with 200 writers to 200 target files.
 It is very likely that neither 8 nor 16 disks in RAID5 will be able to
sync this rate due to excessive head seeking, as I previously mentioned.
 Having LVM layered between XFS and mdraid will make this situation even
worse.

> This is the pressure testing, just verify the reliability of the
> system, we will not use it in real envrionment, 100 video streams
> writen is our goal. 

So now you have an aggregate 25 MB/s random write workload with 100
writers.  This is still going to likely be too much for an 8 or 16 disk
RAID5 array for the same reason as 200 threads--too many disk seeks.

> is there any clue for optimize the application?

This is the Linux-raid list.  We can help you optimize your RAID, and
since many of us use XFS, we can help you there as well.  I've never
seen your application so it is impossible to suggest how to optimize it.
 As far as optimizing your 16 disks with mdraid and XFS, I've already
given you pointers on how you need to configure your storage for optimal
performance with this workload.  However, experience tells me you simply
don't have enough spindles to sync all these parallel writers if you
also need redundancy.

The fact that you're disabling barriers likely in lieu of BBWC seems to
indicate you're not terribly worried about losing all your data in the
event of a drive failure.  If this is the case, 16 x 7.2k spindles with
6 parallel writers per spindle at .25 MB/s each might work.  Simply
create an md linear array with 16 disks, no LVM, format it with

$ mkfs.xfs -d agcount=48 /dev/md0

and configure your application to write each video stream file to a
different directory.  Writing to multiple directories is what drives XFS
parallelism.  This will allow for 96 parallel write streams, 6 to each
drive concurrently, for 6*.25 MB/s = 1.5MB/s per drive, 24 MB/s
aggregate.  This might keep seeks low enough to achieve the parallel
throughput you need.

I'm guessing since you didn't comment on my previous recommendation that
you don't yet understand the benefits of linear concatenation, and using
XFS allocation groups to drive parallelism, instead of blindly assuming
striping will do the job--it often cannot.  XFS on linear array can
decrease the seek rate on each drive substantially vs striped RAID.
Keeping the seek rate low increases throughput of parallel writers.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Thank you very much!
I got it, so we can remove the Volume Group and Logical Volume to save resource.
And i will try RAID5 with 16 disks to write 96 total streams again.

I used the Linux kernel 2.6.26.4.

And we do not have BBWC

The application has 16kb cache per stream, Is it possible to optimize
it if we use 32kb or 64kb cache?

On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 3/31/2012 8:16 PM, daobang wang wrote:
>> Thanks to Mathias and stan, Here is the detail of the configuration.
>>
>> 1. RAID5 with 8 2TB ST32000644NS disks , i can extend to 16 disks.
>> the RAID5 created with Chunk Size of 64K and left-symmetric
>>
>> 2. Volume Group on RAID5 with full capacity
>>
>> 3. Logical Volume on the Volume Group with full capacity
>
> LVM will create unnecessary overhead for this workload.  Do not use it.
>  Directly format the md device with XFS and proper alignment.  Again,
> mkfs.xfs will do this automatically.
>
>> 4. XFS filesystem created on the Logical Volume with option "-f -i
>> size=512", and mount option is "-t xfs -o
>> defaults,usrquota,grpquota,noatime,nodiratime,nobarrier,delaylog,logbsize=262144",
>
> What kernel version are you using?  logbsize=262144 has been the default
> for quite some time now.
>
> NEVER disable barriers unless you have a quality BBWC RAID controller
> and a good working UPS.  Are these 8 disks connected to a BBWC RAID
> card?  Have you verified the write back cache is working properly?
>
>> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
>> files on the XFS.
>
> This is a 50 MB/s raw stream rate with 200 writers to 200 target files.
>  It is very likely that neither 8 nor 16 disks in RAID5 will be able to
> sync this rate due to excessive head seeking, as I previously mentioned.
>  Having LVM layered between XFS and mdraid will make this situation even
> worse.
>
>> This is the pressure testing, just verify the reliability of the
>> system, we will not use it in real envrionment, 100 video streams
>> writen is our goal.
>
> So now you have an aggregate 25 MB/s random write workload with 100
> writers.  This is still going to likely be too much for an 8 or 16 disk
> RAID5 array for the same reason as 200 threads--too many disk seeks.
>
>> is there any clue for optimize the application?
>
> This is the Linux-raid list.  We can help you optimize your RAID, and
> since many of us use XFS, we can help you there as well.  I've never
> seen your application so it is impossible to suggest how to optimize it.
>  As far as optimizing your 16 disks with mdraid and XFS, I've already
> given you pointers on how you need to configure your storage for optimal
> performance with this workload.  However, experience tells me you simply
> don't have enough spindles to sync all these parallel writers if you
> also need redundancy.
>
> The fact that you're disabling barriers likely in lieu of BBWC seems to
> indicate you're not terribly worried about losing all your data in the
> event of a drive failure.  If this is the case, 16 x 7.2k spindles with
> 6 parallel writers per spindle at .25 MB/s each might work.  Simply
> create an md linear array with 16 disks, no LVM, and format it with
>
> $ mkfs.xfs -d agcount=48 /dev/md0
>
> and configure you application to write each video stream file to a
> different directory.  This will allow for 96 total streams, 6 being
> written to each drive concurrently, for 6*.25 MB/s = 1.5MB/s per drive.
>
>

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 3/31/2012 9:05 PM, daobang wang wrote:
> There is another problem occurred, it seems that the file system was
> damaged when the pressure is very high, 

What kernel version are you using?  Did you get an oops?  What's in dmesg?

> it reported input/output error

Actual errors would be very helpful.

> when i typed ls or other command, and I tried to repair it with
> xfs_repair /dev/vg00/lv0000, the xfs_repir alloc memory failed, we

Did XFS automatically unmount the filesystem?  If not, the error
reported may not indicate a problem with the filesystem.  XFS shuts
filesystems down when it encounters serious problems.

> have 4GB memory on the machine, and the logical volume was a little
> more than 15TB, Could it be repair successfully if we have enough
> memory?

Hard to say.  Depends on what happened and the extent of the damage, if
any.  You've presented no log or debug information.  I would think 4GB
should be plenty to run xfs_repair.  Try

$ xfs_repair -n -vv -m 1 /dev/vg00/lv0000

dmem = in the output tells you how much RAM is needed for xfs_repair.
If it's more than 2GB and you're using a PAE kernel, switch to a 64 bit
kernel and 64 bit userland.  If dmem is over 4GB then you need more
DIMMs in the machine.  Or maybe simply dropping caches before running
xfs_repair might help:

$ echo 3 > /proc/sys/vm/drop_caches

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/1/2012 12:12 AM, daobang wang wrote:
> Thank you very much!
> I got it, so we can remove the Volume Group and Logical Volume to save resource.
> And i will try RAID5 with 16 disks to write 96 total streams again.

Why do you keep insisting on RAID5?!?!  It is not suitable for your
workload.  It sucks Monday through Saturday and twice on Sunday for this
workload.

Test your 16 drive RAID5 array head to head with the linear array + XFS
architecture I gave you instructions to create, and report back your
results.

> I used the Linux kernel 2.6.26.4.

Which distro?

2.6.26 is *ancient* and has storage layer bugs.  It does NOT have
delaylog, which was introduced in 2.6.35, and wasn't fully performant
until 2.6.38+.

You're building and testing a new platform with a terribly obsolete
distribution.  You need a much newer kernel and distro.  3.0.x would be
best.  Debian 6.0.4 with a backport 3.0.x kernel would be a good start.

> And we do not have BBWC

Then you must re-enable barriers or perennially suffer more filesystem
problems.  I simply cannot emphasize enough how critical write barriers
are to filesystem consistency.

> The application has 16kb cache per stream, Is it possible to optimize
> it if we use 32kb or 64kb cache?

No.  Read the app's documentation.  This caching is to prevent dropped
frames in the recorded file.  Increasing this value won't affect disk
performance.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

So sorry, the kernel version should be 2.6.36.4, and we do not use
distro, we compiled the kernel codes and user space codes.

I'm duplicating the input/output error issue, the system was
restarted, I will dump the dmesg log if i can duplicate it.

Thanks again,
Daobang Wang.

On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/1/2012 12:12 AM, daobang wang wrote:
>> Thank you very much!
>> I got it, so we can remove the Volume Group and Logical Volume to save
>> resource.
>> And i will try RAID5 with 16 disks to write 96 total streams again.
>
> Why do you keep insisting on RAID5?!?!  It is not suitable for your
> workload.  It sucks Monday through Saturday and twice on Sunday for this
> workload.
>
> Test your 16 drive RAID5 array head to head with the linear array + XFS
> architecture I gave you instructions to create, and report back your
> results.
>
>> I used the Linux kernel 2.6.26.4.
>
> Which distro?
>
> 2.6.26 is *ancient* and has storage layer bugs.  It does NOT have
> delaylog, which was introduced in 2.6.35, and wasn't fully performant
> until 2.6.38+.
>
> You're building and testing a new platform with a terribly obsolete
> distribution.  You need a much newer kernel and distro.  3.0.x would be
> best.  Debian 6.0.4 with a backport 3.0.x kernel would be a good start.
>
>> And we do not have BBWC
>
> Then you must re-enable barriers or perennially suffer more filesystem
> problems.  I simply cannot emphasize enough how critical write barriers
> are to filesystem consistency.
>
>> The application has 16kb cache per stream, Is it possible to optimize
>> it if we use 32kb or 64kb cache?
>
> No.  Read the app's documentation.  This caching is to prevent dropped
> frames in the recorded file.  Increasing this value won't affect disk
> performance.
>
> --
> Stan
>

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

I have the different opinion, the application does not write the disk
directly, disk IOs will be merged before writen in kernel, just we can
not caculate how many IOs will be merged.

On 4/1/12, daobang wang <wangdb1981@gmail.com> wrote:
> So sorry, the kernel version should be 2.6.36.4, and we do not use
> distro, we compiled the kernel codes and user space codes.
>
> I'm duplicating the input/output error issue, the system was
> restarted, I will dump the dmesg log if i can duplicate it.
>
> Thanks again,
> Daobang Wang.
>
> On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>> On 4/1/2012 12:12 AM, daobang wang wrote:
>>> Thank you very much!
>>> I got it, so we can remove the Volume Group and Logical Volume to save
>>> resource.
>>> And i will try RAID5 with 16 disks to write 96 total streams again.
>>
>> Why do you keep insisting on RAID5?!?!  It is not suitable for your
>> workload.  It sucks Monday through Saturday and twice on Sunday for this
>> workload.
>>
>> Test your 16 drive RAID5 array head to head with the linear array + XFS
>> architecture I gave you instructions to create, and report back your
>> results.
>>
>>> I used the Linux kernel 2.6.26.4.
>>
>> Which distro?
>>
>> 2.6.26 is *ancient* and has storage layer bugs.  It does NOT have
>> delaylog, which was introduced in 2.6.35, and wasn't fully performant
>> until 2.6.38+.
>>
>> You're building and testing a new platform with a terribly obsolete
>> distribution.  You need a much newer kernel and distro.  3.0.x would be
>> best.  Debian 6.0.4 with a backport 3.0.x kernel would be a good start.
>>
>>> And we do not have BBWC
>>
>> Then you must re-enable barriers or perennially suffer more filesystem
>> problems.  I simply cannot emphasize enough how critical write barriers
>> are to filesystem consistency.
>>
>>> The application has 16kb cache per stream, Is it possible to optimize
>>> it if we use 32kb or 64kb cache?
>>
>> No.  Read the app's documentation.  This caching is to prevent dropped
>> frames in the recorded file.  Increasing this value won't affect disk
>> performance.
>>
>> --
>> Stan
>>
>

Re: RAID5 created by 8 disks works with xfs

[Marcus Sorensen]

Streaming workloads don't benefit much from writeback cache.
Writeback can absorb spikes, but if you have a constant load that goes
beyond what your disks can handle, you'll have good performance
exactly to the point where your writeback is full. Once you hit
dirty_bytes, dirty_ratio, or the timeout, your system will be crushed
with I/O beyond recovery. It's best to limit your writeback cache to a
relatively small number with such a constant IO load.

You are right that merging could help to some degree, but you likely
won't be merging I/Os from separate streams, so your workload is still
terribly random and you just end up with larger random I/Os. I don't
think it will make up for the difference between your workload and
your configuration.

On Sun, Apr 1, 2012 at 12:20 AM, daobang wang <wangdb1981@gmail.com> wrote:
> I have the different opinion, the application does not write the disk
> directly, disk IOs will be merged before writen in kernel, just we can
> not caculate how many IOs will be merged.
>
> On 4/1/12, daobang wang <wangdb1981@gmail.com> wrote:
>> So sorry, the kernel version should be 2.6.36.4, and we do not use
>> distro, we compiled the kernel codes and user space codes.
>>
>> I'm duplicating the input/output error issue, the system was
>> restarted, I will dump the dmesg log if i can duplicate it.
>>
>> Thanks again,
>> Daobang Wang.
>>
>> On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>> On 4/1/2012 12:12 AM, daobang wang wrote:
>>>> Thank you very much!
>>>> I got it, so we can remove the Volume Group and Logical Volume to save
>>>> resource.
>>>> And i will try RAID5 with 16 disks to write 96 total streams again.
>>>
>>> Why do you keep insisting on RAID5?!?!  It is not suitable for your
>>> workload.  It sucks Monday through Saturday and twice on Sunday for this
>>> workload.
>>>
>>> Test your 16 drive RAID5 array head to head with the linear array + XFS
>>> architecture I gave you instructions to create, and report back your
>>> results.
>>>
>>>> I used the Linux kernel 2.6.26.4.
>>>
>>> Which distro?
>>>
>>> 2.6.26 is *ancient* and has storage layer bugs.  It does NOT have
>>> delaylog, which was introduced in 2.6.35, and wasn't fully performant
>>> until 2.6.38+.
>>>
>>> You're building and testing a new platform with a terribly obsolete
>>> distribution.  You need a much newer kernel and distro.  3.0.x would be
>>> best.  Debian 6.0.4 with a backport 3.0.x kernel would be a good start.
>>>
>>>> And we do not have BBWC
>>>
>>> Then you must re-enable barriers or perennially suffer more filesystem
>>> problems.  I simply cannot emphasize enough how critical write barriers
>>> are to filesystem consistency.
>>>
>>>> The application has 16kb cache per stream, Is it possible to optimize
>>>> it if we use 32kb or 64kb cache?
>>>
>>> No.  Read the app's documentation.  This caching is to prevent dropped
>>> frames in the recorded file.  Increasing this value won't affect disk
>>> performance.
>>>
>>> --
>>> Stan
>>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 created by 8 disks works with xfs

[John Robinson]

On 01/04/2012 05:52, Stan Hoeppner wrote:
[...]
>> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
>> files on the XFS.
>
> This is a 50 MB/s raw stream rate with 200 writers to 200 target files.
>   It is very likely that neither 8 nor 16 disks in RAID5 will be able to
> sync this rate due to excessive head seeking, as I previously mentioned.

Handwaving slightly here, but I wonder whether using a lot of write 
cache will help aggregate writes here, since it's not actually random 
writes. If I were to model it as 250MB written every 5 seconds from 200 
writers, that might be able to be persuaded into 40 writes per second, 
which shouldn't cause excessive head seeking. It would be helpful if 
there were also enough additional cache that the reads for any RAID5 
read-modify-write cycles mostly came from cache and didn't need seeks. 
It should be possible to set this up such that, in the event of a crash 
or hardware failure, the filesystem and files are consistent, though 
obviously the last 5 seconds of each file is likely to be lost.

Having said that, even if it can be persuaded to work, I agree that it's 
still the wrong way to go; RAID10 or indeed RAID1+linear look like much 
better solutions.

Cheers,

John.

Re: RAID5 created by 8 disks works with xfs

[David Brown]

On 01/04/12 07:40, Stan Hoeppner wrote:
> On 4/1/2012 12:12 AM, daobang wang wrote:
>> Thank you very much!
>> I got it, so we can remove the Volume Group and Logical Volume to save resource.
>> And i will try RAID5 with 16 disks to write 96 total streams again.
>
> Why do you keep insisting on RAID5?!?!  It is not suitable for your
> workload.  It sucks Monday through Saturday and twice on Sunday for this
> workload.
>

My thoughts on this setup are that RAID5 (or RAID6) is a poor choice - 
it can quickly cause a mess for streaming writes.  Even if the streams 
themselves can mostly end up as full stripe writes, odd writes such as 
the end of a file, metadata writes, log data writes, etc., will mean 
read-modify-write operations that will cripple the write performance for 
the rest of the operations.

And if a disk fails so that you are running degraded, it will be hopeless.

Either drop the redundancy requirement entirely (maybe by making sure 
other backups are in order), or double the spindles and use RAID1 / RAID10.


For an application like this, it would probably make sense to put the 
xfs log (and the mdraid bitmap file, if you are using one) on a separate 
disk - perhaps a small SSD.

Re: RAID5 created by 8 disks works with xfs

[John Robinson]

On 01/04/2012 11:33, David Brown wrote:
[...]
> For an application like this, it would probably make sense to put the
> xfs log (and the mdraid bitmap file, if you are using one) on a separate
> disk - perhaps a small SSD.

Now that you've suggested that, it occurs to me that for an application 
like this, the OP might be better doing his multiple slow streams to a 
spool folder on SSD, and copying over to the big array of spinning rust 
when each stream completes. 200 streams of 500MB is 100GB of data, so a 
pair of slightly larger SSDs in RAID1 (or RAID10, to balance the reads 
coming off) would do nicely as a spool area.

This might also be a good application for bcache, FlashCache or whatever.

Cheers,

John.

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/1/2012 1:20 AM, daobang wang wrote:
> I have the different opinion, the application does not write the disk
> directly, disk IOs will be merged before writen in kernel, just we can
> not caculate how many IOs will be merged.

Your original workload description was writing 200 or 100 video streams
to 200 or 100 different files in parallel.  What you state above is
different than the original.  Maybe you could give us a little more
detail about how this application actually does its file IO.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/1/2012 2:08 AM, Marcus Sorensen wrote:
> Streaming workloads don't benefit much from writeback cache.
> Writeback can absorb spikes, but if you have a constant load that goes
> beyond what your disks can handle, you'll have good performance
> exactly to the point where your writeback is full. Once you hit
> dirty_bytes, dirty_ratio, or the timeout, your system will be crushed
> with I/O beyond recovery. It's best to limit your writeback cache to a
> relatively small number with such a constant IO load.

My comments WRT battery or flash backed write cache, whether write back
or write through, were strictly related to running with XFS barriers
disabled.  The only scenaio where you can safely disable XFS barriers is
when you have a properly functioning BBWC RAID controller, whether an
HBA, or a host independent external array such as a SAN controller.

Of course, I agree 100% that write cache yields little benefit with high
throughput workloads, and especially those generating high seek rates to
boot.  The workload described is many parallel streaming writes of .25
MB/s each.  If we use 96 streams, that's "only" 24 MB/s aggregate.  But
as each of the 16 drives will likely be hitting its seek ceiling of
~150/s using XFS on striped RAID, the aggregate throughput of the 15
RAID5 spindles will probably be less than 10 MB/s.

Using the linear array with XFS instead of RAID5 will eliminate much of
the head seeking, increasing throughput.  The increase may not be a
huge, but it will be enough to handle many more parallel write streams
than RAID5 before the drives hit their seek ceiling.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/1/2012 5:33 AM, David Brown wrote:

> For an application like this, it would probably make sense to put the
> xfs log (and the mdraid bitmap file, if you are using one) on a separate
> disk - perhaps a small SSD.

XFS only journals metadata changes.  Thus an external journal is not
needed here as there is no metadata in the workload.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[David Brown]

On 01/04/2012 14:28, John Robinson wrote:
> On 01/04/2012 11:33, David Brown wrote:
> [...]
>> For an application like this, it would probably make sense to put the
>> xfs log (and the mdraid bitmap file, if you are using one) on a separate
>> disk - perhaps a small SSD.
>
> Now that you've suggested that, it occurs to me that for an application
> like this, the OP might be better doing his multiple slow streams to a
> spool folder on SSD, and copying over to the big array of spinning rust
> when each stream completes. 200 streams of 500MB is 100GB of data, so a
> pair of slightly larger SSDs in RAID1 (or RAID10, to balance the reads
> coming off) would do nicely as a spool area.

If the OP wants to do that, he must be careful when looking at the rated 
write speeds for the SSDs.  SSDs need a bit of a break on occasion to 
get time to do their garbage collection - their long-term streamed write 
speed is therefore lower than their short-term write speeds (and that's 
the figure that will be given in the spec).  But if you take that into 
account, and short-partition the SSD's a little (i.e., leave 10% 
unpartitioned, and put the raid1 on the partition rather than the disk) 
so that there are always plenty of free blocks when they are needed, 
that could be a very good solution.

>
> This might also be a good application for bcache, FlashCache or whatever.
>
> Cheers,
>
> John.
>

Re: RAID5 created by 8 disks works with xfs

[David Brown]

On 02/04/2012 07:43, Stan Hoeppner wrote:
> On 4/1/2012 5:33 AM, David Brown wrote:
>
>> For an application like this, it would probably make sense to put the
>> xfs log (and the mdraid bitmap file, if you are using one) on a separate
>> disk - perhaps a small SSD.
>
> XFS only journals metadata changes.  Thus an external journal is not
> needed here as there is no metadata in the workload.
>

Won't there still be metadata changes as each file is closed and a new 
one started, or as more space is allocated to the files being written (I 
know xfs has delayed allocation and other tricks to minimise this, but 
it can't be eliminated entirely).  Each time the metadata or log needs 
written, you'll get a big discontinuity in the writing as the disks 
heads need to jump around (especially for RAID5!).

It would make sense to have a separate disk (again, SSD is logical) for 
the OS and other files anyway, keeping the big array for the data. 
Putting the xfs log there too would surely be a small but helpful step?

Re: RAID5 created by 8 disks works with xfs

[John Robinson]

On 02/04/2012 04:15, Jack Wang wrote:
> 2012/4/1 John Robinson <john.robinson@anonymous.org.uk
> <mailto:john.robinson@anonymous.org.uk>>
[...]
>     Now that you've suggested that, it occurs to me that for an
>     application like this, the OP might be better doing his multiple
>     slow streams to a spool folder on SSD, and copying over to the big
>     array of spinning rust when each stream completes. 200 streams of
>     500MB is 100GB of data, so a pair of slightly larger SSDs in RAID1
>     (or RAID10, to balance the reads coming off) would do nicely as a
>     spool area.
>
>     This might also be a good application for bcache, FlashCache or
>     whatever.
>
> Using bcache/flashcache is what I'm considering, does above
> configuration option still needed?

Using a pair of SSDs? I think if you're adding a cache to a redundant 
array, you ought to make the cache redundant too.

Using RAID10 to balance reads coming off the redundant pair of SSDs? 
That would depend on how bcache or flashcache is written; if it's 
single-threaded or does large copies from cache to the backing, then 
yes, you may see better performance or at least more balanced usage of 
the two SSDs, and surely won't see worse performance.

Cheers,

John.

Re: RAID5 created by 8 disks works with xfs

[Jack Wang]

2012/4/2 John Robinson <john.robinson@anonymous.org.uk>
>
> On 02/04/2012 04:15, Jack Wang wrote:
>>
>> 2012/4/1 John Robinson <john.robinson@anonymous.org.uk
>> <mailto:john.robinson@anonymous.org.uk>>
>
> [...]
>>
>>    Now that you've suggested that, it occurs to me that for an
>>    application like this, the OP might be better doing his multiple
>>    slow streams to a spool folder on SSD, and copying over to the big
>>    array of spinning rust when each stream completes. 200 streams of
>>    500MB is 100GB of data, so a pair of slightly larger SSDs in RAID1
>>    (or RAID10, to balance the reads coming off) would do nicely as a
>>    spool area.
>>
>>    This might also be a good application for bcache, FlashCache or
>>    whatever.
>>
>> Using bcache/flashcache is what I'm considering, does above
>> configuration option still needed?
>
>
> Using a pair of SSDs? I think if you're adding a cache to a redundant array, you ought to make the cache redundant too.
>
> Using RAID10 to balance reads coming off the redundant pair of SSDs? That would depend on how bcache or flashcache is written; if it's single-threaded or does large copies from cache to the backing, then yes, you may see better performance or at least more balanced usage of the two SSDs, and surely won't see worse performance.
>
> Cheers,
>
> John.


John,

Thanks for reply , I mean using SSD as cache device, and raid5
16*7200rpm disks, is this works for streaming workload above?

I know bcache is try to turn rand write to seqencial write, but most
ssd is not good at RAND write as I know.

Cheers,

Jack
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 created by 8 disks works with xfs

[John Robinson]

On 02/04/2012 11:01, Jack Wang wrote:
> 2012/4/2 John Robinson<john.robinson@anonymous.org.uk>
>> On 02/04/2012 04:15, Jack Wang wrote:
>>> 2012/4/1 John Robinson<john.robinson@anonymous.org.uk
>>> <mailto:john.robinson@anonymous.org.uk>>
>> [...]
>>>     This might also be a good application for bcache, FlashCache or
>>>     whatever.
>>>
>>> Using bcache/flashcache is what I'm considering, does above
>>> configuration option still needed?
[...]
> Thanks for reply , I mean using SSD as cache device, and raid5
> 16*7200rpm disks, is this works for streaming workload above?
>
> I know bcache is try to turn rand write to seqencial write, but most
> ssd is not good at RAND write as I know.

I don't know where you got that from; SSDs are absolutely brilliant at 
small random writes and reads, hundreds of times better than spinning 
hard discs. bcache and flashcache were written specifically to take 
advantage of SSDs' high I/Os per second for small (random) reads and writes.

I wouldn't touch RAID5 for anything other than 3 drives; I'd use RAID6 
or RAID60 for 4-plus where I didn't need the better small/random write 
performance of RAID10. I haven't actually used bcache or flashcache, 
which is why I introduced them with "might".

My fantasy configuration in your 16-drive chassis would be 2 6-drive 
RAID6s, striped together in RAID0, with an SSD cache over the top built 
from two SSDs in RAID10 (or if I was feeling really paranoid, 3 SSDs in 
RAID10,n3), with the remaining slot containing a hot spare for the RAID6s.

Cheers,

John.

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/2/2012 2:04 AM, David Brown wrote:
> On 02/04/2012 07:43, Stan Hoeppner wrote:
>> On 4/1/2012 5:33 AM, David Brown wrote:
>>
>>> For an application like this, it would probably make sense to put the
>>> xfs log (and the mdraid bitmap file, if you are using one) on a separate
>>> disk - perhaps a small SSD.
>>
>> XFS only journals metadata changes.  Thus an external journal is not
>> needed here as there is no metadata in the workload.
>>
> 
> Won't there still be metadata changes as each file is closed and a new
> one started, 

Yes, creating one file requires one journal operation consisting of a
few dozen to a few hundred bytes of journal data.  With delaylog this
will very likely be consolidated into a single IO to the journal, or a
handful of IOs.  With the workload in question the number of journal IOs
is trivial.

> or as more space is allocated to the files being written (I

Allocation occurs once when the file is created.

> know xfs has delayed allocation and other tricks to minimise this, but
> it can't be eliminated entirely).  Each time the metadata or log needs
> written, you'll get a big discontinuity in the writing as the disks
> heads need to jump around (especially for RAID5!).

The ratio of file IOs to metadata IOs in this workload is well north of
millions to one.  Again, the metadata IOPS in this workload are trivial.
 RAID level has no bearing on metadata IOPS here.  Though it does for
the actual workload, as I already discussed in detail.

> It would make sense to have a separate disk (again, SSD is logical) for
> the OS and other files anyway, keeping the big array for the data.
> Putting the xfs log there too would surely be a small but helpful step?

No it doesn't make sense.  See above.  Additionally, an external journal
can only be used if the journal device is behind battery backed RAID
cache.  Due to a design issue, XFS cannot perform barrier operations on
an external journal device, only to an internal journal.  Without
barriers or BBWC you have the opportunity for constant filesystem
corruption.  The OP doesn't have a BBWC RAID controller, nor any RAID
controller at all.  So an external journal isn't an option here anyway.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/2/2012 5:28 AM, John Robinson wrote:

> My fantasy configuration in your 16-drive chassis would be 2 6-drive
> RAID6s, striped together in RAID0, with an SSD cache over the top built
> from two SSDs in RAID10 (or if I was feeling really paranoid, 3 SSDs in
> RAID10,n3), with the remaining slot containing a hot spare for the RAID6s.

This advice is not sound.  The introductory "My fantasy" tends to put an
exclamation point on this fact.  This configuration is completely
unsuitable for the workload described.

I already described the most optimal configuration for this workload,
given the OP's current hardware, including all the commands to build it.
 The staged SSD idea has merit, as well as the SSD write cache idea.  If
going this route, two SLC SSDs should simply be mirrored.  Using 3 of
them or a RAID 1E variant on two of them is silly.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Thanks to all. Because the end-user is adhere to use RAID5, I need to
test how many video streams could be run on it really, and give the
suggestion to them. Thanks again.

Best Wishes,
Daobang Wang.

On 4/2/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/1/2012 2:08 AM, Marcus Sorensen wrote:
>> Streaming workloads don't benefit much from writeback cache.
>> Writeback can absorb spikes, but if you have a constant load that goes
>> beyond what your disks can handle, you'll have good performance
>> exactly to the point where your writeback is full. Once you hit
>> dirty_bytes, dirty_ratio, or the timeout, your system will be crushed
>> with I/O beyond recovery. It's best to limit your writeback cache to a
>> relatively small number with such a constant IO load.
>
> My comments WRT battery or flash backed write cache, whether write back
> or write through, were strictly related to running with XFS barriers
> disabled.  The only scenaio where you can safely disable XFS barriers is
> when you have a properly functioning BBWC RAID controller, whether an
> HBA, or a host independent external array such as a SAN controller.
>
> Of course, I agree 100% that write cache yields little benefit with high
> throughput workloads, and especially those generating high seek rates to
> boot.  The workload described is many parallel streaming writes of .25
> MB/s each.  If we use 96 streams, that's "only" 24 MB/s aggregate.  But
> as each of the 16 drives will likely be hitting its seek ceiling of
> ~150/s using XFS on striped RAID, the aggregate throughput of the 15
> RAID5 spindles will probably be less than 10 MB/s.
>
> Using the linear array with XFS instead of RAID5 will eliminate much of
> the head seeking, increasing throughput.  The increase may not be a
> huge, but it will be enough to handle many more parallel write streams
> than RAID5 before the drives hit their seek ceiling.
>
> --
> Stan
>

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/5/2012 1:48 AM, daobang wang wrote:
> Hi stan,
> 
>      I duplicated the input/output error issue, about the detail
> operations and logs, please see the attachments, Is there any way to
> fix this? thanks!
> 
> Best Regards,
> Daobang Wang.


These fiilesystem issues have nothing to do with linux-raid.  I'm
copying the XFS mailing list which is where this discussion should be
taking place from this point forward.  Please reply-to-all, and paste
the output you previously attached, but inline this time.

Also, since the XFS folks are unfamiliar with what you're doing up to
this point, please provide a basic description of your hardware/storage
setup, kernel version, mdraid configuration, xfs_info output as well as
your fstab XFS mount options, and a description of your workload.

My best guess at this point as to the du and ls errors is that your
application is not behaving properly, or you're still running with XFS
barriers disabled, which, I say _loudly_ for the 2nd time, you should
NOT do in the absence of BBWC, which you stated you do not have.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/5/2012 1:48 AM, daobang wang wrote:
> Hi stan,
> 
>      I duplicated the input/output error issue, about the detail
> operations and logs, please see the attachments, Is there any way to
> fix this? thanks!
> 
> Best Regards,
> Daobang Wang.


These fiilesystem issues have nothing to do with linux-raid.  I'm
copying the XFS mailing list which is where this discussion should be
taking place from this point forward.  Please reply-to-all, and paste
the output you previously attached, but inline this time.

Also, since the XFS folks are unfamiliar with what you're doing up to
this point, please provide a basic description of your hardware/storage
setup, kernel version, mdraid configuration, xfs_info output as well as
your fstab XFS mount options, and a description of your workload.

My best guess at this point as to the du and ls errors is that your
application is not behaving properly, or you're still running with XFS
barriers disabled, which, I say _loudly_ for the 2nd time, you should
NOT do in the absence of BBWC, which you stated you do not have.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Hi All,

    I have found the solution, i updated the xfsprogs from 2.10.1 to
3.1.5, and could repair it, Thanks.

Best Wishes,
Daobang Wang.

On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/5/2012 1:48 AM, daobang wang wrote:
>> Hi stan,
>>
>>      I duplicated the input/output error issue, about the detail
>> operations and logs, please see the attachments, Is there any way to
>> fix this? thanks!
>>
>> Best Regards,
>> Daobang Wang.
>
>
> These fiilesystem issues have nothing to do with linux-raid.  I'm
> copying the XFS mailing list which is where this discussion should be
> taking place from this point forward.  Please reply-to-all, and paste
> the output you previously attached, but inline this time.
>
> Also, since the XFS folks are unfamiliar with what you're doing up to
> this point, please provide a basic description of your hardware/storage
> setup, kernel version, mdraid configuration, xfs_info output as well as
> your fstab XFS mount options, and a description of your workload.
>
> My best guess at this point as to the du and ls errors is that your
> application is not behaving properly, or you're still running with XFS
> barriers disabled, which, I say _loudly_ for the 2nd time, you should
> NOT do in the absence of BBWC, which you stated you do not have.
>
> --
> Stan
>

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Hi All,

    I have found the solution, i updated the xfsprogs from 2.10.1 to
3.1.5, and could repair it, Thanks.

Best Wishes,
Daobang Wang.

On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/5/2012 1:48 AM, daobang wang wrote:
>> Hi stan,
>>
>>      I duplicated the input/output error issue, about the detail
>> operations and logs, please see the attachments, Is there any way to
>> fix this? thanks!
>>
>> Best Regards,
>> Daobang Wang.
>
>
> These fiilesystem issues have nothing to do with linux-raid.  I'm
> copying the XFS mailing list which is where this discussion should be
> taking place from this point forward.  Please reply-to-all, and paste
> the output you previously attached, but inline this time.
>
> Also, since the XFS folks are unfamiliar with what you're doing up to
> this point, please provide a basic description of your hardware/storage
> setup, kernel version, mdraid configuration, xfs_info output as well as
> your fstab XFS mount options, and a description of your workload.
>
> My best guess at this point as to the du and ls errors is that your
> application is not behaving properly, or you're still running with XFS
> barriers disabled, which, I say _loudly_ for the 2nd time, you should
> NOT do in the absence of BBWC, which you stated you do not have.
>
> --
> Stan
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

There is another issue, i updated xfsprog from 2.10.1 to 3.1.5, and
found i could not make the xfs filesystem when the logical volume size
large than 8TB with command mkfs.xfs -f -i size=512
/dev/vg+vg00+20120406101850/lv+nxx+lv0000, this command seems hang, it
did not return for a long time, is there any parameter i should
adjust?

Thank you very much

Best Regards,
Daobang Wang.

On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
> Hi All,
>
>     I have found the solution, i updated the xfsprogs from 2.10.1 to
> 3.1.5, and could repair it, Thanks.
>
> Best Wishes,
> Daobang Wang.
>
> On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>> On 4/5/2012 1:48 AM, daobang wang wrote:
>>> Hi stan,
>>>
>>>      I duplicated the input/output error issue, about the detail
>>> operations and logs, please see the attachments, Is there any way to
>>> fix this? thanks!
>>>
>>> Best Regards,
>>> Daobang Wang.
>>
>>
>> These fiilesystem issues have nothing to do with linux-raid.  I'm
>> copying the XFS mailing list which is where this discussion should be
>> taking place from this point forward.  Please reply-to-all, and paste
>> the output you previously attached, but inline this time.
>>
>> Also, since the XFS folks are unfamiliar with what you're doing up to
>> this point, please provide a basic description of your hardware/storage
>> setup, kernel version, mdraid configuration, xfs_info output as well as
>> your fstab XFS mount options, and a description of your workload.
>>
>> My best guess at this point as to the du and ls errors is that your
>> application is not behaving properly, or you're still running with XFS
>> barriers disabled, which, I say _loudly_ for the 2nd time, you should
>> NOT do in the absence of BBWC, which you stated you do not have.
>>
>> --
>> Stan
>>
>

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

There is another issue, i updated xfsprog from 2.10.1 to 3.1.5, and
found i could not make the xfs filesystem when the logical volume size
large than 8TB with command mkfs.xfs -f -i size=512
/dev/vg+vg00+20120406101850/lv+nxx+lv0000, this command seems hang, it
did not return for a long time, is there any parameter i should
adjust?

Thank you very much

Best Regards,
Daobang Wang.

On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
> Hi All,
>
>     I have found the solution, i updated the xfsprogs from 2.10.1 to
> 3.1.5, and could repair it, Thanks.
>
> Best Wishes,
> Daobang Wang.
>
> On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>> On 4/5/2012 1:48 AM, daobang wang wrote:
>>> Hi stan,
>>>
>>>      I duplicated the input/output error issue, about the detail
>>> operations and logs, please see the attachments, Is there any way to
>>> fix this? thanks!
>>>
>>> Best Regards,
>>> Daobang Wang.
>>
>>
>> These fiilesystem issues have nothing to do with linux-raid.  I'm
>> copying the XFS mailing list which is where this discussion should be
>> taking place from this point forward.  Please reply-to-all, and paste
>> the output you previously attached, but inline this time.
>>
>> Also, since the XFS folks are unfamiliar with what you're doing up to
>> this point, please provide a basic description of your hardware/storage
>> setup, kernel version, mdraid configuration, xfs_info output as well as
>> your fstab XFS mount options, and a description of your workload.
>>
>> My best guess at this point as to the du and ls errors is that your
>> application is not behaving properly, or you're still running with XFS
>> barriers disabled, which, I say _loudly_ for the 2nd time, you should
>> NOT do in the absence of BBWC, which you stated you do not have.
>>
>> --
>> Stan
>>
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: RAID5 created by 8 disks works with xfs

[Jack Wang]

using bigger inode size or inode64 does help?

So your environment is NVR software running in Linux ,100+ D1 streamer
directly write to filesystem on top of 16 SATA disks (with raid5 and
vg)?

2012/4/6 daobang wang <wangdb1981@gmail.com>:
> There is another issue, i updated xfsprog from 2.10.1 to 3.1.5, and
> found i could not make the xfs filesystem when the logical volume size
> large than 8TB with command mkfs.xfs -f -i size=512
> /dev/vg+vg00+20120406101850/lv+nxx+lv0000, this command seems hang, it
> did not return for a long time, is there any parameter i should
> adjust?
>
> Thank you very much
>
> Best Regards,
> Daobang Wang.
>
> On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
>> Hi All,
>>
>>     I have found the solution, i updated the xfsprogs from 2.10.1 to
>> 3.1.5, and could repair it, Thanks.
>>
>> Best Wishes,
>> Daobang Wang.
>>
>> On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>> On 4/5/2012 1:48 AM, daobang wang wrote:
>>>> Hi stan,
>>>>
>>>>      I duplicated the input/output error issue, about the detail
>>>> operations and logs, please see the attachments, Is there any way to
>>>> fix this? thanks!
>>>>
>>>> Best Regards,
>>>> Daobang Wang.
>>>
>>>
>>> These fiilesystem issues have nothing to do with linux-raid.  I'm
>>> copying the XFS mailing list which is where this discussion should be
>>> taking place from this point forward.  Please reply-to-all, and paste
>>> the output you previously attached, but inline this time.
>>>
>>> Also, since the XFS folks are unfamiliar with what you're doing up to
>>> this point, please provide a basic description of your hardware/storage
>>> setup, kernel version, mdraid configuration, xfs_info output as well as
>>> your fstab XFS mount options, and a description of your workload.
>>>
>>> My best guess at this point as to the du and ls errors is that your
>>> application is not behaving properly, or you're still running with XFS
>>> barriers disabled, which, I say _loudly_ for the 2nd time, you should
>>> NOT do in the absence of BBWC, which you stated you do not have.
>>>
>>> --
>>> Stan
>>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 created by 8 disks works with xfs

[Jack Wang]

using bigger inode size or inode64 does help?

So your environment is NVR software running in Linux ,100+ D1 streamer
directly write to filesystem on top of 16 SATA disks (with raid5 and
vg)?

2012/4/6 daobang wang <wangdb1981@gmail.com>:
> There is another issue, i updated xfsprog from 2.10.1 to 3.1.5, and
> found i could not make the xfs filesystem when the logical volume size
> large than 8TB with command mkfs.xfs -f -i size=512
> /dev/vg+vg00+20120406101850/lv+nxx+lv0000, this command seems hang, it
> did not return for a long time, is there any parameter i should
> adjust?
>
> Thank you very much
>
> Best Regards,
> Daobang Wang.
>
> On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
>> Hi All,
>>
>>     I have found the solution, i updated the xfsprogs from 2.10.1 to
>> 3.1.5, and could repair it, Thanks.
>>
>> Best Wishes,
>> Daobang Wang.
>>
>> On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>> On 4/5/2012 1:48 AM, daobang wang wrote:
>>>> Hi stan,
>>>>
>>>>      I duplicated the input/output error issue, about the detail
>>>> operations and logs, please see the attachments, Is there any way to
>>>> fix this? thanks!
>>>>
>>>> Best Regards,
>>>> Daobang Wang.
>>>
>>>
>>> These fiilesystem issues have nothing to do with linux-raid.  I'm
>>> copying the XFS mailing list which is where this discussion should be
>>> taking place from this point forward.  Please reply-to-all, and paste
>>> the output you previously attached, but inline this time.
>>>
>>> Also, since the XFS folks are unfamiliar with what you're doing up to
>>> this point, please provide a basic description of your hardware/storage
>>> setup, kernel version, mdraid configuration, xfs_info output as well as
>>> your fstab XFS mount options, and a description of your workload.
>>>
>>> My best guess at this point as to the du and ls errors is that your
>>> application is not behaving properly, or you're still running with XFS
>>> barriers disabled, which, I say _loudly_ for the 2nd time, you should
>>> NOT do in the absence of BBWC, which you stated you do not have.
>>>
>>> --
>>> Stan
>>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Hi Jacky,

     Yes, the environment is like your description, i will try with
your suggestion. thanks a lot.

On 4/6/12, Jack Wang <jack.wang.usish@gmail.com> wrote:
> using bigger inode size or inode64 does help?
>
> So your environment is NVR software running in Linux ,100+ D1 streamer
> directly write to filesystem on top of 16 SATA disks (with raid5 and
> vg)?
>
> 2012/4/6 daobang wang <wangdb1981@gmail.com>:
>> There is another issue, i updated xfsprog from 2.10.1 to 3.1.5, and
>> found i could not make the xfs filesystem when the logical volume size
>> large than 8TB with command mkfs.xfs -f -i size=512
>> /dev/vg+vg00+20120406101850/lv+nxx+lv0000, this command seems hang, it
>> did not return for a long time, is there any parameter i should
>> adjust?
>>
>> Thank you very much
>>
>> Best Regards,
>> Daobang Wang.
>>
>> On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
>>> Hi All,
>>>
>>>     I have found the solution, i updated the xfsprogs from 2.10.1 to
>>> 3.1.5, and could repair it, Thanks.
>>>
>>> Best Wishes,
>>> Daobang Wang.
>>>
>>> On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>>> On 4/5/2012 1:48 AM, daobang wang wrote:
>>>>> Hi stan,
>>>>>
>>>>>      I duplicated the input/output error issue, about the detail
>>>>> operations and logs, please see the attachments, Is there any way to
>>>>> fix this? thanks!
>>>>>
>>>>> Best Regards,
>>>>> Daobang Wang.
>>>>
>>>>
>>>> These fiilesystem issues have nothing to do with linux-raid.  I'm
>>>> copying the XFS mailing list which is where this discussion should be
>>>> taking place from this point forward.  Please reply-to-all, and paste
>>>> the output you previously attached, but inline this time.
>>>>
>>>> Also, since the XFS folks are unfamiliar with what you're doing up to
>>>> this point, please provide a basic description of your hardware/storage
>>>> setup, kernel version, mdraid configuration, xfs_info output as well as
>>>> your fstab XFS mount options, and a description of your workload.
>>>>
>>>> My best guess at this point as to the du and ls errors is that your
>>>> application is not behaving properly, or you're still running with XFS
>>>> barriers disabled, which, I say _loudly_ for the 2nd time, you should
>>>> NOT do in the absence of BBWC, which you stated you do not have.
>>>>
>>>> --
>>>> Stan
>>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Hi Jacky,

     Yes, the environment is like your description, i will try with
your suggestion. thanks a lot.

On 4/6/12, Jack Wang <jack.wang.usish@gmail.com> wrote:
> using bigger inode size or inode64 does help?
>
> So your environment is NVR software running in Linux ,100+ D1 streamer
> directly write to filesystem on top of 16 SATA disks (with raid5 and
> vg)?
>
> 2012/4/6 daobang wang <wangdb1981@gmail.com>:
>> There is another issue, i updated xfsprog from 2.10.1 to 3.1.5, and
>> found i could not make the xfs filesystem when the logical volume size
>> large than 8TB with command mkfs.xfs -f -i size=512
>> /dev/vg+vg00+20120406101850/lv+nxx+lv0000, this command seems hang, it
>> did not return for a long time, is there any parameter i should
>> adjust?
>>
>> Thank you very much
>>
>> Best Regards,
>> Daobang Wang.
>>
>> On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
>>> Hi All,
>>>
>>>     I have found the solution, i updated the xfsprogs from 2.10.1 to
>>> 3.1.5, and could repair it, Thanks.
>>>
>>> Best Wishes,
>>> Daobang Wang.
>>>
>>> On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>>> On 4/5/2012 1:48 AM, daobang wang wrote:
>>>>> Hi stan,
>>>>>
>>>>>      I duplicated the input/output error issue, about the detail
>>>>> operations and logs, please see the attachments, Is there any way to
>>>>> fix this? thanks!
>>>>>
>>>>> Best Regards,
>>>>> Daobang Wang.
>>>>
>>>>
>>>> These fiilesystem issues have nothing to do with linux-raid.  I'm
>>>> copying the XFS mailing list which is where this discussion should be
>>>> taking place from this point forward.  Please reply-to-all, and paste
>>>> the output you previously attached, but inline this time.
>>>>
>>>> Also, since the XFS folks are unfamiliar with what you're doing up to
>>>> this point, please provide a basic description of your hardware/storage
>>>> setup, kernel version, mdraid configuration, xfs_info output as well as
>>>> your fstab XFS mount options, and a description of your workload.
>>>>
>>>> My best guess at this point as to the du and ls errors is that your
>>>> application is not behaving properly, or you're still running with XFS
>>>> barriers disabled, which, I say _loudly_ for the 2nd time, you should
>>>> NOT do in the absence of BBWC, which you stated you do not have.
>>>>
>>>> --
>>>> Stan
>>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

I did more tests, and found the mkfs.xfs did not hang.

I tried to make the xfs on 7TB logical volume, both 2.10.1 and 3.1.5
mkfs.xfs -f dev will work well, less than one minute.

But when i tried to make xfs on 8TB logical volume, 2.10.1 works well,
less than 1 minute, but 3.1.5 needs 5 minutes.

On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
> Hi Jacky,
>
>      Yes, the environment is like your description, i will try with
> your suggestion. thanks a lot.
>
> On 4/6/12, Jack Wang <jack.wang.usish@gmail.com> wrote:
>> using bigger inode size or inode64 does help?
>>
>> So your environment is NVR software running in Linux ,100+ D1 streamer
>> directly write to filesystem on top of 16 SATA disks (with raid5 and
>> vg)?
>>
>> 2012/4/6 daobang wang <wangdb1981@gmail.com>:
>>> There is another issue, i updated xfsprog from 2.10.1 to 3.1.5, and
>>> found i could not make the xfs filesystem when the logical volume size
>>> large than 8TB with command mkfs.xfs -f -i size=512
>>> /dev/vg+vg00+20120406101850/lv+nxx+lv0000, this command seems hang, it
>>> did not return for a long time, is there any parameter i should
>>> adjust?
>>>
>>> Thank you very much
>>>
>>> Best Regards,
>>> Daobang Wang.
>>>
>>> On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
>>>> Hi All,
>>>>
>>>>     I have found the solution, i updated the xfsprogs from 2.10.1 to
>>>> 3.1.5, and could repair it, Thanks.
>>>>
>>>> Best Wishes,
>>>> Daobang Wang.
>>>>
>>>> On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>>>> On 4/5/2012 1:48 AM, daobang wang wrote:
>>>>>> Hi stan,
>>>>>>
>>>>>>      I duplicated the input/output error issue, about the detail
>>>>>> operations and logs, please see the attachments, Is there any way to
>>>>>> fix this? thanks!
>>>>>>
>>>>>> Best Regards,
>>>>>> Daobang Wang.
>>>>>
>>>>>
>>>>> These fiilesystem issues have nothing to do with linux-raid.  I'm
>>>>> copying the XFS mailing list which is where this discussion should be
>>>>> taking place from this point forward.  Please reply-to-all, and paste
>>>>> the output you previously attached, but inline this time.
>>>>>
>>>>> Also, since the XFS folks are unfamiliar with what you're doing up to
>>>>> this point, please provide a basic description of your
>>>>> hardware/storage
>>>>> setup, kernel version, mdraid configuration, xfs_info output as well
>>>>> as
>>>>> your fstab XFS mount options, and a description of your workload.
>>>>>
>>>>> My best guess at this point as to the du and ls errors is that your
>>>>> application is not behaving properly, or you're still running with XFS
>>>>> barriers disabled, which, I say _loudly_ for the 2nd time, you should
>>>>> NOT do in the absence of BBWC, which you stated you do not have.
>>>>>
>>>>> --
>>>>> Stan
>>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

I did more tests, and found the mkfs.xfs did not hang.

I tried to make the xfs on 7TB logical volume, both 2.10.1 and 3.1.5
mkfs.xfs -f dev will work well, less than one minute.

But when i tried to make xfs on 8TB logical volume, 2.10.1 works well,
less than 1 minute, but 3.1.5 needs 5 minutes.

On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
> Hi Jacky,
>
>      Yes, the environment is like your description, i will try with
> your suggestion. thanks a lot.
>
> On 4/6/12, Jack Wang <jack.wang.usish@gmail.com> wrote:
>> using bigger inode size or inode64 does help?
>>
>> So your environment is NVR software running in Linux ,100+ D1 streamer
>> directly write to filesystem on top of 16 SATA disks (with raid5 and
>> vg)?
>>
>> 2012/4/6 daobang wang <wangdb1981@gmail.com>:
>>> There is another issue, i updated xfsprog from 2.10.1 to 3.1.5, and
>>> found i could not make the xfs filesystem when the logical volume size
>>> large than 8TB with command mkfs.xfs -f -i size=512
>>> /dev/vg+vg00+20120406101850/lv+nxx+lv0000, this command seems hang, it
>>> did not return for a long time, is there any parameter i should
>>> adjust?
>>>
>>> Thank you very much
>>>
>>> Best Regards,
>>> Daobang Wang.
>>>
>>> On 4/6/12, daobang wang <wangdb1981@gmail.com> wrote:
>>>> Hi All,
>>>>
>>>>     I have found the solution, i updated the xfsprogs from 2.10.1 to
>>>> 3.1.5, and could repair it, Thanks.
>>>>
>>>> Best Wishes,
>>>> Daobang Wang.
>>>>
>>>> On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>>>> On 4/5/2012 1:48 AM, daobang wang wrote:
>>>>>> Hi stan,
>>>>>>
>>>>>>      I duplicated the input/output error issue, about the detail
>>>>>> operations and logs, please see the attachments, Is there any way to
>>>>>> fix this? thanks!
>>>>>>
>>>>>> Best Regards,
>>>>>> Daobang Wang.
>>>>>
>>>>>
>>>>> These fiilesystem issues have nothing to do with linux-raid.  I'm
>>>>> copying the XFS mailing list which is where this discussion should be
>>>>> taking place from this point forward.  Please reply-to-all, and paste
>>>>> the output you previously attached, but inline this time.
>>>>>
>>>>> Also, since the XFS folks are unfamiliar with what you're doing up to
>>>>> this point, please provide a basic description of your
>>>>> hardware/storage
>>>>> setup, kernel version, mdraid configuration, xfs_info output as well
>>>>> as
>>>>> your fstab XFS mount options, and a description of your workload.
>>>>>
>>>>> My best guess at this point as to the du and ls errors is that your
>>>>> application is not behaving properly, or you're still running with XFS
>>>>> barriers disabled, which, I say _loudly_ for the 2nd time, you should
>>>>> NOT do in the absence of BBWC, which you stated you do not have.
>>>>>
>>>>> --
>>>>> Stan
>>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/6/2012 1:49 AM, daobang wang wrote:
> I did more tests, and found the mkfs.xfs did not hang.
> 
> I tried to make the xfs on 7TB logical volume, both 2.10.1 and 3.1.5
> mkfs.xfs -f dev will work well, less than one minute.
> 
> But when i tried to make xfs on 8TB logical volume, 2.10.1 works well,
> less than 1 minute, but 3.1.5 needs 5 minutes.

Once you provide the information I requested I'd be glad to help you
further.  Without that information I'm wasting my time.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/6/2012 1:49 AM, daobang wang wrote:
> I did more tests, and found the mkfs.xfs did not hang.
> 
> I tried to make the xfs on 7TB logical volume, both 2.10.1 and 3.1.5
> mkfs.xfs -f dev will work well, less than one minute.
> 
> But when i tried to make xfs on 8TB logical volume, 2.10.1 works well,
> less than 1 minute, but 3.1.5 needs 5 minutes.

Once you provide the information I requested I'd be glad to help you
further.  Without that information I'm wasting my time.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Hi stan,

    Thank you for your reply, the envrionment is the same that i
mentioned before,  the user allows the cache data to miss, so i did
not disable the barriers, they just demands that the filesystem should
be repaired after system restarted. We did not deal with the fstab
now, it is the next step work.

    And about the mkfs.xfs slow issue, we found the logsize is too
large, it caused making the xfs filesystem slowly.

    Thanks again.

Have a good day.
Daobang Wang.

On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/6/2012 1:49 AM, daobang wang wrote:
>> I did more tests, and found the mkfs.xfs did not hang.
>>
>> I tried to make the xfs on 7TB logical volume, both 2.10.1 and 3.1.5
>> mkfs.xfs -f dev will work well, less than one minute.
>>
>> But when i tried to make xfs on 8TB logical volume, 2.10.1 works well,
>> less than 1 minute, but 3.1.5 needs 5 minutes.
>
> Once you provide the information I requested I'd be glad to help you
> further.  Without that information I'm wasting my time.
>
> --
> Stan
>

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Hi stan,

    Thank you for your reply, the envrionment is the same that i
mentioned before,  the user allows the cache data to miss, so i did
not disable the barriers, they just demands that the filesystem should
be repaired after system restarted. We did not deal with the fstab
now, it is the next step work.

    And about the mkfs.xfs slow issue, we found the logsize is too
large, it caused making the xfs filesystem slowly.

    Thanks again.

Have a good day.
Daobang Wang.

On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/6/2012 1:49 AM, daobang wang wrote:
>> I did more tests, and found the mkfs.xfs did not hang.
>>
>> I tried to make the xfs on 7TB logical volume, both 2.10.1 and 3.1.5
>> mkfs.xfs -f dev will work well, less than one minute.
>>
>> But when i tried to make xfs on 8TB logical volume, 2.10.1 works well,
>> less than 1 minute, but 3.1.5 needs 5 minutes.
>
> Once you provide the information I requested I'd be glad to help you
> further.  Without that information I'm wasting my time.
>
> --
> Stan
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/6/2012 3:45 AM, daobang wang wrote:
> Hi stan,
> 
>     Thank you for your reply, the envrionment is the same that i
> mentioned before,  [...]

There's seems to be some kind of communication failure here.  I'll make
this really simple.  Please fill in the blanks.

$ uname -a

$ xfs_info [device]

$ cat /etc/fstab|grep xfs

$ mdadm --detail /dev/mdX


Thanks.

-- 
Stan

Re: RAID5 created by 8 disks works with xfs

[Stan Hoeppner]

On 4/6/2012 3:45 AM, daobang wang wrote:
> Hi stan,
> 
>     Thank you for your reply, the envrionment is the same that i
> mentioned before,  [...]

There's seems to be some kind of communication failure here.  I'll make
this really simple.  Please fill in the blanks.

$ uname -a

$ xfs_info [device]

$ cat /etc/fstab|grep xfs

$ mdadm --detail /dev/mdX


Thanks.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Hi stan,

    So sorry to reply this so late, the original envrionment is not
exist now, i build a same one, please see the detail output.

1. uname -a
Linux nsspioneer 2.6.36.4-v64 #1 SMP Fri Apr 13 10:16:12 CST 2012
x86_64 x86_64 x86_64 GNU/Linux

2.xfs_info /dev/vg+vg00+20120418101725/lv+nxx+lv0000
meta-data=/dev/vg+vg00+20120418101725/lv+nxx+lv0000 isize=256
agcount=13, agsize=268435440 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=3418374144, imaxpct=5
         =                       sunit=16     swidth=128 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

3. the command "cat /etc/fstab | grep xfs" has no output, we did not
save it in fstab

4. mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Wed Apr 18 10:02:26 2012
     Raid Level : raid5
     Array Size : 13673525376 (13040.09 GiB 14001.69 GB)
  Used Dev Size : 1953360768 (1862.87 GiB 2000.24 GB)
   Raid Devices : 8
  Total Devices : 8
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Apr 18 10:12:22 2012
          State : online
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : md+LD00+120418100226
           UUID : bbddf00e:779ca156:bb4f7d10:285e4fcf
         Events : 195

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       8       48        2      active sync   /dev/sdd
       3       8       64        3      active sync   /dev/sde
       4       8       80        4      active sync   /dev/sdf
       5       8       96        5      active sync   /dev/sdg
       6       8      112        6      active sync   /dev/sdh
       8       8      128        7      active sync   /dev/sdi

On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/6/2012 3:45 AM, daobang wang wrote:
>> Hi stan,
>>
>>     Thank you for your reply, the envrionment is the same that i
>> mentioned before,  [...]
>
> There's seems to be some kind of communication failure here.  I'll make
> this really simple.  Please fill in the blanks.
>
> $ uname -a
>
> $ xfs_info [device]
>
> $ cat /etc/fstab|grep xfs
>
> $ mdadm --detail /dev/mdX
>
>
> Thanks.
>
> --
> Stan
>
>

Re: RAID5 created by 8 disks works with xfs

[daobang wang]

Hi stan,

    So sorry to reply this so late, the original envrionment is not
exist now, i build a same one, please see the detail output.

1. uname -a
Linux nsspioneer 2.6.36.4-v64 #1 SMP Fri Apr 13 10:16:12 CST 2012
x86_64 x86_64 x86_64 GNU/Linux

2.xfs_info /dev/vg+vg00+20120418101725/lv+nxx+lv0000
meta-data=/dev/vg+vg00+20120418101725/lv+nxx+lv0000 isize=256
agcount=13, agsize=268435440 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=3418374144, imaxpct=5
         =                       sunit=16     swidth=128 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

3. the command "cat /etc/fstab | grep xfs" has no output, we did not
save it in fstab

4. mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Wed Apr 18 10:02:26 2012
     Raid Level : raid5
     Array Size : 13673525376 (13040.09 GiB 14001.69 GB)
  Used Dev Size : 1953360768 (1862.87 GiB 2000.24 GB)
   Raid Devices : 8
  Total Devices : 8
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Apr 18 10:12:22 2012
          State : online
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : md+LD00+120418100226
           UUID : bbddf00e:779ca156:bb4f7d10:285e4fcf
         Events : 195

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       8       48        2      active sync   /dev/sdd
       3       8       64        3      active sync   /dev/sde
       4       8       80        4      active sync   /dev/sdf
       5       8       96        5      active sync   /dev/sdg
       6       8      112        6      active sync   /dev/sdh
       8       8      128        7      active sync   /dev/sdi

On 4/6/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/6/2012 3:45 AM, daobang wang wrote:
>> Hi stan,
>>
>>     Thank you for your reply, the envrionment is the same that i
>> mentioned before,  [...]
>
> There's seems to be some kind of communication failure here.  I'll make
> this really simple.  Please fill in the blanks.
>
> $ uname -a
>
> $ xfs_info [device]
>
> $ cat /etc/fstab|grep xfs
>
> $ mdadm --detail /dev/mdX
>
>
> Thanks.
>
> --
> Stan
>
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs