Array read 3 time slower to read after data fully loaded

Array read 3 time slower to read after data fully loaded

[Zoltan Szecsei]

Hi,
Can anyone please explain this to me?

For the last >30 hours I have been loading some 2.5TB of 200-400MB tiff 
images, from another PC on our internal network, onto this new array.
Yesterday, during this load process, I copied 10 of these files (about 
2.5GB) from the array onto the /home directory of the same system.

It took 33 seconds, equating to some 75MB/s.

This morning I came in to find that the load process had finished, so I 
deleted the 10 test files from /home, and redid the identical copy test 
(using bash-history, so: identical).

It took 123 seconds, equating to some 20MB/s.

Que? (as in Manuel from Faulty Towers :-)  )


Thanks & regards,
Zoltan


root@gs0:/home/geograph# uname -a
Linux gs0 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 19:52:42 UTC 2010 
x86_64 GNU/Linux
root@gs0:/home/geograph# mdadm -V
mdadm - v3.1.4 - 31st August 2010
root@gs0:/home/geograph# lvm version
   LVM version:     2.02.54(1) (2009-10-26)
   Library version: 1.02.39 (2009-10-26)
   Driver version:  4.15.0
root@gs0:/home/geograph# df  #ext4
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdi7            426559760   9494844 395396872   3% /home
/dev/mapper/geo_vg0-geo_lvm0
                      11536635672 2530042680 8889387468  23% /mnt/geo_lvm0
root@gs0:/home/geograph#


-- 

===========================================
Zoltan Szecsei PrGISc [PGP0031]
Geograph (Pty) Ltd.
P.O. Box 7, Muizenberg 7950, South Africa.

65 Main Road, Muizenberg 7945
Western Cape, South Africa.

34° 6'16.35"S 18°28'5.62"E

Tel: +27-21-7884897  Mobile: +27-83-6004028
Fax: +27-86-6115323     www.geograph.co.za
===========================================



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1153 / Virus Database: 424/3262 - Release Date: 11/17/10

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Array read 3 time slower to read after data fully loaded

[Michal Soltys]

On 10-11-18 06:30, Zoltan Szecsei wrote:
> Hi,
> Can anyone please explain this to me?
>
> For the last >30 hours I have been loading some 2.5TB of 200-400MB tiff
> images, from another PC on our internal network, onto this new array.
> Yesterday, during this load process, I copied 10 of these files (about
> 2.5GB) from the array onto the /home directory of the same system.
>
> It took 33 seconds, equating to some 75MB/s.
>
> This morning I came in to find that the load process had finished, so I
> deleted the 10 test files from /home, and redid the identical copy test
> (using bash-history, so: identical).
>
> It took 123 seconds, equating to some 20MB/s.
>
> Que? (as in Manuel from Faulty Towers :-) )
>

Is the situation with reading into /dev/null instead of /home the same ?

As this is reading issue, one thing that came to my mind:

- make sure the kernel is _not_ compiled with  CONFIG_MULTICORE_RAID456. 
It's afaik still experimental and can cause severe slowdowns

RE: Array read 3 time slower to read after data fully loaded

[David Lethe]

Throughput is a function of block size & chunk size.  For operations such as copying large files, then larger block/chunk sizes mean
higher throughput .. at the COST of I/Os per second.

There is no free lunch. Before doing any tuning, ask yourself what the normal mix of random, sequential, read, write, large and small
block I/O is.  If you are constantly moving large files around, then by all means redo the RAID and file system setup, but if this exercise is an exception, then you may be better off leaving things as they are.
________________________________________
From: linux-raid-owner@vger.kernel.org [linux-raid-owner@vger.kernel.org] On Behalf Of Michal Soltys [soltys@ziu.info]
Sent: Thursday, November 18, 2010 2:43 PM
To: Zoltan Szecsei
Cc: linux-raid@vger.kernel.org
Subject: Re: Array read 3 time slower to read after data fully loaded

On 10-11-18 06:30, Zoltan Szecsei wrote:
> Hi,
> Can anyone please explain this to me?
>
> For the last >30 hours I have been loading some 2.5TB of 200-400MB tiff
> images, from another PC on our internal network, onto this new array.
> Yesterday, during this load process, I copied 10 of these files (about
> 2.5GB) from the array onto the /home directory of the same system.
>
> It took 33 seconds, equating to some 75MB/s.
>
> This morning I came in to find that the load process had finished, so I
> deleted the 10 test files from /home, and redid the identical copy test
> (using bash-history, so: identical).
>
> It took 123 seconds, equating to some 20MB/s.
>
> Que? (as in Manuel from Faulty Towers :-) )
>

Is the situation with reading into /dev/null instead of /home the same ?

As this is reading issue, one thing that came to my mind:

- make sure the kernel is _not_ compiled with  CONFIG_MULTICORE_RAID456.
It's afaik still experimental and can cause severe slowdowns

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Array read 3 time slower to read after data fully loaded

[Neil Brown]

On Thu, 18 Nov 2010 07:30:14 +0200
Zoltan Szecsei <zoltans@geograph.co.za> wrote:

> Hi,
> Can anyone please explain this to me?
> 
> For the last >30 hours I have been loading some 2.5TB of 200-400MB tiff 
> images, from another PC on our internal network, onto this new array.
> Yesterday, during this load process, I copied 10 of these files (about 
> 2.5GB) from the array onto the /home directory of the same system.
> 
> It took 33 seconds, equating to some 75MB/s.
> 
> This morning I came in to find that the load process had finished, so I 
> deleted the 10 test files from /home, and redid the identical copy test 
> (using bash-history, so: identical).
> 
> It took 123 seconds, equating to some 20MB/s.
> 
> Que? (as in Manuel from Faulty Towers :-)  )

Maybe the first time some of the files were in RAM... how much RAM do you
have, and how recently that the files been loaded before your copied them?

NeilBrown - who has never been to Barcelona.

> 
> 
> Thanks & regards,
> Zoltan
> 
> 
> root@gs0:/home/geograph# uname -a
> Linux gs0 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 19:52:42 UTC 2010 
> x86_64 GNU/Linux
> root@gs0:/home/geograph# mdadm -V
> mdadm - v3.1.4 - 31st August 2010
> root@gs0:/home/geograph# lvm version
>    LVM version:     2.02.54(1) (2009-10-26)
>    Library version: 1.02.39 (2009-10-26)
>    Driver version:  4.15.0
> root@gs0:/home/geograph# df  #ext4
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sdi7            426559760   9494844 395396872   3% /home
> /dev/mapper/geo_vg0-geo_lvm0
>                       11536635672 2530042680 8889387468  23% /mnt/geo_lvm0
> root@gs0:/home/geograph#
> 
>