RE: raid0 and iscsi

RE: raid0 and iscsi

[jshankar]

Hello Neil/Rob,

I am using IP based network(2 Intel Ethernet card Pro 1000/T).
CPUinfo is as below
:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 8
model name      : Pentium III (Coppermine)
stepping        : 1
cpu MHz         : 526.186
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov
pat pse36 mmx fxsr sse
bogomips        : 1044.22



Can u please explain me  about the below fuctionality:
2>If my chunk size is 8 byte. Does that mean it will write 8 byte into device 
0
and then into device 1. Is the write request to the disk in synchronous or
asynchronous mode.

3> Is wite_disk_sb in md.c responsible for writing into disk??. Is fsync_dev
responsible for synhronous write ??. If so can i change to asynchronous 
write.I
will really appreciate if somebody can tell me what all routines I need to go
through to figure out the functionality of raid0 behaviourand how it write to 
disk.


Thanks
Jay






===== Original Message From rob <rob@fantinibakery.com> =====
>Hello,
> What kind of network are you uasing for the data transfer?
>
> The rest of post does not answer your specific questions, but is to
>show the transfer speeds on our network to backup servers using hardware
>and software raid.
>
> Both systems use Mandrake 9.1.  network is a fast-lan. speed 100.
>
>Bkup1 is scsi with Mylex hardware raid, athlon 1800 cpu.  array has 4
>9-gb lvd disks.
>
>Bkup2 is a new test server using software raid.  dual 750 intel xeon,
>4- 9-gb se scsi disks.
>
> I transfer data 30-40 times per day to each server.  500-mb hourly on
>the average. ususal tme is less than a min.
> The test uses rsync to transfer data.  rsync uses a lot of cpu so the
>speed difference may be more due to cpu than the difference of raid type.
>
> the command used: rsync -e ssh -a --stats  /bkup3/  fbc5:/bkup3/
>
>here is the result of sending approx 3-gb of data to each:
>
>-------------------------------------------------
> this is the output of an rsync transfer to the backup software-raid server:
>
>Mon Oct 13 07:36:41 EDT 2003
>rsync[21832] (sender) heap statistics:
>  arena:        7996056   (bytes from sbrk)
>  ordblks:           59   (chunks not in use)
>  smblks:             2
>  hblks:              1   (chunks from mmap)
>  hblkhd:        516096   (bytes from mmap)
>  usmblks:            0
>  fsmblks:           80
>  uordblks:     7821736   (bytes used)
>  fordblks:      174320   (bytes free)
>  keepcost:      128840   (bytes in releasable chunk)
>
>Number of files: 65699
>Number of files transferred: 45739
>Total file size: 2707740663 bytes
>Total transferred file size: 2707630760 bytes
>Literal data: 2707630760 bytes
>Matched data: 0 bytes
>File list size: 1,925,417
>Total bytes written: 2,711,680,565
>Total bytes read: 731844
>
>wrote 2711680565 bytes  read 731844 bytes  5,046,348.67 bytes/sec
>total size is 2707740663  speedup is 1.00
>Mon Oct 13 07:45:38 EDT 2003
>
>------------------------------------
>to the hardware raid server:
>Mon Oct 13 07:23:19 EDT 2003
>rsync[19963] (sender) heap statistics:
>  arena:        7996056   (bytes from sbrk)
>  ordblks:           59   (chunks not in use)
>  smblks:             2
>  hblks:              1   (chunks from mmap)
>  hblkhd:        516096   (bytes from mmap)
>  usmblks:            0
>  fsmblks:           80
>  uordblks:     7821736   (bytes used)
>  fordblks:      174320   (bytes free)
>  keepcost:      128840   (bytes in releasable chunk)
>
>Number of files: 65699
>Number of files transferred: 45739
>Total file size: 2707740663 bytes
>Total transferred file size: 2707630760 bytes
>Literal data: 2707630760 bytes
>Matched data: 0 bytes
>File list size: 1925417
>Total bytes written: 2,711,680,565
>Total bytes read: 731844
>
>wrote 2711680565 bytes  read 731844 bytes  8,384,582.41 bytes/sec
>total size is 2707740663  speedup is 1.00
>Mon Oct 13 07:28:42 EDT 2003
>
>
>----------------
>
>
>
>Jayshankar Nair wrote:
>
>>Hello,
>>
>>
>>I was testing the raid0 configuration for the iscsi device ( /dev/sdb and
>>/dev/sdc).
>>
>>For writing a 512MB in a LAN environment it was taking 24 hrs. i was going
>>through the source code of raid0.c and certain things doesn't made sense to 
me.
>>
>>1> what role does the Hash bit play.
>>2>If my chunk size is 8 byte. Does that mean it will write 8 byte into 
device 0
>>and then into device 1. Is the write request to the disk in synchronous or
>>asynchronous mode.
>>
>>
>>3> Is wite_disk_sb in md.c responsible for writing into disk??. Is fsync_dev
>>responsible for synhronous write ??. If so can i change to asynchronous 
write.I
>>will really appreciate if somebody can tell me what all routines I need to 
go
>>through to figure out the functionality of raid0 behaviour.
>>
>>Thanks
>>Jayshankar
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: raid0 and iscsi

[jshankar]

Hello Neil,


Thanks for the reply.

I was working on the internet scsi(iscsi drivers provided by intel) and 
software
raid configuration for my thesis in Storage Area Network.

The problems right now i am facing is
1> For a local disk write  for 512 mb in DELL 1.2 Ghz it takes me 1 min.
2> For a single iscsi device write, the transfer of data takes place at
4000 bytes/sec.
2> For raid configuration, the rate is pretty slow ( 40 bytes/sec).
One thing i have observe is that write to one of the device( from a total of 2 
device)  takes for few minutes and then stop.


I was trying to figure what routines in the operating system code, I might 
need
to look into to understand a problem if there is one.

 Some more problems.
If one of the network comes down, the mkfs for the raid , hangs.


>A raid0 array can be made of a number of drives of differing sizes.
>To accomodate this we divide the address space into several blocks.
>The first block is striped across all drives to the size of the
>smallest.  The next block is striped across the remaining drives to
>the size of the next smallest, etc.

Is there a relation between chunk size and block size of raid. Is this similar 
to buffer size( write(fd,&buffer,buffer_size) and block size of filesystem.

-Jay




>===== Original Message From Neil Brown <neilb@cse.unsw.edu.au> =====
>On Sunday October 12, jshankar@CS.ColoState.EDU wrote:
>> Hello,
>>
>>
>> I was testing the raid0 configuration for the iscsi device ( /dev/sdb and
>> /dev/sdc).
>>
>> For writing a 512MB in a LAN environment it was taking 24 hrs. i was going
>> through the source code of raid0.c and certain things doesn't made sense to 
me.
>>
>> 1> what role does the Hash bit play.
>
>A raid0 array can be made of a number of drives of differing sizes.
>To accomodate this we divide the address space into several blocks.
>The first block is striped across all drives to the size of the
>smallest.  The next block is striped across the remaining drives to
>the size of the next smallest, etc.
>
>Mapping from a virtual device address to block and thence a drive and
>offset is not straight forward.  It requires a table search.  The hash
>table helps accelerate this search.
>
>
>> 2>If my chunk size is 8 byte. Does that mean it will write 8 byte into 
device 0
>> and then into device 1. Is the write request to the disk in synchronous or
>> asynchronous mode.
>>
>You cannot have an 8byte chunk size.  4K is the minimum.
>With an 8K chunk size and 2 devices,
> sectors between 0 and 8K, 16k and 24K, 32K and 40K, 48K and 56K etc
>    are written to the first device.
> sectors between 8K and 16K, 28K and 32K, 40K and 48K etc
>    are written to the second device.
>
>raid0 does not impose any synchronisation.  Writes are only
>synchronous if the filesystem waits for them.  raid0 never waits.
>
>>
>> 3> Is wite_disk_sb in md.c responsible for writing into disk??. Is 
fsync_dev
>> responsible for synhronous write ??. If so can i change to asynchronous 
write.I
>> will really appreciate if somebody can tell me what all routines I need to 
go
>> through to figure out the functionality of raid0 behaviour.
>
>write_disk_sb is for writing the raid superblock to disk.  It doesn't
>happen often.
>fsync_dev is fairly irrelevant. You can safely ignore it.
>
>I hope that helps.
>
>NeilBrown
>
>>
>> Thanks
>> Jayshankar
>>
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

raid0 and iscsi

[Jayshankar Nair]

Hello,


I was testing the raid0 configuration for the iscsi device ( /dev/sdb and 
/dev/sdc).

For writing a 512MB in a LAN environment it was taking 24 hrs. i was going
through the source code of raid0.c and certain things doesn't made sense to me.

1> what role does the Hash bit play.
2>If my chunk size is 8 byte. Does that mean it will write 8 byte into device 0 
and then into device 1. Is the write request to the disk in synchronous or
asynchronous mode. 


3> Is wite_disk_sb in md.c responsible for writing into disk??. Is fsync_dev 
responsible for synhronous write ??. If so can i change to asynchronous write.I 
will really appreciate if somebody can tell me what all routines I need to go 
through to figure out the functionality of raid0 behaviour.

Thanks
Jayshankar

Re: raid0 and iscsi

[rob]

Hello,
 What kind of network are you uasing for the data transfer?

 The rest of post does not answer your specific questions, but is to 
show the transfer speeds on our network to backup servers using hardware 
and software raid.  

 Both systems use Mandrake 9.1.  network is a fast-lan. speed 100.

Bkup1 is scsi with Mylex hardware raid, athlon 1800 cpu.  array has 4  
9-gb lvd disks.

Bkup2 is a new test server using software raid.  dual 750 intel xeon,  
4- 9-gb se scsi disks.

 I transfer data 30-40 times per day to each server.  500-mb hourly on 
the average. ususal tme is less than a min.
 The test uses rsync to transfer data.  rsync uses a lot of cpu so the 
speed difference may be more due to cpu than the difference of raid type.

 the command used: rsync -e ssh -a --stats  /bkup3/  fbc5:/bkup3/

here is the result of sending approx 3-gb of data to each:
 
-------------------------------------------------
 this is the output of an rsync transfer to the backup software-raid server:

Mon Oct 13 07:36:41 EDT 2003
rsync[21832] (sender) heap statistics:
  arena:        7996056   (bytes from sbrk)
  ordblks:           59   (chunks not in use)
  smblks:             2
  hblks:              1   (chunks from mmap)
  hblkhd:        516096   (bytes from mmap)
  usmblks:            0
  fsmblks:           80
  uordblks:     7821736   (bytes used)
  fordblks:      174320   (bytes free)
  keepcost:      128840   (bytes in releasable chunk)

Number of files: 65699
Number of files transferred: 45739
Total file size: 2707740663 bytes
Total transferred file size: 2707630760 bytes
Literal data: 2707630760 bytes
Matched data: 0 bytes
File list size: 1,925,417
Total bytes written: 2,711,680,565
Total bytes read: 731844

wrote 2711680565 bytes  read 731844 bytes  5,046,348.67 bytes/sec
total size is 2707740663  speedup is 1.00
Mon Oct 13 07:45:38 EDT 2003

------------------------------------
to the hardware raid server:
Mon Oct 13 07:23:19 EDT 2003
rsync[19963] (sender) heap statistics:
  arena:        7996056   (bytes from sbrk)
  ordblks:           59   (chunks not in use)
  smblks:             2
  hblks:              1   (chunks from mmap)
  hblkhd:        516096   (bytes from mmap)
  usmblks:            0
  fsmblks:           80
  uordblks:     7821736   (bytes used)
  fordblks:      174320   (bytes free)
  keepcost:      128840   (bytes in releasable chunk)

Number of files: 65699
Number of files transferred: 45739
Total file size: 2707740663 bytes
Total transferred file size: 2707630760 bytes
Literal data: 2707630760 bytes
Matched data: 0 bytes
File list size: 1925417
Total bytes written: 2,711,680,565
Total bytes read: 731844

wrote 2711680565 bytes  read 731844 bytes  8,384,582.41 bytes/sec
total size is 2707740663  speedup is 1.00
Mon Oct 13 07:28:42 EDT 2003


----------------



Jayshankar Nair wrote:

>Hello,
>
>
>I was testing the raid0 configuration for the iscsi device ( /dev/sdb and 
>/dev/sdc).
>
>For writing a 512MB in a LAN environment it was taking 24 hrs. i was going
>through the source code of raid0.c and certain things doesn't made sense to me.
>
>1> what role does the Hash bit play.
>2>If my chunk size is 8 byte. Does that mean it will write 8 byte into device 0 
>and then into device 1. Is the write request to the disk in synchronous or
>asynchronous mode. 
>
>
>3> Is wite_disk_sb in md.c responsible for writing into disk??. Is fsync_dev 
>responsible for synhronous write ??. If so can i change to asynchronous write.I 
>will really appreciate if somebody can tell me what all routines I need to go 
>through to figure out the functionality of raid0 behaviour.
>
>Thanks
>Jayshankar
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  
>

Re: raid0 and iscsi

[rob]

the test I did faulty.  I mistakenly sent data to a non-raid disk on the 
software raid computer.
for kicks I'll retest and post the results sometime soon.

>
> the command used: rsync -e ssh -a --stats  /bkup3/  fbc5:/bkup3/
>
> here is the result of sending approx 3-gb of data to each:

Re: raid0 and iscsi

[rob]

here is result of writting 3-gb over lan :

1- to h/w raid system:  8,384,582.41 bytes/sec

2- to ide disk: 5,046,348.67 bytes/sec

3- to software raid system: 6,678,231.03 bytes/sec

 the h/w system has fasrer disks than the s/w raid. 


>
> the test I did faulty.  I mistakenly sent data to a non-raid disk on 
> the software raid computer.
> for kicks I'll retest and post the results sometime soon.
>
>>
>> the command used: rsync -e ssh -a --stats  /bkup3/  fbc5:/bkup3/
>>
>> here is the result of sending approx 3-gb of data to each:
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid0 and iscsi

[Neil Brown]

On Sunday October 12, jshankar@CS.ColoState.EDU wrote:
> Hello,
> 
> 
> I was testing the raid0 configuration for the iscsi device ( /dev/sdb and 
> /dev/sdc).
> 
> For writing a 512MB in a LAN environment it was taking 24 hrs. i was going
> through the source code of raid0.c and certain things doesn't made sense to me.
> 
> 1> what role does the Hash bit play.

A raid0 array can be made of a number of drives of differing sizes.
To accomodate this we divide the address space into several blocks.
The first block is striped across all drives to the size of the
smallest.  The next block is striped across the remaining drives to
the size of the next smallest, etc.

Mapping from a virtual device address to block and thence a drive and
offset is not straight forward.  It requires a table search.  The hash
table helps accelerate this search.


> 2>If my chunk size is 8 byte. Does that mean it will write 8 byte into device 0 
> and then into device 1. Is the write request to the disk in synchronous or
> asynchronous mode. 
> 
You cannot have an 8byte chunk size.  4K is the minimum.
With an 8K chunk size and 2 devices,
 sectors between 0 and 8K, 16k and 24K, 32K and 40K, 48K and 56K etc
    are written to the first device.
 sectors between 8K and 16K, 28K and 32K, 40K and 48K etc
    are written to the second device.

raid0 does not impose any synchronisation.  Writes are only
synchronous if the filesystem waits for them.  raid0 never waits.

> 
> 3> Is wite_disk_sb in md.c responsible for writing into disk??. Is fsync_dev 
> responsible for synhronous write ??. If so can i change to asynchronous write.I 
> will really appreciate if somebody can tell me what all routines I need to go 
> through to figure out the functionality of raid0 behaviour.

write_disk_sb is for writing the raid superblock to disk.  It doesn't
happen often.
fsync_dev is fairly irrelevant. You can safely ignore it.

I hope that helps.

NeilBrown

> 
> Thanks
> Jayshankar
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html