raw devices

raw devices

[simon]

I have been performing some tests on raw devices, the results of 
which I do not understand. I have 4 FC busses, each with one  
SCSI disk, connect to a Linux system running 2.4.16. I use the raw 
command to bind /dev/raw1 - /dev/raw4 to each of the devices. 
With one process per raw device, running large sequential reads, I 
got a total throughput  of 340 Megabytes per second. I also 
observed 85% CPU idle. Following this I performed some more 
tests and then returned to this one. This time the total had gone 
down to 180 and there was no free CPU. I realized that the first 
time I ran the tests, each of the disks that the raw devices were 
mapped to were mounted. I then verified that this data was being 
transferred along the FC bus using an analyzer while the devices 
were mounted. Can anyone explain this to me ? I find it hard to 
believe that the disk should be permitted to be mounted when 
using raw device mappings. If the disks should not be mounted 
why is there such a great performance difference ?

Many Thanks

Simon.
__________________________

Simon Haynes - Baydel 
Phone : 44 (0) 1372 378811
Email : simon@baydel.com
__________________________

Re: raw devices

[Andrea Arcangeli]

On Thu, Dec 20, 2001 at 12:22:33PM -0000, simon@baydel.com wrote:
> I have been performing some tests on raw devices, the results of 
> which I do not understand. I have 4 FC busses, each with one  
> SCSI disk, connect to a Linux system running 2.4.16. I use the raw 
> command to bind /dev/raw1 - /dev/raw4 to each of the devices. 
> With one process per raw device, running large sequential reads, I 
> got a total throughput  of 340 Megabytes per second. I also 
> observed 85% CPU idle. Following this I performed some more 
> tests and then returned to this one. This time the total had gone 
> down to 180 and there was no free CPU. I realized that the first 
> time I ran the tests, each of the disks that the raw devices were 
> mapped to were mounted. I then verified that this data was being 
> transferred along the FC bus using an analyzer while the devices 
> were mounted. Can anyone explain this to me ? I find it hard to 
> believe that the disk should be permitted to be mounted when 
> using raw device mappings. If the disks should not be mounted 
> why is there such a great performance difference ?

because when mounted rawio uses a blocksize larger than the
softblocksize (for no good reason, but that's another issue).

in short when the disk is mouned a single bh was doing I/O on a page (I
guess you were using 4k blocksize in the fs), while when it was
unmounted it was doing I/O on only 512bytes, so you needed 8 times more
bh and CPU to do the same I/O. 

there are many ways to fix this, I guess it would be nice to get the bio
stuff optimized in 2.5 first (bio will be even faster than your rawio
with the fs mounted, because a single metadata entity will be able to do
I/O on more than one page, that's the whole point of the bio design
changes). Then in 2.5 rawio can also be obsoleted and modularized.
O_DIRECT will have to relax its granularity requirements for both fs and
blkdev and so then it will overlap enterely the rawio chardevice
functionality.

btw, if you try to use O_DIRECT on the fs with your current kernel, you
should be just able to read at 340mbyte/sec with most of the CPU idle.

Andrea

Re: raw devices

[simon]

Andrea,

This clears this up, thanks very much. I have tried using the 
O_DIRECT and it is almost as fast as the mounted raw devices.
It sounds like 2.5 is set to improve on this. Do you know if there 
have been any suggestions regarding using mounting options on a 
filesystem so that none of the files on that filesystem are cached ?

Thanks again 

Simon. 


On 20 Dec 2001, at 18:05, Andrea Arcangeli wrote:

> On Thu, Dec 20, 2001 at 12:22:33PM -0000, simon@baydel.com wrote:
> > I have been performing some tests on raw devices, the results of 
> > which I do not understand. I have 4 FC busses, each with one  
> > SCSI disk, connect to a Linux system running 2.4.16. I use the raw 
> > command to bind /dev/raw1 - /dev/raw4 to each of the devices. 
> > With one process per raw device, running large sequential reads, I 
> > got a total throughput  of 340 Megabytes per second. I also 
> > observed 85% CPU idle. Following this I performed some more 
> > tests and then returned to this one. This time the total had gone 
> > down to 180 and there was no free CPU. I realized that the first 
> > time I ran the tests, each of the disks that the raw devices were 
> > mapped to were mounted. I then verified that this data was being 
> > transferred along the FC bus using an analyzer while the devices 
> > were mounted. Can anyone explain this to me ? I find it hard to 
> > believe that the disk should be permitted to be mounted when 
> > using raw device mappings. If the disks should not be mounted 
> > why is there such a great performance difference ?
> 
> because when mounted rawio uses a blocksize larger than the
> softblocksize (for no good reason, but that's another issue).
> 
> in short when the disk is mouned a single bh was doing I/O on a page (I
> guess you were using 4k blocksize in the fs), while when it was
> unmounted it was doing I/O on only 512bytes, so you needed 8 times more
> bh and CPU to do the same I/O. 
> 
> there are many ways to fix this, I guess it would be nice to get the bio
> stuff optimized in 2.5 first (bio will be even faster than your rawio
> with the fs mounted, because a single metadata entity will be able to do
> I/O on more than one page, that's the whole point of the bio design
> changes). Then in 2.5 rawio can also be obsoleted and modularized.
> O_DIRECT will have to relax its granularity requirements for both fs and
> blkdev and so then it will overlap enterely the rawio chardevice
> functionality.
> 
> btw, if you try to use O_DIRECT on the fs with your current kernel, you
> should be just able to read at 340mbyte/sec with most of the CPU idle.
> 
> Andrea


__________________________

Simon Haynes - Baydel 
Phone : 44 (0) 1372 378811
Email : simon@baydel.com
__________________________