Writing on raw device with software RAID 0 is slow

Writing on raw device with software RAID 0 is slow

[Martin Rauh]

Hello,

Writing to an software RAID 0 containing 4 SCSI discs is very fast.
I get transfer rates of about 100 MBytes/s. The filesystem on the RAID
is ext2.

Writing to the same RAID directly (that means on the raw device without
a filesystem) works
but gives low transfer rates of about 31 MBytes/s.

Any explanation for that?

thanks and greetings,

Martin Rauh

Re: Writing on raw device with software RAID 0 is slow

[Stephen C. Tweedie]

Hi,

On Wed, Feb 28, 2001 at 03:58:11PM +0100, Martin Rauh wrote:
> 
> Writing to an software RAID 0 containing 4 SCSI discs is very fast.
> I get transfer rates of about 100 MBytes/s. The filesystem on the RAID
> is ext2.
> 
> Writing to the same RAID directly (that means on the raw device without
> a filesystem) works
> but gives low transfer rates of about 31 MBytes/s.
> 
> Any explanation for that?

Raw IO is always synchronous: it gets flushed to disk before the write
returns.  You don't get any write-behind with raw IO, so the smaller
the blocksize you write in, the slower things get.

--Stephen

Re: Writing on raw device with software RAID 0 is slow

[Ben LaHaise]

Hello all,

On Thu, 1 Mar 2001, Stephen C. Tweedie wrote:

> Raw IO is always synchronous: it gets flushed to disk before the write
> returns.  You don't get any write-behind with raw IO, so the smaller
> the blocksize you write in, the slower things get.

More importantly, the mainstream raw io code only writes in 64KB chunks
that are unpipelined, which can lead to writes not hitting the drive
before the sector passes under the rw head.  You can work around this to
some extent by issuing multiple writes (via threads, or the aio work I've
done) at the expense of atomicity.  Also, before we allow locking of
arbitrary larger ios in main memory, we need bean counting to prevent the
obvious DoSes.

		-ben

Re: Writing on raw device with software RAID 0 is slow

[Stephen C. Tweedie]

Hi,

On Thu, Mar 01, 2001 at 10:44:38AM -0500, Ben LaHaise wrote:
> 
> On Thu, 1 Mar 2001, Stephen C. Tweedie wrote:
> 
> > Raw IO is always synchronous: it gets flushed to disk before the write
> > returns.  You don't get any write-behind with raw IO, so the smaller
> > the blocksize you write in, the slower things get.
> 
> More importantly, the mainstream raw io code only writes in 64KB chunks
> that are unpipelined, which can lead to writes not hitting the drive
> before the sector passes under the rw head.  You can work around this to
> some extent by issuing multiple writes (via threads, or the aio work I've
> done) at the expense of atomicity.  Also, before we allow locking of
> arbitrary larger ios in main memory, we need bean counting to prevent the
> obvious DoSes.

Yep.  There shouldn't be any problem increasing the 64KB size, it's
only the lack of accounting for the pinned memory which stopped me
increasing it by default.

--Stephen

Re: Writing on raw device with software RAID 0 is slow

[Ben LaHaise]

On Thu, 1 Mar 2001, Stephen C. Tweedie wrote:

> Yep.  There shouldn't be any problem increasing the 64KB size, it's
> only the lack of accounting for the pinned memory which stopped me
> increasing it by default.

Actually, how about making it a sysctl?  That's probably the most
reasonable approach for now since the optimal size depends on hardware.

		-ben

Re: Writing on raw device with software RAID 0 is slow

[Stephen C. Tweedie]

Hi,

On Thu, Mar 01, 2001 at 11:08:13AM -0500, Ben LaHaise wrote:
> On Thu, 1 Mar 2001, Stephen C. Tweedie wrote:
> 
> Actually, how about making it a sysctl?  That's probably the most
> reasonable approach for now since the optimal size depends on hardware.

Fine with me.

--Stephen

Re: Writing on raw device with software RAID 0 is slow

[Douglas Gilbert]

 
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Ben LaHaise wrote:
> On Thu, 1 Mar 2001, Stephen C. Tweedie wrote:
> 
> > Yep.  There shouldn't be any problem increasing the 64KB size, it's
> > only the lack of accounting for the pinned memory which stopped me
> > increasing it by default.
> 
> Actually, how about making it a sysctl?  That's probably the most
> reasonable approach for now since the optimal size depends on hardware.

Something else may slow down raw IO. A buffer
that looks contiguous in the user space typically looks
quite splintered from the kernel's perspective. This
means that a buffer of 64 KB in the user space ends
up being a scatter gather list of 16 elements (assuming
PAGE_SIZE of 4KB) en route to the IDE or SCSI subsystem.

Now one SCSI adapter that I have examined must push each
scatter gather element through its firmware to the DMA 
engine which can only hold one element at a time. 
That takes time.

Doug Gilbert