DMA for ide-scsi?

DMA for ide-scsi?

[Mikael Pettersson]

On Sat, 13 Sep 2003 08:29:18 +0200, Jens Axboe <axboe@suse.de> wrote:
>> Actually with 2.6, you no longer need ide-scsi.  You'll need to upgrade 
>> your cdrecord tools and probably your burning GUI, if you use one.  I've 
>> been burning that way for several months now.  (I'm using xcdroast, 
>> though I need to start it with "-n" since I'm using cdrecord 2.01a18.)  
>> This actually works better for me than ide-scsi as for some reason it 
>> uses less CPU.
>
>That's because it _is_ faster. It contains no silly memory allocations
>for the buffer and data copying in the kernel, the data is mapped from
>the user buffer and DMA'ed directly from there. It also uses DMA where
>ide-scsi wont.

That begs the question: why won't ide-scsi do DMA?
I understand you'd rather see it disappear (:->) but since I use
it for other ATAPI devices as well, I'd like to see it maintained
and fully operational. Having DMA in ide-scsi would be nice.

(And the concept of using a SCSI API to ATA devices is in itself
not broken, even if the implementation has some problems.)

/Mikael

Re: DMA for ide-scsi?

[Bartlomiej Zolnierkiewicz]

On Saturday 13 of September 2003 13:01, Mikael Pettersson wrote:
> On Sat, 13 Sep 2003 08:29:18 +0200, Jens Axboe <axboe@suse.de> wrote:
> >> Actually with 2.6, you no longer need ide-scsi.  You'll need to upgrade
> >> your cdrecord tools and probably your burning GUI, if you use one.  I've
> >> been burning that way for several months now.  (I'm using xcdroast,
> >> though I need to start it with "-n" since I'm using cdrecord 2.01a18.)
> >> This actually works better for me than ide-scsi as for some reason it
> >> uses less CPU.
> >
> >That's because it _is_ faster. It contains no silly memory allocations
> >for the buffer and data copying in the kernel, the data is mapped from
> >the user buffer and DMA'ed directly from there. It also uses DMA where
> >ide-scsi wont.
>
> That begs the question: why won't ide-scsi do DMA?
> I understand you'd rather see it disappear (:->) but since I use
> it for other ATAPI devices as well, I'd like to see it maintained
> and fully operational. Having DMA in ide-scsi would be nice.

ide-scsi maintainer position is available :-).

> (And the concept of using a SCSI API to ATA devices is in itself
> not broken, even if the implementation has some problems.)

It is not broken short-term, however it is broken long-term.

> /Mikael

Re: DMA for ide-scsi?

[Alan Cox]

On Sad, 2003-09-13 at 12:01, Mikael Pettersson wrote:
> That begs the question: why won't ide-scsi do DMA?

Because nobody added it. Its that simple

> I understand you'd rather see it disappear (:->) but since I use
> it for other ATAPI devices as well, I'd like to see it maintained
> and fully operational. Having DMA in ide-scsi would be nice.

I don't see it vanishing either - people abuse IDE (especially SATA) for
weird stuff like high end scanners which want to use ide-scsi for sg. I
agree with Bart about ide-scsi for disks. For tape ide-tape isnt good
enough for newer stuff but that could be fixed either way

> (And the concept of using a SCSI API to ATA devices is in itself
> not broken, even if the implementation has some problems.)

ATA drives don't generally talk ATAPI as well so you have protocol
stuff. For Serial ATA that is what the current SATA code everyone is
using does and each SATA vendor has followed the same path because our
existing PATA code 

In 2.7 the SCSI layer split can get finished so it seperates "scsi the
protocol" from "queueing engine and handling for an intelligent
controller".

Right now its not too bad - error handling is entirely pluggable for
example.

2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jeff Garzik]

On Sat, Sep 13, 2003 at 07:04:36PM +0100, Alan Cox wrote:
> In 2.7 the SCSI layer split can get finished so it seperates "scsi the
> protocol" from "queueing engine and handling for an intelligent
> controller".

Yep, this is exactly my plan.  And exactly why libata must use
the SCSI layer in 2.4 and 2.6, and in 2.7 use the "queueing engine ..."
stuff that you describe.

There are a lot of pieces in the SCSI layer that I want to move "up" to
the block layer in 2.7:  ->queuecommand "queue one" callback and loop
(Jens has code for this already), error handling thread helper code,
low-level device driver registration structure[1], personality code
(disk, cdrom, tape, ....), support for host controllers which may have
host _or_ device (or both!) TCQ limits, and on and on.

That's a ton of code, and IMO it's not feasible to (a) recreate it
for 2.6 libata, or (b) do the "move up" in 2.6.0-testX and change ide,
scsi, block, ... drivers to use the new helpers and new structure.
Changes are too big this late in the 2.6 game.

As of next week (I'm presenting this at Intel Developer Forum), libata
will support "AHCI", Intel's next generation SATA controller.  Each PCI
device supports up to 32 SATA ports (one SATA device each).  Each port
supports up to 32 outstanding ATA commands (i.e. 32 tags), including a
64-bit-DMA-capable scatter-gather table.  The S/G doesn't have the silly
64K segment boundary worries, either.

Promise hardware is similar -- up to 10 "packets" (taskfiles) can be
queued per host... not per device.  SCSI layer handles this with just a
few knob-turns.

For 2.6, libata (unfortunately) requires the SCSI layer for ATA
devices, and libata drives real hardware that noone else can drive.

For 2.7, when all this code "moves up" -- basically adding a bunch of
helper functions to the block layer -- libata won't need to treat ATA
devices as SCSI devices.

Some developers have rightfully pointed out that disks going from
/dev/hdX to /dev/sdX might create some user confusion.  This hasn't been
the case in practice.  Partly because LABEL= is fairly prevalent, and
partly because libata is used for "only SATA" scenarios, which is by
definition new hardware.


On to /dev/disk...
Another interesting part of this "moving up" is that I want to unify the
personality code.  The various tape, cdrom, and disk modules for ide,
scsi, and others are quite similar.  And if you look closer at today's
block layer, you see that already everything is designed around "struct
request".  So I envision a more top-down structure, with helper
functions and callbacks combining to form:  /dev/disk, /dev/cdrom,
/dev/tape, ...

The generic "disk" personality code would take care of allocating block
device major/minors (i.e. register_blkdev duties), and generating
requests.  Existing ->prep_rq helpers will fill in the specific details.
Subsystem-specfic ioctls can be delivered as REQ_SPECIAL.

cdrom, tape, and friends follow a similar pattern.

"What about /dev/hda!"  Answer:  a simple remapping block device, which
simulates /dev/hdXX or /dev/sdXX, by remapping all requests to
/dev/diskXX.

So the next time you hear me say "libata must wait for 2.7 to ditch SCSI"
this is what I mean ;-)  These aren't unachieveable goals, either.  It's
mostly code shuffling.  This code mostly already exists today.

	Jeff


[1] low-level driver registers a set of callbacks, which are
either implemented in the driver itself (cciss, cpqarray) or mostly
library-based helper functions (ATA or SCSI).

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jeff Garzik]

Oh, and I'm pondering the best way to deliver out-of-bang ATA taskfiles
and SCSI cdbs to a device.  (for the uninitiated, this is lower level
than block devices / cdrom devices / etc.)

 ... AF_BLOCK is not out of the question ;-)

	Jeff

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jeff Garzik]

On Sat, Sep 13, 2003 at 03:01:31PM -0400, Jeff Garzik wrote:
> Oh, and I'm pondering the best way to deliver out-of-bang ATA taskfiles

lol.  That would be "out-of-band".

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Bartlomiej Zolnierkiewicz]

On Saturday 13 of September 2003 20:49, Jeff Garzik wrote:
> On Sat, Sep 13, 2003 at 07:04:36PM +0100, Alan Cox wrote:
> > In 2.7 the SCSI layer split can get finished so it seperates "scsi the
> > protocol" from "queueing engine and handling for an intelligent
> > controller".
>
> Yep, this is exactly my plan.  And exactly why libata must use
> the SCSI layer in 2.4 and 2.6, and in 2.7 use the "queueing engine ..."
> stuff that you describe.
>
> There are a lot of pieces in the SCSI layer that I want to move "up" to
> the block layer in 2.7:  ->queuecommand "queue one" callback and loop
> (Jens has code for this already), error handling thread helper code,
> low-level device driver registration structure[1], personality code
> (disk, cdrom, tape, ....), support for host controllers which may have
> host _or_ device (or both!) TCQ limits, and on and on.
>
> That's a ton of code, and IMO it's not feasible to (a) recreate it
> for 2.6 libata, or (b) do the "move up" in 2.6.0-testX and change ide,
> scsi, block, ... drivers to use the new helpers and new structure.
> Changes are too big this late in the 2.6 game.
>
> As of next week (I'm presenting this at Intel Developer Forum), libata
> will support "AHCI", Intel's next generation SATA controller.  Each PCI
> device supports up to 32 SATA ports (one SATA device each).  Each port
> supports up to 32 outstanding ATA commands (i.e. 32 tags), including a
> 64-bit-DMA-capable scatter-gather table.  The S/G doesn't have the silly
> 64K segment boundary worries, either.
>
> Promise hardware is similar -- up to 10 "packets" (taskfiles) can be
> queued per host... not per device.  SCSI layer handles this with just a
> few knob-turns.
>
> For 2.6, libata (unfortunately) requires the SCSI layer for ATA
> devices, and libata drives real hardware that noone else can drive.
>
> For 2.7, when all this code "moves up" -- basically adding a bunch of
> helper functions to the block layer -- libata won't need to treat ATA
> devices as SCSI devices.

s/ATA/SATA/

ATA and SATA will still need their own driver(s) aware of driver-model,
sysfs, ATA quirks/tuning etc.  I am working on this part currently, so you can
concentrate on new, sexy SATA, leaving all dirty, legacy ATA for me.

For all other stuff described in your mail I can only say: HELL YEAH!.

> Some developers have rightfully pointed out that disks going from
> /dev/hdX to /dev/sdX might create some user confusion.  This hasn't been
> the case in practice.  Partly because LABEL= is fairly prevalent, and
> partly because libata is used for "only SATA" scenarios, which is by
> definition new hardware.
>
>
> On to /dev/disk...
> Another interesting part of this "moving up" is that I want to unify the
> personality code.  The various tape, cdrom, and disk modules for ide,
> scsi, and others are quite similar.  And if you look closer at today's
> block layer, you see that already everything is designed around "struct
> request".  So I envision a more top-down structure, with helper
> functions and callbacks combining to form:  /dev/disk, /dev/cdrom,
> /dev/tape, ...
>
> The generic "disk" personality code would take care of allocating block
> device major/minors (i.e. register_blkdev duties), and generating
> requests.  Existing ->prep_rq helpers will fill in the specific details.
> Subsystem-specfic ioctls can be delivered as REQ_SPECIAL.
>
> cdrom, tape, and friends follow a similar pattern.
>
> "What about /dev/hda!"  Answer:  a simple remapping block device, which
> simulates /dev/hdXX or /dev/sdXX, by remapping all requests to
> /dev/diskXX.
>
> So the next time you hear me say "libata must wait for 2.7 to ditch SCSI"
> this is what I mean ;-)  These aren't unachieveable goals, either.  It's
> mostly code shuffling.  This code mostly already exists today.
>
> 	Jeff
>
>
> [1] low-level driver registers a set of callbacks, which are
> either implemented in the driver itself (cciss, cpqarray) or mostly
> library-based helper functions (ATA or SCSI).

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jeff Garzik]

On Sat, Sep 13, 2003 at 09:24:05PM +0200, Bartlomiej Zolnierkiewicz wrote:
> On Saturday 13 of September 2003 20:49, Jeff Garzik wrote:
> > For 2.6, libata (unfortunately) requires the SCSI layer for ATA
> > devices, and libata drives real hardware that noone else can drive.
> >
> > For 2.7, when all this code "moves up" -- basically adding a bunch of
> > helper functions to the block layer -- libata won't need to treat ATA
> > devices as SCSI devices.
> 
> s/ATA/SATA/
> 
> ATA and SATA will still need their own driver(s) aware of driver-model,
> sysfs, ATA quirks/tuning etc.

Agreed.  Though I think some of your work in sysfs area can be made
common.


> I am working on this part currently, so you can
> concentrate on new, sexy SATA, leaving all dirty, legacy ATA for me.

Sounds good to me ;-)  Though I'll definitely want to work together with
you on several issues in 2.7...


> For all other stuff described in your mail I can only say: HELL YEAH!.

;-)

	Jeff

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jens Axboe]

On Sat, Sep 13 2003, Jeff Garzik wrote:
> Oh, and I'm pondering the best way to deliver out-of-bang ATA taskfiles
> and SCSI cdbs to a device.  (for the uninitiated, this is lower level
> than block devices / cdrom devices / etc.)
> 
>  ... AF_BLOCK is not out of the question ;-)

Eh... I wont comment on that. I think we are way into Garzik lala land
there :)

I'd prefer just keeping sg_io_hdr, but dumping sg. A fully fledged bsg
(block sg) implementation. That way programs continue to work like
before on ATAPI/SCSI, for ATA we can use it as a task file transport.

-- 
Jens Axboe

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jeff Garzik]

On Mon, Sep 15, 2003 at 09:34:45AM +0200, Jens Axboe wrote:
> On Sat, Sep 13 2003, Jeff Garzik wrote:
> > Oh, and I'm pondering the best way to deliver out-of-bang ATA taskfiles
> > and SCSI cdbs to a device.  (for the uninitiated, this is lower level
> > than block devices / cdrom devices / etc.)
> > 
> >  ... AF_BLOCK is not out of the question ;-)
> 
> Eh... I wont comment on that. I think we are way into Garzik lala land
> there :)
> 
> I'd prefer just keeping sg_io_hdr, but dumping sg. A fully fledged bsg
> (block sg) implementation. That way programs continue to work like
> before on ATAPI/SCSI, for ATA we can use it as a task file transport.

I don't propose dumping the ugly "submit cdb/taskfile" ioctls, but we do
need to deprecate them.  The ioctls are awful for throughput, async
queueing, and the like.  And of course in general, ioctls are evil :)

And we should deprecate them with a solution that aligns what with Linus
described in Dec 2001 on lkml:  a chrdev where userland write(2)s cdbs
and taskfiles, and read(2)s the results.  This is where my thinking
picked up:  if we are creating a chrdev to send "packets" and receive
responses to those packets............  <insert conclusion here>

	Jeff

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jens Axboe]

On Tue, Sep 16 2003, Jeff Garzik wrote:
> On Mon, Sep 15, 2003 at 09:34:45AM +0200, Jens Axboe wrote:
> > On Sat, Sep 13 2003, Jeff Garzik wrote:
> > > Oh, and I'm pondering the best way to deliver out-of-bang ATA taskfiles
> > > and SCSI cdbs to a device.  (for the uninitiated, this is lower level
> > > than block devices / cdrom devices / etc.)
> > > 
> > >  ... AF_BLOCK is not out of the question ;-)
> > 
> > Eh... I wont comment on that. I think we are way into Garzik lala land
> > there :)
> > 
> > I'd prefer just keeping sg_io_hdr, but dumping sg. A fully fledged bsg
> > (block sg) implementation. That way programs continue to work like
> > before on ATAPI/SCSI, for ATA we can use it as a task file transport.
> 
> I don't propose dumping the ugly "submit cdb/taskfile" ioctls, but we do
> need to deprecate them.  The ioctls are awful for throughput, async
> queueing, and the like.  And of course in general, ioctls are evil :)
> 
> And we should deprecate them with a solution that aligns what with Linus
> described in Dec 2001 on lkml:  a chrdev where userland write(2)s cdbs
> and taskfiles, and read(2)s the results.  This is where my thinking
> picked up:  if we are creating a chrdev to send "packets" and receive
> responses to those packets............  <insert conclusion here>

== bsg, block sg. Did you read what I wrote? :). I started implementing
this and have something that barely works. You just bind a block device
to a /dev/sg* char device and use read/write on that. Aka sg.

I don't want ioctls command submission interfaces more than you do.

-- 
Jens Axboe

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jeff Garzik]

Jens Axboe wrote:
> On Tue, Sep 16 2003, Jeff Garzik wrote:

>>And we should deprecate them with a solution that aligns what with Linus
>>described in Dec 2001 on lkml:  a chrdev where userland write(2)s cdbs
>>and taskfiles, and read(2)s the results.  This is where my thinking
>>picked up:  if we are creating a chrdev to send "packets" and receive
>>responses to those packets............  <insert conclusion here>
> 
> 
> == bsg, block sg. Did you read what I wrote? :). I started implementing
> this and have something that barely works. You just bind a block device
> to a /dev/sg* char device and use read/write on that. Aka sg.

sg needs some modifications -- for example it errors out instead of 
sleeps on queue full -- but sounds good to me.


> I don't want ioctls command submission interfaces more than you do.

Groovy.

	Jeff

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Alan Cox]

On Sad, 2003-09-20 at 19:28, Jeff Garzik wrote:
> sg needs some modifications -- for example it errors out instead of 
> sleeps on queue full -- but sounds good to me.

Is that an error and change in behaviour ?

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jeff Garzik]

Alan Cox wrote:
> On Sad, 2003-09-20 at 19:28, Jeff Garzik wrote:
> 
>>sg needs some modifications -- for example it errors out instead of 
>>sleeps on queue full -- but sounds good to me.
> 
> 
> Is that an error and change in behaviour ?


No and yes.  :)

Current sg breaks the Unix model of write(2)... you shouldn't error out 
if the queue will "probably" become available again.

	Jeff

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Alan Cox]

On Sad, 2003-09-20 at 23:22, Jeff Garzik wrote:
> > Is that an error and change in behaviour ?
> 
> 
> No and yes.  :)
> 
> Current sg breaks the Unix model of write(2)... you shouldn't error out 
> if the queue will "probably" become available again.

Thats what I thought - because apps want to know if the queue is full
when doing raw scsi work some of the time. Would the change break any
known apps - if not it seems fine (providing O_NDELAY is supported)

Re: 2.7 block ramblings (was Re: DMA for ide-scsi?)

[Jens Axboe]

On Sat, Sep 20 2003, Jeff Garzik wrote:
> Jens Axboe wrote:
> >On Tue, Sep 16 2003, Jeff Garzik wrote:
> 
> >>And we should deprecate them with a solution that aligns what with Linus
> >>described in Dec 2001 on lkml:  a chrdev where userland write(2)s cdbs
> >>and taskfiles, and read(2)s the results.  This is where my thinking
> >>picked up:  if we are creating a chrdev to send "packets" and receive
> >>responses to those packets............  <insert conclusion here>
> >
> >
> >== bsg, block sg. Did you read what I wrote? :). I started implementing
> >this and have something that barely works. You just bind a block device
> >to a /dev/sg* char device and use read/write on that. Aka sg.
> 
> sg needs some modifications -- for example it errors out instead of 
> sleeps on queue full -- but sounds good to me.

Definitely. bsg will be a new implementation from scratch, also dumping
a lot of really (imo) useless "features" that clutter up the code. And
yes, of course it should honor the typical write(2) model :)

-- 
Jens Axboe