[RFC] {read,write}s{b,w,l} or iobarrier_*()

[RFC] {read,write}s{b,w,l} or iobarrier_*()

[Benjamin Herrenschmidt]

Hi !

Here I need people to agree so we decide once for all what
we want and move further (this has been discussed several
times already withotu agreement).

So, the "generic" problem is: reading/writing from/to a FIFO
that is larger than 8 bits wide on a big endian machine requires
more than just a loop of reads or writes. Those IO functions
are doing byteswap (and barriers on every call) while the FIFO
access wants typically to be not byteswapped nor have barriers
on every access.

For "PIO", we already provide the "s" functions {in,out}s{b,w,l}
that are implemented by every arch and provide drivers with a
good abstraction to use (see ide_insw/ide_outsw in 2.5 which
should just be a simple insw/outsw for normal interfaces for
example).

However, we don't provide a similar abstraction for MMIO.
Typical cases where we want this is MMIO versions of the
IDE iops (we probably want to provide generic impl of both
PIO and MMIO ones in ide-iops, not just PIO); some sound
cards, etc...

A driver needing that today will have to either use a loop
of {read,write}{b,w,l} and undo byteswapping (ugh), or try
to re-implement it using raw_{read,write}{b,w,l} and adding
proper iobarrier_* where needed. But the iobarrier functions
aren't provided by all archs which means ugly #ifdef salad,
and I hate having to put the burden of knowing everything
about barriers one each single driver maintainer.

So we have 2 solutions here (one of which I prefer, but I
still want the debate open here):

 - Have all archs provide {read,write}s{b,w,l} functions.
Those will hide all of the details of bytewapping & barriers
from the drivers and can be used as-is for things like IDE
MMIO iops.

 - Have all archs provide iobarrier_* functions. Here, drivers
would still have to re-implement the transfer loops with
raw_{read,write}{b,w,l} and do proper use of iobarrier_*.

Ben.

Re: [RFC] {read,write}s{b,w,l} or iobarrier_*()

[Jeff Garzik]

Benjamin Herrenschmidt wrote:
> So we have 2 solutions here (one of which I prefer, but I
> still want the debate open here):
> 
>  - Have all archs provide {read,write}s{b,w,l} functions.
> Those will hide all of the details of bytewapping & barriers
> from the drivers and can be used as-is for things like IDE
> MMIO iops.

I prefer this solution...


>  - Have all archs provide iobarrier_* functions. Here, drivers
> would still have to re-implement the transfer loops with
> raw_{read,write}{b,w,l} and do proper use of iobarrier_*.

I have a tulip patch from Peter de Shivjer (sp?) that adds 
iobarrier_rw() and I think it looks ugly as sin.  I would much prefer 
the first solution...

	Jeff

Re: [RFC] {read,write}s{b,w,l} or iobarrier_*()

[David S. Miller]

   From: Jeff Garzik <jgarzik@pobox.com>
   Date: Thu, 26 Sep 2002 12:23:41 -0400

   Benjamin Herrenschmidt wrote:
   >  - Have all archs provide {read,write}s{b,w,l} functions.
   > Those will hide all of the details of bytewapping & barriers
   > from the drivers and can be used as-is for things like IDE
   > MMIO iops.
   
   I prefer this solution...

I'm starting to think about taking back all the previous
arguments I had against this idea.  It's starting to sound
like the preferred way to go.

Re: [RFC] {read,write}s{b,w,l} or iobarrier_*()

[Andre Hedrick]

On Thu, 26 Sep 2002, David S. Miller wrote:

>    From: Jeff Garzik <jgarzik@pobox.com>
>    Date: Thu, 26 Sep 2002 12:23:41 -0400
> 
>    Benjamin Herrenschmidt wrote:
>    >  - Have all archs provide {read,write}s{b,w,l} functions.
>    > Those will hide all of the details of bytewapping & barriers
>    > from the drivers and can be used as-is for things like IDE
>    > MMIO iops.
>    
>    I prefer this solution...
> 
> I'm starting to think about taking back all the previous
> arguments I had against this idea.  It's starting to sound
> like the preferred way to go.

Hi Dave,

One of the main reasons for changing the core iops of the ata/ide driver
results from newer HBA's either supporting dual transports modes or in
some case exclusive MMIO, similar to current IOMIO x86 HBA's today.

Additionally, I suspect the current dual path for execution may end up
going N-ways.  As there are a few PATA hosts and in the future all SATA
hosts will convert to the FP-DMA (First Party DMA) messaging service
protocol.

This will then make (3) different execution transports.  The bad news is
during transition period (just now starting) it will become more painful
if we do not make the drive independent of iops regardless of arch/bus
issues.

One of the key ideas hottly debated between Ben, Jeff, and myself were the
impacts resulting from this change in direction.  Obviously Ben lives in a
world of MMIO and has suffered with a driver which was designed around
IOMIO.  Thus all the bastardized macros we all hate resulted from
x86-centric lameness (industry driven for the most part, yeah me included).

Now if we expand the issue into Jeff's world of the net-stack drivers, he
banged me over the head with the issue of "pci-posting delays".  Ben got
his shots in also about the issue, too.  Thus the resulting io_barrier
additions by Ben to the original ATA-driver transformation, can be extened
to the Net-Drivers.  Oh and the slope is increasing fast now.

So if it works for ATA and NET, could it not migrate to all hardware?
If it makes it to all hardware, should it not be coupled to the BUS?
If it is coupled to the BUS, are there going to be problems with
exceptions?

Well once I leave ATA (storage in general), I am not really able to
discuss those issues from a position of first hand experience.  So here is
my nickel spent.

Cheers,

Andre Hedrick
LAD Storage Consulting Group

Re: [RFC] {read,write}s{b,w,l} or iobarrier_*()

[Geert Uytterhoeven]

On Thu, 26 Sep 2002, Jeff Garzik wrote:
> Benjamin Herrenschmidt wrote:
> > So we have 2 solutions here (one of which I prefer, but I
> > still want the debate open here):
> > 
> >  - Have all archs provide {read,write}s{b,w,l} functions.
> > Those will hide all of the details of bytewapping & barriers
> > from the drivers and can be used as-is for things like IDE
> > MMIO iops.
> 
> I prefer this solution...
> 
> 
> >  - Have all archs provide iobarrier_* functions. Here, drivers
> > would still have to re-implement the transfer loops with
> > raw_{read,write}{b,w,l} and do proper use of iobarrier_*.
> 
> I have a tulip patch from Peter de Shivjer (sp?) that adds 

Peter De Schrijver, I assume.

> iobarrier_rw() and I think it looks ugly as sin.  I would much prefer 
> the first solution...

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: [RFC] {read,write}s{b,w,l} or iobarrier_*()

[Benjamin Herrenschmidt]

>
>Now if we expand the issue into Jeff's world of the net-stack drivers, he
>banged me over the head with the issue of "pci-posting delays".  Ben got
>his shots in also about the issue, too.  Thus the resulting io_barrier
>additions by Ben to the original ATA-driver transformation, can be extened
>to the Net-Drivers.  Oh and the slope is increasing fast now.

Actually, the iobarrier is a slightly different issue, it's not
related pci posting delays. (The only sane way to deal with them
is to do a read() from the same bus path).

The iobarrier's are a CPU-specific things that basically tell the
CPU not to re-order accesses around the barrier. This is necessary
for reads & writes to IO locations to be done in the order they
are written and not be speculated on CPUs that can do that sort
of things (PPC is one). By default, IO macros like {in,out}* and
{read,write}* do that implicitely (at least the PPC impl. of them
does the barriers) in addition to byteswap. But the raw_* versions
that we would need to avoid byteswap in implementing MMIO insw/outsw
also don't do the barriers, thus the macro I had to add.

Anyway, looks like we finally agree on getting {read,write}s{b,w,l}
in. I'll post a patch implementing them for PPC by tomorrow. That
will make things easier for IDE and possibly other drivers as well.

BTW. Jeff: the tulip patch don't really need those explicit
barriers from what I understand of it, you can probably merge the
patch without them.

Ben.