[RFC] I/O Access Abstractions

[RFC] I/O Access Abstractions

[David Howells]

In conjunction with David Woodhouse (dwmw2@redhat.com) and Arjan Van De Ven
(arjan@redhat.com), I've come up with a way to abstract I/O accesses in the
Linux kernel whilst trying to keep overheads minimal. These would be
particularly useful on many non-i386 platforms.

Any comments would be greatly appreciated.

The example code I've written can be downloaded from:

	ftp://infradead.org/pub/people/dwh/ioaccess.tar.bz2

Cheers,
David Howells
dhowells@redhat.com
_______________________________________________________________________________

			   I/O ACCESS ABSTRACTIONS
			   =======================

PURPOSES
========

 * Makes all peripheral input and output accesses look the same.

   - I/O ports
   - Memory-mapped I/O registers
   - PCI Configuration space
   - Device Specific registers

 * Can hide all the strange hoops that some architectures have to jump through
   to provide certain bus access services. For example:

   (1) The SH arch has no I/O port space in the same way that the i386 bus
       does. It instead maps a window in the physical memory space to PCI IO
       port accesses.

   (2) The AM33 arch also has no I/O port space, and likewise maps a window in
       memory space to PCI IO port addresses. Furthermore, the PCI _memory_
       space doesn't correspond on 1:1 basis with the CPU's memory space. At
       any one time, a 64Mb window is mapped from a fixed location in the CPU
       memory space into a mobile location on the PCI memory space. This can
       be moved by means of a control register on the host bridge.

 * Can hide the exact details of the layout of a devices register space as it
   varies from arch to arch. For instance, on the PC, a the 16550 compatible
   serial ports have 8 1-byte registers adjacent to each other in the I/O port
   space, whereas another arch may have the same 8 1-byte registers but at
   intervals of 4-bytes in memory space.

 * The operations by which a device can be accessed can be bound at runtime
   rather than at compile time without the need for conditional branches (this
   should simplify serial.c immensely).

 * Permit "iotrace" on specific resources by operation table substitution.

 * Provide a method of device emulation.

 * Potentially permit transparent byte swapping.


DISADVANTAGES
=============

 * Indirection. It will incur an overhead of a few extra cycles on the i386
   (for the call and return). This can be minimised by carefully crafting bits
   of inline assembly and carefully choosing what registers are used to pass
   what values. This may also be offset under certain circumstances by the
   removal of conditional branches (for example in serial.c).

   Also on an i386, the actual I/O instruction itself is going to take a
   comparatively long time anyway, given the speed differential between CPU
   and external buses.

   On other archs where the indirection exists anyway, there shouldn't be any
   overhead


IMPLEMENTATION
==============

 * The resource structure gains a pointer to a table of operations.

 * The table of resource operations includes 3 single-input functions, 3
   single-output functions, 3 string-input functions and 3 string-output
   functions. Each set of functions includes variation for byte, word and
   dword granularity.

 * Inline functions are provided on a per-arch basis that take a resource
   structure amongst their arguments, dereference it to pull out the
   appropriate function address, set up the registers and then call that
   function. For example:

	struct resource *csr;
	__u32 x = inputw(csr,0x10);
	outputw(cst,0x10,x|0x0020);

 * Macros are provided for generating and using calling convention translation
   stubs for making it possible to write operation functions in C (which if
   necessary will be assembly functions):

	__u16 __iocall pcnet32_inputw(struct resource *p, unsigned offset);
	__DECLARE_INPUT_CALLBACK(w,pcnet32_inputw);
	struct resource_ops pcnet32_ops = {
		__USE_INPUT_CALLBACK(b,pcnet32_inputb),
		__USE_INPUT_CALLBACK(w,pcnet32_inputw),
		__USE_INPUT_CALLBACK(l,pcnet32_inputl),
	};

   Note that on most archs (where there are enough registers) these macros
   will do nothing but plug the user-supplied function straight in.

 * There are three ops tables supplied for the i386:

   - I/O port accesses				[ioaccess-io.S]
   - memory-mapped I/O accesses			[ioaccess-mem.S]
   - PCI type 1 config space accesses		[ioaccess-pciconf1.S]

   Note that the PCI access example does not have the complete set of
   functions at the moment.


EXTENSIONS
==========

 * Driver-specific operations can be provided (for example CSR register access
   functions in the AMD PCnet32 driver).


ISSUES
======

Having discussed this with others the following points have been raised:

 * Some i386 support routines have been written in assembly and optimised to
   have the smallest latency possible. However:

   * These use a non-standard calling convention to call out to the backend
     functions - basically all the values to be passed are put in appropriate
     registers (including ESI/EDI), and some of these registers are clobbered
     or preserved in non-standard ways.

   * As a consequence, generic driver-supplied C functions require a small
     wrapper/thunk/stub to convert the calling convention back into the C one.

   * On an arch that passes sufficient arguments in registers with no
     particular purposes assigned to those registers, no stub will be
     required.

 * Opinion is divided as to whether an optimised assembly should be included
   in the first attempt. It may be better to use just standard C function
   pointers straight off.

 * If optimised assembly is used, it may be worth not immediately providing
   the ability to generate stubs for user-defined operations.

 * It may be worth adding mass register <-> memory transfer instructions
   (equivalent to memcpy).

 * It may be worth providing inline function or macros that, depending on the
   value of a global configuration setting, compile-time switch between doing,
   for example, inb and inputb or, say, between readb and inputb.

Re: [RFC] I/O Access Abstractions

[Alan Cox]

>    Also on an i386, the actual I/O instruction itself is going to take a
>    comparatively long time anyway, given the speed differential between CPU
>    and external buses.

PCI memory (and sometimes I/O) writes are posted, Since x86 memory writes are
also parallelisable instructions and since the CPU merely has to retire the
writes in order your stall basically doesnt exist.

High end devices tend to have you writing them to them and they DMA into your
RAM so its a concern

Re: [RFC] I/O Access Abstractions

[David Woodhouse]

alan@lxorguk.ukuu.org.uk said:
>  PCI memory (and sometimes I/O) writes are posted, Since x86 memory
> writes are also parallelisable instructions and since the CPU merely
> has to retire the writes in order your stall basically doesnt exist.

True. I can envisage a situation where the overhead of the function call is
noticeable. It would be interesting to see if it actually happens in real
life. If so, and if we set up the abstraction properly, we can keep
the accesses inline for those architectures where it's a win.

	#ifdef CONFIG_VIRTUAL_BUS
	#define resource_readb(res, x) res->readb(res, x)
	#else
	#define resource_readb(res, x) readb(x)
	#endif

Having per-resource I/O methods would help us to remove some of the
cruft which is accumulating in various non-x86 code. Note that the below is
the _core_ routines for _one_ board - I'm not even including the extra
indirection through the machine vector here....

static inline volatile __u16 *
port2adr(unsigned int port)
{
        if (port > 0x2000)
                return (volatile __u16 *) (PA_MRSHPC + (port - 0x2000));
        else if (port >= 0x1000)
                return (volatile __u16 *) (PA_83902 + (port << 1));
        else if (sh_pcic_io_start <= port && port <= sh_pcic_io_stop)
                return (volatile __u16 *) (sh_pcic_io_wbase + (port &~ 1));
        else
                return (volatile __u16 *) (PA_SUPERIO + (port << 1));
}

static inline int
shifted_port(unsigned long port)
{
        /* For IDE registers, value is not shifted */
        if ((0x1f0 <= port && port < 0x1f8) || port == 0x3f6)
                return 0;
        else
                return 1;
}

unsigned char se_inb(unsigned long port)
{
        unsigned long pciiobrSet;
        volatile unsigned long pciIoArea;
        unsigned char pciIoData;

        if (PXSEG(port))
                return *(volatile unsigned char *)port;
#if defined(CONFIG_CPU_SUBTYPE_SH7751) && defined(CONFIG_PCI)
        else if((port >= PCIBIOS_MIN_IO) && (port <= PCIBIOS_MIN_IO + SH7751_PCI_IO_SIZE)) {
                pciiobrSet  = (port & 0xFFFC0000);
                *PCIIOBR    = pciiobrSet;
                pciIoArea   = port & 0x0003FFFF;
                pciIoArea  += PCI_IO_AREA;
                pciIoData   = *((unsigned char*)pciIoArea);   
                return (unsigned char)pciIoData;
        }
#endif /* defined(CONFIG_CPU_SUBTYPE_SH7751) && defined(CONFIG_PCI) */
        else if (sh_pcic_io_start <= port && port <= sh_pcic_io_stop)
                return *(__u8 *) (sh_pcic_io_wbase + 0x40000 + port); 
        else if (shifted_port(port))
                return (*port2adr(port) >> 8); 
        else
                return (*port2adr(port))&0xff; 
}




--
dwmw2

Re: [RFC] I/O Access Abstractions

[Jes Sorensen]

>>>>> "David" == David Woodhouse <dwmw2@infradead.org> writes:

David> Having per-resource I/O methods would help us to remove some of
David> the cruft which is accumulating in various non-x86 code. Note
David> that the below is the _core_ routines for _one_ board - I'm not
David> even including the extra indirection through the machine vector
David> here....

Have you considered the method used by the 8390 Ethernet driver?
For each device, add a pointer to the registers and a register shift.

I really don't like hacing virtual access functions that makes memory
mapped I/O look the same as I/O operations. For memory mapped I/O you
want to be able to smart optimizations to reduce the access on the PCI
bus (or similar).

Jes

Re: [RFC] I/O Access Abstractions

[David Howells]

Jes Sorensen <jes@sunsite.dk> wrote:
> >>>>> "David" == David Woodhouse <dwmw2@infradead.org> writes:
> 
> David> Having per-resource I/O methods would help us to remove some of
> David> the cruft which is accumulating in various non-x86 code. Note
> David> that the below is the _core_ routines for _one_ board - I'm not
> David> even including the extra indirection through the machine vector
> David> here....
> 
> Have you considered the method used by the 8390 Ethernet driver?
> For each device, add a pointer to the registers and a register shift.

And also flags to specify which address space the I/O ports reside in, and
which how to adjust the host-PCI bridge to bring the appropriate bit of the
PCI address space into view through the CPU address space window. Don't laugh
- it happens.

> I really don't like hacing virtual access functions that makes memory
> mapped I/O look the same as I/O operations.

Why not? It makes drivers a simpler and more flexible if they can treat
different types of resource in the same way. serial.c is a really good example
of this:

| static _INLINE_ unsigned int serial_in(struct async_struct *info, int offset)
| {
|         switch (info->io_type) {
| #ifdef CONFIG_HUB6
|         case SERIAL_IO_HUB6:
|                 outb(info->hub6 - 1 + offset, info->port);
|                 return inb(info->port+1);
| #endif
|         case SERIAL_IO_MEM:
|                 return readb((unsigned long) info->iomem_base +
|                              (offset<<info->iomem_reg_shift));
| #ifdef CONFIG_SERIAL_GSC
|         case SERIAL_IO_GSC:
|                 return gsc_readb(info->iomem_base + offset);
| #endif
|         default:
|                 return inb(info->port + offset);
|         }
| }

The switch has to be performed at runtime - so you get at least one
conditional branch in your code, probably more. My proposal would replace the
conditional branch with a subroutine (which lacks the pipline stall).

Also a number of drivers (eg: ne2k-pci) have to be bound to compile time to
either I/O space or memory space. This would allow you to do it at runtime
without incurring conditional branching.

> For memory mapped I/O you want to be able to smart optimizations to reduce
> the access on the PCI bus (or similar).

I wouldn't have thought you'd want to reduce the number of accesses to the PCI
bus. If, for example, you did to writes to a memory-mapped I/O register, you
don't want the first to be optimised away.

David

Re: [RFC] I/O Access Abstractions

[Jes Sorensen]

>>>>> "David" == David Howells <dhowells@redhat.com> writes:

David> Jes Sorensen <jes@sunsite.dk> wrote:
>>  Have you considered the method used by the 8390 Ethernet driver?
>> For each device, add a pointer to the registers and a register
>> shift.

David> And also flags to specify which address space the I/O ports
David> reside in, and which how to adjust the host-PCI bridge to bring
David> the appropriate bit of the PCI address space into view through
David> the CPU address space window. Don't laugh - it happens.

Hmm, I am shocked ;-)

>> I really don't like hacing virtual access functions that makes
>> memory mapped I/O look the same as I/O operations.

David> Why not? It makes drivers a simpler and more flexible if they
David> can treat different types of resource in the same way. serial.c
David> is a really good example of this:

It will also degrade performance if you introduce all these weird
tests or function calls in the hot path.

David> The switch has to be performed at runtime - so you get at least
David> one conditional branch in your code, probably more. My proposal
David> would replace the conditional branch with a subroutine (which
David> lacks the pipline stall).

Actually on some architectures you'd prefer the conditional branch
over the subroutine call when you can do predication like on the ia64.

David> Also a number of drivers (eg: ne2k-pci) have to be bound to
David> compile time to either I/O space or memory space. This would
David> allow you to do it at runtime without incurring conditional
David> branching.

But it's going to cost for the ones who do not support this.

>> For memory mapped I/O you want to be able to smart optimizations to
>> reduce the access on the PCI bus (or similar).

David> I wouldn't have thought you'd want to reduce the number of
David> accesses to the PCI bus. If, for example, you did to writes to
David> a memory-mapped I/O register, you don't want the first to be
David> optimised away.

One good example I can think off is when you update a dma descriptor
in PCI shared memory space (__raw_writel()). In that case you want to
be able to use posted writes so all writes is done in one PCI write
transaction instead of individual ones.

Jes

Re: [RFC] I/O Access Abstractions

[David Woodhouse]

jes@sunsite.dk said:
>  But it's going to cost for the ones who do not support this. 

You don't need to make it out-of-line for all cases - or indeed in any case
where it isn't out-of-line already. Some architectures may have only IO
calls out-of-line (many already do). Some may have MMIO calls out-of-line
too - some already do that too.

It would just be nice to have a standard way of doing it, and in particular
it would be nice to pass the struct resource into the out-of-line functions
in the case where they _are_ out of line, so that the Iyou/O functions don't
have to play evil tricks with the numbers they're given to work out which bus
the caller wanted to talk to.

#ifdef OUT_OF_LINE_MMIO
#define res_readb(res, adr) (res->access_ops->readb(res, adr)
#else
#define res_readb(res, adr) readb(adr)
#endif

etc.

--
dwmw2

Re: [RFC] I/O Access Abstractions

[David Howells]

> #ifdef OUT_OF_LINE_MMIO
> #define res_readb(res, adr) (res->access_ops->readb(res, adr)
> #else
> #define res_readb(res, adr) readb(adr)
> #endif

I think the second #define should be:

	#define res_readb(res, adr) readb(res->start+adr)

for consistency.

David

Re: [RFC] I/O Access Abstractions

[David Woodhouse]

dhowells@redhat.com said:
>  I think the second #define should be:
> 	#define res_readb(res, adr) readb(res->start+adr)
> for consistency. 

You're right that it should be consistent. But it doesn't really matter
whether we pass an offset within the resource, or whether we continue to
pass the full 'bus address'. The driver doesn't even need to care - it just
adds the register offset to whatever opaque cookie it's given as the address
of that resource anyway.

That's really an orthogonal issue. The _important_ bit is that we pass the
resource to the I/O functions, so that in the case where they're
out-of-line, they don't need to play silly buggers with the numbers they're
given just to work out which bus they should be talking to.

--
dwmw2

Re: [RFC] I/O Access Abstractions

[Alan Cox]

> dhowells@redhat.com said:
> >  I think the second #define should be:
> > 	#define res_readb(res, adr) readb(res->start+adr)
> > for consistency. 
> 
> You're right that it should be consistent. But it doesn't really matter
> whether we pass an offset within the resource, or whether we continue to

The question I think being ignored here is. Why not leave things as is. The
multiple bus stuff is a port specific detail hidden behind readb() and friends.

On the HP PA32 its already hiding controller number encodings and generating
multiple cycles under spinlocks for PCI I/O space and the devices dont know
about it

Re: [RFC] I/O Access Abstractions

[David Woodhouse]

alan@lxorguk.ukuu.org.uk said:
>  The question I think being ignored here is. Why not leave things as
> is.

Because if we just pass in this one extra piece of information which is
normally already available in the driver, we can avoid a whole lot of ugly
cruft in the out-of-line functions by plugging in the correct out-of-line 
function to match the resource. 

> The multiple bus stuff is a port specific detail hidden behind
> readb() and friends. 

The alternative view is that the _single_ bus stuff is a port-specific
detail which has permeated all the drivers and forced the non-i386
architectures' I/O functions to have to try to work out which bus they're
talking to when the driver could have just passed that information to them.

--
dwmw2

Re: [RFC] I/O Access Abstractions

[Alan Cox]

> Because if we just pass in this one extra piece of information which is
> normally already available in the driver, we can avoid a whole lot of ugly
> cruft in the out-of-line functions by plugging in the correct out-of-line 
> function to match the resource. 

Case 1:
	You pass a single cookie to the readb code
	Odd platforms decode it

Case 2:
	You carry around bus number information all throughout
		each driver
	You keep putting it on/off the stack
	You keep it in structures
	You do complex generic locking for hotplug 'just in case'


I think I prefer case 1.

Re: [RFC] I/O Access Abstractions

[Russell King]

On Mon, Jul 02, 2001 at 05:56:56PM +0100, Alan Cox wrote:
> Case 1:
> 	You pass a single cookie to the readb code
> 	Odd platforms decode it

Last time I checked, ioremap didn't work for inb() and outb().

--
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: [RFC] I/O Access Abstractions

[Jeff Garzik]

Russell King wrote:
> 
> On Mon, Jul 02, 2001 at 05:56:56PM +0100, Alan Cox wrote:
> > Case 1:
> >       You pass a single cookie to the readb code
> >       Odd platforms decode it
> 
> Last time I checked, ioremap didn't work for inb() and outb().

It should :)

-- 
Jeff Garzik      | "I respect faith, but doubt is
Building 1024    |  what gives you an education."
MandrakeSoft     |           -- Wilson Mizner

Re: [RFC] I/O Access Abstractions

[Alan Cox]

> > >       You pass a single cookie to the readb code
> > >       Odd platforms decode it
> > 
> > Last time I checked, ioremap didn't work for inb() and outb().
> 
> It should :)

it doesnt need to.

pci_find_device returns the io address and can return a cookie, ditto 
isapnp etc

Re: [RFC] I/O Access Abstractions

[Benjamin Herrenschmidt]

>> > Last time I checked, ioremap didn't work for inb() and outb().
>> 
>> It should :)
>
>it doesnt need to.
>
>pci_find_device returns the io address and can return a cookie, ditto 
>isapnp etc

Yes, but doing that require 2 annoying things:

 - Parsing of this cookie on each inx/outx access, which can
take a bit of time (typically looking up the host bridge)

 - On machines with PIO mapped in CPU mem space and several
(or large) IO regions, they must all be mapped all the time,
which is a waste of kernel virtual space.

Why not, at least for 2.5, define a kind of pioremap that
would be the equivalent of ioremap for PIO ?

In fact, I'd rather have all this abstracted in a

ioremap_resource(struct resource *, int flags)
iounmap_resource(struct resource *)

("flags" is just an idea that could be used to pass things
like specific caching attributes, or whatever makes sense to
a given arch).

The distinction between inx/oux & readx/writex would still
make sense at least for x86.

Ben.

Re: [RFC] I/O Access Abstractions

[Benjamin Herrenschmidt]

>Last time I checked, ioremap didn't work for inb() and outb().

ioremap itself cannot work for inb/outb as they are different
address spaces with potentially overlapping addresses, I don't
see how a single function would handle both... except if we
pass it a struct resource instead of the address.

Ben.

Re: [RFC] I/O Access Abstractions

[Alan Cox]

>  - Parsing of this cookie on each inx/outx access, which can
> take a bit of time (typically looking up the host bridge)

It depends on the implementation obviously, but its typically something like

	take lock
	writew(port&0xFFFF, port&0xFFFF0000);
	writew(data, port&0xFFFF0000+1);
	drop lock

Assuming you can drop the bridges on 64K boundaries in pci mem space, or
one extra deref and a register load if not.

Can you give me an idea of what sort of cookie decoding a PPC/PMac would need
and why - Im working off things like pa-risc so I dont have a full picture.

Re: [RFC] I/O Access Abstractions

[Benjamin Herrenschmidt]

>
>Can you give me an idea of what sort of cookie decoding a PPC/PMac would need
>and why - Im working off things like pa-risc so I dont have a full picture.

Each domain provide an IO space (size depends on the bridge, recent Apple
UniNorth hosts have 16Mb per domain). 

That IO space can be in any location (depends on the box, bridge config,
..), so basically, we must assume that each host bridge can have it's IO
space anywhere in CPU mem space.

Currently, we store the physical address of those in our pci_controller
structure, and ioremap all of them. One is picked up as the "ISA" io base
(for VGA and such things as legacy devices on non-pmac PPCs). That
isa_io_base is used as an offset to inx/outx, and all PCI IO_RESOURCES
are fixed up to be their real virtual address offset'ed with isa_io_base.
(A bit weird but works and we have only an addition in inx/outx).

I'm more concerned about having all that space mapped permanently in
kernel virtual space. I'd prefer mapping on-demand, and that would
require a specific ioremap for IOs.

Ben.

Re: [RFC] I/O Access Abstractions

[Jeff Garzik]

Alan Cox wrote:
> 
> > > >       You pass a single cookie to the readb code
> > > >       Odd platforms decode it
> > >
> > > Last time I checked, ioremap didn't work for inb() and outb().
> >
> > It should :)
> 
> it doesnt need to.
> 
> pci_find_device returns the io address and can return a cookie, ditto
> isapnp etc

Is the idea here to mitigate the amount of driver code changes, or
something else?

If you are sticking a cookie in there behind the scenes, why go ahead
and use ioremap?

We -already- have a system which does remapping and returns cookies and
such for PCI mem regions.  Why not use it for I/O regions too?

	Jeff


-- 
Jeff Garzik      | "I respect faith, but doubt is
Building 1024    |  what gives you an education."
MandrakeSoft     |           -- Wilson Mizner

Re: [RFC] I/O Access Abstractions

[David Howells]

> The question I think being ignored here is. Why not leave things as is. The
> multiple bus stuff is a port specific detail hidden behind readb() and
> friends.

This isn't so much for the case where the address generation is done by a
simple addition. That could be optimised away by the compiler with an entirely
inline function (as per David Woodhouse's suggestion).

It's far more important for non-x86 platforms which only have a single address
space and have to fold multiple external address spaces into it.

For example, one board I've got doesn't allow you to do a straight
memory-mapped I/O access to your PCI device directly, but have to reposition a
window in the CPU's memory space over part of the PCI memory space first, and
then hold a spinlock whilst you do it.

David

Re: [RFC] I/O Access Abstractions

[Jeff Garzik]

David Howells wrote:
> For example, one board I've got doesn't allow you to do a straight
> memory-mapped I/O access to your PCI device directly, but have to reposition a
> window in the CPU's memory space over part of the PCI memory space first, and
> then hold a spinlock whilst you do it.

Yuck.  Does that wind up making MMIO slower than PIO, on this board?

-- 
Jeff Garzik      | "I respect faith, but doubt is
Building 1024    |  what gives you an education."
MandrakeSoft     |           -- Wilson Mizner

Re: [RFC] I/O Access Abstractions

[David Howells]

> Case 1:
> 	You pass a single cookie to the readb code
> 	Odd platforms decode it

As opposed to passing a cookie (struct resource) and an offset, and letting
the compiler do the addition it'd do anyway or eliminate the cookie directly
on platforms where this is suitable.

> Case 2:
> 	You carry around bus number information all throughout
> 		each driver

Eh? Who said anything about bus number info? Just the information in the
resource structure.

> 	You keep putting it on/off the stack

Why should I want to do that? You've got to keep the base address of your
resource space somewhere anyway, so you could just replace it with a pointer
to the resource struct (which you've already got). Plus, I can pass this in a
register to any behind the scenes function.

In my example code, in the really simple cases (most of them), there were no
pushes and pops.

> 	You keep it in structures

Doesn't everyone? Apart from those that use global variables, I suppose, but
surely they're limited in reusability.

> 	You do complex generic locking for hotplug 'just in case'

Eh? No I wasn't, but under some circumstances one might have to do that
anyway, and so the out-of-line functions may be the best place to do that.

David

Re: [RFC] I/O Access Abstractions

[David Howells]

> David Howells wrote:
> > For example, one board I've got doesn't allow you to do a straight
> > memory-mapped I/O access to your PCI device directly, but have to
> > reposition a window in the CPU's memory space over part of the PCI memory
> > space first, and then hold a spinlock whilst you do it.
> 
> Yuck.  Does that wind up making MMIO slower than PIO, on this board?

The mapping is not symmetrical (things on the PCI bus can see more of the CPU
bus at any one time than the reverse, so DMA isn't a problem), and at the
moment I have only one device on it that I'm actually using (the ethernet
chipset).

David

Re: [RFC] I/O Access Abstractions

[David Howells]

Jeff Garzik <jgarzik@mandrakesoft.com> wrote:
> Russell King wrote:
> > 
> > On Mon, Jul 02, 2001 at 05:56:56PM +0100, Alan Cox wrote:
> > > Case 1:
> > >       You pass a single cookie to the readb code
> > >       Odd platforms decode it
> > 
> > Last time I checked, ioremap didn't work for inb() and outb().
> 
> It should :)

Surely it shouldn't... ioremap() is for mapping "memory-mapped I/O" resources
into the kernel's virtual memory scheme (at least on the i386 arch). There's
no way to tell the CPU/MMU that a particular pages should assert the IO access
pin rather than memory access pin (or however it is done externally).

David

Re: [RFC] I/O Access Abstractions

[Jeff Garzik]

David Howells wrote:
> 
> Jeff Garzik <jgarzik@mandrakesoft.com> wrote:
> > Russell King wrote:
> > >
> > > On Mon, Jul 02, 2001 at 05:56:56PM +0100, Alan Cox wrote:
> > > > Case 1:
> > > >       You pass a single cookie to the readb code
> > > >       Odd platforms decode it
> > >
> > > Last time I checked, ioremap didn't work for inb() and outb().
> >
> > It should :)
> 
> Surely it shouldn't... ioremap() is for mapping "memory-mapped I/O" resources
> into the kernel's virtual memory scheme (at least on the i386 arch). There's
> no way to tell the CPU/MMU that a particular pages should assert the IO access
> pin rather than memory access pin (or however it is done externally).

The "at least on the i386 arch" part is the key caveat.  On PPC AFAIK,
PIO is remapped and treated very similarly to MMIO.  ioremap on x86, for
PIO, could probably be a no-op, simply returning the same address it was
given.  For other arches which want to do more complex mappings, ioremap
is IMHO the perfect part of the API for the job.

Basically I don't understand the following train of thought:

* We needed to remap MMIO, therefore ioremap was created.
* Now, we need to remap PIO too [on some arches].  Let's hide the
remapping in arch-specific code.

That's an understandable train of thought from an
implement-it-now-in-2.4 standpoint, but not from a
2.5-design-something-better standpoint.

	Jeff


-- 
Jeff Garzik      | "I respect faith, but doubt is
Building 1024    |  what gives you an education."
MandrakeSoft     |           -- Wilson Mizner

Re: [RFC] I/O Access Abstractions

[Jeff Garzik]

I also point out that using ioremap for PIO adds flexibility while
keeping most drivers relatively unchanged.  Everyone uses a base address
anyway, so whether its obtained directly (address from PCI BAR) or
indirectly (via ioremap), you already store it and use it.

Further, code lacking ioremap for PIO (100% of PIO code, at present)
does not require a flag day.  Drivers can be transitioned as foreign
arches start supporting ioremap for PIO... if ioremap is no-op on x86,
drivers continue to work on x86 before and after the update.  Assuming a
stored not hardcoded base address (common case), the only change to a
driver is in probe and remove, nowhere else.

-- 
Jeff Garzik      | "I respect faith, but doubt is
Building 1024    |  what gives you an education."
MandrakeSoft     |           -- Wilson Mizner

Re: [RFC] I/O Access Abstractions

[David Howells]

Jeff Garzik <jgarzik@mandrakesoft.com> wrote:
> Alan Cox wrote:
> > 
> > > > >       You pass a single cookie to the readb code
> > > > >       Odd platforms decode it
> > > >
> > > > Last time I checked, ioremap didn't work for inb() and outb().
> > >
> > > It should :)
> > 
> > it doesnt need to.
> > 
> > pci_find_device returns the io address and can return a cookie, ditto
> > isapnp etc
> 
> Is the idea here to mitigate the amount of driver code changes, or
> something else?

A number of things:

 * It ought to be easier to deal with having a number of ways of accessing
   resources if you just had an ops table with each struct resource. This
   would allow the bus driver to provide each resource of each device on that
   bus with a set of operations for accessing registers within that resource,
   no matter what the actual mechanism for doing so. For example:

	* direct I/O port accesses
	* direct memory-mapped I/O accesses
	* address port/data port accesses (eg: pcnet32, PCI config, etc.)
	* hide sparse I/O port accesses (eg: non-adjacent 16550 registers)
	* small-window CPU->PCI bus mapping
	* emulation of I/O port range through memory window

   Note that any number of these things can exist on the same architecture, or
   even the same platform (the GEODE GX1 & SC1200 being excellent examples).

 * It should make drivers easier to write: they don't have to worry about
   whether a resource refers to memory or to I/O or to something more exotic.

 * It makes some drivers more flexible. For example, the ne2k-pci driver has
   to be set at _compile_ time to use _either_ I/O ports _or_ memory. It'd
   make Linux installation more better if _both_ were supported.

 * It'd allow some drivers to be massively cleaned up (serial.c).

 * Permit transparent byte-swapping.

There are some other minor benefits too, but they're far less important (such
as I/O access tracing).

> If you are sticking a cookie in there behind the scenes, why go ahead
> and use ioremap?
> 
> We -already- have a system which does remapping and returns cookies and
> such for PCI mem regions.  Why not use it for I/O regions too?

You can't remap I/O ports into memory space on the i386. You can't easily
remap PCI devices if the all PCI devices peer through a small sliding window
in the CPU address space.

David

Re: [RFC] I/O Access Abstractions

[David Howells]

> I also point out that using ioremap for PIO adds flexibility while
> keeping most drivers relatively unchanged.  Everyone uses a base address
> anyway, so whether its obtained directly (address from PCI BAR) or
> indirectly (via ioremap), you already store it and use it.

I see what you're getting at at last:-) I didn't quite pick up on the fact
that you'd still have to go through readb/writeb and their ilk to access the
code.

Of course, however, this still requires cookie decoding to be done in readb
and writeb (even on the i386). So why not use resource struct?

David

Re: [RFC] I/O Access Abstractions

[Jeff Garzik]

David Howells wrote:
> Of course, however, this still requires cookie decoding to be done in readb
> and writeb (even on the i386). So why not use resource struct?

IMHO that makes the operation too heavyweight on architectures where
that level of abstraction is not needed.

-- 
Jeff Garzik      | "I respect faith, but doubt is
Building 1024    |  what gives you an education."
MandrakeSoft     |           -- Wilson Mizner

Re: [RFC] I/O Access Abstractions

[Alan Cox]

> For example, one board I've got doesn't allow you to do a straight
> memory-mapped I/O access to your PCI device directly, but have to reposition a
> window in the CPU's memory space over part of the PCI memory space first, and
> then hold a spinlock whilst you do it.

What does this prove. PA-RISC has this problem in reverse for I/O cycle access
to PCI slots on hppa1.1 at least. Cookies work _fine_

And by the time you are taking a spinlock who cares about the add, you can do
that while the bus transactions for the atomic op are completing

On the other hand each call, each push of resource * pointers costs real clocks
on x86

Re: [RFC] I/O Access Abstractions

[Alan Cox]

> I'm more concerned about having all that space mapped permanently in
> kernel virtual space. I'd prefer mapping on-demand, and that would
> require a specific ioremap for IOs.

I have no problem with the idea of a function to indicate which I/O maps you
are and are not using. But passing resource structs around is way too heavy

Alan

Re: [RFC] I/O Access Abstractions

[Benjamin Herrenschmidt]

>> I'm more concerned about having all that space mapped permanently in
>> kernel virtual space. I'd prefer mapping on-demand, and that would
>> require a specific ioremap for IOs.
>
>I have no problem with the idea of a function to indicate which I/O maps you
>are and are not using. But passing resource structs around is way too heavy

Too heavy for inx/outx, I agree, but why too heavy to ioremap ? That would
make a clean abstract implementation, with a semantic like "prepare this
resource for use by the driver". A kind of generic ioremap for IOs, resources,
and whatever another bus type may want to define, returning a token that is
to be passed to readx/writex in all cases but PIO, where it's passed to
inx/outx. That souds to me like the most flexible mecanism, and the small 
bit of overhead of passing the resource pointer is done _once_, usually
at driver init.

Something like

   iomap_resource(struct resource *);
   iounmap_resource(struct resource *);

Eventually, we can have it more fine grained in case where the driver don't
need the entire resource (maybe useful for framebuffers exporting very large
double-endian apertures where only one half is needed).

   iomap_resource(struct resource *, unsigned long offset, unsigned long
size);
   iounmap_resource(struct resource *, unsigned long offset, unsigned
long size);
  
The implementation would just call ioremap/iounmap for memory type resources,
and the identity for IO resources on x86. Other archs can then play whatever
tricks, like placing cookies in there.

One thing I have in mind here is the ability for things like embedded that
can have weird bus types, to have additional flags in the resources taken
into account by iomap/unmap to locate the proper bus, or build the proper
cookie that will be used by inx/outx, or define some access attributes
depending on other resource flags (write combine ?). 

Ben.

Re: [RFC] I/O Access Abstractions

[Geert Uytterhoeven]

On Tue, 3 Jul 2001, David Howells wrote:
> > The question I think being ignored here is. Why not leave things as is. The
> > multiple bus stuff is a port specific detail hidden behind readb() and
> > friends.
> 
> This isn't so much for the case where the address generation is done by a
> simple addition. That could be optimised away by the compiler with an entirely
> inline function (as per David Woodhouse's suggestion).
> 
> It's far more important for non-x86 platforms which only have a single address
> space and have to fold multiple external address spaces into it.
> 
> For example, one board I've got doesn't allow you to do a straight
> memory-mapped I/O access to your PCI device directly, but have to reposition a
> window in the CPU's memory space over part of the PCI memory space first, and
> then hold a spinlock whilst you do it.

This is a common practice on NEC PCI host bridges: usually you have 2 `windows'
to the PCI bus only, so you can have direct access to only two of PCI memory,
PCI I/O and PCI config spaces at the same time. If you need access to the
third, you have to reconfigure the windows.  Usually you configure the windows
to have direct access to PCI memory and PCI I/O spaces. So PCI config space
takes the hit. If you have only one window, YMMV.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: [RFC] I/O Access Abstractions

[Geert Uytterhoeven]

On Tue, 3 Jul 2001, David Howells wrote:
>  * It should make drivers easier to write: they don't have to worry about
>    whether a resource refers to memory or to I/O or to something more exotic.
> 
>  * It makes some drivers more flexible. For example, the ne2k-pci driver has
>    to be set at _compile_ time to use _either_ I/O ports _or_ memory. It'd
>    make Linux installation more better if _both_ were supported.
> 
>  * It'd allow some drivers to be massively cleaned up (serial.c).

And the IDE driver.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: [RFC] I/O Access Abstractions

[David Howells]

Russell King <rmk@arm.linux.org.uk> wrote:
> They _ARE_ different, because people connect these chips in many different
> ways.  For example:

Also hence the mess in serial.c. On the board I'm currently dealing with, the
PC16550 chip is connected to the memory space with registers at 4-byte
intervals because the chip is an 8-bit chip connected to a 32-bit data bus
(and so ignored addr lines A0 and A1). Whereas on the PC, various serial
register sets appear generally in I/O port space at 1-byte intervals from some
base address.

This could be greatly simplified with the method I'm proposing. The resource
access function could make the address space for that device to be eight
consecutive registers regardless of the reality.

David