General porting question

General porting question

[Jacky Lam]

Dear all,

    Could anyone give me some notes/idea what should be take care to port an
x86 PCI card driver to powerpc walnut platform? I try several
cards(ethernet, audio) which run well on my PC by cross-compiling the
drivers. They can be recognized successfully, but none of them works. Walnut
seems don't be able to receive any interrupt from the cards.

    I guess there must be something about PCI implementation in PPC
different from x86. Is there any expert on PPC PCI can tell me what happen?
Or any related documentation can help?

    Thanks.

Best regards,
Jacky


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: General porting question

[Bret Indrelee]

On Thu, 9 Oct 2003, Jacky Lam wrote:
> Dear all,
>
>     Could anyone give me some notes/idea what should be take care to port an
> x86 PCI card driver to powerpc walnut platform? I try several
> cards(ethernet, audio) which run well on my PC by cross-compiling the
> drivers. They can be recognized successfully, but none of them works. Walnut
> seems don't be able to receive any interrupt from the cards.
>
>     I guess there must be something about PCI implementation in PPC
> different from x86. Is there any expert on PPC PCI can tell me what happen?
> Or any related documentation can help?

Best guess, the driver doesn't handle any of the endian issues.

PPC is a big-endian processor, x86 is little endian. Most likely you've
got data that needs to be byte swapped in order to work. Best way I know
of to find out if this is the case is using a PCI bus analyzer to
watch the PCI bus.

-Bret

--
Bret Indrelee                 QLogic Corporation
Bret.Indrelee@qlogic.com      6321 Bury Drive, St 13, Eden Prairie, MN 55346


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: General porting question

[Matt Porter]

On Thu, Oct 09, 2003 at 12:11:09PM +0800, Jacky Lam wrote:
>
> Dear all,
>
>     Could anyone give me some notes/idea what should be take care to port an
> x86 PCI card driver to powerpc walnut platform? I try several
> cards(ethernet, audio) which run well on my PC by cross-compiling the
> drivers. They can be recognized successfully, but none of them works. Walnut
> seems don't be able to receive any interrupt from the cards.
>
>     I guess there must be something about PCI implementation in PPC
> different from x86. Is there any expert on PPC PCI can tell me what happen?
> Or any related documentation can help?

Documentation/DMA-mapping.txt and IO-mapping.txt are of general
interest when dealing with DMA capable devices and address translation.
Unfortunately, the docs/APIs aren't yet complete with respect to some
platforms. On non cache coherent processors like PPC4xx/8xx, consistent
memory is allocated from vmalloc space.  The virt_to_*/*_to_virt family
of APIs is only valid for staticly mapped kernel system memory addresses
(KERNELBASE -> KERNELBASE+<size of sysmem>).

A quick inspection shows that es1371 does some things that won't work on
a 405.  It takes a pci_alloc_consistent buffer and does a virt_to_page.
That will be bogus since the buffer's virtual address is in vmalloc
space on 4xx.  In addition, it performs a virt_to_phys and then uses
remap_page_range on that...that will be bogus as well.  A bus_to_virt
on the pci_alloc_consistent buffer's dma_addr will work and then
virt_to_page() can be used on that virtual address.  In the same
fashion one could do a virt_to_phys(bus_to_virt(dma_addr) to get
the correct physical address to be passed to remap_page_range.

Someday we will get the DMA API to be completely cross architecture
so driver. It's much better than the situation in older kernels. :)

On the other hand, emu10k1 works out of the box...SB Live.

-Matt

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: General porting question

[Jacky Lam]

> A quick inspection shows that es1371 does some things that won't work on
> a 405.  It takes a pci_alloc_consistent buffer and does a virt_to_page.
> That will be bogus since the buffer's virtual address is in vmalloc
> space on 4xx.  In addition, it performs a virt_to_phys and then uses
> remap_page_range on that...that will be bogus as well.  A bus_to_virt
> on the pci_alloc_consistent buffer's dma_addr will work and then
> virt_to_page() can be used on that virtual address.  In the same
> fashion one could do a virt_to_phys(bus_to_virt(dma_addr) to get
> the correct physical address to be passed to remap_page_range.
>

    That means in short:

    pci_alloc_consistent() ->virt_to_page() should be replaced by
pci_alloc_consistent()->bus_to_virt()->virt_to_page().

    Similarly, for any PCI related memory space. Before using
virt_to_*/*_to_virt(), I should use bus_to_*/*_to_bus() on the argument
first.

    Is it right?

    Thanks for your help.

Best regards,
Jacky


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: General porting question

[Matt Porter]

On Thu, Oct 09, 2003 at 02:18:28PM +0800, Jacky Lam wrote:
>
> > A quick inspection shows that es1371 does some things that won't work on
> > a 405.  It takes a pci_alloc_consistent buffer and does a virt_to_page.
> > That will be bogus since the buffer's virtual address is in vmalloc
> > space on 4xx.  In addition, it performs a virt_to_phys and then uses
> > remap_page_range on that...that will be bogus as well.  A bus_to_virt
> > on the pci_alloc_consistent buffer's dma_addr will work and then
> > virt_to_page() can be used on that virtual address.  In the same
> > fashion one could do a virt_to_phys(bus_to_virt(dma_addr) to get
> > the correct physical address to be passed to remap_page_range.
> >
>
>     That means in short:
>
>     pci_alloc_consistent() ->virt_to_page() should be replaced by
> pci_alloc_consistent()->bus_to_virt()->virt_to_page().
>
>     Similarly, for any PCI related memory space. Before using
> virt_to_*/*_to_virt(), I should use bus_to_*/*_to_bus() on the argument
> first.

Almost.

You can not use virt_to_* on the address returned by
pci_alloc_consistent().

You need to hold on to the dma_addr_t returned, perform
a bus_to_virt(dma_addr_t) and then a virt_to_* will work
on that address.

However, I'm not sure of the "PCI related memory space" you
are referring to.  "PCI memory space" has a distinct meaning
and the virt_to_*/*_to_bus APIs don't work there.  Those APIs
are only valid for system memory that is accessible through
inbound transactions by a PCI bus master.

You can use virt_to_* on an address returned from kmalloc
or __get_free_pages.

-Matt

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: General porting question

[Jacky Lam]

> Almost.
>
> You can not use virt_to_* on the address returned by
> pci_alloc_consistent().

    Why?

    By the way, this problem will only affect the consistency of dma buffer.
In my case, it will only cause wrong output sound. But my card seems don't
consume the dma and doesn't give any interrupt in return. It's fine on PC
and I can receive interrupt if I write to the card's register to force an
interrupt. What other possible porting problem can be here? Really
strange....

>
> You need to hold on to the dma_addr_t returned, perform
> a bus_to_virt(dma_addr_t) and then a virt_to_* will work
> on that address.

    umm....then does   virt_to_phys(bus_to_virt(dma_addr_t))==dma_addr_t
                    or
                bus_to_virt(dma_addr_t)==pci_alloc_consistent() ?
>
> However, I'm not sure of the "PCI related memory space" you
> are referring to.  "PCI memory space" has a distinct meaning
> and the virt_to_*/*_to_bus APIs don't work there.  Those APIs
> are only valid for system memory that is accessible through
> inbound transactions by a PCI bus master.
>
> You can use virt_to_* on an address returned from kmalloc
> or __get_free_pages.

    I also don't understand....Could you or someone "invent" these things
kindly give some brief explaination here (especially for the reason how
consistent_alloc() is implemented)? I think it will benefit many PPC
beginner. Thanks very much.

Best regards,
Jacky


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: General porting question

[Matt Porter]

On Thu, Oct 09, 2003 at 11:36:07PM +0800, Jacky Lam wrote:
>
> > Almost.
> >
> > You can not use virt_to_* on the address returned by
> > pci_alloc_consistent().
>
>     Why?

Because virt_to_*() is only defined for staticly mapped kernel virtual
addresses...consistent_alloc() is not guaranteed to return a staticly
mapped kernel virtual address so you can't use virt_to_*().  It is not
a generic address translation API.

>     By the way, this problem will only affect the consistency of dma buffer.
> In my case, it will only cause wrong output sound. But my card seems don't
> consume the dma and doesn't give any interrupt in return. It's fine on PC
> and I can receive interrupt if I write to the card's register to force an
> interrupt. What other possible porting problem can be here? Really
> strange....

Endianness issues as another person pointed out.  You still have to
solve these address munging issues.

> > You need to hold on to the dma_addr_t returned, perform
> > a bus_to_virt(dma_addr_t) and then a virt_to_* will work
> > on that address.
>
>     umm....then does   virt_to_phys(bus_to_virt(dma_addr_t))==dma_addr_t

No, dma_addr_t is not necessary they same as the physical address.
Your platform may have physical addresses mapped 1:1 with bus (PCI)
addresses but that is not always the case (PReP, many peer-to-peer
PCI systems. etc).

>                     or
>                 bus_to_virt(dma_addr_t)==pci_alloc_consistent() ?

No, the virtual address you get by doing a bus_to_virt(dma_addr_t)
is != to the address that will be returned by pci_alloc_consistent()
(on 4xx/8xx).

> > However, I'm not sure of the "PCI related memory space" you
> > are referring to.  "PCI memory space" has a distinct meaning
> > and the virt_to_*/*_to_bus APIs don't work there.  Those APIs
> > are only valid for system memory that is accessible through
> > inbound transactions by a PCI bus master.
> >
> > You can use virt_to_* on an address returned from kmalloc
> > or __get_free_pages.
>
>     I also don't understand....Could you or someone "invent" these things
> kindly give some brief explaination here (especially for the reason how
> consistent_alloc() is implemented)? I think it will benefit many PPC
> beginner. Thanks very much.

*consistent_alloc() has a "special" implementation for 4xx/8xx because
they do not snoop memory access by external bus masters.  In order to
allocate "consistent" memory it is necessary to create a cache inhibited
mapping to system memory in the vmalloc area.  Virtual addresses in
the vmalloc area cannot be translated using the virt_to_* APIs.

There are a number of threads on this topic if you check the archives.

-Matt

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: General porting question

[Jacky Lam]

> > > You can not use virt_to_* on the address returned by
> > > pci_alloc_consistent().
> >
> >     Why?
>
> Because virt_to_*() is only defined for staticly mapped kernel virtual
> addresses...consistent_alloc() is not guaranteed to return a staticly
> mapped kernel virtual address so you can't use virt_to_*().  It is not
> a generic address translation API.
>
> >     By the way, this problem will only affect the consistency of dma
buffer.
> > In my case, it will only cause wrong output sound. But my card seems
don't
> > consume the dma and doesn't give any interrupt in return. It's fine on
PC
> > and I can receive interrupt if I write to the card's register to force
an
> > interrupt. What other possible porting problem can be here? Really
> > strange....
>
> Endianness issues as another person pointed out.  You still have to
> solve these address munging issues.

    Thanks. I will spend more time to understand those mapping stuff.....I
still cannot understand very clearly...

    Concerning endianness, I think the read/write to register of PCI card is
ok because it is done by inl()/outl() which already handled the endian
conversion. The remaining is the data written to dma buffer. I don't care it
now because the card seems don't start to consume the data.

    I look through the code many times. I can't find out any porting related
issues beside endianness and memory mapping that will make x86 and PPC
different. Any idea?

Best regards,
Jacky

P.S.: By the way, in embedded system, I know how much PCI memory are used.
Why don't we just give up some on board memory, say 1MB, and map "simply"
the memory with uncachable flag. Then this region of memory can be use by
PCI cards without need to take care all those mapping stuffs.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/