RE: understanding an IO question

RE: understanding an IO question

[Charles Krinke]

Thanks Kumar. Let me gnaw on these notions over the next few days and I
will probably come back and ask another question or two, but now I have
enough meat to go look into what is happening in the Linux
initialization.

Charles

Re: understanding an IO question

[Kumar Gala]

On Feb 23, 2007, at 8:16 AM, Charles Krinke wrote:

>
> Dear Kumar:
>
> Thank you very much for your advice on addressing our PCI device. It
> turned out that the OutboundTranslation windows were the key.
>
> In uboot, I setup two windows to allow accesses to 32bit cpu space
> 0x8400_000 & 0x8800_0000 to be translated to 64bit DAC (Dual Address
> Cycles) at 0x3_0000_0000 & 0x1_0000_0000 respectively, and both the  
> 8241
> and 8541 do this quite well.
>
> Once the potar, powbar and other outbound translation address windows
> were reset to the same values in Linux that they had in uboot,  
> outbound
> transactions work fine and address the registers and memory in our  
> ASIC.

Good to hear.

> Next, I need to study the difference between the DMA engines in the  
> 8241
> and 8541 to accomplish a similar mission. This one may be a bit  
> trickier
> and you advice would be greatly appreciated.
>
> The 8241 DMA engine understands 64bit DMA  in that it has a high  
> source
> & destination address register for the upper 32bits of a 64bit  
> address.
> Unfortunately, the 8541 does not. It only supports 32bit DMA.

This is doable, however I haven't done it myself.

> So, my assumption is that there is some way to use a different  
> outbound
> address translation window and emulate 64bit DMA transactions using  
> the
> 32bit DMA engine in the 8541.

Yep. If you have the POTARs setup to access the 64 bit PCI address  
space you want than you just DMA to the 'internal' address that the  
POTAR is pointing to.

I suggest a simple direct mode test first to make sure everything  
appears to be working properly (maybe via hand in u-boot with memory  
write commands)

> I don't know if you have studied this part at all, but if you or  
> anyone
> would be willing to offer a few hints, I would be appreciative.

- k

RE: understanding an IO question

[Charles Krinke]

Dear Kumar:

Thank you very much for your advice on addressing our PCI device. It
turned out that the OutboundTranslation windows were the key.

In uboot, I setup two windows to allow accesses to 32bit cpu space
0x8400_000 & 0x8800_0000 to be translated to 64bit DAC (Dual Address
Cycles) at 0x3_0000_0000 & 0x1_0000_0000 respectively, and both the 8241
and 8541 do this quite well.

Once the potar, powbar and other outbound translation address windows
were reset to the same values in Linux that they had in uboot, outbound
transactions work fine and address the registers and memory in our ASIC.

Next, I need to study the difference between the DMA engines in the 8241
and 8541 to accomplish a similar mission. This one may be a bit trickier
and you advice would be greatly appreciated.

The 8241 DMA engine understands 64bit DMA  in that it has a high source
& destination address register for the upper 32bits of a 64bit address.
Unfortunately, the 8541 does not. It only supports 32bit DMA.

So, my assumption is that there is some way to use a different outbound
address translation window and emulate 64bit DMA transactions using the
32bit DMA engine in the 8541.

I don't know if you have studied this part at all, but if you or anyone
would be willing to offer a few hints, I would be appreciative.

Charles

Re: understanding an IO question

[Kumar Gala]

> Dear Kumar:
>
> I hope all is well with you and I appreciate your taking time to  
> have a
> mini-dialog with me today.

No problem.  If you're ever in Austin, buy me a beer ;)
> Here's the deal. First of all, we are using the linux-2.6.17.11 code
> base from kernel.org, which was the latest one when this project
> started. We have validated that our driver, with no modifications,  
> works
> fine on our existing production 8241 board with this kernel. This  
> driver
> does use ioremap_nocache(), request_region() and other common
> driver-type calls, so I suspect the issue is the kernel is not setup
> quite right for the driver, not that the driver needs modification for
> the kernel.
>
> Starting from that premise, I know that we depend on outbound address
> translation windows and any necessary TLB setup to get access to  
> the CPU
> address region around 0x8400_0000.
>
> The reason for the outbound address translation windows has to do with
> the fact that we use a DAC (Dual Address Cycle) to a 64bit PCI address
> at 0x1_0000_0000 or 0x3_0000_0000 depending on whether we are
> writing/reading to the window at 0x8800_0000 or 0x8400_0000.

This does all seem odd.  The fact that it works fine from u-boot but  
not the kernel gets me.

> One thing that puzzles me is that a 'cat /proc/iomem' shows a PCI  
> window
> is setup in Linux, like this
>
>>> cat /proc/iomem
>>> 80000000-9fffffff : PCI1 host bridge
>
> I don't see anywhere in the Linux-2.6.17.11 code base where this  
> window
> is setup. Perhaps you could point me towards the section of code where
> this is done.

This is done in arch/ppc/syslib/ppc85xx_setup.c in mpc85xx_setup_hose 
().  And now that I look at it I think I was lying before about not  
setting up outbound windows.  (I'm thinking newer kernels).

Maybe something in mpc85xx_setup_hose() is causing you issues.  I'd  
suggest dumping the pci law* from u-boot and doing the same in the  
kernel and make sure they match.

> On the ioremap()/ioremap_nocache() issue. I did look at the source in
> the kernel for ioremp_nocache() and it is #defined to be ioremap(),  
> so I
> think it is safe to assume they are the same.

Ok, just thinking of what might be going on.

- k

RE: understanding an IO question

[Charles Krinke]

You have to use ioremap() to get TLB and page mappings for any IO.  =20
We setup some initial kernel mappings to allow access to system =20
memory but thats it.  Everything else should go through some mapping =20
function.

> 2. "Are Outbound Address Translation windows reset by the arch/ppc/...
> Linux initialization and if so, where. If they are reset, where is the
> best place to put them back to the state I had set them up in uboot.

Nope, currently the address translation windows at the system level =20
are not touched by arch/ppc (unless done in your own code).

Have you tried plain old ioremap() [not _nocached] ?

Dear Kumar:

I hope all is well with you and I appreciate your taking time to have a
mini-dialog with me today.

Here's the deal. First of all, we are using the linux-2.6.17.11 code
base from kernel.org, which was the latest one when this project
started. We have validated that our driver, with no modifications, works
fine on our existing production 8241 board with this kernel. This driver
does use ioremap_nocache(), request_region() and other common
driver-type calls, so I suspect the issue is the kernel is not setup
quite right for the driver, not that the driver needs modification for
the kernel.

Starting from that premise, I know that we depend on outbound address
translation windows and any necessary TLB setup to get access to the CPU
address region around 0x8400_0000.=20

The reason for the outbound address translation windows has to do with
the fact that we use a DAC (Dual Address Cycle) to a 64bit PCI address
at 0x1_0000_0000 or 0x3_0000_0000 depending on whether we are
writing/reading to the window at 0x8800_0000 or 0x8400_0000.

One thing that puzzles me is that a 'cat /proc/iomem' shows a PCI window
is setup in Linux, like this

>> cat /proc/iomem
>> 80000000-9fffffff : PCI1 host bridge

I don't see anywhere in the Linux-2.6.17.11 code base where this window
is setup. Perhaps you could point me towards the section of code where
this is done.

On the ioremap()/ioremap_nocache() issue. I did look at the source in
the kernel for ioremp_nocache() and it is #defined to be ioremap(), so I
think it is safe to assume they are the same.

Charles

Re: understanding an IO question

[Kumar Gala]

On Feb 21, 2007, at 8:14 AM, Charles Krinke wrote:

> I apologize for not making myself clearer. This is a kernel driver
> issue, not a user space issue. The issue is addressing the  
> registers in
> a PCI device in its driver.
>
> We have a functioning code base that works with Linux 2.6.17.11 on our
> 8241 platform. We have built a new board using the 8541. Although the
> registers are addressable from uboot and can be read/written, once  
> Linux
> boots, the registers cannot be addressed.
>
> I suspect this to be in either TLB1 setup or Outbound Address
> Translation windows.
>
> So,... my questions really are
>
> 1. "Where are the TLB's setup in the 85xx chip during Linux
> initialization for devices that may be using PCI memory in the
> 0x8000_0000 - 0x9fff_ffff address space region?". I can see that some
> reset of TLB's occurs in head_fsl_booke.S and perhaps that is part of
> the answer. The other part may be in where TLB's are then setup again
> after they are reset in Linux.

You have to use ioremap() to get TLB and page mappings for any IO.   
We setup some initial kernel mappings to allow access to system  
memory but thats it.  Everything else should go through some mapping  
function.

> 2. "Are Outbound Address Translation windows reset by the arch/ppc/...
> Linux initialization and if so, where. If they are reset, where is the
> best place to put them back to the state I had set them up in uboot.

Nope, currently the address translation windows at the system level  
are not touched by arch/ppc (unless done in your own code).

Have you tried plain old ioremap() [not _nocached] ?

- k

>
> Charles
>
>
>
> -----Original Message-----
> From: Ami Levi Waves [mailto:ami@waves.com]
> Sent: Wednesday, February 21, 2007 12:51 AM
> To: Charles Krinke; linuxppc-embedded@ozlabs.org
> Subject: RE: understanding an IO question
>
> The way we do it:
>
> Fisrt open a memory device:
>
> 	Handle = open("/dev/mem", O_RDWR);
>
> Than map the address using:
>
> 	mmap(NULL, nAdress, PORT_READ | PROT_WRITE, MAP_SHARED, Handle,
> your
> base address);
>
> mmap returns a virtual pointer, pointing to your HW.
>
> Ami
>
>> -----Original Message-----
>> From: linuxppc-embedded-bounces+ami=waves.com@ozlabs.org
> [mailto:linuxppc-
>> embedded-bounces+ami=waves.com@ozlabs.org] On Behalf Of Charles  
>> Krinke
>> Sent: Wednesday, February 21, 2007 2:14 AM
>> To: linuxppc-embedded@ozlabs.org
>> Subject: understanding an IO question
>>
>> I have a board using an 8541 which is based on the mpc85xx_cds  
>> bsp. It
>> has one PCI device connected to the 8541, an ASIC we make that  
>> maps in
>> resources for its registers at 0x8400_0000 - 0x8400_0038.
>>
>> The driver for our ASIC works in an 8241 design and I am trying to  
>> get
>> its register access working in the 8541 and need a little additional
>> understanding.
>>
>> I can see in cat /proc/iomem where we have memory apparently  
>> allocated
>> for our registers, like this.
>>
>> cat /proc/iomem
>> 80000000-9fffffff : PCI1 host bridge
>>   84000000-84000038 : ichar
>>   88000000-8fffffff : ichar
>> e0003000-e00030ff : fsl-i2c.1
>> e0004500-e0004507 : serial
>>
>> In our driver, I do an ioremap_nocache with 0x8400_0000 as an  
>> argument
>> to the ioremap and am returned a virtual address of the form
>> 0xc905_8000. Unfortunately, I cannot seem to access my registers as
> they
>> all return zero when accessed with the iowrite32/ioread32 helper
>> functions.
>>
>> A little further. We do use the Outbound Address Translation window
>> feature in the 8541 to get to our PCI device and I do set this up in
>> uboot and I can read and write registers on our ASIC from uboot. Its
>> just in the Linux driver that I seem to be having trouble.
>>
>> So,... I am hoping someone could give me a couple of pointers on  
>> where
>> to look next for the missing piece of why I cannot access the
> registers
>> on our ASIC from Linux whereas things were working fine from uboot
>> before booting Linux.
>>
>> Charles
>> _______________________________________________
>> Linuxppc-embedded mailing list
>> Linuxppc-embedded@ozlabs.org
>> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded

Re: understanding an IO question

[Florian Boelstler]

Charles Krinke schrieb:
> I apologize for not making myself clearer. This is a kernel driver
> issue, not a user space issue. The issue is addressing the registers in
> a PCI device in its driver.
> 
> We have a functioning code base that works with Linux 2.6.17.11 on our
> 8241 platform. We have built a new board using the 8541. Although the
> registers are addressable from uboot and can be read/written, once Linux
> boots, the registers cannot be addressed.
> 
> I suspect this to be in either TLB1 setup or Outbound Address
> Translation windows.
> 
> So,... my questions really are 
> 2. "Are Outbound Address Translation windows reset by the arch/ppc/...
> Linux initialization and if so, where. If they are reset, where is the
> best place to put them back to the state I had set them up in uboot.

Well may be Kumar is the one to answer that at best.
However I am working on a solution, which uses MPC8540/MPC8541 and I 
know for sure that POWBARs and friends aren't reset during Linux boot 
*when PCI is completely disabled* in kernel config.

You may check if the appropriate set of POTAR, POWBAR and POWAR 
registers are still configured after boot.
POTAR reflects your external device address, POWBAR your local one (e.g. 
0x84000000, but only the 20 most-significant bits).
POWAR must be set to something like 0x80044013 (for a 1MB window).
Also a set of LAWBAR/LAWAR must appropriately match these settings.

See "Reference Manual" for details.

Good luck,

   Florian

RE: understanding an IO question

[Charles Krinke]

I apologize for not making myself clearer. This is a kernel driver
issue, not a user space issue. The issue is addressing the registers in
a PCI device in its driver.

We have a functioning code base that works with Linux 2.6.17.11 on our
8241 platform. We have built a new board using the 8541. Although the
registers are addressable from uboot and can be read/written, once Linux
boots, the registers cannot be addressed.

I suspect this to be in either TLB1 setup or Outbound Address
Translation windows.

So,... my questions really are=20

1. "Where are the TLB's setup in the 85xx chip during Linux
initialization for devices that may be using PCI memory in the
0x8000_0000 - 0x9fff_ffff address space region?". I can see that some
reset of TLB's occurs in head_fsl_booke.S and perhaps that is part of
the answer. The other part may be in where TLB's are then setup again
after they are reset in Linux.

2. "Are Outbound Address Translation windows reset by the arch/ppc/...
Linux initialization and if so, where. If they are reset, where is the
best place to put them back to the state I had set them up in uboot.

Charles



-----Original Message-----
From: Ami Levi Waves [mailto:ami@waves.com]=20
Sent: Wednesday, February 21, 2007 12:51 AM
To: Charles Krinke; linuxppc-embedded@ozlabs.org
Subject: RE: understanding an IO question

The way we do it:

Fisrt open a memory device:

	Handle =3D open("/dev/mem", O_RDWR);

Than map the address using:

	mmap(NULL, nAdress, PORT_READ | PROT_WRITE, MAP_SHARED, Handle,
your
base address);

mmap returns a virtual pointer, pointing to your HW.

Ami

> -----Original Message-----
> From: linuxppc-embedded-bounces+ami=3Dwaves.com@ozlabs.org
[mailto:linuxppc-
> embedded-bounces+ami=3Dwaves.com@ozlabs.org] On Behalf Of Charles =
Krinke
> Sent: Wednesday, February 21, 2007 2:14 AM
> To: linuxppc-embedded@ozlabs.org
> Subject: understanding an IO question
>=20
> I have a board using an 8541 which is based on the mpc85xx_cds bsp. It
> has one PCI device connected to the 8541, an ASIC we make that maps in
> resources for its registers at 0x8400_0000 - 0x8400_0038.
>=20
> The driver for our ASIC works in an 8241 design and I am trying to get
> its register access working in the 8541 and need a little additional
> understanding.
>=20
> I can see in cat /proc/iomem where we have memory apparently allocated
> for our registers, like this.
>=20
> cat /proc/iomem
> 80000000-9fffffff : PCI1 host bridge
>   84000000-84000038 : ichar
>   88000000-8fffffff : ichar
> e0003000-e00030ff : fsl-i2c.1
> e0004500-e0004507 : serial
>=20
> In our driver, I do an ioremap_nocache with 0x8400_0000 as an argument
> to the ioremap and am returned a virtual address of the form
> 0xc905_8000. Unfortunately, I cannot seem to access my registers as
they
> all return zero when accessed with the iowrite32/ioread32 helper
> functions.
>=20
> A little further. We do use the Outbound Address Translation window
> feature in the 8541 to get to our PCI device and I do set this up in
> uboot and I can read and write registers on our ASIC from uboot. Its
> just in the Linux driver that I seem to be having trouble.
>=20
> So,... I am hoping someone could give me a couple of pointers on where
> to look next for the missing piece of why I cannot access the
registers
> on our ASIC from Linux whereas things were working fine from uboot
> before booting Linux.
>=20
> Charles
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded

RE: understanding an IO question

[Ami Levi Waves]

The way we do it:

Fisrt open a memory device:

	Handle =3D open("/dev/mem", O_RDWR);

Than map the address using:

	mmap(NULL, nAdress, PORT_READ | PROT_WRITE, MAP_SHARED, Handle, your
base address);

mmap returns a virtual pointer, pointing to your HW.

Ami

> -----Original Message-----
> From: linuxppc-embedded-bounces+ami=3Dwaves.com@ozlabs.org =
[mailto:linuxppc-
> embedded-bounces+ami=3Dwaves.com@ozlabs.org] On Behalf Of Charles =
Krinke
> Sent: Wednesday, February 21, 2007 2:14 AM
> To: linuxppc-embedded@ozlabs.org
> Subject: understanding an IO question
>=20
> I have a board using an 8541 which is based on the mpc85xx_cds bsp. It
> has one PCI device connected to the 8541, an ASIC we make that maps in
> resources for its registers at 0x8400_0000 - 0x8400_0038.
>=20
> The driver for our ASIC works in an 8241 design and I am trying to get
> its register access working in the 8541 and need a little additional
> understanding.
>=20
> I can see in cat /proc/iomem where we have memory apparently allocated
> for our registers, like this.
>=20
> cat /proc/iomem
> 80000000-9fffffff : PCI1 host bridge
>   84000000-84000038 : ichar
>   88000000-8fffffff : ichar
> e0003000-e00030ff : fsl-i2c.1
> e0004500-e0004507 : serial
>=20
> In our driver, I do an ioremap_nocache with 0x8400_0000 as an argument
> to the ioremap and am returned a virtual address of the form
> 0xc905_8000. Unfortunately, I cannot seem to access my registers as =
they
> all return zero when accessed with the iowrite32/ioread32 helper
> functions.
>=20
> A little further. We do use the Outbound Address Translation window
> feature in the 8541 to get to our PCI device and I do set this up in
> uboot and I can read and write registers on our ASIC from uboot. Its
> just in the Linux driver that I seem to be having trouble.
>=20
> So,... I am hoping someone could give me a couple of pointers on where
> to look next for the missing piece of why I cannot access the =
registers
> on our ASIC from Linux whereas things were working fine from uboot
> before booting Linux.
>=20
> Charles
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded

understanding an IO question

[Charles Krinke]

I have a board using an 8541 which is based on the mpc85xx_cds bsp. It
has one PCI device connected to the 8541, an ASIC we make that maps in
resources for its registers at 0x8400_0000 - 0x8400_0038.

The driver for our ASIC works in an 8241 design and I am trying to get
its register access working in the 8541 and need a little additional
understanding.

I can see in cat /proc/iomem where we have memory apparently allocated
for our registers, like this.

cat /proc/iomem
80000000-9fffffff : PCI1 host bridge
  84000000-84000038 : ichar
  88000000-8fffffff : ichar
e0003000-e00030ff : fsl-i2c.1
e0004500-e0004507 : serial

In our driver, I do an ioremap_nocache with 0x8400_0000 as an argument
to the ioremap and am returned a virtual address of the form
0xc905_8000. Unfortunately, I cannot seem to access my registers as they
all return zero when accessed with the iowrite32/ioread32 helper
functions.

A little further. We do use the Outbound Address Translation window
feature in the 8541 to get to our PCI device and I do set this up in
uboot and I can read and write registers on our ASIC from uboot. Its
just in the Linux driver that I seem to be having trouble.

So,... I am hoping someone could give me a couple of pointers on where
to look next for the missing piece of why I cannot access the registers
on our ASIC from Linux whereas things were working fine from uboot
before booting Linux.

Charles