Questions on interrupt vector assignment on MPC8641D

Questions on interrupt vector assignment on MPC8641D

[david.hagood]

I'm new to developing on the PPC (most of my experience is on x86), and I
have a couple of questions about mapping of interrupts on the PPC,
specifically on the Freescale 8641D embedded CPU.

1) How does one allocate interrupt vectors for the embedded devices, such
as the MPIC? All the kernel how-tos are really targeting the x86 and PCI,
so I've been unable to find a good document on "that which is not an X86".

2) Are there any good guides to programming the PPC generally, and the
MPC8641 specifically, that I can go read to answer my other questions?

I'm trying to write a driver to map the 8641D's PCIe controllers *as
endpoints* into something frobablbe from user space - to allow setting the
BARs and outbound address translation units via sysfs files, and to allow
mapping of inbound interrupts to sysfs "doorbell"-style files. If I can
get it worked out I want to submit it back to the kernel for general
distribution.

If anybody has any insights, I'd greatly appreciate it if you'd copy my
work email (david.hagood at aeroflex.com) as well (I am using my personal
email as work insisted upon adding a one paragraph "wart" to all outbound
email that isn't acceptable for a list like this.)

Also, if anybody out there has experience with the 8641D PEX's in endpoint
mode and wouldn't mind answering a couple of questions about programming
the OWBARs, I'd be greatly obliged.

RE: Questions on interrupt vector assignment on MPC8641D

[Anderson, Trevor]

IRQ assignments for MPC8641D are "virtual", meaning "made up" and quite di=
fficult to determine by looking through code.
But I believe the plan goes something like this:

  IRQ
---------
    0           No interrupt
  1 - 15        External interrupts (only 1..12 would be used)
 16 - 127       Internal interrupts - see Table 9-2 of 8641D ref man,
            Add 15 to the numbers shown
128 - 255       Yours, or your BSP's, to do with as you like
256 - 511       Message Signaled Interrupts

If you look in the sample device trees for 8641 cards you'll see that
the interrupt values cited there for various devices match the 16..127
group above.



> -----Original Message-----
> From: linuxppc-dev-bounces+tanderson=3Dcurtisswright.com@lists.ozlabs.or=
g [mailto:linuxppc-dev-
> bounces+tanderson=3Dcurtisswright.com@lists.ozlabs.org] On Behalf Of dav=
id.hagood@gmail.com
> Sent: Tuesday, September 21, 2010 7:12 AM
> To: linuxppc-dev@lists.ozlabs.org
> Subject: Questions on interrupt vector assignment on MPC8641D
>
> I'm new to developing on the PPC (most of my experience is on x86), and =
I
> have a couple of questions about mapping of interrupts on the PPC,
> specifically on the Freescale 8641D embedded CPU.
>
> 1) How does one allocate interrupt vectors for the embedded devices, suc=
h
> as the MPIC? All the kernel how-tos are really targeting the x86 and PCI=
,
> so I've been unable to find a good document on "that which is not an X86=
".
>
> 2) Are there any good guides to programming the PPC generally, and the
> MPC8641 specifically, that I can go read to answer my other questions?
>
> I'm trying to write a driver to map the 8641D's PCIe controllers *as
> endpoints* into something frobablbe from user space - to allow setting t=
he
> BARs and outbound address translation units via sysfs files, and to allo=
w
> mapping of inbound interrupts to sysfs "doorbell"-style files. If I can
> get it worked out I want to submit it back to the kernel for general
> distribution.
>
> If anybody has any insights, I'd greatly appreciate it if you'd copy my
> work email (david.hagood at aeroflex.com) as well (I am using my persona=
l
> email as work insisted upon adding a one paragraph "wart" to all outboun=
d
> email that isn't acceptable for a list like this.)
>
> Also, if anybody out there has experience with the 8641D PEX's in endpoi=
nt
> mode and wouldn't mind answering a couple of questions about programming=

> the OWBARs, I'd be greatly obliged.
>
>
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev

_______________________________________________________________________
This e-mail and any files transmitted with it are proprietary and intended=
 solely for the use of the individual or entity to whom they are addressed=
. If you have reason to believe that you have received this e-mail in erro=
r, please notify the sender and destroy this email and any attached files.=
 Please note that any views or opinions presented in this e-mail are solel=
y those of=20the author and do not necessarily represent those of the Curt=
iss-Wright Corporation or any of its subsidiaries.  Documents attached her=
eto may contain technology subject to government export regulations. Recip=
ient is solely responsible for ensuring that any re-export, transfer or di=
sclosure of this information is in accordance with applicable government e=
xport regulations.  The recipient should check this e-mail and any attachm=
ents for the presence of viruses. Curtiss-Wright Corporation and its subsi=
diaries accept no liability for any damage caused by any virus transmitted=
 by this e-mail.

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Tue, 21 Sep 2010 17:37:15 -0400
"Anderson, Trevor" <tanderson@curtisswright.com> wrote:

> IRQ assignments for MPC8641D are "virtual", 

This is true for powerpc in general.

> meaning "made up" and quite difficult to determine by looking through code.
> But I believe the plan goes something like this:
> 
>   IRQ
> ---------
>     0           No interrupt
>   1 - 15        External interrupts (only 1..12 would be used)
>  16 - 127       Internal interrupts - see Table 9-2 of 8641D ref man,
>             Add 15 to the numbers shown
> 128 - 255       Yours, or your BSP's, to do with as you like
> 256 - 511       Message Signaled Interrupts
> 
> If you look in the sample device trees for 8641 cards you'll see that
> the interrupt values cited there for various devices match the 16..127
> group above.

There are two different numberspaces involved here.

The MPIC interrupt numberspace in the device tree (which is not
virtual; it is a private numberspace to MPIC) is based on the offset of
the registers for that interrupt source. External interrupts start at
zero (which is valid), internal at 16, and special things like MSIs at
higher numbers (I don't think it's quite 256).

The numberspace for virtual interrupts, which is what you pass to
request_irq(), is dynamically allocated.  You shouldn't need to know
anything about it other than that zero is invalid, and you get the
virtual IRQ from irq_of_parse_and_map() or another such mapping
function.

In neither case is there a range that is set aside for you "to do with
as you like".  If you have another interrupt controller and need a
block of virtual interrupts, register the controller with
irq_alloc_host() and you'll get a block assigned.

-Scott

RE: Questions on interrupt vector assignment on MPC8641D

[Chen, Tiejun]

> -----Original Message-----
> From:=20
> linuxppc-dev-bounces+tiejun.chen=3Dwindriver.com@lists.ozlabs.or
> g=20
> [mailto:linuxppc-dev-bounces+tiejun.chen=3Dwindriver.com@lists.o
> zlabs.org] On Behalf Of Scott Wood
> Sent: Wednesday, September 22, 2010 6:07 AM
> To: Anderson, Trevor
> Cc: david.hagood@gmail.com; linuxppc-dev@lists.ozlabs.org
> Subject: Re: Questions on interrupt vector assignment on MPC8641D
>=20
> On Tue, 21 Sep 2010 17:37:15 -0400
> "Anderson, Trevor" <tanderson@curtisswright.com> wrote:
>=20
> > IRQ assignments for MPC8641D are "virtual",
>=20
> This is true for powerpc in general.
>=20
> > meaning "made up" and quite difficult to determine by=20
> looking through code.
> > But I believe the plan goes something like this:
> >=20
> >   IRQ
> > ---------
> >     0           No interrupt
> >   1 - 15        External interrupts (only 1..12 would be used)
> >  16 - 127       Internal interrupts - see Table 9-2 of=20
> 8641D ref man,
> >             Add 15 to the numbers shown
> > 128 - 255       Yours, or your BSP's, to do with as you like
> > 256 - 511       Message Signaled Interrupts
> >=20
> > If you look in the sample device trees for 8641 cards=20
> you'll see that=20
> > the interrupt values cited there for various devices match=20
> the 16..127=20
> > group above.
>=20
> There are two different numberspaces involved here.
>=20
> The MPIC interrupt numberspace in the device tree (which is=20
> not virtual; it is a private numberspace to MPIC) is based on=20
> the offset of the registers for that interrupt source.=20
> External interrupts start at zero (which is valid), internal=20
> at 16, and special things like MSIs at higher numbers (I=20
> don't think it's quite 256).

Agreed.=20

This should be depend on the given platform. For example 0xe0~0xe7 are
ranged for MSI on Freescale targets. And it can avoid easily to conflict
those used lower numbers when the special MSI interrupt numbers are
higher.=20

Tiejun

>=20
> The numberspace for virtual interrupts, which is what you=20
> pass to request_irq(), is dynamically allocated.  You=20
> shouldn't need to know anything about it other than that zero=20
> is invalid, and you get the virtual IRQ from=20
> irq_of_parse_and_map() or another such mapping function.
>=20
> In neither case is there a range that is set aside for you=20
> "to do with as you like".  If you have another interrupt=20
> controller and need a block of virtual interrupts, register=20
> the controller with
> irq_alloc_host() and you'll get a block assigned.
>=20
> -Scott
>=20
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
>=20

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

> On Tue, 21 Sep 2010 17:37:15 -0400
> The MPIC interrupt numberspace in the device tree (which is not
> virtual; it is a private numberspace to MPIC) is based on the offset of
> the registers for that interrupt source. External interrupts start at
> zero (which is valid), internal at 16, and special things like MSIs at
> higher numbers (I don't think it's quite 256).

OK, so I'm slowly wrapping my head around this (OT: Has anybody considered
sending this information to the folks doing the Linux Device Drivers
books? They are just a bit x86 centric right now...).

As I understand, what I have to do is somehow get a device_node *, then
make a call to irq_of_parse_and_map() to convert that into a system IRQ.

What I am doing right now is:
device_node *mpic = of_find_node_by_type(0,"open-pic");
irq = irq_of_parse_and_map(mpic,256);

While I get a pointer from of_find_node_by_type, when I try to map IRQ
#256 via the irq_of_parse_and_map function, I get a zero back from it.

So I guess my questions would be:
1) should I be frobbing the PIC for this, or should I be looking up some
other device?
2) How do I know for certain that 256 is the right value for the first MSI
signaled via MSIR0?

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Thu, 7 Oct 2010 15:12:26 -0500
<david.hagood@gmail.com> wrote:

> > On Tue, 21 Sep 2010 17:37:15 -0400
> > The MPIC interrupt numberspace in the device tree (which is not
> > virtual; it is a private numberspace to MPIC) is based on the offset of
> > the registers for that interrupt source. External interrupts start at
> > zero (which is valid), internal at 16, and special things like MSIs at
> > higher numbers (I don't think it's quite 256).
> 
> OK, so I'm slowly wrapping my head around this (OT: Has anybody considered
> sending this information to the folks doing the Linux Device Drivers
> books? They are just a bit x86 centric right now...).
> 
> As I understand, what I have to do is somehow get a device_node *, then
> make a call to irq_of_parse_and_map() to convert that into a system IRQ.
> 
> What I am doing right now is:
> device_node *mpic = of_find_node_by_type(0,"open-pic");
> irq = irq_of_parse_and_map(mpic,256);

This is asking for the 256th specifier in the interrupts property in
the mpic node -- not what you want.

Ideally you would have a node for your device with an interrupt
specifier that you could look up with irq_of_parse_and_map().  But if
not, you can use irq_create_mapping() to directly convert an MPIC IRQ
number to a virtual IRQ.

What interrupt are you trying to select with 256?  That doesn't look
like a valid MPIC interrupt number.

> 2) How do I know for certain that 256 is the right value for the first MSI
> signaled via MSIR0?

It isn't.  The vector/priority register (MSIVPR0) is at offset
0x51c00.  Each interrupt source is 32 bytes.  The first interrupt
source is at 0x50000.

So the interrupt number is (0x51c00 - 0x50000) / 32 = 224.

BTW, the MSIs are already described in an msi node in the device tree.
Are you trying to do something different with them than PCI MSIs?  If
not, is the existing MSI driver not working?  If you are trying to do
something different, you should bind to that node yourself and use
irq_of_parse_and_map() on it.

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

> On Thu, 7 Oct 2010 15:12:26 -0500.
>
> BTW, the MSIs are already described in an msi node in the device tree.
As far as I can tell, ONLY in root complex mode, not in endpoint mode,
which is what I am working with.

What I want is a means by which the system root complex can generate one
or more interrupts INTO the 8641 when it is operating in endpoint mode.

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

First of all - where is all of this documented? There seems to be a great
deal of "oral tradition" type knowledge here, but is any of it actually
written down somewhere? (see below for examples)

> On Thu, 7 Oct 2010 15:12:26 -0500
> This is asking for the 256th specifier in the interrupts property in
> the mpic node -- not what you want.

That was from some of the previous emails in this thread.

> Ideally you would have a node for your device with an interrupt
> specifier that you could look up with irq_of_parse_and_map().

OK, and how do these devices come into being? From what I can tell, they
are defined by uBoot, and if uBoot doesn't define it, then you are out of
luck.

But even assuming you can define these nodes at run time, as far as I can
see, you are right back to the question of "and how do I know what the
mappings are so I can create the node?" You haven't answered the question,
you've just moved where you are asking it.


> The vector/priority register (MSIVPR0) is at offset
> 0x51c00.  Each interrupt source is 32 bytes.  The first interrupt
> source is at 0x50000.
>
> So the interrupt number is (0x51c00 - 0x50000) / 32 = 224.

OK, so that's how you get the MPIC IRQ number. It's a pity that the key
piece of information - "MPIC IRQ Numbers are defined by the interrupt
vector table starting at 0x50000 and incrementing by 0x20 - see table 9.3
of the MPD8641D manual for details" wasn't spelled out anywhere I could
find.

Given the data I had: the kernel sources, the MPD8641D manual, and not
much else, I had three unknowns:
* a table of config registers, that could map an set of events into an
arbitrary vector number.
* The standard kernel "request_irq" API that takes that arbitrary vector
number
* A set of APIs (irq_of_parse_and_map(), irq_create_mapping) that were,
for all intents, undocumented (just "here's the parameters", no
description of when and how to use them), that took a ???? and returned a
???? (because, with the documentation I had, that's basically all I could
say about them).

There was no documentation that said how to compute the MPIC interrupt
vector, no documentation that said which API to use to convert that to an
kernel IRQ, nothing.

> BTW, the MSIs are already described in an msi node in the device tree.

As I stated previously - not that I can see. Neither does
/proc/device_tree contain such a listing, nor does doing a
"of_find_node_by_name(0,"msi");" nor "of_find_node_by_type(0,"msi");" find
them.

It may be they are defined in PCI Root Complex mode, but again - I am
writing code to handle Endpoint mode, which, as far as I can tell from the
kernel sources, is NOT handled in any standardized way.


Again, it would be nice were this sort of thing documented somewhere. If
it is, I'd love a link - I've tried every combination of search terms I
can think of to find it to no avail.

If it isn't documented, might I suggest that either a) creating a
dedicated "Programming PPC Embedded devices and their components under
Linux" be created, or b) an appropriate chapter or chapters be drafted and
submitted to the Linux Device Drivers book.


Don't take this message the wrong way - I do want to say "THANK YOU FOR
THE HELP!" - I am just trying to point out why somebody with a fair amount
of Linux kernel experience and a great deal of general embedded experience
is having great trouble doing a simple thing.

I'll try creating a mapping, binding to it, and seeing what happens.

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

OK, using 224 as the MPIC interrupt number, and attempting to map it via
irq_create_mapping(0,224) gives me a kernel seg fault:

Unable to handle kernel paging request for data at address 0x00000000
Faulting instruction address: 0xc0016540
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT SMP NR_CPUS=2 EP8641A
Modules linked in: Endpoint_driver(+)
NIP: c0016540 LR: c0050b38 CTR: c00163b8
REGS: ef8c1ab0 TRAP: 0300   Not tainted  (2.6.26.2-ep1.10)
MSR: 00001032 <ME,IR,DR>  CR: 24024482  XER: 00000000
DAR: 00000000, DSISR: 40000000
TASK = eedec6a0[60] 'insmod' THREAD: ef8c0000 CPU: 1
GPR00: 00000000 ef8c1b60 eedec6a0 ffffffea 00000000 00000000 c035c17c
000000e0
GPR08: 00400000 00000000 c035c020 00000000 24024422 100a7264 00000000
00000095
GPR16: 00000095 00000000 0000000d 00000124 f102fd98 00000000 c004a4fc
f102c000
GPR24: 0000001a f1038400 00000000 c031e610 fffffffa 00009032 c031e5e0
000000e0
NIP [c0016540] mpic_set_irq_type+0x188/0x1c4
LR [c0050b38] set_irq_type+0x84/0xc8
Call Trace:
[ef8c1b60] [c0050b1c] set_irq_type+0x68/0xc8 (unreliable)
[ef8c1b80] [c001572c] mpic_host_map+0xec/0xf4
[ef8c1b90] [c00064a0] irq_setup_virq+0x60/0x98
[ef8c1bb0] [c00065dc] irq_create_mapping+0x104/0x114
[ef8c1bd0] [f1034798] Endpoint_device_ctor+0x258/0x3b8 [Endpoint_driver]

Reading the source (since these calls don't seem to have any other
documentation), it *looks* like it ought to be valid to call
irq_create_mapping with a null irq_host *host - which is good, since I
can't see anything obvious that would give me some other irq_host (again,
since none of this seems to be documented anywhere).

I am assuming the default host used by the system would be correct.

However, then I get this fault.

Re: Questions on interrupt vector assignment on MPC8641D

[tiejun.chen]

david.hagood@gmail.com wrote:
> First of all - where is all of this documented? There seems to be a great
> deal of "oral tradition" type knowledge here, but is any of it actually
> written down somewhere? (see below for examples)
> 
>> On Thu, 7 Oct 2010 15:12:26 -0500
>> This is asking for the 256th specifier in the interrupts property in
>> the mpic node -- not what you want.
> 
> That was from some of the previous emails in this thread.
> 
>> Ideally you would have a node for your device with an interrupt
>> specifier that you could look up with irq_of_parse_and_map().
> 
> OK, and how do these devices come into being? From what I can tell, they
> are defined by uBoot, and if uBoot doesn't define it, then you are out of
> luck.

You should define MSI device nodes on your target dts. And you can refer to the
file, mpc8572ds.dts.

Often U-Boot dose not generate MSI information and embed that to dtb.

> 
> But even assuming you can define these nodes at run time, as far as I can
> see, you are right back to the question of "and how do I know what the
> mappings are so I can create the node?" You haven't answered the question,
> you've just moved where you are asking it.

I think you can check fsl_msi.c to figure out what you want.
> 
> 
>> The vector/priority register (MSIVPR0) is at offset
>> 0x51c00.  Each interrupt source is 32 bytes.  The first interrupt
>> source is at 0x50000.
>>
>> So the interrupt number is (0x51c00 - 0x50000) / 32 = 224.
> 
> OK, so that's how you get the MPIC IRQ number. It's a pity that the key
> piece of information - "MPIC IRQ Numbers are defined by the interrupt
> vector table starting at 0x50000 and incrementing by 0x20 - see table 9.3
> of the MPD8641D manual for details" wasn't spelled out anywhere I could
> find.
> 
> Given the data I had: the kernel sources, the MPD8641D manual, and not
> much else, I had three unknowns:
> * a table of config registers, that could map an set of events into an
> arbitrary vector number.
> * The standard kernel "request_irq" API that takes that arbitrary vector
> number
> * A set of APIs (irq_of_parse_and_map(), irq_create_mapping) that were,
> for all intents, undocumented (just "here's the parameters", no
> description of when and how to use them), that took a ???? and returned a
> ???? (because, with the documentation I had, that's basically all I could
> say about them).

Firstly you should use irq_of_parse_and_map()/irq_create_mapping() to map the
real hardware irq to virtual irq. Then use request_irq() with the virtual irq to
hook your interrupt handler.

> 
> There was no documentation that said how to compute the MPIC interrupt
> vector, no documentation that said which API to use to convert that to an
> kernel IRQ, nothing.

Often most latest classic books/articles always use x86 code as an example to
clarify Linux. So you have to understand something on PPC via codes. But I think
you will benefit more information from the codes than documents :)

> 
>> BTW, the MSIs are already described in an msi node in the device tree.
> 
> As I stated previously - not that I can see. Neither does
> /proc/device_tree contain such a listing, nor does doing a
> "of_find_node_by_name(0,"msi");" nor "of_find_node_by_type(0,"msi");" find
> them.
> 
> It may be they are defined in PCI Root Complex mode, but again - I am
> writing code to handle Endpoint mode, which, as far as I can tell from the
> kernel sources, is NOT handled in any standardized way.

Maybe you can check the file, ppc4xx_pci.c since ppc4xx also can support EP as I
previously said. And sounds Scott will do something to support EP for Freescale
chip.

Looks you want to your host root complex to trigger MSI to mpc8641 EP target? If
so I'm a bit confused since MSIs should be delivered to to the Root Complex
resided on your host.

Tiejun

> 
> 
> Again, it would be nice were this sort of thing documented somewhere. If
> it is, I'd love a link - I've tried every combination of search terms I
> can think of to find it to no avail.
> 
> If it isn't documented, might I suggest that either a) creating a
> dedicated "Programming PPC Embedded devices and their components under
> Linux" be created, or b) an appropriate chapter or chapters be drafted and
> submitted to the Linux Device Drivers book.
> 
> 
> Don't take this message the wrong way - I do want to say "THANK YOU FOR
> THE HELP!" - I am just trying to point out why somebody with a fair amount
> of Linux kernel experience and a great deal of general embedded experience
> is having great trouble doing a simple thing.
> 
> I'll try creating a mapping, binding to it, and seeing what happens.
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
> 

Re: Questions on interrupt vector assignment on MPC8641D

[tiejun.chen]

david.hagood@gmail.com wrote:
> OK, using 224 as the MPIC interrupt number, and attempting to map it via
> irq_create_mapping(0,224) gives me a kernel seg fault:

This should not be correct without initialing MSI for MPIC host. As I comment on
another email, please refer to the file, arch/powerpc/sysdev/fsl_msi.c.

-Tiejun

> 
> Unable to handle kernel paging request for data at address 0x00000000
> Faulting instruction address: 0xc0016540
> Oops: Kernel access of bad area, sig: 11 [#1]
> PREEMPT SMP NR_CPUS=2 EP8641A
> Modules linked in: Endpoint_driver(+)
> NIP: c0016540 LR: c0050b38 CTR: c00163b8
> REGS: ef8c1ab0 TRAP: 0300   Not tainted  (2.6.26.2-ep1.10)
> MSR: 00001032 <ME,IR,DR>  CR: 24024482  XER: 00000000
> DAR: 00000000, DSISR: 40000000
> TASK = eedec6a0[60] 'insmod' THREAD: ef8c0000 CPU: 1
> GPR00: 00000000 ef8c1b60 eedec6a0 ffffffea 00000000 00000000 c035c17c
> 000000e0
> GPR08: 00400000 00000000 c035c020 00000000 24024422 100a7264 00000000
> 00000095
> GPR16: 00000095 00000000 0000000d 00000124 f102fd98 00000000 c004a4fc
> f102c000
> GPR24: 0000001a f1038400 00000000 c031e610 fffffffa 00009032 c031e5e0
> 000000e0
> NIP [c0016540] mpic_set_irq_type+0x188/0x1c4
> LR [c0050b38] set_irq_type+0x84/0xc8
> Call Trace:
> [ef8c1b60] [c0050b1c] set_irq_type+0x68/0xc8 (unreliable)
> [ef8c1b80] [c001572c] mpic_host_map+0xec/0xf4
> [ef8c1b90] [c00064a0] irq_setup_virq+0x60/0x98
> [ef8c1bb0] [c00065dc] irq_create_mapping+0x104/0x114
> [ef8c1bd0] [f1034798] Endpoint_device_ctor+0x258/0x3b8 [Endpoint_driver]
> 
> Reading the source (since these calls don't seem to have any other
> documentation), it *looks* like it ought to be valid to call
> irq_create_mapping with a null irq_host *host - which is good, since I
> can't see anything obvious that would give me some other irq_host (again,
> since none of this seems to be documented anywhere).
> 
> I am assuming the default host used by the system would be correct.
> 
> However, then I get this fault.
> 
> 
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
> 

Re: Questions on interrupt vector assignment on MPC8641D

[David Hagood]

On Mon, 2010-10-11 at 17:51 +0800, tiejun.chen wrote:
> 
> You should define MSI device nodes on your target dts. And you can refer to the
> file, mpc8572ds.dts.
> 
> Often U-Boot dose not generate MSI information and embed that to dtb.
> 
> > 
> > But even assuming you can define these nodes at run time, as far as I can
> > see, you are right back to the question of "and how do I know what the
> > mappings are so I can create the node?" You haven't answered the question,
> > you've just moved where you are asking it.
> 
> I think you can check fsl_msi.c to figure out what you want.
<snip>

> Often most latest classic books/articles always use x86 code as an
example to
> clarify Linux. So you have to understand something on PPC via codes.
But I think
> you will benefit more information from the codes than documents :)
> 

RTFS. That wouldn't be bad advice, if the source were actually
commented. None of these APIS have any meaningful comments in them,
around them, or anywhere near them. You'd think we were back in the old
BASIC days, when comments occupied run-time memory. Code can tell me
WHAT is being done, but not WHY, and not what assumptions are being
made.

So when you say "You should define MSI device nodes on your target dts."
we go right back to my comment of "You haven't answered the question,
you've just moved where you are asking it."

> 
> Firstly you should use irq_of_parse_and_map()/irq_create_mapping() to map the
> real hardware irq to virtual irq. Then use request_irq() with the virtual irq to
> hook your interrupt handler.
And when I do that, I get a segfault as my follow-on mail reports.

> 
> Maybe you can check the file, ppc4xx_pci.c since ppc4xx also can support EP as I
> previously said. And sounds Scott will do something to support EP for Freescale
> chip.
I will look at that file.

> 
> Looks you want to your host root complex to trigger MSI to mpc8641 EP target? If
> so I'm a bit confused since MSIs should be delivered to to the Root Complex
> resided on your host.
I want the host to be able to interrupt the PPC. Since this is PCIe, the
only way that will be working is for the root complex to do a write to
somewhere on the PPC. This is basically the same mechanism as MSI, only
with the PPC as the target of the write rather than the root complex.
Obviously, the rest of the PCI devices will be sending interrupts to the
root complex, some via MSI. That has nothing to do with this discussion.

Since the PPC is in endpoint mode, the MSI hardware on the PPC won't be
used. The hardware is writable from the root complex via the PPC BAR0,
so I see no reason not to reuse it to trigger interrupts on the PPC from
the Root complex.

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

> You should define MSI device nodes on your target dts. And you can refer
> to the
> file, mpc8572ds.dts.

I see nothing in that file that defines any MSIs. I see code that looks
like it maps ROOT COMPLEX MODE interrupts on regular PCI interfaces, which
IS NOT WHAT I AM DOING.

Since it seems I have been unclear, let me state this as clearly as possible.

I AM DOING ENDPOINT MODE.


> I think you can check fsl_msi.c to figure out what you want.
Except that file doesn't exist in the 2.6.26 kernel with which I am working.

> Firstly you should use irq_of_parse_and_map()/irq_create_mapping() to map
> the
> real hardware irq to virtual irq. Then use request_irq() with the virtual
> irq to
> hook your interrupt handler.

Except I have no device to pass irq_of_parse_and_map, and when I call
irq_create_mapping it seg faults.

>
> Maybe you can check the file, ppc4xx_pci.c since ppc4xx also can support
> EP as I
> previously said.

It only "supports" EP mode in the sense that it sets the hardware up to
export a BAR, and does nothing else. It doesn't provide any means for any
other code to actually DO anything in that mode. It doesn't define a way
to be interrupted by the root complex, nor to do anything when that
interrupt happens. It doesn't supply any meaningful framework to get any
data from the Root Complex.

That's the whole reason I am trying to make a piece of code that actually
WILL make those things available in a useful fashion.

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Sat, 9 Oct 2010 10:52:49 -0500
<david.hagood@gmail.com> wrote:

> First of all - where is all of this documented? There seems to be a great
> deal of "oral tradition" type knowledge here, but is any of it actually
> written down somewhere? (see below for examples)

Documentation/powerpc/dts-bindings/fsl/mpic.txt

Plus the chip manual, for the register offsets.

> > On Thu, 7 Oct 2010 15:12:26 -0500
> > This is asking for the 256th specifier in the interrupts property in
> > the mpic node -- not what you want.
> 
> That was from some of the previous emails in this thread.
> 
> > Ideally you would have a node for your device with an interrupt
> > specifier that you could look up with irq_of_parse_and_map().
> 
> OK, and how do these devices come into being? From what I can tell, they
> are defined by uBoot, and if uBoot doesn't define it, then you are out of
> luck.

More commonly they're statically defined in the dts file, not
dynamically created by u-boot.

If it's not in the dts, add it.  If for whatever reason that's not an
option, you can use irq_create_mapping() as I mentioned in the previous
e-mail.

> * A set of APIs (irq_of_parse_and_map(), irq_create_mapping) that were,
> for all intents, undocumented (just "here's the parameters", no
> description of when and how to use them), that took a ???? and returned a
> ???? (because, with the documentation I had, that's basically all I could
> say about them).

Yes, there are many parts of the kernel that could use better
documentation.

> > BTW, the MSIs are already described in an msi node in the device tree.
> 
> As I stated previously - not that I can see.

What board are you using?  What kernel?

If you grep arch/powerpc/boot/dts for msi in a reasonably recent kernel
you should find msi nodes.

> It may be they are defined in PCI Root Complex mode, 

No, it's a separate node.

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Mon, 11 Oct 2010 17:51:39 +0800
"tiejun.chen" <tiejun.chen@windriver.com> wrote:

> Maybe you can check the file, ppc4xx_pci.c since ppc4xx also can support EP as I
> previously said. And sounds Scott will do something to support EP for Freescale
> chip.

What did I get signed up for? :-)

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

Re-ordering your questions a bit:

> What board are you using?  What kernel?

One of 2 boards: Either an Embedded Planet or a Performance Tech uTCA
board based on the MPC8641D, running the 2.6.26 as supplied by EP.


> On Sat, 9 Oct 2010 10:52:49 -0500
> Documentation/powerpc/dts-bindings/fsl/mpic.txt
Not present in the version I have.

>
> Plus the chip manual, for the register offsets.
I have that now, so at least one part of the fog is a bit less dense.

> If it's not in the dts, add it.  If for whatever reason that's not an
> option, you can use irq_create_mapping() as I mentioned in the previous
> e-mail.
And as I've said previously, I have no good info on HOW to add the nodes,
WHAT to add, or WHERE. You may as well be saying "Bargle the Narbog".

And when I try to use irq_create_mapping() it seg faults, which doesn't
exactly help me get my interrupt hooked up.

(crotchety old man mode)Kids these day - when I was a kid, you just
grabbed the IRQ vector into your assembly code and away you went. Now GET
OFF MY YARD!(/crotchety old man mode).

> If you grep arch/powerpc/boot/dts for msi in a reasonably recent kernel
> you should find msi nodes.

ddhagood@WIC-102362:..Workspace/Linux_Kernel_for_PPC> grep msi
arch/powerpc/boot/dts  -ir
arch/powerpc/boot/dts/glacier.dts:			enable-msi-hole;
arch/powerpc/boot/dts/taishan.dts:			enable-msi-hole;
arch/powerpc/boot/dts/canyonlands.dts:			enable-msi-hole;
arch/powerpc/boot/dts/katmai.dts:			enable-msi-hole;

> What did I get signed up for? :-)

It sounds like Tiejun thinks you might be working on, well, basically what
I am working on - a generic interface to allow user space to support being
a endpoint, with the ability to generate interrupts to the host root
complex, get interrupts from the host root complex, provide memory to be
accessed by the host root complex via the PPC's BARs, and to access the
host root complex's PCI address space via the OATMU windows. Something to
allow a person to use a PPC SOC as an endpoint, and to get on with the job
of providing functionality rather than having to deal with esoterica.

I'm hoping to make an interface generic enough to support not only the
PPC, but other devices as well (things I cannot go into due to NDAs at
this time). As a systems programmer, I am tired of re-inventing the wheel
when dealing with being an endpoint - I want to invent the wheel, and push
it uphill to the greater Linux community so that a) others are spared the
pain I am suffering, and b) maybe other SOC vendors will implement my
interface (and thus I won't have to).

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Mon, 11 Oct 2010 17:55:07 +0800
"tiejun.chen" <tiejun.chen@windriver.com> wrote:

> david.hagood@gmail.com wrote:
> > OK, using 224 as the MPIC interrupt number, and attempting to map it via
> > irq_create_mapping(0,224) gives me a kernel seg fault:
> 
> This should not be correct without initialing MSI for MPIC host. As I comment on
> another email, please refer to the file, arch/powerpc/sysdev/fsl_msi.c.

The MSI driver is not going to do anything that would change whether
that call to irq_create_mapping() works or not.  fsl_msi.c would also
probably require some changes to be useful in endpoint mode.

> > Reading the source (since these calls don't seem to have any other
> > documentation), it *looks* like it ought to be valid to call
> > irq_create_mapping with a null irq_host *host - which is good, since I
> > can't see anything obvious that would give me some other irq_host (again,
> > since none of this seems to be documented anywhere).

There's mpic stuff in the call trace, so the NULL host was OK.

Look in arch/powerpc/platforms/86xx/pic.c.  What is the second-to-last
parameter of mpc86xx_init_irq() in your kernel tree?  It's 256 in
current upstream -- this is the number of IRQ sources the MPIC driver
will handle.

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Mon, 11 Oct 2010 12:02:15 -0500
<david.hagood@gmail.com> wrote:

> Re-ordering your questions a bit:
> 
> > What board are you using?  What kernel?
> 
> One of 2 boards: Either an Embedded Planet or a Performance Tech uTCA
> board based on the MPC8641D, running the 2.6.26 as supplied by EP.

That's a very old kernel.  Have you contacted EP support?

You may be better off trying to migrate anything board-specific to the
latest kernel, rather than do new development on the vendor kernel,
especially if you're looking for community support, and definitely if
you're trying to do something that could be merged upstream to help
others.

The MPC8641D itself is supported in mainline Linux.

> > On Sat, 9 Oct 2010 10:52:49 -0500
> > Documentation/powerpc/dts-bindings/fsl/mpic.txt
> Not present in the version I have.

We didn't have a time machine, so we were only able to add things to new
releases, not old ones. :-)

Should it have been documented from the start?  Ideally, yes, and we're
pickier now than we were then about insisting on bindings being
documented before they're used.  But nothing's perfect.

> > If it's not in the dts, add it.  If for whatever reason that's not an
> > option, you can use irq_create_mapping() as I mentioned in the previous
> > e-mail.
> And as I've said previously, I have no good info on HOW to add the nodes,
> WHAT to add, or WHERE. You may as well be saying "Bargle the Narbog".

There are no nodes to add with irq_create_mapping().

Just bargle the narbog.

> And when I try to use irq_create_mapping() it seg faults, which doesn't
> exactly help me get my interrupt hooked up.

Well, I didn't know it was going to do that...  Your usage of
irq_create_mapping() seems fine, there's just something going wrong.
See my other reply about the number of MPIC interrupts.

> > If you grep arch/powerpc/boot/dts for msi in a reasonably recent kernel
> > you should find msi nodes.
> 
> ddhagood@WIC-102362:..Workspace/Linux_Kernel_for_PPC> grep msi
> arch/powerpc/boot/dts  -ir
> arch/powerpc/boot/dts/glacier.dts:			enable-msi-hole;
> arch/powerpc/boot/dts/taishan.dts:			enable-msi-hole;
> arch/powerpc/boot/dts/canyonlands.dts:			enable-msi-hole;
> arch/powerpc/boot/dts/katmai.dts:			enable-msi-hole;

"reasonably recent kernel"

$ grep -rI msi arch/powerpc/boot/dts/|wc -l
85

> > What did I get signed up for? :-)
> 
> It sounds like Tiejun thinks you might be working on, well, basically what
> I am working on - a generic interface to allow user space to support being
> a endpoint,

I am not working on that -- not sure how I gave that impression.

> with the ability to generate interrupts to the host root
> complex, get interrupts from the host root complex, provide memory to be
> accessed by the host root complex via the PPC's BARs, and to access the
> host root complex's PCI address space via the OATMU windows. Something to
> allow a person to use a PPC SOC as an endpoint, and to get on with the job
> of providing functionality rather than having to deal with esoterica.

That does sound like it would be a useful addition.

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[tiejun.chen]

Scott Wood wrote:
> On Mon, 11 Oct 2010 17:51:39 +0800
> "tiejun.chen" <tiejun.chen@windriver.com> wrote:
> 
>> Maybe you can check the file, ppc4xx_pci.c since ppc4xx also can support EP as I
>> previously said. And sounds Scott will do something to support EP for Freescale
>> chip.
> 
> What did I get signed up for? :-)

As far as I recalled you ever comment you will support EP when you replied to
David in the previous email. If no I may misunderstand something so sorry for
this :)

Tiejun

> 
> -Scott
> 
> 

Re: Questions on interrupt vector assignment on MPC8641D

[tiejun.chen]

tiejun.chen wrote:
> david.hagood@gmail.com wrote:
>> OK, using 224 as the MPIC interrupt number, and attempting to map it via
>> irq_create_mapping(0,224) gives me a kernel seg fault:
> 
> This should not be correct without initialing MSI for MPIC host. As I comment on
> another email, please refer to the file, arch/powerpc/sysdev/fsl_msi.c.

With Scott's comment I remember the NULL host would be always 'irq_default_host'
at last. The 'irq_default_host' should be your MPIC host so it should be fine.

> 
> -Tiejun
> 
>> Unable to handle kernel paging request for data at address 0x00000000
>> Faulting instruction address: 0xc0016540
>> Oops: Kernel access of bad area, sig: 11 [#1]
>> PREEMPT SMP NR_CPUS=2 EP8641A
>> Modules linked in: Endpoint_driver(+)
>> NIP: c0016540 LR: c0050b38 CTR: c00163b8
>> REGS: ef8c1ab0 TRAP: 0300   Not tainted  (2.6.26.2-ep1.10)
>> MSR: 00001032 <ME,IR,DR>  CR: 24024482  XER: 00000000
>> DAR: 00000000, DSISR: 40000000
>> TASK = eedec6a0[60] 'insmod' THREAD: ef8c0000 CPU: 1
>> GPR00: 00000000 ef8c1b60 eedec6a0 ffffffea 00000000 00000000 c035c17c
					^
					The return value, r3, should be -22, '-EINVAL'.

And refer to the call trace and the source codes
------
int mpic_set_irq_type(unsigned int virq, unsigned int flow_type)
{
        struct mpic *mpic = mpic_from_irq(virq);
        unsigned int src = mpic_irq_to_hw(virq);
        struct irq_desc *desc = get_irq_desc(virq);
        unsigned int vecpri, vold, vnew;

        DBG("mpic: set_irq_type(mpic:@%p,virq:%d,src:0x%x,type:0x%x)\n",
            mpic, virq, src, flow_type);

        if (src >= mpic->irq_count)
                return -EINVAL;
			^
			I think this OOPS may be from here.

You can enable DBG on this file, arch/powerpc/sysdev/mpic.c, to track 'src' and
'mpic->irq_count' on your old tree.

-Tiejun

>> 000000e0
>> GPR08: 00400000 00000000 c035c020 00000000 24024422 100a7264 00000000
>> 00000095
>> GPR16: 00000095 00000000 0000000d 00000124 f102fd98 00000000 c004a4fc
>> f102c000
>> GPR24: 0000001a f1038400 00000000 c031e610 fffffffa 00009032 c031e5e0
>> 000000e0
>> NIP [c0016540] mpic_set_irq_type+0x188/0x1c4
>> LR [c0050b38] set_irq_type+0x84/0xc8
>> Call Trace:
>> [ef8c1b60] [c0050b1c] set_irq_type+0x68/0xc8 (unreliable)
>> [ef8c1b80] [c001572c] mpic_host_map+0xec/0xf4
>> [ef8c1b90] [c00064a0] irq_setup_virq+0x60/0x98
>> [ef8c1bb0] [c00065dc] irq_create_mapping+0x104/0x114
>> [ef8c1bd0] [f1034798] Endpoint_device_ctor+0x258/0x3b8 [Endpoint_driver]
>>
>> Reading the source (since these calls don't seem to have any other
>> documentation), it *looks* like it ought to be valid to call
>> irq_create_mapping with a null irq_host *host - which is good, since I
>> can't see anything obvious that would give me some other irq_host (again,
>> since none of this seems to be documented anywhere).
>>
>> I am assuming the default host used by the system would be correct.
>>
>> However, then I get this fault.
>>
>>
>>
>> _______________________________________________
>> Linuxppc-dev mailing list
>> Linuxppc-dev@lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/linuxppc-dev
>>
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
> 

Re: Questions on interrupt vector assignment on MPC8641D

[tiejun.chen]

Scott Wood wrote:
> On Mon, 11 Oct 2010 12:02:15 -0500
> <david.hagood@gmail.com> wrote:
> 
>> Re-ordering your questions a bit:
>>
>>> What board are you using?  What kernel?
>> One of 2 boards: Either an Embedded Planet or a Performance Tech uTCA
>> board based on the MPC8641D, running the 2.6.26 as supplied by EP.
> 
> That's a very old kernel.  Have you contacted EP support?
> 
> You may be better off trying to migrate anything board-specific to the
> latest kernel, rather than do new development on the vendor kernel,
> especially if you're looking for community support, and definitely if
> you're trying to do something that could be merged upstream to help
> others.
> 
> The MPC8641D itself is supported in mainline Linux.
> 
>>> On Sat, 9 Oct 2010 10:52:49 -0500
>>> Documentation/powerpc/dts-bindings/fsl/mpic.txt
>> Not present in the version I have.
> 
> We didn't have a time machine, so we were only able to add things to new
> releases, not old ones. :-)
> 
> Should it have been documented from the start?  Ideally, yes, and we're
> pickier now than we were then about insisting on bindings being
> documented before they're used.  But nothing's perfect.

On the *latest* tree you can find more than your tree:
------
Documentation/powerpc/dts-bindings/fsl/msi-pic.txt
Documentation/powerpc/booting-without-of.txt

If you want to understand dts in detail you can refer to the file,
------
http://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.0.pdf

> 
>>> If it's not in the dts, add it.  If for whatever reason that's not an
>>> option, you can use irq_create_mapping() as I mentioned in the previous
>>> e-mail.
>> And as I've said previously, I have no good info on HOW to add the nodes,
>> WHAT to add, or WHERE. You may as well be saying "Bargle the Narbog".
> 
> There are no nodes to add with irq_create_mapping().

Sure.

> 
> Just bargle the narbog.
> 
>> And when I try to use irq_create_mapping() it seg faults, which doesn't
>> exactly help me get my interrupt hooked up.
> 
> Well, I didn't know it was going to do that...  Your usage of
> irq_create_mapping() seems fine, there's just something going wrong.

The 'NULL' host would be 'irq_default_host'. For OOPS you can check my another
email and hope its useful.

-Tiejun

> See my other reply about the number of MPIC interrupts.

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

> There's mpic stuff in the call trace, so the NULL host was OK.
>
> Look in arch/powerpc/platforms/86xx/pic.c.  What is the second-to-last
> parameter of mpc86xx_init_irq() in your kernel tree?  It's 256 in
> current upstream -- this is the number of IRQ sources the MPIC driver
> will handle.
>

One, thanks for your help - I do appreciate it.

In the tree I am working from, the call to mpc86xx_init_irq() function is
in a BSP file (ep8641a.c), and the line was

 mpic1 = mpic_alloc(np, res.start,
			MPIC_PRIMARY | MPIC_WANTS_RESET | MPIC_BIG_ENDIAN,
			16, NR_IRQS-4,
			" MPIC     ");

I tried changing that to
mpic1 = mpic_alloc(np, res.start,
			MPIC_PRIMARY | MPIC_WANTS_RESET | MPIC_BIG_ENDIAN,
			16, 256,
			" MPIC     ");

With no meaningful changes.

I wonder about the next lines:


	mpic_assign_isu(mpic1, 0, res.start + 0x10000);

	/* 48 Internal Interrupts */
	mpic_assign_isu(mpic1, 1, res.start + 0x10200);
	mpic_assign_isu(mpic1, 2, res.start + 0x10400);
	mpic_assign_isu(mpic1, 3, res.start + 0x10600);

	/* 16 External interrupts
	 * Moving them from [0 - 15] to [64 - 79]
	 */
	mpic_assign_isu(mpic1, 4, res.start + 0x10000);


Looking at the code, and where it appears to be faulting, it looks like
its in kernel/irq/chip.c:


int set_irq_type(unsigned int irq, unsigned int type)
{
	struct irq_desc *desc;
	unsigned long flags;
	int ret = -ENXIO;

	if (irq >= NR_IRQS) {
		printk(KERN_ERR "Trying to set irq type for IRQ%d\n", irq);
		return -ENODEV;
	}

	desc = irq_desc + irq;
------------------------
	if (desc->chip->set_type) {
		spin_lock_irqsave(&desc->lock, flags);
		ret = desc->chip->set_type(irq, type);
------------------------


		spin_unlock_irqrestore(&desc->lock, flags);
	}
	return ret;
}

My conjecture is that desc->chip isn't set. Is mpic_assign_isu the
function that does that?

(yes, I know - update your kernel. I am 2 weeks from a delivery, I have to
merge driver changes to sRIO in along with all of this....)

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Tue, 12 Oct 2010 15:55:28 -0500
<david.hagood@gmail.com> wrote:

> I wonder about the next lines:
> 
> 
> 	mpic_assign_isu(mpic1, 0, res.start + 0x10000);
> 
> 	/* 48 Internal Interrupts */
> 	mpic_assign_isu(mpic1, 1, res.start + 0x10200);
> 	mpic_assign_isu(mpic1, 2, res.start + 0x10400);
> 	mpic_assign_isu(mpic1, 3, res.start + 0x10600);
> 
> 	/* 16 External interrupts
> 	 * Moving them from [0 - 15] to [64 - 79]
> 	 */
> 	mpic_assign_isu(mpic1, 4, res.start + 0x10000);

No mainline 86xx boards do that, even in 2.6.26.  I suspect you need to
either get rid of the isu stuff altogether, or add a mapping for the
MSI interrupts.

> Looking at the code, and where it appears to be faulting, it looks like
> its in kernel/irq/chip.c:
> 
> 
> int set_irq_type(unsigned int irq, unsigned int type)
> {
> 	struct irq_desc *desc;
> 	unsigned long flags;
> 	int ret = -ENXIO;
> 
> 	if (irq >= NR_IRQS) {
> 		printk(KERN_ERR "Trying to set irq type for IRQ%d\n", irq);
> 		return -ENODEV;
> 	}
> 
> 	desc = irq_desc + irq;
> ------------------------
> 	if (desc->chip->set_type) {
> 		spin_lock_irqsave(&desc->lock, flags);
> 		ret = desc->chip->set_type(irq, type);
> ------------------------
> 
> 
> 		spin_unlock_irqrestore(&desc->lock, flags);
> 	}
> 	return ret;
> }
> 
> My conjecture is that desc->chip isn't set. Is mpic_assign_isu the
> function that does that?

That happens in set_irq_chip_and_handler(), called from mpic_host_map()
-- just a few lines before calling set_irq_type().

The crash is happening somewhere in mpic_set_irq_type():
> NIP [c0016540] mpic_set_irq_type+0x188/0x1c4

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

On Mon, 2010-10-11 at 09:44 -0500, david.hagood@gmail.com wrote:
> > You should define MSI device nodes on your target dts. And you can refer
> > to the
> > file, mpc8572ds.dts.
> 
> I see nothing in that file that defines any MSIs. I see code that looks
> like it maps ROOT COMPLEX MODE interrupts on regular PCI interfaces, which
> IS NOT WHAT I AM DOING.
> 
> Since it seems I have been unclear, let me state this as clearly as possible.
> 
> I AM DOING ENDPOINT MODE.

Shouting is not going to make you any friends.

cheers



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Questions on interrupt vector assignment on MPC8641D

[tiejun.chen]

Scott Wood wrote:
> On Tue, 12 Oct 2010 15:55:28 -0500
> <david.hagood@gmail.com> wrote:
> 
>> I wonder about the next lines:
>>
>>
>> 	mpic_assign_isu(mpic1, 0, res.start + 0x10000);
>>
>> 	/* 48 Internal Interrupts */
>> 	mpic_assign_isu(mpic1, 1, res.start + 0x10200);
>> 	mpic_assign_isu(mpic1, 2, res.start + 0x10400);
>> 	mpic_assign_isu(mpic1, 3, res.start + 0x10600);
>>
>> 	/* 16 External interrupts
>> 	 * Moving them from [0 - 15] to [64 - 79]
>> 	 */
>> 	mpic_assign_isu(mpic1, 4, res.start + 0x10000);
> 
> No mainline 86xx boards do that, even in 2.6.26.  I suspect you need to
> either get rid of the isu stuff altogether, or add a mapping for the
> MSI interrupts.
> 
>> Looking at the code, and where it appears to be faulting, it looks like
>> its in kernel/irq/chip.c:
>>
>>
>> int set_irq_type(unsigned int irq, unsigned int type)
>> {
>> 	struct irq_desc *desc;
>> 	unsigned long flags;
>> 	int ret = -ENXIO;
>>
>> 	if (irq >= NR_IRQS) {
>> 		printk(KERN_ERR "Trying to set irq type for IRQ%d\n", irq);
>> 		return -ENODEV;
>> 	}
>>
>> 	desc = irq_desc + irq;
>> ------------------------
>> 	if (desc->chip->set_type) {
>> 		spin_lock_irqsave(&desc->lock, flags);
>> 		ret = desc->chip->set_type(irq, type);
>> ------------------------
>>
>>
>> 		spin_unlock_irqrestore(&desc->lock, flags);
>> 	}
>> 	return ret;
>> }
>>
>> My conjecture is that desc->chip isn't set. Is mpic_assign_isu the
>> function that does that?
> 
> That happens in set_irq_chip_and_handler(), called from mpic_host_map()
> -- just a few lines before calling set_irq_type().
> 
> The crash is happening somewhere in mpic_set_irq_type():

Agreed. That is just where I pointed out on my email replied for OOPS. To enable
DBG to figure out 'src' and 'mpic->irq_count' from the file,
arch/powerpc/sysdev/mpic.c,    .
======
int mpic_set_irq_type(unsigned int virq, unsigned int flow_type)
{
	......
	if (src >= mpic->irq_count)
		return -EINVAL;
			^
			I think this OOPS may be from here.


Tiejun

>> NIP [c0016540] mpic_set_irq_type+0x188/0x1c4
> 
> -Scott
> 
> 

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Wed, 13 Oct 2010 09:17:01 +0800
"tiejun.chen" <tiejun.chen@windriver.com> wrote:

> Scott Wood wrote:
> > The crash is happening somewhere in mpic_set_irq_type():
> 
> Agreed. That is just where I pointed out on my email replied for OOPS. To enable
> DBG to figure out 'src' and 'mpic->irq_count' from the file,
> arch/powerpc/sysdev/mpic.c,    .
> ======
> int mpic_set_irq_type(unsigned int virq, unsigned int flow_type)
> {
> 	......
> 	if (src >= mpic->irq_count)
> 		return -EINVAL;
> 			^
> 			I think this OOPS may be from here.

No, it's after that.  His board code is using the mpic's "isu" remapping
mechanism, and the MSIs aren't covered, so those registers aren't
ioremapped.

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

Do you have any suggestions on where to find out what parameters I should
pass to mpic_assign_isu() to map the interrupts? It's another of those
charmingly undocumented functions, even in the most recent git pull of the
mainline kernel.

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Wed, 13 Oct 2010 12:08:16 -0500
<david.hagood@gmail.com> wrote:

> Do you have any suggestions on where to find out what parameters I should
> pass to mpic_assign_isu() to map the interrupts? It's another of those
> charmingly undocumented functions, even in the most recent git pull of the
> mainline kernel.

I'd just rip the whole thing out of the board code, and pass zero in
isu_size to mpic_alloc(), if you can undo whatever is depending on the
remapping.

Otherwise, I'm guessing from the lines that you posted that you want to
add:

mpic_assign_isu(mpic1, 14, res.start + 0x11c00);

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

> On Wed, 13 Oct 2010 12:08:16 -0500
> I'd just rip the whole thing out of the board code, and pass zero in
> isu_size to mpic_alloc(), if you can undo whatever is depending on the
> remapping.

OK, what I did was to change

mpic1 = mpic_alloc(np, res.start,
			MPIC_PRIMARY | MPIC_WANTS_RESET | MPIC_BIG_ENDIAN,
			16, 256,
			" MPIC     ");

to
mpic1 = mpic_alloc(np, res.start,
	MPIC_PRIMARY | MPIC_WANTS_RESET | MPIC_BIG_ENDIAN,
	0, 256,
	" MPIC     ");

and remove all the calls to mpic_assign_isu.

When I did this, the board booted, but when I then tried to map my
interrupt, I got a segfault:

unsigned mpic_vector = mpic_reg_to_irq(MSIVPR0);

      printk(KERN_INFO "Endpoint %s: got MPIC IRQ %d\n",name,mpic_vector);

      dis->irq = irq_create_mapping(0,mpic_vector);

Led to:
Using EP8641A machine description
Total memory = 1024MB; using 2048kB for hash table (at cfa00000)
<snip>
mpic: Setting up MPIC " MPIC     " version 1.2 at e0040000, max 2 CPUs
mpic: ISU size: 88, shift: 7, mask: 7f
mpic: Initializing for 88 sources
<snip>
RAMDISK: Compressed image found at block 0
VFS: Mounted root (ext2 filesystem).
Freeing unused kernel memory: 164k init
mpic_enable_irq timeout
PCI endpoint driver. Ver. 0.01
Copyright (C) 2010, Aeroflex, Inc
compiled Oct 12 2010 15:26:57
Endpoint: PORBMSR 8F300001, PORDEVSR 0AA58740
Endpoint: PORPLLSR 0020000A
Endpoint: PORDEVSR 0AA58740
Endpoint: PORDBGMSR 0E000000
Endpoint: PORIMPCR 0000007F
Endpoint: PORCIR 2CFFFFFF
SerDes1 PEX, SerDes2 sRIO 3.125Gbps
EP: from 00000000 to 1FFFFFFF is DDR1
EP: from F0000000 to FFFFFFFF is LBC
EP: from C0000000 to DFFFFFFF is sRIO
EP: from A0000000 to BFFFFFFF is PEX1
EP: from E2000000 to E2FFFFFF is PEX1
EP: from 20000000 to 3FFFFFFF is DDR2
Endpoint PEX1 eedb1800: PEX_CONFIG 00000028
Endpoint PEX1: alloc LAW space A0000000 - A0000FFF
Endpoint PEX1: got MPIC IRQ 224
Unable to handle kernel paging request for data at address 0x00000c00
Faulting instruction address: 0xc0016540
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT SMP NR_CPUS=2 EP8641A
Modules linked in: Endpoint_driver(+)
NIP: c0016540 LR: c0050b38 CTR: c00163b8
REGS: eed41ab0 TRAP: 0300   Not tainted  (2.6.26.2-ep1.10)
MSR: 00001032 <ME,IR,DR>  CR: 24024482  XER: 00000000
DAR: 00000c00, DSISR: 40000000
TASK = eed82050[59] 'insmod' THREAD: eed40000 CPU: 0
GPR00: 00000c00 eed41b60 eed82050 ffffffea 00000000 00000000 c035c17c
000000e0
GPR08: 00400000 00000000 c035c020 00000c00 24024422 100a7264 00000000
00000095
GPR16: 00000095 00000000 0000000d 00000124 f102fdb8 00000000 c004a4fc
f102c000
GPR24: 0000001a f1038428 00000000 c031e610 fffffffa 00009032 c031e5e0
000000e0
NIP [c0016540] mpic_set_irq_type+0x188/0x1c4
LR [c0050b38] set_irq_type+0x84/0xc8
Call Trace:
[eed41b60] [c0050b1c] set_irq_type+0x68/0xc8 (unreliable)
[eed41b80] [c001572c] mpic_host_map+0xec/0xf4
[eed41b90] [c00064a0] irq_setup_virq+0x60/0x98
[eed41bb0] [c00065dc] irq_create_mapping+0x104/0x114
[eed41bd0] [f1034798] Endpoint_device_ctor+0x258/0x3b4 [Endpoint_driver]
[eed41c10] [f1034d3c] Endpoint_device_find+0x448/0x63c [Endpoint_driver]
[eed41e40] [f103a028] Endpoint_init_module+0x28/0x60 [Endpoint_driver]
[eed41e60] [c004bcb4] sys_init_module+0xd4/0x1c1c
[eed41f40] [c0011278] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff787ec
    LR = 0x10010e2c
Instruction dump:
90080018 41bdff34 4bffffd4 7c0938ae 2f800000 540b073e 5400073a 40befef8
4bfffeec 7d26482e 7c095a14 7c0004ac <7d29582e> 0c090000 4c00012c 5520028e
---[ end trace a3d033ca1998c9f0 ]---
note: insmod[59] exited with preempt_count 1

>
> Otherwise, I'm guessing from the lines that you posted that you want to
> add:
>
> mpic_assign_isu(mpic1, 14, res.start + 0x11c00);
>
Actually, I tried this first. So I guess the second parameter is interrupt
vector #>>16, and the third is the address base of the registers to
generate that interrupt.

When I did this, the kernel wouldn't even boot - it seg'ed on that line.

Re: Questions on interrupt vector assignment on MPC8641D

[tiejun.chen]

Scott Wood wrote:
> On Wed, 13 Oct 2010 09:17:01 +0800
> "tiejun.chen" <tiejun.chen@windriver.com> wrote:
> 
>> Scott Wood wrote:
>>> The crash is happening somewhere in mpic_set_irq_type():
>> Agreed. That is just where I pointed out on my email replied for OOPS. To enable
>> DBG to figure out 'src' and 'mpic->irq_count' from the file,
>> arch/powerpc/sysdev/mpic.c,    .
>> ======
>> int mpic_set_irq_type(unsigned int virq, unsigned int flow_type)
>> {
>> 	......
>> 	if (src >= mpic->irq_count)
>> 		return -EINVAL;
>> 			^
>> 			I think this OOPS may be from here.
> 
> No, it's after that.  His board code is using the mpic's "isu" remapping

I means OOPS is *from* here. According to David's call trace,
mpic_set_irq_type() is the last issued function. And that corresponding return
value, R3, is '0xffffffea', -22, and also '-EINVAL'. If everything is OK, I
think we should not be failed with returning '-EINVAL' here. Right? So I think
we should dump 'src' (mpic_irq_to_hw(virq)) and 'mpic->irq_count'. Then figure
out why 'src' >= 'mpic->irq_count'. I think this can make our debug life easier.

Tiejun

> mechanism, and the MSIs aren't covered, so those registers aren't
> ioremapped.
> 
> -Scott
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
> 

Re: Questions on interrupt vector assignment on MPC8641D

[tiejun.chen]

tiejun.chen wrote:
> Scott Wood wrote:
>> On Wed, 13 Oct 2010 09:17:01 +0800
>> "tiejun.chen" <tiejun.chen@windriver.com> wrote:
>>
>>> Scott Wood wrote:
>>>> The crash is happening somewhere in mpic_set_irq_type():
>>> Agreed. That is just where I pointed out on my email replied for OOPS. To enable
>>> DBG to figure out 'src' and 'mpic->irq_count' from the file,
>>> arch/powerpc/sysdev/mpic.c,    .
>>> ======
>>> int mpic_set_irq_type(unsigned int virq, unsigned int flow_type)
>>> {
>>> 	......
>>> 	if (src >= mpic->irq_count)
>>> 		return -EINVAL;
>>> 			^
>>> 			I think this OOPS may be from here.
>> No, it's after that.  His board code is using the mpic's "isu" remapping
> 
> I means OOPS is *from* here. According to David's call trace,
> mpic_set_irq_type() is the last issued function. And that corresponding return
> value, R3, is '0xffffffea', -22, and also '-EINVAL'. If everything is OK, I
> think we should not be failed with returning '-EINVAL' here. Right? So I think
> we should dump 'src' (mpic_irq_to_hw(virq)) and 'mpic->irq_count'. Then figure
> out why 'src' >= 'mpic->irq_count'. I think this can make our debug life easier.
> 

Certainly I'm missing something since I have no any real environment. So maybe
we can use gdb to track this panic as normal :)
# gdb vmlinux
(gdb) list *mpic_set_irq_type+0x188/

Tiejun

> Tiejun
> 
>> mechanism, and the MSIs aren't covered, so those registers aren't
>> ioremapped.
>>
>> -Scott
>>
>> _______________________________________________
>> Linuxppc-dev mailing list
>> Linuxppc-dev@lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/linuxppc-dev
>>
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
> 

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Thu, 14 Oct 2010 11:27:09 +0800
"tiejun.chen" <tiejun.chen@windriver.com> wrote:

> tiejun.chen wrote:
> > Scott Wood wrote:
> >> On Wed, 13 Oct 2010 09:17:01 +0800
> >> "tiejun.chen" <tiejun.chen@windriver.com> wrote:
> >>
> >>> Scott Wood wrote:
> >>>> The crash is happening somewhere in mpic_set_irq_type():
> >>> Agreed. That is just where I pointed out on my email replied for OOPS. To enable
> >>> DBG to figure out 'src' and 'mpic->irq_count' from the file,
> >>> arch/powerpc/sysdev/mpic.c,    .
> >>> ======
> >>> int mpic_set_irq_type(unsigned int virq, unsigned int flow_type)
> >>> {
> >>> 	......
> >>> 	if (src >= mpic->irq_count)
> >>> 		return -EINVAL;
> >>> 			^
> >>> 			I think this OOPS may be from here.
> >> No, it's after that.  His board code is using the mpic's "isu" remapping
> > 
> > I means OOPS is *from* here. According to David's call trace,
> > mpic_set_irq_type() is the last issued function. And that corresponding return
> > value, R3, is '0xffffffea', -22, and also '-EINVAL'.

Just because that value is in r3 doesn't mean that src >=
mpic->irq_count.

Consider something like:

cmplw	r4, r5
li	r3, -EINVAL
bgelr
...


-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

I may have a clue (you might not think so, but...):

I've configured the init thusly:
mpic1 = mpic_alloc(np, res.start,
	MPIC_PRIMARY | MPIC_WANTS_RESET | MPIC_BIG_ENDIAN ,
	0, 256,
	" MPIC     ");

Which, as I read the code, should disable the ISU stuff.

I've seeing this on boot:
mpic: Setting up MPIC " MPIC     " version 1.2 at e0040000, max 2 CPUs
mpic: ISU size: 88, shift: 7, mask: 7f
mpic: Initializing for 88 sources

Now, since the interrupt number I want is 224, which, last time I checked,
was > 88, this may be the root cause.

As I read the code:
        /* Read feature register, calculate num CPUs and, for non-ISU
	 * MPICs, num sources as well. On ISU MPICs, sources are counted
	 * as ISUs are added
	 */
	greg_feature = mpic_read(mpic->gregs, MPIC_INFO(GREG_FEATURE_0));
	mpic->num_cpus = ((greg_feature & MPIC_GREG_FEATURE_LAST_CPU_MASK)
			  >> MPIC_GREG_FEATURE_LAST_CPU_SHIFT) + 1;
	if (isu_size == 0)
		mpic->num_sources =
			((greg_feature & MPIC_GREG_FEATURE_LAST_SRC_MASK)
			 >> MPIC_GREG_FEATURE_LAST_SRC_SHIFT) + 1;
So it would seem to me that the "greg_feature" is saying I have 88
interrupts.

I've tried setting the ISU size to 256:
mpic1 = mpic_alloc(np, res.start,
	MPIC_PRIMARY | MPIC_WANTS_RESET | MPIC_BIG_ENDIAN ,
	256, 256,
	" MPIC     ");
And that kills the kernel as we init the mpic.

SO, I guess the question in, what sets "greg_feature", as it would seem to
be incorrect.

Or, am I on the wrong trail?

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Thu, 14 Oct 2010 11:22:45 -0500
<david.hagood@gmail.com> wrote:

> As I read the code:
>         /* Read feature register, calculate num CPUs and, for non-ISU
> 	 * MPICs, num sources as well. On ISU MPICs, sources are counted
> 	 * as ISUs are added
> 	 */
> 	greg_feature = mpic_read(mpic->gregs, MPIC_INFO(GREG_FEATURE_0));
> 	mpic->num_cpus = ((greg_feature & MPIC_GREG_FEATURE_LAST_CPU_MASK)
> 			  >> MPIC_GREG_FEATURE_LAST_CPU_SHIFT) + 1;
> 	if (isu_size == 0)
> 		mpic->num_sources =
> 			((greg_feature & MPIC_GREG_FEATURE_LAST_SRC_MASK)
> 			 >> MPIC_GREG_FEATURE_LAST_SRC_SHIFT) + 1;
> So it would seem to me that the "greg_feature" is saying I have 88
> interrupts.
> 
> I've tried setting the ISU size to 256:
> mpic1 = mpic_alloc(np, res.start,
> 	MPIC_PRIMARY | MPIC_WANTS_RESET | MPIC_BIG_ENDIAN ,
> 	256, 256,
> 	" MPIC     ");
> And that kills the kernel as we init the mpic.
> 
> SO, I guess the question in, what sets "greg_feature", as it would seem to
> be incorrect.

It comes from FRR[NIRQ].  It seems that this chip takes a
less-than-useful interpretation of what that field means -- it gives
the actual number of sources, not the size of the sparsely populated
array.

If you look at current kernels, you'll find an MPIC_BROKEN_FRR_NIRQS
flag to work around this.

It's not very clear to me what distinction the MPIC code is
trying to make between irq_count and num_sources in the first place,
though.

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

Hallelujah and Huzzah! I finally got my vector!

I back-ported the MPIC_BROKEN_FRR_NIRQS flag and code to our kernel, and
the kernel is now letting me have my vector! Now I can actually see if the
dang thing works!

THANK YOU EVERYBODY for putting up with me on this!

> It comes from FRR[NIRQ].  It seems that this chip takes a
> less-than-useful interpretation of what that field means -- it gives
> the actual number of sources, not the size of the sparsely populated
> array.
Perhaps you might want to have a talk with your cow-orkers there, Scott,
since this is a Freescale part.

> It's not very clear to me what distinction the MPIC code is
> trying to make between irq_count and num_sources in the first place,
> though.
/me looks at Scott's email again.

If you, working FOR Freescale, and following the Linux development
(presumably for some time) are confused, imagine what I've been going
through!

Hot damn, and time for a quick version control commit, a hot lunch, and
really testing the code.

Thanks again!

Re: Questions on interrupt vector assignment on MPC8641D

[Scott Wood]

On Thu, 14 Oct 2010 12:20:55 -0500
<david.hagood@gmail.com> wrote:

> > It comes from FRR[NIRQ].  It seems that this chip takes a
> > less-than-useful interpretation of what that field means -- it gives
> > the actual number of sources, not the size of the sparsely populated
> > array.
> Perhaps you might want to have a talk with your cow-orkers there, Scott,
> since this is a Freescale part.

Well, it was a coworker who added the workaround, so I assume we're
already aware of the issue.

The description of NIRQ is vague enough that it's hard to argue that
Linux's expectations of what it means are justified.

-Scott

Re: Questions on interrupt vector assignment on MPC8641D

[david.hagood]

> Well, it was a coworker who added the workaround, so I assume we're
> already aware of the issue.
>
> The description of NIRQ is vague enough that it's hard to argue that
> Linux's expectations of what it means are justified.
>
Well, I now can actually interrupt the PPC from my host processor!

(well, it segfaulted, but's my fault for not actually initializing my work
queue structure. But I can get the interrupt into the part!)

So thanks to all.

I hope to get my code working, get past my delivery deadline on it, clean
it up so it matches The One True Way Of Laying Out Kernel Code, and post
it here for everybody to point and laugh at. Maybe around Christmas.

Re: Questions on interrupt vector assignment on MPC8641D

[tiejun.chen]

Scott Wood wrote:
> On Thu, 14 Oct 2010 11:27:09 +0800
> "tiejun.chen" <tiejun.chen@windriver.com> wrote:
> 
>> tiejun.chen wrote:
>>> Scott Wood wrote:
>>>> On Wed, 13 Oct 2010 09:17:01 +0800
>>>> "tiejun.chen" <tiejun.chen@windriver.com> wrote:
>>>>
>>>>> Scott Wood wrote:
>>>>>> The crash is happening somewhere in mpic_set_irq_type():
>>>>> Agreed. That is just where I pointed out on my email replied for OOPS. To enable
>>>>> DBG to figure out 'src' and 'mpic->irq_count' from the file,
>>>>> arch/powerpc/sysdev/mpic.c,    .
>>>>> ======
>>>>> int mpic_set_irq_type(unsigned int virq, unsigned int flow_type)
>>>>> {
>>>>> 	......
>>>>> 	if (src >= mpic->irq_count)
>>>>> 		return -EINVAL;
>>>>> 			^
>>>>> 			I think this OOPS may be from here.
>>>> No, it's after that.  His board code is using the mpic's "isu" remapping
>>> I means OOPS is *from* here. According to David's call trace,
>>> mpic_set_irq_type() is the last issued function. And that corresponding return
>>> value, R3, is '0xffffffea', -22, and also '-EINVAL'.
> 
> Just because that value is in r3 doesn't mean that src >=
> mpic->irq_count.
> 
> Consider something like:
> 
> cmplw	r4, r5
> li	r3, -EINVAL
> bgelr

Right absolutely and got it.

Thanks again
Tiejun

> ...
> 
> 
> -Scott
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
>