multiple separate pci bridges ...

multiple separate pci bridges ...

[Sven Luther]

Hello,

I am currently trying to port linux to the Pegasos 2, which uses the
Marvell Discovery 2 chip, and has two independent pci controllers, of
which one is faked as an agp bus. This is with a modified 2.4.23 kernel
from the linuxppc_2_4 branch.

Anyway, these two independent controllers both have one bus 0 on them :

PCI bus 0 controlled by pci at 80000000
PCI bus 0 controlled by pci at c0000000

(This coming from chrp_find_bridges).

For the pci bus, at 80000000, a simple indirect access can be used, and
i setup a specialized io ops for the faked agp bus since it needs
special care.

Later, in pcibios_init, there is a problem in the pci_scan_bus. The
first bus has no problems :

Scanning bus 00
Found 00:00 [11ab/6460] 000600 00
Found 00:08 [1106/3044] 000c00 00
Found 00:28 [1000/0001] 000100 00
Found 00:60 [1106/8231] 000601 00
Found 00:61 [1106/0571] 000101 00
Found 00:62 [1106/3038] 000c03 00
Found 00:63 [1106/3038] 000c03 00
Found 00:64 [1106/8235] 000000 00
Found 00:65 [1106/3058] 000401 00
Found 00:66 [1106/3068] 000780 00
Found 00:68 [1106/3065] 000200 00
Fixups for bus 00
Bus scan for 00 returning with max=00

But the second fails with :

PCI: Bus 00 already known

Which comes because in drivers/pci/pci.c:pci_bus_exists does handle only
buses, and thus know nothing of separate pci bridges with the same bus
number, and thus the kernel dies when accessing bus->xxx something in
pcibios_init.

Now, i know that the powermacs in particular, and maybe others, also
have this case of different same numbered pci buses with different base
addresses, and that this kind of situation has already been solved.

So, what is the workaround here, and where does it get set.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

On Fri, 2004-01-02 at 05:11, Sven Luther wrote:
>
> I am currently trying to port linux to the Pegasos 2, which uses the
> Marvell Discovery 2 chip, and has two independent pci controllers,
> of which one is faked as an agp bus. This is with a modified 2.4.23
> kernel from the linuxppc_2_4 branch.

Why "faked" ? It's actually fairly sane to have the AGP bus be a
separate PCI host controller... That's how it's done on pmacs.

> Anyway, these two independent controllers both have one bus 0 on them :
>
> PCI bus 0 controlled by pci at 80000000
> PCI bus 0 controlled by pci at c0000000
>
> (This coming from chrp_find_bridges).

Typical junk, Apple does the same. You want PCI domains or bus
renumbering. No PCI domains in 2.4 though, so you want bus renumbering.
Look at what pmac_pci.c does for uninorth which has 3 root busses whose
firmware bus numbers all start at 0

> For the pci bus, at 80000000, a simple indirect access can be used,
> and i setup a specialized io ops for the faked agp bus since it needs
> special care.

What do you mean ? config ops ? You can have different ops per host,
again, look at what pmac_pci.c does

> Later, in pcibios_init, there is a problem in the pci_scan_bus. The
> first bus has no problems :
>
> Scanning bus 00
> Found 00:00 [11ab/6460] 000600 00
> Found 00:08 [1106/3044] 000c00 00
> Found 00:28 [1000/0001] 000100 00
> Found 00:60 [1106/8231] 000601 00
> Found 00:61 [1106/0571] 000101 00
> Found 00:62 [1106/3038] 000c03 00
> Found 00:63 [1106/3038] 000c03 00
> Found 00:64 [1106/8235] 000000 00
> Found 00:65 [1106/3058] 000401 00
> Found 00:66 [1106/3068] 000780 00
> Found 00:68 [1106/3065] 000200 00
> Fixups for bus 00
> Bus scan for 00 returning with max=00
>
> But the second fails with :
>
> PCI: Bus 00 already known
>
> Which comes because in drivers/pci/pci.c:pci_bus_exists does handle
> only buses, and thus know nothing of separate pci bridges with the
> same bus number, and thus the kernel dies when accessing bus->xxx
> something in pcibios_init.

Yes, in 2.4, you need to do some renumbering.

> Now, i know that the powermacs in particular, and maybe others, also
> have this case of different same numbered pci buses with different
> base addresses, and that this kind of situation has already been
> solved.
>
> So, what is the workaround here, and where does it get set.

Basically, you do the bus numbering yourself. You never care about the
number of the "root" bus since you basically just send type 0 config
cycles to it, but you do care about bus numbers in PCI<->PCI bridges.
You trigger the proper renumbering code by setting pci_assign_all_busses
to 1.

So basically, what we do is, we keep a "next_busno" variable which is
0 at first. When probing a host, we assign it that bus number (0 for
the first host typically). Then, we call pci_scan_bus(). This one will
do the renumbering for P2P bridges when pci_assign_all_busses is true.
It will also tell us what was the "last" bus number found on the host.
We use that+1 to get the new next_busno and start again for the next
host.

On your side, you don't have much else to do that set
pci_assign_all_busses to 1, and create 2 pci_controller structures with
different config cycle ops.

Note that in 2.4, there is a small risk when doing bus renumbering with
PCI-PCI bridges that you can get temporarily overlapping bus numbers
during the probe, thus screwing it up. this typically happens on
Xserves. The problem is when scanning a given segment with P2P bridges,
when old & new numbers overlap, a not-yet scanned bridge may be set to
the same number as one we are currently scanning after having renumbered
it, thus screwing things up. I think 2.6 is protected against this
problem. For 2.4, what I do is hack to force a 0x10 bus number offset
between hosts.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

Happy new year to you.

On Fri, Jan 02, 2004 at 03:03:43PM +1100, Benjamin Herrenschmidt wrote:
> On Fri, 2004-01-02 at 05:11, Sven Luther wrote:
> >
> > I am currently trying to port linux to the Pegasos 2, which uses the
> > Marvell Discovery 2 chip, and has two independent pci controllers,
> > of which one is faked as an agp bus. This is with a modified 2.4.23
> > kernel from the linuxppc_2_4 branch.
>
> Why "faked" ? It's actually fairly sane to have the AGP bus be a
> separate PCI host controller... That's how it's done on pmacs.

There is some extra magic that needs to be done. In particular
interrupts need to be disabled before being able to write to the config
space of the agp bridge.

> > Anyway, these two independent controllers both have one bus 0 on
> > them :
> >
> > PCI bus 0 controlled by pci at 80000000
> > PCI bus 0 controlled by pci at c0000000
> >
> > (This coming from chrp_find_bridges).
>
> Typical junk, Apple does the same. You want PCI domains or bus
> renumbering. No PCI domains in 2.4 though, so you want bus
> renumbering.

Yeah.

> Look at what pmac_pci.c does for uninorth which has 3 root busses
> whose firmware bus numbers all start at 0

Ok, thanks.

> > For the pci bus, at 80000000, a simple indirect access can be used,
> > and i setup a specialized io ops for the faked agp bus since it
> > needs special care.
>
> What do you mean ? config ops ? You can have different ops per host,
> again, look at what pmac_pci.c does

Yep, that i did already. And that is what i did, i used the indirect pci
ops for the first bus, but specialized stuff for the second bus, due to
the above mentioned stuff.

> > Later, in pcibios_init, there is a problem in the pci_scan_bus. The
> > first bus has no problems :
> >
> > Scanning bus 00
> > Found 00:00 [11ab/6460] 000600 00
> > Found 00:08 [1106/3044] 000c00 00
> > Found 00:28 [1000/0001] 000100 00
> > Found 00:60 [1106/8231] 000601 00
> > Found 00:61 [1106/0571] 000101 00
> > Found 00:62 [1106/3038] 000c03 00
> > Found 00:63 [1106/3038] 000c03 00
> > Found 00:64 [1106/8235] 000000 00
> > Found 00:65 [1106/3058] 000401 00
> > Found 00:66 [1106/3068] 000780 00
> > Found 00:68 [1106/3065] 000200 00
> > Fixups for bus 00
> > Bus scan for 00 returning with max=00
> >
> > But the second fails with :
> >
> > PCI: Bus 00 already known
> >
> > Which comes because in drivers/pci/pci.c:pci_bus_exists does handle
> > only buses, and thus know nothing of separate pci bridges with the
> > same bus number, and thus the kernel dies when accessing bus->xxx
> > something in pcibios_init.
>
> Yes, in 2.4, you need to do some renumbering.

Ok.

> > Now, i know that the powermacs in particular, and maybe others, also
> > have this case of different same numbered pci buses with different
> > base addresses, and that this kind of situation has already been
> > solved.
> >
> > So, what is the workaround here, and where does it get set.
>
> Basically, you do the bus numbering yourself. You never care about the
> number of the "root" bus since you basically just send type 0 config
> cycles to it, but you do care about bus numbers in PCI<->PCI bridges.

Mmm, since these are two CPU<->PCI bridges, i don't care about that,
right. PCI<->PCI bridges are ones which stand behind a PCI<->PCI bridge,
right ?

> You trigger the proper renumbering code by setting
> pci_assign_all_busses to 1.

Ok. This must have been the magic bit i was searching for.

> So basically, what we do is, we keep a "next_busno" variable which is
> 0 at first. When probing a host, we assign it that bus number (0 for
> the first host typically). Then, we call pci_scan_bus(). This one will
> do the renumbering for P2P bridges when pci_assign_all_busses is true.
> It will also tell us what was the "last" bus number found on the host.
> We use that+1 to get the new next_busno and start again for the next
> host.

Ok. Need to look again though to find the place where the arch specific
place is to modify this. As i remember, pci_scan_bus is called from
drivers/pci/pci.c, and i shouldn't need to modify this.

> On your side, you don't have much else to do that set
> pci_assign_all_busses to 1, and create 2 pci_controller structures
> with different config cycle ops.

Yep. I guess the pci_assign_all_busses can be set in chrp_find_bridges,
in the pegasos2 special case. The rest i have already handled.

> Note that in 2.4, there is a small risk when doing bus renumbering
> with PCI-PCI bridges that you can get temporarily overlapping bus
> numbers during the probe, thus screwing it up. this typically happens
> on

Not a problem in this case though, right ? There is only one bus for
each controller, and the controller are sitting directly after the CPU.

> Xserves. The problem is when scanning a given segment with P2P
> bridges, when old & new numbers overlap, a not-yet scanned bridge
> may be set to the same number as one we are currently scanning after
> having renumbered

Ok, do we need to detect this, or will it just fail or something ?

> it, thus screwing things up. I think 2.6 is protected against this
> problem. For 2.4, what I do is hack to force a 0x10 bus number offset
> between hosts.

Mmm. Will look into that.

Anyway, i will not be able to implement it until sunday at the earliest.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> There is some extra magic that needs to be done. In particular
> interrupts need to be disabled before being able to write to the config
> space of the agp bridge.

That can be completely local to the config ops. of that pci_controller
instance.

> > What do you mean ? config ops ? You can have different ops per host,
> > again, look at what pmac_pci.c does
>
> Yep, that i did already. And that is what i did, i used the indirect pci
> ops for the first bus, but specialized stuff for the second bus, due to
> the above mentioned stuff.

Good.

> Mmm, since these are two CPU<->PCI bridges, i don't care about that,
> right. PCI<->PCI bridges are ones which stand behind a PCI<->PCI bridge,
> right ?

No, no, ... PCI<->PCI bridges can happen still... some PCI cards can
have one on-board for example.

> > You trigger the proper renumbering code by setting pci_assign_all_busses
> > to 1.
>
> Ok. This must have been the magic bit i was searching for.

Probably :)

> > So basically, what we do is, we keep a "next_busno" variable which is
> > 0 at first. When probing a host, we assign it that bus number (0 for
> > the first host typically). Then, we call pci_scan_bus(). This one will
> > do the renumbering for P2P bridges when pci_assign_all_busses is true.
> > It will also tell us what was the "last" bus number found on the host.
> > We use that+1 to get the new next_busno and start again for the next
> > host.
>
> Ok. Need to look again though to find the place where the arch specific
> place is to modify this. As i remember, pci_scan_bus is called from
> drivers/pci/pci.c, and i shouldn't need to modify this.

No, look what I do when setting up uninorth in pmac_pci.c

> > On your side, you don't have much else to do that set
> > pci_assign_all_busses to 1, and create 2 pci_controller structures with
> > different config cycle ops.
>
> Yep. I guess the pci_assign_all_busses can be set in chrp_find_bridges,
> in the pegasos2 special case. The rest i have already handled.

Yes.

> > Note that in 2.4, there is a small risk when doing bus renumbering with
> > PCI-PCI bridges that you can get temporarily overlapping bus numbers
> > during the probe, thus screwing it up. this typically happens on
>
> Not a problem in this case though, right ? There is only one bus for
> each controller, and the controller are sitting directly after the CPU.

And ? You can still have P2P bridges sitting on one of your PCI busses.

> > Xserves. The problem is when scanning a given segment with P2P bridges,
> > when old & new numbers overlap, a not-yet scanned bridge may be set to
> > the same number as one we are currently scanning after having renumbered
>
> Ok, do we need to detect this, or will it just fail or something ?

It will do random crap on PCI probe

> > it, thus screwing things up. I think 2.6 is protected against this
> > problem. For 2.4, what I do is hack to force a 0x10 bus number offset
> > between hosts.
>
> Mmm. Will look into that.
>
> Anyway, i will not be able to implement it until sunday at the earliest.
>
> Friendly,
>
> Sven Luther
--
Benjamin Herrenschmidt <benh@kernel.crashing.org>

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Rob Baxter]

Sven,

Synergy Microsystems has designs with the Discovery 2 as well.  We worked
around this by setting pci_assign_all_busses to true and with modifications
to the platform specific config cycle routines (only included one as an
example):

int
gemini_pcibios_bus_fixup(struct pci_dev *dev, struct pci_controller *hose)
{
        return ((dev->bus->number == hose->first_busno) ? 0 : dev->bus->number);
}

int
gemini_pcibios_read_config_byte(struct pci_dev *dev, int offset, u8 *val)
{
        unsigned long reg;
        unsigned int addr;
        int bus;
        struct pci_controller *hose;

        hose = dev->sysdata;
        bus = gemini_pcibios_bus_fixup(dev, hose);
        addr = pci_config_type1_addr(bus, dev->devfn, offset & ~(0x3));

        reg = config_read(addr, hose->cfg_addr, hose->cfg_data);
        *val = ((reg >> ((offset & 0x3) << 3)) & 0xff);
        return PCIBIOS_SUCCESSFUL;
}

It is also necessary to set the first and last bus number fields of the
hose structure correctly in your platform specific code.  These will get
over written later by the PCI driver code, and in our case, to the same
values.  Our platform specific code does a preliminary scan of the PCI
buses to find the first and last bus numbers.

This will also work when a PtP bridge is located on either of the primary
PCI buses of the Discovery.  We're using a derivative of a 2.4.19 kernel.

I'm open to a better approach...

Rob Baxter

On Thu, Jan 01, 2004 at 07:11:45PM +0100, Sven Luther wrote:
>
> I am currently trying to port linux to the Pegasos 2, which uses the
> Marvell Discovery 2 chip, and has two independent pci controllers,
> of which one is faked as an agp bus. This is with a modified 2.4.23
> kernel from the linuxppc_2_4 branch.
>
> Anyway, these two independent controllers both have one bus 0 on them :
>
> PCI bus 0 controlled by pci at 80000000
> PCI bus 0 controlled by pci at c0000000
>
> (This coming from chrp_find_bridges).
>
> For the pci bus, at 80000000, a simple indirect access can be used,
> and i setup a specialized io ops for the faked agp bus since it needs
> special care.
>
> Later, in pcibios_init, there is a problem in the pci_scan_bus. The
> first bus has no problems :
>
> Scanning bus 00
> Found 00:00 [11ab/6460] 000600 00
> Found 00:08 [1106/3044] 000c00 00
> Found 00:28 [1000/0001] 000100 00
> Found 00:60 [1106/8231] 000601 00
> Found 00:61 [1106/0571] 000101 00
> Found 00:62 [1106/3038] 000c03 00
> Found 00:63 [1106/3038] 000c03 00
> Found 00:64 [1106/8235] 000000 00
> Found 00:65 [1106/3058] 000401 00
> Found 00:66 [1106/3068] 000780 00
> Found 00:68 [1106/3065] 000200 00
> Fixups for bus 00
> Bus scan for 00 returning with max=00
>
> But the second fails with :
>
> PCI: Bus 00 already known
>
> Which comes because in drivers/pci/pci.c:pci_bus_exists does handle
> only buses, and thus know nothing of separate pci bridges with the
> same bus number, and thus the kernel dies when accessing bus->xxx
> something in pcibios_init.
>
> Now, i know that the powermacs in particular, and maybe others, also
> have this case of different same numbered pci buses with different
> base addresses, and that this kind of situation has already been
> solved.
>
> So, what is the workaround here, and where does it get set.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Geert Uytterhoeven]

On Fri, 2 Jan 2004, Sven Luther wrote:
> Happy new year to you.

Happy New Year!

> On Fri, Jan 02, 2004 at 03:03:43PM +1100, Benjamin Herrenschmidt wrote:
> > On Fri, 2004-01-02 at 05:11, Sven Luther wrote:
> > > I am currently trying to port linux to the Pegasos 2, which
> > > uses the Marvell Discovery 2 chip, and has two independent pci
> > > controllers, of which one is faked as an agp bus. This is with a
> > > modified 2.4.23 kernel from the linuxppc_2_4 branch.
> >
> > Why "faked" ? It's actually fairly sane to have the AGP bus be a
> > separate PCI host controller... That's how it's done on pmacs.
>
> There is some extra magic that needs to be done. In particular
> interrupts need to be disabled before being able to write to the
> config space of the agp bridge.

I seem to remember the upper layer already takes care of disabling interrupts?
Or am I mistaken?

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

On Sat, 2004-01-03 at 02:18, Rob Baxter wrote:
> Sven,
>
> Synergy Microsystems has designs with the Discovery 2 as well.  We worked
> around this by setting pci_assign_all_busses to true and with modifications
> to the platform specific config cycle routines (only included one as an
> example):
>
> int
> gemini_pcibios_bus_fixup(struct pci_dev *dev, struct pci_controller *hose)
> {
>         return ((dev->bus->number == hose->first_busno) ? 0 : dev->bus->number);
> }
>
> int
> gemini_pcibios_read_config_byte(struct pci_dev *dev, int offset, u8 *val)
> {
>         unsigned long reg;
>         unsigned int addr;
>         int bus;
>         struct pci_controller *hose;
>
>         hose = dev->sysdata;
>         bus = gemini_pcibios_bus_fixup(dev, hose);
>         addr = pci_config_type1_addr(bus, dev->devfn, offset & ~(0x3));
>
>         reg = config_read(addr, hose->cfg_addr, hose->cfg_data);
>         *val = ((reg >> ((offset & 0x3) << 3)) & 0xff);
>         return PCIBIOS_SUCCESSFUL;
> }

Hrm... Why this bus fixup thing ? Why not just clearly writing
something like

if (dev->bus->number == hose->first_busno)
	discovery2_config_type0_addr(....);
else
	discovery2_config_type1_addr(....);

And those could be inline...

Also, the way you define functions with a name that "looks" like a
generic function (pci_config_type1_addr) is rather ugly...


> It is also necessary to set the first and last bus number fields of the
> hose structure correctly in your platform specific code.  These will get
> over written later by the PCI driver code, and in our case, to the same
> values.  Our platform specific code does a preliminary scan of the PCI
> buses to find the first and last bus numbers.

Why would you need to set them correctly ? The PCI code will set them
to sensible values before doing the bus scan and will fixup the last
busno afterward for each controller, you do not need any pre-scan or
whatever and you can leave those values to 0 at init time.

> This will also work when a PtP bridge is located on either of the primary
> PCI buses of the Discovery.  We're using a derivative of a 2.4.19 kernel.

Yup, that should work fine.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Rob Baxter]

On Sat, Jan 03, 2004 at 10:56:57AM +1100, Benjamin Herrenschmidt wrote:
> On Sat, 2004-01-03 at 02:18, Rob Baxter wrote:
> > Sven,
> >
> > Synergy Microsystems has designs with the Discovery 2 as well.  We worked
> > around this by setting pci_assign_all_busses to true and with modifications
> > to the platform specific config cycle routines (only included one as an
> > example):
> >
> > int
> > gemini_pcibios_bus_fixup(struct pci_dev *dev, struct pci_controller *hose)
> > {
> >         return ((dev->bus->number == hose->first_busno) ? 0 : dev->bus->number);
> > }
> >
> > int
> > gemini_pcibios_read_config_byte(struct pci_dev *dev, int offset, u8 *val)
> > {
> >         unsigned long reg;
> >         unsigned int addr;
> >         int bus;
> >         struct pci_controller *hose;
> >
> >         hose = dev->sysdata;
> >         bus = gemini_pcibios_bus_fixup(dev, hose);
> >         addr = pci_config_type1_addr(bus, dev->devfn, offset & ~(0x3));
> >
> >         reg = config_read(addr, hose->cfg_addr, hose->cfg_data);
> >         *val = ((reg >> ((offset & 0x3) << 3)) & 0xff);
> >         return PCIBIOS_SUCCESSFUL;
> > }
>
> Hrm... Why this bus fixup thing ? Why not just clearly writing
> something like
>
> if (dev->bus->number == hose->first_busno)
> 	discovery2_config_type0_addr(....);
> else
> 	discovery2_config_type1_addr(....);
>
> And those could be inline...

I like this, thank you, a much cleaner approach.

>
> Also, the way you define functions with a name that "looks" like a
> generic function (pci_config_type1_addr) is rather ugly...

It's a local macro, I agree, it's ugly, it was there before my time:

#define pci_config_type1_addr(bus,dev,offset) \
        (0x80000000 | ((bus)<<16) | ((dev)<<8) | (offset))

#define pci_config_type0_addr(dev,offset) \
        ((1 << PCI_SLOT(dev)) | (PCI_FUNC(dev) << 8) | offset)

>
>
> > It is also necessary to set the first and last bus number fields of the
> > hose structure correctly in your platform specific code.  These will get
> > over written later by the PCI driver code, and in our case, to the same
> > values.  Our platform specific code does a preliminary scan of the PCI
> > buses to find the first and last bus numbers.
>
> Why would you need to set them correctly ? The PCI code will set them
> to sensible values before doing the bus scan and will fixup the last
> busno afterward for each controller, you do not need any pre-scan or
> whatever and you can leave those values to 0 at init time.

This was necessary because the PCI driver (scan/enumeration code) was using
these same config calls that was being fixed up.  (A chicken and egg issue?)

>
> > This will also work when a PtP bridge is located on either of the primary
> > PCI buses of the Discovery.  We're using a derivative of a 2.4.19 kernel.
>
> Yup, that should work fine.
>
> Ben.

--
-Rob

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> This was necessary because the PCI driver (scan/enumeration code) was using
> these same config calls that was being fixed up.  (A chicken and egg issue?)

This should still not be necessary... Before scanning a bus, the PCI code
will properly set first_busno, thus the config access methods will work.

The only problem I spotted in 2.4 (and afaik, that is fixed in 2.6) is
that when P2P bridges are moved around, we could end up with a temporary
wrong state where 2 of these on a given bus segment would try to decode
the same bus numbers, thus screwing up probe. This is why I added a
skew value of 0x10 between each host on some machines.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

Hello,

I managed to boot into linux, altough i didn't tried yet with a root
filesystem, so i don't know if it will work.

Still, i get :

PCI: Probing PCI hardware
Scanning bus 00
Found 00:00 [11ab/6460] 000600 00
Found 00:08 [1106/3044] 000c00 00
Found 00:28 [1000/0001] 000100 00
Found 00:60 [1106/8231] 000601 00
Found 00:61 [1106/0571] 000101 00
Found 00:62 [1106/3038] 000c03 00
Found 00:63 [1106/3038] 000c03 00
Found 00:64 [1106/8235] 000000 00
Found 00:65 [1106/3058] 000401 00
Found 00:66 [1106/3068] 000780 00
Found 00:68 [1106/3065] 000200 00
Fixups for bus 00
Bus scan for 00 returning with max=00
Scanning bus 10
Fixups for bus 10
Bus scan for 10 returning with max=10
PCI: Cannot allocate resource region 4 of device 00:00.0

So, the 0x10 incrementation is fine, but the agp bus is not working
correctly, and thus the agp graphic card not recognized (and thus no
radeonfb).

That said, i believe the 00:00.0 (should that not be 10:00.0 because of
the renumbering) is the Marvell Discovery II pci controller, and the
actual agp card should be found just behind.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> PCI: Probing PCI hardware
> Scanning bus 00
> Found 00:00 [11ab/6460] 000600 00
> Found 00:08 [1106/3044] 000c00 00
> Found 00:28 [1000/0001] 000100 00
> Found 00:60 [1106/8231] 000601 00
> Found 00:61 [1106/0571] 000101 00
> Found 00:62 [1106/3038] 000c03 00
> Found 00:63 [1106/3038] 000c03 00
> Found 00:64 [1106/8235] 000000 00
> Found 00:65 [1106/3058] 000401 00
> Found 00:66 [1106/3068] 000780 00
> Found 00:68 [1106/3065] 000200 00
> Fixups for bus 00
> Bus scan for 00 returning with max=00
> Scanning bus 10
> Fixups for bus 10
> Bus scan for 10 returning with max=10
> PCI: Cannot allocate resource region 4 of device 00:00.0
>
> So, the 0x10 incrementation is fine, but the agp bus is not working
> correctly, and thus the agp graphic card not recognized (and thus no
> radeonfb).
>
> That said, i believe the 00:00.0 (should that not be 10:00.0 because of
> the renumbering) is the Marvell Discovery II pci controller, and the
> actual agp card should be found just behind.

What video card is this ?

It's possible that your config ops for the AGP don't work properly. Make
sure you are really generating type 0 config cycles.

Also, this "cannot allocate resource region" smells bad. Can
you check what the BARs of the host bridge are supposed to mean ?

I hope it's not yet another brain dead bridge that puts the system memory
in a BAR... in which case we'll have to "hide" it from linux PCI subsystem.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Mon, Jan 05, 2004 at 08:45:24AM +1100, Benjamin Herrenschmidt wrote:
>
>
> > PCI: Probing PCI hardware
> > Scanning bus 00
> > Found 00:00 [11ab/6460] 000600 00
> > Found 00:08 [1106/3044] 000c00 00
> > Found 00:28 [1000/0001] 000100 00
> > Found 00:60 [1106/8231] 000601 00
> > Found 00:61 [1106/0571] 000101 00
> > Found 00:62 [1106/3038] 000c03 00
> > Found 00:63 [1106/3038] 000c03 00
> > Found 00:64 [1106/8235] 000000 00
> > Found 00:65 [1106/3058] 000401 00
> > Found 00:66 [1106/3068] 000780 00
> > Found 00:68 [1106/3065] 000200 00
> > Fixups for bus 00
> > Bus scan for 00 returning with max=00
> > Scanning bus 10
> > Fixups for bus 10
> > Bus scan for 10 returning with max=10
> > PCI: Cannot allocate resource region 4 of device 00:00.0
> >
> > So, the 0x10 incrementation is fine, but the agp bus is not working
> > correctly, and thus the agp graphic card not recognized (and thus no
> > radeonfb).
> >
> > That said, i believe the 00:00.0 (should that not be 10:00.0 because of
> > the renumbering) is the Marvell Discovery II pci controller, and the
> > actual agp card should be found just behind.
>
> What video card is this ?

A Radeon 9200 SE 64M, from Xpert Vision. The kernel is patched for
Radeon 9200 SE support. (Both i have are Yd models).

> It's possible that your config ops for the AGP don't work properly. Make
> sure you are really generating type 0 config cycles.

Ok, i will check again, altough no more tonight.

> Also, this "cannot allocate resource region" smells bad. Can
> you check what the BARs of the host bridge are supposed to mean ?

Ok, will do.

> I hope it's not yet another brain dead bridge that puts the system memory
> in a BAR... in which case we'll have to "hide" it from linux PCI subsystem.

Mmm, seems  Rob Baxter from Synergy Microsystems are already running
linux on systems with the Discovery II north bridge, so i don't think
this is it. I will check this tomorrow though.

Thanks again for your help,

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Rob Baxter]

On Sat, Jan 03, 2004 at 12:12:23PM +1100, Benjamin Herrenschmidt wrote:
>
> > This was necessary because the PCI driver (scan/enumeration code) was using
> > these same config calls that was being fixed up.  (A chicken and egg issue?)
>
> This should still not be necessary... Before scanning a bus, the PCI code
> will properly set first_busno, thus the config access methods will work.
>
> The only problem I spotted in 2.4 (and afaik, that is fixed in 2.6) is
> that when P2P bridges are moved around, we could end up with a temporary
> wrong state where 2 of these on a given bus segment would try to decode
> the same bus numbers, thus screwing up probe. This is why I added a
> skew value of 0x10 between each host on some machines.
>
> Ben.

The problem I saw with the dual PCI buses is that the second Discovery PCI
bridge was assigned a bus number of 01 (i.e., first_busno), or 02 if a PtP
bridge was located off of the first PCI bus of the Discovery, prior to the
scan.  The Discovery does not respond to PCI configuration cycles when the
bus number is not zero--the reason for bus fixup routine.

--
-Rob

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> The problem I saw with the dual PCI buses is that the second Discovery PCI
> bridge was assigned a bus number of 01 (i.e., first_busno), or 02 if a PtP
> bridge was located off of the first PCI bus of the Discovery, prior to the
> scan.  The Discovery does not respond to PCI configuration cycles when the
> bus number is not zero--the reason for bus fixup routine.

Sure, you need to issue a type-0 config cycle on a host bridge for devices
that are on the same segment... it's possible that in your case, a type
0 and and type 1 are the same format with bus 0 for the first, in which
case, just do that (no need to "fixup" the bus number, it's really not
related to bus numbers, but to the type of config cycles issued to the
PCI bus)

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Sun, Jan 04, 2004 at 11:06:08PM +0100, Sven Luther wrote:
>
> On Mon, Jan 05, 2004 at 08:45:24AM +1100, Benjamin Herrenschmidt wrote:
> >
> >
> > > PCI: Probing PCI hardware
> > > Scanning bus 00
> > > Found 00:00 [11ab/6460] 000600 00
> > > Found 00:08 [1106/3044] 000c00 00
> > > Found 00:28 [1000/0001] 000100 00
> > > Found 00:60 [1106/8231] 000601 00
> > > Found 00:61 [1106/0571] 000101 00
> > > Found 00:62 [1106/3038] 000c03 00
> > > Found 00:63 [1106/3038] 000c03 00
> > > Found 00:64 [1106/8235] 000000 00
> > > Found 00:65 [1106/3058] 000401 00
> > > Found 00:66 [1106/3068] 000780 00
> > > Found 00:68 [1106/3065] 000200 00
> > > Fixups for bus 00
> > > Bus scan for 00 returning with max=00
> > > Scanning bus 10
> > > Fixups for bus 10
> > > Bus scan for 10 returning with max=10
> > > PCI: Cannot allocate resource region 4 of device 00:00.0
> > >
> > > So, the 0x10 incrementation is fine, but the agp bus is not working
> > > correctly, and thus the agp graphic card not recognized (and thus no
> > > radeonfb).
> > >
> > > That said, i believe the 00:00.0 (should that not be 10:00.0 because of
> > > the renumbering) is the Marvell Discovery II pci controller, and the
> > > actual agp card should be found just behind.
> >
> > What video card is this ?
>
> A Radeon 9200 SE 64M, from Xpert Vision. The kernel is patched for
> Radeon 9200 SE support. (Both i have are Yd models).

Ok, i got it to work finally, at least upto asking me for a root
filesystem, which i don't yet have on this harddisk.

Now, I have some technical questions about how to best do a few things.

I was recommended to set the ppc_md.pci_exclude_device so that the
device 0 (the marvell bridge itself) is not seen by linux. I did this by
following the 4xx example, which should be ok.

But, i have to do access some address which i need to ioremap. I created
a hose->cfg_peg2_magic to to put this ioremapped address in. I guess
this is not the most clean way of doing this or something, any hint on
how to best do it ? I need to set this in chrp_find_bridges or something
such, and use the address in the read/write_config functions.

Also, i had to manually set hose->bus_offset = 0x10, since that didn't
seem to be set automatically. I don't know why though.

And finally, about the stuff which blocked me most, i am somewhat
bewildered. I use this :

        data = 0x80000000 | ((dev->bus->number - hose->bus_offset) << 16)
		| (dev->devfn << 8) | (offset & 0xff);

As the address to write to, but the indirect pci stuff uses offset &
0xfc, which blanks bits 0 & 1 for pci config type selection. What am i
misunderstanding or doing wrong here ?

A, and a last question to Rob Baxter, did you manage to get the builtin
gigabyte ethernet port to work, and if yes, with which code, an
existing driver or some home built driver. Can you eventually share the
code or something such ?

Anyway, thanks for your help.

Friendly,

Sven Luther


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> Ok, i got it to work finally, at least upto asking me for a root
> filesystem, which i don't yet have on this harddisk.
>
> Now, I have some technical questions about how to best do a few things.
>
> I was recommended to set the ppc_md.pci_exclude_device so that the
> device 0 (the marvell bridge itself) is not seen by linux. I did this by
> following the 4xx example, which should be ok.

Actually, just hiding the BARs with a quirk should be enough (what
I do for the CPC710)...

> But, i have to do access some address which i need to ioremap. I created
> a hose->cfg_peg2_magic to to put this ioremapped address in. I guess
> this is not the most clean way of doing this or something, any hint on
> how to best do it ? I need to set this in chrp_find_bridges or something
> such, and use the address in the read/write_config functions.

What for ? you can eventually ioremap some lower address in
host->cfg_addr or cfg_data and then use offsets from these addresses,
though it's not recommended to ioremap too large spaces. You can also
just ioremap that in a global once... I do that for a few things in
pmac_feature.c, like the northbridge registers.

> Also, i had to manually set hose->bus_offset = 0x10, since that didn't
> seem to be set automatically. I don't know why though.

Why do you need hose->bus_offset ? For indirect_pci ? Well... that's
a kludge, you should rather fix indirect_pci to use first_busno
instead... I don't know where this bus_offset comes from in the first
place, it's defined in pci_controller but not used at all in 2.6...

(BTW. You should really work on 2.6, not 2.4...)

> And finally, about the stuff which blocked me most, i am somewhat
> bewildered. I use this :
>
>         data = 0x80000000 | ((dev->bus->number - hose->bus_offset) << 16)
> 		| (dev->devfn << 8) | (offset & 0xff);
>
> As the address to write to, but the indirect pci stuff uses offset &
> 0xfc, which blanks bits 0 & 1 for pci config type selection. What am i
> misunderstanding or doing wrong here ?

Normally, you write the offset 32 bits aligned, and then you add those
2 missing bits to the cycle used to actually read the config space
(access to the data register).

But then... read your bridge spec.

> A, and a last question to Rob Baxter, did you manage to get the builtin
> gigabyte ethernet port to work, and if yes, with which code, an

gigabit you mean ? :) gigabyte would be nice though...

> existing driver or some home built driver. Can you eventually share the
> code or something such ?

What cell is it ?

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Marcus Barrow]

	Hi Sven:

	I think there is some basic brokeness in the Discovery
	specific code. I've been meaning to go and look at some
	of the original code, to see if some stuff was left on
	the floor.

	On our evalution boards the P2P registers are not set
	up. So you indeed have two PCI bus 0. The P2P register
	for PCI bus 1 should be initialized to identify it as
	bus 1, or whatever the right value is considering
	any more bridges found on bus 0.

	If you insert pci cards which contain bridges on them,
	they only work on the bus 0 slots...

	Do you have something like ppcboot or uboot starting
	your system or just a kernel? If you are using a real
	boot rom, you could try initializing the p2p registers
	by hand, then booting...

	I'm hoping to work on this in the next few days.

				Regards, Marcus Barrow




On Mon, 2004-01-05 at 11:40, Sven Luther wrote:
>
> On Sun, Jan 04, 2004 at 11:06:08PM +0100, Sven Luther wrote:
> >
> > On Mon, Jan 05, 2004 at 08:45:24AM +1100, Benjamin Herrenschmidt wrote:
> > >
> > >
> > > > PCI: Probing PCI hardware
> > > > Scanning bus 00
> > > > Found 00:00 [11ab/6460] 000600 00
> > > > Found 00:08 [1106/3044] 000c00 00
> > > > Found 00:28 [1000/0001] 000100 00
> > > > Found 00:60 [1106/8231] 000601 00
> > > > Found 00:61 [1106/0571] 000101 00
> > > > Found 00:62 [1106/3038] 000c03 00
> > > > Found 00:63 [1106/3038] 000c03 00
> > > > Found 00:64 [1106/8235] 000000 00
> > > > Found 00:65 [1106/3058] 000401 00
> > > > Found 00:66 [1106/3068] 000780 00
> > > > Found 00:68 [1106/3065] 000200 00
> > > > Fixups for bus 00
> > > > Bus scan for 00 returning with max=00
> > > > Scanning bus 10
> > > > Fixups for bus 10
> > > > Bus scan for 10 returning with max=10
> > > > PCI: Cannot allocate resource region 4 of device 00:00.0
> > > >
> > > > So, the 0x10 incrementation is fine, but the agp bus is not working
> > > > correctly, and thus the agp graphic card not recognized (and thus no
> > > > radeonfb).
> > > >
> > > > That said, i believe the 00:00.0 (should that not be 10:00.0 because of
> > > > the renumbering) is the Marvell Discovery II pci controller, and the
> > > > actual agp card should be found just behind.
> > >
> > > What video card is this ?
> >
> > A Radeon 9200 SE 64M, from Xpert Vision. The kernel is patched for
> > Radeon 9200 SE support. (Both i have are Yd models).
>
> Ok, i got it to work finally, at least upto asking me for a root
> filesystem, which i don't yet have on this harddisk.
>
> Now, I have some technical questions about how to best do a few things.
>
> I was recommended to set the ppc_md.pci_exclude_device so that the
> device 0 (the marvell bridge itself) is not seen by linux. I did this by
> following the 4xx example, which should be ok.
>
> But, i have to do access some address which i need to ioremap. I created
> a hose->cfg_peg2_magic to to put this ioremapped address in. I guess
> this is not the most clean way of doing this or something, any hint on
> how to best do it ? I need to set this in chrp_find_bridges or something
> such, and use the address in the read/write_config functions.
>
> Also, i had to manually set hose->bus_offset = 0x10, since that didn't
> seem to be set automatically. I don't know why though.
>
> And finally, about the stuff which blocked me most, i am somewhat
> bewildered. I use this :
>
>         data = 0x80000000 | ((dev->bus->number - hose->bus_offset) << 16)
> 		| (dev->devfn << 8) | (offset & 0xff);
>
> As the address to write to, but the indirect pci stuff uses offset &
> 0xfc, which blanks bits 0 & 1 for pci config type selection. What am i
> misunderstanding or doing wrong here ?
>
> A, and a last question to Rob Baxter, did you manage to get the builtin
> gigabyte ethernet port to work, and if yes, with which code, an
> existing driver or some home built driver. Can you eventually share the
> code or something such ?
>
> Anyway, thanks for your help.
>
> Friendly,
>
> Sven Luther
>
>
>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Tue, Jan 06, 2004 at 08:28:15AM +1100, Benjamin Herrenschmidt wrote:
>
> > Ok, i got it to work finally, at least upto asking me for a root
> > filesystem, which i don't yet have on this harddisk.
> >
> > Now, I have some technical questions about how to best do a few things.
> >
> > I was recommended to set the ppc_md.pci_exclude_device so that the
> > device 0 (the marvell bridge itself) is not seen by linux. I did this by
> > following the 4xx example, which should be ok.
>
> Actually, just hiding the BARs with a quirk should be enough (what
> I do for the CPC710)...

Mmm, will look into this. Actually, linux should not write to those, but
read access should be ok.

> > But, i have to do access some address which i need to ioremap. I created
> > a hose->cfg_peg2_magic to to put this ioremapped address in. I guess
> > this is not the most clean way of doing this or something, any hint on
> > how to best do it ? I need to set this in chrp_find_bridges or something
> > such, and use the address in the read/write_config functions.
>
> What for ? you can eventually ioremap some lower address in

Well, these are the magic let's transform a standard pci bus into an agp
one, i don't know exactly what they do, just that i have to do it.

> host->cfg_addr or cfg_data and then use offsets from these addresses,
> though it's not recommended to ioremap too large spaces. You can also

Well, i tried ioremapping 0x10000, but without much success
(cfg_addr/data is at cf8/cfc, and this magic stuff is at f118/f11c). I
just got a sig 11 OOPS, so ...

> just ioremap that in a global once... I do that for a few things in
> pmac_feature.c, like the northbridge registers.

Yeah, i have seen.

> > Also, i had to manually set hose->bus_offset = 0x10, since that didn't
> > seem to be set automatically. I don't know why though.
>
> Why do you need hose->bus_offset ? For indirect_pci ? Well... that's

Nope, i checked, and it was trying to read the bus 0x10, while the bus
is in reallity 0. That said, if i understood stuff correctly, it should
be ok, since type 0 config ignores bus and device, right ?

> a kludge, you should rather fix indirect_pci to use first_busno
> instead... I don't know where this bus_offset comes from in the first
> place, it's defined in pci_controller but not used at all in 2.6...

Yeah, nor in 2.4, if my grep is correct. I can't use indirect_pci for
the second bus though.

> (BTW. You should really work on 2.6, not 2.4...)

Ah, yes, but 2.4 means suppoprt for debian-installer. Once that works, i
will work on 2.6.x. Also, 2.4 provides me a known working base.

> > And finally, about the stuff which blocked me most, i am somewhat
> > bewildered. I use this :
> >
> >         data = 0x80000000 | ((dev->bus->number - hose->bus_offset) << 16)
> > 		| (dev->devfn << 8) | (offset & 0xff);
> >
> > As the address to write to, but the indirect pci stuff uses offset &
> > 0xfc, which blanks bits 0 & 1 for pci config type selection. What am i
> > misunderstanding or doing wrong here ?
>
> Normally, you write the offset 32 bits aligned, and then you add those
> 2 missing bits to the cycle used to actually read the config space
> (access to the data register).
>
> But then... read your bridge spec.

This is which bewilders me, the bridge spec speak about type 0 and type
1, and using the last 2 bits for this.

> > A, and a last question to Rob Baxter, did you manage to get the builtin
> > gigabyte ethernet port to work, and if yes, with which code, an
>
> gigabit you mean ? :) gigabyte would be nice though...

Yeah, whatever.

> > existing driver or some home built driver. Can you eventually share the
> > code or something such ?
>
> What cell is it ?

The Marvell Discovery II has (one, two or three) gigabit ethernets
included, these do not appear on the pci bus, but need to be programmed
directly with the NB registers. On the other hand, the sk98 driver
provides driver for various Marvell gigabit ethernet.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> Mmm, will look into this. Actually, linux should not write to those, but
> read access should be ok.

If the BARs contain address ranges that will confuse linux resource
management (like RAM location), then you have to hide them completely.

> Well, i tried ioremapping 0x10000, but without much success
> (cfg_addr/data is at cf8/cfc, and this magic stuff is at f118/f11c). I
> just got a sig 11 OOPS, so ...

I don't fully understand... If you are doing IO cycles, there's no
ioremapping needed at all at this point, just use the one that is done
once for the bridge IO space. neither cf8/cfc not f118 look like
addresses that can be ioremap'ed anyway...

> > just ioremap that in a global once... I do that for a few things in
> > pmac_feature.c, like the northbridge registers.
>
> Yeah, i have seen.
>
> > > Also, i had to manually set hose->bus_offset = 0x10, since that didn't
> > > seem to be set automatically. I don't know why though.
> >
> > Why do you need hose->bus_offset ? For indirect_pci ? Well... that's
>
> Nope, i checked, and it was trying to read the bus 0x10, while the bus
> is in reallity 0. That said, if i understood stuff correctly, it should
> be ok, since type 0 config ignores bus and device, right ?

Yes. And for type 1, just issue a cycle to (bus - hose->first_busno)

> > a kludge, you should rather fix indirect_pci to use first_busno
> > instead... I don't know where this bus_offset comes from in the first
> > place, it's defined in pci_controller but not used at all in 2.6...
>
> Yeah, nor in 2.4, if my grep is correct. I can't use indirect_pci for
> the second bus though.
>
> > (BTW. You should really work on 2.6, not 2.4...)
>
> Ah, yes, but 2.4 means suppoprt for debian-installer. Once that works, i
> will work on 2.6.x. Also, 2.4 provides me a known working base.

I still don't like the idea of new dev. beeing done on 2.4...

> This is which bewilders me, the bridge spec speak about type 0 and type
> 1, and using the last 2 bits for this.

Yes. Usually, you don't put the last 2 bits of the offsets in cfg_addr,
you crop that value to the closest 32 bits boundary (or 64 bits in some
chips), and then you "offset" the cfg_data access.

For example, for a 8 bits access at an offset of 7, you would use 4
as the offset in cfg_addr, and then do a 8 bits access at cfg_data + 3


> The Marvell Discovery II has (one, two or three) gigabit ethernets
> included, these do not appear on the pci bus, but need to be programmed
> directly with the NB registers. On the other hand, the sk98 driver
> provides driver for various Marvell gigabit ethernet.

Argh ????? They don't appear on PCI ? What piece of SHIT is this bridge ?

Really totally insane.

I strongly suggest you still match them as PCI devices, eventually using
the bridge device itself as the match for the driver.... The DMA mapping
ops are designed to work with PCI anyway.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Mon, Jan 05, 2004 at 04:38:50PM -0500, Marcus Barrow wrote:
>
> 	Hi Sven:
>
> 	I think there is some basic brokeness in the Discovery
> 	specific code. I've been meaning to go and look at some
> 	of the original code, to see if some stuff was left on
> 	the floor.

Mmm. What code are you speaking about here ?

> 	On our evalution boards the P2P registers are not set
> 	up. So you indeed have two PCI bus 0. The P2P register
> 	for PCI bus 1 should be initialized to identify it as
> 	bus 1, or whatever the right value is considering
> 	any more bridges found on bus 0.

Mmm, i understand that the 2.6.x pci domain will make this not
necessary. I might be wrong though.

> 	If you insert pci cards which contain bridges on them,
> 	they only work on the bus 0 slots...

bus 1 is used as an agp bus anyway, and i doubt there are many agp
graphic cards out there with pci bridges on them. When i inserted my
Appian Jeronimo 2000 in that board, it refused to powerup at all, but
then, maybe it is drawing too much power or something. After all, it has
a 3dlabs gamma and two permedia3 on it.

> 	Do you have something like ppcboot or uboot starting
> 	your system or just a kernel? If you are using a real

Well, there is a customized SmartFirmware OpenFirmware on the board. I
even have (non-free though) access to the source of it. The board is
mostly a CHRP design.

> 	boot rom, you could try initializing the p2p registers
> 	by hand, then booting...

The Firmware guy told me don't touch it though.

> 	I'm hoping to work on this in the next few days.

Ok, great.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Tue, Jan 06, 2004 at 09:12:06AM +1100, Benjamin Herrenschmidt wrote:
>
> > Mmm, will look into this. Actually, linux should not write to those, but
> > read access should be ok.
>
> If the BARs contain address ranges that will confuse linux resource
> management (like RAM location), then you have to hide them completely.

Mmm, will check.

BTW, X has problems with pci config space access. It simply opens the
/proc/bus/pci/<bus>/<dev>.<func> stuff, and is not happy with the
result.

Is the above just a plain ioremap of config space or something such, or
does the reads there use the pci access functions ?

> > Well, i tried ioremapping 0x10000, but without much success
> > (cfg_addr/data is at cf8/cfc, and this magic stuff is at f118/f11c). I
> > just got a sig 11 OOPS, so ...
>
> I don't fully understand... If you are doing IO cycles, there's no
> ioremapping needed at all at this point, just use the one that is done
> once for the bridge IO space. neither cf8/cfc not f118 look like
> addresses that can be ioremap'ed anyway...

Well, it is f1000cf8/cfc and f100f118/f11c naturally, and the second is
needed to enable a hardware kludge to make the second pci bus look like
agp. I don't really have the details of such, but i need to write to the
f118/f11c before/after each pci access to the second pci bus, as well as
disable the CPU interrupts for doing the access.

> > > just ioremap that in a global once... I do that for a few things in
> > > pmac_feature.c, like the northbridge registers.
> >
> > Yeah, i have seen.
> >
> > > > Also, i had to manually set hose->bus_offset = 0x10, since that didn't
> > > > seem to be set automatically. I don't know why though.
> > >
> > > Why do you need hose->bus_offset ? For indirect_pci ? Well... that's
> >
> > Nope, i checked, and it was trying to read the bus 0x10, while the bus
> > is in reallity 0. That said, if i understood stuff correctly, it should
> > be ok, since type 0 config ignores bus and device, right ?
>
> Yes. And for type 1, just issue a cycle to (bus - hose->first_busno)

How are the config access functions told that they shall do a type 0 or
1 access ? I have seen there is something in the pci_indirect functions
that do this, but not sure i can do it here.

> > > a kludge, you should rather fix indirect_pci to use first_busno
> > > instead... I don't know where this bus_offset comes from in the first
> > > place, it's defined in pci_controller but not used at all in 2.6...
> >
> > Yeah, nor in 2.4, if my grep is correct. I can't use indirect_pci for
> > the second bus though.
> >
> > > (BTW. You should really work on 2.6, not 2.4...)
> >
> > Ah, yes, but 2.4 means suppoprt for debian-installer. Once that works, i
> > will work on 2.6.x. Also, 2.4 provides me a known working base.
>
> I still don't like the idea of new dev. beeing done on 2.4...

Well, once i get it working, i will go on to 2.6, promised :))

That said, i was a bit disapointed by 2.6. If you remember, i had some
problems with 2.4, since the second IDE bus uses a different interrupt
than the first one, so i used a kludge in the via-ide driver to make it
work. The 2.6 ide driver has per ide channel setup, but still does get
the primary ide interrupt for the second channel too.

> > This is which bewilders me, the bridge spec speak about type 0 and type
> > 1, and using the last 2 bits for this.
>
> Yes. Usually, you don't put the last 2 bits of the offsets in cfg_addr,
> you crop that value to the closest 32 bits boundary (or 64 bits in some
> chips), and then you "offset" the cfg_data access.
>
> For example, for a 8 bits access at an offset of 7, you would use 4
> as the offset in cfg_addr, and then do a 8 bits access at cfg_data + 3

Ok, understand better now.

> > The Marvell Discovery II has (one, two or three) gigabit ethernets
> > included, these do not appear on the pci bus, but need to be programmed
> > directly with the NB registers. On the other hand, the sk98 driver
> > provides driver for various Marvell gigabit ethernet.
>
> Argh ????? They don't appear on PCI ? What piece of SHIT is this bridge ?

Well, the Discovery II use a internal crossbar switch, and the ethernet
are on the same level as the pci buses. This makes for very efficient
networking i guess, but has problems. In fact, each of these ones has
the same priority as each pci bus.

I believe it should be possible to have each of these appear as an
independent pci bus or something ?

> Really totally insane.

Well, i am no hardware guy, so i cannot make judgement on this, but i
guess there are worse hardware out there.

> I strongly suggest you still match them as PCI devices, eventually using
> the bridge device itself as the match for the driver.... The DMA mapping
> ops are designed to work with PCI anyway.

Mmm. Will have to look into that.

Again, thanks for your help,

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

>
> Mmm, i understand that the 2.6.x pci domain will make this not
> necessary. I might be wrong though.

What do those register actually do ? Is there a spec of that
host bridge I can download somewhere ?

> > 	If you insert pci cards which contain bridges on them,
> > 	they only work on the bus 0 slots...
>
> bus 1 is used as an agp bus anyway, and i doubt there are many agp
> graphic cards out there with pci bridges on them. When i inserted my
> Appian Jeronimo 2000 in that board, it refused to powerup at all, but
> then, maybe it is drawing too much power or something. After all, it has
> a 3dlabs gamma and two permedia3 on it.
>
> > 	Do you have something like ppcboot or uboot starting
> > 	your system or just a kernel? If you are using a real
>
> Well, there is a customized SmartFirmware OpenFirmware on the board. I
> even have (non-free though) access to the source of it. The board is
> mostly a CHRP design.

It has an x86 BIOS emulator for running the firmwares of the video cards
?

> 	boot rom, you could try initializing the p2p registers
> > 	by hand, then booting...
>
> The Firmware guy told me don't touch it though.
>
> > 	I'm hoping to work on this in the next few days.
>
> Ok, great.
>
> Friendly,
>
> Sven Luther
>
--
Benjamin Herrenschmidt <benh@kernel.crashing.org>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

On Tue, 2004-01-06 at 18:39, Sven Luther wrote:
> On Tue, Jan 06, 2004 at 09:12:06AM +1100, Benjamin Herrenschmidt wrote:
> >
> > > Mmm, will look into this. Actually, linux should not write to those, but
> > > read access should be ok.
> >
> > If the BARs contain address ranges that will confuse linux resource
> > management (like RAM location), then you have to hide them completely.
>
> Mmm, will check.
>
> BTW, X has problems with pci config space access. It simply opens the
> /proc/bus/pci/<bus>/<dev>.<func> stuff, and is not happy with the
> result.
>
> Is the above just a plain ioremap of config space or something such, or
> does the reads there use the pci access functions ?

They should use the PCI access function, could be your access size
handling that is wrong (the offset masking stuff), or maybe XFree
shokes on the host bridge

> > > Well, i tried ioremapping 0x10000, but without much success
> > > (cfg_addr/data is at cf8/cfc, and this magic stuff is at f118/f11c). I
> > > just got a sig 11 OOPS, so ...
> >
> > I don't fully understand... If you are doing IO cycles, there's no
> > ioremapping needed at all at this point, just use the one that is done
> > once for the bridge IO space. neither cf8/cfc not f118 look like
> > addresses that can be ioremap'ed anyway...
>
> Well, it is f1000cf8/cfc and f100f118/f11c naturally, and the second is
> needed to enable a hardware kludge to make the second pci bus look like
> agp. I don't really have the details of such, but i need to write to the
> f118/f11c before/after each pci access to the second pci bus, as well as
> disable the CPU interrupts for doing the access.
>
> > > > just ioremap that in a global once... I do that for a few things in
> > > > pmac_feature.c, like the northbridge registers.
> > >
> > > Yeah, i have seen.
> > >
> > > > > Also, i had to manually set hose->bus_offset = 0x10, since that didn't
> > > > > seem to be set automatically. I don't know why though.
> > > >
> > > > Why do you need hose->bus_offset ? For indirect_pci ? Well... that's
> > >
> > > Nope, i checked, and it was trying to read the bus 0x10, while the bus
> > > is in reallity 0. That said, if i understood stuff correctly, it should
> > > be ok, since type 0 config ignores bus and device, right ?
> >
> > Yes. And for type 1, just issue a cycle to (bus - hose->first_busno)
>
> How are the config access functions told that they shall do a type 0 or
> 1 access ? I have seen there is something in the pci_indirect functions
> that do this, but not sure i can do it here.

Type 0 is an access to the primary segment (doesn't contain a bus
number), type 1 is to be forwarded to another bus segment by a P2P
bridge. So for anything directly attached to the host bridge, it's a
type 0 access. Anything else is type 1. Typically, if the bus number of
your "target" == hose->first_busno, it's type 0, else type 1

> > > > a kludge, you should rather fix indirect_pci to use first_busno
> > > > instead... I don't know where this bus_offset comes from in the first
> > > > place, it's defined in pci_controller but not used at all in 2.6...
> > >
> > > Yeah, nor in 2.4, if my grep is correct. I can't use indirect_pci for
> > > the second bus though.
> > >
> > > > (BTW. You should really work on 2.6, not 2.4...)
> > >
> > > Ah, yes, but 2.4 means suppoprt for debian-installer. Once that works, i
> > > will work on 2.6.x. Also, 2.4 provides me a known working base.
> >
> > I still don't like the idea of new dev. beeing done on 2.4...
>
> Well, once i get it working, i will go on to 2.6, promised :))
>
> That said, i was a bit disapointed by 2.6. If you remember, i had some
> problems with 2.4, since the second IDE bus uses a different interrupt
> than the first one, so i used a kludge in the via-ide driver to make it
> work. The 2.6 ide driver has per ide channel setup, but still does get
> the primary ide interrupt for the second channel too.

Well, there is no clean way currently to deal with that issue. If I get
a board, I can try to hack something better :)

> > > This is which bewilders me, the bridge spec speak about type 0 and type
> > > 1, and using the last 2 bits for this.
> >
> > Yes. Usually, you don't put the last 2 bits of the offsets in cfg_addr,
> > you crop that value to the closest 32 bits boundary (or 64 bits in some
> > chips), and then you "offset" the cfg_data access.
> >
> > For example, for a 8 bits access at an offset of 7, you would use 4
> > as the offset in cfg_addr, and then do a 8 bits access at cfg_data + 3
>
> Ok, understand better now.
>
> > > The Marvell Discovery II has (one, two or three) gigabit ethernets
> > > included, these do not appear on the pci bus, but need to be programmed
> > > directly with the NB registers. On the other hand, the sk98 driver
> > > provides driver for various Marvell gigabit ethernet.
> >
> > Argh ????? They don't appear on PCI ? What piece of SHIT is this bridge ?
>
> Well, the Discovery II use a internal crossbar switch, and the ethernet
> are on the same level as the pci buses. This makes for very efficient
> networking i guess, but has problems. In fact, each of these ones has
> the same priority as each pci bus.
>
> I believe it should be possible to have each of these appear as an
> independent pci bus or something ?

They could have appeared as on-chip PCI devices on a "pseudo-bus", but
we can eventually just match with the host's PCI device.

> > Really totally insane.
>
> Well, i am no hardware guy, so i cannot make judgement on this, but i
> guess there are worse hardware out there.

Yah, sure, all hardware is broken some way :)

> > I strongly suggest you still match them as PCI devices, eventually using
> > the bridge device itself as the match for the driver.... The DMA mapping
> > ops are designed to work with PCI anyway.
>
> Mmm. Will have to look into that.
>
> Again, thanks for your help,
>
> Friendly,
>
> Sven Luther
--
Benjamin Herrenschmidt <benh@kernel.crashing.org>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Tue, Jan 06, 2004 at 07:00:24PM +1100, Benjamin Herrenschmidt wrote:
> On Tue, 2004-01-06 at 18:39, Sven Luther wrote:
> > On Tue, Jan 06, 2004 at 09:12:06AM +1100, Benjamin Herrenschmidt wrote:
> > >
> > > > Mmm, will look into this. Actually, linux should not write to those, but
> > > > read access should be ok.
> > >
> > > If the BARs contain address ranges that will confuse linux resource
> > > management (like RAM location), then you have to hide them completely.
> >
> > Mmm, will check.
> >
> > BTW, X has problems with pci config space access. It simply opens the
> > /proc/bus/pci/<bus>/<dev>.<func> stuff, and is not happy with the
> > result.
> >
> > Is the above just a plain ioremap of config space or something such, or
> > does the reads there use the pci access functions ?
>
> They should use the PCI access function, could be your access size
> handling that is wrong (the offset masking stuff), or maybe XFree
> shokes on the host bridge

Don't exactly know what happens, it dies with :

(II) Host-to-PCI bridge:
(II) Bus 16: bridge is at (0:0:0), (-1,16,0), BCTRL: 0x0008 (VGA_EN is set)
(II) Bus 16 I/O range:
	[0] -1	0	0x00000000 - 0x0000ffff (0x10000) IX[B]
(II) Bus 16 non-prefetchable memory range:
	[0] -1	0	0x00000000 - 0xffffffff (0x0) MX[B]
(II) Bus 16 prefetchable memory range:
	[0] -1	0	0x00000000 - 0xffffffff (0x0) MX[B]

(full log on : http://zapek.com/misc/XFree86.0.log)

Need to install my box, and do some testing myself. X doesn't kill the
box though, which is rather nice.

> Type 0 is an access to the primary segment (doesn't contain a bus
> number), type 1 is to be forwarded to another bus segment by a P2P
> bridge. So for anything directly attached to the host bridge, it's a
> type 0 access. Anything else is type 1. Typically, if the bus number of
> your "target" == hose->first_busno, it's type 0, else type 1

Yep, except we have two pci controllers, and it should be type 0 for
both of them.

> > That said, i was a bit disapointed by 2.6. If you remember, i had some
> > problems with 2.4, since the second IDE bus uses a different interrupt
> > than the first one, so i used a kludge in the via-ide driver to make it
> > work. The 2.6 ide driver has per ide channel setup, but still does get
> > the primary ide interrupt for the second channel too.
>
> Well, there is no clean way currently to deal with that issue. If I get
> a board, I can try to hack something better :)

Will see. How long would you need the board, and you are in Australia
still, right ?

> > > Argh ????? They don't appear on PCI ? What piece of SHIT is this bridge ?
> >
> > Well, the Discovery II use a internal crossbar switch, and the ethernet
> > are on the same level as the pci buses. This makes for very efficient
> > networking i guess, but has problems. In fact, each of these ones has
> > the same priority as each pci bus.
> >
> > I believe it should be possible to have each of these appear as an
> > independent pci bus or something ?
>
> They could have appeared as on-chip PCI devices on a "pseudo-bus", but
> we can eventually just match with the host's PCI device.

Ok. but this can also be faked or something ? But, how can we match with
the host PCI device, if we are going to hide it ?

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Tue, Jan 06, 2004 at 06:56:05PM +1100, Benjamin Herrenschmidt wrote:
>
> >
> > Mmm, i understand that the 2.6.x pci domain will make this not
> > necessary. I might be wrong though.
>
> What do those register actually do ? Is there a spec of that
> host bridge I can download somewhere ?

There is a spec, but under NDA, i don't know if you can freely get a
hand on it.

> > > 	If you insert pci cards which contain bridges on them,
> > > 	they only work on the bus 0 slots...
> >
> > bus 1 is used as an agp bus anyway, and i doubt there are many agp
> > graphic cards out there with pci bridges on them. When i inserted my
> > Appian Jeronimo 2000 in that board, it refused to powerup at all, but
> > then, maybe it is drawing too much power or something. After all, it has
> > a 3dlabs gamma and two permedia3 on it.
> >
> > > 	Do you have something like ppcboot or uboot starting
> > > 	your system or just a kernel? If you are using a real
> >
> > Well, there is a customized SmartFirmware OpenFirmware on the board. I
> > even have (non-free though) access to the source of it. The board is
> > mostly a CHRP design.
>
> It has an x86 BIOS emulator for running the firmwares of the video cards ?

Yep, that too. Not sure if the standard OF graphic cards would work too,
i have none of such to test though. tdfx fbdev seems broken with a G4,
since it uses the old access macros, and doesn't do eieios or something
such. Didn't look at recent code though, and don't have voodoo cards
handy anyway.

That said, the OF only uses textmode, since fb8 support would mean vesa
3.0 compliant card limit.

There are some embedded drivers too, but they are not activated i think.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Geert Uytterhoeven]

On Tue, 6 Jan 2004, Sven Luther wrote:
> On Tue, Jan 06, 2004 at 07:00:24PM +1100, Benjamin Herrenschmidt wrote:
> > Type 0 is an access to the primary segment (doesn't contain a bus
> > number), type 1 is to be forwarded to another bus segment by a P2P
> > bridge. So for anything directly attached to the host bridge, it's a
> > type 0 access. Anything else is type 1. Typically, if the bus number of
> > your "target" == hose->first_busno, it's type 0, else type 1
>
> Yep, except we have two pci controllers, and it should be type 0 for
> both of them.

Yep, two hoses, with different hose->first_busno.

> > They could have appeared as on-chip PCI devices on a "pseudo-bus", but
> > we can eventually just match with the host's PCI device.
>
> Ok. but this can also be faked or something ? But, how can we match with
> the host PCI device, if we are going to hide it ?

Hide the memory BAR only, not the full PCI device.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Tue, Jan 06, 2004 at 03:40:30PM +0100, Geert Uytterhoeven wrote:
> On Tue, 6 Jan 2004, Sven Luther wrote:
> > On Tue, Jan 06, 2004 at 07:00:24PM +1100, Benjamin Herrenschmidt wrote:
> > > Type 0 is an access to the primary segment (doesn't contain a bus
> > > number), type 1 is to be forwarded to another bus segment by a P2P
> > > bridge. So for anything directly attached to the host bridge, it's a
> > > type 0 access. Anything else is type 1. Typically, if the bus number of
> > > your "target" == hose->first_busno, it's type 0, else type 1
> >
> > Yep, except we have two pci controllers, and it should be type 0 for
> > both of them.
>
> Yep, two hoses, with different hose->first_busno.

A, yes, right.

> > > They could have appeared as on-chip PCI devices on a "pseudo-bus", but
> > > we can eventually just match with the host's PCI device.
> >
> > Ok. but this can also be faked or something ? But, how can we match with
> > the host PCI device, if we are going to hide it ?
>
> Hide the memory BAR only, not the full PCI device.

Ok, will try that.

BTW, is there any reason the L2 cache is disabled by default in the
2.4.x kernels ?

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Rob Baxter]

On Tue, Jan 06, 2004 at 03:45:45PM +0100, Sven Luther wrote:
>
> On Tue, Jan 06, 2004 at 03:40:30PM +0100, Geert Uytterhoeven wrote:
> > On Tue, 6 Jan 2004, Sven Luther wrote:
> > > On Tue, Jan 06, 2004 at 07:00:24PM +1100, Benjamin Herrenschmidt wrote:
> > > > Type 0 is an access to the primary segment (doesn't contain a bus
> > > > number), type 1 is to be forwarded to another bus segment by a P2P
> > > > bridge. So for anything directly attached to the host bridge, it's a
> > > > type 0 access. Anything else is type 1. Typically, if the bus number of
> > > > your "target" == hose->first_busno, it's type 0, else type 1
> > >
> > > Yep, except we have two pci controllers, and it should be type 0 for
> > > both of them.
> >
> > Yep, two hoses, with different hose->first_busno.
>
> A, yes, right.
>
> > > > They could have appeared as on-chip PCI devices on a "pseudo-bus", but
> > > > we can eventually just match with the host's PCI device.
> > >
> > > Ok. but this can also be faked or something ? But, how can we match with
> > > the host PCI device, if we are going to hide it ?
> >
> > Hide the memory BAR only, not the full PCI device.
>
> Ok, will try that.

We hide all of the PCI resources (i.e., PCI BARs) by initializing all of
resources to zero.  This allows us to load drivers (e.g., ethernet) specific
to the Discovery.  Registers internal to the Discovery (via the PowerPC
bus) need to be initialized and queried for memory, PCI I/O, PCI memory,
and PCI prefecthable memory mappings (hopefully your firmware does this).

Here's the code from our pcibios_fixup:

	dev = NULL;
	while ((dev = pci_find_device(PCI_VENDOR_ID_GALILEO,
				PCI_DEVICE_ID_GALILEO_GT64260, dev))) {
		for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
			dev->resource[i].flags = 0;
			dev->resource[i].start = 0;
			dev->resource[i].end = 0;
		}
	}

>
> BTW, is there any reason the L2 cache is disabled by default in the
> 2.4.x kernels ?

We have it initialized and enabled.

>
> Friendly,
>
> Sven Luther
>

--
-Rob Baxter

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Tue, Jan 06, 2004 at 08:33:05AM -0700, Rob Baxter wrote:
>
> On Tue, Jan 06, 2004 at 03:45:45PM +0100, Sven Luther wrote:
> >
> > On Tue, Jan 06, 2004 at 03:40:30PM +0100, Geert Uytterhoeven wrote:
> > > On Tue, 6 Jan 2004, Sven Luther wrote:
> > > > On Tue, Jan 06, 2004 at 07:00:24PM +1100, Benjamin Herrenschmidt wrote:
> > > > > Type 0 is an access to the primary segment (doesn't contain
> > > > > a bus number), type 1 is to be forwarded to another bus
> > > > > segment by a P2P bridge. So for anything directly attached
> > > > > to the host bridge, it's a type 0 access. Anything else is
> > > > > type 1. Typically, if the bus number of your "target" ==
> > > > > hose->first_busno, it's type 0, else type 1
> > > >
> > > > Yep, except we have two pci controllers, and it should be type 0
> > > > for both of them.
> > >
> > > Yep, two hoses, with different hose->first_busno.
> >
> > A, yes, right.
> >
> > > > > They could have appeared as on-chip PCI devices on a
> > > > > "pseudo-bus", but we can eventually just match with the host's
> > > > > PCI device.
> > > >
> > > > Ok. but this can also be faked or something ? But, how can we
> > > > match with the host PCI device, if we are going to hide it ?
> > >
> > > Hide the memory BAR only, not the full PCI device.
> >
> > Ok, will try that.
>
> We hide all of the PCI resources (i.e., PCI BARs) by initializing all
> of resources to zero. This allows us to load drivers (e.g., ethernet)
> specific to the Discovery. Registers internal to the Discovery (via
> the PowerPC bus) need to be initialized and queried for memory, PCI
> I/O, PCI memory, and PCI prefecthable memory mappings (hopefully your
> firmware does this).
>
> Here's the code from our pcibios_fixup:
>
> 	dev = NULL;
> 	while ((dev = pci_find_device(PCI_VENDOR_ID_GALILEO,
> 				PCI_DEVICE_ID_GALILEO_GT64260, dev))) {
> 		for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
> 			dev->resource[i].flags = 0;
> 			dev->resource[i].start = 0;
> 			dev->resource[i].end = 0;
> 		}
> 	}

Ok, thanks, will try it out.

BTW, could you please do something to not reject my incoming email ?
Black listing frances biggest ISP is somewhat silly, and my last
experience with such black lister was not conclusive. They blacklisted
it on the basis of 50 or so spams received over one year period, most of
them not even comming from wanadoo but from other ISPs.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> Here's the code from our pcibios_fixup:
>
> 	dev = NULL;
> 	while ((dev = pci_find_device(PCI_VENDOR_ID_GALILEO,
> 				PCI_DEVICE_ID_GALILEO_GT64260, dev))) {
> 		for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
> 			dev->resource[i].flags = 0;
> 			dev->resource[i].start = 0;
> 			dev->resource[i].end = 0;
> 		}
> 	}

pcibios_fixup isn't the right place to do that ;) You should do this
from a pci quirk imho.

note that there's still a problem with XFree which will "see"  those
BARs and, according to the log posted by Sven, shoke. Sven, can you
try "hiding" the host bridge completely from the config ops and see
if that helps with XFree ? That's not a very good solution though,
we'll have to do something different about it. Now if only XFree
stopped mucking with the PCI bus...

> > BTW, is there any reason the L2 cache is disabled by default in the
> > 2.4.x kernels ?
>
> We have it initialized and enabled.

The kernel doesn't do anything to the L2 cache, it all depends what
you firmware does to it.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Marcus Barrow]

	The L2 (and L3 if you have one) is enabled
	some twenty lines into "ev64260_setup_arch()".
	Now it only sets the the enable bit, perhaps
	you also need to set bits for size and clock?

	Or possibly, i'm using some defunct tree...

				Marcus




On Tue, 2004-01-06 at 16:37, Benjamin Herrenschmidt wrote:
>
> > Here's the code from our pcibios_fixup:
> >
> > 	dev = NULL;
> > 	while ((dev = pci_find_device(PCI_VENDOR_ID_GALILEO,
> > 				PCI_DEVICE_ID_GALILEO_GT64260, dev))) {
> > 		for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
> > 			dev->resource[i].flags = 0;
> > 			dev->resource[i].start = 0;
> > 			dev->resource[i].end = 0;
> > 		}
> > 	}
>
> pcibios_fixup isn't the right place to do that ;) You should do this
> from a pci quirk imho.
>
> note that there's still a problem with XFree which will "see"  those
> BARs and, according to the log posted by Sven, shoke. Sven, can you
> try "hiding" the host bridge completely from the config ops and see
> if that helps with XFree ? That's not a very good solution though,
> we'll have to do something different about it. Now if only XFree
> stopped mucking with the PCI bus...
>
> > > BTW, is there any reason the L2 cache is disabled by default in the
> > > 2.4.x kernels ?
> >
> > We have it initialized and enabled.
>
> The kernel doesn't do anything to the L2 cache, it all depends what
> you firmware does to it.
>
> Ben.
>
>
>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Rob Baxter]

On Wed, Jan 07, 2004 at 08:37:36AM +1100, Benjamin Herrenschmidt wrote:
>
> > Here's the code from our pcibios_fixup:
> >
> > 	dev = NULL;
> > 	while ((dev = pci_find_device(PCI_VENDOR_ID_GALILEO,
> > 				PCI_DEVICE_ID_GALILEO_GT64260, dev))) {
> > 		for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
> > 			dev->resource[i].flags = 0;
> > 			dev->resource[i].start = 0;
> > 			dev->resource[i].end = 0;
> > 		}
> > 	}
>
> pcibios_fixup isn't the right place to do that ;) You should do this
> from a pci quirk imho.

agreed

>
> note that there's still a problem with XFree which will "see"  those
> BARs and, according to the log posted by Sven, shoke. Sven, can you
> try "hiding" the host bridge completely from the config ops and see
> if that helps with XFree ? That's not a very good solution though,
> we'll have to do something different about it. Now if only XFree
> stopped mucking with the PCI bus...

I have no problem with XFree working with a CT69030 (chips_drv.o) PMC
graphics card and an USB keyboard and mouse setup.

>
> > > BTW, is there any reason the L2 cache is disabled by default in the
> > > 2.4.x kernels ?
> >
> > We have it initialized and enabled.
>
> The kernel doesn't do anything to the L2 cache, it all depends what
> you firmware does to it.

That's true, but for the ppc stuff, there is initialization routines within
the kernel source.

>
> Ben.
>
>

--
-Rob

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

-
> > The kernel doesn't do anything to the L2 cache, it all depends what
> > you firmware does to it.
>
> That's true, but for the ppc stuff, there is initialization routines within
> the kernel source.

Yes, those are initially here to allow change of the setting via
/proc/sys/kernel/l2cr. I also used them to force passing a different
l2cr values via BootX and for setting up the L2 on the second CPU of
SMP machines and to restore it on wakeup from sleep on laptops

Nothing prevents you, of course, from setting L2 in your arch code
as well using those functions, but it's preferred to have the
firmware set it up properly in the first place.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Wed, Jan 07, 2004 at 08:37:36AM +1100, Benjamin Herrenschmidt wrote:
>
> > Here's the code from our pcibios_fixup:
> >
> > 	dev = NULL;
> > 	while ((dev = pci_find_device(PCI_VENDOR_ID_GALILEO,
> > 				PCI_DEVICE_ID_GALILEO_GT64260, dev))) {
> > 		for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
> > 			dev->resource[i].flags = 0;
> > 			dev->resource[i].start = 0;
> > 			dev->resource[i].end = 0;
> > 		}
> > 	}
>
> pcibios_fixup isn't the right place to do that ;) You should do this
> from a pci quirk imho.
>
> note that there's still a problem with XFree which will "see"  those
> BARs and, according to the log posted by Sven, shoke. Sven, can you

Not exactly, but i have yet to investigate this issue.

> try "hiding" the host bridge completely from the config ops and see
> if that helps with XFree ? That's not a very good solution though,
> we'll have to do something different about it. Now if only XFree
> stopped mucking with the PCI bus...

Well, XFree86 doesn't seem to see the device 0, since i hide it from the
config ops. Here is a lspci output :

00:01.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev 46)
00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c810 (rev 23)
00:0c.0 ISA bridge: VIA Technologies, Inc. VT8231 [PCI-to-ISA Bridge] (rev 10)
00:0c.1 IDE interface: VIA Technologies, Inc.  VT82C586A/B/VT82C686/A/B/VT8233/A/C/VT8235 PIPC Bus Master IDE (rev 06)
00:0c.2 USB Controller: VIA Technologies, Inc. USB (rev 1e)
00:0c.3 USB Controller: VIA Technologies, Inc. USB (rev 1e)
00:0c.4 Non-VGA unclassified device: VIA Technologies, Inc. VT8235 ACPI (rev 10)
00:0c.5 Multimedia audio controller: VIA Technologies, Inc. VT82C686 AC97 Audio Controller (rev 40)
00:0c.6 Communication controller: VIA Technologies, Inc. Intel 537 [AC97 Modem] (rev 20)
00:0d.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 51)
10:08.0 VGA compatible controller: ATI Technologies Inc: Unknown device 5964 (rev 01)

I am now building X 4.4.0 yesterdays CVS checkout, and will investigate
what exactly goes wrong.

Also, i retire what i said about voodoo support, it works fine with
tdfxfb.

> > > BTW, is there any reason the L2 cache is disabled by default in the
> > > 2.4.x kernels ?
> >
> > We have it initialized and enabled.
>
> The kernel doesn't do anything to the L2 cache, it all depends what
> you firmware does to it.

Mmm, ok. I still have to pass it the l2cr=0x80000000 boot arg, but maybe
it would be easier to add it to the default boot args or something.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> Mmm, ok. I still have to pass it the l2cr=0x80000000 boot arg, but maybe
> it would be easier to add it to the default boot args or something.

No, your platform code can initialize it directly, but CC a flame to
the firmware folks for not doing it right in the first place

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Wed, Jan 07, 2004 at 01:36:23PM +1100, Benjamin Herrenschmidt wrote:
>
> > Mmm, ok. I still have to pass it the l2cr=0x80000000 boot arg, but maybe
> > it would be easier to add it to the default boot args or something.
>
> No, your platform code can initialize it directly, but CC a flame to
> the firmware folks for not doing it right in the first place

Ok, i will, thanks again for your help.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michael Schmitz]

> note that there's still a problem with XFree which will "see"  those
> BARs and, according to the log posted by Sven, shoke. Sven, can you
> try "hiding" the host bridge completely from the config ops and see
> if that helps with XFree ? That's not a very good solution though,
> we'll have to do something different about it. Now if only XFree
> stopped mucking with the PCI bus...

Is this a problem with overlapping PCI resources again (thought that was
fixed in 2.4 anyway)? Resolving the overlap helped there. What does XFree
do, simply disable resources?

	Michael


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

On Wed, 2004-01-07 at 20:02, Michael Schmitz wrote:
> > note that there's still a problem with XFree which will "see"  those
> > BARs and, according to the log posted by Sven, shoke. Sven, can you
> > try "hiding" the host bridge completely from the config ops and see
> > if that helps with XFree ? That's not a very good solution though,
> > we'll have to do something different about it. Now if only XFree
> > stopped mucking with the PCI bus...
>
> Is this a problem with overlapping PCI resources again (thought that was
> fixed in 2.4 anyway)? Resolving the overlap helped there. What does XFree
> do, simply disable resources?

There should be no overlap.... Let me check the log ...

(II) Host-to-PCI bridge:
(II) Bus 16: bridge is at (0:0:0), (-1,16,0), BCTRL: 0x0008 (VGA_EN is
set)
(II) Bus 16 I/O range:
        [0] -1  0       0x00000000 - 0x0000ffff (0x10000) IX[B]
(II) Bus 16 non-prefetchable memory range:
        [0] -1  0       0x00000000 - 0xffffffff (0x0) MX[B]
(II) Bus 16 prefetchable memory range:
        [0] -1  0       0x00000000 - 0xffffffff (0x0) MX[B]

The problem is on the host bridge BARs again. I don't know what they
are supposed to do, you need to look at the bridge spec, but you
definitely need to hide them some way...

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Wed, Jan 07, 2004 at 08:23:57PM +1100, Benjamin Herrenschmidt wrote:
> On Wed, 2004-01-07 at 20:02, Michael Schmitz wrote:
> > > note that there's still a problem with XFree which will "see"  those
> > > BARs and, according to the log posted by Sven, shoke. Sven, can you
> > > try "hiding" the host bridge completely from the config ops and see
> > > if that helps with XFree ? That's not a very good solution though,
> > > we'll have to do something different about it. Now if only XFree
> > > stopped mucking with the PCI bus...
> >
> > Is this a problem with overlapping PCI resources again (thought that was
> > fixed in 2.4 anyway)? Resolving the overlap helped there. What does XFree
> > do, simply disable resources?
>
> There should be no overlap.... Let me check the log ...
>
> (II) Host-to-PCI bridge:
> (II) Bus 16: bridge is at (0:0:0), (-1,16,0), BCTRL: 0x0008 (VGA_EN is
> set)
> (II) Bus 16 I/O range:
>         [0] -1  0       0x00000000 - 0x0000ffff (0x10000) IX[B]
> (II) Bus 16 non-prefetchable memory range:
>         [0] -1  0       0x00000000 - 0xffffffff (0x0) MX[B]
> (II) Bus 16 prefetchable memory range:
>         [0] -1  0       0x00000000 - 0xffffffff (0x0) MX[B]
>
> The problem is on the host bridge BARs again. I don't know what they
> are supposed to do, you need to look at the bridge spec, but you
> definitely need to hide them some way...

What i don't know is why X looks at them, even if it didn't appear on
the pci devices list. i am currently building X 4.4 out of CVS to follow
up on where exactly the problem is.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michael Schmitz]

> > Is this a problem with overlapping PCI resources again (thought that was
> > fixed in 2.4 anyway)? Resolving the overlap helped there. What does XFree
> > do, simply disable resources?
>
> There should be no overlap.... Let me check the log ...
>
> (II) Host-to-PCI bridge:
> (II) Bus 16: bridge is at (0:0:0), (-1,16,0), BCTRL: 0x0008 (VGA_EN is
> set)
> (II) Bus 16 I/O range:
>         [0] -1  0       0x00000000 - 0x0000ffff (0x10000) IX[B]
> (II) Bus 16 non-prefetchable memory range:
>         [0] -1  0       0x00000000 - 0xffffffff (0x0) MX[B]
> (II) Bus 16 prefetchable memory range:
>         [0] -1  0       0x00000000 - 0xffffffff (0x0) MX[B]
>
> The problem is on the host bridge BARs again. I don't know what they

That's the weird (-1,busno,0)? My values are 0,busno,1 (plain old
gracke). The rest looks perfectly OK...

> are supposed to do, you need to look at the bridge spec, but you
> definitely need to hide them some way...

Thought I spotted something familiar, but this looks new.

	Michael


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Wed, Jan 07, 2004 at 08:37:36AM +1100, Benjamin Herrenschmidt wrote:
>
> > Here's the code from our pcibios_fixup:
> >
> > 	dev = NULL;
> > 	while ((dev = pci_find_device(PCI_VENDOR_ID_GALILEO,
> > 				PCI_DEVICE_ID_GALILEO_GT64260, dev))) {
> > 		for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
> > 			dev->resource[i].flags = 0;
> > 			dev->resource[i].start = 0;
> > 			dev->resource[i].end = 0;
> > 		}
> > 	}
>
> pcibios_fixup isn't the right place to do that ;) You should do this
> from a pci quirk imho.

Mmm, what do you mean by a pci quirk ?

Anyway, i did that from pcibios_fixup, as well as blocking all pci
writes to device 0 on both buses and reads to all functions of device 0
on both bus, except function 0, for which i allow reads of the first 15
bytes, and return 0 for the others, as i am told i should do by the
hardware guys.

This works fine for the fbdev, but X is not at all happy with it, well,
it seems to work, but there is nothing outputed on the screen :(, but
that is something i probably have to solve with XFree86.

> note that there's still a problem with XFree which will "see"  those
> BARs and, according to the log posted by Sven, shoke. Sven, can you
> try "hiding" the host bridge completely from the config ops and see
> if that helps with XFree ? That's not a very good solution though,
> we'll have to do something different about it. Now if only XFree
> stopped mucking with the PCI bus...

So, yes, that does help with XFree86, altough i get a strange message i
have to search the origin for :

  (WW) ****INVALID IO ALLOCATION**** b: 0x1000 e:0x10ff correcting

Also, with voodoo 3 cards, the tdfx driver dies with :

  (EE) TDFX(0): No valid PIO address in PCI config space

But Radeon works.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Tue, Jan 13, 2004 at 10:56:38AM +0100, Sven Luther wrote:
>
> On Wed, Jan 07, 2004 at 08:37:36AM +1100, Benjamin Herrenschmidt wrote:
> >
> > > Here's the code from our pcibios_fixup:
> > >
> > > 	dev = NULL;
> > > 	while ((dev = pci_find_device(PCI_VENDOR_ID_GALILEO,
> > > 				PCI_DEVICE_ID_GALILEO_GT64260, dev))) {
> > > 		for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
> > > 			dev->resource[i].flags = 0;
> > > 			dev->resource[i].start = 0;
> > > 			dev->resource[i].end = 0;
> > > 		}
> > > 	}
> >
> > pcibios_fixup isn't the right place to do that ;) You should do this
> > from a pci quirk imho.

Damn, i specifically return 0 when reading bytes >15 on the device 0,
func 0, but hexdump /proc/bus/pci/10/00.0 still gives me :

0000000 ab11 6064 0700 b022 0300 0006 2000 0000
0000010 20df 65c3 20df 65c3 20df 65c3 20df 65c3
*
0000040

Should the second line not show only 0s ?

Also, GALILEO is now known as MARVELL, is it ok to add the additional
PCI_VENDOR_ID_MARVELL as the same value as the PCI_VENDOR_ID_GALILEO
above, and also use PCI_DEVICE_ID_MAVELL_... ?

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Tue, Jan 06, 2004 at 03:40:30PM +0100, Geert Uytterhoeven wrote:
>
> On Tue, 6 Jan 2004, Sven Luther wrote:
> > On Tue, Jan 06, 2004 at 07:00:24PM +1100, Benjamin Herrenschmidt wrote:
> > > Type 0 is an access to the primary segment (doesn't contain a bus
> > > number), type 1 is to be forwarded to another bus segment by a P2P
> > > bridge. So for anything directly attached to the host bridge, it's a
> > > type 0 access. Anything else is type 1. Typically, if the bus number of
> > > your "target" == hose->first_busno, it's type 0, else type 1
> >
> > Yep, except we have two pci controllers, and it should be type 0 for
> > both of them.
>
> Yep, two hoses, with different hose->first_busno.
>
> > > They could have appeared as on-chip PCI devices on a "pseudo-bus", but
> > > we can eventually just match with the host's PCI device.
> >
> > Ok. but this can also be faked or something ? But, how can we match with
> > the host PCI device, if we are going to hide it ?
>
> Hide the memory BAR only, not the full PCI device.

Ok, i have done this. I have followed the recomendations of the hardware
guy, and did the following :

  device 0, function 0 : read access to bytes 0-15, read returns 0 on
  regs > 15. Write access fails silently.

  device 0, function 1-7 : both read and writes fail with
  PCIBIOS_DEVICE_NOT_FOUND.

Additionnally, i have nullified the ressources with :
	dev->resource[i].flags = 0;
	dev->resource[i].start = 0;
	dev->resource[i].end = 0;
from the pcibios_fixup (not the right place, but i don't know where it
should be done instead).

This allows me to see the host bridge on both buses, and radeonfb works
fine :

$ lspci
00:00.0 Host bridge: Galileo Technology Ltd.: Unknown device 6460 (rev 03)
00:01.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev 46)
00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c810 (rev 23)
00:0c.0 ISA bridge: VIA Technologies, Inc. VT8231 [PCI-to-ISA Bridge] (rev 10)
00:0c.1 IDE interface: VIA Technologies, Inc.  VT82C586A/B/VT82C686/A/B/VT8233/A/C/VT8235 PIPC Bus Master IDE (rev 06)
00:0c.2 USB Controller: VIA Technologies, Inc. USB (rev 1e)
00:0c.3 USB Controller: VIA Technologies, Inc. USB (rev 1e)
00:0c.4 Non-VGA unclassified device: VIA Technologies, Inc. VT8235 ACPI (rev 10)
00:0c.5 Multimedia audio controller: VIA Technologies, Inc. VT82C686 AC97 Audio Controller (rev 40)
00:0c.6 Communication controller: VIA Technologies, Inc. Intel 537 [AC97 Modem] (rev 20)
00:0d.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 51)
10:00.0 Host bridge: Galileo Technology Ltd.: Unknown device 6460 (rev 03)
10:08.0 VGA compatible controller: ATI Technologies Inc: Unknown device 5964 (rev 01)

But :

$ hexdump /proc/bus/pci/10/00.0
0000000 ab11 6064 0700 b022 0300 0006 2000 0000
0000010 20df 85cf 20df 85cf 20df 85cf 20df 85cf
*
0000040

Why in hell do i have 0x20df85cf on all four bars of the config space
addresses 0x10 to 0x1f ?

Finally, X works, altough DRI freezes after a second or two with my
radeon 9200SE, while it works for a Radeon 7500, but this is probably a
DRI issue.

What makes me wonder, is that X needs around 20 seconds to launch, which
could not be something normal. It usually takes 5-8 seconds only.
Something strange is going on.

Thanks for your help.

Friendly,

Sven Luther


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michel Dänzer]

On Sun, 2004-01-18 at 13:15, Sven Luther wrote:
>
> Finally, X works, altough DRI freezes after a second or two with my
> radeon 9200SE, while it works for a Radeon 7500, but this is probably a
> DRI issue.

Could also be AGP related.


> What makes me wonder, is that X needs around 20 seconds to launch, which
> could not be something normal. It usually takes 5-8 seconds only.

DDC timeouts? As Ben pointed out, one can only speculate without seeing
the server log.


--
Earthling Michel Dänzer      |     Debian (powerpc), X and DRI developer
Libre software enthusiast    |   http://svcs.affero.net/rm.php?r=daenzer


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Sun, Jan 18, 2004 at 02:00:18PM +0100, Michel Dänzer wrote:
> On Sun, 2004-01-18 at 13:15, Sven Luther wrote:
> >
> > Finally, X works, altough DRI freezes after a second or two with my
> > radeon 9200SE, while it works for a Radeon 7500, but this is probably a
> > DRI issue.
>
> Could also be AGP related.

Using PCI gart only. (No real AGP on this box, only a pci bus faked as
AGP one.

> > What makes me wonder, is that X needs around 20 seconds to launch, which
> > could not be something normal. It usually takes 5-8 seconds only.
>
> DDC timeouts? As Ben pointed out, one can only speculate without seeing
> the server log.

Maybe, but it worked well with my previous board. See the log in my
response to Ben.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Mon, Jan 19, 2004 at 08:12:25PM +1100, Benjamin Herrenschmidt wrote:
> On Mon, 2004-01-19 at 00:14, Sven Luther wrote:
>
> > Using PCI gart only. (No real AGP on this box, only a pci bus faked as
> > AGP one.
>
> Makes no sense. It's either an AGP bus or not an AGP bus. AGP is just a
> superset of PCI so there is no such thing as "PCI bus faked as AGP one".

It is a PCI bus (well actually, PCI-X could be supported), but it is
not. The connector is an AGP one, and there is some magic chip or
something which you need to access to do AGP-like config accessing. I
have no idea of the details of it though, only that you need to write
the f118/f11c address before/after each agp config space address.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> It is a PCI bus (well actually, PCI-X could be supported), but it is
> not. The connector is an AGP one, and there is some magic chip or
> something which you need to access to do AGP-like config accessing. I
> have no idea of the details of it though, only that you need to write
> the f118/f11c address before/after each agp config space address.

WTF ? There is no such thing like AGP-like config access... there is
real AGP cycles but from what you say, it doesn't seem to work.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> Ok, i have done this. I have followed the recomendations of the hardware
> guy, and did the following :
>
>   device 0, function 0 : read access to bytes 0-15, read returns 0 on
>   regs > 15. Write access fails silently.

Show the code (just in case...)

>   device 0, function 1-7 : both read and writes fail with
>   PCIBIOS_DEVICE_NOT_FOUND.
>
> Additionnally, i have nullified the ressources with :
> 	dev->resource[i].flags = 0;
> 	dev->resource[i].start = 0;
> 	dev->resource[i].end = 0;
> from the pcibios_fixup (not the right place, but i don't know where it
> should be done instead).

A quirk is one of the functions in the table in pci.c but you can
use the fixup too at this point...

> This allows me to see the host bridge on both buses, and radeonfb works
> fine :
>
> $ lspci
> 00:00.0 Host bridge: Galileo Technology Ltd.: Unknown device 6460 (rev 03)
> 00:01.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev 46)
> 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c810 (rev 23)
> 00:0c.0 ISA bridge: VIA Technologies, Inc. VT8231 [PCI-to-ISA Bridge] (rev 10)
> 00:0c.1 IDE interface: VIA Technologies, Inc.  VT82C586A/B/VT82C686/A/B/VT8233/A/C/VT8235 PIPC Bus Master IDE (rev 06)
> 00:0c.2 USB Controller: VIA Technologies, Inc. USB (rev 1e)
> 00:0c.3 USB Controller: VIA Technologies, Inc. USB (rev 1e)
> 00:0c.4 Non-VGA unclassified device: VIA Technologies, Inc. VT8235 ACPI (rev 10)
> 00:0c.5 Multimedia audio controller: VIA Technologies, Inc. VT82C686 AC97 Audio Controller (rev 40)
> 00:0c.6 Communication controller: VIA Technologies, Inc. Intel 537 [AC97 Modem] (rev 20)
> 00:0d.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 51)
> 10:00.0 Host bridge: Galileo Technology Ltd.: Unknown device 6460 (rev 03)
> 10:08.0 VGA compatible controller: ATI Technologies Inc: Unknown device 5964 (rev 01)
>
> But :
>
> $ hexdump /proc/bus/pci/10/00.0
> 0000000 ab11 6064 0700 b022 0300 0006 2000 0000
> 0000010 20df 85cf 20df 85cf 20df 85cf 20df 85cf
> *
> 0000040
>
> Why in hell do i have 0x20df85cf on all four bars of the config space
> addresses 0x10 to 0x1f ?

I don't know, make sure you are "filtering" things properly... You
should return 0xff on anything after 0x10 I suppose (though you
may want to return 0 for the BARs, I have to dbl check).

> Finally, X works, altough DRI freezes after a second or two with my
> radeon 9200SE, while it works for a Radeon 7500, but this is probably a
> DRI issue.

Which version of DRI ? Do you have the interrupt routing working
properly ?

> What makes me wonder, is that X needs around 20 seconds to launch, which
> could not be something normal. It usually takes 5-8 seconds only.
> Something strange is going on.

Yah, XFree log could be useful.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

On Mon, 2004-01-19 at 00:14, Sven Luther wrote:

> Using PCI gart only. (No real AGP on this box, only a pci bus faked as
> AGP one.

Makes no sense. It's either an AGP bus or not an AGP bus. AGP is just a
superset of PCI so there is no such thing as "PCI bus faked as AGP one".

> > > What makes me wonder, is that X needs around 20 seconds to launch, which
> > > could not be something normal. It usually takes 5-8 seconds only.
> >
> > DDC timeouts? As Ben pointed out, one can only speculate without seeing
> > the server log.
>
> Maybe, but it worked well with my previous board. See the log in my
> response to Ben.
>
> Friendly,
>
> Sven Luther
--
Benjamin Herrenschmidt <benh@kernel.crashing.org>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Mon, Jan 19, 2004 at 09:59:20AM +1100, Benjamin Herrenschmidt wrote:
>
> > It is a PCI bus (well actually, PCI-X could be supported), but it is
> > not. The connector is an AGP one, and there is some magic chip or
> > something which you need to access to do AGP-like config accessing. I
> > have no idea of the details of it though, only that you need to write
> > the f118/f11c address before/after each agp config space address.
>
> WTF ? There is no such thing like AGP-like config access... there is
> real AGP cycles but from what you say, it doesn't seem to work.

Yeah, exactly my impression on this. The only thing i was told about
this, is that you have to do this black magic (disable interrupts, write
to f118, do the access, write to f11c, enable interrupts) for each
access to the secondary bus config space.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Tue, Jan 20, 2004 at 08:35:06AM +1100, Benjamin Herrenschmidt wrote:
>
> > Mmm, should it also be needed to put 0 in the Memory Space and IO space
> > bits of the Command register ?
>
> Probably a good idea, yes.

Mmm.

> > Also, about the Interrupt line, i believe that there is no interrupt
> > used by the northbridge at this time. We were not speaking about the pci
> > config space of the graphic card, but of the bridge, so it should have
> > no influence with DRI or anything else, right ?
>
> Yup.

BTW, xfree86 with voodoo cards and the tdfx driver dies with :

  (EE) TDFX(0): No valid PIO address in PCI config space

But again, this is with the graphic card which should not be touched
here. Tdfxfb is ok though, as is X with the fbdev driver.

Friendly,

Sven Luther


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> Mmm, should it also be needed to put 0 in the Memory Space and IO space
> bits of the Command register ?

Probably a good idea, yes.

> Also, about the Interrupt line, i believe that there is no interrupt
> used by the northbridge at this time. We were not speaking about the pci
> config space of the graphic card, but of the bridge, so it should have
> no influence with DRI or anything else, right ?

Yup.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michel Dänzer]

On Mon, 2004-01-19 at 15:16, Sven Luther wrote:
> BTW, is it time to move this thread to the dri-devel list or something ?

It wouldn't be really on topic there either, so maybe just take it off
list?


> With radeon it does work. I believe this is because the ati module
> lacks knowledge about the 5964 to choose the submodule ?

Right, I forgot to merge that.


> That said, DRI doesn't work, because it lacks the 9200SE recognition
> code, i suppose. Should it be ok to apply the same patch that was posted
> on the debian-x list to your dri-trunk package also ?

RV280 chips should be treated correctly AFAICT. How does it fail?


--
Earthling Michel Dänzer      |     Debian (powerpc), X and DRI developer
Libre software enthusiast    |   http://svcs.affero.net/rm.php?r=daenzer


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

BTW, is it time to move this thread to the dri-devel list or something ?

On Mon, Jan 19, 2004 at 03:02:28PM +0100, Michel Dänzer wrote:
>
> On Mon, 2004-01-19 at 14:48, Sven Luther wrote:
> > On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> > >
> > > You may want to try my dri-trunk-sid packages.
> >
> > Just tried (all three packages) and my Radeon 9200 SE is not recognized :
> >
> > (II) PCI: 10:08:0: chip 1002,5964 card 148c,2073 rev 01 class 03,00,00 hdr 00
> > ...
> > (WW) ATI:  PCI/AGP Mach64 in slot 16:8:0 could not be detected!
> > (EE) No devices detected.
>
> Hmm.
>
> daenzer@thor|14:50:01> grep 5964 /var/log/XFree86.0.log
>         ATI Radeon 9200 5962 (AGP), ATI Radeon 9200SE 5964 (AGP),
>
> If you have Driver "ati" in the Device section, does "radeon" make a
> difference?

Yep, it makes a difference. With radeon it does work. I believe this is
because the ati module lacks knowledge about the 5964 to choose the
submodule ?

Also, by doing this, the startup time of X is also much faster, it
brings the startup time down from 20s, which i feel is to much, to
10-12s.

That said, DRI doesn't work, because it lacks the 9200SE recognition
code, i suppose. Should it be ok to apply the same patch that was posted
on the debian-x list to your dri-trunk package also ?

> > Time for you to make another snapshot ?
>
> Give me the time, and/or fix all the bugs (or at least the important
> ones) in current DRI/Mesa CVS...

:))

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michel Dänzer]

On Mon, 2004-01-19 at 14:48, Sven Luther wrote:
> On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> >
> > You may want to try my dri-trunk-sid packages.
>
> Just tried (all three packages) and my Radeon 9200 SE is not recognized :
>
> (II) PCI: 10:08:0: chip 1002,5964 card 148c,2073 rev 01 class 03,00,00 hdr 00
> ...
> (WW) ATI:  PCI/AGP Mach64 in slot 16:8:0 could not be detected!
> (EE) No devices detected.

Hmm.

daenzer@thor|14:50:01> grep 5964 /var/log/XFree86.0.log
        ATI Radeon 9200 5962 (AGP), ATI Radeon 9200SE 5964 (AGP),

If you have Driver "ati" in the Device section, does "radeon" make a
difference?

> Time for you to make another snapshot ?

Give me the time, and/or fix all the bugs (or at least the important
ones) in current DRI/Mesa CVS...


--
Earthling Michel Dänzer      |     Debian (powerpc), X and DRI developer
Libre software enthusiast    |   http://svcs.affero.net/rm.php?r=daenzer


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Mon, Jan 19, 2004 at 02:54:32PM +0100, Geert Uytterhoeven wrote:
> On Mon, 19 Jan 2004, Sven Luther wrote:
> > On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> > > You may want to try my dri-trunk-sid packages.
> >
> > Just tried (all three packages) and my Radeon 9200 SE is not recognized :
> >
> > (II) PCI: 10:08:0: chip 1002,5964 card 148c,2073 rev 01 class 03,00,00 hdr 00
> > ...
> > (WW) ATI:  PCI/AGP Mach64 in slot 16:8:0 could not be detected!
>
> That's a warning from the Mach64 driver, not from the Radeon driver?

Yes, indeed it is. And in fact i see :

(II) ATI: ATI driver (version 6.5.3) for chipset: ati
...
(II) RADEON: Driver for ATI Radeon chipsets: ATI Radeon QD (AGP),
...
        ATI Radeon 9200 5962 (AGP), ATI Radeon 9200SE 5964 (AGP),

Which is my graphic card, but then :

        ATI Radeon 9800XT NJ (AGP)
(II) Primary Device is: PCI 10:08:0
(II) ATI:  Candidate "Device" section "Generic Video Card".
(WW) ATI:  PCI/AGP Mach64 in slot 16:8:0 could not be detected!
(EE) No devices detected.

So, i wonder what is going on.

Full log can be found at :

  http://people.debian.org/~luther/XFree86.Michel.log

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Geert Uytterhoeven]

On Mon, 19 Jan 2004, Sven Luther wrote:
> On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> > You may want to try my dri-trunk-sid packages.
>
> Just tried (all three packages) and my Radeon 9200 SE is not recognized :
>
> (II) PCI: 10:08:0: chip 1002,5964 card 148c,2073 rev 01 class 03,00,00 hdr 00
> ...
> (WW) ATI:  PCI/AGP Mach64 in slot 16:8:0 could not be detected!

That's a warning from the Mach64 driver, not from the Radeon driver?

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> On Sun, 2004-01-18 at 15:44, Sven Luther wrote:
> > >
> > > > Finally, X works, altough DRI freezes after a second or two with my
> > > > radeon 9200SE, while it works for a Radeon 7500, but this is probably a
> > > > DRI issue.
>
> [...]
>
> > As for the DRI version, i use the drm module from the linuxppc-2.4 tree,
> > using the v2.4.24 TAG to checkout, and the rest of the XFree86 stuff,
> > including the mesa libraries, from the 4.3.0-0pre1v5 experimental
> > package, rebuild with the Radeon 9200SE patch from Michel Daenzer.
>
> You may want to try my dri-trunk-sid packages.

Just tried (all three packages) and my Radeon 9200 SE is not recognized :

(II) PCI: 10:08:0: chip 1002,5964 card 148c,2073 rev 01 class 03,00,00 hdr 00
...
(WW) ATI:  PCI/AGP Mach64 in slot 16:8:0 could not be detected!
(EE) No devices detected.

Time for you to make another snapshot ? These where with the 2003.10.05
snapshot.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Mon, Jan 19, 2004 at 10:41:19PM +1100, Benjamin Herrenschmidt wrote:
>
> On Mon, 2004-01-19 at 21:08, Geert Uytterhoeven wrote:
> > On Mon, 19 Jan 2004, Benjamin Herrenschmidt wrote:
> > > > I am returning 0 for all of function 0. I dropped the whole struct
> > > > pci_dev ressource thingy, and they are well nullified. My limited
> > > > understanding of those pci issues let me make a guess though. I think
> > > > that either the stuff in the struct pci_dev is set later on (the BARs
> > > > are modifiable i think), or those values are read from the struct
> > > > pci_dev before i nullify them.
> > >
> > > pci_dev resources are read from the BARs and the sizing mecanism
> > > uses the BARs too (you should filter out writes too btw). If you
> > > properly filter things out, there should be no problem.
> >
> > So shouldn't reads from the BAR return -1 instead of 0, and writes be ignored?
>
> Writes have to be ignored, I don't remember if an unused BAR should
> return 0xffffffff or 0 (check the spec). (The call shall not fail,
> the BAR space has to be implemented).

Mmm, should it also be needed to put 0 in the Memory Space and IO space
bits of the Command register ?

Also, about the Interrupt line, i believe that there is no interrupt
used by the northbridge at this time. We were not speaking about the pci
config space of the graphic card, but of the bridge, so it should have
no influence with DRI or anything else, right ?

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

On Mon, 2004-01-19 at 21:08, Geert Uytterhoeven wrote:
> On Mon, 19 Jan 2004, Benjamin Herrenschmidt wrote:
> > > I am returning 0 for all of function 0. I dropped the whole struct
> > > pci_dev ressource thingy, and they are well nullified. My limited
> > > understanding of those pci issues let me make a guess though. I think
> > > that either the stuff in the struct pci_dev is set later on (the BARs
> > > are modifiable i think), or those values are read from the struct
> > > pci_dev before i nullify them.
> >
> > pci_dev resources are read from the BARs and the sizing mecanism
> > uses the BARs too (you should filter out writes too btw). If you
> > properly filter things out, there should be no problem.
>
> So shouldn't reads from the BAR return -1 instead of 0, and writes be ignored?

Writes have to be ignored, I don't remember if an unused BAR should
return 0xffffffff or 0 (check the spec). (The call shall not fail,
the BAR space has to be implemented).

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Geert Uytterhoeven]

On Mon, 19 Jan 2004, Benjamin Herrenschmidt wrote:
> > I am returning 0 for all of function 0. I dropped the whole struct
> > pci_dev ressource thingy, and they are well nullified. My limited
> > understanding of those pci issues let me make a guess though. I think
> > that either the stuff in the struct pci_dev is set later on (the BARs
> > are modifiable i think), or those values are read from the struct
> > pci_dev before i nullify them.
>
> pci_dev resources are read from the BARs and the sizing mecanism
> uses the BARs too (you should filter out writes too btw). If you
> properly filter things out, there should be no problem.

So shouldn't reads from the BAR return -1 instead of 0, and writes be ignored?

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Mon, Jan 19, 2004 at 12:33:24AM +0100, Michel Dänzer wrote:
>
> On Sun, 2004-01-18 at 23:20, Sven Luther wrote:
> > On Sun, Jan 18, 2004 at 07:24:44PM +0100, Michel Dänzer wrote:
> > > On Sun, 2004-01-18 at 18:28, Sven Luther wrote:
> > > > On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> > > > > On Sun, 2004-01-18 at 15:44, Sven Luther wrote:
> > > > > >
> > > > > > The freeze happens when i first launch glxinfo, or when i first start
> > > > > > moving a window around (using a debian/unstable default gnome desktop).
> > > > > > I don't remember well, but i think it would also freeze when let running
> > > > > > for a time, but i am not sure.
> > >
> > > It would be quite interesting to know, see below.
> >
> > But then, maybe it has something to do with the GL screensaver or
> > something such.
>
> Don't run them, then? *shrug*

Mmm, i only have xscreensaver installer, not the gl ones. Should be ok.

But then, the DRI server corrupts the fbdev screen, and the font memory
when i /etc/init.d/gdm stop :/

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

> .../...

The code you posted is awfully ugly...

I'll look at it in detail later.

> I am returning 0 for all of function 0. I dropped the whole struct
> pci_dev ressource thingy, and they are well nullified. My limited
> understanding of those pci issues let me make a guess though. I think
> that either the stuff in the struct pci_dev is set later on (the BARs
> are modifiable i think), or those values are read from the struct
> pci_dev before i nullify them.

pci_dev resources are read from the BARs and the sizing mecanism
uses the BARs too (you should filter out writes too btw). If you
properly filter things out, there should be no problem.

> > > Finally, X works, altough DRI freezes after a second or two with my
> > > radeon 9200SE, while it works for a Radeon 7500, but this is probably a
> > > DRI issue.
> >
> > Which version of DRI ? Do you have the interrupt routing working
> > properly ?
>
> Mmm, maybe i should also allow to read (and write ?) the config 32-bit
> word at 0x3c, those include the Interrupt Line and Pin, as well as the
> Max_lat and Min_Gnt.

Interrupt pin is mostly useless. You may want to fill interrupt line
of the PCI cards with the value assigned by OF (or in any case, at
least make sure pci_dev->irq is properly filled).

> Maybe some of the first 16 bytes would also need to be modifiable, and
> there should be no harm in allowing read of the subsytem id and vendor
> id ?

Of what ? the bridge ? You surely need to let the system access the AGP
portion of it btw...

> As for the DRI version, i use the drm module from the linuxppc-2.4 tree,
> using the v2.4.24 TAG to checkout, and the rest of the XFree86 stuff,
> including the mesa libraries, from the 4.3.0-0pre1v5 experimental
> package, rebuild with the Radeon 9200SE patch from Michel Daenzer.

Use the DRM module from Michel snapshot, might help...

> The freeze happens when i first launch glxinfo, or when i first start
> moving a window around (using a debian/unstable default gnome desktop).
> I don't remember well, but i think it would also freeze when let running
> for a time, but i am not sure. The box is still available trough ssh,
> but killing the X server doesn't restore the fbdev console, and freeze
> the box.

Could be irq not working...

> > > What makes me wonder, is that X needs around 20 seconds to launch, which
> > > could not be something normal. It usually takes 5-8 seconds only.
> > > Something strange is going on.
> >
> > Yah, XFree log could be useful.
>
> Ok, i uploaded it at http://people.debian.org/~luther/XFree86.0.log,
> since the list was not able to cope with it. I didn't see anything anormal
> there though. Altough some of the bus ressources seem to be checked even
> if i nullified them.
>
> Friendly,
>
> Sven Luther
--
Benjamin Herrenschmidt <benh@kernel.crashing.org>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michel Dänzer]

On Mon, 2004-01-19 at 00:42, Benjamin Herrenschmidt wrote:
> >
> > Only the 3D drivers use the IRQ, it's not involved when moving around
> > windows. Or do you mean a badly configured IRQ causing a flood or
> > something?
>
> Oh, you don't use the IRQ for throttling the ring ?

No, that's only rarely needed anyway, if ever (a while ago, I discovered
that the DRM code to detect a full ring had been broken for a while, but
it didn't seem to cause any problems :).

> Only for VBL ?

That, frame throttling (to prevent too many frames getting queued) and
waiting for rendering to finish.


--
Earthling Michel Dänzer      |     Debian (powerpc), X and DRI developer
Libre software enthusiast    |   http://svcs.affero.net/rm.php?r=daenzer


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

>
> Only the 3D drivers use the IRQ, it's not involved when moving around
> windows. Or do you mean a badly configured IRQ causing a flood or
> something?

Oh, you don't use the IRQ for throttling the ring ? Only for VBL ?

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michel Dänzer]

On Sun, 2004-01-18 at 23:20, Sven Luther wrote:
> On Sun, Jan 18, 2004 at 07:24:44PM +0100, Michel Dänzer wrote:
> > On Sun, 2004-01-18 at 18:28, Sven Luther wrote:
> > > On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> > > > On Sun, 2004-01-18 at 15:44, Sven Luther wrote:
> > > > >
> > > > > The freeze happens when i first launch glxinfo, or when i first start
> > > > > moving a window around (using a debian/unstable default gnome desktop).
> > > > > I don't remember well, but i think it would also freeze when let running
> > > > > for a time, but i am not sure.
> >
> > It would be quite interesting to know, see below.
>
> But then, maybe it has something to do with the GL screensaver or
> something such.

Don't run them, then? *shrug*


> Altough this problem doesn't occur for the Radeon 7500 on the same board.

I suspect that's more luck than anything else at this point...

> I believe it may be triggered by some problem which is present in
> the R200 code path, and which is present in the R100 software T&L (since
> i think the 7000 and 7200 lack a T&L engine).

The 7200 has one, and the 2D acceleration code (which moves windows
around) doesn't have any core specific paths because the 2D engines are
mostly identical.

> > It might also occur when the ring wraps around; whether or not it also
> > happens when the server is idling would give a hint about this, or even
> > better instrumenting the wraparound handling code in the DRM.
>
> Ok, i will look into this and do some experiments. What is the size of
> the ring anyway ? 4K or something such ?

daenzer@thor|0:29:29> grep 'ring buffer' /var/log/XFree86.0.log
(II) RADEON(0): Using 1 MB for the ring buffer


--
Earthling Michel Dänzer      |     Debian (powerpc), X and DRI developer
Libre software enthusiast    |   http://svcs.affero.net/rm.php?r=daenzer


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michel Dänzer]

On Mon, 2004-01-19 at 10:11, Benjamin Herrenschmidt wrote:
>
> > The freeze happens when i first launch glxinfo, or when i first start
> > moving a window around (using a debian/unstable default gnome desktop).
> > I don't remember well, but i think it would also freeze when let running
> > for a time, but i am not sure. The box is still available trough ssh,
> > but killing the X server doesn't restore the fbdev console, and freeze
> > the box.
>
> Could be irq not working...

Only the 3D drivers use the IRQ, it's not involved when moving around
windows. Or do you mean a badly configured IRQ causing a flood or
something?


--
Earthling Michel Dänzer      |     Debian (powerpc), X and DRI developer
Libre software enthusiast    |   http://svcs.affero.net/rm.php?r=daenzer


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Mon, Jan 19, 2004 at 08:11:31PM +1100, Benjamin Herrenschmidt wrote:
>
> > .../...
>
> The code you posted is awfully ugly...

Comes from multiple testing and such, once it find the final one, i will
do some cleanup or something. That said, it is not all that more ugly as
the stuff in indirect_pci.c i used as example.

> I'll look at it in detail later.
>
> > I am returning 0 for all of function 0. I dropped the whole struct
> > pci_dev ressource thingy, and they are well nullified. My limited
> > understanding of those pci issues let me make a guess though. I think
> > that either the stuff in the struct pci_dev is set later on (the BARs
> > are modifiable i think), or those values are read from the struct
> > pci_dev before i nullify them.
>
> pci_dev resources are read from the BARs and the sizing mecanism
> uses the BARs too (you should filter out writes too btw). If you
> properly filter things out, there should be no problem.

The BARs are the ones at address 0x10 to 0x27 of the pci config space,
right ? Only 0 is read back when looking at them.

> > > > Finally, X works, altough DRI freezes after a second or two with my
> > > > radeon 9200SE, while it works for a Radeon 7500, but this is probably a
> > > > DRI issue.
> > >
> > > Which version of DRI ? Do you have the interrupt routing working
> > > properly ?
> >
> > Mmm, maybe i should also allow to read (and write ?) the config 32-bit
> > word at 0x3c, those include the Interrupt Line and Pin, as well as the
> > Max_lat and Min_Gnt.
>
> Interrupt pin is mostly useless. You may want to fill interrupt line
> of the PCI cards with the value assigned by OF (or in any case, at
> least make sure pci_dev->irq is properly filled).

Ok.

> > Maybe some of the first 16 bytes would also need to be modifiable, and
> > there should be no harm in allowing read of the subsytem id and vendor
> > id ?
>
> Of what ? the bridge ? You surely need to let the system access the AGP
> portion of it btw...

There is no AGP portion of it, it is a plain pci bus with some config
space reading magic i don't know the details, and a AGP (3.3V)
connector.

> > As for the DRI version, i use the drm module from the linuxppc-2.4 tree,
> > using the v2.4.24 TAG to checkout, and the rest of the XFree86 stuff,
> > including the mesa libraries, from the 4.3.0-0pre1v5 experimental
> > package, rebuild with the Radeon 9200SE patch from Michel Daenzer.
>
> Use the DRM module from Michel snapshot, might help...

Will try tomorrow. But then, it seems that the default for debian will
be to use the kernel included DRM.

> > The freeze happens when i first launch glxinfo, or when i first start
> > moving a window around (using a debian/unstable default gnome desktop).
> > I don't remember well, but i think it would also freeze when let running
> > for a time, but i am not sure. The box is still available trough ssh,
> > but killing the X server doesn't restore the fbdev console, and freeze
> > the box.
>
> Could be irq not working...

Which would make sense with the above remarq of the interrupt line.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Sun, Jan 18, 2004 at 07:24:44PM +0100, Michel Dänzer wrote:
> On Sun, 2004-01-18 at 18:28, Sven Luther wrote:
> > On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> > > On Sun, 2004-01-18 at 15:44, Sven Luther wrote:
> > > > >
> > > > The freeze happens when i first launch glxinfo, or when i first start
> > > > moving a window around (using a debian/unstable default gnome desktop).
> > > > I don't remember well, but i think it would also freeze when let running
> > > > for a time, but i am not sure.
>
> It would be quite interesting to know, see below.

But then, maybe it has something to do with the GL screensaver or
something such. The problem is that i have this box on the analog entry
of my dual entry monitor, and mostly work on the box connected to the
DVI connector, so it usually died while i was not looking.

> > > > The box is still available trough ssh, but killing the X server
> > > > doesn't restore the fbdev console, and freeze the box.
> > >
> > > Sounds like a typical chip lockup, possibly caused by the chip not
> > > reading from the CP ring what we're writing to it.
> >
> > Well, not sure. Compared to the pegasos 1 situation, where nothing
> > showed up on the screen, except the hardware cursor, Things seem to work
> > out fine at first, and only certain operation make it happen. The gdm
> > prompt comes up, you can log in, then once logged in, you can open a
> > gnome-terminal, and even do some stuff. Once you try moving the window
> > though, it starts moving, but then quickly freezes.
>
> Because the RENDER extension isn't accelerated, that might be the first
> time the ring is heavily used.

Yep. Altough this problem doesn't occur for the Radeon 7500 on the same
board. I believe it may be triggered by some problem which is present in
the R200 code path, and which is present in the R100 software T&L (since
i think the 7000 and 7200 lack a T&L engine).

> It might also occur when the ring wraps around; whether or not it also
> happens when the server is idling would give a hint about this, or even
> better instrumenting the wraparound handling code in the DRM.

Ok, i will look into this and do some experiments. What is the size of
the ring anyway ? 4K or something such ?

> > I believe there is maybe more a problem with the CPU and the graphic
> > chip being in disacord over the size of the CP ring.
>
> I'd expect problems everywhere if that code wasn't correct.

Mmm.

> > Come to think of it, i have three negative tries (Radeon 7000, 7200 and
> > 9200SE) and one positive (Radeon 7500), or at least claimed such, the
> > glxgears numbers obtained with the Radeon 7500 where not all that high
> > (maybe 300 or so), but still higher than the software only numbers i
> > obtain on my box with Radeon 9200 SE.
>
> PCI GART isn't very fast in general, it's even mysteriously slow on some
> systems, see the dri-devel list archives.

Ok. There is no other chance anyway, since there is no true AGP bus on
these boxes. The nortbridge used is a communication controller after
all.

> > Also, the Radeon 9200 SE has only 64bit memory interface. I don't know
> > about 7000 and 7200, but maybe this is also the case for them, while the
> > 7500 could have 128bit memory interface ? Not sure, will investigate.
>
> I doubt that matters.

I doubt it too. But then, if the chip measures data in multiples of the
memory words or something such, it may have an influence.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michel Dänzer]

On Sun, 2004-01-18 at 18:28, Sven Luther wrote:
> On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> > On Sun, 2004-01-18 at 15:44, Sven Luther wrote:
> > > >
> > > The freeze happens when i first launch glxinfo, or when i first start
> > > moving a window around (using a debian/unstable default gnome desktop).
> > > I don't remember well, but i think it would also freeze when let running
> > > for a time, but i am not sure.

It would be quite interesting to know, see below.

> > > The box is still available trough ssh, but killing the X server
> > > doesn't restore the fbdev console, and freeze the box.
> >
> > Sounds like a typical chip lockup, possibly caused by the chip not
> > reading from the CP ring what we're writing to it.
>
> Well, not sure. Compared to the pegasos 1 situation, where nothing
> showed up on the screen, except the hardware cursor, Things seem to work
> out fine at first, and only certain operation make it happen. The gdm
> prompt comes up, you can log in, then once logged in, you can open a
> gnome-terminal, and even do some stuff. Once you try moving the window
> though, it starts moving, but then quickly freezes.

Because the RENDER extension isn't accelerated, that might be the first
time the ring is heavily used.

It might also occur when the ring wraps around; whether or not it also
happens when the server is idling would give a hint about this, or even
better instrumenting the wraparound handling code in the DRM.

> I believe there is maybe more a problem with the CPU and the graphic
> chip being in disacord over the size of the CP ring.

I'd expect problems everywhere if that code wasn't correct.


> Come to think of it, i have three negative tries (Radeon 7000, 7200 and
> 9200SE) and one positive (Radeon 7500), or at least claimed such, the
> glxgears numbers obtained with the Radeon 7500 where not all that high
> (maybe 300 or so), but still higher than the software only numbers i
> obtain on my box with Radeon 9200 SE.

PCI GART isn't very fast in general, it's even mysteriously slow on some
systems, see the dri-devel list archives.


> Also, the Radeon 9200 SE has only 64bit memory interface. I don't know
> about 7000 and 7200, but maybe this is also the case for them, while the
> 7500 could have 128bit memory interface ? Not sure, will investigate.

I doubt that matters.


--
Earthling Michel Dänzer      |     Debian (powerpc), X and DRI developer
Libre software enthusiast    |   http://svcs.affero.net/rm.php?r=daenzer


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Sun, Jan 18, 2004 at 05:33:21PM +0100, Michel Dänzer wrote:
> On Sun, 2004-01-18 at 15:44, Sven Luther wrote:
> > >
> > > > Finally, X works, altough DRI freezes after a second or two with my
> > > > radeon 9200SE, while it works for a Radeon 7500, but this is probably a
> > > > DRI issue.
>
> [...]
>
> > As for the DRI version, i use the drm module from the linuxppc-2.4 tree,
> > using the v2.4.24 TAG to checkout, and the rest of the XFree86 stuff,
> > including the mesa libraries, from the 4.3.0-0pre1v5 experimental
> > package, rebuild with the Radeon 9200SE patch from Michel Daenzer.
>
> You may want to try my dri-trunk-sid packages.

Ok, will give them a try.

> > The freeze happens when i first launch glxinfo, or when i first start
> > moving a window around (using a debian/unstable default gnome desktop).
> > I don't remember well, but i think it would also freeze when let running
> > for a time, but i am not sure. The box is still available trough ssh,
> > but killing the X server doesn't restore the fbdev console, and freeze
> > the box.
>
> Sounds like a typical chip lockup, possibly caused by the chip not
> reading from the CP ring what we're writing to it.

Well, not sure. Compared to the pegasos 1 situation, where nothing
showed up on the screen, except the hardware cursor, Things seem to work
out fine at first, and only certain operation make it happen. The gdm
prompt comes up, you can log in, then once logged in, you can open a
gnome-terminal, and even do some stuff. Once you try moving the window
though, it starts moving, but then quickly freezes. I believe there is
maybe more a problem with the CPU and the graphic chip being in disacord
over the size of the CP ring.

Come to think of it, i have three negative tries (Radeon 7000, 7200 and
9200SE) and one positive (Radeon 7500), or at least claimed such, the
glxgears numbers obtained with the Radeon 7500 where not all that high
(maybe 300 or so), but still higher than the software only numbers i
obtain on my box with Radeon 9200 SE.

Also, the Radeon 9200 SE has only 64bit memory interface. I don't know
about 7000 and 7200, but maybe this is also the case for them, while the
7500 could have 128bit memory interface ? Not sure, will investigate.

> Do you also have a server log with the DRI enabled?

Wait a minute, i can send them to you. I also had logs of the of the
7500 and 7000, but had them in /tmp and rebooted my box yesterday :/.

Anyway, here are the three DRI enabled logs :

http://people.debian.org/~luther/XFree86.DRI.log
	this one is while it is loading.
http://people.debian.org/~luther/XFree86.DRI2.log
	this one is ones loged in gnome, opened a terminal, and started
	typing in it (ls only).
http://people.debian.org/~luther/XFree86.DRI3.log
	this one is after having moved the gnome-terminal and locking up
	X. Hardware cursor still works. i first moved the gnome-terminal
	a bit, and it worked, but i insisted and it locked up.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Michel Dänzer]

On Sun, 2004-01-18 at 15:44, Sven Luther wrote:
> >
> > > Finally, X works, altough DRI freezes after a second or two with my
> > > radeon 9200SE, while it works for a Radeon 7500, but this is probably a
> > > DRI issue.

[...]

> As for the DRI version, i use the drm module from the linuxppc-2.4 tree,
> using the v2.4.24 TAG to checkout, and the rest of the XFree86 stuff,
> including the mesa libraries, from the 4.3.0-0pre1v5 experimental
> package, rebuild with the Radeon 9200SE patch from Michel Daenzer.

You may want to try my dri-trunk-sid packages.

> The freeze happens when i first launch glxinfo, or when i first start
> moving a window around (using a debian/unstable default gnome desktop).
> I don't remember well, but i think it would also freeze when let running
> for a time, but i am not sure. The box is still available trough ssh,
> but killing the X server doesn't restore the fbdev console, and freeze
> the box.

Sounds like a typical chip lockup, possibly caused by the chip not
reading from the CP ring what we're writing to it.

Do you also have a server log with the DRI enabled?


--
Earthling Michel Dänzer      |     Debian (powerpc), X and DRI developer
Libre software enthusiast    |   http://svcs.affero.net/rm.php?r=daenzer


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Sven Luther]

On Mon, Jan 19, 2004 at 10:24:52AM +1100, Benjamin Herrenschmidt wrote:
>
> > Ok, i have done this. I have followed the recomendations of the hardware
> > guy, and did the following :
> >
> >   device 0, function 0 : read access to bytes 0-15, read returns 0 on
> >   regs > 15. Write access fails silently.
>
> Show the code (just in case...)


  #define PEG2_PCI_OP(rw, size, type, op, mask, btrw)
  int __chrp
  peg2_##rw##_config_##size(struct pci_dev *dev, int offset, type val)
  {
          struct pci_controller *hose = dev->sysdata;
          u32 msr;
          u32 data;
          volatile unsigned int *f118 = hose->cfg_peg2_magic;
          volatile unsigned int *f11c = (unsigned int *) f118 + 1;

          /* We won't write on device 0 and only read byte 0-15 of func 0 */
          if (dev->bus->number == 0 || dev->bus->number == 0x10) {
                  if (dev->devfn == 0) {
                          if (!(btrw & 1) && (offset > 15)) {
                                  val = (type) 0x0;
                                  return PCIBIOS_SUCCESSFUL;
                          } else if (btrw & 1) {
                                  return PCIBIOS_SUCCESSFUL;
                          }
                  } else if ((dev->devfn >> 3) == 0) {
                          return PCIBIOS_DEVICE_NOT_FOUND;
                  }
          }

	  if (btrw & 2) {
                /* Disable cpu interrupts */
                msr = mfmsr();
                mtmsr(msr & ~MSR_EE);
                /* Enable PCI -> AGP idsel mapping */
                cfg_write(0x8000, f118, u32, out_be32);
          }

          data = 0x80000000 | ((dev->bus->number - hose->bus_offset) << 16)
                    | (dev->devfn << 8) | (offset & 0xff);
          cfg_write(data, hose->cfg_addr, u32, out_le32);
          cfg_##rw(val, hose->cfg_data + (offset & mask), type, op);

          if (btrw & 2) {
                /* Enable PCI -> AGP idsel mapping */
                cfg_write(0x8000, f11c, u32, out_be32);
                /* Disable cpu interrupts */
                mtmsr(msr);
          }
          return PCIBIOS_SUCCESSFUL;
  }

btrw is 0 for reads and 1 for writes. The second bit (0 or 2) also
encodes if we are on the pci or agp bus, to enable th agp config access
magic.

> >   device 0, function 1-7 : both read and writes fail with
> >   PCIBIOS_DEVICE_NOT_FOUND.
> >
> > Additionnally, i have nullified the ressources with :
> > 	dev->resource[i].flags = 0;
> > 	dev->resource[i].start = 0;
> > 	dev->resource[i].end = 0;
> > from the pcibios_fixup (not the right place, but i don't know where it
> > should be done instead).
>
> A quirk is one of the functions in the table in pci.c but you can
> use the fixup too at this point...

Ok.

> > This allows me to see the host bridge on both buses, and radeonfb works
> > fine :
> >
> > $ lspci
> > 00:00.0 Host bridge: Galileo Technology Ltd.: Unknown device 6460 (rev 03)
> > 00:01.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev 46)
> > 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c810 (rev 23)
> > 00:0c.0 ISA bridge: VIA Technologies, Inc. VT8231 [PCI-to-ISA Bridge] (rev 10)
> > 00:0c.1 IDE interface: VIA Technologies, Inc.  VT82C586A/B/VT82C686/A/B/VT8233/A/C/VT8235 PIPC Bus Master IDE (rev 06)
> > 00:0c.2 USB Controller: VIA Technologies, Inc. USB (rev 1e)
> > 00:0c.3 USB Controller: VIA Technologies, Inc. USB (rev 1e)
> > 00:0c.4 Non-VGA unclassified device: VIA Technologies, Inc. VT8235 ACPI (rev 10)
> > 00:0c.5 Multimedia audio controller: VIA Technologies, Inc. VT82C686 AC97 Audio Controller (rev 40)
> > 00:0c.6 Communication controller: VIA Technologies, Inc. Intel 537 [AC97 Modem] (rev 20)
> > 00:0d.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 51)
> > 10:00.0 Host bridge: Galileo Technology Ltd.: Unknown device 6460 (rev 03)
> > 10:08.0 VGA compatible controller: ATI Technologies Inc: Unknown device 5964 (rev 01)
> >
> > But :
> >
> > $ hexdump /proc/bus/pci/10/00.0
> > 0000000 ab11 6064 0700 b022 0300 0006 2000 0000
> > 0000010 20df 85cf 20df 85cf 20df 85cf 20df 85cf
> > *
> > 0000040
> >
> > Why in hell do i have 0x20df85cf on all four bars of the config space
> > addresses 0x10 to 0x1f ?
>
> I don't know, make sure you are "filtering" things properly... You
> should return 0xff on anything after 0x10 I suppose (though you
> may want to return 0 for the BARs, I have to dbl check).

I am returning 0 for all of function 0. I dropped the whole struct
pci_dev ressource thingy, and they are well nullified. My limited
understanding of those pci issues let me make a guess though. I think
that either the stuff in the struct pci_dev is set later on (the BARs
are modifiable i think), or those values are read from the struct
pci_dev before i nullify them.

> > Finally, X works, altough DRI freezes after a second or two with my
> > radeon 9200SE, while it works for a Radeon 7500, but this is probably a
> > DRI issue.
>
> Which version of DRI ? Do you have the interrupt routing working
> properly ?

Mmm, maybe i should also allow to read (and write ?) the config 32-bit
word at 0x3c, those include the Interrupt Line and Pin, as well as the
Max_lat and Min_Gnt.

Maybe some of the first 16 bytes would also need to be modifiable, and
there should be no harm in allowing read of the subsytem id and vendor
id ?

As for the DRI version, i use the drm module from the linuxppc-2.4 tree,
using the v2.4.24 TAG to checkout, and the rest of the XFree86 stuff,
including the mesa libraries, from the 4.3.0-0pre1v5 experimental
package, rebuild with the Radeon 9200SE patch from Michel Daenzer.

The freeze happens when i first launch glxinfo, or when i first start
moving a window around (using a debian/unstable default gnome desktop).
I don't remember well, but i think it would also freeze when let running
for a time, but i am not sure. The box is still available trough ssh,
but killing the X server doesn't restore the fbdev console, and freeze
the box.

> > What makes me wonder, is that X needs around 20 seconds to launch, which
> > could not be something normal. It usually takes 5-8 seconds only.
> > Something strange is going on.
>
> Yah, XFree log could be useful.

Ok, i uploaded it at http://people.debian.org/~luther/XFree86.0.log,
since the list was not able to cope with it. I didn't see anything anormal
there though. Altough some of the bus ressources seem to be checked even
if i nullified them.

Friendly,

Sven Luther

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

On Wed, 2004-01-07 at 08:09, Marcus Barrow wrote:
> oops, sorry i sent wrong note previously...
>
>         I enabled the debugging option in drivers/pci.c and rebooted.
>         Here is the pertinent output. As you can see, the two pci busses
>         have different bus numbers (in dev->bus->number ).
>
>         We use a modified version of the Marvell eval. board code. It
>         doesn't seem to have been changed for some time now. Perhaps you
>         need to look at your "agp ops"? Is this helpfull?

You do NOT care about bus numbers on the primary segment of a given
PCI domain. These are type 0 config cycles and so do not carry any
kind of bus numbers.

If the host bridge do try to muck with those, then it's not fully
on spec, but then, that can be worked around.

I'd still like to see the docs of this chipset though...

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Benjamin Herrenschmidt]

>         Would you care to rethink those comments?
>
>         It seems unfair to call a chip shit, when you don't have
>         a manual for it and appear to lack any knowledge of it's
>         internals. Do the math for 3 GigE ports running line speed with
>         64 byte packets, and full bandwidth on 2 PCI busses...

It would still have been easy to have them "show up" as PCI devices
for the sake of proper discovery (and eventually mapping their control
registers as PCI devices), without impairing their ability to use
a fast path for DMA transfers.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Marcus Barrow]

oops, sorry i sent wrong note previously...

        I enabled the debugging option in drivers/pci.c and rebooted.
        Here is the pertinent output. As you can see, the two pci busses
        have different bus numbers (in dev->bus->number ).

        We use a modified version of the Marvell eval. board code. It
        doesn't seem to have been changed for some time now. Perhaps you
        need to look at your "agp ops"? Is this helpfull?

PCI: Probing PCI hardware
Scanning bus 00
Found 00:60 [8086/1008] 000200 00
Fixups for bus 00
Bus scan for 00 returning with max=00
Scanning bus 01
Found 01:38 [1077/2312] 000c04 00
Found 01:39 [1077/2312] 000c04 00
Fixups for bus 01
Bus scan for 01 returning with max=01

Ben writes>
           >Argh ????? They don't appear on PCI ? What piece of SHIT is
           >this bridge ?

           >Really totally insane.


        Would you care to rethink those comments?

        It seems unfair to call a chip shit, when you don't have
        a manual for it and appear to lack any knowledge of it's
        internals. Do the math for 3 GigE ports running line speed with
        64 byte packets, and full bandwidth on 2 PCI busses...

SVEN writes>

        >Well, the Discovery II use a internal crossbar switch, and the
        >ethernet are on the same level as the pci buses. This makes for
        >very efficient networking i guess, but has problems. In fact,
        >each of these ones has the same priority as each pci bus.

        Each of the ports of the crossbar has a programmable arbiter.
        (at least on Discovery, which is what we use.)

On Thu, 2004-01-01 at 13:11, Sven Luther wrote:
>
> I am currently trying to port linux to the Pegasos 2, which uses the
> Marvell Discovery 2 chip, and has two independent pci controllers,
> of which one is faked as an agp bus. This is with a modified 2.4.23
> kernel from the linuxppc_2_4 branch.
>
> Anyway, these two independent controllers both have one bus 0 on them:
>
> PCI bus 0 controlled by pci at 80000000
> PCI bus 0 controlled by pci at c0000000
>
> (This coming from chrp_find_bridges).
>
> For the pci bus, at 80000000, a simple indirect access can be used,
> and i setup a specialized io ops for the faked agp bus since it needs
> special care.
>
> Later, in pcibios_init, there is a problem in the pci_scan_bus. The
> first bus has no problems :
>
> Scanning bus 00
> Found 00:00 [11ab/6460] 000600 00
> Found 00:08 [1106/3044] 000c00 00
> Found 00:28 [1000/0001] 000100 00
> Found 00:60 [1106/8231] 000601 00
> Found 00:61 [1106/0571] 000101 00
> Found 00:62 [1106/3038] 000c03 00
> Found 00:63 [1106/3038] 000c03 00
> Found 00:64 [1106/8235] 000000 00
> Found 00:65 [1106/3058] 000401 00
> Found 00:66 [1106/3068] 000780 00
> Found 00:68 [1106/3065] 000200 00
> Fixups for bus 00
> Bus scan for 00 returning with max=00
>
> But the second fails with :
>
> PCI: Bus 00 already known
>
> Which comes because in drivers/pci/pci.c:pci_bus_exists does handle
> only buses, and thus know nothing of separate pci bridges with the
> same bus number, and thus the kernel dies when accessing bus->xxx
> something in pcibios_init.
>
> Now, i know that the powermacs in particular, and maybe others, also
> have this case of different same numbered pci buses with different
> base addresses, and that this kind of situation has already been
> solved.
>
> So, what is the workaround here, and where does it get set.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: multiple separate pci bridges ...

[Marcus Barrow]

        Hi Sven:

        I think there is some basic brokeness in the Discovery
        specific code. I've been meaning to go and look at some
        of the original code, to see if some stuff was left on
        the floor.

        On our evalution boards the P2P registers are not set
        up. So you indeed have two PCI bus 0. The P2P register
        for PCI bus 1 should be initialized to identify it as
        bus 1, or whatever the right value is considering
        any more bridges found on bus 0.

        If you insert pci cards which contain bridges on them,
        they only work on the bus 0 slots...

        Do you have something like ppcboot or uboot starting
        your system or just a kernel? If you are using a real
        boot rom, you could try initializing the p2p registers
        by hand, then booting...

        I'm hoping to work on this in the next few days.

                                Regards, Marcus Barrow




On Mon, 2004-01-05 at 11:40, Sven Luther wrote:
>
> On Sun, Jan 04, 2004 at 11:06:08PM +0100, Sven Luther wrote:
> >
> > On Mon, Jan 05, 2004 at 08:45:24AM +1100, Benjamin Herrenschmidt
wrote:
> > >
> > >
> > > > PCI: Probing PCI hardware
> > > > Scanning bus 00
> > > > Found 00:00 [11ab/6460] 000600 00
> > > > Found 00:08 [1106/3044] 000c00 00
> > > > Found 00:28 [1000/0001] 000100 00
> > > > Found 00:60 [1106/8231] 000601 00
> > > > Found 00:61 [1106/0571] 000101 00
> > > > Found 00:62 [1106/3038] 000c03 00
> > > > Found 00:63 [1106/3038] 000c03 00
> > > > Found 00:64 [1106/8235] 000000 00
> > > > Found 00:65 [1106/3058] 000401 00
> > > > Found 00:66 [1106/3068] 000780 00
> > > > Found 00:68 [1106/3065] 000200 00
> > > > Fixups for bus 00
> > > > Bus scan for 00 returning with max=00
> > > > Scanning bus 10
> > > > Fixups for bus 10
> > > > Bus scan for 10 returning with max=10
> > > > PCI: Cannot allocate resource region 4 of device 00:00.0
> > > >
> > > > So, the 0x10 incrementation is fine, but the agp bus is not
working
> > > > correctly, and thus the agp graphic card not recognized (and
thus no
> > > > radeonfb).
> > > >
> > > > That said, i believe the 00:00.0 (should that not be 10:00.0
because of
> > > > the renumbering) is the Marvell Discovery II pci controller, and
the
> > > > actual agp card should be found just behind.
> > >
> > > What video card is this ?
> >
> > A Radeon 9200 SE 64M, from Xpert Vision. The kernel is patched for
> > Radeon 9200 SE support. (Both i have are Yd models).
>
> Ok, i got it to work finally, at least upto asking me for a root
> filesystem, which i don't yet have on this harddisk.
>
> Now, I have some technical questions about how to best do a few
things.
>
> I was recommended to set the ppc_md.pci_exclude_device so that the
> device 0 (the marvell bridge itself) is not seen by linux. I did this
by
> following the 4xx example, which should be ok.
>
> But, i have to do access some address which i need to ioremap. I
created
> a hose->cfg_peg2_magic to to put this ioremapped address in. I guess
> this is not the most clean way of doing this or something, any hint on
> how to best do it ? I need to set this in chrp_find_bridges or
something
> such, and use the address in the read/write_config functions.
>
> Also, i had to manually set hose->bus_offset = 0x10, since that didn't
> seem to be set automatically. I don't know why though.
>
> And finally, about the stuff which blocked me most, i am somewhat
> bewildered. I use this :
>
>         data = 0x80000000 | ((dev->bus->number - hose->bus_offset) <<
16)
>               | (dev->devfn << 8) | (offset & 0xff);
>
> As the address to write to, but the indirect pci stuff uses offset &
> 0xfc, which blanks bits 0 & 1 for pci config type selection. What am i
> misunderstanding or doing wrong here ?
>
> A, and a last question to Rob Baxter, did you manage to get the
builtin
> gigabyte ethernet port to work, and if yes, with which code, an
> existing driver or some home built driver. Can you eventually share
the
> code or something such ?
>
> Anyway, thanks for your help.
>
> Friendly,
>
> Sven Luther
>
>
>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/