Going beyond 256 PCI buses

Going beyond 256 PCI buses

[Tom Gall]

<Forgive if this is a dub... but the message I composed yesterday didn't
appear to be posted>

  Anyway, Hi All,

  I was wondering if there are any other folks out there like me who
have the 256 PCI bus limit looking at them straight in the face? If so,
it'd be nice to collaborate and come up with a more general solution
that would hopefully work towards the greater good.

  I live in ppc64 land which is a new arch that the linux kernel has
been ported to. The boxes we run on tend to be big.

  The box that I'm wrestling with, has a setup where each PHB has an
additional id, then each PHB can have up to 256 buses.  So when you are
talking to a device, the scheme is phbid, bus, dev etc etc. Pretty easy
really.

  I am getting for putting something like this into the kernel at large,
it would probably be best to have a CONFIG_GREATER_THAN_256_BUSES or
some such.

  Anyways, thoughts? opinions?

--
Hakuna Matata,

Tom

-----------------------------------------------------------
ppc64 Maintainer     IBM Linux Technology Center
"My heart is human, my blood is boiling, my brain IBM" -- Mr Roboto
tgall@rochcivictheatre.org
tom_gall@vnet.ibm.com

Re: Going beyond 256 PCI buses

[Albert D. Cahalan]

Tom Gall writes:

>   I was wondering if there are any other folks out there like me who
> have the 256 PCI bus limit looking at them straight in the face?

I might. The need to reserve bus numbers for hot-plug looks like
a quick way to waste all 256 bus numbers.

> each PHB has an
> additional id, then each PHB can have up to 256 buses.

Try not to think of him as a PHB with an extra id. Lots of people
have weird collections. If your boss wants to collect buses, well,
that's his business. Mine likes boats. It's not a big deal, really.

(Did you not mean your pointy-haired boss has mental problems?)

Re: Going beyond 256 PCI buses

[Tom Gall]

"Albert D. Cahalan" wrote:
> 
> Tom Gall writes:
> 
> >   I was wondering if there are any other folks out there like me who
> > have the 256 PCI bus limit looking at them straight in the face?
> 
> I might. The need to reserve bus numbers for hot-plug looks like
> a quick way to waste all 256 bus numbers.

Hi Albert,

  yeah I'll be worring about this one too in time. Two birds with one stone
wouldn't be a bad thing. Hopefully it doesn't translate into needing a
significantly larger stone. 

> > each PHB has an
> > additional id, then each PHB can have up to 256 buses.
> 
> Try not to think of him as a PHB with an extra id. Lots of people
> have weird collections. If your boss wants to collect buses, well,
> that's his business. Mine likes boats. It's not a big deal, really.

Heh.... PHB==Primary Host Bridge  ... but I'll be sure to pass the word onto my
PHB that there's a used greyhound sale... <bah-bum-bum-tshhh>

Anyway, it really is a new id, at least for the implementation on this box. So
PHB0 could have 256 buses, and PHB1 could have 10 buses, PHB2 could have ....
you get the idea.

Hot plug would still have the problem in that it'd have 256 bus numbers in the
namespace of a PHB to manage. Hot plug under a different PHB would have another
256 to play with.

Regards,

Tom

-- 
Tom Gall - PPC64 Maintainer      "Where's the ka-boom? There was
Linux Technology Center           supposed to be an earth
(w) tom_gall@vnet.ibm.com         shattering ka-boom!"
(w) 507-253-4558                 -- Marvin Martian
(h) tgall@rochcivictheatre.org
http://www.ibm.com/linux/ltc/projects/ppc

Re: Going beyond 256 PCI buses

[Jeff Garzik]

Tom Gall wrote:
>   The box that I'm wrestling with, has a setup where each PHB has an
> additional id, then each PHB can have up to 256 buses.  So when you are
> talking to a device, the scheme is phbid, bus, dev etc etc. Pretty easy
> really.
> 
>   I am getting for putting something like this into the kernel at large,
> it would probably be best to have a CONFIG_GREATER_THAN_256_BUSES or
> some such.

We don't need such a CONFIG_xxx at all.  The current PCI core code
should scale up just fine.

According to the PCI spec it is -impossible- to have more than 256 buses
on a single "hose", so you simply have to implement multiple hoses, just
like Alpha (and Sparc64?) already do.  That's how the hardware is forced
to implement it...

	Jeff


-- 
Jeff Garzik      | Andre the Giant has a posse.
Building 1024    |
MandrakeSoft     |

Re: Going beyond 256 PCI buses

[David S. Miller]

Jonathan Lundell writes:
 > That's right, of course. A small problem is that dev->slot_name 
 > becomes ambiguous, since it doesn't have any hose identification. Nor 
 > does it have any room for the hose id; it's fixed at 8 chars, and 
 > fully used (bb:dd.f\0).

Sure, Jeff and I already said that slot name strings need to change.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Jonathan Lundell]

At 7:24 AM -0700 2001-06-14, David S. Miller wrote:
>2) Figure out what to do wrt. sys_pciconfig_{read,write}()
>
>    They ought to really be deprecated and the /proc/bus/pci
>    way is the expected way to go about doing these things.
>    The procfs interface is more intelligent, less clumsy, and
>    even allows you to mmap() PCI cards portably (instead of
>    doing crap like mmap()s on /dev/mem, yuck).
>
>    Actually, there only real issue here is what happens when
>    userspace does PCI config space reads to things which talk
>    about "bus numbers" since those will be 8-bit as this is a
>    hardware imposed type.  These syscalls take long args already
>    so they could use >256 bus numbers just fine.
>
>Basically, this 256 bus limit in Linux is a complete fallacy.

There's also the question of pci_ops() (PCI config space access 
routines). Alpha (titan in this example) uses:

	struct pci_controler *hose = dev->sysdata;

but the general idea is that you need the hose info to do 
config-space access on PCI.

The 256-bus fallacy is caused in part by an ambiguity in what we mean 
by "bus". In PCI spec talk, a bus is a very specify thing with an 
8-bit bus number, and this is reflected in various registers and 
address formats. On the other hand we have the general concept of 
buses, which includes the possibility of multiple PCI controllers, 
and thus multiple domains of 256 buses each.

As I recall, even a midline chipset such as the ServerWorks LE 
supports the use of two north bridges, which implies two PCI bus 
domains.

I'd favor adopting something like the Alpha approach, but with a 
dedicated struct pci_device item (and with "controller" spelled 
correctly). And I suppose "domain" sounds a little more sober than 
"hose". But since the domain information is necessary to access 
configuration registers, it really needs to be included in struct 
pci_device.
-- 
/Jonathan Lundell.

Re: Going beyond 256 PCI buses

[Jonathan Lundell]

At 12:03 PM -0700 2001-06-14, David S. Miller wrote:
>Jonathan Lundell writes:
>  > As I recall, even a midline chipset such as the ServerWorks LE
>  > supports the use of two north bridges, which implies two PCI bus
>  > domains.
>
>It hides this fact by making config space accesses respond in such a
>way that it appears that it is all behind one PCI controller.  The
>BIOS even avoids allowing any of the MEM and I/O resources from
>overlapping.

So we end up with a single domain and max 256 buses. Still, it's not 
behavior one can count on. Sun's U2P PCI controller certainly creates 
a new PCI domain for each controller. It's easier in architectures 
other than IA32, in a way, since they typically don't have the 64KB 
IO-space addressing limitation that makes heavily bridged systems 
problematical on IA32 (one tends to run out of IO space).
-- 
/Jonathan Lundell.

Re: Going beyond 256 PCI buses

[David S. Miller]

Jeff Garzik writes:
 > According to the PCI spec it is -impossible- to have more than 256 buses
 > on a single "hose", so you simply have to implement multiple hoses, just
 > like Alpha (and Sparc64?) already do.  That's how the hardware is forced
 > to implement it...

Right, what userspace had to become aware of are "PCI domains" which
is just another fancy term for a "hose" or "controller".

All you have to do is (right now, the kernel supports this fully)
open up a /proc/bus/pci/${BUS}/${DEVICE} node and then go:

	domain = ioctl(fd, PCIIOC_CONTROLLER, 0);

Viola.

There are only two real issues:

1) Extending the type bus numbers use inside the kernel.

   Basically how most multi-controller platforms work now
   is they allocate bus numbers in the 256 bus space as
   controllers are probed.  If we change the internal type
   used by the kernel to "u32" or whatever, we expand that
   available space accordingly.

   For the lazy, basically go into include/linux/pci.h
   and change the "unsigned char"s in struct pci_bus into
   some larger type.  This is mindless work.

2) Figure out what to do wrt. sys_pciconfig_{read,write}()

   They ought to really be deprecated and the /proc/bus/pci
   way is the expected way to go about doing these things.
   The procfs interface is more intelligent, less clumsy, and
   even allows you to mmap() PCI cards portably (instead of
   doing crap like mmap()s on /dev/mem, yuck).

   Actually, there only real issue here is what happens when
   userspace does PCI config space reads to things which talk
   about "bus numbers" since those will be 8-bit as this is a
   hardware imposed type.  These syscalls take long args already
   so they could use >256 bus numbers just fine.

Basically, this 256 bus limit in Linux is a complete fallacy.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Jeff Garzik]

"David S. Miller" wrote:
> 1) Extending the type bus numbers use inside the kernel.
> 
>    Basically how most multi-controller platforms work now
>    is they allocate bus numbers in the 256 bus space as
>    controllers are probed.  If we change the internal type
>    used by the kernel to "u32" or whatever, we expand that
>    available space accordingly.
> 
>    For the lazy, basically go into include/linux/pci.h
>    and change the "unsigned char"s in struct pci_bus into
>    some larger type.  This is mindless work.

Why do you want to make the bus number larger than the PCI bus number
register?

It seems like adding 'unsigned int domain_num' makes more sense, and is
more correct.  Maybe that implies fixing up other code to use a
(domain,bus) pair, but that's IMHO a much better change than totally
changing the interpretation of pci_bus::bus_number...


> 2) Figure out what to do wrt. sys_pciconfig_{read,write}()

3) (tiny issue) Change pci_dev::slot_name such that it includes the
domain number.  This is passed to userspace by SCSI and net drivers as a
way to allow userspace to associate a kernel interface with a bus
device.


> Basically, this 256 bus limit in Linux is a complete fallacy.

yep

Regards,

	Jeff


-- 
Jeff Garzik      | Andre the Giant has a posse.
Building 1024    |
MandrakeSoft     |

Re: Going beyond 256 PCI buses

[David S. Miller]

Jeff Garzik writes:
 > Why do you want to make the bus number larger than the PCI bus number
 > register?

This isn't it.  What I'm trying to provoke thought on is
"is there a way to make mindless apps using these syscalls
work transparently"

I think the answer is no.  Apps should really fetch info out
of /proc/bus/pci and use the controller ioctl.

But someone could surprise me :-)

 > It seems like adding 'unsigned int domain_num' makes more sense, and is
 > more correct.  Maybe that implies fixing up other code to use a
 > (domain,bus) pair, but that's IMHO a much better change than totally
 > changing the interpretation of pci_bus::bus_number...

Correct, I agree.  But I don't even believe we should be sticking
the domain thing into struct pci_bus.

It's a platform thing.  Most platforms have a single domain, so why
clutter up struct pci_bus with this value?  By this reasoning we could
say that since it's arch-specific, this stuff belongs in sysdata or
wherever.

And this is what is happening right now.  So in essence, the work is
done :-)  The only "limiting factor" is that x86 doesn't support
multiple domains as some other platforms do.  So all these hot-plug
patches just need to use domains properly, and perhaps add domain
support to X86 when one of these hot-plug capable controllers are
being used.

 > > 2) Figure out what to do wrt. sys_pciconfig_{read,write}()
 > 
 > 3) (tiny issue) Change pci_dev::slot_name such that it includes the
 > domain number.  This is passed to userspace by SCSI and net drivers as a
 > way to allow userspace to associate a kernel interface with a bus
 > device.

Sure.  It's an address and the domain is part of the address.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Jeff Garzik]

"David S. Miller" wrote:
> Jeff Garzik writes:
>  > Why do you want to make the bus number larger than the PCI bus number
>  > register?
> 
> This isn't it.  What I'm trying to provoke thought on is
> "is there a way to make mindless apps using these syscalls
> work transparently"
> 
> I think the answer is no.  Apps should really fetch info out
> of /proc/bus/pci and use the controller ioctl.
> 
> But someone could surprise me :-)

yeah, those syscalls weren't built with much eye towards the future. 
And I don't think they are present in other OS's either...


>  > It seems like adding 'unsigned int domain_num' makes more sense, and is
>  > more correct.  Maybe that implies fixing up other code to use a
>  > (domain,bus) pair, but that's IMHO a much better change than totally
>  > changing the interpretation of pci_bus::bus_number...
> 
> Correct, I agree.  But I don't even believe we should be sticking
> the domain thing into struct pci_bus.
> 
> It's a platform thing.  Most platforms have a single domain, so why
> clutter up struct pci_bus with this value?  By this reasoning we could
> say that since it's arch-specific, this stuff belongs in sysdata or
> wherever.

Pretty much any arch with a PCI slot can have multiple domains, now that
hotplug controllers are out and about.  So it seems a generic enough
concept to me...


> And this is what is happening right now.  So in essence, the work is
> done :-)  The only "limiting factor" is that x86 doesn't support
> multiple domains as some other platforms do.  So all these hot-plug
> patches just need to use domains properly, and perhaps add domain
> support to X86 when one of these hot-plug capable controllers are
> being used.

point.

Regards,

	Jeff


-- 
Jeff Garzik      | Andre the Giant has a posse.
Building 1024    |
MandrakeSoft     |

Re: Going beyond 256 PCI buses

[Jeff Garzik]

Jeff Garzik wrote:
> Pretty much any arch with a PCI slot can have multiple domains, now that
> hotplug controllers are out and about.  So it seems a generic enough
> concept to me...

Um, correction:  that is assuming a certain implementation...  you can
certainly implement a hotplug controller as another PCI-PCI bridge,
AFAICS, too.

-- 
Jeff Garzik      | Andre the Giant has a posse.
Building 1024    |
MandrakeSoft     |

Re: Going beyond 256 PCI buses

[Albert D. Cahalan]

David S. Miller writes:
> Jeff Garzik writes:

>> According to the PCI spec it is -impossible- to have more than 256
>> buses on a single "hose", so you simply have to implement multiple
>> hoses, just like Alpha (and Sparc64?) already do.  That's how the
>> hardware is forced to implement it...
>
> Right, what userspace had to become aware of are "PCI domains" which
> is just another fancy term for a "hose" or "controller".
>
> All you have to do is (right now, the kernel supports this fully)
> open up a /proc/bus/pci/${BUS}/${DEVICE} node and then go:
> 
> 	domain = ioctl(fd, PCIIOC_CONTROLLER, 0);
>
> Viola.
>
> There are only two real issues:

No, three.

0) The API needs to be taken out and shot.

   You've added an ioctl. This isn't just any ioctl. It's a
   wicked nasty ioctl. It's an OH MY GOD YOU CAN'T BE SERIOUS
   ioctl by any standard.

   Consider the logical tree:
   hose -> bus -> slot -> function -> bar

   Well, the hose and bar are missing. You specify the middle
   three parts in the filename (with slot and function merged),
   then use an ioctl to specify the hose and bar.

   Doing the whole thing by filename would be better. Else
   why not just say "screw it", open /proc/pci, and do the
   whole thing by ioctl? Using ioctl for both the most and
   least significant parts of the path while using a path
   for the middle part is Wrong, Bad, Evil, and Broken.

   Fix:

   /proc/bus/PCI/0/0/3/0/config   config space
   /proc/bus/PCI/0/0/3/0/0        the first bar
   /proc/bus/PCI/0/0/3/0/1        the second bar
   /proc/bus/PCI/0/0/3/0/driver   info about the driver, if any
   /proc/bus/PCI/0/0/3/0/event    hot-plug, messages from driver...

   Then we have arch-specific MMU cruft. For example the PowerPC
   defines bits that affect caching, ordering, and merging policy.
   The chips from IBM also define an endianness bit. I don't think
   this ought to be an ioctl either. Maybe mmap() flags would be
   reasonable. This isn't just for PCI; one might do an anon mmap
   with pages locked and cache-incoherent for better performance.

> 1) Extending the type bus numbers use inside the kernel.
...
> 2) Figure out what to do wrt. sys_pciconfig_{read,write}()
...

Re: Going beyond 256 PCI buses

[David S. Miller]

Albert D. Cahalan writes:
 >    You've added an ioctl. This isn't just any ioctl. It's a
 >    wicked nasty ioctl. It's an OH MY GOD YOU CAN'T BE SERIOUS
 >    ioctl by any standard.

It's an ioctl which allows things to work properly in the framework we
currently have.

 >    Fix:
 > 
 >    /proc/bus/PCI/0/0/3/0/config   config space

Which breaks xfree86 instantly.  This fix is unacceptable.

In fact, the current ioctl/mmap machanism was discussed with and
agreed to by the PPC, Alpha, and Sparc64 folks.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Albert D. Cahalan]

David S. Miller writes:
> Albert D. Cahalan writes:

>>    You've added an ioctl. This isn't just any ioctl. It's a
>>    wicked nasty ioctl. It's an OH MY GOD YOU CAN'T BE SERIOUS
>>    ioctl by any standard.
>
> It's an ioctl which allows things to work properly in the
> framework we currently have.

It's a hack that keeps us stuck with the existing mistakes.
We need a transition path to get us away from the old mess.

>>    Fix:
>>
>>    /proc/bus/PCI/0/0/3/0/config   config space
>
> Which breaks xfree86 instantly.  This fix is unacceptable.

Nope. Keep /proc/bus/pci until Linux 3.14 if you like.
The above is /proc/bus/PCI. That's "PCI", not "pci".
We still have /proc/pci after all.

> In fact, the current ioctl/mmap machanism was discussed with and
> agreed to by the PPC, Alpha, and Sparc64 folks.

Did you somehow miss when Linus scolded you a few weeks ago?
How about asking somebody who helps maintain /proc, like Al Viro?

Re: Going beyond 256 PCI buses

[David S. Miller]

Albert D. Cahalan writes:
 > >>    /proc/bus/PCI/0/0/3/0/config   config space
 > >
 > > Which breaks xfree86 instantly.  This fix is unacceptable.
 > 
 > Nope. Keep /proc/bus/pci until Linux 3.14 if you like.
 > The above is /proc/bus/PCI. That's "PCI", not "pci".
 > We still have /proc/pci after all.

Oh I see.

Well, xfree86 and other programs aren't going to look there, so
something had to be done about the existing /proc/bus/pci/* hierarchy.

To be honest, xfree86 needs the controller information not for the
sake of device probing, it needs it to detect resource conflicts.

Anyways, I agree with your change, sure.

 > Did you somehow miss when Linus scolded you a few weeks ago?

You mean specifically what?

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Jeff Garzik]

Thinking a bit more independently of bus type, and with an eye toward's
2.5's s/pci_dev/device/ and s/pci_driver/driver/, would it be useful to
go ahead and codify the concept of PCI domains into a more generic
concept of bus tree numbers?  (or something along those lines)  That
would allow for a more general picture of the entire system's device
tree, across buses.

First sbus bus is tree-0, first PCI bus tree is tree-1, second PCI bus
tree is tree-2, ...

-- 
Jeff Garzik      | Andre the Giant has a posse.
Building 1024    |
MandrakeSoft     |

Re: Going beyond 256 PCI buses

[David S. Miller]

Jeff Garzik writes:
 > Thinking a bit more independently of bus type, and with an eye toward's
 > 2.5's s/pci_dev/device/ and s/pci_driver/driver/, would it be useful to
 > go ahead and codify the concept of PCI domains into a more generic
 > concept of bus tree numbers?  (or something along those lines)  That
 > would allow for a more general picture of the entire system's device
 > tree, across buses.
 > 
 > First sbus bus is tree-0, first PCI bus tree is tree-1, second PCI bus
 > tree is tree-2, ...

If you're going to do something like this, ie. true hierarchy, why not
make one tree which is "system", right? Use /proc/bus/${controllernum}
ala:

/proc/bus/0/type	--> "sbus", "pci", "zorro", etc.
/proc/bus/0/*		--> for type == "pci" ${bus}/${dev}.${fn}
			    for type == "sbus" ${slot}
			    ...

How about this?

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Jeff Garzik]

"David S. Miller" wrote:
> 
> Jeff Garzik writes:
>  > Thinking a bit more independently of bus type, and with an eye toward's
>  > 2.5's s/pci_dev/device/ and s/pci_driver/driver/, would it be useful to
>  > go ahead and codify the concept of PCI domains into a more generic
>  > concept of bus tree numbers?  (or something along those lines)  That
>  > would allow for a more general picture of the entire system's device
>  > tree, across buses.
>  >
>  > First sbus bus is tree-0, first PCI bus tree is tree-1, second PCI bus
>  > tree is tree-2, ...
> 
> If you're going to do something like this, ie. true hierarchy, why not
> make one tree which is "system", right? Use /proc/bus/${controllernum}
> ala:
> 
> /proc/bus/0/type        --> "sbus", "pci", "zorro", etc.
> /proc/bus/0/*           --> for type == "pci" ${bus}/${dev}.${fn}
>                             for type == "sbus" ${slot}
>                             ...
> 
> How about this?

ok with me.  would bus #0 be the system or root bus?  that would be my
preference, in a tiered system like this.

-- 
Jeff Garzik      | Andre the Giant has a posse.
Building 1024    |
MandrakeSoft     |

Re: Going beyond 256 PCI buses

[David S. Miller]

Jeff Garzik writes:
 > ok with me.  would bus #0 be the system or root bus?  that would be my
 > preference, in a tiered system like this.

Bus 0 is controller 0, of whatever bus type that happens to be.
If we want to do something special we could create something
like /proc/bus/root or whatever, but I feel this unnecessary.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Benjamin Herrenschmidt]

>
>Bus 0 is controller 0, of whatever bus type that happens to be.
>If we want to do something special we could create something
>like /proc/bus/root or whatever, but I feel this unnecessary.

<old rant>

Mostly, except for one thing: legacy devices expecting ISA-like
ops on a given domain which currently need some way to know
what PCI bus hold the ISA bus. 

While we are at it, I'd be really glad if we could agree on a
way to abstract the current PIO scheme to understand the fact
that any domain can actually have "legacy ISA-like" devices.

One example is that any domain can have a VGA controller that
requires a bit of legacy PIO & ISA-mem stuff. In the same vein,
any domain can have an ISA-bridge used to wired 16bits devices

Another example is an embedded device which could use the
domain abstraction to represent different IO busses on which
old-style 16bits chips are wired.

I beleive there will always be need for some platform specific
hacking at probe-time to handle those, but we can at least make
the inx/outx functions/macros compatible with such a scheme,
possibly by requesting an ioremap equivalent to be done so that
we stop passing them real PIO addresses, but a cookie obtained
in various platform specific ways.

For the case of PCI (which would handle both the VGA case and
the multiple PCI<->ISA bridge case), one possibility is to
provide a function returning resources for the "legacy" PIO
and MMIO regions if any on a given domain. This is especially
true for ISA-memory (used mostly for VGA) as host controllers
for non-x86 platforms usually have a special window somewhere
in the bus space for generating <64k mem cycles on the PCI bus.

</old rant>

Ben.

Re: Going beyond 256 PCI buses

[Jeff Garzik]

Benjamin Herrenschmidt wrote:
> While we are at it, I'd be really glad if we could agree on a
> way to abstract the current PIO scheme to understand the fact
> that any domain can actually have "legacy ISA-like" devices.

ioremap for outb/outw/outl.

IMHO of course.

I think rth requested pci_ioremap also...

-- 
Jeff Garzik      | Andre the Giant has a posse.
Building 1024    |
MandrakeSoft     |

Re: Going beyond 256 PCI buses

[David S. Miller]

Jeff Garzik writes:
 > I think rth requested pci_ioremap also...

It really isn't needed, and I understand why Linus didn't like the
idea either.  Because you can encode the bus etc. info into the
resource addresses themselves.

On sparc64 we just so happen to stick raw physical addresses into the
resources, but that is just one way of implementing it.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Benjamin Herrenschmidt]

>It really isn't needed, and I understand why Linus didn't like the
>idea either.  Because you can encode the bus etc. info into the
>resource addresses themselves.
>
>On sparc64 we just so happen to stick raw physical addresses into the
>resources, but that is just one way of implementing it.

That would be fine for PIO on PCI, but still is an issue for
VGA-like devices that need to issue some "legacy" cycles on
a given domain. Currently, on PPC, inx/outx will only go to
one bus (arbitrarily choosen during boot) because of that,
meaning that we can't have 2 VGA cards on 2 different domains

That's why I'd love to see a review of the "legacy" (ISA) stuff
in general. I understand that can require a bit of updating of
a lot of legacy drivers to do the proper ioremap's, but that would
help a lot, including some weird embedded archs which love using
those cheap 16 bits devices on all sorts of custom busses. In
those case, only the probe part will have to be hacked since the
drivers will all cleanly use a "base" obtained from that probe-time
ioremap before doing inx/outx.

I'd be happy to help bringing drivers up-to-date (however, I don't
have an x86 box to test with) once we agree on the way do go.

Ben.

Re: Going beyond 256 PCI buses

[David S. Miller]

Benjamin Herrenschmidt writes:
 > That would be fine for PIO on PCI, but still is an issue for
 > VGA-like devices that need to issue some "legacy" cycles on
 > a given domain. Currently, on PPC, inx/outx will only go to
 > one bus (arbitrarily choosen during boot) because of that,
 > meaning that we can't have 2 VGA cards on 2 different domains

It's funny you mention this because I have been working on something
similar recently.  Basically making xfree86 int10 and VGA poking happy
on sparc64.

But this has no real use in the kernel.  (actually I take this back,
read below)

You have a primary VGA device, that is the one the bios (boot
firmware, whatever you want to call it) enables to respond to I/O and
MEM accesses, the rest are configured to VGA pallette snoop and that's
it.  The primary VGA device is the kernel console (unless using some
fbcon driver of course), and that's that.

The secondary VGA devices are only interesting to things like the X
server, and xfree86 does all the enable/disable/bridge-forward-vga
magic when doing multi-head.

Perhaps, you might need to program the VGA resources of some device to
use it in a fbcon driver (ie. to init it or set screen crt parameters,
I believe the tdfx requires the latter which is why I'm having a devil
of a time getting it to work on my sparc64 box).  This would be a
seperate issue, and I would not mind at all seeing an abstraction for
this sort of thing, let us call it:

	struct pci_vga_resource {
		struct resource io, mem;
	};

	int pci_route_vga(struct pci_dev *pdev, struct pci_vga_resource *res);
	pci_restore_vga(void);

So you'd go:

	struct pci_vga_resource vga_res;
	int err;

	err = pci_route_vga(tdfx_pdev, &vga_res);

	if (err)
		barf();
	vga_ports = ioremap(vga_res.io.start, vga_res.io.end-vga_res.io.start+1);
	program_video_crtc_params(vga_ports);
	iounmap(vga_ports);
	vga_fb = ioremap(vga_res.mem.start, vga_res.mem.end-vga_res.mem.start+1);
	clear_vga_fb(vga_fb);
	iounmap(vga_fb);

	pci_restore_vga();
	
pci_route_vga does several things:

1) It saves the current VGA routing information.
2) It configures busses and VGA devices such that PDEV responds to
   VGA accesses, and other VGA devices just VGA palette snoop.
3) Fills in the pci_vga_resources with
   io: 0x320-->0x340 in domain PDEV lives, vga I/O regs
   mem: 0xa0000-->0xc0000 in domain PDEV lives, video ram

pci_restore_vga, as the name suggests, restores things back to how
they were before the pci_route_vga() call.  Maybe also some semaphore
so only one driver can do this at once and you can't drop the
semaphore without calling pci_restore_vga().  VC switching into the X
server would need to grab this thing too.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Benjamin Herrenschmidt]

>It's funny you mention this because I have been working on something
>similar recently.  Basically making xfree86 int10 and VGA poking happy
>on sparc64.

Heh, world is small ;)

>But this has no real use in the kernel.  (actually I take this back,
>read below)

yup, fbcon at least... 

>You have a primary VGA device, that is the one the bios (boot
>firmware, whatever you want to call it) enables to respond to I/O and
>MEM accesses, the rest are configured to VGA pallette snoop and that's
>it.  The primary VGA device is the kernel console (unless using some
>fbcon driver of course), and that's that.

Yup, fbcon is what I have in mind here

>The secondary VGA devices are only interesting to things like the X
>server, and xfree86 does all the enable/disable/bridge-forward-vga
>magic when doing multi-head.

and multihead fbcon. 

>Perhaps, you might need to program the VGA resources of some device to
>use it in a fbcon driver (ie. to init it or set screen crt parameters,
>I believe the tdfx requires the latter which is why I'm having a devil
>of a time getting it to work on my sparc64 box).  This would be a
>seperate issue, and I would not mind at all seeing an abstraction for
>this sort of thing, let us call it:
>
>	struct pci_vga_resource {
>		struct resource io, mem;
>	};
>
>	int pci_route_vga(struct pci_dev *pdev, struct pci_vga_resource *res);
>	pci_restore_vga(void);
>
> [.../...]

Well... that would work for VGA itself (note that this semaphore
you are talking about should be shared some way with the /proc
interface so XFree can be properly sync'ed as well).

But I still think it may be useful to generalize the idea to 
all kind of legacy IO & PIOs. I definitely agree that VGA is a kind
of special case, mostly because of the necessary exclusion on
the VGA IO response.

But what about all those legacy drivers that will issue inx/outx
calls without an ioremap ? Should they call ioremap with hard-coded
legacy addresses ? There are chipsets containing things like legacy
timers, legacy keyboard controllers, etc... and in some (rare I admit)
cases, those may be scattered (or multiplied) on various domains. 
If we decide we don't handle those, then well, I won't argue more
(it's mostly an estethic rant on my side ;), but the problem of
wether they should call ioremap or not is there, and since the
ISA bus can be "mapped" anywhere in the bus space by the host bridge,
there need to be a way to retreive the ISA resources in general for
a given domain.

That's why I'd suggest something like 

pci_get_isa_mem(struct resource* isa_mem);
pci_get_isa_io(struct resource* isa_io);

(I prefer 2 different functions as some platforms like powermac just
don't provide the ISA mem space at all, there's no way to generate
a memory cycle in the low-address range on the PCI bus of those and
they don't have a PCI<->ISA bridge), so I like having the ability of
one of the functions returning an error and not the other.

Also, having the same ioremap() call for both mem IO and PIO means
that things like 0xc0000 cannot be interpreted. It's a valid ISA-mem
address in the VGA space and a valid PIO address on a PCI bus that
supports >64k of PIO space.

I beleive it would make things clearer (and probably implementation
simpler) to separate ioremap and pioremap.

Ben.

>So you'd go:
>
>	struct pci_vga_resource vga_res;
>	int err;
>
>	err = pci_route_vga(tdfx_pdev, &vga_res);
>
>	if (err)
>		barf();
>	vga_ports = ioremap(vga_res.io.start, vga_res.io.end-vga_res.io.start+1);
>	program_video_crtc_params(vga_ports);
>	iounmap(vga_ports);
>	vga_fb = ioremap(vga_res.mem.start, vga_res.mem.end-vga_res.mem.start+1);
>	clear_vga_fb(vga_fb);
>	iounmap(vga_fb);
>
>	pci_restore_vga();
>	
>pci_route_vga does several things:
>
>1) It saves the current VGA routing information.
>2) It configures busses and VGA devices such that PDEV responds to
>   VGA accesses, and other VGA devices just VGA palette snoop.
>3) Fills in the pci_vga_resources with
>   io: 0x320-->0x340 in domain PDEV lives, vga I/O regs
>   mem: 0xa0000-->0xc0000 in domain PDEV lives, video ram
>
>pci_restore_vga, as the name suggests, restores things back to how
>they were before the pci_route_vga() call.  Maybe also some semaphore
>so only one driver can do this at once and you can't drop the
>semaphore without calling pci_restore_vga().  VC switching into the X
>server would need to grab this thing too.

Re: Going beyond 256 PCI buses

[David S. Miller]

Benjamin Herrenschmidt writes:
 > Well... that would work for VGA itself (note that this semaphore
 > you are talking about should be shared some way with the /proc
 > interface so XFree can be properly sync'ed as well).

Once XFree takes ownership of the VC it has control of all the
graphics cards in the system.  It does this before it does any poking
around.

 > But I still think it may be useful to generalize the idea to 
 > all kind of legacy IO & PIOs. I definitely agree that VGA is a kind
 > of special case, mostly because of the necessary exclusion on
 > the VGA IO response.

I don't think this is the way to go.

First, what would you use it for?  Let's check some examples:

1) "legacy serial on Super I/O"

   This is best handled by a asm/serial.h mechanism, something akin
   to how asm/floppy.h works.  Let the platform say how to get at
   the ports.

   Wait, there already is a asm/serial.h that does just this :-)

2) "floppy"

   See asm/floppy.h :-)

3) "PS/2 keyboard and mouse"

   I believe there is an asm/keyboard.h for the keyboard side of this.
   Yes, and it has macros for the register accesses.

4) "legacy keyboard beeper"

   Make an asm/kbdbeep.h or something.

Add whatever else you might be interested that things tend to
inb/outb.

And if your concern is having multiple of these in your system, the
only ones that make sense are floppy and serial and those are handled
just fine by the asm/serial.h mechanism.

This way of doing this allows 16550's, floppies, etc. to be handled on
any bus whatsoever.

I mean, besides this and VGA what is left and even matters?

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Benjamin Herrenschmidt]

>Add whatever else you might be interested that things tend to
>inb/outb.
>
>And if your concern is having multiple of these in your system, the
>only ones that make sense are floppy and serial and those are handled
>just fine by the asm/serial.h mechanism.
>
>This way of doing this allows 16550's, floppies, etc. to be handled on
>any bus whatsoever.
>
>I mean, besides this and VGA what is left and even matters?

Ok, I capitulate ;)

So basically, all is needed is to enforce those drivers to use
ioremap before doing their IOs.

I still think there's a potential difficulty with having the same
ioremap function for both MMIO and PIO as the address space may overlap.

For once, the x86 enters the dance as it has really separate bus spaces for
them. Other archs can work around this by using the physical address
where the PIO is mapped in the IO resources.

Ben.

VGA handling was [Re: Going beyond 256 PCI buses]

[James Simmons]

> The secondary VGA devices are only interesting to things like the X
> server, and xfree86 does all the enable/disable/bridge-forward-vga
> magic when doing multi-head.

   Actually this weekend I'm working on this for the console system. I
plan to get my TNT card and 3Dfx card both working at the same time both in
VGA text mode. If you also really want to get multiple card in vga text
mode it is going to get far more complex in a multihead besides the
problem of multiple buses.

   First the approach I'm taking is be able to pass to vgacon different
register and memory regions as well as different layouts for vga text
mode. Different cards have different memory layouts for vga text mode. I
discovered this the hard way :-( This way I can get these VGA regions from
the pci bus and pass this to vgacon. This approach will solve alot of
problems. We still havethe problem of multibuses which I haven't yet
figured out. 

Re: VGA handling was [Re: Going beyond 256 PCI buses]

[David S. Miller]

James Simmons writes:
 >    Actually this weekend I'm working on this for the console system. I
 > plan to get my TNT card and 3Dfx card both working at the same time both in
 > VGA text mode. If you also really want to get multiple card in vga text
 > mode it is going to get far more complex in a multihead besides the
 > problem of multiple buses.

You going to have to enable/disable I/O, MEM access, and VGA pallette
snooping in the PCI_COMMAND register of both boards every time you go
from rendering text on one to rendering text on the other.  If there
are bridges leading to either device, you may need to fiddle with VGA
forwarding during each switch as well.

You'll also need a semaphore or similar to control this "active VGA"
state.

Really, I don't think this is all that good of an idea.

Later,
David S. Miller
davem@redhat.com

Re: VGA handling was [Re: Going beyond 256 PCI buses]

[James Simmons]

> You going to have to enable/disable I/O, MEM access, and VGA pallette
> snooping in the PCI_COMMAND register of both boards every time you go
> from rendering text on one to rendering text on the other.  If there
> are bridges leading to either device, you may need to fiddle with VGA
> forwarding during each switch as well.
> 
> You'll also need a semaphore or similar to control this "active VGA"
> state.
> 
> Really, I don't think this is all that good of an idea.

Yes I know. Also each card needs it own special functions to handle
programming the CRTC, SEQ registers etc. Perhaps for real multihead
support I guess the user will have to use fbdev. vgacon can just exist for
single head systems. I guess it is time to let vga go. It is old technology. 

Re: VGA handling was [Re: Going beyond 256 PCI buses]

[Pavel Machek]

Hi!

> Yes I know. Also each card needs it own special functions to handle
> programming the CRTC, SEQ registers etc. Perhaps for real multihead
> support I guess the user will have to use fbdev. vgacon can just exist for
> single head systems. I guess it is time to let vga go. It is old technology. 

It is still faster than fbdev, and not all cards have fbdev drivers. This
should work with any vga card, right?
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: VGA handling was [Re: Going beyond 256 PCI buses]

[Albert D. Cahalan]

David S. Miller writes:

> You going to have to enable/disable I/O, MEM access, and VGA pallette
> snooping in the PCI_COMMAND register of both boards every time you go
> from rendering text on one to rendering text on the other.  If there
> are bridges leading to either device, you may need to fiddle with VGA
> forwarding during each switch as well.
> 
> You'll also need a semaphore or similar to control this "active VGA"
> state.
> 
> Really, I don't think this is all that good of an idea.

It might not be so bad if you assume that one doesn't blast away
with interleaved operations between the displays. Lazy switching
means you can do a whole user-interface action without needing
to muck with bridges. By "user-interface action" I mean something
like a "make menuconfig" screen refresh.

Re: VGA handling was [Re: Going beyond 256 PCI buses]

[Matan Ziv-Av]

On Thu, 14 Jun 2001, David S. Miller wrote:

> James Simmons writes:
>  >  Actually this weekend I'm working on this for the console system. I
>  > plan to get my TNT card and 3Dfx card both working at the same time both in
>  > VGA text mode. If you also really want to get multiple card in vga text
>  > mode it is going to get far more complex in a multihead besides the
>  > problem of multiple buses.
> 
> You going to have to enable/disable I/O, MEM access, and VGA pallette
> snooping in the PCI_COMMAND register of both boards every time you go
> from rendering text on one to rendering text on the other.If there
> are bridges leading to either device, you may need to fiddle with VGA
> forwarding during each switch as well.
> 
> You'll also need a semaphore or similar to control this "active VGA"
> state.

Why do that? You ignore the vga regions of all the cards except for the
primary, and program all other cards by accessing their PCI mapped
regions, which are programmed not to overlap, so they are completely
independent.
This is what nvvgacon does for using text mode of secondary nvidia card. 

-- 
Matan Ziv-Av.                         matan@svgalib.org

Re: Going beyond 256 PCI buses

[David S. Miller]

Benjamin Herrenschmidt writes:
 > I beleive there will always be need for some platform specific
 > hacking at probe-time to handle those, but we can at least make
 > the inx/outx functions/macros compatible with such a scheme,
 > possibly by requesting an ioremap equivalent to be done so that
 > we stop passing them real PIO addresses, but a cookie obtained
 > in various platform specific ways.

The cookie can be encoded into the address itself.

This is why readl() etc. take one arg, the address, not a billion
other arguments like some systems do.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Benjamin Herrenschmidt]

> > I beleive there will always be need for some platform specific
> > hacking at probe-time to handle those, but we can at least make
> > the inx/outx functions/macros compatible with such a scheme,
> > possibly by requesting an ioremap equivalent to be done so that
> > we stop passing them real PIO addresses, but a cookie obtained
> > in various platform specific ways.
>
>The cookie can be encoded into the address itself.
>
>This is why readl() etc. take one arg, the address, not a billion
>other arguments like some systems do.

Right, I don't want an additional parameter. Just a clear definition
that, like read/writex(), the in/outx() functions "address" parameter
is not a magic IO address, but a cookie obtained from an ioremap-like
function.

Now, the parameter passed to that ioremap-like function are a different
story. I beleive it could be something like

isa_pioremap(domain, address, size)  for ISA-like PIO
isa_ioremap(domain, address, size)   for ISA-like mem

domain can be 0 for "default" case (or maybe -1 as 0 is a valid
domain number and we may want the default domain to be another one)

For PCI PIOs, we need a pioremap(pci_dev, address, size) counterpart
to ioremap, as mapping of IO space is usually different on each domain.

A nice side-effect of enforcing those rules is that we no-longer need
to have the entire IO space of all domain beeing mapped in kernel virtual
space all the time as it's the case now (at least on PPC), thus saving
kernel virtual address space.

Now, we can argue on the "pioremap" name itself ;)

A typical use is a fbdev driver for a VGA-compatible PCI card. Some of
these still require some "legacy" access before beeing fully useable with
PCI MMIO only. Such driver can use isa_pioremap & isa_ioremap with the
domain number extracted from the pci_dev of the card in order to generate
those necessary cycles.

Ben.

Re: Going beyond 256 PCI buses

[Jeff Garzik]

"David S. Miller" wrote:
> 
> Jeff Garzik writes:
>  > ok with me.  would bus #0 be the system or root bus?  that would be my
>  > preference, in a tiered system like this.
> 
> Bus 0 is controller 0, of whatever bus type that happens to be.
> If we want to do something special we could create something
> like /proc/bus/root or whatever, but I feel this unnecessary.

Basically I would prefer some sort of global tree so we can figure out a
sane ordering for PM.  Power down the pcmcia cards before the add-on
card containing a PCI-pcmcia bridge, that sort of thing.  Cross-bus-type
ordering.

-- 
Jeff Garzik      | Andre the Giant has a posse.
Building 1024    |
MandrakeSoft     |

Re: Going beyond 256 PCI buses

[Benjamin Herrenschmidt]

>"David S. Miller" wrote:
>> 
>> Jeff Garzik writes:
>>  > ok with me.  would bus #0 be the system or root bus?  that would be my
>>  > preference, in a tiered system like this.
>> 
>> Bus 0 is controller 0, of whatever bus type that happens to be.
>> If we want to do something special we could create something
>> like /proc/bus/root or whatever, but I feel this unnecessary.
>
>Basically I would prefer some sort of global tree so we can figure out a
>sane ordering for PM.  Power down the pcmcia cards before the add-on
>card containing a PCI-pcmcia bridge, that sort of thing.  Cross-bus-type
>ordering.

Welcome to the PM tree....

What I would have liked, but it looks like our current design is not
going that way, would have been a tree structure for the PM notifiers
from the beginning. And instead of having various kind of callbacks
(like PCI suspend/resume/whatever, others for USB, FW, etc..), we
can just have a PM notifier node for each device and have notifiers
handle calling their childs.

That also allow bus "controllers" (in general) to broadcast specific
messages to their childs for things that don't fit in the D0..D3
scheme.

For PCI, instead of having the PCI layer itself having one node and
call the tree, I'd rather see it having one node per pci_dev, and 
layout them according to the PCI tree by default. I can see (and
already know of) cases where the PM tree is _not_ the PCI tree
(because power/reset lines are wired to various ASICs on a motherboard),
and having this PM tree structure separate allow the arch to 
influence it if necessary.

It's simple (a notifier node is a lightweight structure, only one
callback function is implemented, only a few messages are usually
needed to be handled in a given node).

Ben.

Re: Going beyond 256 PCI buses

[Geert Uytterhoeven]

On Thu, 14 Jun 2001, David S. Miller wrote:
> Albert D. Cahalan writes:
>  > >>    /proc/bus/PCI/0/0/3/0/config   config space
>  > >
>  > > Which breaks xfree86 instantly.  This fix is unacceptable.
>  > 
>  > Nope. Keep /proc/bus/pci until Linux 3.14 if you like.
>  > The above is /proc/bus/PCI. That's "PCI", not "pci".
>  > We still have /proc/pci after all.
> 
> Oh I see.
> 
> Well, xfree86 and other programs aren't going to look there, so
> something had to be done about the existing /proc/bus/pci/* hierarchy.
> 
> To be honest, xfree86 needs the controller information not for the
> sake of device probing, it needs it to detect resource conflicts.

Well, those resource conflicts shouldn't be there in the first place. They
should be handled by the OS.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: Going beyond 256 PCI buses

[David S. Miller]

Geert Uytterhoeven writes:
 > Well, those resource conflicts shouldn't be there in the first place. They
 > should be handled by the OS.

I agree, completely.

But this doesn't solve the issue of reserving resources inside the
X server.  Ie. making sure one driver doesn't take the register area
another driver is using.

You still need to get the controller number into the resources to do
that properly.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[David S. Miller]

Jonathan Lundell writes:
 > As I recall, even a midline chipset such as the ServerWorks LE 
 > supports the use of two north bridges, which implies two PCI bus 
 > domains.

It hides this fact by making config space accesses respond in such a
way that it appears that it is all behind one PCI controller.  The
BIOS even avoids allowing any of the MEM and I/O resources from
overlapping.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[David S. Miller]

Jonathan Lundell writes:
 > It's easier in architectures other than IA32, in a way, since they
 > typically don't have the 64KB IO-space addressing limitation that
 > makes heavily bridged systems problematical on IA32 (one tends to
 > run out of IO space).

Right, I was even going to mention this.

But nothing stops an ia32 PCI controller vendor from doing what the
Sun PCI controller does, which is to make I/O space memory mapped.
Well, one thing stops them, no OS would support this from the get
go.

Luckily, it would be quite easy to make Linux handle this kind of
thing on x86 since other platforms built up the infrastructure
needed to make it work.

Later,
David S. Miller
davem@redhat.com

Re: Going beyond 256 PCI buses

[Jonathan Lundell]

At 10:14 AM -0400 2001-06-14, Jeff Garzik wrote:
>According to the PCI spec it is -impossible- to have more than 256 buses
>on a single "hose", so you simply have to implement multiple hoses, just
>like Alpha (and Sparc64?) already do.  That's how the hardware is forced
>to implement it...

That's right, of course. A small problem is that dev->slot_name 
becomes ambiguous, since it doesn't have any hose identification. Nor 
does it have any room for the hose id; it's fixed at 8 chars, and 
fully used (bb:dd.f\0).
-- 
/Jonathan Lundell.

Re: Going beyond 256 PCI buses

[Jeff Garzik]

Jonathan Lundell wrote:
> 
> At 10:14 AM -0400 2001-06-14, Jeff Garzik wrote:
> >According to the PCI spec it is -impossible- to have more than 256 buses
> >on a single "hose", so you simply have to implement multiple hoses, just
> >like Alpha (and Sparc64?) already do.  That's how the hardware is forced
> >to implement it...
> 
> That's right, of course. A small problem is that dev->slot_name
> becomes ambiguous, since it doesn't have any hose identification. Nor
> does it have any room for the hose id; it's fixed at 8 chars, and
> fully used (bb:dd.f\0).

Ouch.  Good point.  Well, extending that field's size shouldn't break
anything except binary modules (which IMHO means, it doesn't break
anything).

	Jeff


-- 
Jeff Garzik      | Andre the Giant has a posse.
Building 1024    |
MandrakeSoft     |