* Does my understanding correct?
@ 2012-04-27 9:27 Richard Yang
2012-04-27 14:10 ` Jiang Liu
2012-04-27 14:17 ` Bjorn Helgaas
0 siblings, 2 replies; 16+ messages in thread
From: Richard Yang @ 2012-04-27 9:27 UTC (permalink / raw)
To: linux-pci; +Cc: bhelgaas
All,
I draw a picture about the physical layout on a part of the pci system.
Each block represent a bridge or a device.
Am I correct?
+-----------------------------+
| |
| |
| Bus#2 |
| ------------------ |
| |
| |
| |
| +-------------+
| |DownStream |02:00.0 pci_dev
+---------------+-------+-----+
| Bus#3
---------------+------------------------------------------+---
| |
| |
+---------------+-------+-----+ +---------------+----+--+-----+
| |UpStream |03:00.0 | |UpStream |03:01.0
| +-------------+ | +-------------+ a normal device
| | | |
| | | |
| Bus#4 | | |
| ------------------- | | |
| +----------+ | |
| |DownStream| | |
+-----------------------+-----+ +-----------------------------+
|
| Bus#5
------------+------------
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-04-27 9:27 Does my understanding correct? Richard Yang
@ 2012-04-27 14:10 ` Jiang Liu
2012-04-27 15:55 ` Bjorn Helgaas
2012-04-27 14:17 ` Bjorn Helgaas
1 sibling, 1 reply; 16+ messages in thread
From: Jiang Liu @ 2012-04-27 14:10 UTC (permalink / raw)
To: Richard Yang; +Cc: linux-pci, bhelgaas
For PCI, there's no internal bus Bus#4 in the left bottom device.
On 04/27/2012 05:27 PM, Richard Yang wrote:
> All,
>
> I draw a picture about the physical layout on a part of the pci system.
>
> Each block represent a bridge or a device.
> Am I correct?
>
> +-----------------------------+
> | |
> | |
> | Bus#2 |
> | ------------------ |
> | |
> | |
> | |
> | +-------------+
> | |DownStream |02:00.0 pci_dev
> +---------------+-------+-----+
> | Bus#3
> ---------------+------------------------------------------+---
> | |
> | |
> +---------------+-------+-----+ +---------------+----+--+-----+
> | |UpStream |03:00.0 | |UpStream |03:01.0
> | +-------------+ | +-------------+ a normal device
> | | | |
> | | | |
> | Bus#4 | | |
> | ------------------- | | |
> | +----------+ | |
> | |DownStream| | |
> +-----------------------+-----+ +-----------------------------+
> |
> | Bus#5
> ------------+------------
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-04-27 9:27 Does my understanding correct? Richard Yang
2012-04-27 14:10 ` Jiang Liu
@ 2012-04-27 14:17 ` Bjorn Helgaas
2012-04-28 5:01 ` Richard Yang
2012-04-28 8:21 ` Richard Yang
1 sibling, 2 replies; 16+ messages in thread
From: Bjorn Helgaas @ 2012-04-27 14:17 UTC (permalink / raw)
To: Richard Yang; +Cc: linux-pci
T24gRnJpLCBBcHIgMjcsIDIwMTIgYXQgMzoyNyBBTSwgUmljaGFyZCBZYW5nCjx3ZWl5YW5nQGxp
bnV4LnZuZXQuaWJtLmNvbT4gd3JvdGU6Cj4gQWxsLAo+Cj4gSSBkcmF3IGEgcGljdHVyZSBhYm91
dCB0aGUgcGh5c2ljYWwgbGF5b3V0IG9uIGEgcGFydCBvZiB0aGUgcGNpIHN5c3RlbS4KPgo+IEVh
Y2ggYmxvY2sgcmVwcmVzZW50IGEgYnJpZGdlIG9yIGEgZGV2aWNlLgo+IEFtIEkgY29ycmVjdD8K
Pgo+IKAgoCstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsKPiCgIKB8IKAgoCCgIKAgoCCg
IKAgoCCgIKAgoCCgIKAgoCB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+
IKAgoHwgoCCgIKBCdXMjMiCgIKAgoCCgIKAgoCCgIKAgoHwKPiCgIKB8IKAgLS0tLS0tLS0tLS0t
LS0tLS0tIKAgoCCgIKB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+IKAg
oHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPiCgIKB8IKAgoCCgIKAgoCCgIKAgoCCg
IKAgoCCgIKAgoCB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgICstLS0tLS0tLS0tLS0tKwo+IKAgoHwg
oCCgIKAgoCCgIKAgoCB8RG93blN0cmVhbSCgIHwwMjowMC4wIHBjaV9kZXYKPiCgIKArLS0tLS0t
LS0tLS0tLS0tKy0tLS0tLS0rLS0tLS0rCj4goCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgfCCg
IEJ1cyMzCj4goCCgIKAgoCCgIKAgLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0KPiCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKB8
IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgfAo+IKAgoCCgIKAgoCCg
IKAgoCCgIKAgoCCgIKAgoHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCg
IKB8Cj4goCCgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tKy0tLS0tKyCgIKAgoCCgIKAgoCCgICst
LS0tLS0tLS0tLS0tLS0rLS0tLSstLSstLS0tLSsKPiCgIKB8IKAgoCCgIKAgoCCgIKAgfFVwU3Ry
ZWFtIKAgoCB8MDM6MDAuMCCgIKAgoCCgfCCgIKAgoCCgIKAgoCCgIHxVcFN0cmVhbSCgIKAgfDAz
OjAxLjAKPiCgIKB8IKAgoCCgIKAgoCCgIKAgKy0tLS0tLS0tLS0tLS0rIKAgoCCgIKAgoCCgIKAg
fCCgIKAgoCCgIKAgoCCgICstLS0tLS0tLS0tLS0tKyBhIG5vcm1hbCBkZXZpY2UKPiCgIKB8IKAg
oCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCg
IKAgoCCgIKAgoCCgIKAgfAo+IKAgoHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwgoCCg
IKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8Cj4goCCgfCCgIKAgoCCg
IKAgoCBCdXMjNCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAg
oCCgIKAgoCCgIHwKPiCgIKB8IKAgoCAtLS0tLS0tLS0tLS0tLS0tLS0tIKAgoCB8IKAgoCCgIKAg
oCCgIKAgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+IKAgoHwgoCCgIKAgoCCgIKAg
oCCgIKArLS0tLS0tLS0tLSsgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCg
IKAgoCB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgIKAgoHxEb3duU3RyZWFtfCCgIKAgoCCgIKAgoCCg
IHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPiCgIKArLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0rLS0tLS0rIKAgoCCgIKAgoCCgIKAgKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
Kwo+IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoHwKPiCgIKAgoCCgIKAgoCCgIKAgoCCgIKAg
oCCgIKB8IEJ1cyM1Cj4goCCgIKAgoCCgIKAgoCCgLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLQoK
SSBhc3N1bWUgeW91ciBxdWVzdGlvbiByZWxhdGVzIHRvIHRoZSBTdHJhdHVzIGZ0U2VydmVyIHRv
cG9sb2d5LiAgSWYKc28sIHRoZSBsc3BjaSBkZXRhaWxzIG1pZ2h0IGNsYXJpZnkgdGhpbmdzLgoK
SW4gdGhhdCBzeXN0ZW0sIG15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdCAwMzowMS4wIGlzIGEgZG93
bnN0cmVhbSBwb3J0LApub3QgYW4gdXBzdHJlYW0gcG9ydC4KCkkgdGhpbmsgeW91ciBwaWN0dXJl
IGlzIHNsaWdodGx5IG1pc2xlYWRpbmcgYmVjYXVzZSBQQ0llIGxpbmtzIGFyZSBub3QKYnVzZXM7
IHRoZXkncmUgcG9pbnQtdG8tcG9pbnQgbGlua3MgYmV0d2VlbiB0d28gZGV2aWNlcy4gIFlvdSd2
ZSBkcmF3bgojMyBhbmQgIzUgYXMgYnVzZXMgdGhhdCBjYW4gaGF2ZSBzZXZlcmFsIGRldmljZXMg
b24gdGhlbSwgd2hpY2ggaXMgbm90CnJlYWxseSB0aGUgY2FzZS4gIFRoZSBsaW5rIGZyb20gYSBk
b3duc3RyZWFtIHBvcnQgc2hvdWxkIGxlYWQgdG8KZXhhY3RseSBvbmUgZGV2aWNlLgoKVGhhdCdz
IG9uZSB0aGluZyB0aGF0J3Mgc3RyYW5nZSBpbiB0aGUgZnRTZXJ2ZXIgdG9wb2xvZ3k6IGFwcGFy
ZW50bHkKdGhlcmUgYXJlICp0d28qIGRldmljZXMgb24gYnVzIDAzOiB0aGUgMDM6MDAuMCB1cHN0
cmVhbSBwb3J0IGFuZCB0aGUKMDM6MDEuMCBkb3duc3RyZWFtIHBvcnQuICBJIHRoaW5rIDAzOjAw
LjAgaXMgdGhlIHVwc3RyZWFtIHBvcnQgb2YgYQpQQ0llIHN3aXRjaCwgd2hpY2ggaXMgcGVyZmVj
dGx5IG5vcm1hbC4gIE15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdAowMzowMS4wIGlzIGFub3RoZXIg
KmRvd25zdHJlYW0qIHBvcnQgdGhhdCBsZWFkcyB0byBzZXZlcmFsIG1vcmUKZGV2aWNlcyAoVVNC
LCBOSUMsIGV0YykuCgpCam9ybgo=
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-04-27 14:10 ` Jiang Liu
@ 2012-04-27 15:55 ` Bjorn Helgaas
0 siblings, 0 replies; 16+ messages in thread
From: Bjorn Helgaas @ 2012-04-27 15:55 UTC (permalink / raw)
To: Jiang Liu; +Cc: Richard Yang, linux-pci
T24gRnJpLCBBcHIgMjcsIDIwMTIgYXQgODoxMCBBTSwgSmlhbmcgTGl1IDxsaXVqOTdAZ21haWwu
Y29tPiB3cm90ZToKPiBGb3IgUENJLCB0aGVyZSdzIG5vIGludGVybmFsIGJ1cyBCdXMjNCBpbiB0
aGUgbGVmdCBib3R0b20gZGV2aWNlLgoKWWVzLCB0aGVyZSBpcy4gIFRoZSBsZWZ0IGJvdHRvbSBi
b3ggaXMgYSBQQ0llIHN3aXRjaCB3aXRoIHVwc3RyZWFtCnBvcnQgMDM6MDAuMC4gIFRoZSB1cHN0
cmVhbSBwb3J0J3Mgc2Vjb25kYXJ5IGJ1cyBpcyAwNCwgd2hpY2ggaXMgdGhlCnN3aXRjaCdzIGlu
dGVybmFsIGJ1cy4gIFRoZSBzd2l0Y2gncyBkb3duc3RyZWFtIHBvcnRzIHdvdWxkIGJlLCBmb3IK
ZXhhbXBsZSwgMDQ6MDAuMCwgMDQ6MDEuMCwgMDQ6MDIuMCwgZXRjIChvdGhlciBudW1iZXJpbmdz
IGFyZQpwb3NzaWJsZSkuICBJZiB0aGUgYm94IGxhYmVsbGVkICJEb3duc3RyZWFtIiBpcyAwNDow
MC4wLCBpdHMgc2Vjb25kYXJ5CmJ1cyB3b3VsZCBiZSAwNS4KCkFzIEkgbWVudGlvbmVkIGJlZm9y
ZSwgSSB0aGluayB0aGUgYnVzIDA1IGxpbmUgaXMgbWlzbGVhZGluZywgYmVjYXVzZQppdCdzIGRy
YXduIGFzIGEgYnVzIHdoZXJlIG11bHRpcGxlIGRldmljZXMgY291bGQgYmUgYXR0YWNoZWQuICBJ
dApzaG91bGQgaW5zdGVhZCBiZSBhIHBvaW50LXRvLXBvaW50IGxpbmsgdG8gYSBzaW5nbGUgZGV2
aWNlLiAgKEV4Y2VwdAppbiB0aGlzIFN0cmF0dXMgc3lzdGVtLCB3aGljaCBzZWVtcyBicm9rZW4g
YmVjYXVzZSBpdCAqZG9lcyogaGF2ZSB0d28KZGV2aWNlcyBhdHRhY2hlZCB0byB0aGUgc2FtZSBk
b3duc3RyZWFtIHBvcnQuKQoKPiBPbiAwNC8yNy8yMDEyIDA1OjI3IFBNLCBSaWNoYXJkIFlhbmcg
d3JvdGU6Cj4+IEFsbCwKPj4KPj4gSSBkcmF3IGEgcGljdHVyZSBhYm91dCB0aGUgcGh5c2ljYWwg
bGF5b3V0IG9uIGEgcGFydCBvZiB0aGUgcGNpIHN5c3RlbS4KPj4KPj4gRWFjaCBibG9jayByZXBy
ZXNlbnQgYSBicmlkZ2Ugb3IgYSBkZXZpY2UuCj4+IEFtIEkgY29ycmVjdD8KPj4KPj4goCCgICst
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAg
oCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4goCCg
IHwgoCCgIKBCdXMjMiCgIKAgoCCgIKAgoCCgIKAgoHwKPj4goCCgIHwgoCAtLS0tLS0tLS0tLS0t
LS0tLS0goCCgIKAgoHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4g
oCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAg
oCCgIKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCArLS0tLS0tLS0tLS0tLSsK
Pj4goCCgIHwgoCCgIKAgoCCgIKAgoCB8RG93blN0cmVhbSCgIHwwMjowMC4wIHBjaV9kZXYKPj4g
oCCgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLSstLS0tLSsKPj4goCCgIKAgoCCgIKAgoCCgIKAg
oCCgIKAgoCCgIHwgoCBCdXMjMwo+PiCgIKAgoCCgIKAgoCCgLS0tLS0tLS0tLS0tLS0tKy0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0KPj4goCCgIKAgoCCgIKAg
oCCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCg
IKB8Cj4+IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAg
oCCgIKAgoCCgIKAgoCCgIKAgoCCgfAo+PiCgIKAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tKy0t
LS0tKyCgIKAgoCCgIKAgoCCgICstLS0tLS0tLS0tLS0tLS0rLS0tLSstLSstLS0tLSsKPj4goCCg
IHwgoCCgIKAgoCCgIKAgoCB8VXBTdHJlYW0goCCgIHwwMzowMC4wIKAgoCCgIKB8IKAgoCCgIKAg
oCCgIKAgfFVwU3RyZWFtIKAgoCB8MDM6MDEuMAo+PiCgIKAgfCCgIKAgoCCgIKAgoCCgICstLS0t
LS0tLS0tLS0tKyCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCArLS0tLS0tLS0tLS0tLSsg
YSBub3JtYWwgZGV2aWNlCj4+IKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8IKAg
oCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+PiCgIKAgfCCgIKAg
oCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCg
IKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgQnVzIzQgoCCgIKAgoCCgIHwgoCCg
IKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8Cj4+IKAgoCB8IKAgoCAt
LS0tLS0tLS0tLS0tLS0tLS0tIKAgoCB8IKAgoCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIKAg
oCCgIKAgoCCgIKAgfAo+PiCgIKAgfCCgIKAgoCCgIKAgoCCgIKAgoCstLS0tLS0tLS0tKyCgIKAg
oCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAg
oCCgIKAgoCCgIKB8RG93blN0cmVhbXwgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCg
IKAgoCCgIKAgoCB8Cj4+IKAgoCArLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0rIKAgoCCg
IKAgoCCgIKAgKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKwo+PiCgIKAgoCCgIKAgoCCg
IKAgoCCgIKAgoCCgIKAgfAo+PiCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfCBCdXMjNQo+
PiCgIKAgoCCgIKAgoCCgIKAgLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLQo+Pgo+Pgo+Cg==
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-04-27 14:17 ` Bjorn Helgaas
@ 2012-04-28 5:01 ` Richard Yang
2012-04-28 7:21 ` Richard Yang
2012-04-30 15:56 ` Bjorn Helgaas
2012-04-28 8:21 ` Richard Yang
1 sibling, 2 replies; 16+ messages in thread
From: Richard Yang @ 2012-04-28 5:01 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci
On Fri, Apr 27, 2012 at 08:17:48AM -0600, Bjorn Helgaas wrote:
>On Fri, Apr 27, 2012 at 3:27 AM, Richard Yang
><weiyang@linux.vnet.ibm.com> wrote:
>
>I assume your question relates to the Stratus ftServer topology. If
>so, the lspci details might clarify things.
>
Yes, my picture is a little bit related to your previous mail.
While my intention is to find out how the physical world is represented
in the kernel.
Below is a typical topology in PCIe spec r3.0.
+------------------+
| |
| RC |
| Bus#0 |
| ------------- |
| |
+-+-----+--------+-+
00:0.0 | | | 00:02.0
+---------+---------+ | | | +------------+-------------+
| +------+ | +-------| PCIe 2 PCI Bridge |
| PCIe Endpoint | | | |
+-------------------+ | | Bus#2 |
| | -------------- |
| +-------+---------------+--+
| | |
| 00:01.0 |02:00.0 |02:01.0
+------------+-------------+ +-------+------+ +---+-------+
| | |PCI dev | |PCI dev |
| Switch | | | | |
| Bus#1 | | | | |
| --------------- | +--------------+ +-----------+
| |
+------------------------+-+
| |
| |
| 01:00.0 | 01:01.0
+---------+-------+ +--------+----------------+
| | | |
| PCI Endpoint | | PCIe Endpoint |
| | | |
| | | |
+-----------------+ +-------------------------+
Do you think the current assignment of bus number and pci_dev is
correct?
>In that system, my understanding is that 03:01.0 is a downstream port,
>not an upstream port.
>
>I think your picture is slightly misleading because PCIe links are not
>buses; they're point-to-point links between two devices. You've drawn
>#3 and #5 as buses that can have several devices on them, which is not
>really the case. The link from a downstream port should lead to
>exactly one device.
>
>That's one thing that's strange in the ftServer topology: apparently
>there are *two* devices on bus 03: the 03:00.0 upstream port and the
>03:01.0 downstream port. I think 03:00.0 is the upstream port of a
>PCIe switch, which is perfectly normal. My understanding is that
>03:01.0 is another *downstream* port that leads to several more
>devices (USB, NIC, etc).
>
>Bjorn
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-04-28 5:01 ` Richard Yang
@ 2012-04-28 7:21 ` Richard Yang
2012-04-30 15:56 ` Bjorn Helgaas
1 sibling, 0 replies; 16+ messages in thread
From: Richard Yang @ 2012-04-28 7:21 UTC (permalink / raw)
To: Richard Yang; +Cc: Bjorn Helgaas, linux-pci
On Sat, Apr 28, 2012 at 01:01:27PM +0800, Richard Yang wrote:
>On Fri, Apr 27, 2012 at 08:17:48AM -0600, Bjorn Helgaas wrote:
>>On Fri, Apr 27, 2012 at 3:27 AM, Richard Yang
>><weiyang@linux.vnet.ibm.com> wrote:
>>
>>I assume your question relates to the Stratus ftServer topology. If
>>so, the lspci details might clarify things.
>>
>Yes, my picture is a little bit related to your previous mail.
>While my intention is to find out how the physical world is represented
>in the kernel.
>
>Below is a typical topology in PCIe spec r3.0.
>
> +------------------+
> | |
> | RC |
> | Bus#0 |
> | ------------- |
> | |
> +-+-----+--------+-+
> 00:0.0 | | | 00:02.0
> +---------+---------+ | | | +------------+-------------+
> | +------+ | +-------| PCIe 2 PCI Bridge |
> | PCIe Endpoint | | | |
> +-------------------+ | | Bus#2 |
> | | -------------- |
> | +-------+---------------+--+
> | | |
> | 00:01.0 |02:00.0 |02:01.0
> +------------+-------------+ +-------+------+ +---+-------+
> | | |PCI dev | |PCI dev |
> | Switch | | | | |
> | Bus#1 | | | | |
> | --------------- | +--------------+ +-----------+
> | |
> +------------------------+-+
> | |
> | |
> | 01:00.0 | 01:01.0
> +---------+-------+ +--------+----------------+
> | | | |
> | PCI Endpoint | | PCIe Endpoint |
> | | | |
> | | | |
> +-----------------+ +-------------------------+
>
>Do you think the current assignment of bus number and pci_dev is
>correct?
>
BTW, if the chart is correct, 01:00.0 reprents Switch DownStream Port
or the PCI Endpoint?
>
>>In that system, my understanding is that 03:01.0 is a downstream port,
>>not an upstream port.
>>
>>I think your picture is slightly misleading because PCIe links are not
>>buses; they're point-to-point links between two devices. You've drawn
>>#3 and #5 as buses that can have several devices on them, which is not
>>really the case. The link from a downstream port should lead to
>>exactly one device.
>>
>>That's one thing that's strange in the ftServer topology: apparently
>>there are *two* devices on bus 03: the 03:00.0 upstream port and the
>>03:01.0 downstream port. I think 03:00.0 is the upstream port of a
>>PCIe switch, which is perfectly normal. My understanding is that
>>03:01.0 is another *downstream* port that leads to several more
>>devices (USB, NIC, etc).
>>
>>Bjorn
>
>--
>Richard Yang
>Help you, Help me
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-04-27 14:17 ` Bjorn Helgaas
2012-04-28 5:01 ` Richard Yang
@ 2012-04-28 8:21 ` Richard Yang
1 sibling, 0 replies; 16+ messages in thread
From: Richard Yang @ 2012-04-28 8:21 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci
On Fri, Apr 27, 2012 at 08:17:48AM -0600, Bjorn Helgaas wrote:
>I assume your question relates to the Stratus ftServer topology. If
>so, the lspci details might clarify things.
>
>In that system, my understanding is that 03:01.0 is a downstream port,
>not an upstream port.
>
>I think your picture is slightly misleading because PCIe links are not
>buses; they're point-to-point links between two devices. You've drawn
>#3 and #5 as buses that can have several devices on them, which is not
>really the case. The link from a downstream port should lead to
>exactly one device.
Hmm... if this is a PCI bridge, then it will have both internal bus and
externel bus? or no internal pci bus, just one external bus?
>
>That's one thing that's strange in the ftServer topology: apparently
>there are *two* devices on bus 03: the 03:00.0 upstream port and the
>03:01.0 downstream port. I think 03:00.0 is the upstream port of a
>PCIe switch, which is perfectly normal. My understanding is that
>03:01.0 is another *downstream* port that leads to several more
>devices (USB, NIC, etc).
>
>Bjorn
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-04-28 5:01 ` Richard Yang
2012-04-28 7:21 ` Richard Yang
@ 2012-04-30 15:56 ` Bjorn Helgaas
2012-05-02 6:24 ` Richard Yang
1 sibling, 1 reply; 16+ messages in thread
From: Bjorn Helgaas @ 2012-04-30 15:56 UTC (permalink / raw)
To: Richard Yang; +Cc: linux-pci
On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
<weiyang@linux.vnet.ibm.com> wrote:
> On Fri, Apr 27, 2012 at 08:17:48AM -0600, Bjorn Helgaas wrote:
>>On Fri, Apr 27, 2012 at 3:27 AM, Richard Yang
>><weiyang@linux.vnet.ibm.com> wrote:
>>
>>I assume your question relates to the Stratus ftServer topology. If
>>so, the lspci details might clarify things.
>>
> Yes, my picture is a little bit related to your previous mail.
> While my intention is to find out how the physical world is represented
> in the kernel.
>
> Below is a typical topology in PCIe spec r3.0.
>
> +------------------+
> | |
> | RC |
> | Bus#0 |
> | ------------- |
> | |
> +-+-----+--------+-+
> 00:0.0 | | | 00:02.0
> +---------+---------+ | | | +------------+-------------+
> | +------+ | +-------| PCIe 2 PCI Bridge |
> | PCIe Endpoint | | | |
> +-------------------+ | | Bus#2 |
> | | -------------- |
> | +-------+---------------+--+
> | | |
> | 00:01.0 |02:00.0 |02:01.0
> +------------+-------------+ +-------+------+ +---+-------+
> | | |PCI dev | |PCI dev |
> | Switch | | | | |
> | Bus#1 | | | | |
> | --------------- | +--------------+ +-----------+
> | |
> +------------------------+-+
> | |
> | |
> | 01:00.0 | 01:01.0
> +---------+-------+ +--------+----------------+
> | | | |
> | PCI Endpoint | | PCIe Endpoint |
> | | | |
> | | | |
> +-----------------+ +-------------------------+
>
> Do you think the current assignment of bus number and pci_dev is
> correct?
I think assignments shown for the PCIe-to-PCI bridge are OK, although
I would draw it like this because the bridge originates a single bus
02 that may have multiple devices attached to it (this side is PCI,
not PCIe, so it really is a shared bus):
^
|
+--------+--------+
| 00:02.0 |
| PCIe-PCI bridge |
| |
+--------+--------+
|
|
+---------------------+ Bus 02
| |
| |
| |
+----v----+ +----v----+
| 02:00.0 | | 02:01.0 |
+---------+ +---------+
I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec
1.3.3 of the PCIe r3 spec:
^
|
+-------------------------------|------------------------------+
| | |
| +----+----+ |
| | virtual | |
| | PCI-PCI | |
| | bridge | |
| +----+----+ |
| | |
| | |
| | |
| +----------------------------------------+ |
| | | | |
| | | | |
| | | | |
| +----+----+ +----+----+ +----++---+ |
| | virtual | | virtual | | virtual | |
| | PCI-PCI | | PCI-PCI | | PCI-PCI | |
| | bridge | | bridge | | bridge | |
| +----+----+ +----+----+ +----+----+ |
| | | | |
| | | | |
+----------|--------------------|-------------------|----------+
| | |
v v v
A PCIe switch appears as two or more PCI-PCI bridges. One is
associated with the upstream port; the others with the downstream
ports.
A bridge always has a primary side and a secondary side. In your
diagram, the bridge associated with the upstream port would be 00:01.0
(primary bus 00) and could have a secondary bus of 03 (since 02 is
already consumed by the PCIe-PCI bridge).
The bridges associated with the downstream ports are all logically on
bus 03. Their primary bus number would be 03; they might be 03:00.0,
03:01.0, 03:02.0, etc. Each would have its own secondary bus number,
for example 04, 05, 06. That secondary bus number is for the
downstream link from the corresponding downstream port.
The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0
(or these could be the upstream ports of more PCIe switches).
Bjorn
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-04-30 15:56 ` Bjorn Helgaas
@ 2012-05-02 6:24 ` Richard Yang
2012-05-02 14:59 ` Bjorn Helgaas
0 siblings, 1 reply; 16+ messages in thread
From: Richard Yang @ 2012-05-02 6:24 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci
On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote:
>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
><weiyang@linux.vnet.ibm.com> wrote:
>
Thanks for your nice chart.
>I think assignments shown for the PCIe-to-PCI bridge are OK, although
>I would draw it like this because the bridge originates a single bus
>02 that may have multiple devices attached to it (this side is PCI,
>not PCIe, so it really is a shared bus):
>
> ^
> |
> +--------+--------+
> | 00:02.0 |
> | PCIe-PCI bridge |
> | |
> +--------+--------+
> |
> |
> +---------------------+ Bus 02
> | |
> | |
> | |
> +----v----+ +----v----+
> | 02:00.0 | | 02:01.0 |
> +---------+ +---------+
>
So for this case, there is not internal bus, while this is really a
physical shared bus, not a logical one.
>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec
>1.3.3 of the PCIe r3 spec:
^
|
+-------------------------------|------------------------------+
| | |
| +----+----+ |
| | virtual | |
| | PCI-PCI | |
| | bridge | |
| +----+----+ |
| | |
| |Bus#3 |
| | |
| +----------------------------------------+ |
| | | | |
| | | | |
| |03:00.0 |03:01.0 |03:02.0 |
| +----+----+ +----+----+ +----++---+ |
| | virtual | | virtual | | virtual | |
| | PCI-PCI | | PCI-PCI | | PCI-PCI | |
| | bridge | | bridge | | bridge | |
| +----+----+ +----+----+ +----+----+ |
| |Bus#4? | | |
| -----+------- | | |
+----------|--------------------|-------------------|----------+
| | |
v v v
>A PCIe switch appears as two or more PCI-PCI bridges. One is
>associated with the upstream port; the others with the downstream
>ports.
>
>A bridge always has a primary side and a secondary side. In your
>diagram, the bridge associated with the upstream port would be 00:01.0
>(primary bus 00) and could have a secondary bus of 03 (since 02 is
>already consumed by the PCIe-PCI bridge).
Hmm... I am confused why is 03. 02 is used but 01 is not used.
Switch should be configured after PCIe2PCI bridge?
>
>The bridges associated with the downstream ports are all logically on
>bus 03. Their primary bus number would be 03; they might be 03:00.0,
>03:01.0, 03:02.0, etc. Each would have its own secondary bus number,
>for example 04, 05, 06. That secondary bus number is for the
>downstream link from the corresponding downstream port.
Hmm, as you mentioned in previous letter, PCIe is an point-to-point
protocol, then the secondary bus should reside in the Switch?
Do you think my Bus#4 is correct?
>
>The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0
>(or these could be the upstream ports of more PCIe switches).
So below the PCIe downstream port, there is only on PCIe device.
The whole bus is occupied by this device?
That is why the device could have upto 256 functions?
>
>Bjorn
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-05-02 6:24 ` Richard Yang
@ 2012-05-02 14:59 ` Bjorn Helgaas
2012-05-02 21:05 ` Don Dutile
2012-05-03 6:21 ` Richard Yang
0 siblings, 2 replies; 16+ messages in thread
From: Bjorn Helgaas @ 2012-05-02 14:59 UTC (permalink / raw)
To: Richard Yang; +Cc: linux-pci
On Wed, May 2, 2012 at 12:24 AM, Richard Yang
<weiyang@linux.vnet.ibm.com> wrote:
> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote:
>>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
>><weiyang@linux.vnet.ibm.com> wrote:
>>
> Thanks for your nice chart.
>>I think assignments shown for the PCIe-to-PCI bridge are OK, although
>>I would draw it like this because the bridge originates a single bus
>>02 that may have multiple devices attached to it (this side is PCI,
>>not PCIe, so it really is a shared bus):
>>
>> ^
>> |
>> +--------+--------+
>> | 00:02.0 |
>> | PCIe-PCI bridge |
>> | |
>> +--------+--------+
>> |
>> |
>> +---------------------+ Bus 02
>> | |
>> | |
>> | |
>> +----v----+ +----v----+
>> | 02:00.0 | | 02:01.0 |
>> +---------+ +---------+
>>
> So for this case, there is not internal bus, while this is really a
> physical shared bus, not a logical one.
Yes. The downstream side of the PCIe-PCI bridge is PCI.
>>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec
>>1.3.3 of the PCIe r3 spec:
>
> ^
> |
> +-------------------------------|------------------------------+
> | | |
> | +----+----+ |
> | | virtual | |
> | | PCI-PCI | |
> | | bridge | |
> | +----+----+ |
> | | |
> | |Bus#3 |
> | | |
> | +----------------------------------------+ |
> | | | | |
> | | | | |
> | |03:00.0 |03:01.0 |03:02.0 |
> | +----+----+ +----+----+ +----++---+ |
> | | virtual | | virtual | | virtual | |
> | | PCI-PCI | | PCI-PCI | | PCI-PCI | |
> | | bridge | | bridge | | bridge | |
> | +----+----+ +----+----+ +----+----+ |
> | |Bus#4? | | |
> | -----+------- | | |
> +----------|--------------------|-------------------|----------+
> | | |
> v v v
>
>
>>A PCIe switch appears as two or more PCI-PCI bridges. One is
>>associated with the upstream port; the others with the downstream
>>ports.
>>
>>A bridge always has a primary side and a secondary side. In your
>>diagram, the bridge associated with the upstream port would be 00:01.0
>>(primary bus 00) and could have a secondary bus of 03 (since 02 is
>>already consumed by the PCIe-PCI bridge).
> Hmm... I am confused why is 03. 02 is used but 01 is not used.
> Switch should be configured after PCIe2PCI bridge?
It's likely that the PCIe switch would be configured first, since its
device number is lower, but that is not a requirement. The
requirement is that the bus number ranges consumed by bridges be
non-overlapping. In this case (using your original topology plus my
PCIe switch diagram), we'd have:
1. an endpoint at 00:00.0 -- consumes no additional buses
2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
Bus 03 is the internal PCIe switch bus that connects the upstream port
to the downstream ports. Buses 04, 05, and 06 are the links
originating from the downstream ports.
>>The bridges associated with the downstream ports are all logically on
>>bus 03. Their primary bus number would be 03; they might be 03:00.0,
>>03:01.0, 03:02.0, etc. Each would have its own secondary bus number,
>>for example 04, 05, 06. That secondary bus number is for the
>>downstream link from the corresponding downstream port.
> Hmm, as you mentioned in previous letter, PCIe is an point-to-point
> protocol, then the secondary bus should reside in the Switch?
> Do you think my Bus#4 is correct?
No. Bus 04 is a PCIe link that connects the downstream port (03:00.0)
to a single PCIe device. That device (04:00) could be a PCIe
endpoint, or it could be the upstream port of another PCIe switch.
(If it is a switch, more bus numbers would be required.)
>>The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0
>>(or these could be the upstream ports of more PCIe switches).
> So below the PCIe downstream port, there is only on PCIe device.
> The whole bus is occupied by this device?
> That is why the device could have upto 256 functions?
Yes, if it supports ARI.
Bjorn
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-05-02 14:59 ` Bjorn Helgaas
@ 2012-05-02 21:05 ` Don Dutile
2012-05-03 6:21 ` Richard Yang
1 sibling, 0 replies; 16+ messages in thread
From: Don Dutile @ 2012-05-02 21:05 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci
On 05/02/2012 10:59 AM, Bjorn Helgaas wrote:
> On Wed, May 2, 2012 at 12:24 AM, Richard Yang
> <weiyang@linux.vnet.ibm.com> wrote:
>> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote:
>>> On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
>>> <weiyang@linux.vnet.ibm.com> wrote:
>>>
>> Thanks for your nice chart.
>>> I think assignments shown for the PCIe-to-PCI bridge are OK, although
>>> I would draw it like this because the bridge originates a single bus
>>> 02 that may have multiple devices attached to it (this side is PCI,
>>> not PCIe, so it really is a shared bus):
>>>
>>> ^
>>> |
>>> +--------+--------+
>>> | 00:02.0 |
>>> | PCIe-PCI bridge |
>>> | |
>>> +--------+--------+
>>> |
>>> |
>>> +---------------------+ Bus 02
>>> | |
>>> | |
>>> | |
>>> +----v----+ +----v----+
>>> | 02:00.0 | | 02:01.0 |
>>> +---------+ +---------+
>>>
>> So for this case, there is not internal bus, while this is really a
>> physical shared bus, not a logical one.
>
> Yes. The downstream side of the PCIe-PCI bridge is PCI.
>
>>> I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec
>>> 1.3.3 of the PCIe r3 spec:
>>
>> ^
>> |
>> +-------------------------------|------------------------------+
>> | | |
>> | +----+----+ |
>> | | virtual | |
>> | | PCI-PCI | |
>> | | bridge | |
>> | +----+----+ |
>> | | |
>> | |Bus#3 |
>> | | |
>> | +----------------------------------------+ |
>> | | | | |
>> | | | | |
>> | |03:00.0 |03:01.0 |03:02.0 |
>> | +----+----+ +----+----+ +----++---+ |
>> | | virtual | | virtual | | virtual | |
>> | | PCI-PCI | | PCI-PCI | | PCI-PCI | |
>> | | bridge | | bridge | | bridge | |
>> | +----+----+ +----+----+ +----+----+ |
>> | |Bus#4? | | |
>> | -----+------- | | |
>> +----------|--------------------|-------------------|----------+
>> | | |
>> v v v
>>
>>
>>> A PCIe switch appears as two or more PCI-PCI bridges. One is
>>> associated with the upstream port; the others with the downstream
>>> ports.
>>>
>>> A bridge always has a primary side and a secondary side. In your
>>> diagram, the bridge associated with the upstream port would be 00:01.0
>>> (primary bus 00) and could have a secondary bus of 03 (since 02 is
>>> already consumed by the PCIe-PCI bridge).
>> Hmm... I am confused why is 03. 02 is used but 01 is not used.
>> Switch should be configured after PCIe2PCI bridge?
>
> It's likely that the PCIe switch would be configured first, since its
> device number is lower, but that is not a requirement. The
> requirement is that the bus number ranges consumed by bridges be
> non-overlapping. In this case (using your original topology plus my
> PCIe switch diagram), we'd have:
>
> 1. an endpoint at 00:00.0 -- consumes no additional buses
> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
>
> Bus 03 is the internal PCIe switch bus that connects the upstream port
> to the downstream ports. Buses 04, 05, and 06 are the links
> originating from the downstream ports.
>
>>> The bridges associated with the downstream ports are all logically on
>>> bus 03. Their primary bus number would be 03; they might be 03:00.0,
>>> 03:01.0, 03:02.0, etc. Each would have its own secondary bus number,
>>> for example 04, 05, 06. That secondary bus number is for the
>>> downstream link from the corresponding downstream port.
>> Hmm, as you mentioned in previous letter, PCIe is an point-to-point
>> protocol, then the secondary bus should reside in the Switch?
>> Do you think my Bus#4 is correct?
>
> No. Bus 04 is a PCIe link that connects the downstream port (03:00.0)
> to a single PCIe device. That device (04:00) could be a PCIe
> endpoint, or it could be the upstream port of another PCIe switch.
> (If it is a switch, more bus numbers would be required.)
>
>>> The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0
>>> (or these could be the upstream ports of more PCIe switches).
>> So below the PCIe downstream port, there is only on PCIe device.
>> The whole bus is occupied by this device?
>> That is why the device could have upto 256 functions?
>
> Yes, if it supports ARI.
>
minor tweak: if the device *and* the downstream PCIe bridge support ARI...
both must exist. I know of one such module, e.g., non-ARI PCIe switch w/PCIe endpoints)
and the endpoints have SRIOV/VF support... not a very smart hw combo. .. nuf said.
> Bjorn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-05-02 14:59 ` Bjorn Helgaas
2012-05-02 21:05 ` Don Dutile
@ 2012-05-03 6:21 ` Richard Yang
2012-05-03 16:39 ` Bjorn Helgaas
1 sibling, 1 reply; 16+ messages in thread
From: Richard Yang @ 2012-05-03 6:21 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci
On Wed, May 02, 2012 at 08:59:40AM -0600, Bjorn Helgaas wrote:
>On Wed, May 2, 2012 at 12:24 AM, Richard Yang
><weiyang@linux.vnet.ibm.com> wrote:
>> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote:
>>>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
>>><weiyang@linux.vnet.ibm.com> wrote:
>>>
>> Thanks for your nice chart.
>>>I think assignments shown for the PCIe-to-PCI bridge are OK, although
>>>I would draw it like this because the bridge originates a single bus
>>>02 that may have multiple devices attached to it (this side is PCI,
>>>not PCIe, so it really is a shared bus):
>>>
>>> ^
>>> |
>>> +--------+--------+
>>> | 00:02.0 |
>>> | PCIe-PCI bridge |
>>> | |
>>> +--------+--------+
>>> |
>>> |
>>> +---------------------+ Bus 02
>>> | |
>>> | |
>>> | |
>>> +----v----+ +----v----+
>>> | 02:00.0 | | 02:01.0 |
>>> +---------+ +---------+
>>>
>> So for this case, there is not internal bus, while this is really a
>> physical shared bus, not a logical one.
>
>Yes. The downstream side of the PCIe-PCI bridge is PCI.
>
>>>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec
>>>1.3.3 of the PCIe r3 spec:
>>
>>>A PCIe switch appears as two or more PCI-PCI bridges. One is
>>>associated with the upstream port; the others with the downstream
>>>ports.
>>>
>>>A bridge always has a primary side and a secondary side. In your
>>>diagram, the bridge associated with the upstream port would be 00:01.0
>>>(primary bus 00) and could have a secondary bus of 03 (since 02 is
>>>already consumed by the PCIe-PCI bridge).
>> Hmm... I am confused why is 03. 02 is used but 01 is not used.
>> Switch should be configured after PCIe2PCI bridge?
>
>It's likely that the PCIe switch would be configured first, since its
>device number is lower, but that is not a requirement. The
>requirement is that the bus number ranges consumed by bridges be
>non-overlapping. In this case (using your original topology plus my
>PCIe switch diagram), we'd have:
>
> 1. an endpoint at 00:00.0 -- consumes no additional buses
> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
So this is a sequence issue. Below is also a valid configuration.
2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus)
>
>Bus 03 is the internal PCIe switch bus that connects the upstream port
>to the downstream ports. Buses 04, 05, and 06 are the links
>originating from the downstream ports.
>
>>>The bridges associated with the downstream ports are all logically on
>>>bus 03. Their primary bus number would be 03; they might be 03:00.0,
>>>03:01.0, 03:02.0, etc. Each would have its own secondary bus number,
>>>for example 04, 05, 06. That secondary bus number is for the
>>>downstream link from the corresponding downstream port.
>> Hmm, as you mentioned in previous letter, PCIe is an point-to-point
>> protocol, then the secondary bus should reside in the Switch?
>> Do you think my Bus#4 is correct?
>
>No. Bus 04 is a PCIe link that connects the downstream port (03:00.0)
>to a single PCIe device. That device (04:00) could be a PCIe
>endpoint, or it could be the upstream port of another PCIe switch.
>(If it is a switch, more bus numbers would be required.)
^
|
+-------------------------------|------------------------------+
| | |
| +----+----+ |
| | virtual | |
| | PCI-PCI | |
| | bridge | |
| +----+----+ |
| | |
| |Bus#3 |
| | |
| +----------------------------------------+ |
| | | | |
| | | | |
| |03:00.0 |03:01.0 |03:02.0 |
| +----+----+ +----+----+ +----++---+ |
| | virtual | | virtual | | virtual | |
| | PCI-PCI | | PCI-PCI | | PCI-PCI | |
| | bridge | | bridge | | bridge | |
| +----+----+ +----+----+ +----+----+ |
| | | | |
| | | | |
+----------|--------------------|-------------------|----------+
| Bus#4? | |
v v v
So the link itself is Bus#4?
If there is no device under this link, will Bus#4 appear in kernel?
And then the PCIe switch is represented by 4 pci_dev structure and each
is bridge type?
>
>>>The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0
>>>(or these could be the upstream ports of more PCIe switches).
>> So below the PCIe downstream port, there is only on PCIe device.
>> The whole bus is occupied by this device?
>> That is why the device could have upto 256 functions?
>
>Yes, if it supports ARI.
>
>Bjorn
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-05-03 6:21 ` Richard Yang
@ 2012-05-03 16:39 ` Bjorn Helgaas
2012-05-04 2:11 ` Richard Yang
` (2 more replies)
0 siblings, 3 replies; 16+ messages in thread
From: Bjorn Helgaas @ 2012-05-03 16:39 UTC (permalink / raw)
To: Richard Yang; +Cc: linux-pci
On Thu, May 3, 2012 at 12:21 AM, Richard Yang
<weiyang@linux.vnet.ibm.com> wrote:
> On Wed, May 02, 2012 at 08:59:40AM -0600, Bjorn Helgaas wrote:
>>On Wed, May 2, 2012 at 12:24 AM, Richard Yang
>><weiyang@linux.vnet.ibm.com> wrote:
>>> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote:
>>>>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
>>>><weiyang@linux.vnet.ibm.com> wrote:
>>>>
>>> Thanks for your nice chart.
>>>>I think assignments shown for the PCIe-to-PCI bridge are OK, although
>>>>I would draw it like this because the bridge originates a single bus
>>>>02 that may have multiple devices attached to it (this side is PCI,
>>>>not PCIe, so it really is a shared bus):
>>>>
>>>> ^
>>>> |
>>>> +--------+--------+
>>>> | 00:02.0 |
>>>> | PCIe-PCI bridge |
>>>> | |
>>>> +--------+--------+
>>>> |
>>>> |
>>>> +---------------------+ Bus 02
>>>> | |
>>>> | |
>>>> | |
>>>> +----v----+ +----v----+
>>>> | 02:00.0 | | 02:01.0 |
>>>> +---------+ +---------+
>>>>
>>> So for this case, there is not internal bus, while this is really a
>>> physical shared bus, not a logical one.
>>
>>Yes. The downstream side of the PCIe-PCI bridge is PCI.
>>
>>>>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec
>>>>1.3.3 of the PCIe r3 spec:
>>>
>>>>A PCIe switch appears as two or more PCI-PCI bridges. One is
>>>>associated with the upstream port; the others with the downstream
>>>>ports.
>>>>
>>>>A bridge always has a primary side and a secondary side. In your
>>>>diagram, the bridge associated with the upstream port would be 00:01.0
>>>>(primary bus 00) and could have a secondary bus of 03 (since 02 is
>>>>already consumed by the PCIe-PCI bridge).
>>> Hmm... I am confused why is 03. 02 is used but 01 is not used.
>>> Switch should be configured after PCIe2PCI bridge?
>>
>>It's likely that the PCIe switch would be configured first, since its
>>device number is lower, but that is not a requirement. The
>>requirement is that the bus number ranges consumed by bridges be
>>non-overlapping. In this case (using your original topology plus my
>>PCIe switch diagram), we'd have:
>>
>> 1. an endpoint at 00:00.0 -- consumes no additional buses
>> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
>> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
> So this is a sequence issue. Below is also a valid configuration.
> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05]
> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus)
Yes, this would also be valid.
>>Bus 03 is the internal PCIe switch bus that connects the upstream port
>>to the downstream ports. Buses 04, 05, and 06 are the links
>>originating from the downstream ports.
>>
>>>>The bridges associated with the downstream ports are all logically on
>>>>bus 03. Their primary bus number would be 03; they might be 03:00.0,
>>>>03:01.0, 03:02.0, etc. Each would have its own secondary bus number,
>>>>for example 04, 05, 06. That secondary bus number is for the
>>>>downstream link from the corresponding downstream port.
>>> Hmm, as you mentioned in previous letter, PCIe is an point-to-point
>>> protocol, then the secondary bus should reside in the Switch?
>>> Do you think my Bus#4 is correct?
>>
>>No. Bus 04 is a PCIe link that connects the downstream port (03:00.0)
>>to a single PCIe device. That device (04:00) could be a PCIe
>>endpoint, or it could be the upstream port of another PCIe switch.
>>(If it is a switch, more bus numbers would be required.)
>
>
> ^
> |
> +-------------------------------|------------------------------+
> | | |
> | +----+----+ |
> | | virtual | |
> | | PCI-PCI | |
> | | bridge | |
> | +----+----+ |
> | | |
> | |Bus#3 |
> | | |
> | +----------------------------------------+ |
> | | | | |
> | | | | |
> | |03:00.0 |03:01.0 |03:02.0 |
> | +----+----+ +----+----+ +----++---+ |
> | | virtual | | virtual | | virtual | |
> | | PCI-PCI | | PCI-PCI | | PCI-PCI | |
> | | bridge | | bridge | | bridge | |
> | +----+----+ +----+----+ +----+----+ |
> | | | | |
> | | | | |
> +----------|--------------------|-------------------|----------+
> | Bus#4? | |
> v v v
>
> So the link itself is Bus#4?
Exactly.
> If there is no device under this link, will Bus#4 appear in kernel?
Yes, I'm pretty sure we build the bus 04 pci_bus structure, then scan
bus 04 for devices. Even if we find none, the pci_bus struct for bus
04 remains.
> And then the PCIe switch is represented by 4 pci_dev structure and each
> is bridge type?
Yes. And each has an associated pci_bus struct for its secondary bus.
Bjorn
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-05-03 16:39 ` Bjorn Helgaas
@ 2012-05-04 2:11 ` Richard Yang
2012-05-06 15:21 ` Richard Yang
2012-05-07 3:00 ` Richard Yang
2 siblings, 0 replies; 16+ messages in thread
From: Richard Yang @ 2012-05-04 2:11 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci
On Thu, May 03, 2012 at 10:39:42AM -0600, Bjorn Helgaas wrote:
>On Thu, May 3, 2012 at 12:21 AM, Richard Yang
><weiyang@linux.vnet.ibm.com> wrote:
>> On Wed, May 02, 2012 at 08:59:40AM -0600, Bjorn Helgaas wrote:
>>>On Wed, May 2, 2012 at 12:24 AM, Richard Yang
>>><weiyang@linux.vnet.ibm.com> wrote:
>>>> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote:
>>>>>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
>>>>><weiyang@linux.vnet.ibm.com> wrote:
>>>>>
>>>> Thanks for your nice chart.
>>>>>I think assignments shown for the PCIe-to-PCI bridge are OK, although
>>>>>I would draw it like this because the bridge originates a single bus
>>>>>02 that may have multiple devices attached to it (this side is PCI,
>>>>>not PCIe, so it really is a shared bus):
>>>>>
>>>>> ^
>>>>> |
>>>>> +--------+--------+
>>>>> | 00:02.0 |
>>>>> | PCIe-PCI bridge |
>>>>> | |
>>>>> +--------+--------+
>>>>> |
>>>>> |
>>>>> +---------------------+ Bus 02
>>>>> | |
>>>>> | |
>>>>> | |
>>>>> +----v----+ +----v----+
>>>>> | 02:00.0 | | 02:01.0 |
>>>>> +---------+ +---------+
>>>>>
>>>> So for this case, there is not internal bus, while this is really a
>>>> physical shared bus, not a logical one.
>>>
>>>Yes. The downstream side of the PCIe-PCI bridge is PCI.
>>>
>>>>>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec
>>>>>1.3.3 of the PCIe r3 spec:
>>>>
>>>>>A PCIe switch appears as two or more PCI-PCI bridges. One is
>>>>>associated with the upstream port; the others with the downstream
>>>>>ports.
>>>>>
>>>>>A bridge always has a primary side and a secondary side. In your
>>>>>diagram, the bridge associated with the upstream port would be 00:01.0
>>>>>(primary bus 00) and could have a secondary bus of 03 (since 02 is
>>>>>already consumed by the PCIe-PCI bridge).
>>>> Hmm... I am confused why is 03. 02 is used but 01 is not used.
>>>> Switch should be configured after PCIe2PCI bridge?
>>>
>>>It's likely that the PCIe switch would be configured first, since its
>>>device number is lower, but that is not a requirement. The
>>>requirement is that the bus number ranges consumed by bridges be
>>>non-overlapping. In this case (using your original topology plus my
>>>PCIe switch diagram), we'd have:
>>>
>>> 1. an endpoint at 00:00.0 -- consumes no additional buses
>>> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
>>> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
>
>> So this is a sequence issue. Below is also a valid configuration.
>> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05]
>> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus)
>
>Yes, this would also be valid.
>
>>>Bus 03 is the internal PCIe switch bus that connects the upstream port
>>>to the downstream ports. Buses 04, 05, and 06 are the links
>>>originating from the downstream ports.
>>>
>>>>>The bridges associated with the downstream ports are all logically on
>>>>>bus 03. Their primary bus number would be 03; they might be 03:00.0,
>>>>>03:01.0, 03:02.0, etc. Each would have its own secondary bus number,
>>>>>for example 04, 05, 06. That secondary bus number is for the
>>>>>downstream link from the corresponding downstream port.
>>>> Hmm, as you mentioned in previous letter, PCIe is an point-to-point
>>>> protocol, then the secondary bus should reside in the Switch?
>>>> Do you think my Bus#4 is correct?
>>>
>>>No. Bus 04 is a PCIe link that connects the downstream port (03:00.0)
>>>to a single PCIe device. That device (04:00) could be a PCIe
>>>endpoint, or it could be the upstream port of another PCIe switch.
>>>(If it is a switch, more bus numbers would be required.)
>>
>>
>> ^
>> |
>> +-------------------------------|------------------------------+
>> | | |
>> | +----+----+ |
>> | | virtual | |
>> | | PCI-PCI | |
>> | | bridge | |
>> | +----+----+ |
>> | | |
>> | |Bus#3 |
>> | | |
>> | +----------------------------------------+ |
>> | | | | |
>> | | | | |
>> | |03:00.0 |03:01.0 |03:02.0 |
>> | +----+----+ +----+----+ +----++---+ |
>> | | virtual | | virtual | | virtual | |
>> | | PCI-PCI | | PCI-PCI | | PCI-PCI | |
>> | | bridge | | bridge | | bridge | |
>> | +----+----+ +----+----+ +----+----+ |
>> | | | | |
>> | | | | |
>> +----------|--------------------|-------------------|----------+
>> | Bus#4? | |
>> v v v
>>
>> So the link itself is Bus#4?
>
>Exactly.
>
>> If there is no device under this link, will Bus#4 appear in kernel?
>
>Yes, I'm pretty sure we build the bus 04 pci_bus structure, then scan
>bus 04 for devices. Even if we find none, the pci_bus struct for bus
>04 remains.
>
>> And then the PCIe switch is represented by 4 pci_dev structure and each
>> is bridge type?
>
>Yes. And each has an associated pci_bus struct for its secondary bus.
Bjorn
Thanks a lot :)
Your patient reply answer my question clearly.
Those questions confused me a lot.
Now I think I get a more clear view about how pci system is represented
in the kernel.
Thanks again.
>
>Bjorn
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-05-03 16:39 ` Bjorn Helgaas
2012-05-04 2:11 ` Richard Yang
@ 2012-05-06 15:21 ` Richard Yang
2012-05-07 3:00 ` Richard Yang
2 siblings, 0 replies; 16+ messages in thread
From: Richard Yang @ 2012-05-06 15:21 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci
On Thu, May 03, 2012 at 10:39:42AM -0600, Bjorn Helgaas wrote:
>On Thu, May 3, 2012 at 12:21 AM, Richard Yang
><weiyang@linux.vnet.ibm.com> wrote:
>> On Wed, May 02, 2012 at 08:59:40AM -0600, Bjorn Helgaas wrote:
>>>On Wed, May 2, 2012 at 12:24 AM, Richard Yang
>>><weiyang@linux.vnet.ibm.com> wrote:
>>>> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote:
>>>>>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
>>>>><weiyang@linux.vnet.ibm.com> wrote:
>>>>>
>>>> Thanks for your nice chart.
>>>>>I think assignments shown for the PCIe-to-PCI bridge are OK, although
>>>>>I would draw it like this because the bridge originates a single bus
>>>>>02 that may have multiple devices attached to it (this side is PCI,
>>>>>not PCIe, so it really is a shared bus):
>>>>>
>>>>> ^
>>>>> |
>>>>> +--------+--------+
>>>>> | 00:02.0 |
>>>>> | PCIe-PCI bridge |
>>>>> | |
>>>>> +--------+--------+
>>>>> |
>>>>> |
>>>>> +---------------------+ Bus 02
>>>>> | |
>>>>> | |
>>>>> | |
>>>>> +----v----+ +----v----+
>>>>> | 02:00.0 | | 02:01.0 |
>>>>> +---------+ +---------+
>>>>>
>>>> So for this case, there is not internal bus, while this is really a
>>>> physical shared bus, not a logical one.
>>>
>>>Yes. The downstream side of the PCIe-PCI bridge is PCI.
>>>
>>>>>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec
>>>>>1.3.3 of the PCIe r3 spec:
>>>>
>>>>>A PCIe switch appears as two or more PCI-PCI bridges. One is
>>>>>associated with the upstream port; the others with the downstream
>>>>>ports.
>>>>>
>>>>>A bridge always has a primary side and a secondary side. In your
>>>>>diagram, the bridge associated with the upstream port would be 00:01.0
>>>>>(primary bus 00) and could have a secondary bus of 03 (since 02 is
>>>>>already consumed by the PCIe-PCI bridge).
>>>> Hmm... I am confused why is 03. 02 is used but 01 is not used.
>>>> Switch should be configured after PCIe2PCI bridge?
>>>
>>>It's likely that the PCIe switch would be configured first, since its
>>>device number is lower, but that is not a requirement. The
>>>requirement is that the bus number ranges consumed by bridges be
>>>non-overlapping. In this case (using your original topology plus my
>>>PCIe switch diagram), we'd have:
>>>
>>> 1. an endpoint at 00:00.0 -- consumes no additional buses
>>> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
>>> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
>
>> So this is a sequence issue. Below is also a valid configuration.
>> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05]
>> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus)
>
>Yes, this would also be valid.
>
>>>Bus 03 is the internal PCIe switch bus that connects the upstream port
>>>to the downstream ports. Buses 04, 05, and 06 are the links
>>>originating from the downstream ports.
>>>
>>>>>The bridges associated with the downstream ports are all logically on
>>>>>bus 03. Their primary bus number would be 03; they might be 03:00.0,
>>>>>03:01.0, 03:02.0, etc. Each would have its own secondary bus number,
>>>>>for example 04, 05, 06. That secondary bus number is for the
>>>>>downstream link from the corresponding downstream port.
>>>> Hmm, as you mentioned in previous letter, PCIe is an point-to-point
>>>> protocol, then the secondary bus should reside in the Switch?
>>>> Do you think my Bus#4 is correct?
>>>
>>>No. Bus 04 is a PCIe link that connects the downstream port (03:00.0)
>>>to a single PCIe device. That device (04:00) could be a PCIe
>>>endpoint, or it could be the upstream port of another PCIe switch.
>>>(If it is a switch, more bus numbers would be required.)
>>
>>
>> ^
>> |
>> +-------------------------------|------------------------------+
>> | | |
>> | +----+----+ |
>> | | virtual | |
>> | | PCI-PCI | |
>> | | bridge | |
>> | +----+----+ |
>> | | |
>> | |Bus#3 |
>> | | |
>> | +----------------------------------------+ |
>> | | | | |
>> | | | | |
>> | |03:00.0 |03:01.0 |03:02.0 |
>> | +----+----+ +----+----+ +----++---+ |
>> | | virtual | | virtual | | virtual | |
>> | | PCI-PCI | | PCI-PCI | | PCI-PCI | |
>> | | bridge | | bridge | | bridge | |
>> | +----+----+ +----+----+ +----+----+ |
>> | | | | |
>> | | | | |
>> +----------|--------------------|-------------------|----------+
>> | Bus#4? | |
>> v v v
>>
>> So the link itself is Bus#4?
Bjorn,
Sorry to disturb you again.
If under this link, one SRIOV device is attached.
And VF is enabled, more than 256 VFs.
How the Bus#4 handle those PFs and VFs?
How new Buses is organized?
Or this is not standarded?
>
>Exactly.
>
>> If there is no device under this link, will Bus#4 appear in kernel?
>
>Yes, I'm pretty sure we build the bus 04 pci_bus structure, then scan
>bus 04 for devices. Even if we find none, the pci_bus struct for bus
>04 remains.
>
>> And then the PCIe switch is represented by 4 pci_dev structure and each
>> is bridge type?
>
>Yes. And each has an associated pci_bus struct for its secondary bus.
>
>Bjorn
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct?
2012-05-03 16:39 ` Bjorn Helgaas
2012-05-04 2:11 ` Richard Yang
2012-05-06 15:21 ` Richard Yang
@ 2012-05-07 3:00 ` Richard Yang
2 siblings, 0 replies; 16+ messages in thread
From: Richard Yang @ 2012-05-07 3:00 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci
On Thu, May 03, 2012 at 10:39:42AM -0600, Bjorn Helgaas wrote:
>>
>>
>>
>>
>> +-------------------------------|------------------------------+
>> | | |
>> | +----+----+ |
>> | | virtual | |
>> | | PCI-PCI | |
>> | | bridge | |
>> | +----+----+
>> | | |
>> | |Bus#3
>> | | |
>> | +----------------------------------------+ |
>> | | | | |
>> | | | | |
>> | |03:00.0 |03:01.0 |03:02.0 |
>> | +----+----+ +----+----+ +----++---+ |
>> | | virtual | | virtual | | virtual | |
>> | | PCI-PCI | | PCI-PCI | | PCI-PCI | |
>> | | bridge | | bridge | | bridge | |
>> | +----+----+ +----+----+ +----+----+ |
>> | | | | |
>> | | | | |
>> +----------|--------------------|-------------------|----------+
>> | Bus#4? | |
>> v v v
>>
I read the code of pci_scan_child_bus(), in this function it will call
pci_scan_slot() like this.
for (devfn = 0; devfn < 0x100; devfn += 8)
pci_scan_slot(bus, devfn);
I think this scheme is based on the PCI LB specification.
If the bus->self is ari enabled, one call of pci_scan_slot(bus, 0) will
scan all the functions under this bus.
Then next 2^5-1 times of pci_scan_slot() will configure the function which
has already been configured, or even empty function.
My idea is
if the bus is ari enabled, just one call of pci_scan_slot() will be
called.
Do you think this works? Or may I miss some real case?
>> So the link itself is Bus#4?
>
>Exactly.
>
>> If there is no device under this link, will Bus#4 appear in kernel?
>
>Yes, I'm pretty sure we build the bus 04 pci_bus structure, then scan
>bus 04 for devices. Even if we find none, the pci_bus struct for bus
>04 remains.
>
>> And then the PCIe switch is represented by 4 pci_dev structure and each
>> is bridge type?
>
>Yes. And each has an associated pci_bus struct for its secondary bus.
>
>Bjorn
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2012-05-07 3:00 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-04-27 9:27 Does my understanding correct? Richard Yang
2012-04-27 14:10 ` Jiang Liu
2012-04-27 15:55 ` Bjorn Helgaas
2012-04-27 14:17 ` Bjorn Helgaas
2012-04-28 5:01 ` Richard Yang
2012-04-28 7:21 ` Richard Yang
2012-04-30 15:56 ` Bjorn Helgaas
2012-05-02 6:24 ` Richard Yang
2012-05-02 14:59 ` Bjorn Helgaas
2012-05-02 21:05 ` Don Dutile
2012-05-03 6:21 ` Richard Yang
2012-05-03 16:39 ` Bjorn Helgaas
2012-05-04 2:11 ` Richard Yang
2012-05-06 15:21 ` Richard Yang
2012-05-07 3:00 ` Richard Yang
2012-04-28 8:21 ` Richard Yang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.