* Does my understanding correct?
@ 2012-04-27 9:27 Richard Yang
2012-04-27 14:10 ` Jiang Liu
2012-04-27 14:17 ` Bjorn Helgaas
0 siblings, 2 replies; 16+ messages in thread
From: Richard Yang @ 2012-04-27 9:27 UTC (permalink / raw)
To: linux-pci; +Cc: bhelgaas
All,
I draw a picture about the physical layout on a part of the pci system.
Each block represent a bridge or a device.
Am I correct?
+-----------------------------+
| |
| |
| Bus#2 |
| ------------------ |
| |
| |
| |
| +-------------+
| |DownStream |02:00.0 pci_dev
+---------------+-------+-----+
| Bus#3
---------------+------------------------------------------+---
| |
| |
+---------------+-------+-----+ +---------------+----+--+-----+
| |UpStream |03:00.0 | |UpStream |03:01.0
| +-------------+ | +-------------+ a normal device
| | | |
| | | |
| Bus#4 | | |
| ------------------- | | |
| +----------+ | |
| |DownStream| | |
+-----------------------+-----+ +-----------------------------+
|
| Bus#5
------------+------------
--
Richard Yang
Help you, Help me
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: Does my understanding correct? 2012-04-27 9:27 Does my understanding correct? Richard Yang @ 2012-04-27 14:10 ` Jiang Liu 2012-04-27 15:55 ` Bjorn Helgaas 2012-04-27 14:17 ` Bjorn Helgaas 1 sibling, 1 reply; 16+ messages in thread From: Jiang Liu @ 2012-04-27 14:10 UTC (permalink / raw) To: Richard Yang; +Cc: linux-pci, bhelgaas For PCI, there's no internal bus Bus#4 in the left bottom device. On 04/27/2012 05:27 PM, Richard Yang wrote: > All, > > I draw a picture about the physical layout on a part of the pci system. > > Each block represent a bridge or a device. > Am I correct? > > +-----------------------------+ > | | > | | > | Bus#2 | > | ------------------ | > | | > | | > | | > | +-------------+ > | |DownStream |02:00.0 pci_dev > +---------------+-------+-----+ > | Bus#3 > ---------------+------------------------------------------+--- > | | > | | > +---------------+-------+-----+ +---------------+----+--+-----+ > | |UpStream |03:00.0 | |UpStream |03:01.0 > | +-------------+ | +-------------+ a normal device > | | | | > | | | | > | Bus#4 | | | > | ------------------- | | | > | +----------+ | | > | |DownStream| | | > +-----------------------+-----+ +-----------------------------+ > | > | Bus#5 > ------------+------------ > > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-04-27 14:10 ` Jiang Liu @ 2012-04-27 15:55 ` Bjorn Helgaas 0 siblings, 0 replies; 16+ messages in thread From: Bjorn Helgaas @ 2012-04-27 15:55 UTC (permalink / raw) To: Jiang Liu; +Cc: Richard Yang, linux-pci T24gRnJpLCBBcHIgMjcsIDIwMTIgYXQgODoxMCBBTSwgSmlhbmcgTGl1IDxsaXVqOTdAZ21haWwu Y29tPiB3cm90ZToKPiBGb3IgUENJLCB0aGVyZSdzIG5vIGludGVybmFsIGJ1cyBCdXMjNCBpbiB0 aGUgbGVmdCBib3R0b20gZGV2aWNlLgoKWWVzLCB0aGVyZSBpcy4gIFRoZSBsZWZ0IGJvdHRvbSBi b3ggaXMgYSBQQ0llIHN3aXRjaCB3aXRoIHVwc3RyZWFtCnBvcnQgMDM6MDAuMC4gIFRoZSB1cHN0 cmVhbSBwb3J0J3Mgc2Vjb25kYXJ5IGJ1cyBpcyAwNCwgd2hpY2ggaXMgdGhlCnN3aXRjaCdzIGlu dGVybmFsIGJ1cy4gIFRoZSBzd2l0Y2gncyBkb3duc3RyZWFtIHBvcnRzIHdvdWxkIGJlLCBmb3IK ZXhhbXBsZSwgMDQ6MDAuMCwgMDQ6MDEuMCwgMDQ6MDIuMCwgZXRjIChvdGhlciBudW1iZXJpbmdz IGFyZQpwb3NzaWJsZSkuICBJZiB0aGUgYm94IGxhYmVsbGVkICJEb3duc3RyZWFtIiBpcyAwNDow MC4wLCBpdHMgc2Vjb25kYXJ5CmJ1cyB3b3VsZCBiZSAwNS4KCkFzIEkgbWVudGlvbmVkIGJlZm9y ZSwgSSB0aGluayB0aGUgYnVzIDA1IGxpbmUgaXMgbWlzbGVhZGluZywgYmVjYXVzZQppdCdzIGRy YXduIGFzIGEgYnVzIHdoZXJlIG11bHRpcGxlIGRldmljZXMgY291bGQgYmUgYXR0YWNoZWQuICBJ dApzaG91bGQgaW5zdGVhZCBiZSBhIHBvaW50LXRvLXBvaW50IGxpbmsgdG8gYSBzaW5nbGUgZGV2 aWNlLiAgKEV4Y2VwdAppbiB0aGlzIFN0cmF0dXMgc3lzdGVtLCB3aGljaCBzZWVtcyBicm9rZW4g YmVjYXVzZSBpdCAqZG9lcyogaGF2ZSB0d28KZGV2aWNlcyBhdHRhY2hlZCB0byB0aGUgc2FtZSBk b3duc3RyZWFtIHBvcnQuKQoKPiBPbiAwNC8yNy8yMDEyIDA1OjI3IFBNLCBSaWNoYXJkIFlhbmcg d3JvdGU6Cj4+IEFsbCwKPj4KPj4gSSBkcmF3IGEgcGljdHVyZSBhYm91dCB0aGUgcGh5c2ljYWwg bGF5b3V0IG9uIGEgcGFydCBvZiB0aGUgcGNpIHN5c3RlbS4KPj4KPj4gRWFjaCBibG9jayByZXBy ZXNlbnQgYSBicmlkZ2Ugb3IgYSBkZXZpY2UuCj4+IEFtIEkgY29ycmVjdD8KPj4KPj4goCCgICst LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4goCCg IHwgoCCgIKBCdXMjMiCgIKAgoCCgIKAgoCCgIKAgoHwKPj4goCCgIHwgoCAtLS0tLS0tLS0tLS0t LS0tLS0goCCgIKAgoHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4g oCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCArLS0tLS0tLS0tLS0tLSsK Pj4goCCgIHwgoCCgIKAgoCCgIKAgoCB8RG93blN0cmVhbSCgIHwwMjowMC4wIHBjaV9kZXYKPj4g oCCgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLSstLS0tLSsKPj4goCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIHwgoCBCdXMjMwo+PiCgIKAgoCCgIKAgoCCgLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0KPj4goCCgIKAgoCCgIKAg oCCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCg IKB8Cj4+IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIKAgoCCgIKAgoCCgfAo+PiCgIKAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tKy0t LS0tKyCgIKAgoCCgIKAgoCCgICstLS0tLS0tLS0tLS0tLS0rLS0tLSstLSstLS0tLSsKPj4goCCg IHwgoCCgIKAgoCCgIKAgoCB8VXBTdHJlYW0goCCgIHwwMzowMC4wIKAgoCCgIKB8IKAgoCCgIKAg oCCgIKAgfFVwU3RyZWFtIKAgoCB8MDM6MDEuMAo+PiCgIKAgfCCgIKAgoCCgIKAgoCCgICstLS0t LS0tLS0tLS0tKyCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCArLS0tLS0tLS0tLS0tLSsg YSBub3JtYWwgZGV2aWNlCj4+IKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8IKAg oCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+PiCgIKAgfCCgIKAg oCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgQnVzIzQgoCCgIKAgoCCgIHwgoCCg IKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8Cj4+IKAgoCB8IKAgoCAt LS0tLS0tLS0tLS0tLS0tLS0tIKAgoCB8IKAgoCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIKAgfAo+PiCgIKAgfCCgIKAgoCCgIKAgoCCgIKAgoCstLS0tLS0tLS0tKyCgIKAg oCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAg oCCgIKAgoCCgIKB8RG93blN0cmVhbXwgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCB8Cj4+IKAgoCArLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0rIKAgoCCg IKAgoCCgIKAgKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKwo+PiCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIKAgfAo+PiCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfCBCdXMjNQo+ PiCgIKAgoCCgIKAgoCCgIKAgLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLQo+Pgo+Pgo+Cg== ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-04-27 9:27 Does my understanding correct? Richard Yang 2012-04-27 14:10 ` Jiang Liu @ 2012-04-27 14:17 ` Bjorn Helgaas 2012-04-28 5:01 ` Richard Yang 2012-04-28 8:21 ` Richard Yang 1 sibling, 2 replies; 16+ messages in thread From: Bjorn Helgaas @ 2012-04-27 14:17 UTC (permalink / raw) To: Richard Yang; +Cc: linux-pci T24gRnJpLCBBcHIgMjcsIDIwMTIgYXQgMzoyNyBBTSwgUmljaGFyZCBZYW5nCjx3ZWl5YW5nQGxp bnV4LnZuZXQuaWJtLmNvbT4gd3JvdGU6Cj4gQWxsLAo+Cj4gSSBkcmF3IGEgcGljdHVyZSBhYm91 dCB0aGUgcGh5c2ljYWwgbGF5b3V0IG9uIGEgcGFydCBvZiB0aGUgcGNpIHN5c3RlbS4KPgo+IEVh Y2ggYmxvY2sgcmVwcmVzZW50IGEgYnJpZGdlIG9yIGEgZGV2aWNlLgo+IEFtIEkgY29ycmVjdD8K Pgo+IKAgoCstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsKPiCgIKB8IKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIKAgoCB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+ IKAgoHwgoCCgIKBCdXMjMiCgIKAgoCCgIKAgoCCgIKAgoHwKPiCgIKB8IKAgLS0tLS0tLS0tLS0t LS0tLS0tIKAgoCCgIKB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+IKAg oHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPiCgIKB8IKAgoCCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgICstLS0tLS0tLS0tLS0tKwo+IKAgoHwg oCCgIKAgoCCgIKAgoCB8RG93blN0cmVhbSCgIHwwMjowMC4wIHBjaV9kZXYKPiCgIKArLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0rLS0tLS0rCj4goCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgfCCg IEJ1cyMzCj4goCCgIKAgoCCgIKAgLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0KPiCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKB8 IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgfAo+IKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIKAgoHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCg IKB8Cj4goCCgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tKy0tLS0tKyCgIKAgoCCgIKAgoCCgICst LS0tLS0tLS0tLS0tLS0rLS0tLSstLSstLS0tLSsKPiCgIKB8IKAgoCCgIKAgoCCgIKAgfFVwU3Ry ZWFtIKAgoCB8MDM6MDAuMCCgIKAgoCCgfCCgIKAgoCCgIKAgoCCgIHxVcFN0cmVhbSCgIKAgfDAz OjAxLjAKPiCgIKB8IKAgoCCgIKAgoCCgIKAgKy0tLS0tLS0tLS0tLS0rIKAgoCCgIKAgoCCgIKAg fCCgIKAgoCCgIKAgoCCgICstLS0tLS0tLS0tLS0tKyBhIG5vcm1hbCBkZXZpY2UKPiCgIKB8IKAg oCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIKAgfAo+IKAgoHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwgoCCg IKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8Cj4goCCgfCCgIKAgoCCg IKAgoCBCdXMjNCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIHwKPiCgIKB8IKAgoCAtLS0tLS0tLS0tLS0tLS0tLS0tIKAgoCB8IKAgoCCgIKAg oCCgIKAgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+IKAgoHwgoCCgIKAgoCCgIKAg oCCgIKArLS0tLS0tLS0tLSsgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCg IKAgoCB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgIKAgoHxEb3duU3RyZWFtfCCgIKAgoCCgIKAgoCCg IHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPiCgIKArLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0rLS0tLS0rIKAgoCCgIKAgoCCgIKAgKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t Kwo+IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoHwKPiCgIKAgoCCgIKAgoCCgIKAgoCCgIKAg oCCgIKB8IEJ1cyM1Cj4goCCgIKAgoCCgIKAgoCCgLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLQoK SSBhc3N1bWUgeW91ciBxdWVzdGlvbiByZWxhdGVzIHRvIHRoZSBTdHJhdHVzIGZ0U2VydmVyIHRv cG9sb2d5LiAgSWYKc28sIHRoZSBsc3BjaSBkZXRhaWxzIG1pZ2h0IGNsYXJpZnkgdGhpbmdzLgoK SW4gdGhhdCBzeXN0ZW0sIG15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdCAwMzowMS4wIGlzIGEgZG93 bnN0cmVhbSBwb3J0LApub3QgYW4gdXBzdHJlYW0gcG9ydC4KCkkgdGhpbmsgeW91ciBwaWN0dXJl IGlzIHNsaWdodGx5IG1pc2xlYWRpbmcgYmVjYXVzZSBQQ0llIGxpbmtzIGFyZSBub3QKYnVzZXM7 IHRoZXkncmUgcG9pbnQtdG8tcG9pbnQgbGlua3MgYmV0d2VlbiB0d28gZGV2aWNlcy4gIFlvdSd2 ZSBkcmF3bgojMyBhbmQgIzUgYXMgYnVzZXMgdGhhdCBjYW4gaGF2ZSBzZXZlcmFsIGRldmljZXMg b24gdGhlbSwgd2hpY2ggaXMgbm90CnJlYWxseSB0aGUgY2FzZS4gIFRoZSBsaW5rIGZyb20gYSBk b3duc3RyZWFtIHBvcnQgc2hvdWxkIGxlYWQgdG8KZXhhY3RseSBvbmUgZGV2aWNlLgoKVGhhdCdz IG9uZSB0aGluZyB0aGF0J3Mgc3RyYW5nZSBpbiB0aGUgZnRTZXJ2ZXIgdG9wb2xvZ3k6IGFwcGFy ZW50bHkKdGhlcmUgYXJlICp0d28qIGRldmljZXMgb24gYnVzIDAzOiB0aGUgMDM6MDAuMCB1cHN0 cmVhbSBwb3J0IGFuZCB0aGUKMDM6MDEuMCBkb3duc3RyZWFtIHBvcnQuICBJIHRoaW5rIDAzOjAw LjAgaXMgdGhlIHVwc3RyZWFtIHBvcnQgb2YgYQpQQ0llIHN3aXRjaCwgd2hpY2ggaXMgcGVyZmVj dGx5IG5vcm1hbC4gIE15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdAowMzowMS4wIGlzIGFub3RoZXIg KmRvd25zdHJlYW0qIHBvcnQgdGhhdCBsZWFkcyB0byBzZXZlcmFsIG1vcmUKZGV2aWNlcyAoVVNC LCBOSUMsIGV0YykuCgpCam9ybgo= ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-04-27 14:17 ` Bjorn Helgaas @ 2012-04-28 5:01 ` Richard Yang 2012-04-28 7:21 ` Richard Yang 2012-04-30 15:56 ` Bjorn Helgaas 2012-04-28 8:21 ` Richard Yang 1 sibling, 2 replies; 16+ messages in thread From: Richard Yang @ 2012-04-28 5:01 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci On Fri, Apr 27, 2012 at 08:17:48AM -0600, Bjorn Helgaas wrote: >On Fri, Apr 27, 2012 at 3:27 AM, Richard Yang ><weiyang@linux.vnet.ibm.com> wrote: > >I assume your question relates to the Stratus ftServer topology. If >so, the lspci details might clarify things. > Yes, my picture is a little bit related to your previous mail. While my intention is to find out how the physical world is represented in the kernel. Below is a typical topology in PCIe spec r3.0. +------------------+ | | | RC | | Bus#0 | | ------------- | | | +-+-----+--------+-+ 00:0.0 | | | 00:02.0 +---------+---------+ | | | +------------+-------------+ | +------+ | +-------| PCIe 2 PCI Bridge | | PCIe Endpoint | | | | +-------------------+ | | Bus#2 | | | -------------- | | +-------+---------------+--+ | | | | 00:01.0 |02:00.0 |02:01.0 +------------+-------------+ +-------+------+ +---+-------+ | | |PCI dev | |PCI dev | | Switch | | | | | | Bus#1 | | | | | | --------------- | +--------------+ +-----------+ | | +------------------------+-+ | | | | | 01:00.0 | 01:01.0 +---------+-------+ +--------+----------------+ | | | | | PCI Endpoint | | PCIe Endpoint | | | | | | | | | +-----------------+ +-------------------------+ Do you think the current assignment of bus number and pci_dev is correct? >In that system, my understanding is that 03:01.0 is a downstream port, >not an upstream port. > >I think your picture is slightly misleading because PCIe links are not >buses; they're point-to-point links between two devices. You've drawn >#3 and #5 as buses that can have several devices on them, which is not >really the case. The link from a downstream port should lead to >exactly one device. > >That's one thing that's strange in the ftServer topology: apparently >there are *two* devices on bus 03: the 03:00.0 upstream port and the >03:01.0 downstream port. I think 03:00.0 is the upstream port of a >PCIe switch, which is perfectly normal. My understanding is that >03:01.0 is another *downstream* port that leads to several more >devices (USB, NIC, etc). > >Bjorn -- Richard Yang Help you, Help me ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-04-28 5:01 ` Richard Yang @ 2012-04-28 7:21 ` Richard Yang 2012-04-30 15:56 ` Bjorn Helgaas 1 sibling, 0 replies; 16+ messages in thread From: Richard Yang @ 2012-04-28 7:21 UTC (permalink / raw) To: Richard Yang; +Cc: Bjorn Helgaas, linux-pci On Sat, Apr 28, 2012 at 01:01:27PM +0800, Richard Yang wrote: >On Fri, Apr 27, 2012 at 08:17:48AM -0600, Bjorn Helgaas wrote: >>On Fri, Apr 27, 2012 at 3:27 AM, Richard Yang >><weiyang@linux.vnet.ibm.com> wrote: >> >>I assume your question relates to the Stratus ftServer topology. If >>so, the lspci details might clarify things. >> >Yes, my picture is a little bit related to your previous mail. >While my intention is to find out how the physical world is represented >in the kernel. > >Below is a typical topology in PCIe spec r3.0. > > +------------------+ > | | > | RC | > | Bus#0 | > | ------------- | > | | > +-+-----+--------+-+ > 00:0.0 | | | 00:02.0 > +---------+---------+ | | | +------------+-------------+ > | +------+ | +-------| PCIe 2 PCI Bridge | > | PCIe Endpoint | | | | > +-------------------+ | | Bus#2 | > | | -------------- | > | +-------+---------------+--+ > | | | > | 00:01.0 |02:00.0 |02:01.0 > +------------+-------------+ +-------+------+ +---+-------+ > | | |PCI dev | |PCI dev | > | Switch | | | | | > | Bus#1 | | | | | > | --------------- | +--------------+ +-----------+ > | | > +------------------------+-+ > | | > | | > | 01:00.0 | 01:01.0 > +---------+-------+ +--------+----------------+ > | | | | > | PCI Endpoint | | PCIe Endpoint | > | | | | > | | | | > +-----------------+ +-------------------------+ > >Do you think the current assignment of bus number and pci_dev is >correct? > BTW, if the chart is correct, 01:00.0 reprents Switch DownStream Port or the PCI Endpoint? > >>In that system, my understanding is that 03:01.0 is a downstream port, >>not an upstream port. >> >>I think your picture is slightly misleading because PCIe links are not >>buses; they're point-to-point links between two devices. You've drawn >>#3 and #5 as buses that can have several devices on them, which is not >>really the case. The link from a downstream port should lead to >>exactly one device. >> >>That's one thing that's strange in the ftServer topology: apparently >>there are *two* devices on bus 03: the 03:00.0 upstream port and the >>03:01.0 downstream port. I think 03:00.0 is the upstream port of a >>PCIe switch, which is perfectly normal. My understanding is that >>03:01.0 is another *downstream* port that leads to several more >>devices (USB, NIC, etc). >> >>Bjorn > >-- >Richard Yang >Help you, Help me -- Richard Yang Help you, Help me ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-04-28 5:01 ` Richard Yang 2012-04-28 7:21 ` Richard Yang @ 2012-04-30 15:56 ` Bjorn Helgaas 2012-05-02 6:24 ` Richard Yang 1 sibling, 1 reply; 16+ messages in thread From: Bjorn Helgaas @ 2012-04-30 15:56 UTC (permalink / raw) To: Richard Yang; +Cc: linux-pci On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang <weiyang@linux.vnet.ibm.com> wrote: > On Fri, Apr 27, 2012 at 08:17:48AM -0600, Bjorn Helgaas wrote: >>On Fri, Apr 27, 2012 at 3:27 AM, Richard Yang >><weiyang@linux.vnet.ibm.com> wrote: >> >>I assume your question relates to the Stratus ftServer topology. If >>so, the lspci details might clarify things. >> > Yes, my picture is a little bit related to your previous mail. > While my intention is to find out how the physical world is represented > in the kernel. > > Below is a typical topology in PCIe spec r3.0. > > +------------------+ > | | > | RC | > | Bus#0 | > | ------------- | > | | > +-+-----+--------+-+ > 00:0.0 | | | 00:02.0 > +---------+---------+ | | | +------------+-------------+ > | +------+ | +-------| PCIe 2 PCI Bridge | > | PCIe Endpoint | | | | > +-------------------+ | | Bus#2 | > | | -------------- | > | +-------+---------------+--+ > | | | > | 00:01.0 |02:00.0 |02:01.0 > +------------+-------------+ +-------+------+ +---+-------+ > | | |PCI dev | |PCI dev | > | Switch | | | | | > | Bus#1 | | | | | > | --------------- | +--------------+ +-----------+ > | | > +------------------------+-+ > | | > | | > | 01:00.0 | 01:01.0 > +---------+-------+ +--------+----------------+ > | | | | > | PCI Endpoint | | PCIe Endpoint | > | | | | > | | | | > +-----------------+ +-------------------------+ > > Do you think the current assignment of bus number and pci_dev is > correct? I think assignments shown for the PCIe-to-PCI bridge are OK, although I would draw it like this because the bridge originates a single bus 02 that may have multiple devices attached to it (this side is PCI, not PCIe, so it really is a shared bus): ^ | +--------+--------+ | 00:02.0 | | PCIe-PCI bridge | | | +--------+--------+ | | +---------------------+ Bus 02 | | | | | | +----v----+ +----v----+ | 02:00.0 | | 02:01.0 | +---------+ +---------+ I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec 1.3.3 of the PCIe r3 spec: ^ | +-------------------------------|------------------------------+ | | | | +----+----+ | | | virtual | | | | PCI-PCI | | | | bridge | | | +----+----+ | | | | | | | | | | | +----------------------------------------+ | | | | | | | | | | | | | | | | | +----+----+ +----+----+ +----++---+ | | | virtual | | virtual | | virtual | | | | PCI-PCI | | PCI-PCI | | PCI-PCI | | | | bridge | | bridge | | bridge | | | +----+----+ +----+----+ +----+----+ | | | | | | | | | | | +----------|--------------------|-------------------|----------+ | | | v v v A PCIe switch appears as two or more PCI-PCI bridges. One is associated with the upstream port; the others with the downstream ports. A bridge always has a primary side and a secondary side. In your diagram, the bridge associated with the upstream port would be 00:01.0 (primary bus 00) and could have a secondary bus of 03 (since 02 is already consumed by the PCIe-PCI bridge). The bridges associated with the downstream ports are all logically on bus 03. Their primary bus number would be 03; they might be 03:00.0, 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, for example 04, 05, 06. That secondary bus number is for the downstream link from the corresponding downstream port. The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 (or these could be the upstream ports of more PCIe switches). Bjorn ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-04-30 15:56 ` Bjorn Helgaas @ 2012-05-02 6:24 ` Richard Yang 2012-05-02 14:59 ` Bjorn Helgaas 0 siblings, 1 reply; 16+ messages in thread From: Richard Yang @ 2012-05-02 6:24 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote: >On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang ><weiyang@linux.vnet.ibm.com> wrote: > Thanks for your nice chart. >I think assignments shown for the PCIe-to-PCI bridge are OK, although >I would draw it like this because the bridge originates a single bus >02 that may have multiple devices attached to it (this side is PCI, >not PCIe, so it really is a shared bus): > > ^ > | > +--------+--------+ > | 00:02.0 | > | PCIe-PCI bridge | > | | > +--------+--------+ > | > | > +---------------------+ Bus 02 > | | > | | > | | > +----v----+ +----v----+ > | 02:00.0 | | 02:01.0 | > +---------+ +---------+ > So for this case, there is not internal bus, while this is really a physical shared bus, not a logical one. >I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec >1.3.3 of the PCIe r3 spec: ^ | +-------------------------------|------------------------------+ | | | | +----+----+ | | | virtual | | | | PCI-PCI | | | | bridge | | | +----+----+ | | | | | |Bus#3 | | | | | +----------------------------------------+ | | | | | | | | | | | | |03:00.0 |03:01.0 |03:02.0 | | +----+----+ +----+----+ +----++---+ | | | virtual | | virtual | | virtual | | | | PCI-PCI | | PCI-PCI | | PCI-PCI | | | | bridge | | bridge | | bridge | | | +----+----+ +----+----+ +----+----+ | | |Bus#4? | | | | -----+------- | | | +----------|--------------------|-------------------|----------+ | | | v v v >A PCIe switch appears as two or more PCI-PCI bridges. One is >associated with the upstream port; the others with the downstream >ports. > >A bridge always has a primary side and a secondary side. In your >diagram, the bridge associated with the upstream port would be 00:01.0 >(primary bus 00) and could have a secondary bus of 03 (since 02 is >already consumed by the PCIe-PCI bridge). Hmm... I am confused why is 03. 02 is used but 01 is not used. Switch should be configured after PCIe2PCI bridge? > >The bridges associated with the downstream ports are all logically on >bus 03. Their primary bus number would be 03; they might be 03:00.0, >03:01.0, 03:02.0, etc. Each would have its own secondary bus number, >for example 04, 05, 06. That secondary bus number is for the >downstream link from the corresponding downstream port. Hmm, as you mentioned in previous letter, PCIe is an point-to-point protocol, then the secondary bus should reside in the Switch? Do you think my Bus#4 is correct? > >The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 >(or these could be the upstream ports of more PCIe switches). So below the PCIe downstream port, there is only on PCIe device. The whole bus is occupied by this device? That is why the device could have upto 256 functions? > >Bjorn -- Richard Yang Help you, Help me ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-05-02 6:24 ` Richard Yang @ 2012-05-02 14:59 ` Bjorn Helgaas 2012-05-02 21:05 ` Don Dutile 2012-05-03 6:21 ` Richard Yang 0 siblings, 2 replies; 16+ messages in thread From: Bjorn Helgaas @ 2012-05-02 14:59 UTC (permalink / raw) To: Richard Yang; +Cc: linux-pci On Wed, May 2, 2012 at 12:24 AM, Richard Yang <weiyang@linux.vnet.ibm.com> wrote: > On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote: >>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang >><weiyang@linux.vnet.ibm.com> wrote: >> > Thanks for your nice chart. >>I think assignments shown for the PCIe-to-PCI bridge are OK, although >>I would draw it like this because the bridge originates a single bus >>02 that may have multiple devices attached to it (this side is PCI, >>not PCIe, so it really is a shared bus): >> >> ^ >> | >> +--------+--------+ >> | 00:02.0 | >> | PCIe-PCI bridge | >> | | >> +--------+--------+ >> | >> | >> +---------------------+ Bus 02 >> | | >> | | >> | | >> +----v----+ +----v----+ >> | 02:00.0 | | 02:01.0 | >> +---------+ +---------+ >> > So for this case, there is not internal bus, while this is really a > physical shared bus, not a logical one. Yes. The downstream side of the PCIe-PCI bridge is PCI. >>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec >>1.3.3 of the PCIe r3 spec: > > ^ > | > +-------------------------------|------------------------------+ > | | | > | +----+----+ | > | | virtual | | > | | PCI-PCI | | > | | bridge | | > | +----+----+ | > | | | > | |Bus#3 | > | | | > | +----------------------------------------+ | > | | | | | > | | | | | > | |03:00.0 |03:01.0 |03:02.0 | > | +----+----+ +----+----+ +----++---+ | > | | virtual | | virtual | | virtual | | > | | PCI-PCI | | PCI-PCI | | PCI-PCI | | > | | bridge | | bridge | | bridge | | > | +----+----+ +----+----+ +----+----+ | > | |Bus#4? | | | > | -----+------- | | | > +----------|--------------------|-------------------|----------+ > | | | > v v v > > >>A PCIe switch appears as two or more PCI-PCI bridges. One is >>associated with the upstream port; the others with the downstream >>ports. >> >>A bridge always has a primary side and a secondary side. In your >>diagram, the bridge associated with the upstream port would be 00:01.0 >>(primary bus 00) and could have a secondary bus of 03 (since 02 is >>already consumed by the PCIe-PCI bridge). > Hmm... I am confused why is 03. 02 is used but 01 is not used. > Switch should be configured after PCIe2PCI bridge? It's likely that the PCIe switch would be configured first, since its device number is lower, but that is not a requirement. The requirement is that the bus number ranges consumed by bridges be non-overlapping. In this case (using your original topology plus my PCIe switch diagram), we'd have: 1. an endpoint at 00:00.0 -- consumes no additional buses 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06] 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus) Bus 03 is the internal PCIe switch bus that connects the upstream port to the downstream ports. Buses 04, 05, and 06 are the links originating from the downstream ports. >>The bridges associated with the downstream ports are all logically on >>bus 03. Their primary bus number would be 03; they might be 03:00.0, >>03:01.0, 03:02.0, etc. Each would have its own secondary bus number, >>for example 04, 05, 06. That secondary bus number is for the >>downstream link from the corresponding downstream port. > Hmm, as you mentioned in previous letter, PCIe is an point-to-point > protocol, then the secondary bus should reside in the Switch? > Do you think my Bus#4 is correct? No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) to a single PCIe device. That device (04:00) could be a PCIe endpoint, or it could be the upstream port of another PCIe switch. (If it is a switch, more bus numbers would be required.) >>The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 >>(or these could be the upstream ports of more PCIe switches). > So below the PCIe downstream port, there is only on PCIe device. > The whole bus is occupied by this device? > That is why the device could have upto 256 functions? Yes, if it supports ARI. Bjorn ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-05-02 14:59 ` Bjorn Helgaas @ 2012-05-02 21:05 ` Don Dutile 2012-05-03 6:21 ` Richard Yang 1 sibling, 0 replies; 16+ messages in thread From: Don Dutile @ 2012-05-02 21:05 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci On 05/02/2012 10:59 AM, Bjorn Helgaas wrote: > On Wed, May 2, 2012 at 12:24 AM, Richard Yang > <weiyang@linux.vnet.ibm.com> wrote: >> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote: >>> On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang >>> <weiyang@linux.vnet.ibm.com> wrote: >>> >> Thanks for your nice chart. >>> I think assignments shown for the PCIe-to-PCI bridge are OK, although >>> I would draw it like this because the bridge originates a single bus >>> 02 that may have multiple devices attached to it (this side is PCI, >>> not PCIe, so it really is a shared bus): >>> >>> ^ >>> | >>> +--------+--------+ >>> | 00:02.0 | >>> | PCIe-PCI bridge | >>> | | >>> +--------+--------+ >>> | >>> | >>> +---------------------+ Bus 02 >>> | | >>> | | >>> | | >>> +----v----+ +----v----+ >>> | 02:00.0 | | 02:01.0 | >>> +---------+ +---------+ >>> >> So for this case, there is not internal bus, while this is really a >> physical shared bus, not a logical one. > > Yes. The downstream side of the PCIe-PCI bridge is PCI. > >>> I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec >>> 1.3.3 of the PCIe r3 spec: >> >> ^ >> | >> +-------------------------------|------------------------------+ >> | | | >> | +----+----+ | >> | | virtual | | >> | | PCI-PCI | | >> | | bridge | | >> | +----+----+ | >> | | | >> | |Bus#3 | >> | | | >> | +----------------------------------------+ | >> | | | | | >> | | | | | >> | |03:00.0 |03:01.0 |03:02.0 | >> | +----+----+ +----+----+ +----++---+ | >> | | virtual | | virtual | | virtual | | >> | | PCI-PCI | | PCI-PCI | | PCI-PCI | | >> | | bridge | | bridge | | bridge | | >> | +----+----+ +----+----+ +----+----+ | >> | |Bus#4? | | | >> | -----+------- | | | >> +----------|--------------------|-------------------|----------+ >> | | | >> v v v >> >> >>> A PCIe switch appears as two or more PCI-PCI bridges. One is >>> associated with the upstream port; the others with the downstream >>> ports. >>> >>> A bridge always has a primary side and a secondary side. In your >>> diagram, the bridge associated with the upstream port would be 00:01.0 >>> (primary bus 00) and could have a secondary bus of 03 (since 02 is >>> already consumed by the PCIe-PCI bridge). >> Hmm... I am confused why is 03. 02 is used but 01 is not used. >> Switch should be configured after PCIe2PCI bridge? > > It's likely that the PCIe switch would be configured first, since its > device number is lower, but that is not a requirement. The > requirement is that the bus number ranges consumed by bridges be > non-overlapping. In this case (using your original topology plus my > PCIe switch diagram), we'd have: > > 1. an endpoint at 00:00.0 -- consumes no additional buses > 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06] > 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus) > > Bus 03 is the internal PCIe switch bus that connects the upstream port > to the downstream ports. Buses 04, 05, and 06 are the links > originating from the downstream ports. > >>> The bridges associated with the downstream ports are all logically on >>> bus 03. Their primary bus number would be 03; they might be 03:00.0, >>> 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, >>> for example 04, 05, 06. That secondary bus number is for the >>> downstream link from the corresponding downstream port. >> Hmm, as you mentioned in previous letter, PCIe is an point-to-point >> protocol, then the secondary bus should reside in the Switch? >> Do you think my Bus#4 is correct? > > No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) > to a single PCIe device. That device (04:00) could be a PCIe > endpoint, or it could be the upstream port of another PCIe switch. > (If it is a switch, more bus numbers would be required.) > >>> The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 >>> (or these could be the upstream ports of more PCIe switches). >> So below the PCIe downstream port, there is only on PCIe device. >> The whole bus is occupied by this device? >> That is why the device could have upto 256 functions? > > Yes, if it supports ARI. > minor tweak: if the device *and* the downstream PCIe bridge support ARI... both must exist. I know of one such module, e.g., non-ARI PCIe switch w/PCIe endpoints) and the endpoints have SRIOV/VF support... not a very smart hw combo. .. nuf said. > Bjorn > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-05-02 14:59 ` Bjorn Helgaas 2012-05-02 21:05 ` Don Dutile @ 2012-05-03 6:21 ` Richard Yang 2012-05-03 16:39 ` Bjorn Helgaas 1 sibling, 1 reply; 16+ messages in thread From: Richard Yang @ 2012-05-03 6:21 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci On Wed, May 02, 2012 at 08:59:40AM -0600, Bjorn Helgaas wrote: >On Wed, May 2, 2012 at 12:24 AM, Richard Yang ><weiyang@linux.vnet.ibm.com> wrote: >> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote: >>>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang >>><weiyang@linux.vnet.ibm.com> wrote: >>> >> Thanks for your nice chart. >>>I think assignments shown for the PCIe-to-PCI bridge are OK, although >>>I would draw it like this because the bridge originates a single bus >>>02 that may have multiple devices attached to it (this side is PCI, >>>not PCIe, so it really is a shared bus): >>> >>> ^ >>> | >>> +--------+--------+ >>> | 00:02.0 | >>> | PCIe-PCI bridge | >>> | | >>> +--------+--------+ >>> | >>> | >>> +---------------------+ Bus 02 >>> | | >>> | | >>> | | >>> +----v----+ +----v----+ >>> | 02:00.0 | | 02:01.0 | >>> +---------+ +---------+ >>> >> So for this case, there is not internal bus, while this is really a >> physical shared bus, not a logical one. > >Yes. The downstream side of the PCIe-PCI bridge is PCI. > >>>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec >>>1.3.3 of the PCIe r3 spec: >> >>>A PCIe switch appears as two or more PCI-PCI bridges. One is >>>associated with the upstream port; the others with the downstream >>>ports. >>> >>>A bridge always has a primary side and a secondary side. In your >>>diagram, the bridge associated with the upstream port would be 00:01.0 >>>(primary bus 00) and could have a secondary bus of 03 (since 02 is >>>already consumed by the PCIe-PCI bridge). >> Hmm... I am confused why is 03. 02 is used but 01 is not used. >> Switch should be configured after PCIe2PCI bridge? > >It's likely that the PCIe switch would be configured first, since its >device number is lower, but that is not a requirement. The >requirement is that the bus number ranges consumed by bridges be >non-overlapping. In this case (using your original topology plus my >PCIe switch diagram), we'd have: > > 1. an endpoint at 00:00.0 -- consumes no additional buses > 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06] > 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus) So this is a sequence issue. Below is also a valid configuration. 2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05] 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus) > >Bus 03 is the internal PCIe switch bus that connects the upstream port >to the downstream ports. Buses 04, 05, and 06 are the links >originating from the downstream ports. > >>>The bridges associated with the downstream ports are all logically on >>>bus 03. Their primary bus number would be 03; they might be 03:00.0, >>>03:01.0, 03:02.0, etc. Each would have its own secondary bus number, >>>for example 04, 05, 06. That secondary bus number is for the >>>downstream link from the corresponding downstream port. >> Hmm, as you mentioned in previous letter, PCIe is an point-to-point >> protocol, then the secondary bus should reside in the Switch? >> Do you think my Bus#4 is correct? > >No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) >to a single PCIe device. That device (04:00) could be a PCIe >endpoint, or it could be the upstream port of another PCIe switch. >(If it is a switch, more bus numbers would be required.) ^ | +-------------------------------|------------------------------+ | | | | +----+----+ | | | virtual | | | | PCI-PCI | | | | bridge | | | +----+----+ | | | | | |Bus#3 | | | | | +----------------------------------------+ | | | | | | | | | | | | |03:00.0 |03:01.0 |03:02.0 | | +----+----+ +----+----+ +----++---+ | | | virtual | | virtual | | virtual | | | | PCI-PCI | | PCI-PCI | | PCI-PCI | | | | bridge | | bridge | | bridge | | | +----+----+ +----+----+ +----+----+ | | | | | | | | | | | +----------|--------------------|-------------------|----------+ | Bus#4? | | v v v So the link itself is Bus#4? If there is no device under this link, will Bus#4 appear in kernel? And then the PCIe switch is represented by 4 pci_dev structure and each is bridge type? > >>>The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 >>>(or these could be the upstream ports of more PCIe switches). >> So below the PCIe downstream port, there is only on PCIe device. >> The whole bus is occupied by this device? >> That is why the device could have upto 256 functions? > >Yes, if it supports ARI. > >Bjorn -- Richard Yang Help you, Help me ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-05-03 6:21 ` Richard Yang @ 2012-05-03 16:39 ` Bjorn Helgaas 2012-05-04 2:11 ` Richard Yang ` (2 more replies) 0 siblings, 3 replies; 16+ messages in thread From: Bjorn Helgaas @ 2012-05-03 16:39 UTC (permalink / raw) To: Richard Yang; +Cc: linux-pci On Thu, May 3, 2012 at 12:21 AM, Richard Yang <weiyang@linux.vnet.ibm.com> wrote: > On Wed, May 02, 2012 at 08:59:40AM -0600, Bjorn Helgaas wrote: >>On Wed, May 2, 2012 at 12:24 AM, Richard Yang >><weiyang@linux.vnet.ibm.com> wrote: >>> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote: >>>>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang >>>><weiyang@linux.vnet.ibm.com> wrote: >>>> >>> Thanks for your nice chart. >>>>I think assignments shown for the PCIe-to-PCI bridge are OK, although >>>>I would draw it like this because the bridge originates a single bus >>>>02 that may have multiple devices attached to it (this side is PCI, >>>>not PCIe, so it really is a shared bus): >>>> >>>> ^ >>>> | >>>> +--------+--------+ >>>> | 00:02.0 | >>>> | PCIe-PCI bridge | >>>> | | >>>> +--------+--------+ >>>> | >>>> | >>>> +---------------------+ Bus 02 >>>> | | >>>> | | >>>> | | >>>> +----v----+ +----v----+ >>>> | 02:00.0 | | 02:01.0 | >>>> +---------+ +---------+ >>>> >>> So for this case, there is not internal bus, while this is really a >>> physical shared bus, not a logical one. >> >>Yes. The downstream side of the PCIe-PCI bridge is PCI. >> >>>>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec >>>>1.3.3 of the PCIe r3 spec: >>> >>>>A PCIe switch appears as two or more PCI-PCI bridges. One is >>>>associated with the upstream port; the others with the downstream >>>>ports. >>>> >>>>A bridge always has a primary side and a secondary side. In your >>>>diagram, the bridge associated with the upstream port would be 00:01.0 >>>>(primary bus 00) and could have a secondary bus of 03 (since 02 is >>>>already consumed by the PCIe-PCI bridge). >>> Hmm... I am confused why is 03. 02 is used but 01 is not used. >>> Switch should be configured after PCIe2PCI bridge? >> >>It's likely that the PCIe switch would be configured first, since its >>device number is lower, but that is not a requirement. The >>requirement is that the bus number ranges consumed by bridges be >>non-overlapping. In this case (using your original topology plus my >>PCIe switch diagram), we'd have: >> >> 1. an endpoint at 00:00.0 -- consumes no additional buses >> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06] >> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus) > So this is a sequence issue. Below is also a valid configuration. > 2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05] > 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus) Yes, this would also be valid. >>Bus 03 is the internal PCIe switch bus that connects the upstream port >>to the downstream ports. Buses 04, 05, and 06 are the links >>originating from the downstream ports. >> >>>>The bridges associated with the downstream ports are all logically on >>>>bus 03. Their primary bus number would be 03; they might be 03:00.0, >>>>03:01.0, 03:02.0, etc. Each would have its own secondary bus number, >>>>for example 04, 05, 06. That secondary bus number is for the >>>>downstream link from the corresponding downstream port. >>> Hmm, as you mentioned in previous letter, PCIe is an point-to-point >>> protocol, then the secondary bus should reside in the Switch? >>> Do you think my Bus#4 is correct? >> >>No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) >>to a single PCIe device. That device (04:00) could be a PCIe >>endpoint, or it could be the upstream port of another PCIe switch. >>(If it is a switch, more bus numbers would be required.) > > > ^ > | > +-------------------------------|------------------------------+ > | | | > | +----+----+ | > | | virtual | | > | | PCI-PCI | | > | | bridge | | > | +----+----+ | > | | | > | |Bus#3 | > | | | > | +----------------------------------------+ | > | | | | | > | | | | | > | |03:00.0 |03:01.0 |03:02.0 | > | +----+----+ +----+----+ +----++---+ | > | | virtual | | virtual | | virtual | | > | | PCI-PCI | | PCI-PCI | | PCI-PCI | | > | | bridge | | bridge | | bridge | | > | +----+----+ +----+----+ +----+----+ | > | | | | | > | | | | | > +----------|--------------------|-------------------|----------+ > | Bus#4? | | > v v v > > So the link itself is Bus#4? Exactly. > If there is no device under this link, will Bus#4 appear in kernel? Yes, I'm pretty sure we build the bus 04 pci_bus structure, then scan bus 04 for devices. Even if we find none, the pci_bus struct for bus 04 remains. > And then the PCIe switch is represented by 4 pci_dev structure and each > is bridge type? Yes. And each has an associated pci_bus struct for its secondary bus. Bjorn ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-05-03 16:39 ` Bjorn Helgaas @ 2012-05-04 2:11 ` Richard Yang 2012-05-06 15:21 ` Richard Yang 2012-05-07 3:00 ` Richard Yang 2 siblings, 0 replies; 16+ messages in thread From: Richard Yang @ 2012-05-04 2:11 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci On Thu, May 03, 2012 at 10:39:42AM -0600, Bjorn Helgaas wrote: >On Thu, May 3, 2012 at 12:21 AM, Richard Yang ><weiyang@linux.vnet.ibm.com> wrote: >> On Wed, May 02, 2012 at 08:59:40AM -0600, Bjorn Helgaas wrote: >>>On Wed, May 2, 2012 at 12:24 AM, Richard Yang >>><weiyang@linux.vnet.ibm.com> wrote: >>>> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote: >>>>>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang >>>>><weiyang@linux.vnet.ibm.com> wrote: >>>>> >>>> Thanks for your nice chart. >>>>>I think assignments shown for the PCIe-to-PCI bridge are OK, although >>>>>I would draw it like this because the bridge originates a single bus >>>>>02 that may have multiple devices attached to it (this side is PCI, >>>>>not PCIe, so it really is a shared bus): >>>>> >>>>> ^ >>>>> | >>>>> +--------+--------+ >>>>> | 00:02.0 | >>>>> | PCIe-PCI bridge | >>>>> | | >>>>> +--------+--------+ >>>>> | >>>>> | >>>>> +---------------------+ Bus 02 >>>>> | | >>>>> | | >>>>> | | >>>>> +----v----+ +----v----+ >>>>> | 02:00.0 | | 02:01.0 | >>>>> +---------+ +---------+ >>>>> >>>> So for this case, there is not internal bus, while this is really a >>>> physical shared bus, not a logical one. >>> >>>Yes. The downstream side of the PCIe-PCI bridge is PCI. >>> >>>>>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec >>>>>1.3.3 of the PCIe r3 spec: >>>> >>>>>A PCIe switch appears as two or more PCI-PCI bridges. One is >>>>>associated with the upstream port; the others with the downstream >>>>>ports. >>>>> >>>>>A bridge always has a primary side and a secondary side. In your >>>>>diagram, the bridge associated with the upstream port would be 00:01.0 >>>>>(primary bus 00) and could have a secondary bus of 03 (since 02 is >>>>>already consumed by the PCIe-PCI bridge). >>>> Hmm... I am confused why is 03. 02 is used but 01 is not used. >>>> Switch should be configured after PCIe2PCI bridge? >>> >>>It's likely that the PCIe switch would be configured first, since its >>>device number is lower, but that is not a requirement. The >>>requirement is that the bus number ranges consumed by bridges be >>>non-overlapping. In this case (using your original topology plus my >>>PCIe switch diagram), we'd have: >>> >>> 1. an endpoint at 00:00.0 -- consumes no additional buses >>> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06] >>> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus) > >> So this is a sequence issue. Below is also a valid configuration. >> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05] >> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus) > >Yes, this would also be valid. > >>>Bus 03 is the internal PCIe switch bus that connects the upstream port >>>to the downstream ports. Buses 04, 05, and 06 are the links >>>originating from the downstream ports. >>> >>>>>The bridges associated with the downstream ports are all logically on >>>>>bus 03. Their primary bus number would be 03; they might be 03:00.0, >>>>>03:01.0, 03:02.0, etc. Each would have its own secondary bus number, >>>>>for example 04, 05, 06. That secondary bus number is for the >>>>>downstream link from the corresponding downstream port. >>>> Hmm, as you mentioned in previous letter, PCIe is an point-to-point >>>> protocol, then the secondary bus should reside in the Switch? >>>> Do you think my Bus#4 is correct? >>> >>>No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) >>>to a single PCIe device. That device (04:00) could be a PCIe >>>endpoint, or it could be the upstream port of another PCIe switch. >>>(If it is a switch, more bus numbers would be required.) >> >> >> ^ >> | >> +-------------------------------|------------------------------+ >> | | | >> | +----+----+ | >> | | virtual | | >> | | PCI-PCI | | >> | | bridge | | >> | +----+----+ | >> | | | >> | |Bus#3 | >> | | | >> | +----------------------------------------+ | >> | | | | | >> | | | | | >> | |03:00.0 |03:01.0 |03:02.0 | >> | +----+----+ +----+----+ +----++---+ | >> | | virtual | | virtual | | virtual | | >> | | PCI-PCI | | PCI-PCI | | PCI-PCI | | >> | | bridge | | bridge | | bridge | | >> | +----+----+ +----+----+ +----+----+ | >> | | | | | >> | | | | | >> +----------|--------------------|-------------------|----------+ >> | Bus#4? | | >> v v v >> >> So the link itself is Bus#4? > >Exactly. > >> If there is no device under this link, will Bus#4 appear in kernel? > >Yes, I'm pretty sure we build the bus 04 pci_bus structure, then scan >bus 04 for devices. Even if we find none, the pci_bus struct for bus >04 remains. > >> And then the PCIe switch is represented by 4 pci_dev structure and each >> is bridge type? > >Yes. And each has an associated pci_bus struct for its secondary bus. Bjorn Thanks a lot :) Your patient reply answer my question clearly. Those questions confused me a lot. Now I think I get a more clear view about how pci system is represented in the kernel. Thanks again. > >Bjorn -- Richard Yang Help you, Help me ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-05-03 16:39 ` Bjorn Helgaas 2012-05-04 2:11 ` Richard Yang @ 2012-05-06 15:21 ` Richard Yang 2012-05-07 3:00 ` Richard Yang 2 siblings, 0 replies; 16+ messages in thread From: Richard Yang @ 2012-05-06 15:21 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci On Thu, May 03, 2012 at 10:39:42AM -0600, Bjorn Helgaas wrote: >On Thu, May 3, 2012 at 12:21 AM, Richard Yang ><weiyang@linux.vnet.ibm.com> wrote: >> On Wed, May 02, 2012 at 08:59:40AM -0600, Bjorn Helgaas wrote: >>>On Wed, May 2, 2012 at 12:24 AM, Richard Yang >>><weiyang@linux.vnet.ibm.com> wrote: >>>> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote: >>>>>On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang >>>>><weiyang@linux.vnet.ibm.com> wrote: >>>>> >>>> Thanks for your nice chart. >>>>>I think assignments shown for the PCIe-to-PCI bridge are OK, although >>>>>I would draw it like this because the bridge originates a single bus >>>>>02 that may have multiple devices attached to it (this side is PCI, >>>>>not PCIe, so it really is a shared bus): >>>>> >>>>> ^ >>>>> | >>>>> +--------+--------+ >>>>> | 00:02.0 | >>>>> | PCIe-PCI bridge | >>>>> | | >>>>> +--------+--------+ >>>>> | >>>>> | >>>>> +---------------------+ Bus 02 >>>>> | | >>>>> | | >>>>> | | >>>>> +----v----+ +----v----+ >>>>> | 02:00.0 | | 02:01.0 | >>>>> +---------+ +---------+ >>>>> >>>> So for this case, there is not internal bus, while this is really a >>>> physical shared bus, not a logical one. >>> >>>Yes. The downstream side of the PCIe-PCI bridge is PCI. >>> >>>>>I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec >>>>>1.3.3 of the PCIe r3 spec: >>>> >>>>>A PCIe switch appears as two or more PCI-PCI bridges. One is >>>>>associated with the upstream port; the others with the downstream >>>>>ports. >>>>> >>>>>A bridge always has a primary side and a secondary side. In your >>>>>diagram, the bridge associated with the upstream port would be 00:01.0 >>>>>(primary bus 00) and could have a secondary bus of 03 (since 02 is >>>>>already consumed by the PCIe-PCI bridge). >>>> Hmm... I am confused why is 03. 02 is used but 01 is not used. >>>> Switch should be configured after PCIe2PCI bridge? >>> >>>It's likely that the PCIe switch would be configured first, since its >>>device number is lower, but that is not a requirement. The >>>requirement is that the bus number ranges consumed by bridges be >>>non-overlapping. In this case (using your original topology plus my >>>PCIe switch diagram), we'd have: >>> >>> 1. an endpoint at 00:00.0 -- consumes no additional buses >>> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06] >>> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus) > >> So this is a sequence issue. Below is also a valid configuration. >> 2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05] >> 3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus) > >Yes, this would also be valid. > >>>Bus 03 is the internal PCIe switch bus that connects the upstream port >>>to the downstream ports. Buses 04, 05, and 06 are the links >>>originating from the downstream ports. >>> >>>>>The bridges associated with the downstream ports are all logically on >>>>>bus 03. Their primary bus number would be 03; they might be 03:00.0, >>>>>03:01.0, 03:02.0, etc. Each would have its own secondary bus number, >>>>>for example 04, 05, 06. That secondary bus number is for the >>>>>downstream link from the corresponding downstream port. >>>> Hmm, as you mentioned in previous letter, PCIe is an point-to-point >>>> protocol, then the secondary bus should reside in the Switch? >>>> Do you think my Bus#4 is correct? >>> >>>No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) >>>to a single PCIe device. That device (04:00) could be a PCIe >>>endpoint, or it could be the upstream port of another PCIe switch. >>>(If it is a switch, more bus numbers would be required.) >> >> >> ^ >> | >> +-------------------------------|------------------------------+ >> | | | >> | +----+----+ | >> | | virtual | | >> | | PCI-PCI | | >> | | bridge | | >> | +----+----+ | >> | | | >> | |Bus#3 | >> | | | >> | +----------------------------------------+ | >> | | | | | >> | | | | | >> | |03:00.0 |03:01.0 |03:02.0 | >> | +----+----+ +----+----+ +----++---+ | >> | | virtual | | virtual | | virtual | | >> | | PCI-PCI | | PCI-PCI | | PCI-PCI | | >> | | bridge | | bridge | | bridge | | >> | +----+----+ +----+----+ +----+----+ | >> | | | | | >> | | | | | >> +----------|--------------------|-------------------|----------+ >> | Bus#4? | | >> v v v >> >> So the link itself is Bus#4? Bjorn, Sorry to disturb you again. If under this link, one SRIOV device is attached. And VF is enabled, more than 256 VFs. How the Bus#4 handle those PFs and VFs? How new Buses is organized? Or this is not standarded? > >Exactly. > >> If there is no device under this link, will Bus#4 appear in kernel? > >Yes, I'm pretty sure we build the bus 04 pci_bus structure, then scan >bus 04 for devices. Even if we find none, the pci_bus struct for bus >04 remains. > >> And then the PCIe switch is represented by 4 pci_dev structure and each >> is bridge type? > >Yes. And each has an associated pci_bus struct for its secondary bus. > >Bjorn -- Richard Yang Help you, Help me ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-05-03 16:39 ` Bjorn Helgaas 2012-05-04 2:11 ` Richard Yang 2012-05-06 15:21 ` Richard Yang @ 2012-05-07 3:00 ` Richard Yang 2 siblings, 0 replies; 16+ messages in thread From: Richard Yang @ 2012-05-07 3:00 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci On Thu, May 03, 2012 at 10:39:42AM -0600, Bjorn Helgaas wrote: >> >> >> >> >> +-------------------------------|------------------------------+ >> | | | >> | +----+----+ | >> | | virtual | | >> | | PCI-PCI | | >> | | bridge | | >> | +----+----+ >> | | | >> | |Bus#3 >> | | | >> | +----------------------------------------+ | >> | | | | | >> | | | | | >> | |03:00.0 |03:01.0 |03:02.0 | >> | +----+----+ +----+----+ +----++---+ | >> | | virtual | | virtual | | virtual | | >> | | PCI-PCI | | PCI-PCI | | PCI-PCI | | >> | | bridge | | bridge | | bridge | | >> | +----+----+ +----+----+ +----+----+ | >> | | | | | >> | | | | | >> +----------|--------------------|-------------------|----------+ >> | Bus#4? | | >> v v v >> I read the code of pci_scan_child_bus(), in this function it will call pci_scan_slot() like this. for (devfn = 0; devfn < 0x100; devfn += 8) pci_scan_slot(bus, devfn); I think this scheme is based on the PCI LB specification. If the bus->self is ari enabled, one call of pci_scan_slot(bus, 0) will scan all the functions under this bus. Then next 2^5-1 times of pci_scan_slot() will configure the function which has already been configured, or even empty function. My idea is if the bus is ari enabled, just one call of pci_scan_slot() will be called. Do you think this works? Or may I miss some real case? >> So the link itself is Bus#4? > >Exactly. > >> If there is no device under this link, will Bus#4 appear in kernel? > >Yes, I'm pretty sure we build the bus 04 pci_bus structure, then scan >bus 04 for devices. Even if we find none, the pci_bus struct for bus >04 remains. > >> And then the PCIe switch is represented by 4 pci_dev structure and each >> is bridge type? > >Yes. And each has an associated pci_bus struct for its secondary bus. > >Bjorn -- Richard Yang Help you, Help me ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Does my understanding correct? 2012-04-27 14:17 ` Bjorn Helgaas 2012-04-28 5:01 ` Richard Yang @ 2012-04-28 8:21 ` Richard Yang 1 sibling, 0 replies; 16+ messages in thread From: Richard Yang @ 2012-04-28 8:21 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Richard Yang, linux-pci On Fri, Apr 27, 2012 at 08:17:48AM -0600, Bjorn Helgaas wrote: >I assume your question relates to the Stratus ftServer topology. If >so, the lspci details might clarify things. > >In that system, my understanding is that 03:01.0 is a downstream port, >not an upstream port. > >I think your picture is slightly misleading because PCIe links are not >buses; they're point-to-point links between two devices. You've drawn >#3 and #5 as buses that can have several devices on them, which is not >really the case. The link from a downstream port should lead to >exactly one device. Hmm... if this is a PCI bridge, then it will have both internal bus and externel bus? or no internal pci bus, just one external bus? > >That's one thing that's strange in the ftServer topology: apparently >there are *two* devices on bus 03: the 03:00.0 upstream port and the >03:01.0 downstream port. I think 03:00.0 is the upstream port of a >PCIe switch, which is perfectly normal. My understanding is that >03:01.0 is another *downstream* port that leads to several more >devices (USB, NIC, etc). > >Bjorn -- Richard Yang Help you, Help me ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2012-05-07 3:00 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-04-27 9:27 Does my understanding correct? Richard Yang 2012-04-27 14:10 ` Jiang Liu 2012-04-27 15:55 ` Bjorn Helgaas 2012-04-27 14:17 ` Bjorn Helgaas 2012-04-28 5:01 ` Richard Yang 2012-04-28 7:21 ` Richard Yang 2012-04-30 15:56 ` Bjorn Helgaas 2012-05-02 6:24 ` Richard Yang 2012-05-02 14:59 ` Bjorn Helgaas 2012-05-02 21:05 ` Don Dutile 2012-05-03 6:21 ` Richard Yang 2012-05-03 16:39 ` Bjorn Helgaas 2012-05-04 2:11 ` Richard Yang 2012-05-06 15:21 ` Richard Yang 2012-05-07 3:00 ` Richard Yang 2012-04-28 8:21 ` Richard Yang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.