[question]: Query regarding the PCI addresses

[question]: Query regarding the PCI addresses

[Wenchao Hao]

Since linux identify PCI peripheral by [domain:bus:device:function] 
number like following,

# lspci -D
0000:00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge
0000:00:01.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
0000:00:02.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub 
PCI Express Root Port 0 (rev 02)
0000:00:02.1 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub 
PCI Express Root Port 0 (rev 02)
0000:00:02.2 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub 
PCI Express Root Port 0 (rev 02)
0000:00:02.3 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub 
PCI Express Root Port 0 (rev 02)
0000:01:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
0000:02:01.0 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) 
USB2 EHCI Controller (rev 10)
0000:02:02.0 Unclassified device [00ff]: Virtio: Virtio memory balloon
0000:02:03.0 SCSI storage controller: Virtio: Virtio SCSI
0000:02:04.0 Display controller: Virtio: Virtio GPU (rev 01)
0000:03:00.0 Ethernet controller: Virtio: Virtio network device (rev 01)

Here are my questions: Are these [domain:bus:device:function] number 
come from hardware's
physical connection or allocated by software dynamic? If hardware do not 
change, can we
guarantee these number do not change after system reboot? If they are 
not fixed, then is there
anyway I can get a fixed ID which can indicate physical connection.

Thanks, Hope to get your help.

Re: [question]: Query regarding the PCI addresses

[Bjorn Helgaas]

[+cc Matthew for "lspci -P"]

On Wed, Jul 14, 2021 at 02:33:37PM +0800, Wenchao Hao wrote:
> Since linux identify PCI peripheral by [domain:bus:device:function] number
> like following,
> 
> # lspci -D
> 0000:00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge
> 0000:00:01.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
> 0000:00:02.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI
> Express Root Port 0 (rev 02)
> 0000:00:02.1 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI
> Express Root Port 0 (rev 02)
> 0000:00:02.2 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI
> Express Root Port 0 (rev 02)
> 0000:00:02.3 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI
> Express Root Port 0 (rev 02)
> 0000:01:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
> 0000:02:01.0 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M)
> USB2 EHCI Controller (rev 10)
> 0000:02:02.0 Unclassified device [00ff]: Virtio: Virtio memory balloon
> 0000:02:03.0 SCSI storage controller: Virtio: Virtio SCSI
> 0000:02:04.0 Display controller: Virtio: Virtio GPU (rev 01)
> 0000:03:00.0 Ethernet controller: Virtio: Virtio network device (rev 01)
> 
> Here are my questions: Are these [domain:bus:device:function] number
> come from hardware's physical connection or allocated by software
> dynamic?

The device and function numbers are completely determined by the
hardware and are not programmable.

Bus numbers are programmable and are determined by the Secondary Bus
Number of the bridge leading to the device.  These bus numbers are
generally programmed by the BIOS on x86.  It's possible for Linux to
reprogram them, but it generally leaves them alone if they are valid.

On x86 with ACPI, the domain number comes from the _SEG method of the
PNP0A03 device that describes the host bridge.  This may correspond to
a programmable hardware register, but that isn't visible to the OS and
Linux has no way to change it.

> If hardware do not change, can we guarantee these number do not
> change after system reboot?

For the typical x86 system with ACPI, this is really a question for
the BIOS.  If the hardware doesn't change, I would *expect* the
domain/bus/device/function numbers to stay the same, but only BIOS
folks can answer this definitively.

> If they are not fixed, then is there anyway I can get a fixed ID
> which can indicate physical connection.

You can look at the "lspci -P" option.  I'm not really familiar with
this, but I think Matthew (cc'd) implemented it.

Bjorn

Re: [question]: Query regarding the PCI addresses

[Keith Busch]

On Wed, Jul 14, 2021 at 11:54:27AM -0500, Bjorn Helgaas wrote:
> On Wed, Jul 14, 2021 at 02:33:37PM +0800, Wenchao Hao wrote:
> 
> > If they are not fixed, then is there anyway I can get a fixed ID
> > which can indicate physical connection.
> 
> You can look at the "lspci -P" option.  I'm not really familiar with
> this, but I think Matthew (cc'd) implemented it.

That option shows the parent devices for each listed device, but that
may not produce the same output if BDf doesn't always enumerate the
same.

I think Wenchao was seeking some invariant device identification that
can be used to look up its BDf. There's no PCI level requirement for
uniquely identifying a specific device across changing topologies, so I
don't think this is generically possible.

Re: [question]: Query regarding the PCI addresses

[Wenchao Hao]

On 2021/7/15 0:54, Bjorn Helgaas wrote:
> [+cc Matthew for "lspci -P"]
>
> On Wed, Jul 14, 2021 at 02:33:37PM +0800, Wenchao Hao wrote:
>> Since linux identify PCI peripheral by [domain:bus:device:function] number
>> like following,
>>
>> # lspci -D
>> 0000:00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge
>> 0000:00:01.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
>> 0000:00:02.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI
>> Express Root Port 0 (rev 02)
>> 0000:00:02.1 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI
>> Express Root Port 0 (rev 02)
>> 0000:00:02.2 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI
>> Express Root Port 0 (rev 02)
>> 0000:00:02.3 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI
>> Express Root Port 0 (rev 02)
>> 0000:01:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
>> 0000:02:01.0 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M)
>> USB2 EHCI Controller (rev 10)
>> 0000:02:02.0 Unclassified device [00ff]: Virtio: Virtio memory balloon
>> 0000:02:03.0 SCSI storage controller: Virtio: Virtio SCSI
>> 0000:02:04.0 Display controller: Virtio: Virtio GPU (rev 01)
>> 0000:03:00.0 Ethernet controller: Virtio: Virtio network device (rev 01)
>>
>> Here are my questions: Are these [domain:bus:device:function] number
>> come from hardware's physical connection or allocated by software
>> dynamic?
> The device and function numbers are completely determined by the
> hardware and are not programmable.
>
> Bus numbers are programmable and are determined by the Secondary Bus
> Number of the bridge leading to the device.  These bus numbers are
> generally programmed by the BIOS on x86.  It's possible for Linux to
> reprogram them, but it generally leaves them alone if they are valid.
>
> On x86 with ACPI, the domain number comes from the _SEG method of the
> PNP0A03 device that describes the host bridge.  This may correspond to
> a programmable hardware register, but that isn't visible to the OS and
> Linux has no way to change it.
>
>> If hardware do not change, can we guarantee these number do not
>> change after system reboot?
> For the typical x86 system with ACPI, this is really a question for
> the BIOS.  If the hardware doesn't change, I would *expect* the
> domain/bus/device/function numbers to stay the same, but only BIOS
> folks can answer this definitively.
>
>> If they are not fixed, then is there anyway I can get a fixed ID
>> which can indicate physical connection.
> You can look at the "lspci -P" option.  I'm not really familiar with
> this, but I think Matthew (cc'd) implemented it.
>
> Bjorn
> .

Thanks Bjorn for your great analysis and I would ask BIOS for their 
suggestions.
I think we can put some constraints such as make BIOS give a stable bus
number on our users to meet their needs of finding a stable path,
although this method looks ugly.

Wenchao

Re: [question]: Query regarding the PCI addresses

[Wenchao Hao]

On 2021/7/15 1:26, Keith Busch wrote:
> On Wed, Jul 14, 2021 at 11:54:27AM -0500, Bjorn Helgaas wrote:
>> On Wed, Jul 14, 2021 at 02:33:37PM +0800, Wenchao Hao wrote:
>>
>>> If they are not fixed, then is there anyway I can get a fixed ID
>>> which can indicate physical connection.
>> You can look at the "lspci -P" option.  I'm not really familiar with
>> this, but I think Matthew (cc'd) implemented it.
> That option shows the parent devices for each listed device, but that
> may not produce the same output if BDf doesn't always enumerate the
> same.
>
> I think Wenchao was seeking some invariant device identification that
> can be used to look up its BDf. There's no PCI level requirement for
> uniquely identifying a specific device across changing topologies, so I
> don't think this is generically possible.
> .

Yes, I want a way to access device which can keep unchanged, a
direction is according to hardware. If we have anyway which makes
it possible for software can describe hardware connection would
satisfy our demand.
Wenchao

Re: [question]: Query regarding the PCI addresses

[Bjorn Helgaas]

On Fri, Jul 16, 2021 at 10:04:51PM +0800, Wenchao Hao wrote:
> On 2021/7/15 1:26, Keith Busch wrote:
> > On Wed, Jul 14, 2021 at 11:54:27AM -0500, Bjorn Helgaas wrote:
> > > On Wed, Jul 14, 2021 at 02:33:37PM +0800, Wenchao Hao wrote:
> > > 
> > > > If they are not fixed, then is there anyway I can get a fixed ID
> > > > which can indicate physical connection.
> > > You can look at the "lspci -P" option.  I'm not really familiar with
> > > this, but I think Matthew (cc'd) implemented it.
> > That option shows the parent devices for each listed device, but that
> > may not produce the same output if BDf doesn't always enumerate the
> > same.
> > 
> > I think Wenchao was seeking some invariant device identification that
> > can be used to look up its BDf. There's no PCI level requirement for
> > uniquely identifying a specific device across changing topologies, so I
> > don't think this is generically possible.
> 
> Yes, I want a way to access device which can keep unchanged, a
> direction is according to hardware. If we have anyway which makes
> it possible for software can describe hardware connection would
> satisfy our demand.

I don't know whether this would be useful, but PCI does define an
optional "Device Serial Number" extended capability.  It has issues
like the fact that many devices don't support it at all, and even on
devices that do support it, the serial number may not actually be
unique.  There is minimal support for this in Linux (pci_get_dsn()),
but it is currently not exposed to userspace via sysfs.

Bjorn

Re: [question]: Query regarding the PCI addresses

[Wenchao Hao]

On 2021/7/16 22:25, Bjorn Helgaas wrote:
> On Fri, Jul 16, 2021 at 10:04:51PM +0800, Wenchao Hao wrote:
>> On 2021/7/15 1:26, Keith Busch wrote:
>>> On Wed, Jul 14, 2021 at 11:54:27AM -0500, Bjorn Helgaas wrote:
>>>> On Wed, Jul 14, 2021 at 02:33:37PM +0800, Wenchao Hao wrote:
>>>>
>>>>> If they are not fixed, then is there anyway I can get a fixed ID
>>>>> which can indicate physical connection.
>>>> You can look at the "lspci -P" option.  I'm not really familiar with
>>>> this, but I think Matthew (cc'd) implemented it.
>>> That option shows the parent devices for each listed device, but that
>>> may not produce the same output if BDf doesn't always enumerate the
>>> same.
>>>
>>> I think Wenchao was seeking some invariant device identification that
>>> can be used to look up its BDf. There's no PCI level requirement for
>>> uniquely identifying a specific device across changing topologies, so I
>>> don't think this is generically possible.
>> Yes, I want a way to access device which can keep unchanged, a
>> direction is according to hardware. If we have anyway which makes
>> it possible for software can describe hardware connection would
>> satisfy our demand.
> I don't know whether this would be useful, but PCI does define an
> optional "Device Serial Number" extended capability.  It has issues
> like the fact that many devices don't support it at all, and even on
> devices that do support it, the serial number may not actually be
> unique.  There is minimal support for this in Linux (pci_get_dsn()),
> but it is currently not exposed to userspace via sysfs.
>
> Bjorn
> .

This "Device Serial Number" seems can not satisfy our demand because it 
is looks
not a general ID. According to BIOS conclusion, if we can the hardware keeps
unchanged, the BUS number would not change. We would put some 
restrictions as
workaround temporarily.

Thanks a lot for your suggestions.