PCIe bus (re-)numbering

PCIe bus (re-)numbering

[Ruud]

Hello all,

Not a patch, not a complaint: a start of a discussion on PCIe bus
renumbering and bus numbering in general..

For bigger PCIe chassis I notice it contains lots of levels of PCIe
switches e.g. as per
commit lopg https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/commit/?h=for-pci-v4.3-rc1&id=d3934f379e3a35aed05b53aeb49b5fb872c55aa1

Imagine this kind of tree is behind a hot-plug interface at
[1c.0-[01-10]]  and is plugged in later. Like

pci tree:
-[0000:00]-+-00.0
           +-1c.0-[01-10]--+-00.0-[02-10]--+-01.0-[03]----00.0  PLX
Technology, Inc. Device 87b1
           |               |
+-02.0-[04-09]--+-00.0-[05-09]--+-01.0-[06]----00.0  PLX Technology,
Inc. Device 87b1
           |               |               |               |
    +-02.0-[07]----00.0  Broadcom Corporation Device 8650
           |               |               |               |
    +-03.0-[08]--
           |               |               |               |
    \-04.0-[09]----00.0  Altera Corporation Device 0201
           |               |               |               +-00.1  PLX
Technology, Inc. Device 87d0
           |               |               |               +-00.2  PLX
Technology, Inc. Device 87d0
           |               |               |               +-00.3  PLX
Technology, Inc. Device 87d0
           |               |               |               \-00.4  PLX
Technology, Inc. Device 87d0
           |               |
+-03.0-[0a-0f]--+-00.0-[0b-0f]--+-01.0-[0c]----00.0  PLX Technology,
Inc. Device 87b1
           |               |               |               |
    +-02.0-[0d]----00.0  Broadcom Corporation Device 8650
           |               |               |               |
    +-03.0-[0e]--
           |               |               |               |
    \-04.0-[0f]----00.0  Altera Corporation Device 0201
           |               |               |               +-00.1  PLX
Technology, Inc. Device 87d0
           |               |               |               +-00.2  PLX
Technology, Inc. Device 87d0
           |               |               |               +-00.3  PLX
Technology, Inc. Device 87d0
           |               |               |               \-00.4  PLX
Technology, Inc. Device 87d0
           |               |               \-04.0-[10]--
           |               +-00.1  PLX Technology, Inc. Device 87d0
           |               +-00.2  PLX Technology, Inc. Device 87d0
           |               +-00.3  PLX Technology, Inc. Device 87d0
           |               \-00.4  PLX Technology, Inc. Device 87d0
           +-1c.3-[11]----00.0

The current algorithm seems to allocate 8 extra busnumbers at the
hotplug switch, but clearly 8 is not sufficient for the whole tree
when it is discovered after initial numbering has been assigned. As
the PCIe routing requires the bus numbers to be consecutive as it
describes ranges there are not that many allocation strategies for bus
numbers. It is impossible to predict at boot-time which switch will
require lots of busses and which do not.

A solution is static assignment (e.g. as described by
http://article.gmane.org/gmane.linux.kernel.pci/45212), but it seems
not convenient to me.

I got the impression the most elegant way is to renumber, but at the
same time I doubt. Would the BIOS become confused? Currently the
kernel becomes confused as it renumbers the ethernet interfaces when
the bus-numbers change. Several drivers seem to be locked to the
device by its geographical routing (aka bus << 16 | device << 11 |
function << 8 ). I got the impression that this is the root of the
evil as the bus need not be as constant as expected.

E.g. the Broadcom device at bus 07 device 00 function 0 could just as
well be at bus 08 device 00 function 0 when an extra busnumber is
assigned to another switch.

Would it be an idea to describe the geographical location for this
device as the full chain..
[0000:00].[1c.0].[00.0].[02.0].[00.0].[02.0].[00.0]. This would be
invariant for busnumbering. Device drivers if in need for the bus
number (why would they in the first place?) could determine the actual
bus number at that moment in time. As a result ethernet interface
renaming would perhaps also not happen?

I am not that deep in the material yet (in respect to the kernel
code). But I got the feeling that by allowing renumbering the
assignment procedure can be greatly simplified and become more robust
for big PCIe configs... probably moving complexity to other parts like
ethernet naming.

What does the community think?

Best regards,

Ruud

Re: PCIe bus (re-)numbering

[Yinghai Lu]

On Sat, Sep 19, 2015 at 1:20 AM, Ruud <netwerkforens@gmail.com> wrote:
> Hello all,
>
> Not a patch, not a complaint: a start of a discussion on PCIe bus
> renumbering and bus numbering in general..
>
>
> The current algorithm seems to allocate 8 extra busnumbers at the
> hotplug switch, but clearly 8 is not sufficient for the whole tree
> when it is discovered after initial numbering has been assigned. As
> the PCIe routing requires the bus numbers to be consecutive as it
> describes ranges there are not that many allocation strategies for bus
> numbers. It is impossible to predict at boot-time which switch will
> require lots of busses and which do not.

Well, if you need more than 8 bus number then practical way is
booting with pcie switch and late only hot-remove and host-add
instead of code hot-add.

>
> A solution is static assignment (e.g. as described by
> http://article.gmane.org/gmane.linux.kernel.pci/45212), but it seems
> not convenient to me.

Interesting, one patch in that thread looks like try to use bus range blindly.
so it is no go.

>
> I got the impression the most elegant way is to renumber, but at the
> same time I doubt. Would the BIOS become confused? Currently the
> kernel becomes confused as it renumbers the ethernet interfaces when
> the bus-numbers change. Several drivers seem to be locked to the
> device by its geographical routing (aka bus << 16 | device << 11 |
> function << 8 ). I got the impression that this is the root of the
> evil as the bus need not be as constant as expected.

Do you mean changing bus number without unloading driver ?

No, you can not do that.

some device firmware like lsi cards, if you change it's primary bus number,
the device will stop working, but that is another problem.

For cold hot add several pcie switches, right way would be have a script:
1. remove related children devices.
2. use setpci to clear bus number register
3. remove bridge devices
4. do pci rescan.

Yinghai

Re: PCIe bus (re-)numbering

[Ruud]

>> The current algorithm seems to allocate 8 extra busnumbers at the
>> hotplug switch, but clearly 8 is not sufficient for the whole tree
>> when it is discovered after initial numbering has been assigned. As
>> the PCIe routing requires the bus numbers to be consecutive as it
>> describes ranges there are not that many allocation strategies for bus
>> numbers. It is impossible to predict at boot-time which switch will
>> require lots of busses and which do not.
>
> Well, if you need more than 8 bus number then practical way is
> booting with pcie switch and late only hot-remove and host-add
> instead of code hot-add.

The current procedure I follow is to boot with two PCIe switches in the host.
(one at the root complex level, intel based, one level above PLX
based, and the whole tree in the chassis).

- I turn off the chassis (as it conflicts with the BIOS :( )
- Reboot into linux.
- remove the intel based switch (has no relevant childs) (echo 1
>.../remove  sorry for the missing numbers its weekend)
- turn on chassis
- rescan starting at the root complex  (echo 1 > .../rescan )

During the rescan, it will map in the original busnumber-range which
is too small. I understand from your email that by clearing the
busnumber range in the switch (perhaps both host switces), the kernel
will pick a different range which is not clamped in by the other
busnumbers of surrounding other switches?

I will test next monday.

What I did get to work is the following procedure:

- I turn off the chassis (as it conflicts with the BIOS :(  )
- Reboot into GRUB
- turn on chassis
- Boot linux with parameter pci=assign-busses (BIOS will have
configured the switches in the host without a serious busnumber range)
This procedure is very inconvenient as the host is operated headless.

What almost works is the following procedure:

- I turn off the chassis (as it conflicts with the BIOS :(  )
- Boot linux with parameter pci=assign-busses (BIOS will have
configured the switches in the host without a serious busnumber range)
- remove the intel based switch (has no relevant childs) (echo 1
>.../remove  sorry for the missing numbers its weekend)
- turn on chassis
- rescan starting at the root complex  (echo 1 > .../rescan )
During rescan the numbering is messed up, and dmesg fills up with
ethernet renaming "errors", didn;t dare to look at other side-effects.

>
>>
>
> Do you mean changing bus number without unloading driver ?
>
> No, you can not do that.
>
> some device firmware like lsi cards, if you change it's primary bus number,
> the device will stop working, but that is another problem.
>

Are these settings in the binary driver? I do not see that much need
for a driver to use the geographical addressing after the BAR's have
been set. I thus wondered if it is feasable to hide the geographical
addressing from the driver and offer an API for it from the PCIe layer
to the drivers...

Just a thought.

Best regards,

Ruud

Re: PCIe bus (re-)numbering

[Yinghai Lu]

On Sun, Sep 20, 2015 at 2:17 AM, Ruud <netwerkforens@gmail.com> wrote:
>
> The current procedure I follow is to boot with two PCIe switches in the host.
> (one at the root complex level, intel based, one level above PLX
> based, and the whole tree in the chassis).
>
> - I turn off the chassis (as it conflicts with the BIOS :( )
> - Reboot into linux.
> - remove the intel based switch (has no relevant childs) (echo 1
>>.../remove  sorry for the missing numbers its weekend)
> - turn on chassis
> - rescan starting at the root complex  (echo 1 > .../rescan )
>
> During the rescan, it will map in the original busnumber-range which
> is too small. I understand from your email that by clearing the
> busnumber range in the switch (perhaps both host switces), the kernel
> will pick a different range which is not clamped in by the other
> busnumbers of surrounding other switches?

Yes.

Only need to clear root port.

here the scripts that I used to test busn_alloc and other mmio
resource allocation.
The system could have 8 peer pci root buses.

#
# for x4-4, x4-8, x5-4, x5-8

BUSES='00 20 40 60 80 a0 c0 e0'
DEV_FUNCS='02.0 03.2'

echo "Remove all child devices at first"
for BUS in $BUSES; do
    for DEV_FUNC in $DEV_FUNCS; do
      NAME=0000:"$BUS":"$DEV_FUNC"
      LINE=`/sbin/lspci -nn -s $NAME | wc -l`
      if [ $LINE -eq 0 ]; then
        continue
      fi
      echo $NAME
      NA=`find /sys/devices/pci0000:"$BUS"/"$NAME"/*/remove -name "remove"`
      for N in $NA; do
        echo $N
        echo -n 1 > "$N"
        sleep 1s
      done
    done
done

sleep 5s

echo "Clear bridge mmio BARs and busn"
for BUS in $BUSES; do
    for DEV_FUNC in $DEV_FUNCS; do
      NAME=0000:"$BUS":"$DEV_FUNC"
      LINE=`/sbin/lspci -nn -s $NAME | wc -l`
      if [ $LINE -eq 0 ]; then
        continue
      fi
      echo $NAME
      /sbin/setpci -s $NAME 0x20.l=0
      /sbin/setpci -s $NAME 0x24.l=0
      /sbin/setpci -s $NAME 0x28.l=0
      /sbin/setpci -s $NAME 0x2c.l=0
      /sbin/setpci -s $NAME 0x18.w=0
      /sbin/setpci -s $NAME 0x1a.b=0
      N=`find /sys/devices/pci0000:"$BUS"/"$NAME"/remove -name "remove"`
      echo $N
      echo -n 1 > "$N"
      sleep 1s
    done
done


>
> I will test next monday.

Good. Please check current upstream and my tree for-pci-v4.3-rc1 branch.

>
> What I did get to work is the following procedure:
>> - I turn off the chassis (as it conflicts with the BIOS :(  )
> - Boot linux with parameter pci=assign-busses (BIOS will have
> configured the switches in the host without a serious busnumber range)
> - remove the intel based switch (has no relevant childs) (echo 1
>>.../remove  sorry for the missing numbers its weekend)
> - turn on chassis
> - rescan starting at the root complex  (echo 1 > .../rescan )
> During rescan the numbering is messed up, and dmesg fills up with
> ethernet renaming "errors", didn;t dare to look at other side-effects.

assign-busses may be too destructive.  May make some card firmware
not happy.

>
>>
>>>
>>
>> Do you mean changing bus number without unloading driver ?
>>
>> No, you can not do that.
>>
>> some device firmware like lsi cards, if you change it's primary bus number,
>> the device will stop working, but that is another problem.
>>
>
> Are these settings in the binary driver? I do not see that much need
> for a driver to use the geographical addressing after the BAR's have
> been set. I thus wondered if it is feasable to hide the geographical
> addressing from the driver and offer an API for it from the PCIe layer
> to the drivers...

Card firmware.  Assume those card firmware would trap pci conf read cycle
and compare something inside.
The only workaround that I found is reset the link to make firmware rebooting.
but that will have problem if you are use the disk as root etc.

Thanks

Yinghai

Re: PCIe bus (re-)numbering

[Ruud]

>       /sbin/setpci -s $NAME 0x1a.b=0
>       N=`find /sys/devices/pci0000:"$BUS"/"$NAME"/remove -name "remove"`
>       echo $N
>       echo -n 1 > "$N"
>       sleep 1s
>     done
> done
>

Thanks for the script!

>
>>
>> I will test next monday.
>
> Good. Please check current upstream and my tree for-pci-v4.3-rc1 branch.


Sorry for the rookie question: which version do you mean by "current
upstream", the current main trunk rc?.
I am currently using for-pci-v4.3-rc1 for testing.

>> Are these settings in the binary driver? I do not see that much need
>> for a driver to use the geographical addressing after the BAR's have
>> been set. I thus wondered if it is feasable to hide the geographical
>> addressing from the driver and offer an API for it from the PCIe layer
>> to the drivers...
>
> Card firmware.  Assume those card firmware would trap pci conf read cycle
> and compare something inside.
> The only workaround that I found is reset the link to make firmware rebooting.
> but that will have problem if you are use the disk as root etc.
>

I do not completely understand the remark you make, are you suggesting
the firmware running on the PCIe card (embedded in a CPU on the card
itself) is acting on the content of the busnumber or do you mean x86
extension rom code acting on the busnumber.

Although it is probably irrelevant for the problem I like to solve, I
still like to understand all details.


> Thanks
>
> Yinghai

BR,

Ruud

Re: PCIe bus (re-)numbering

[Ruud]

2015-09-21 9:49 GMT+02:00 Ruud <netwerkforens@gmail.com>:
>>       /sbin/setpci -s $NAME 0x1a.b=0
>>       N=`find /sys/devices/pci0000:"$BUS"/"$NAME"/remove -name "remove"`
>>       echo $N
>>       echo -n 1 > "$N"
>>       sleep 1s
>>     done
>> done
>>
>
> Thanks for the script!
>
>>
>>>
>>> I will test next monday.
>>


Test result based upon

http://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
7e3fad2 (for-pci-v4.3-rc1)

Unfortunately your script hangs the system, I modified it for my
situation (only one root complex, 1 PCI switch will be affected).
 It seems to aggressive on the BAR's..tried to troubleshoot but don't
understand the details on

      /sbin/setpci -s $NAME 0x20.l=0
      /sbin/setpci -s $NAME 0x24.l=0
      /sbin/setpci -s $NAME 0x28.l=0
      /sbin/setpci -s $NAME 0x2c.l=0

why you zero these (besides the bus numbers).

I got the full system working by just zeroing the busnumbers on the
switch at root complex level, remove the switch and rescan it. In this
procedure I need to take care of the some physical aspects: I can not
turn on the chassis when the PCIe switch is removed, some interrupt
comes through and will result in a kernel panic.

Thus the procedure that works is:
1) Chassis off
2) Boot linux
3) Chassis on
4) setpci busnrs to 0
5) remove switch
6) rescan

Cards recognised and functional, memory regions look ok, running out
of iospace but not much can be done about that.

Same procedure on 3.2.0 kernel (debian stock)  results in all cards
being recognized (enum ok) but fails to assign BAR's (tries to fit
them below 4G)


Best regards,

Ruud

Re: PCIe bus (re-)numbering

[Yinghai Lu]

On Mon, Sep 21, 2015 at 12:49 AM, Ruud <netwerkforens@gmail.com> wrote:
> Sorry for the rookie question: which version do you mean by "current
> upstream", the current main trunk rc?.

current linus tree like v4.2 or v4.3-rc1 etc

Re: PCIe bus (re-)numbering

[Yinghai Lu]

On Mon, Sep 21, 2015 at 7:06 AM, Ruud <netwerkforens@gmail.com> wrote:
> 2015-09-21 9:49 GMT+02:00 Ruud <netwerkforens@gmail.com>:

> Test result based upon
>
> http://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
> 7e3fad2 (for-pci-v4.3-rc1)
>
> Unfortunately your script hangs the system, I modified it for my
> situation (only one root complex, 1 PCI switch will be affected).
>  It seems to aggressive on the BAR's..tried to troubleshoot but don't
> understand the details on
>
>       /sbin/setpci -s $NAME 0x20.l=0
>       /sbin/setpci -s $NAME 0x24.l=0
>       /sbin/setpci -s $NAME 0x28.l=0
>       /sbin/setpci -s $NAME 0x2c.l=0
>
> why you zero these (besides the bus numbers).

so the realloc will start from clean.

>
> I got the full system working by just zeroing the busnumbers on the
> switch at root complex level, remove the switch and rescan it.

Good.

> In this
> procedure I need to take care of the some physical aspects: I can not
> turn on the chassis when the PCIe switch is removed, some interrupt
> comes through and will result in a kernel panic.

?

>
> Thus the procedure that works is:
> 1) Chassis off
> 2) Boot linux
> 3) Chassis on
> 4) setpci busnrs to 0
> 5) remove switch
> 6) rescan

4, 5 is reversed?

>
> Cards recognised and functional, memory regions look ok, running out
> of iospace but not much can be done about that.
>
> Same procedure on 3.2.0 kernel (debian stock)  results in all cards
> being recognized (enum ok) but fails to assign BAR's (tries to fit
> them below 4G)

Could be resource allocation problem.

Thanks

Yinghai

Re: PCIe bus (re-)numbering

[Ruud]

>>
>> Thus the procedure that works is:
>> 1) Chassis off
>> 2) Boot linux
>> 3) Chassis on
>> 4) setpci busnrs to 0
>> 5) remove switch
>> 6) rescan
>
> 4, 5 is reversed?

I can not setpci on a removed device, afaik.... for that reason I
reset the busses before removing the switch (not physically remove but
echo 1 >../remove).

For the kernel crash issue I will try to explain better, and try to
get a kernel crash dump to post some logging. It seems that the switch
will still fire hotplug events even after being removed (see lspci -tv
below). I zero-ed the busnumber for bus 04 (02.0-[04-17])  without
removing the underlying bus 05 device 00.0 and switches below.

(for illustration, lspci -tv from a 3.2 kernel, hand edited as the
original picture has already discovered the subordinate busses)
            +-02.0-[04-17]----00.0-[05-17]--+-04.0-[06-16]----00.0-[07-16]---
             |                               |
      \-08.0-[11-16]----

Sorry for the delay, but I didn't manage to create a crashdump yet :(
and the BMC is not cooperative in getting the serial console kernel
working either.
Will not go into details on my attempts for the kernel crash logging
to work as it is out of scope for the discussion..

Bet regards, Ruud

Re: PCIe bus (re-)numbering

[Yinghai Lu]

On Tue, Sep 29, 2015 at 7:04 AM, Ruud <netwerkforens@gmail.com> wrote:
>
> (for illustration, lspci -tv from a 3.2 kernel, hand edited as the
> original picture has already discovered the subordinate busses)
>             +-02.0-[04-17]----00.0-[05-17]--+-04.0-[06-16]----00.0-[07-16]---
>              |                               |
>       \-08.0-[11-16]----
>

please try to post lspci -tv and whole boot log.

Yinghai