RE: [PATCH v9 00/26] PCI: Allow BAR movement during boot and hotplug

[David Laight <David.Laight@ACULAB.COM>]

> Actually, we are hotplugging not only endpoints, but nested PCIe
> switches as well: those are PCIe JBOD chassis (with NVMes and SAS
> drives).

From what I remember of playing with some PCI hot-plug hardware
and cardbus extenders to PCI chassis many years ago bus numbers
were actually a big problem.

A surprising number of io cards contain a bridge - so need a
bus number if hot-plugged.

(In spite of the marketing hype hot-plug basically let you remove
a working card and replace it with an identical one!
Modern drivers and OS are likely to handle the errors from
faulty cards better.)

The initial allocation of bus-numbers could easily spread out
the unused bus numbers.
Doing that for memory resources may have other problems
(You probably don't want to allocate the entire range to the
root hub?)

Are the bus numbers exposed as keys/filename in /sys ?
In which case changing them after boot is problematic.
You'd need a map of virtual bus numbers to physical ones.

As well as your 'suspend/resume' sequence, it should
be possible to send a card-remove/insert sequence to
an idle driver.

There is another case were BARs might need moving.
The PCIe spec doesn't allow very long (200ms?) from
reset removal (which might be close to power on) and
the requirement for endpoints to answer config cycles.
If you have to load a large FPGA from a serial EEPROM
this is actually a real problem.
So if the OS does a full probe of the PCI devices it may
find endpoints (or even bridges) that weren't actually there
when the BIOS (or equivalent) did its earlier probe.
Finding space in the middle of the pci devices for an endpoint
that wants two 1MB BARs is unlikely to suceed!

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)