Are back to back PCIe BAR allocations supported by Linux?

Are back to back PCIe BAR allocations supported by Linux?

[Christian König]

Hello everyone,

we recently had a bug report of a system which works fine when a PCIe 
hotplug device is connected on boot, but fails to initialize if those 
device are disconnected and then reconnected again.

During investigation I've found that Linux isn't able to assign the BARs 
of the device correctly while reconnecting. The problem seems to be that 
the Linux PCI code doesn't seem to use back to back BAR allocations.

Now what's back to back BAR allocation? Let's assume you have two 
devices with a 256MiB BAR and a 2MiB BAR each behind a common upstream 
bridge.

The configuration Linux seems to use is the following:
Device A - 256MiB BAR
Device A -     2MiB BAR
Padding     254MiB
Device B - 256MiB BAR
Device B -     2MIB BAR

With padding this results in at least 770MiB address space requirement 
for the common upstream bridge, with alignment this is probably more 
like 1GiB.

The BIOS on the other hand seems to be capable of configuring the BARs 
like this:

Device A - 256MiB BAR
Device A -     2MiB BAR
Padding     252MiB
Device B -     2MIB BAR
Device B - 256MiB BAR

The result is that you only need 768MiB address space for the upstream 
bridge which then perfectly fits into what is assigned for hotplug here.

Is that already supported by the Linux PCIe code? If yes then how? I've 
tried to read a bit into the BAR allocation code, but it is kind of hard 
to understand.

Regards,
Christian.

Re: Are back to back PCIe BAR allocations supported by Linux?

[Christian König]

Ping?

Am 01.04.21 um 14:03 schrieb Christian König:
> Hello everyone,
>
> we recently had a bug report of a system which works fine when a PCIe 
> hotplug device is connected on boot, but fails to initialize if those 
> device are disconnected and then reconnected again.
>
> During investigation I've found that Linux isn't able to assign the 
> BARs of the device correctly while reconnecting. The problem seems to 
> be that the Linux PCI code doesn't seem to use back to back BAR 
> allocations.
>
> Now what's back to back BAR allocation? Let's assume you have two 
> devices with a 256MiB BAR and a 2MiB BAR each behind a common upstream 
> bridge.
>
> The configuration Linux seems to use is the following:
> Device A - 256MiB BAR
> Device A -     2MiB BAR
> Padding     254MiB
> Device B - 256MiB BAR
> Device B -     2MIB BAR
>
> With padding this results in at least 770MiB address space requirement 
> for the common upstream bridge, with alignment this is probably more 
> like 1GiB.
>
> The BIOS on the other hand seems to be capable of configuring the BARs 
> like this:
>
> Device A - 256MiB BAR
> Device A -     2MiB BAR
> Padding     252MiB
> Device B -     2MIB BAR
> Device B - 256MiB BAR
>
> The result is that you only need 768MiB address space for the upstream 
> bridge which then perfectly fits into what is assigned for hotplug here.
>
> Is that already supported by the Linux PCIe code? If yes then how? 
> I've tried to read a bit into the BAR allocation code, but it is kind 
> of hard to understand.
>
> Regards,
> Christian.

Re: Are back to back PCIe BAR allocations supported by Linux?

[Bjorn Helgaas]

On Thu, Apr 01, 2021 at 02:03:34PM +0200, Christian König wrote:
> Hello everyone,
> 
> we recently had a bug report of a system which works fine when a PCIe
> hotplug device is connected on boot, but fails to initialize if those device
> are disconnected and then reconnected again.
> 
> During investigation I've found that Linux isn't able to assign the BARs of
> the device correctly while reconnecting. The problem seems to be that the
> Linux PCI code doesn't seem to use back to back BAR allocations.
> 
> Now what's back to back BAR allocation? Let's assume you have two devices
> with a 256MiB BAR and a 2MiB BAR each behind a common upstream bridge.
> 
> The configuration Linux seems to use is the following:
> Device A - 256MiB BAR
> Device A -     2MiB BAR
> Padding     254MiB
> Device B - 256MiB BAR
> Device B -     2MIB BAR
> 
> With padding this results in at least 770MiB address space requirement for
> the common upstream bridge, with alignment this is probably more like 1GiB.
> 
> The BIOS on the other hand seems to be capable of configuring the BARs like
> this:
> 
> Device A - 256MiB BAR
> Device A -     2MiB BAR
> Padding     252MiB
> Device B -     2MIB BAR
> Device B - 256MiB BAR
> 
> The result is that you only need 768MiB address space for the upstream
> bridge which then perfectly fits into what is assigned for hotplug here.
> 
> Is that already supported by the Linux PCIe code? If yes then how? I've
> tried to read a bit into the BAR allocation code, but it is kind of hard to
> understand.

I'm sorry that I don't have a ready answer.  The Linux resource
allocation code is indeed very difficult to understand.  I'm quite
sure it is incapable of doing optimal assignments.

If the BIOS has already assigned working values, Linux generally
doesn't change them.  But when you hot-remove and then reconnect a
device, the BIOS assignments are lost, of course, and it's not
surprising that Linux doesn't recreate them exactly.  I certainly
agree that if it worked before, Linux *should* be able to make it work
again.

Bjorn