Re: [PATCH 08/15] PCI: mvebu: Propagate errors when updating PCI_IO_BASE and PCI_MEM_BASE registers

[Bjorn Helgaas <helgaas@kernel.org>]

On Sat, Jan 08, 2022 at 12:46:58AM +0100, Pali Rohár wrote:
> On Friday 07 January 2022 17:16:17 Bjorn Helgaas wrote:
> > On Fri, Jan 07, 2022 at 11:28:26PM +0100, Pali Rohár wrote:
> > > On Friday 07 January 2022 15:55:04 Bjorn Helgaas wrote:
> > > > On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
> > > > > Properly propagate failure from mvebu_pcie_add_windows() function back to
> > > > > the caller mvebu_pci_bridge_emul_base_conf_write() and correctly updates
> > > > > PCI_IO_BASE, PCI_MEM_BASE and PCI_IO_BASE_UPPER16 registers on error.
> > > > > On error set base value higher than limit value which indicates that
> > > > > address range is disabled. 
> > > > 
> > > > Does the spec say that if software programs something invalid,
> > > > hardware should proactively set the base and limit registers to
> > > > disable the window?
> > > 
> > > No. But this patch address something totally different. Software can do
> > > fully valid operation, e.g. try to set forwarding memory window as large
> > > as possible. But because this driver "emulates" pci bridge by calling
> > > software/kernel function (mvebu_pcie_add_windows), some operations which
> > > in real HW cannot happen, are possible in software.
> > > 
> > > For example there are limitations in sizes of forwarding memory windows,
> > > because it is done by mvebu-mbus driver, which is responsible for
> > > configuring mapping and forwarding of PCIe I/O and MEM windows. And due
> > > to Marvell HW, there are restrictions which are not in PCIe HW.
> > > 
> > > Currently if such error happens, obviously kernel is not able to set
> > > PCIe windows and it just print warnings to dmesg. Trying to access these
> > > windows would result in the worst case in crashes.
> > > 
> > > With this change when mvebu_pcie_add_windows() function fails then into
> > > emulated config space is put information that particular forwarding
> > > window is disabled. I think that it is better to indicate it in config
> > > space what is the current "reality" of hardware configuration. If window
> > > is disabled in real-HW (meaning in mvebu-mbus driver) then show it also
> > > in emulated config space of pci bridge.
> > > 
> > > Do you have better idea what should emulated pci bridge do, if software
> > > try to set fully valid configuration of forwarding window, but it is not
> > > possible to achieve it (even compliant PCI bridge must be able to do
> > > it)?
> > 
> > On an ACPI system, the host bridge window sizes are constrained by the
> > host bridge _CRS method.  I assume there's a similar constraint in DT.
> > 
> > Is the fact that mvebu_pcie_add_windows() can fail a symptom of a DT
> > that describes more available space than mvebu-bus can map?
> 
> Memory maps for mvebu are more complicated. There is no explicit size in
> DT ranges property as it is dynamically allocated by mvebu-mbus:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts/armada-385.dtsi?h=v5.15#n47

I wish I knew how to really interpret those "ranges" properties.  (Is
there a good description in Documentation/ somewhere?  All I've found
so far is https://elinux.org/Device_Tree_Usage, which is good, but
doesn't match this example completely.)

I see:

  pciec: pcie {
    ranges = <...>;
    pcie1: pcie@1,0 {
      ranges = <0x82000000 0 0 0x82000000 0x1 0 1 0
	        0x81000000 0 0 0x81000000 0x1 0 1 0>;
    };
    pcie2: pcie@2,0 {
      ranges = <0x82000000 0 0 0x82000000 0x2 0 1 0
	        0x81000000 0 0 0x81000000 0x2 0 1 0>;
    };
    pcie3: pcie@3,0 {
      ranges = <0x82000000 0 0 0x82000000 0x3 0 1 0
	        0x81000000 0 0 0x81000000 0x3 0 1 0>;
    };
    pcie4: pcie@4,0 {
      ranges = <0x82000000 0 0 0x82000000 0x4 0 1 0
	        0x81000000 0 0 0x81000000 0x4 0 1 0>;
    };
  };

What does this look like in dmesg, i.e., what CPU address ranges are
mapped to what PCI bus addresses?

Are pcie1, pcie2, etc Root Ports?  Or are they each separate host
bridges (they each have "bus-range = <0x00 0xff>")?

Is space from pciec dynamically assigned to pcie1, pcie2, etc?  If so,
I assume there are more restrictions on the size and alignment than on
PCI bridge windows, which allow size/alignment down to 1MB?

I'm trying to see how this could be described in ACPI because that's a
fairly general model that accommodates most machines.  Possibly
describing mvebu in ACPI would involve losing some flexibility.

Bjorn