Re: [PATCH 00/12] sunxi: Devicetree sync from Linux v5.18-rc1

[Andre Przywara <andre.przywara@arm.com>]

On Fri, 29 Apr 2022 21:38:58 -0500
Samuel Holland <samuel@sholland.org> wrote:

Hi Samuel,

> On 4/29/22 7:08 PM, Andre Przywara wrote:
> > On Fri, 29 Apr 2022 14:14:19 -0400
> > Tom Rini <trini@konsulko.com> wrote:
> > 
> > Hi,
> >   
> >> On Fri, Apr 29, 2022 at 06:05:03PM +0200, Mark Kettenis wrote:  
> >>>> Date: Fri, 29 Apr 2022 11:31:00 -0400
> >>>> From: Tom Rini <trini@konsulko.com>
> >>>>
> >>>> On Fri, Apr 29, 2022 at 04:25:51PM +0100, Andre Przywara wrote:    
> >>>>> On Fri, 29 Apr 2022 10:57:10 -0400
> >>>>> Tom Rini <trini@konsulko.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>     
> >>>>>> On Fri, Apr 29, 2022 at 03:51:59PM +0100, Andre Przywara wrote:    
> >>>>>>> On Wed, 27 Apr 2022 15:31:19 -0500
> >>>>>>> Samuel Holland <samuel@sholland.org> wrote:
> >>>>>>>
> >>>>>>> Hi Samuel, Tom,
> >>>>>>>       
> >>>>>>>> This series brings all of our devicetrees up to date with Linux.
> >>>>>>>>
> >>>>>>>> Older SoCs (before A83T) have not been synchronized in over 3 years.
> >>>>>>>> And I don't have any of this hardware to test. But there are not major
> >>>>>>>> changes to those devicetrees either.
> >>>>>>>>
> >>>>>>>> The big motivation for including older SoCs in this update is converting
> >>>>>>>> the USB PHY driver to get its VBUS detection GPIO/regulator from the
> >>>>>>>> devicetree instead of from a pin name in Kconfig. Many older boards had
> >>>>>>>> those properties added or fixed since the last devicetree sync. This PHY
> >>>>>>>> driver change is necessary to complete the DM_GPIO migration.
> >>>>>>>>
> >>>>>>>> A couple of breaking changes were made to several SoCs' devicetrees in
> >>>>>>>> Linux relating to the "r_intc" interrupt controller. New kernels support
> >>>>>>>> old devicetrees, but not the other way around. So to be most compatible
> >>>>>>>> and avoid regressions, those changes are skipped here.      
> >>>>>>>
> >>>>>>> Many thanks for considering this! I just skimmed over the A64 and H6
> >>>>>>> patches, and this is indeed the only difference.
> >>>>>>>
> >>>>>>> But while I love this pragmatic approach, and would be happy to take this,
> >>>>>>> this goes against our own rules, and more importantly against Tom's one's:
> >>>>>>> to take only direct DT file copies from the kernel tree.
> >>>>>>>
> >>>>>>> Tom, can you give your opinion here? As Samuel mentioned above, the
> >>>>>>> current mainline DTs wouldn't boot on older kernels (the changes affect
> >>>>>>> critical devices), so this spoils stable distro and installer kernels,
> >>>>>>> when using $fdtcontroladdr, for instance when booting via UEFI.
> >>>>>>>
> >>>>>>> As a side effect of always defining SYS_SOC to "sunxi", we cannot easily
> >>>>>>> use per-SoC DT overrides using sun50i-a64-u-boot.dtsi, for instance.
> >>>>>>>
> >>>>>>> For context, those changed properties were in the mainline kernel tree at
> >>>>>>> some point, but have been amended since. So it's not some random change.      
> >>>>>>
> >>>>>> So, this is I guess a bit annoying.  But, we aren't at the point where
> >>>>>> the common use case is the downstream OS using the DTB we've loaded and
> >>>>>> are using, are we?  I mean, we can't be, as ours are so far out of date,
> >>>>>> so this will only be an option when we use a recent DT ourself.  So we
> >>>>>> should be able to sync in the changes and update our code, as they can't
> >>>>>> be using $fdtcontroladdr in this case, right?  Or am I missing the use
> >>>>>> case that's in the wild atm?  Thanks!    
> >>>>>
> >>>>> While it sounds like the DTs are wildly out of date, this mostly affects
> >>>>> secondary functionality. The mainline updates for the 64-bit SoCs are:
> >>>>> - H6: adding the VP9 video h/w codec and an additional wakeup timer
> >>>>> - A64: adding GPU DVFS, adding DRAM DVFS, add support for secondary
> >>>>> digital audio interfaces, plus the wakeup timer
> >>>>> Also there are cosmetic changes, like changing node names to make them
> >>>>> binding compliant.  
> 
> The SoCs where the DTs are wildly out of date (v4.18-rc3) are:
> A10 A10s/A13 A31 A20 A80 A23/A33 A83T
> 
> The SoCs with the r_intc binding change are:
> A31 A23/A33 A83T A64 H3/H5 H6
> 
> For the SoCs which are in both lists, yes, it is unlikely that anyone is using
> $fdtcontroladdr for Linux. So we could probably fully update those DTs. That
> leaves just A64, H3/H5, and H6 which would temporarily need to exclude the
> r_intc-related changes.

Yes, I don't really care about those older SoCs, devices using them are
probably not really distro / UEFI material anyway.

> >>>>> So those DT updates are really only important for mobile devices like the
> >>>>> Pinephone, which probably don't use UEFI booting.  
> 
> We would really like to use UEFI booting on the PinePhone, and the out-of-date
> devicetree is one thing blocking that. We need to use $fdtcontroladdr to pick up
> the CPU idle states that are added at runtime by TF-A.

Yes, I was wondering about that. I could imagine that suspend/resume is
a killer feature for the PinePhone. It probably sounds useful to fully
update just the Pinephone .dts, giving up compatibility for older
kernels. IIUC the PinePhone doesn't run normal "desktop" distros, but
relies more on custom OSes, tailored to a Phone use case in general?
What kernels are those OSes using?
The only caveat would be that this adds to the mess and increases the
diff to mainline, but maybe this could be solved by a
sun50i-a64-pinephone-u-boot.dtsi?

> >>>>> At the moment I boot distro grubs and installers just fine, and without
> >>>>> losing any real functionality (minus suspend/resume, maybe). The
> >>>>> out-of-the-box default boot works now, and would break when pulling in the
> >>>>> pure mainline DTs. Plus FreeBSD (which relies more heavily on UEFI, IIUC),
> >>>>> can only deal with the older DTs (#interrupt-cells for r_intc must be 2).  
> 
> FreeBSD already supports the new binding for forwarding interrupts (everything
> except NMI). Any version with this change should boot fine:
> 
> https://cgit.freebsd.org/src/commit/?id=993e8236c30a

Ah, sorry, my bad, I was looking at a stale repo (they stopped updating
the original master branch and switched to main, for technical reasons).

> >>>> I guess the first point is, yes, we should sync in what we can sync in,
> >>>> to bring things closer to proper alignment.  I further guess that given
> >>>> that we have to support both "new Linux" and "not Linux", we have to
> >>>> keep the old style DT information instead as that's how compatibility is
> >>>> supposed to be handled?  I'm adding in Rob here since this still reads a
> >>>> bit confusing as to what's supposed to happen, but maybe we also just
> >>>> need to check in with some other-OS folks to see what their plan is?    
> >>>
> >>> My goal with OpenBSD has always been to make the OS boot with the DT
> >>> built into U-Boot, but to allow users to use a more up-to-date Linux
> >>> DT by putting the apropriate .dtb file on the ESP.  However it is easy
> >>> to miss changes that break backwards compatibility of the bindings in
> >>> the noise of other changes.  So in many cases we only notice this when
> >>> the changes make it into U-Boot and we update the OpenBSD U-Boot port.
> >>>
> >>> I'll drag out one of my A64 boards and see what needs to be done to
> >>> support the routing of these interrupts through r_intc.  
> > 
> > In FreeBSD the change would be fairly small, I think: just ignoring the
> > first parameter of an r_intc interrupt specifier when it advertises
> > #interrupt-cells = <3>.  
> 
> See above, FreeBSD already supports this.
> 
> > In OpenBSD I don't find the allwinner,sun6i-a31-r-intc (or any other
> > intc related) compatible string at all, and so far we just lose the NMI
> > from the PMIC. But this would radically change with the new DT: now the
> > two PIOs and the RTC are routed through that IRQ controller, so they
> > would probably fail probing.
> >   
> >> So, does that mean the plan is to keep the r_intc changes out of U-Boot
> >> for now, but we can sync the rest, and come up with a plan to fully
> >> update in time?  
> > 
> > That's one possible solution, yes, and so far the easiest, it provides
> > a good balance between features and compatibility.  
> 
> This was my understanding of the plan as well.
> 
> > Theoretically we can never fully sync, unless we decide to no longer
> > support those older OSes (older Linux kernels and (current) *BSD).  
> 
> Do we have any guidance for when this could be? After the n+1 LTS kernel/BSD
> release? After the distro/BSD installers update their kernels?

More the latter, I'd say when major distros stop shipping those
old kernels in relevant releases. Especially Debian is one to keep an
eye on I guess, since they are on 5.10 *currently*, and their installer
properly stays there for a while. Ubuntu 20.04 shipped with 5.4, and I'd
like to support that say at least one more year still. Don't really
keep track of the kernels in other distros, but I think these two are
among the more conservative ones.

Samuel, since I have you here: With your new hat Linux hat on, can you
say whether incompatible DT changes won't happen in the future anymore?
From experience I'd say there are ways to avoid them, though possibly at
some cost (less clean DT, or deviating from some DT rules).

> > One thing we could explore is patching the DT at runtime, but U-Boot
> > cannot know if the OS supports the new style or not, so it has to be
> > manually triggered.  
> 
> Right, automatically handling this is not really feasible. Users that need the
> r_intc changes for suspend/resume will have to load a DTB or overlay from disk.
> (We could possibly build in such an overlay, and load it based on some
> environment variable, but this seems like little benefit when most users load a
> DTB from disk anyway.)

But loading from disk would lose any manipulation that previous
firmware did, for instance TF-A. Plus I think reserved memory is not
properly propagated, at least last time I checked. Also the DT would
really need to be loaded by U-Boot, loading it via grub would lose even
more manipulations like the DRAM size and MAC address.

So I believe an overlay is the way to go. I have a patch sitting here
that applies all .dtbo files found in a directory on some block device
(e.g. "fdt apply_all mmc 0:1 overlays/"), would that help?

Cheers,
Andre