32-bit DMA limit for devices (and drivers)

[Mark Kettenis <mark.kettenis@xs4all.nl>]

> From: Bin Meng <bmeng.cn@gmail.com>
> Date: Sat, 1 May 2021 19:45:02 +0800
> 
> On Fri, Apr 30, 2021 at 7:22 PM Andre Przywara <andre.przywara@arm.com> wrote:
> >
> > Hi,
> >
> > We now see the first Allwinner devices [1] having DRAM located above
> > 4GB in address space (4GB DRAM starting at 1GB). After one fix[2]
> > this works somewhat fine, but the sun8i-emac network device is still
> > limited to 32-bit DMA addresses. With U-Boot relocating itself (plus
> > stack and heap) to the end of DRAM, it now runs completely beyond 4GB
> > on those machines, so not giving pure 32-bit addresses for buffers
> > anymore.
> > In Linux we handle this easily by just keeping the default DMA
> > mask at 32 bits, and letting the DMA framework deal with the nasty
> > details.
> >
> > I was wondering how this should be handled in U-Boot? The straight
> > forward solution would be:
> > - Let the driver allocate the RX and TX buffers separately, placing them
> >   below 4GB in the address space (using lmb_reserve(), I guess?)
> > - Use those RX buffers and hand the addresses back to the upper layers.
> > - We already copy TX packets, so this would also be covered, in this
> >   situation. Other drivers might need to introduce copying.
> >
> > This sounds like a common problem, so I was wondering if there is a
> > more generic solution to this? Maybe there are already platforms or
> > devices affected? Or should the whole heap and stack be moved below 4GB
> > (if this is easily possible)?
> 
> My understanding is that the relocated address of U-Boot should be
> below 4GB then there is no problem for the 32-bit DMA. I thought this
> is a rule to be followed by every board, but this is not the case on
> your board?

Yes, that was my impression as well.  And I think that would work fine
on this board as there is plenty of DRAM below 4GB.  And this can be
achieved by implementing the board_get_usable_ram_top() function.

As I indicated in my reply, some care is needed in the EFI subsystem,
but there already is a solution for that.  There is
CONFIG_EFI_LOADER_BOUNCE_BUFFER, but that might not actually be needed
in this case.  By default the EFI subsystem will mark all conventional
memory above "ram_top" as EFI_BOOT_SERVICES_DATA.  So EFI applications
uch as OS loaders will not allocate that memory until they've called
ExitBootServices() at which point U-Boot will be completely out of the
picture.

> > In our case we make the buffers part of our priv struct, so should
> > there be an option to let the priv_auto allocation come from below 4GB?
> >
> > Grateful for any input on this!
> 
> Regards,
> Bin
>