32-bit DMA limit for devices (and drivers)

[Andre Przywara <andre.przywara@arm.com>]

On Sat, 1 May 2021 14:23:32 +0200 (CEST)
Mark Kettenis <mark.kettenis@xs4all.nl> wrote:

Hi,

> > From: Bin Meng <bmeng.cn@gmail.com>
> > Date: Sat, 1 May 2021 19:45:02 +0800
> > 
> > On Fri, Apr 30, 2021 at 7:22 PM Andre Przywara <andre.przywara@arm.com> wrote:  
> > >
> > > Hi,
> > >
> > > We now see the first Allwinner devices [1] having DRAM located above
> > > 4GB in address space (4GB DRAM starting at 1GB). After one fix[2]
> > > this works somewhat fine, but the sun8i-emac network device is still
> > > limited to 32-bit DMA addresses. With U-Boot relocating itself (plus
> > > stack and heap) to the end of DRAM, it now runs completely beyond 4GB
> > > on those machines, so not giving pure 32-bit addresses for buffers
> > > anymore.
> > > In Linux we handle this easily by just keeping the default DMA
> > > mask at 32 bits, and letting the DMA framework deal with the nasty
> > > details.
> > >
> > > I was wondering how this should be handled in U-Boot? The straight
> > > forward solution would be:
> > > - Let the driver allocate the RX and TX buffers separately, placing them
> > >   below 4GB in the address space (using lmb_reserve(), I guess?)
> > > - Use those RX buffers and hand the addresses back to the upper layers.
> > > - We already copy TX packets, so this would also be covered, in this
> > >   situation. Other drivers might need to introduce copying.
> > >
> > > This sounds like a common problem, so I was wondering if there is a
> > > more generic solution to this? Maybe there are already platforms or
> > > devices affected? Or should the whole heap and stack be moved below 4GB
> > > (if this is easily possible)?  
> > 
> > My understanding is that the relocated address of U-Boot should be
> > below 4GB then there is no problem for the 32-bit DMA. I thought this
> > is a rule to be followed by every board, but this is not the case on
> > your board?

Bin, interesting, where is this coming from? Was this originally for
32-bit CPUs with some address extension (PAE/LPAE)? I think on *sane*
64-bit systems there would be no need for this restriction, except maybe
for this 32-bit DMA limitation (which is more of a device problem).

> Yes, that was my impression as well.  And I think that would work fine
> on this board as there is plenty of DRAM below 4GB.  And this can be
> achieved by implementing the board_get_usable_ram_top() function.

Ah, I think this is the thing I missed and was looking for:
So we *can* restrict everything *U-Boot* to 32 bits and save us a lot of
hassle.

Thanks for that hint!
 
> As I indicated in my reply, some care is needed in the EFI subsystem,
> but there already is a solution for that.  There is
> CONFIG_EFI_LOADER_BOUNCE_BUFFER, but that might not actually be needed
> in this case.  By default the EFI subsystem will mark all conventional
> memory above "ram_top" as EFI_BOOT_SERVICES_DATA.  So EFI applications
> uch as OS loaders will not allocate that memory until they've called
> ExitBootServices() at which point U-Boot will be completely out of the
> picture.

Oh nice, this looks like what I need. So EFI apps would never use this
memory for I/O buffers.

So I gave this a try and this solves my problem quite neatly: Linux
sees the full DRAM, but U-Boot never touches anything beyond 4GB.
Briefly tested Linux with both EFI and booti.
Will include the board_get_usable_ram_top() implementation in the v2 of
my 4GB enablement patch.

Thanks again!

Cheers,
Andre

> 
> > > In our case we make the buffers part of our priv struct, so should
> > > there be an option to let the priv_auto allocation come from below 4GB?
> > >
> > > Grateful for any input on this!  
> > 
> > Regards,
> > Bin
> >