Re: [PATCH 2/2] swiotlb: Add swiotlb=nobounce debug option

[Robin Murphy <robin.murphy@arm.com>]

Hi Geert,

On 07/11/16 15:41, Geert Uytterhoeven wrote:
> Hi Robin,
> 
> On Tue, Nov 1, 2016 at 12:46 PM, Robin Murphy <robin.murphy@arm.com> wrote:
>>>>> To aid debugging and catch devices not supporting DMA to memory outside
>>>>> the 32-bit address space, add a kernel command line option
>>>>> "swiotlb=nobounce", which disables the use of bounce buffers.
>>>>> If specified, trying to map memory that cannot be used with DMA will
>>>>> fail, and a warning will be printed (rate-limited).
>>>>
>>>> This rationale seems questionable - how useful is non-deterministic
>>>> behaviour for debugging really? What you end up with is DMA sometimes
>>>> working or sometimes not depending on whether allocations happen to
>>>> naturally fall below 4GB or not. In my experience, that in itself can be
>>>> a pain in the arse to debug.
>>>
>>> It immediately triggered for me, though:
>>>
>>>     rcar-dmac e7300000.dma-controller: Cannot do DMA to address
>>> 0x000000067a9b7000
>>>     ravb e6800000.ethernet: Cannot do DMA to address 0x000000067aa07780
>>>
>>>> Most of the things you might then do to make things more deterministic
>>>> again (like making the default DMA mask tiny or hacking out all the
>>>> system's 32-bit addressable RAM) are also generally sufficient to make
>>>> DMA fail earlier and make this option moot anyway. What's the specific
>>>> use case motivating this?
>>>
>>> My use case is finding which drivers and DMA engines do not support 64-bit
>>> memory. There's more info in my series "[PATCH/RFC 0/5] arm64: r8a7796: 64-bit
>>> Memory and Ethernet Prototype"
>>> (https://www.mail-archive.com/linux-renesas-soc@vger.kernel.org/msg08393.html)
>>
>> Thanks for the context. I've done very similar things in the past, and
>> my first instinct would be to change the default DMA mask in
>> of_dma_configure() to something which can't reach RAM (e.g. <30 bits),
>> then instrument dma_set_mask() to catch cleverer drivers. That's a
>> straightforward way to get 100% coverage - the problem with simply
>> disabling bounce buffering is that whilst statistically it almost
>> certainly will catch >95% of cases, there will always be some that it
>> won't; if some driver only ever does a single dma_alloc_coherent() early
>> enough that allocations are still fairly deterministic, and always
>> happens to get a 32-bit address on that platform, it's likely to slip
>> through the net.
>>
>> I'm not against the idea of SWIOTLB growing a runtime-disable option,
>> I'm just not sure what situation it's actually the best solution for.
> 
> If I set the DMA mask to a small value, DMA is never used, and SWIOTLB
> always falls back to bounce buffers (and DMAing from the small pool)?

Not quite - I meant setting the default mask to a value small enough
that any attempt to allocate or map within it (whether bounced or
otherwise) cannot possibly succeed. Of course, if you have RAM right
down to address 0 this becomes trickier - I'm not entirely certain that
going to extremes (DMA_BIT_MASK(1), say) wouldn't end up going wrong
somewhere for misleadingly unrelated reasons - but I got the impression
from the mainline DT that RAM on your platform starts at 0x40000000,
hence the 30-bit suggestion.

At that point, you can instantly tell for certain that any driver for
which DMA starts failing is relying on the default mask and definitely
needs looking at for 64-bit support, and anything that doesn't just
needs a simple check of what it's passing to dma_set_*mask*().

Alternatively, the really bulletproof option is to get the firmware to
load the kernel into the >4GB area(s) of RAM (or hack dram_base in
handle_kernel_image() in arch/arm64/kernel/efi-stub.c) and simply remove
any 32-bit-addressable areas from the DT entirely (assuming the EFI
memory map isn't hard-coded). Then SWIOTLB becomes moot as there's
nowhere it can even allocate a usable bounce buffer.

> That's the inverse of what I want to achieve: I want to avoid using the
> bounce feature, to make sure the DMA engine is always used with whatever
> kind of memory.

As I said, it's that "always" that's the problem - there is already a
no-op path through the SWIOTLB code if the buffer happens to lie within
the device's DMA mask to start with, so just making SWIOTLB a no-op does
nothing to reveal anything sufficiently lucky to always hit that path on
current platforms.

Robin.

> 
> Gr{oetje,eeting}s,
> 
>                         Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds
>