Deadlock with CMA and CPU hotplug

Deadlock with CMA and CPU hotplug

[Laura Abbott]

Hi,

We've run into a AB/BA deadlock situation involving a driver lock and
the CPU hotplug lock on a 3.10 based kernel. The situation is this:

CPU 0				CPU 1
-----				----
Start CPU hotplug
mutex_lock(&cpu_hotplug.lock)
Run CPU hotplug notifier
				data for driver comes in
				mutex_lock(&driver_lock)
				driver calls dma_alloc_coherent
				alloc_contig_range
				lru_add_drain_all
				get_online_cpus()
				mutex_lock(&cpu_hotplug.lock)

Driver hotplug notifier runs
mutex_lock(&driver_lock)

The driver itself is out of tree right now[1] and we're looking at
ways to rework the driver. The best option for rework right now
though might result in some performance penalties. The size that's
being allocated can't easily be converted to an atomic allocation either
It seems like this might be a limitation of where CMA/
dma_alloc_coherent could potentially be used and make drivers
unnecessarily aware of CPU hotplug locking.

Does this seem like an actual problem that needs to be fixed or
is trying to use CMA in a CPU hotplug notifier path just asking
for trouble?

Thanks,
Laura

[1] For reference, the driver is a version of
https://lkml.org/lkml/2014/10/7/495 although that particular
posted version allocates memory at probe instead of runtime
and probably doesn't have the deadlock.

-- 
Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a 
Linux Foundation Collaborative Project

Re: Deadlock with CMA and CPU hotplug

[Marek Szyprowski]

Hello,

On 2014-10-22 18:57, Laura Abbott wrote:
> We've run into a AB/BA deadlock situation involving a driver lock and
> the CPU hotplug lock on a 3.10 based kernel. The situation is this:
>
> CPU 0                CPU 1
> -----                ----
> Start CPU hotplug
> mutex_lock(&cpu_hotplug.lock)
> Run CPU hotplug notifier
>                 data for driver comes in
>                 mutex_lock(&driver_lock)
>                 driver calls dma_alloc_coherent
>                 alloc_contig_range
>                 lru_add_drain_all
>                 get_online_cpus()
>                 mutex_lock(&cpu_hotplug.lock)
>
> Driver hotplug notifier runs
> mutex_lock(&driver_lock)
>
> The driver itself is out of tree right now[1] and we're looking at
> ways to rework the driver. The best option for rework right now
> though might result in some performance penalties. The size that's
> being allocated can't easily be converted to an atomic allocation either
> It seems like this might be a limitation of where CMA/
> dma_alloc_coherent could potentially be used and make drivers
> unnecessarily aware of CPU hotplug locking.
>
> Does this seem like an actual problem that needs to be fixed or
> is trying to use CMA in a CPU hotplug notifier path just asking
> for trouble?

IMHO doing any allocation without GFP_ATOMIC from a notifier is asking
for problems. I always considered notifiers as callbacks that might be 
called
directly from i.e. interrupts. I don't know much about your code, but 
maybe it
would be possible to move the problematic code from a notifier to a separate
worker or thread?

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland