Re: [PATCH -next] ashmem: Fix ashmem_shrink deadlock.

[Andrew Morton <akpm@linux-foundation.org>]

On Wed,  1 May 2013 09:56:13 -0400 Robert Love <rlove@google.com> wrote:

> Don't acquire ashmem_mutex in ashmem_shrink if we've somehow recursed into the
> shrinker code from within ashmem. Just bail out, avoiding a deadlock. This is
> fine, as ashmem cache pruning is advisory anyhow.
> 

Sorry, but I don't think "somehow" is an adequate description of a
kernel bug.  The deadlock should be described with specificity, so that
others can understand and review the fix and perhaps suggest
alternative implementations.

Presumably someone is performing a memory allocation while holding
ashmem_mutex.  A more idiomatic way of avoiding a call to direct
reclaim in these circumstances would be for the task to set its
PF_MEMALLOC flag, or to use GFP_ATOMIC.  But without any details that's
as far as I can go.

> --- a/drivers/staging/android/ashmem.c
> +++ b/drivers/staging/android/ashmem.c
> @@ -363,7 +363,11 @@ static int ashmem_shrink(struct shrinker *s, struct shrink_control *sc)
>  	if (!sc->nr_to_scan)
>  		return lru_count;
>  
> -	mutex_lock(&ashmem_mutex);
> +	/* avoid recursing into this code from within ashmem itself */
> +	if (!mutex_trylock(&ashmem_mutex)) {
> +		return -1;
> +	}

This is rather hacky.  It consumes more CPU than the above approaches,
and more stack.

Worst of all, it obviously hasn't met checkpatch.pl ;)