Re: [v2 PATCH] crypto: pcrypt - Avoid deadlock by using per-instance padata queues

[Daniel Jordan <daniel.m.jordan@oracle.com>]

On Wed, Nov 20, 2019 at 02:58:27AM +0800, Herbert Xu wrote:
>  /* Replace the internal control structure with a new one. */
> -static void padata_replace(struct padata_instance *pinst,
> -			   struct parallel_data *pd_new)
> +static int padata_replace_one(struct padata_shell *ps)
>  {
> -	struct parallel_data *pd_old = pinst->pd;
> +	struct parallel_data *pd_old = rcu_dereference_protected(ps->pd, 1);
> +	struct parallel_data *pd_new;
>  	int notification_mask = 0;
>  
> -	pinst->flags |= PADATA_RESET;
> -
> -	rcu_assign_pointer(pinst->pd, pd_new);
> +	pd_new = padata_alloc_pd(ps);
> +	if (!pd_new)
> +		return -ENOMEM;
>  
> -	synchronize_rcu();
> +	rcu_assign_pointer(ps->pd, pd_new);
>  
>  	if (!cpumask_equal(pd_old->cpumask.pcpu, pd_new->cpumask.pcpu))
>  		notification_mask |= PADATA_CPU_PARALLEL;
> @@ -510,10 +515,25 @@ static void padata_replace(struct padata_instance *pinst,
>  	if (atomic_dec_and_test(&pd_old->refcnt))
>  		padata_free_pd(pd_old);
>  
> +	return notification_mask;
> +}
> +
> +static void padata_replace(struct padata_instance *pinst)
> +{
> +	int notification_mask = 0;
> +	struct padata_shell *ps;
> +
> +	pinst->flags |= PADATA_RESET;
> +
> +	list_for_each_entry(ps, &pinst->pslist, list)
> +		notification_mask |= padata_replace_one(ps);
> +
> +	synchronize_rcu();
> +
>  	if (notification_mask)
>  		blocking_notifier_call_chain(&pinst->cpumask_change_notifier,
>  					     notification_mask,
> -					     &pd_new->cpumask);
> +					     &pinst->cpumask);
>  
>  	pinst->flags &= ~PADATA_RESET;
>  }

I think it's possible for a task in padata_do_parallel() racing with another in
padata_replace() to use a pd after free.  The synchronize_rcu() comes after the
pd_old->refcnt's are dec'd.

                                          padata_do_parallel()
                                            rcu_dereference_bh(ps->pd)
                                            // doesn't see PADATA_RESET set
padata_replace()
  pinst->flags |= PADATA_RESET
  padata_replace_one()
    rcu_assign_pointer(ps->pd, pd_new)
    atomic_dec_and_test(&pd_old->refcnt)
      padata_free_pd()
                                            atomic_inc(&pd->refcnt) // too late

If I'm not missing something, one way out is adding a list_head to
parallel_data for remembering the old pd's on a local list in padata_replace()
so that this function can loop over it and drop the refs after
synchronize_rcu().  A padata_do_parallel() call will have then had a chance to
take a ref on the pd it's using.


And, not this patch, but with the removal of flushing it seems there's no need
for PADATA_RESET, so it and its EBUSY error can go away.