Re: + kthread-fix-kthread_mod_delayed_work-vs-kthread_cancel_delayed_work_sync-race.patch added to -mm tree

Re: + kthread-fix-kthread_mod_delayed_work-vs-kthread_cancel_delayed_work_sync-race.patch added to -mm tree

[Oleg Nesterov]

On 05/20, Andrew Morton wrote:
>
> --- a/kernel/kthread.c~kthread-fix-kthread_mod_delayed_work-vs-kthread_cancel_delayed_work_sync-race
> +++ a/kernel/kthread.c
> @@ -1181,6 +1181,19 @@ bool kthread_mod_delayed_work(struct kth
>  		goto out;
>
>  	ret = __kthread_cancel_work(work, true, &flags);
> +
> +	/*
> +	 * Canceling could run in parallel from kthread_cancel_delayed_work_sync
> +	 * and change work's canceling count as the spinlock is released and regain
> +	 * in __kthread_cancel_work so we need to check the count again. Otherwise,
> +	 * we might incorrectly queue the dwork and further cause
> +	 * cancel_delayed_work_sync thread waiting for flush dwork endlessly.
> +	 */
> +	if (work->canceling) {
> +		ret = false;
> +		goto out;
> +	}
> +
>  fast_queue:
>  	__kthread_queue_delayed_work(worker, dwork, delay);

Never looked at this code before, can't review...

but note that another caller of __kthread_queue_delayed_work() needs to
check work->canceling too. So perhaps we should simply add queuing_blocked()
into __kthread_queue_delayed_work() ?

Something like below, uncompiled/untested, most probably incorrect.

Either way, this comment

	 * Return: %true if @dwork was pending and its timer was modified,
	 * %false otherwise.

above kthread_mod_delayed_work looks obviously wrong. Currently it returns
true if this work was pending. With your patch it returns true if it was
pending and not canceling.

With the patch below it returns true if the work was (re)queued successfully,
and this makes more sense to me. But again, I can easily misread this code.

In any case, even if my patch is correct, I won't insist, your fix is
much simpler.

Oleg.

--- x/kernel/kthread.c
+++ x/kernel/kthread.c
@@ -977,7 +977,7 @@ void kthread_delayed_work_timer_fn(struc
 }
 EXPORT_SYMBOL(kthread_delayed_work_timer_fn);
 
-static void __kthread_queue_delayed_work(struct kthread_worker *worker,
+static bool __kthread_queue_delayed_work(struct kthread_worker *worker,
 					 struct kthread_delayed_work *dwork,
 					 unsigned long delay)
 {
@@ -987,6 +987,9 @@ static void __kthread_queue_delayed_work
 	WARN_ON_FUNCTION_MISMATCH(timer->function,
 				  kthread_delayed_work_timer_fn);
 
+	if (queuing_blocked(worker, work))
+		return false;
+
 	/*
 	 * If @delay is 0, queue @dwork->work immediately.  This is for
 	 * both optimization and correctness.  The earliest @timer can
@@ -995,7 +998,7 @@ static void __kthread_queue_delayed_work
 	 */
 	if (!delay) {
 		kthread_insert_work(worker, work, &worker->work_list);
-		return;
+		return true;
 	}
 
 	/* Be paranoid and try to detect possible races already now. */
@@ -1005,6 +1008,7 @@ static void __kthread_queue_delayed_work
 	work->worker = worker;
 	timer->expires = jiffies + delay;
 	add_timer(timer);
+	return true;
 }
 
 /**
@@ -1028,16 +1032,12 @@ bool kthread_queue_delayed_work(struct k
 {
 	struct kthread_work *work = &dwork->work;
 	unsigned long flags;
-	bool ret = false;
+	bool ret;
 
 	raw_spin_lock_irqsave(&worker->lock, flags);
-
-	if (!queuing_blocked(worker, work)) {
-		__kthread_queue_delayed_work(worker, dwork, delay);
-		ret = true;
-	}
-
+	ret = __kthread_queue_delayed_work(worker, dwork, delay);
 	raw_spin_unlock_irqrestore(&worker->lock, flags);
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(kthread_queue_delayed_work);
@@ -1180,9 +1180,9 @@ bool kthread_mod_delayed_work(struct kth
 	if (work->canceling)
 		goto out;
 
-	ret = __kthread_cancel_work(work, true, &flags);
+	__kthread_cancel_work(work, true, &flags);
 fast_queue:
-	__kthread_queue_delayed_work(worker, dwork, delay);
+	ret = __kthread_queue_delayed_work(worker, dwork, delay);
 out:
 	raw_spin_unlock_irqrestore(&worker->lock, flags);
 	return ret;

Re: + kthread-fix-kthread_mod_delayed_work-vs-kthread_cancel_delayed_work_sync-race.patch added to -mm tree

[Petr Mladek]

On Fri 2021-05-21 18:35:27, Oleg Nesterov wrote:
> On 05/20, Andrew Morton wrote:
> >
> > --- a/kernel/kthread.c~kthread-fix-kthread_mod_delayed_work-vs-kthread_cancel_delayed_work_sync-race
> > +++ a/kernel/kthread.c
> > @@ -1181,6 +1181,19 @@ bool kthread_mod_delayed_work(struct kth
> >  		goto out;
> >
> >  	ret = __kthread_cancel_work(work, true, &flags);
> > +
> > +	/*
> > +	 * Canceling could run in parallel from kthread_cancel_delayed_work_sync
> > +	 * and change work's canceling count as the spinlock is released and regain
> > +	 * in __kthread_cancel_work so we need to check the count again. Otherwise,
> > +	 * we might incorrectly queue the dwork and further cause
> > +	 * cancel_delayed_work_sync thread waiting for flush dwork endlessly.
> > +	 */
> > +	if (work->canceling) {
> > +		ret = false;
> > +		goto out;
> > +	}
> > +
> >  fast_queue:
> >  	__kthread_queue_delayed_work(worker, dwork, delay);
> 
> Never looked at this code before, can't review...
> 
> but note that another caller of __kthread_queue_delayed_work() needs to
> check work->canceling too. So perhaps we should simply add queuing_blocked()
> into __kthread_queue_delayed_work() ?

Good point. I do not have strong opinion. But if we move the check
to __kthread_queue_delayed_work() than it would make sense to
move it also into kthread_insert_work() to keep it symmetric.
But then we would do the check twice in some code paths.
Well, it would make the API more safe.


> Something like below, uncompiled/untested, most probably incorrect.
> 
> Either way, this comment
> 
> 	 * Return: %true if @dwork was pending and its timer was modified,
> 	 * %false otherwise.
> 
> above kthread_mod_delayed_work looks obviously wrong. Currently it returns
> true if this work was pending. With your patch it returns true if it was
> pending and not canceling.
>
> With the patch below it returns true if the work was (re)queued successfully,
> and this makes more sense to me. But again, I can easily misread this code.

Your patch changes the semantic. The current semantic is the same for
the workqueue's counter-part mod_delayed_work_on().

It look's weird by it makes sense.

kthread_mod_delayed_work() should always succeed and queue the work
with the new delay. Normally, the only interesting information is
whether the work was canceled (queued but not proceed). It means
that some job was not done.

The only situation when kthread_mod_delayed_work() is not able to
queue the work is when another process is canceling the work at
the same time. But it means that kthread_mod_delayed_work()
and kthread_cancel_delayed_work_sync() are called in parallel.
The result is racy by definition. It means that the code is racy.
And it typically means that the API is used a wrong way.
Note the comment:

 * A special case is when the work is being canceled in parallel.
 * It might be caused either by the real kthread_cancel_delayed_work_sync()
 * or yet another kthread_mod_delayed_work() call. We let the other command
 * win and return %false here. The caller is supposed to synchronize these
 * operations a reasonable way.


But you have a point. The new code returns "false" even when the work
was canceled. It means that the previously queue work was not
proceed.

We should actually keep the "ret" value as is to stay compatible with
workqueue API:

	/*
	 * Canceling could run in parallel from kthread_cancel_delayed_work_sync
	 * and change work's canceling count as the spinlock is released and regain
	 * in __kthread_cancel_work so we need to check the count again. Otherwise,
	 * we might incorrectly queue the dwork and further cause
	 * cancel_delayed_work_sync thread waiting for flush dwork endlessly.
	 *
	 * Keep the ret code. The API primary informs the caller
	 * whether some pending work has been canceled (not proceed).
	 */
	if (work->canceling)
		goto out;

Best Regards,
Petr

Re: + kthread-fix-kthread_mod_delayed_work-vs-kthread_cancel_delayed_work_sync-race.patch added to -mm tree

[Oleg Nesterov]

On 05/24, Petr Mladek wrote:
>
> Your patch changes the semantic. The current semantic is the same for
> the workqueue's counter-part mod_delayed_work_on().

OK, I see, thanks. I was confused by the comment.

> We should actually keep the "ret" value as is to stay compatible with
> workqueue API:
>
> 	/*
> 	 * Canceling could run in parallel from kthread_cancel_delayed_work_sync
> 	 * and change work's canceling count as the spinlock is released and regain
> 	 * in __kthread_cancel_work so we need to check the count again. Otherwise,
> 	 * we might incorrectly queue the dwork and further cause
> 	 * cancel_delayed_work_sync thread waiting for flush dwork endlessly.
> 	 *
> 	 * Keep the ret code. The API primary informs the caller
> 	 * whether some pending work has been canceled (not proceed).
> 	 */
> 	if (work->canceling)
> 		goto out;

Agreed, we should keep the "ret" value.

but unless I am confused again this doesn't match mod_delayed_work_on()
which always returns true if it races with cancel(). Nevermind, I think
this doesn't matter.

Thanks,

Oleg.

Re: + kthread-fix-kthread_mod_delayed_work-vs-kthread_cancel_delayed_work_sync-race.patch added to -mm tree

[Petr Mladek]

Added Tejun into CC because of the workqueue API related question
at the end of the mail.

On Wed 2021-05-26 19:06:06, Oleg Nesterov wrote:
> On 05/24, Petr Mladek wrote:
> >
> > Your patch changes the semantic. The current semantic is the same for
> > the workqueue's counter-part mod_delayed_work_on().
> 
> OK, I see, thanks. I was confused by the comment.
> 
> > We should actually keep the "ret" value as is to stay compatible with
> > workqueue API:
> >
> > 	/*
> > 	 * Canceling could run in parallel from kthread_cancel_delayed_work_sync
> > 	 * and change work's canceling count as the spinlock is released and regain
> > 	 * in __kthread_cancel_work so we need to check the count again. Otherwise,
> > 	 * we might incorrectly queue the dwork and further cause
> > 	 * cancel_delayed_work_sync thread waiting for flush dwork endlessly.
> > 	 *
> > 	 * Keep the ret code. The API primary informs the caller
> > 	 * whether some pending work has been canceled (not proceed).
> > 	 */
> > 	if (work->canceling)
> > 		goto out;
> 
> Agreed, we should keep the "ret" value.

Martin Liu, could you please resend the patch without the "ret =
false" line? See above.

Andrew, could you please remove this patch from the -mm tree for now?

> but unless I am confused again this doesn't match mod_delayed_work_on()
> which always returns true if it races with cancel(). Nevermind, I think
> this doesn't matter.

Good point. I think that it is actually a bug. Most callers ignore
the return code but there is the following user:

static void addrconf_del_dad_work(struct inet6_ifaddr *ifp)
{
	if (cancel_delayed_work(&ifp->dad_work))
		__in6_ifa_put(ifp);
}
static void addrconf_mod_dad_work(struct inet6_ifaddr *ifp,
				   unsigned long delay)
{
	in6_ifa_hold(ifp);
	if (mod_delayed_work(addrconf_wq, &ifp->dad_work, delay))
		in6_ifa_put(ifp);
}

If mod_delayed_work() races with cancel_delayed_work() then both might
return true and call in6_ifa_put(ifp).

I thought that they were serialized by ifp->lock. But, for example,
addrconf_dad_start() calls addrconf_mod_dad_work() after releasing
this lock.

It is possible that they are serialized another way. But I think that
in principle only the one that really cancelled a pending work
should return "true".

Tejun, any opinion?  Feel free to ask for more context.

Best Regards,
Petr