Re: [PATCH 5/5 v0.6] sched/umcg: add Documentation/userspace-api/umcg.txt

From: Peter Oskolkov <posk@posk.io>
To: Thierry Delisle <tdelisle@uwaterloo.ca>
Cc: Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	linux-api@vger.kernel.org, Paul Turner <pjt@google.com>,
	Ben Segall <bsegall@google.com>, Peter Oskolkov <posk@google.com>,
	Andrei Vagin <avagin@google.com>, Jann Horn <jannh@google.com>
Subject: Re: [PATCH 5/5 v0.6] sched/umcg: add Documentation/userspace-api/umcg.txt
Date: Tue, 12 Oct 2021 09:58:35 -0700	[thread overview]
Message-ID: <CAFTs51X0kQLngHYXD-BxBmm6oRyMk1vy9nzaPu4V46PifO99LQ@mail.gmail.com> (raw)
In-Reply-To: <e162fdea-5323-89d2-49d0-3fe56ba2ec3a@uwaterloo.ca>

On Tue, Oct 12, 2021 at 9:25 AM Thierry Delisle <tdelisle@uwaterloo.ca> wrote:

[...]

> Just to be clear, sys_umcg_wait supports an operation that, when called
> from
> a worker, puts the worker to sleep without triggering block detection
> or context-switching back to the server?

Potentially, yes - when a worker wants to yield (e.g. as part of a
custom UMCG-aware mutex/condvar code), and calls into the userspace
scheduler, it may be faster to skip the server wakeup (e.g. reassign
the server to another sleeping worker and wake this worker). This is
not a supported operation right now, but I see how it could be used to
optimize some things in the future.

Do you have any concerns here?

>
>
>
>  >> With that said, I'm a little confused by the usage of "yields" in that
>  >> example. I would expect workers yielding to behave like kernel threads
>  >> calling sched_yield(), i.e., context switch to the server but also be
>  >> immediately added to the idle_workers_ptr.
>  >
>  > I'm not a fan of arguing about how to name things. If the maintainers
>  > ask me to rename wait/wake to park/unpark, I'll do that.
>
> I understand the sentiment, and I'm perfectly happy with the use of
> wait/wake.
> I was exclusively referring to this one use of "yield" in the
> documentation.

I don't see a big difference here, sorry. We are  mixing levels of
abstraction here again, I think. For example, the higher level
userspace scheduling code will have more nuanced treatment of IDLE
workers; but down at the kernel they are all the same: IDLE worker is
a worker that the userspace can "schedule" by marking it RUNNING,
regardless of whether the worker is "parked", or "woke from a blocking
op", or whatever other semantically different state the worker can be.
For the kernel, they are all the same, idle, not runnable, waiting for
the userspace to explicitly "schedule" them.

Similarly, I don't see a need to semantically distinguish "yield" from
"park" at the kernel level of things; this distinction seems to be a
higher-level abstraction that can be properly expressed in the
userspace, and does not need to be explicitly addressed in the kernel
(to make the code faster and simpler, for example).