Re: Memory coherency issue with IO thread offloading?

["Nicholas Piggin" <npiggin@gmail.com>]

On Sat Mar 25, 2023 at 11:20 AM AEST, Jens Axboe wrote:
> On 3/24/23 7:15?PM, Jens Axboe wrote:
> >> Are there any CONFIG options I'd need to trip this?
> > 
> > I don't think you need any special CONFIG options. I'll attach my config
> > here, and I know the default distro one hits it too. But perhaps the
> > mariadb version is not new enough? I think you need 10.6 or above, as
> > will use io_uring by default. What version are you running?
>
> And here's the .config and the patch for using queue_work().

So if you *don't* apply this patch, the work gets queued up with an IO
thread? In io-wq.c? Does that worker end up just doing an io_write()
same as this one?

Can the  queueing cause the creation of an IO thread (if one does not
exist, or all blocked?)

I'm wondering what the practical differences are between this patch and
upstream.

kthread_use_mm() should be basically the same as context switching to an
IO thread. There is maybe a difference in that kthread_switch_mm() has
a 'sync' instruction *after* the MMU is switched to the new thread from
the membarrier code, but a regular context switch might not. The MMU
switch does have an isync() after it though, so loads *should* be
prohibited from moving ahead of that.

Something like this adds a sync roughly where kthread_use_mm() has one.
It's a pretty unlikely shot in the dark though. I'm more inclined to
think the work submission to the IO thread might have a problem.

Thanks,
Nick


diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 488655f2319f..417c0652dc66 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5180,6 +5180,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)
        tick_nohz_task_switch();
        finish_lock_switch(rq);
        finish_arch_post_lock_switch();
+       smp_mb();
        kcov_finish_switch(current);
        /*
         * kmap_local_sched_out() is invoked with rq::lock held and