From: Peter Zijlstra <email@example.com> To: Dave Chinner <firstname.lastname@example.org> Cc: Ming Lei <email@example.com>, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, Jeff Moyer <firstname.lastname@example.org>, Dave Chinner <email@example.com>, Eric Sandeen <firstname.lastname@example.org>, Christoph Hellwig <email@example.com>, Jens Axboe <firstname.lastname@example.org>, Ingo Molnar <email@example.com>, Tejun Heo <firstname.lastname@example.org>, Vincent Guittot <email@example.com> Subject: Re: single aio thread is migrated crazily by scheduler Date: Wed, 20 Nov 2019 20:16:36 +0100 Message-ID: <20191120191636.GI4097@hirez.programming.kicks-ass.net> (raw) In-Reply-To: <20191118204054.GV4614@dread.disaster.area> On Tue, Nov 19, 2019 at 07:40:54AM +1100, Dave Chinner wrote: > On Mon, Nov 18, 2019 at 10:21:21AM +0100, Peter Zijlstra wrote: > > We typically only fall back to the active balancer when there is > > (persistent) imbalance and we fail to migrate anything else (of > > substance). > > > > The tuning mentioned has the effect of less frequent scheduling, IOW, > > leaving (short) tasks on the runqueue longer. This obviously means the > > load-balancer will have a bigger chance of seeing them. > > > > Now; it's been a while since I looked at the workqueue code but one > > possible explanation would be if the kworker that picks up the work item > > is pinned. That would make it runnable but not migratable, the exact > > situation in which we'll end up shooting the current task with active > > balance. > > Yes, that's precisely the problem - work is queued, by default, on a > specific CPU and it will wait for a kworker that is pinned to that I'm thinking the problem is that it doesn't wait. If it went and waited for it, active balance wouldn't be needed, that only works on active tasks. > specific CPU to dispatch it. We've already tested that queuing on a > different CPU (via queue_work_on()) makes the problem largely go > away as the work is not longer queued behind the long running fio > task. > > This, however, is not at viable solution to the problem. The pattern > of a long running process queuing small pieces of individual work > for processing in a separate context is pretty common... Right, but you're putting the scheduler in a bind. By overloading the CPU and only allowing the one task to migrate, it pretty much has no choice left. Anyway, I'm still going to have try and reproduce -- I got side-tracked into a crashing bug, I'll hopefully get back to this tomorrow. Lastly, one other thing to try is -next. Vincent reworked the load-balancer quite a bit.
next prev parent reply index Thread overview: 64+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-11-14 11:31 Ming Lei 2019-11-14 13:14 ` Peter Zijlstra 2019-11-15 0:09 ` Ming Lei 2019-11-15 14:16 ` Ming Lei 2019-11-14 23:54 ` Dave Chinner 2019-11-15 1:08 ` Ming Lei 2019-11-15 4:56 ` Dave Chinner 2019-11-15 7:08 ` Ming Lei 2019-11-15 23:40 ` Dave Chinner 2019-11-16 6:31 ` Ming Lei 2019-11-18 9:21 ` Peter Zijlstra 2019-11-18 14:54 ` Vincent Guittot 2019-11-18 20:40 ` Dave Chinner 2019-11-20 19:16 ` Peter Zijlstra [this message] 2019-11-20 22:03 ` Phil Auld 2019-11-21 4:12 ` Ming Lei 2019-11-21 14:12 ` Phil Auld 2019-11-21 15:02 ` Boaz Harrosh 2019-11-21 16:19 ` Jens Axboe 2019-12-09 16:58 ` Srikar Dronamraju 2019-11-21 22:10 ` Dave Chinner 2019-11-21 13:29 ` Peter Zijlstra 2019-11-21 14:21 ` Phil Auld 2019-12-09 16:51 ` Srikar Dronamraju 2019-12-09 23:17 ` Dave Chinner 2019-12-10 3:27 ` Srikar Dronamraju 2019-12-10 5:43 ` [PATCH v2] sched/core: Preempt current task in favour of bound kthread Srikar Dronamraju 2019-12-10 9:26 ` Peter Zijlstra 2019-12-10 9:33 ` Peter Zijlstra 2019-12-10 10:18 ` Srikar Dronamraju 2019-12-10 10:16 ` Srikar Dronamraju 2019-12-10 9:43 ` Vincent Guittot 2019-12-10 10:11 ` Srikar Dronamraju 2019-12-10 11:02 ` Vincent Guittot 2019-12-10 17:23 ` [PATCH v3] " Srikar Dronamraju 2019-12-11 17:38 ` [PATCH v4] " Srikar Dronamraju 2019-12-11 22:46 ` Dave Chinner 2019-12-12 10:10 ` Peter Zijlstra 2019-12-12 10:14 ` Peter Zijlstra 2019-12-12 10:23 ` Peter Zijlstra 2019-12-12 11:20 ` Vincent Guittot 2019-12-12 13:12 ` Peter Zijlstra 2019-12-12 15:07 ` Srikar Dronamraju 2019-12-12 15:15 ` Peter Zijlstra 2019-12-13 5:32 ` Srikar Dronamraju 2019-11-18 16:26 ` single aio thread is migrated crazily by scheduler Srikar Dronamraju 2019-11-18 21:18 ` Dave Chinner 2019-11-19 8:54 ` Ming Lei [not found] ` <firstname.lastname@example.org> 2019-11-28 9:53 ` Vincent Guittot 2019-12-02 2:46 ` Ming Lei 2019-12-02 4:02 ` Dave Chinner 2019-12-02 4:22 ` Ming Lei 2019-12-02 13:45 ` Vincent Guittot 2019-12-02 21:22 ` Phil Auld 2019-12-03 9:45 ` Vincent Guittot 2019-12-04 13:50 ` Ming Lei 2019-12-02 23:53 ` Dave Chinner 2019-12-03 0:18 ` Ming Lei 2019-12-03 13:34 ` Vincent Guittot 2019-12-02 7:39 ` Juri Lelli 2019-12-02 3:08 ` Dave Chinner [not found] ` <email@example.com> 2019-12-02 23:06 ` Dave Chinner [not found] ` <firstname.lastname@example.org> 2019-12-03 22:29 ` Dave Chinner [not found] ` <email@example.com> 2019-12-04 22:59 ` Dave Chinner
Reply instructions: You may reply publically to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20191120191636.GI4097@hirez.programming.kicks-ass.net \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Linux-XFS Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/linux-xfs/0 linux-xfs/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 linux-xfs linux-xfs/ https://lore.kernel.org/linux-xfs \ email@example.com public-inbox-index linux-xfs Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-xfs AGPL code for this site: git clone https://public-inbox.org/public-inbox.git