From: Long Li <longli@microsoft.com>
To: Sagi Grimberg <sagi@grimberg.me>, Ming Lei <ming.lei@redhat.com>
Cc: Keith Busch <keith.busch@intel.com>,
Hannes Reinecke <hare@suse.com>,
Daniel Lezcano <daniel.lezcano@linaro.org>,
Bart Van Assche <bvanassche@acm.org>,
"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
John Garry <john.garry@huawei.com>,
LKML <linux-kernel@vger.kernel.org>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
Jens Axboe <axboe@fb.com>, Ingo Molnar <mingo@redhat.com>,
Thomas Gleixner <tglx@linutronix.de>,
Christoph Hellwig <hch@lst.de>
Subject: RE: [PATCH 1/4] softirq: implement IRQ flood detection mechanism
Date: Fri, 20 Sep 2019 19:12:04 +0000 [thread overview]
Message-ID: <CY4PR21MB074168DE7729C131CE4394CCCE880@CY4PR21MB0741.namprd21.prod.outlook.com> (raw)
In-Reply-To: <30dc6fa9-ea5e-50d6-56f9-fbc9627d8c29@grimberg.me>
> >> Long, does this patch make any difference?
> >
> > Sagi,
> >
> > Sorry it took a while to bring my system back online.
> >
> > With the patch, the IOPS is about the same drop with the 1st patch. I think
> the excessive context switches are causing the drop in IOPS.
> >
> > The following are captured by "perf sched record" for 30 seconds during
> tests.
> >
> > "perf sched latency"
> > With patch:
> > fio:(82) | 937632.706 ms | 1782255 | avg: 0.209 ms | max: 63.123
> ms | max at: 768.274023 s
> >
> > without patch:
> > fio:(82) |2348323.432 ms | 18848 | avg: 0.295 ms | max: 28.446
> ms | max at: 6447.310255 s
>
> Without patch means the proposed hard-irq patch?
It means the current upstream code without any patch. But It's prone to soft lockup.
Ming's proposed hard-irq patch gets similar results to "without patch", however it fixes the soft lockup.
>
> If we are context switching too much, it means the soft-irq operation is not
> efficient, not necessarily the fact that the completion path is running in soft-
> irq..
>
> Is your kernel compiled with full preemption or voluntary preemption?
The tests are based on Ubuntu 18.04 kernel configuration. Here are the parameters:
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
>
> > Look closer at each CPU, we can see ksoftirqd is competing CPU with
> > fio (and effectively throttle other fio processes) (captured in
> > /sys/kernel/debug/tracing, echo sched:* >set_event)
> >
> > On CPU1 with patch: (note that the prev_state for fio is "R", it's
> preemptively scheduled)
> > <...>-4077 [001] d... 66456.805062: sched_switch: prev_comm=fio
> prev_pid=4077 prev_prio=120 prev_state=R ==> next_comm=ksoftirqd/1
> next_pid=17 next_prio=120
> > <...>-17 [001] d... 66456.805859: sched_switch:
> prev_comm=ksoftirqd/1 prev_pid=17 prev_prio=120 prev_state=S ==>
> next_comm=fio next_pid=4077 next_prio=120
> > <...>-4077 [001] d... 66456.844049: sched_switch: prev_comm=fio
> prev_pid=4077 prev_prio=120 prev_state=R ==> next_comm=ksoftirqd/1
> next_pid=17 next_prio=120
> > <...>-17 [001] d... 66456.844607: sched_switch:
> prev_comm=ksoftirqd/1 prev_pid=17 prev_prio=120 prev_state=S ==>
> next_comm=fio next_pid=4077 next_prio=120
> >
> > On CPU1 without patch: (the prev_state for fio is "S", it's voluntarily
> scheduled)
> > <idle>-0 [001] d... 6725.392308: sched_switch:
> prev_comm=swapper/1 prev_pid=0 prev_prio=120 prev_state=R ==>
> next_comm=fio next_pid=14342 next_prio=120
> > fio-14342 [001] d... 6725.392332: sched_switch: prev_comm=fio
> prev_pid=14342 prev_prio=120 prev_state=S ==> next_comm=swapper/1
> next_pid=0 next_prio=120
> > <idle>-0 [001] d... 6725.392356: sched_switch:
> prev_comm=swapper/1 prev_pid=0 prev_prio=120 prev_state=R ==>
> next_comm=fio next_pid=14342 next_prio=120
> > fio-14342 [001] d... 6725.392425: sched_switch:
> > prev_comm=fio prev_pid=14342 prev_prio=120 prev_state=S ==>
> > next_comm=swapper/1 next_pid=0 next_prio=12
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2019-09-20 19:12 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20190827085344.30799-1-ming.lei@redhat.com>
[not found] ` <20190827085344.30799-2-ming.lei@redhat.com>
[not found] ` <alpine.DEB.2.21.1908271633450.1939@nanos.tec.linutronix.de>
[not found] ` <20190827225827.GA5263@ming.t460p>
[not found] ` <alpine.DEB.2.21.1908280104330.1939@nanos.tec.linutronix.de>
[not found] ` <20190828110633.GC15524@ming.t460p>
[not found] ` <alpine.DEB.2.21.1908281316230.1869@nanos.tec.linutronix.de>
[not found] ` <20190828135054.GA23861@ming.t460p>
[not found] ` <alpine.DEB.2.21.1908281605190.23149@nanos.tec.linutronix.de>
2019-09-03 3:30 ` [PATCH 1/4] softirq: implement IRQ flood detection mechanism Ming Lei
2019-09-03 5:59 ` Daniel Lezcano
2019-09-03 6:31 ` Ming Lei
2019-09-03 6:40 ` Daniel Lezcano
2019-09-03 7:28 ` Ming Lei
2019-09-03 7:50 ` Daniel Lezcano
2019-09-03 9:30 ` Ming Lei
2019-09-04 17:07 ` Bart Van Assche
2019-09-04 17:31 ` Daniel Lezcano
2019-09-04 17:38 ` Bart Van Assche
2019-09-04 18:02 ` Peter Zijlstra
2019-09-04 19:47 ` Bart Van Assche
2019-09-05 9:11 ` Ming Lei
2019-09-05 9:06 ` Ming Lei
2019-09-05 10:37 ` Daniel Lezcano
2019-09-06 1:22 ` Long Li
2019-09-06 4:36 ` Daniel Lezcano
2019-09-06 4:44 ` Long Li
2019-09-06 1:48 ` Ming Lei
2019-09-06 5:14 ` Daniel Lezcano
2019-09-06 18:30 ` Sagi Grimberg
2019-09-06 18:52 ` Keith Busch
2019-09-07 0:01 ` Ming Lei
2019-09-10 3:10 ` Sagi Grimberg
2019-09-18 0:00 ` Long Li
2019-09-20 17:14 ` Sagi Grimberg
2019-09-20 19:12 ` Long Li [this message]
2019-09-20 20:45 ` Sagi Grimberg
2019-09-24 0:57 ` Long Li
2019-09-18 14:37 ` Ming Lei
2019-09-20 17:09 ` Sagi Grimberg
2019-09-06 14:18 ` Keith Busch
2019-09-06 17:50 ` Long Li
2019-09-06 22:19 ` Ming Lei
2019-09-06 22:25 ` Keith Busch
2019-09-06 23:13 ` Ming Lei
2019-09-10 0:24 ` Ming Lei
2019-09-03 8:09 ` Thomas Gleixner
2019-09-03 9:24 ` Ming Lei
[not found] ` <20190827085344.30799-5-ming.lei@redhat.com>
2019-09-06 8:50 ` [PATCH 4/4] genirq: use irq's affinity for threaded irq with IRQF_RESCUE_THREAD John Garry
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CY4PR21MB074168DE7729C131CE4394CCCE880@CY4PR21MB0741.namprd21.prod.outlook.com \
--to=longli@microsoft.com \
--cc=axboe@fb.com \
--cc=bvanassche@acm.org \
--cc=daniel.lezcano@linaro.org \
--cc=hare@suse.com \
--cc=hch@lst.de \
--cc=john.garry@huawei.com \
--cc=keith.busch@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-scsi@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=sagi@grimberg.me \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).