linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Marcelo Tosatti <mtosatti@redhat.com>
To: Oscar Shiang <oscar0225@livemail.tw>
Cc: linux-kernel@vger.kernel.org, Nitesh Lal <nilal@redhat.com>,
	Nicolas Saenz Julienne <nsaenzju@redhat.com>,
	Frederic Weisbecker <frederic@kernel.org>,
	Christoph Lameter <cl@linux.com>,
	Juri Lelli <juri.lelli@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Alex Belits <abelits@belits.com>, Peter Xu <peterx@redhat.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Daniel Bristot de Oliveira <bristot@redhat.com>
Subject: Re: [patch v11 00/13] extensible prctl task isolation interface and vmstat sync
Date: Tue, 8 Mar 2022 10:12:35 -0300	[thread overview]
Message-ID: <YidWQ28Jgs/FpmL1@fuller.cnet> (raw)
In-Reply-To: <TYCP286MB19134DD105CC996925379EC8A1099@TYCP286MB1913.JPNP286.PROD.OUTLOOK.COM>

On Tue, Mar 08, 2022 at 02:32:46PM +0800, Oscar Shiang wrote:
> On Feb 24, 2022, at 1:31 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> > Hi Oscar,
> > 
> > On Sat, Feb 19, 2022 at 04:02:10PM +0800, Oscar Shiang wrote:
> > > Hi Marcelo,
> > > 
> > > I tried to apply your patches to kernel v5.15.18-rt28 and measured
> > > the latencies through oslat [1].
> > > 
> > > It turns out that the peak latency (around 100us) can drop to about 90us.
> > > The result is impressive since I only changed the guest's kernel
> > > instead of installing the patched kernel to both host and guest.
> > > 
> > > However, I am still curious about:
> > > 1) Why did I catch a bigger maximum latency in almost each of the
> > >   results of applying task isolation patches? Or does it come from
> > >   other reasons?
> > 
> > There are a number of things that need to be done in order to have an 
> > "well enough" isolated CPU so you can measure latency reliably:
> > 
> > * Boot a kernel with isolated CPU (or better, use realtime-virtual-host profile of
> > https://github.com/redhat-performance/tuned.git, which does a bunch of
> > other things to avoid interruptions to isolated CPUs).
> > * Apply the userspace patches at https://people.redhat.com/~mtosatti/task-isol-v6-userspace-patches/
> > to util-linux and rt-tests.
> > 
> > Run oslat with chisol:
> > 
> > chisol -q vmstat_sync -I conf oslat -c ...
> > 
> > Where chisol is from patched util-linux and oslat from patched rt-tests.
> > 
> > If you had "-f 1" (FIFO priority), on oslat, then the vmstat work would be hung.
> > 
> > Are you doing those things?
> > 
> > > 2) Why did we only get a 10us improvement on quiescing vmstat?
> > 
> > If you did not have FIFO priority on oslat, then other daemons 
> > could be interrupting it, so better make sure the 10us improvement 
> > you see is due to vmstat_flush workqueue work not executing anymore.
> > 
> > The testcase i use is: 
> > 
> > Stock kernel:
> > 
> > terminal 1: 
> > # oslat -f 1 -c X ...
> > 
> > terminal 2:
> > # echo 1 > /proc/sys/vm/stat_refresh
> > (hang)
> > 
> > Patched kernel:
> > 
> > terminal 1: 
> > # chisol -q vmstat_sync -I conf oslat -f 1 -c X ...
> > 
> > terminal 2:
> > # echo 1 > /proc/sys/vm/stat_refresh
> > # 
> 
> Sure, I did see the terminal hung during oslat with FIFO priority.
> 
> BTW, thanks for providing this test case. I used to run all workload stuff to just
> verify the improvement of task isolation. It is a more straightr- forward way to do.
> 
> > > [1]: The result and the test scripts I used can be found at
> > > https://gist.github.com/OscarShiang/8b530a00f472fd1c39f5979ee601516d#testing-task-isolation-via-oslat
> > 
> > OK, you seem to be doing everything necessary for chisol 
> > to work. Does /proc/pid/task_isolation of the oslat worker thread
> > (note its not the same pid as the main oslat thread) show "vmstat"
> > configured and activated for quiesce?
> 
> The status of task_isolation seems to be set properly with "vmstat" and activated
> 
> > However 100us is really high. You should be able to get < 10us with
> > realtime-virtual-host (i see 4us on an idle system).
> > 
> > The answer might be: because 10us is what it takes to execute
> > vmstat_worker on the isolated CPU (you can verify with tracepoints).
> > 
> > That time depends on the number of per-CPU vmstat variables that need flushing, 
> > i suppose...
> 
> Considering the interferences outside of the KVM, I have redone the measurements
> directly on my x86_64 computer [1].
> 
> As result, most of the latencies are down to 60us (and below). There are still
> some latencies larger than 80us, I am working on and trying to figure out the reason.
> 
> [1]: https://gist.github.com/OscarShiang/202eb691e649557fe3eaa5ec67a5aa82

Oscar,

Did you confirm with hwlatdetect that the BIOS does not have long
running SMIs?

Also, for the software part, you could save time by using the
realtime-virtual-host profile (check /usr/lib/tuned/realtime-virtual-host/
to see what its doing in addition to isolcpus=).


  reply	other threads:[~2022-03-08 17:09 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-02-04 17:35 [patch v11 00/13] extensible prctl task isolation interface and vmstat sync Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 01/13] s390: add support for TIF_TASK_ISOL Marcelo Tosatti
2022-02-07 11:40   ` Sven Schnelle
2022-02-04 17:35 ` [patch v11 02/13] x86: " Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 03/13] add basic task isolation prctl interface Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 04/13] add prctl task isolation prctl docs and samples Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 05/13] task isolation: sync vmstats on return to userspace Marcelo Tosatti
2022-02-07 14:57   ` Frederic Weisbecker
2022-02-07 15:16     ` Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 06/13] procfs: add per-pid task isolation state Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 07/13] task isolation: sync vmstats conditional on changes Marcelo Tosatti
2022-02-07 15:19   ` Frederic Weisbecker
2022-03-10 16:30     ` Marcelo Tosatti
2022-02-07 15:29   ` Frederic Weisbecker
2022-02-04 17:35 ` [patch v11 08/13] task isolation: enable return to userspace processing Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 09/13] task isolation: add preempt notifier to sync per-CPU vmstat dirty info to thread info Marcelo Tosatti
2022-02-07 15:36   ` Frederic Weisbecker
2022-02-04 17:35 ` [patch v11 10/13] KVM: x86: process isolation work from VM-entry code path Marcelo Tosatti
2022-02-07 15:47   ` Frederic Weisbecker
2022-03-10 16:35     ` Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 11/13] mm: vmstat: move need_update Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 12/13] mm: vmstat_refresh: avoid queueing work item if cpu stats are clean Marcelo Tosatti
2022-02-04 17:35 ` [patch v11 13/13] task isolation: only TIF_TASK_ISOL if task isolation is enabled Marcelo Tosatti
2022-02-07 15:38   ` Frederic Weisbecker
2022-02-19  8:02 ` [patch v11 00/13] extensible prctl task isolation interface and vmstat sync Oscar Shiang
2022-02-23 17:31   ` Marcelo Tosatti
2022-03-08  6:32     ` Oscar Shiang
2022-03-08 13:12       ` Marcelo Tosatti [this message]
2022-03-09 15:31         ` Oscar Shiang
2022-03-08  7:20 ` Oscar Shiang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YidWQ28Jgs/FpmL1@fuller.cnet \
    --to=mtosatti@redhat.com \
    --cc=abelits@belits.com \
    --cc=bristot@redhat.com \
    --cc=cl@linux.com \
    --cc=frederic@kernel.org \
    --cc=juri.lelli@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=nilal@redhat.com \
    --cc=nsaenzju@redhat.com \
    --cc=oscar0225@livemail.tw \
    --cc=peterx@redhat.com \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).