From: Alexander Fomichev <fomichev.ru@gmail.com>
To: Mel Gorman <mgorman@suse.de>
Cc: linux-kernel@vger.kernel.org, dmaengine@vger.kernel.org,
linux@yadro.com, Peter Zijlstra <peterz@infradead.org>
Subject: Re: [RFC] Scheduler: DMA Engine regression because of sched/fair changes
Date: Mon, 17 Jan 2022 20:44:19 +0300 [thread overview]
Message-ID: <20220117174419.hqj2mz2ctrjdq57d@yadro.com> (raw)
In-Reply-To: <20220117102701.GQ3301@suse.de>
On Mon, Jan 17, 2022 at 10:27:01AM +0000, Mel Gorman wrote:
> > 1) You're right. When options "noverify=1" and "polling=1" are used.
> > then no performance reducing occurs.
>
> How about just noverify=1 on its own? It's a stronger indicator that
> cache hotness is a factor.
>
With "noverify=1 polled=0" the performance reduction is only 10-20%,
but still exists.
-----< v5.15.8-vanilla >-----
[17057.866760] dmatest: Added 1 threads using dma0chan0
[17060.133880] dmatest: Started 1 threads using dma0chan0
[17060.154343] dmatest: dma0chan0-copy0: summary 1000 tests, 0 failures 49338.85 iops 3157686 KB/s (0)
[17063.737887] dmatest: Added 1 threads using dma0chan0
[17065.113838] dmatest: Started 1 threads using dma0chan0
[17065.137659] dmatest: dma0chan0-copy0: summary 1000 tests, 0 failures 42183.41 iops 2699738 KB/s (0)
[17100.339989] dmatest: Added 1 threads using dma0chan0
[17102.190764] dmatest: Started 1 threads using dma0chan0
[17102.214285] dmatest: dma0chan0-copy0: summary 1000 tests, 0 failures 42844.89 iops 2742073 KB/s (0)
-----< end >-----
-----< 5.15.8-ioat-ptdma-dirty-fix+ >-----
[ 6183.356549] dmatest: Added 1 threads using dma0chan0
[ 6187.868237] dmatest: Started 1 threads using dma0chan0
[ 6187.887389] dmatest: dma0chan0-copy0: summary 1000 tests, 0 failures 52753.74 iops 3376239 KB/s (0)
[ 6201.913154] dmatest: Added 1 threads using dma0chan0
[ 6204.701340] dmatest: Started 1 threads using dma0chan0
[ 6204.720490] dmatest: dma0chan0-copy0: summary 1000 tests, 0 failures 52614.96 iops 3367357 KB/s (0)
[ 6285.114603] dmatest: Added 1 threads using dma0chan0
[ 6287.031875] dmatest: Started 1 threads using dma0chan0
[ 6287.050278] dmatest: dma0chan0-copy0: summary 1000 tests, 0 failures 54939.01 iops 3516097 KB/s (0)
-----< end >-----
> > 2) DMA Engine on certain devices, e.g. Switchtec DMA and AMD PTDMA, is
> > used particularly for off-CPU data transfer via device's NTB to a remote
> > host. In NTRDMA project, which I'm involved to, DMA Engine sends data to
> > remote ring buffer and on data arrival CPU processes local ring buffers.
> >
>
> Is there any impact of the patch in this case? Given that it's a remote
> host, the data is likely cache cold anyway.
>
It's complicated. Currently we have a bunch of problems with the
project. So we do decomposition and try to solve them separately. Here
we faced the DMA Engine issue.
> > 4) Do you mean that with noverify=N and dirty patch, data verification
> > is performed on cached data and thus measured performance is fake?
> >
>
> I think it's the data verification going slower because the tasks are
> not aggressively migrating on interrupt. The flip side is other
> interrupts such as IO completion should not migrate the tasks given that
> the interrupt is not necessarily correlated with data hotness.
>
It's quite strange, because dmatest substitutes verification time from
overall test time. I suspect measurement may be inaccurate.
> > 5) What DMA Engine enabled drivers (and dmatest) should use as design
> > pattern to conform migration/cache behavior? Does scheduler optimisation
> > conflict to DMA Engine performance in general?
> >
>
> I'm not familiar with DMA engine drivers but if they use wake_up
> interfaces then passing WF_SYNC or calling the wake_up_*_sync helpers
> may force the migration.
>
Thanks for the advice. I'll try to check if this is a solution.
--
Regards,
Alexander
next prev parent reply other threads:[~2022-01-17 17:44 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-12 15:26 [RFC] Scheduler: DMA Engine regression because of sched/fair changes Alexander Fomichev
2022-01-12 17:05 ` Mel Gorman
2022-01-17 8:19 ` Alexander Fomichev
2022-01-17 10:27 ` Mel Gorman
2022-01-17 17:44 ` Alexander Fomichev [this message]
[not found] ` <20220118020448.2399-1-hdanton@sina.com>
2022-01-18 10:05 ` Mel Gorman
2022-01-19 12:55 ` Alexander Fomichev
[not found] ` <20220121101217.2849-1-hdanton@sina.com>
2022-01-21 13:46 ` Alexander Fomichev
[not found] ` <20220122233314.2999-1-hdanton@sina.com>
2022-01-28 16:50 ` Alexander Fomichev
2022-02-23 15:24 ` Thorsten Leemhuis
2022-01-16 9:55 ` Thorsten Leemhuis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220117174419.hqj2mz2ctrjdq57d@yadro.com \
--to=fomichev.ru@gmail.com \
--cc=dmaengine@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux@yadro.com \
--cc=mgorman@suse.de \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).