From: Chuck Lever <chuck.lever@oracle.com>
To: Robin Murphy <robin.murphy@arm.com>
Cc: Will Deacon <will@kernel.org>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>
Subject: Re: [RFT PATCH 0/3] Performance regression noted in v5.11-rc after c062db039f40
Date: Tue, 26 Jan 2021 17:32:22 +0000 [thread overview]
Message-ID: <EFC0FF60-778C-434C-8655-6FB6D44E583E@oracle.com> (raw)
In-Reply-To: <8fd70fb7-6865-1b11-2a16-9bf473004662@arm.com>
Hi Robin-
> On Jan 26, 2021, at 11:05 AM, Robin Murphy <robin.murphy@arm.com> wrote:
>
> Implementing .iotlb_sync_map means that a single top-level iommu_map()/iommu_map_sg() call should still only invoke a single "TLB flush" (really, any maintenance required for the IOMMU to start using the new mapping) at the end, regardless of how many internal __iommu_map() calls are made to satisfy the overall request. If you're seeing something other than that behaviour (with this series), that implies we've not got things quite right yet.
The flush is expensive, but it's not the only cost. DMA-mapping a 120KB
SGL in a single domain_mapping() call vs. 30 calls is certainly going to
be a detectable difference.
Naively speaking, if there are more DMA mappings to keep track of because
the IOMMU driver isn't coalescing the SGLs the way it did before, that
might trigger TLB thrashing on the NIC.
> Is there any significant difference between how the NFS read and write paths make their DMA API calls and/or get their scatterlists in the first place, that might help shed some light on the curious half-recovery you got?
There isn't a difference in the RPC-over-RDMA code. Client-side DMA mapping
is handled in net/sunrpc/xprtrdma/frwr_ops.c :: frwr_map() which is used for
both I/O directions.
On the server, the RDMA core r/w API is used for mapping and then posting
RDMA Read and Write work requests. That API appears in
drivers/infiniband/core/rw.c , and as far as I understand, the same mapping
functions are used for both I/O directions.
It's possible that the NIC is doing something different for RDMA Read and
RDMA Write, but I don't have much visibility into that. Reads are very
different from Writes, which are posted.
--
Chuck Lever
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
prev parent reply other threads:[~2021-01-26 17:35 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-25 2:38 [RFT PATCH 0/3] Performance regression noted in v5.11-rc after c062db039f40 Lu Baolu
2021-01-25 2:38 ` [RFT PATCH 1/3] iommu: Move iotlb_sync_map out from __iommu_map Lu Baolu
2021-01-25 2:38 ` [RFT PATCH 2/3] iommu: Add iova and size as parameters in iotlb_sync_map Lu Baolu
2021-01-25 2:38 ` [RFT PATCH 3/3] iommu/vt-d: Add iotlb_sync_map callback Lu Baolu
2021-01-25 17:39 ` [RFT PATCH 0/3] Performance regression noted in v5.11-rc after c062db039f40 Chuck Lever
2021-01-25 19:31 ` Chuck Lever
2021-01-26 6:18 ` Lu Baolu
2021-01-26 15:52 ` Chuck Lever
2021-01-27 1:53 ` Lu Baolu
2021-01-27 2:58 ` Chuck Lever
2021-01-26 16:05 ` Robin Murphy
2021-01-26 17:32 ` Chuck Lever [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=EFC0FF60-778C-434C-8655-6FB6D44E583E@oracle.com \
--to=chuck.lever@oracle.com \
--cc=iommu@lists.linux-foundation.org \
--cc=robin.murphy@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.