From: Robin Murphy <robin.murphy@arm.com>
To: "Petr Tesařík" <petr@tesarici.cz>
Cc: Alexander Lobakin <aleksander.lobakin@intel.com>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
Christoph Hellwig <hch@lst.de>,
Marek Szyprowski <m.szyprowski@samsung.com>,
Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
"Rafael J. Wysocki" <rafael@kernel.org>,
Magnus Karlsson <magnus.karlsson@intel.com>,
Maciej Fijalkowski <maciej.fijalkowski@intel.com>,
Alexander Duyck <alexanderduyck@fb.com>,
bpf@vger.kernel.org, netdev@vger.kernel.org,
iommu@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH net-next 2/7] dma: avoid expensive redundant calls for sync operations
Date: Fri, 26 Jan 2024 19:13:05 +0000 [thread overview]
Message-ID: <1c62d388-a600-40d8-b386-15841cb1af95@arm.com> (raw)
In-Reply-To: <20240126194819.147cb4e2@meshulam.tesarici.cz>
On 26/01/2024 6:48 pm, Petr Tesařík wrote:
> On Fri, 26 Jan 2024 17:21:24 +0000
> Robin Murphy <robin.murphy@arm.com> wrote:
>
>> On 26/01/2024 4:45 pm, Alexander Lobakin wrote:
>>> From: Robin Murphy <robin.murphy@arm.com>
>>> Date: Fri, 26 Jan 2024 15:48:54 +0000
>>>
>>>> On 26/01/2024 1:54 pm, Alexander Lobakin wrote:
>>>>> From: Eric Dumazet <edumazet@google.com>
>>>>>
>>>>> Quite often, NIC devices do not need dma_sync operations on x86_64
>>>>> at least.
>>>>> Indeed, when dev_is_dma_coherent(dev) is true and
>>>>> dev_use_swiotlb(dev) is false, iommu_dma_sync_single_for_cpu()
>>>>> and friends do nothing.
>>>>>
>>>>> However, indirectly calling them when CONFIG_RETPOLINE=y consumes about
>>>>> 10% of cycles on a cpu receiving packets from softirq at ~100Gbit rate.
>>>>> Even if/when CONFIG_RETPOLINE is not set, there is a cost of about 3%.
>>>>>
>>>>> Add dev->skip_dma_sync boolean which is set during the device
>>>>> initialization depending on the setup: dev_is_dma_coherent() for direct
>>>>> DMA, !(sync_single_for_device || sync_single_for_cpu) or positive result
>>>>> from the new callback, dma_map_ops::can_skip_sync for non-NULL DMA ops.
>>>>> Then later, if/when swiotlb is used for the first time, the flag
>>>>> is turned off, from swiotlb_tbl_map_single().
>>>>
>>>> I think you could probably just promote the dma_uses_io_tlb flag from
>>>> SWIOTLB_DYNAMIC to a general SWIOTLB thing to serve this purpose now.
>>>
>>> Nice catch!
>>>
>>>>
>>>> Similarly I don't think a new op is necessary now that we have
>>>> dma_map_ops.flags. A simple static flag to indicate that sync may be> skipped under the same conditions as implied for dma-direct - i.e.
>>>> dev_is_dma_coherent(dev) && !dev->dma_use_io_tlb - seems like it ought
>>>> to suffice.
>>>
>>> In my initial implementation, I used a new dma_map_ops flag, but then I
>>> realized different DMA ops may require or not require syncing under
>>> different conditions, not only dev_is_dma_coherent().
>>> Or am I wrong and they would always be the same?
>>
>> I think it's safe to assume that, as with P2P support, this will only
>> matter for dma-direct and iommu-dma for the foreseeable future, and
>> those do currently share the same conditions as above. Thus we may as
>> well keep things simple for now, and if anything ever does have cause to
>> change, it can be the future's problem to keep this mechanism working as
>> intended.
>
> Can we have a comment that states this assumption along with the flag?
> Because when it breaks, it will keep someone cursing for days why DMA
> sometimes fails on their device before they find out it's not synced.
> And then wondering why the code makes such silly assumptions...
Indeed, apologies if it wasn't totally clear, but I really was implying
a literal "may skip sync if coherent and not using SWIOTLB (which
matches dma-direct)" flag, documented as such, and not trying to dress
it up as anything more generic. I just can't suggest a suitably concise
name for that of the top of my head... :)
Thanks,
Robin.
next prev parent reply other threads:[~2024-01-26 19:13 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-26 13:54 [PATCH net-next 0/7] dma: skip calling no-op sync ops when possible Alexander Lobakin
2024-01-26 13:54 ` [PATCH net-next 1/7] dma: compile-out DMA sync op calls when not used Alexander Lobakin
2024-01-29 6:11 ` Christoph Hellwig
2024-01-29 11:07 ` Alexander Lobakin
2024-01-29 12:15 ` Robin Murphy
2024-01-31 16:52 ` Simon Horman
2024-01-31 17:14 ` Robin Murphy
2024-01-26 13:54 ` [PATCH net-next 2/7] dma: avoid expensive redundant calls for sync operations Alexander Lobakin
2024-01-26 15:48 ` Robin Murphy
2024-01-26 16:45 ` Alexander Lobakin
2024-01-26 17:21 ` Robin Murphy
2024-01-26 18:48 ` Petr Tesařík
2024-01-26 19:13 ` Robin Murphy [this message]
2024-01-29 6:09 ` Christoph Hellwig
2024-01-29 14:36 ` Alexander Lobakin
2024-01-29 16:15 ` Petr Tesařík
2024-01-29 14:07 ` Alexander Lobakin
2024-01-29 14:29 ` Robin Murphy
2024-01-29 14:34 ` Alexander Lobakin
2024-01-26 13:54 ` [PATCH net-next 3/7] iommu/dma: avoid expensive indirect " Alexander Lobakin
2024-01-26 13:54 ` [PATCH net-next 4/7] page_pool: make sure frag API fields don't span between cachelines Alexander Lobakin
2024-01-26 13:54 ` [PATCH net-next 5/7] page_pool: don't use driver-set flags field directly Alexander Lobakin
2024-01-26 13:54 ` [PATCH net-next 6/7] page_pool: check for DMA sync shortcut earlier Alexander Lobakin
2024-01-26 13:54 ` [PATCH net-next 7/7] xsk: use generic DMA sync shortcut instead of a custom one Alexander Lobakin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1c62d388-a600-40d8-b386-15841cb1af95@arm.com \
--to=robin.murphy@arm.com \
--cc=aleksander.lobakin@intel.com \
--cc=alexanderduyck@fb.com \
--cc=bpf@vger.kernel.org \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=gregkh@linuxfoundation.org \
--cc=hch@lst.de \
--cc=iommu@lists.linux.dev \
--cc=joro@8bytes.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=m.szyprowski@samsung.com \
--cc=maciej.fijalkowski@intel.com \
--cc=magnus.karlsson@intel.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=petr@tesarici.cz \
--cc=rafael@kernel.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).