From: Clint Sbisa <csbisa@amazon.com>
To: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
Leon Romanovsky <leon@kernel.org>,
linux-pci@vger.kernel.org, Bjorn Helgaas <helgaas@kernel.org>,
Jason Gunthorpe <jgg@nvidia.com>,
catalin.marinas@arm.com, will@kernel.org,
linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs
Date: Fri, 11 Sep 2020 21:42:25 +0000 [thread overview]
Message-ID: <20200911214225.hml2wbbq2rofn4re@amazon.com> (raw)
In-Reply-To: <3110e00a1f4df7b7359ba4f2b7f86a35aa47405e.camel@kernel.crashing.org>
On Fri, Sep 11, 2020 at 10:39:16AM +1000, Benjamin Herrenschmidt wrote:
> > > > > That's why I looped you in - that's what worries me about
> > > > > "enabling"
> > > > > arch_can_pci_mmap_wc() on arm64. If we enable it and we have perf
> > > > > regressions that's not OK.
> > > > >
> > > > > Or we *can* enable arch_can_pci_mmap_wc() but force the mellanox
> > > > > driver (or more broadly all drivers following this message push
> > > > > semantics) to use "something else" for WC detection.
> > > >
> > > > arch_can_pci_mmap_wc() really only controls the sysfs resource file
> > > > and it seems very unclear who in userspace uses that these days.
> > >
> > > dpdk under some circumstances afaik.
> >
> > And something gross for DMA then? Not sure dpdk is useful without
> > DMA. Why not use CONFIG_VFIO_NOIOMMU for such a non-secure thing?
>
> Clint, can you elaborate on the use case ?
>
The use-case I'm targeting is the ENA pmd in DPDK. For performance reasons
(many of which are very similar to what Jason has described for mlx5), we need
to generate full-sized TLPs instead of many partial TLPs to improve efficiency.
Here's an excerpt describing the write-combine usage from
./Documentation/networking/device_drivers/ethernet/amazon/ena.rst:
- Low Latency Queue (LLQ) mode or "push-mode".
* In this mode the driver pushes the transmit descriptors and the
first 128 bytes of the packet directly to the ENA device memory
space. The rest of the packet payload is fetched by the
device. For this operation mode, the driver uses a dedicated PCI
device memory BAR, which is mapped with write-combine capability.
There's no DMA involved with this BAR-- the driver writes a portion of the
packet contents in addition to the descriptors, which generally increases the
number of TLPs if write-combine isn't used. Furthermore, this BAR is only used
for writes and never for reads.
As Jason noted in the other reply to this email, the Linux ENA driver makes use
of WC by using devm_ioremap_wc(). The DPDK code here uses the same mechanism in
user-space to enable write-combining by mapping the resourceX_wc file if the
driver uses RTE_PCI_DRV_WC_ACTIVATE.
Clint
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-09-11 21:44 UTC|newest]
Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20200831151827.pumm2p54fyj7fz5s@amazon.com>
[not found] ` <20200902113207.GA27676@e121166-lin.cambridge.arm.com>
[not found] ` <20200902142922.xc4x6m33unkzewuh@amazon.com>
2020-09-02 16:47 ` [PATCH] arm64: Enable PCI write-combine resources under sysfs Lorenzo Pieralisi
2020-09-02 17:21 ` Catalin Marinas
2020-09-02 17:54 ` Lorenzo Pieralisi
2020-09-02 23:03 ` Benjamin Herrenschmidt
2020-09-02 23:08 ` Benjamin Herrenschmidt
2020-09-02 23:08 ` Benjamin Herrenschmidt
2020-09-02 23:07 ` Benjamin Herrenschmidt
2020-09-03 11:08 ` Lorenzo Pieralisi
2020-09-03 14:36 ` Clint Sbisa
2020-09-03 22:26 ` Benjamin Herrenschmidt
2020-09-07 23:33 ` Benjamin Herrenschmidt
2020-09-10 9:46 ` Lorenzo Pieralisi
2020-09-10 10:54 ` Leon Romanovsky
2020-09-10 12:37 ` Jason Gunthorpe
2020-09-10 15:17 ` Lorenzo Pieralisi
2020-09-10 17:10 ` Jason Gunthorpe
2020-09-10 21:46 ` Benjamin Herrenschmidt
2020-09-10 23:29 ` Jason Gunthorpe
2020-09-11 0:39 ` Benjamin Herrenschmidt
2020-09-11 14:21 ` Jason Gunthorpe
2020-09-11 21:42 ` Clint Sbisa [this message]
2020-09-14 14:17 ` Jason Gunthorpe
2020-09-14 14:24 ` Clint Sbisa
2020-09-14 14:38 ` Jason Gunthorpe
2020-09-14 21:42 ` Benjamin Herrenschmidt
2020-09-14 22:00 ` Benjamin Herrenschmidt
2020-09-14 22:32 ` Clint Sbisa
2020-09-14 22:57 ` Jason Gunthorpe
2020-09-14 23:25 ` Benjamin Herrenschmidt
2020-09-15 10:18 ` Lorenzo Pieralisi
2020-09-15 11:05 ` Jason Gunthorpe
2020-09-15 23:17 ` Benjamin Herrenschmidt
2020-09-15 23:40 ` Jason Gunthorpe
2020-09-16 7:59 ` Benjamin Herrenschmidt
2020-09-16 12:12 ` Jason Gunthorpe
2020-09-16 14:09 ` Lorenzo Pieralisi
2020-09-16 14:14 ` Jason Gunthorpe
2020-09-16 23:59 ` Benjamin Herrenschmidt
2020-09-17 10:28 ` Lorenzo Pieralisi
2020-09-17 11:32 ` Jason Gunthorpe
2020-09-17 14:01 ` Lorenzo Pieralisi
2020-09-17 16:08 ` Will Deacon
2020-09-16 12:48 ` Leon Romanovsky
2020-09-16 8:33 ` Will Deacon
2020-09-16 8:48 ` Catalin Marinas
2020-09-16 14:15 ` Lorenzo Pieralisi
2020-09-16 17:00 ` Catalin Marinas
2020-09-16 21:29 ` Benjamin Herrenschmidt
2020-09-16 12:08 ` Jason Gunthorpe
2020-09-15 23:00 ` Benjamin Herrenschmidt
2020-09-15 23:12 ` Clint Sbisa
2020-09-14 21:41 ` Benjamin Herrenschmidt
2020-08-21 15:51 Clint Sbisa
2020-08-27 14:41 ` Clint Sbisa
2020-08-31 15:22 ` Clint Sbisa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200911214225.hml2wbbq2rofn4re@amazon.com \
--to=csbisa@amazon.com \
--cc=benh@kernel.crashing.org \
--cc=catalin.marinas@arm.com \
--cc=helgaas@kernel.org \
--cc=jgg@nvidia.com \
--cc=leon@kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=lorenzo.pieralisi@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).