From: Robin Murphy <robin.murphy@arm.com>
To: John Garry <john.garry@huawei.com>,
joro@8bytes.org, will@kernel.org, jejb@linux.ibm.com,
martin.petersen@oracle.com, hch@lst.de, m.szyprowski@samsung.com
Cc: iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
linux-scsi@vger.kernel.org, linuxarm@huawei.com
Subject: Re: [PATCH 3/6] iova: Allow rcache range upper limit to be configurable
Date: Wed, 31 Mar 2021 11:53:34 +0100 [thread overview]
Message-ID: <e2d873d9-3529-caff-d4ae-cca456857ff1@arm.com> (raw)
In-Reply-To: <3375b67f-438c-32d3-a5a6-7e08f37b04e3@huawei.com>
On 2021-03-19 17:26, John Garry wrote:
[...]
>>> @@ -25,7 +25,8 @@ struct iova {
>>> struct iova_magazine;
>>> struct iova_cpu_rcache;
>>> -#define IOVA_RANGE_CACHE_MAX_SIZE 6 /* log of max cached IOVA
>>> range size (in pages) */
>>> +#define IOVA_RANGE_CACHE_DEFAULT_SIZE 6
>>> +#define IOVA_RANGE_CACHE_MAX_SIZE 10 /* log of max cached IOVA range
>>> size (in pages) */
>>
>> No.
>>
>> And why? If we're going to allocate massive caches anyway, whatever is
>> the point of *not* using all of them?
>>
>
> I wanted to keep the same effective threshold for devices today, unless
> set otherwise.
>
> The reason is that I didn't know if a blanket increase would cause
> regressions, and I was taking the super-safe road. Specifically some
> systems may be very IOVA space limited, and just work today by not
> caching large IOVAs.
alloc_iova_fast() will already clear out the caches if space is running
low, so the caching itself shouldn't be an issue.
> And in the precursor thread you wrote "We can't arbitrarily *increase*
> the scope of caching once a domain is active due to the size-rounding-up
> requirement, which would be prohibitive to larger allocations if applied
> universally" (sorry for quoting)
>
> I took the last part to mean that we shouldn't apply this increase in
> threshold globally.
I meant we can't increase the caching threshold as-is once the domain is
in use, because that could result in odd-sized IOVAs previously
allocated above the old threshold being later freed back into caches,
then causing havoc the next time they get allocated (because they're not
as big as the actual size being asked for). However, trying to address
that by just size-aligning everything even above the caching threshold
is liable to waste too much space on IOVA-constrained systems (e.g. a
single 4K video frame may be ~35MB - rounding that up to 64MB each time
would be hard to justify).
It follows from that that there's really no point in decoupling the
rounding-up threshold from the actual caching threshold - you get all
the wastage (both IOVA space and actual memory for the cache arrays) for
no obvious benefit.
>> It only makes sense for a configuration knob to affect the actual
>> rcache and depot allocations - that way, big high-throughput systems
>> with plenty of memory can spend it on better performance, while small
>> systems - that often need IOMMU scatter-gather precisely *because*
>> memory is tight and thus easily fragmented - don't have to pay the
>> (not insignificant) cost for caches they don't need.
>
> So do you suggest to just make IOVA_RANGE_CACHE_MAX_SIZE a kconfig option?
Again, I'm less convinced by Kconfig since I imagine many people tuning
server-class systems for their own particular workloads will be running
standard enterprise distros, so I think end-user-accessible knobs will
be the most valuable. That's not to say that a Kconfig option to set the
default state of a command-line option (as we do elsewhere) won't be
useful for embedded users, cloud providers, etc., just that I'm not sure
it's worth it being the *only* option.
Robin.
next prev parent reply other threads:[~2021-03-31 10:54 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-19 13:25 [PATCH 0/6] dma mapping/iommu: Allow IOMMU IOVA rcache range to be configured John Garry
2021-03-19 13:25 ` [PATCH 1/6] iommu: Move IOVA power-of-2 roundup into allocator John Garry
2021-03-19 16:13 ` Robin Murphy
2021-03-19 16:58 ` John Garry
2021-03-19 19:20 ` Robin Murphy
2021-03-22 15:01 ` John Garry
2021-03-31 9:58 ` Robin Murphy
2021-04-06 16:54 ` John Garry
2021-04-14 17:44 ` John Garry
2021-03-19 13:25 ` [PATCH 2/6] iova: Add a per-domain count of reserved nodes John Garry
2021-03-19 13:25 ` [PATCH 3/6] iova: Allow rcache range upper limit to be configurable John Garry
2021-03-19 16:25 ` Robin Murphy
2021-03-19 17:26 ` John Garry
2021-03-31 10:53 ` Robin Murphy [this message]
2021-03-19 13:25 ` [PATCH 4/6] iommu: Add iommu_dma_set_opt_size() John Garry
2021-03-19 13:25 ` [PATCH 5/6] dma-mapping/iommu: Add dma_set_max_opt_size() John Garry
2021-03-19 17:00 ` Robin Murphy
2021-03-19 18:02 ` John Garry
2021-03-31 8:01 ` Salil Mehta
2021-03-31 8:08 ` Salil Mehta
2021-03-19 13:25 ` [PATCH 6/6] scsi: hisi_sas: Set max optimal DMA size for v3 hw John Garry
2021-03-19 13:40 ` [PATCH 0/6] dma mapping/iommu: Allow IOMMU IOVA rcache range to be configured Christoph Hellwig
2021-03-19 15:42 ` John Garry
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e2d873d9-3529-caff-d4ae-cca456857ff1@arm.com \
--to=robin.murphy@arm.com \
--cc=hch@lst.de \
--cc=iommu@lists.linux-foundation.org \
--cc=jejb@linux.ibm.com \
--cc=john.garry@huawei.com \
--cc=joro@8bytes.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=linuxarm@huawei.com \
--cc=m.szyprowski@samsung.com \
--cc=martin.petersen@oracle.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).