All of lore.kernel.org
 help / color / mirror / Atom feed
From: David Stevens <stevensd@chromium.org>
To: Lu Baolu <baolu.lu@linux.intel.com>
Cc: Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
	Sergey Senozhatsky <senozhatsky@chromium.org>,
	iommu@lists.linux-foundation.org,
	David Stevens <stevensd@google.com>,
	Christoph Hellwig <hch@lst.de>,
	open list <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 0/4] Add dynamic iommu backed bounce buffers
Date: Fri, 9 Jul 2021 15:04:47 +0900	[thread overview]
Message-ID: <CAD=HUj4_mtTPbXBqQke=Q+zK0EuJZEeWOiVkhphAUfvK-DMHVg@mail.gmail.com> (raw)
In-Reply-To: <f6541f4e-648e-d9a0-eda7-b2a117978ebb@linux.intel.com>

On Thu, Jul 8, 2021 at 10:38 PM Lu Baolu <baolu.lu@linux.intel.com> wrote:
>
> Hi David,
>
> I like this idea. Thanks for proposing this.
>
> On 2021/7/7 15:55, David Stevens wrote:
> > Add support for per-domain dynamic pools of iommu bounce buffers to the
> > dma-iommu API. This allows iommu mappings to be reused while still
> > maintaining strict iommu protection. Allocating buffers dynamically
> > instead of using swiotlb carveouts makes per-domain pools more amenable
> > on systems with large numbers of devices or where devices are unknown.
>
> Have you ever considered leveraging the per-device swiotlb memory pool
> added by below series?
>
> https://lore.kernel.org/linux-iommu/20210625123004.GA3170@willie-the-truck/

I'm not sure if that's a good fit. The swiotlb pools are allocated
during device initialization, so they require setting aside the
worst-case amount of memory. That's okay if you only use it with a
small number of devices where you know in advance approximately how
much memory they use. However, it doesn't work as well if you want to
use it with a large number of devices, or with unknown (i.e.
hotplugged) devices.

> >
> > When enabled, all non-direct streaming mappings below a configurable
> > size will go through bounce buffers. Note that this means drivers which
> > don't properly use the DMA API (e.g. i915) cannot use an iommu when this
> > feature is enabled. However, all drivers which work with swiotlb=force
> > should work.
>
> If so, why not making it more scalable by adding a callback into vendor
> iommu drivers? The vendor iommu drivers have enough information to tell
> whether the bounce buffer is feasible for a specific domain.

I'm not very familiar with the specifics of VT-d or restrictions with
the graphics hardware, but at least on the surface it looks like a
limitation of the i915 driver's implementation. The driver uses the
DMA_ATTR_SKIP_CPU_SYNC flag, but never calls the dma_sync functions,
since things are coherent on x86 hardware. However, bounce buffers
violate the driver's assumption that there's no need to sync the CPU
and device domain. I doubt there's an inherent limitation of the
hardware here, it's just how the driver is implemented. Given that, I
don't know if it's something the iommu driver needs to handle.

One potential way this could be addressed would be to add explicit
support to the DMA API for long-lived streaming mappings. Drivers can
get that behavior today via DMA_ATTR_SKIP_CPU_SYNC and dma_sync.
However, the DMA API doesn't really have enough information to treat
ephemeral and long-lived mappings differently. With a new DMA_ATTR
flag for long-lived streaming mappings, the DMA API could skip bounce
buffers. That flag could also be used as a performance optimization in
the various dma-buf implementations, since they seem to mostly fall
into the long-lived streaming category (the handful I checked do call
dma_sync, so there isn't a correctness issue).

-David

> >
> > Bounce buffers serve as an optimization in situations where interactions
> > with the iommu are very costly. For example, virtio-iommu operations in
>
> The simulated IOMMU does the same thing.
>
> It's also an optimization for bare metal in cases where the strict mode
> of cache invalidation is used. CPU moving data is faster than IOMMU
> cache invalidation if the buffer is small.
>
> Best regards,
> baolu

WARNING: multiple messages have this Message-ID
From: David Stevens <stevensd@chromium.org>
To: Lu Baolu <baolu.lu@linux.intel.com>
Cc: David Stevens <stevensd@google.com>,
	open list <linux-kernel@vger.kernel.org>,
	Sergey Senozhatsky <senozhatsky@chromium.org>,
	iommu@lists.linux-foundation.org, Will Deacon <will@kernel.org>,
	Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH 0/4] Add dynamic iommu backed bounce buffers
Date: Fri, 9 Jul 2021 15:04:47 +0900	[thread overview]
Message-ID: <CAD=HUj4_mtTPbXBqQke=Q+zK0EuJZEeWOiVkhphAUfvK-DMHVg@mail.gmail.com> (raw)
In-Reply-To: <f6541f4e-648e-d9a0-eda7-b2a117978ebb@linux.intel.com>

On Thu, Jul 8, 2021 at 10:38 PM Lu Baolu <baolu.lu@linux.intel.com> wrote:
>
> Hi David,
>
> I like this idea. Thanks for proposing this.
>
> On 2021/7/7 15:55, David Stevens wrote:
> > Add support for per-domain dynamic pools of iommu bounce buffers to the
> > dma-iommu API. This allows iommu mappings to be reused while still
> > maintaining strict iommu protection. Allocating buffers dynamically
> > instead of using swiotlb carveouts makes per-domain pools more amenable
> > on systems with large numbers of devices or where devices are unknown.
>
> Have you ever considered leveraging the per-device swiotlb memory pool
> added by below series?
>
> https://lore.kernel.org/linux-iommu/20210625123004.GA3170@willie-the-truck/

I'm not sure if that's a good fit. The swiotlb pools are allocated
during device initialization, so they require setting aside the
worst-case amount of memory. That's okay if you only use it with a
small number of devices where you know in advance approximately how
much memory they use. However, it doesn't work as well if you want to
use it with a large number of devices, or with unknown (i.e.
hotplugged) devices.

> >
> > When enabled, all non-direct streaming mappings below a configurable
> > size will go through bounce buffers. Note that this means drivers which
> > don't properly use the DMA API (e.g. i915) cannot use an iommu when this
> > feature is enabled. However, all drivers which work with swiotlb=force
> > should work.
>
> If so, why not making it more scalable by adding a callback into vendor
> iommu drivers? The vendor iommu drivers have enough information to tell
> whether the bounce buffer is feasible for a specific domain.

I'm not very familiar with the specifics of VT-d or restrictions with
the graphics hardware, but at least on the surface it looks like a
limitation of the i915 driver's implementation. The driver uses the
DMA_ATTR_SKIP_CPU_SYNC flag, but never calls the dma_sync functions,
since things are coherent on x86 hardware. However, bounce buffers
violate the driver's assumption that there's no need to sync the CPU
and device domain. I doubt there's an inherent limitation of the
hardware here, it's just how the driver is implemented. Given that, I
don't know if it's something the iommu driver needs to handle.

One potential way this could be addressed would be to add explicit
support to the DMA API for long-lived streaming mappings. Drivers can
get that behavior today via DMA_ATTR_SKIP_CPU_SYNC and dma_sync.
However, the DMA API doesn't really have enough information to treat
ephemeral and long-lived mappings differently. With a new DMA_ATTR
flag for long-lived streaming mappings, the DMA API could skip bounce
buffers. That flag could also be used as a performance optimization in
the various dma-buf implementations, since they seem to mostly fall
into the long-lived streaming category (the handful I checked do call
dma_sync, so there isn't a correctness issue).

-David

> >
> > Bounce buffers serve as an optimization in situations where interactions
> > with the iommu are very costly. For example, virtio-iommu operations in
>
> The simulated IOMMU does the same thing.
>
> It's also an optimization for bare metal in cases where the strict mode
> of cache invalidation is used. CPU moving data is faster than IOMMU
> cache invalidation if the buffer is small.
>
> Best regards,
> baolu
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

  reply	other threads:[~2021-07-09  6:05 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-07  7:55 David Stevens
2021-07-07  7:55 ` David Stevens
2021-07-07  7:55 ` [PATCH 1/4] dma-iommu: add kalloc gfp flag to alloc helper David Stevens
2021-07-07  7:55   ` David Stevens
2021-07-08 17:22   ` Robin Murphy
2021-07-08 17:22     ` Robin Murphy
2021-07-07  7:55 ` [PATCH 2/4] dma-iommu: replace device arguments David Stevens
2021-07-07  7:55   ` David Stevens
2021-07-07  7:55 ` [PATCH 3/4] dma-iommu: expose a few helper functions to module David Stevens
2021-07-07  7:55   ` David Stevens
2021-07-07  7:55 ` [PATCH 4/4] dma-iommu: Add iommu bounce buffers to dma-iommu api David Stevens
2021-07-07  7:55   ` David Stevens
2021-07-08  9:29 ` [PATCH 0/4] Add dynamic iommu backed bounce buffers Joerg Roedel
2021-07-08  9:29   ` Joerg Roedel
2021-07-08 17:14   ` Robin Murphy
2021-07-08 17:14     ` Robin Murphy
2021-07-09  7:25     ` David Stevens
2021-07-09  7:25       ` David Stevens
2021-07-08 13:38 ` Lu Baolu
2021-07-08 13:38   ` Lu Baolu
2021-07-09  6:04   ` David Stevens [this message]
2021-07-09  6:04     ` David Stevens

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAD=HUj4_mtTPbXBqQke=Q+zK0EuJZEeWOiVkhphAUfvK-DMHVg@mail.gmail.com' \
    --to=stevensd@chromium.org \
    --cc=baolu.lu@linux.intel.com \
    --cc=hch@lst.de \
    --cc=iommu@lists.linux-foundation.org \
    --cc=joro@8bytes.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=senozhatsky@chromium.org \
    --cc=stevensd@google.com \
    --cc=will@kernel.org \
    --subject='Re: [PATCH 0/4] Add dynamic iommu backed bounce buffers' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.