linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Will Deacon <will.deacon@arm.com>
To: Christoph Hellwig <hch@infradead.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	Anshuman Khandual <khandual@linux.vnet.ibm.com>,
	virtualization@lists.linux-foundation.org,
	linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
	aik@ozlabs.ru, robh@kernel.org, joe@perches.com,
	elfring@users.sourceforge.net, david@gibson.dropbear.id.au,
	jasowang@redhat.com, mpe@ellerman.id.au, linuxram@us.ibm.com,
	haren@linux.vnet.ibm.com, paulus@samba.org,
	srikar@linux.vnet.ibm.com, robin.murphy@arm.com,
	jean-philippe.brucker@arm.com, marc.zyngier@arm.com
Subject: Re: [RFC 0/4] Virtio uses DMA API for all devices
Date: Wed, 1 Aug 2018 10:05:35 +0100	[thread overview]
Message-ID: <20180801090535.GB14438@arm.com> (raw)
In-Reply-To: <20180801083639.GF26378@infradead.org>

Hi Christoph,

On Wed, Aug 01, 2018 at 01:36:39AM -0700, Christoph Hellwig wrote:
> On Wed, Aug 01, 2018 at 09:16:38AM +0100, Will Deacon wrote:
> > On arm/arm64, the problem we have is that legacy virtio devices on the MMIO
> > transport (so definitely not PCI) have historically been advertised by qemu
> > as not being cache coherent, but because the virtio core has bypassed DMA
> > ops then everything has happened to work. If we blindly enable the arch DMA
> > ops,
> 
> No one is suggesting that as far as I can tell.

Apologies: it's me that wants the DMA ops enabled to handle legacy devices
behind an IOMMU, but see below.

> > we'll plumb in the non-coherent ops and start getting data corruption,
> > so we do need a way to quirk virtio as being "always coherent" if we want to
> > use the DMA ops (which we do, because our emulation platforms have an IOMMU
> > for all virtio devices).
> 
> From all that I've gather so far: no you do not want that.  We really
> need to figure out virtio "dma" interacts with the host / device.
> 
> If you look at the current iommu spec it does talk of physical address
> with a little careveout for VIRTIO_F_IOMMU_PLATFORM.

That's true, although that doesn't exist in the legacy virtio spec, and we
have an existing emulation platform which puts legacy virtio devices behind
an IOMMU. Currently, Linux is unable to boot on this platform unless the
IOMMU is configured as bypass. If we can use the coherent IOMMU DMA ops,
then it works perfectly.

> So between that and our discussion in this thread and its previous
> iterations I think we need to stick to the current always physical,
> bypass system dma ops mode of virtio operation as the default.

As above -- that means we hang during boot because we get stuck trying to
bring up a virtio-block device whose DMA is aborted by the IOMMU. The easy
answer is "just upgrade to latest virtio and advertise the presence of the
IOMMU". I'm pushing for that in future platforms, but it seems a shame not
to support the current platform, especially given that other systems do have
hacks in mainline to get virtio working.

> We just need to figure out how to deal with devices that deviate
> from the default.  One things is that VIRTIO_F_IOMMU_PLATFORM really
> should become VIRTIO_F_PLATFORM_DMA to cover the cases of non-iommu
> dma tweaks (offsets, cache flushing), which seems well in spirit of
> the original design.  The other issue is VIRTIO_F_IO_BARRIER
> which is very vaguely defined, and which needs a better definition.
> And last but not least we'll need some text explaining the challenges
> of hardware devices - I think VIRTIO_F_PLATFORM_DMA + VIRTIO_F_IO_BARRIER
> is what would basically cover them, but a good description including
> an explanation of why these matter.

I agree that this makes sense for future revisions of virtio (or perhaps
it can just be a clarification to virtio 1.0), but we're still left in the
dark with legacy devices and it would be nice to have them work on the
systems which currently exist, even if it's a legacy-only hack in the arch
code.

Will

  reply	other threads:[~2018-08-01  9:05 UTC|newest]

Thread overview: 119+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-07-20  3:59 [RFC 0/4] Virtio uses DMA API for all devices Anshuman Khandual
2018-07-20  3:59 ` [RFC 1/4] virtio: Define virtio_direct_dma_ops structure Anshuman Khandual
2018-07-30  9:24   ` Christoph Hellwig
2018-07-31  4:01     ` Anshuman Khandual
2018-07-20  3:59 ` [RFC 2/4] virtio: Override device's DMA OPS with virtio_direct_dma_ops selectively Anshuman Khandual
2018-07-28  8:56   ` Anshuman Khandual
2018-07-28 21:16     ` Michael S. Tsirkin
2018-07-30  4:15       ` Anshuman Khandual
2018-07-30  9:30       ` Christoph Hellwig
2018-07-31  6:39         ` Anshuman Khandual
2018-07-30  9:25   ` Christoph Hellwig
2018-07-31  7:00     ` Anshuman Khandual
2018-07-20  3:59 ` [RFC 3/4] virtio: Force virtio core to use DMA API callbacks for all virtio devices Anshuman Khandual
2018-07-20  3:59 ` [RFC 4/4] virtio: Add platform specific DMA API translation for virito devices Anshuman Khandual
2018-07-20 13:15   ` Michael S. Tsirkin
2018-07-23  2:16     ` Anshuman Khandual
2018-07-25  4:30       ` Anshuman Khandual
2018-07-25 13:31       ` Michael S. Tsirkin
2018-07-20 13:16 ` [RFC 0/4] Virtio uses DMA API for all devices Michael S. Tsirkin
2018-07-23  6:28   ` Anshuman Khandual
2018-07-23  9:08     ` Michael S. Tsirkin
2018-07-25  3:26       ` Anshuman Khandual
2018-07-27 11:31         ` Michael S. Tsirkin
2018-07-28  8:37           ` Anshuman Khandual
2018-07-27  9:58 ` Will Deacon
2018-07-27 10:58   ` Anshuman Khandual
2018-07-30  9:34   ` Christoph Hellwig
2018-07-30 10:28     ` Michael S. Tsirkin
2018-07-30 11:18       ` Christoph Hellwig
2018-07-30 13:26         ` Michael S. Tsirkin
2018-07-31 17:30           ` Christoph Hellwig
2018-07-31 20:36             ` Benjamin Herrenschmidt
2018-08-01  8:16               ` Will Deacon
2018-08-01  8:36                 ` Christoph Hellwig
2018-08-01  9:05                   ` Will Deacon [this message]
2018-08-01 22:41                     ` Michael S. Tsirkin
2018-08-01 22:35                   ` Michael S. Tsirkin
2018-08-02 15:24                   ` Benjamin Herrenschmidt
2018-08-02 15:41                     ` Michael S. Tsirkin
2018-08-02 16:01                       ` Benjamin Herrenschmidt
2018-08-02 17:19                         ` Michael S. Tsirkin
2018-08-02 17:53                           ` Benjamin Herrenschmidt
2018-08-02 20:52                             ` Michael S. Tsirkin
2018-08-02 21:13                               ` Benjamin Herrenschmidt
2018-08-02 21:51                                 ` Michael S. Tsirkin
2018-08-03  7:05                                 ` Christoph Hellwig
2018-08-03 15:58                                   ` Benjamin Herrenschmidt
2018-08-03 16:02                                     ` Christoph Hellwig
2018-08-03 18:58                                       ` Benjamin Herrenschmidt
2018-08-04  8:21                                         ` Christoph Hellwig
2018-08-05  1:10                                           ` Benjamin Herrenschmidt
2018-08-05  7:29                                             ` Christoph Hellwig
2018-08-05 21:16                                               ` Benjamin Herrenschmidt
2018-08-05 21:30                                                 ` Benjamin Herrenschmidt
2018-08-06  9:42                                                 ` Christoph Hellwig
2018-08-06 19:52                                                   ` Benjamin Herrenschmidt
2018-08-07  6:21                                                     ` Christoph Hellwig
2018-08-07  6:42                                                       ` Benjamin Herrenschmidt
2018-08-07 13:55                                                         ` Christoph Hellwig
2018-08-07 20:32                                                           ` Benjamin Herrenschmidt
2018-08-08  6:31                                                             ` Christoph Hellwig
2018-08-08 10:07                                                               ` Benjamin Herrenschmidt
2018-08-08 12:30                                                                 ` Christoph Hellwig
2018-08-08 13:18                                                                   ` Benjamin Herrenschmidt
2018-08-08 20:31                                                                     ` Michael S. Tsirkin
2018-08-08 22:13                                                                       ` Benjamin Herrenschmidt
2018-08-09  2:00                                                                         ` Benjamin Herrenschmidt
2018-08-09  5:40                                                                         ` Christoph Hellwig
2018-09-07  0:09                                                                           ` Jiandi An
2018-09-10  6:19                                                                             ` Christoph Hellwig
2018-09-10  8:53                                                                               ` Gerd Hoffmann
2018-08-03 19:07                                     ` Michael S. Tsirkin
2018-08-04  1:11                                       ` Benjamin Herrenschmidt
2018-08-04  1:16                                       ` Benjamin Herrenschmidt
2018-08-05  0:22                                         ` Michael S. Tsirkin
2018-08-05  4:52                                           ` Benjamin Herrenschmidt
2018-08-06 13:46                                             ` Michael S. Tsirkin
2018-08-06 19:56                                               ` Benjamin Herrenschmidt
2018-08-06 20:35                                                 ` Michael S. Tsirkin
2018-08-06 21:26                                                   ` Benjamin Herrenschmidt
2018-08-06 21:46                                                     ` Michael S. Tsirkin
2018-08-06 22:13                                                       ` Benjamin Herrenschmidt
2018-08-06 23:16                                                         ` Benjamin Herrenschmidt
2018-08-06 23:45                                                         ` Michael S. Tsirkin
2018-08-07  0:18                                                           ` Benjamin Herrenschmidt
2018-08-07  6:32                                                           ` Christoph Hellwig
2018-08-07  6:27                                                         ` Christoph Hellwig
2018-08-07  6:44                                                           ` Benjamin Herrenschmidt
2018-08-07  6:18                                                       ` Christoph Hellwig
2018-08-07  6:16                                                     ` Christoph Hellwig
2018-08-06 23:18                                                   ` Benjamin Herrenschmidt
2018-08-07  6:12                                                   ` Christoph Hellwig
2018-08-04  1:18                                       ` Benjamin Herrenschmidt
2018-08-04  1:22                                       ` Benjamin Herrenschmidt
2018-08-05  0:23                                         ` Michael S. Tsirkin
2018-08-03 19:17                                   ` Michael S. Tsirkin
2018-08-04  8:15                                     ` Christoph Hellwig
2018-08-05  0:09                                       ` Michael S. Tsirkin
2018-08-05  1:11                                         ` Benjamin Herrenschmidt
2018-08-05  7:25                                         ` Christoph Hellwig
2018-08-05  0:53                                       ` Benjamin Herrenschmidt
2018-08-05  0:27                 ` Michael S. Tsirkin
2018-08-06 14:05                   ` Will Deacon
2018-08-01 21:56               ` Michael S. Tsirkin
2018-08-02 15:33                 ` Benjamin Herrenschmidt
2018-08-02 20:53                   ` Michael S. Tsirkin
2018-08-03  7:06                     ` Christoph Hellwig
2018-08-02 20:55 ` Michael S. Tsirkin
2018-08-03  2:41   ` Jason Wang
2018-08-03 19:08     ` Michael S. Tsirkin
2018-08-04  1:21       ` Benjamin Herrenschmidt
2018-08-05  0:24         ` Michael S. Tsirkin
2018-08-06  9:02           ` Anshuman Khandual
2018-08-06 13:36             ` Michael S. Tsirkin
2018-08-06 15:24               ` Christoph Hellwig
2018-08-06 16:06                 ` Michael S. Tsirkin
2018-08-06 16:10                   ` Christoph Hellwig
2018-08-06 16:13                     ` Michael S. Tsirkin
2018-08-06 16:34                       ` Christoph Hellwig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180801090535.GB14438@arm.com \
    --to=will.deacon@arm.com \
    --cc=aik@ozlabs.ru \
    --cc=benh@kernel.crashing.org \
    --cc=david@gibson.dropbear.id.au \
    --cc=elfring@users.sourceforge.net \
    --cc=haren@linux.vnet.ibm.com \
    --cc=hch@infradead.org \
    --cc=jasowang@redhat.com \
    --cc=jean-philippe.brucker@arm.com \
    --cc=joe@perches.com \
    --cc=khandual@linux.vnet.ibm.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=linuxram@us.ibm.com \
    --cc=marc.zyngier@arm.com \
    --cc=mpe@ellerman.id.au \
    --cc=mst@redhat.com \
    --cc=paulus@samba.org \
    --cc=robh@kernel.org \
    --cc=robin.murphy@arm.com \
    --cc=srikar@linux.vnet.ibm.com \
    --cc=virtualization@lists.linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).