From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754382AbeDZJJy (ORCPT ); Thu, 26 Apr 2018 05:09:54 -0400 Received: from bombadil.infradead.org ([198.137.202.133]:43168 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753780AbeDZJJt (ORCPT ); Thu, 26 Apr 2018 05:09:49 -0400 Date: Thu, 26 Apr 2018 02:09:42 -0700 From: Christoph Hellwig To: Daniel Vetter Cc: Christoph Hellwig , Thierry Reding , Christian =?iso-8859-1?Q?K=F6nig?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Linux Kernel Mailing List , amd-gfx list , Jerome Glisse , dri-devel , Dan Williams , Logan Gunthorpe , "open list:DMA BUFFER SHARING FRAMEWORK" , iommu@lists.linux-foundation.org, Linux ARM Subject: Re: noveau vs arm dma ops Message-ID: <20180426090942.GA18811@infradead.org> References: <20180425054855.GA17038@infradead.org> <20180425064335.GB28100@infradead.org> <20180425074151.GA2271@ulmo> <20180425085439.GA29996@infradead.org> <20180425100429.GR25142@phenom.ffwll.local> <20180425153312.GD27076@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Apr 25, 2018 at 11:35:13PM +0200, Daniel Vetter wrote: > > get_required_mask() is supposed to tell you if you are safe. However > > we are missing lots of implementations of it for iommus so you might get > > some false negatives, improvements welcome. It's been on my list of > > things to fix in the DMA API, but it is nowhere near the top. > > I hasn't come up in a while in some fireworks, so I honestly don't > remember exactly what the issues have been. But > > commit d766ef53006c2c38a7fe2bef0904105a793383f2 > Author: Chris Wilson > Date: Mon Dec 19 12:43:45 2016 +0000 > > drm/i915: Fallback to single PAGE_SIZE segments for DMA remapping > > and the various bits of code that a > > $ git grep SWIOTLB -- drivers/gpu > > turns up is what we're doing to hack around that stuff. And in general > (there's some exceptions) gpus should be able to address everything, > so I never fully understood where that's even coming from. I'm pretty sure I've seen some oddly low dma masks in GPU drivers. E.g. duplicated in various AMD files: adev->need_dma32 = false; dma_bits = adev->need_dma32 ? 32 : 40; r = pci_set_dma_mask(adev->pdev, DMA_BIT_MASK(dma_bits)); if (r) { adev->need_dma32 = true; dma_bits = 32; dev_warn(adev->dev, "amdgpu: No suitable DMA available.\n"); } synopsis: drivers/gpu/drm/bridge/synopsys/dw-hdmi-i2s-audio.c: pdevinfo.dma_mask = DMA_BIT_MASK(32); drivers/gpu/drm/bridge/synopsys/dw-hdmi.c: pdevinfo.dma_mask = DMA_BIT_MASK(32); drivers/gpu/drm/bridge/synopsys/dw-hdmi.c: pdevinfo.dma_mask = DMA_BIT_MASK(32); etnaviv gets it right: drivers/gpu/drm/etnaviv/etnaviv_gpu.c: u32 dma_mask = (u32)dma_get_required_mask(gpu->dev); But yes, the swiotlb hackery really irks me. I just have some more important and bigger fires to fight first, but I plan to get back to the root cause of that eventually. > > >> - dma api hides the cache flushing requirements from us. GPUs love > >> non-snooped access, and worse give userspace control over that. We want > >> a strict separation between mapping stuff and flushing stuff. With the > >> IOMMU api we mostly have the former, but for the later arch maintainers > >> regularly tells they won't allow that. So we have drm_clflush.c. > > > > The problem is that a cache flushing API entirely separate is hard. That > > being said if you look at my generic dma-noncoherent API series it tries > > to move that way. So far it is in early stages and apparently rather > > buggy unfortunately. > > I'm assuming this stuff here? > > https://lkml.org/lkml/2018/4/20/146 > > Anyway got lost in all that work a bit, looks really nice. That url doesn't seem to work currently. But I am talking about the thread titled '[RFC] common non-cache coherent direct dma mapping ops' > Yeah the above is pretty much what we do on x86. dma-api believes > everything is coherent, so dma_map_sg does the mapping we want and > nothing else (minus swiotlb fun). Cache flushing, allocations, all > done by the driver. Which sounds like the right thing to do to me. > On arm that doesn't work. The iommu api seems like a good fit, except > the dma-api tends to get in the way a bit (drm/msm apparently has > similar problems like tegra), and if you need contiguous memory > dma_alloc_coherent is the only way to get at contiguous memory. There > was a huge discussion years ago about that, and direct cma access was > shot down because it would have exposed too much of the caching > attribute mangling required (most arm platforms need wc-pages to not > be in the kernel's linear map apparently). Simple cma_alloc() doesn't do anything about cache handling, it just is a very dumb allocator for large contiguous regions inside a big pool. I'm not the CMA maintainer, but in general I'd love to see an EXPORT_SYMBOL_GPL slapped onto cma_alloc/release and drivers use that were needed. Using that plus dma_map*/dma_unmap* sounds like a much saner interface than dma_alloc_attrs + DMA_ATTR_NON_CONSISTENT or DMA_ATTR_NO_KERNEL_MAPPING. You don't happen to have a pointer to that previous discussion? > Anything that separate these 3 things more (allocation pools, mapping > through IOMMUs and flushing cpu caches) sounds like the right > direction to me. Even if that throws some portability across platforms > away - drivers who want to control things in this much detail aren't > really portable (without some serious work) anyway. As long as we stay away from the dma coherent allocations separating them is fine, and I'm working towards it. dma coherent allocations on the other hand are very hairy beasts, and provide by very different implementations depending on the architecture, or often even depending on the specifics of individual implementations inside the architecture. So for your GPU/media case it seems like you'd better stay away from them as much as you can. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoph Hellwig Subject: Re: noveau vs arm dma ops Date: Thu, 26 Apr 2018 02:09:42 -0700 Message-ID: <20180426090942.GA18811@infradead.org> References: <20180425054855.GA17038@infradead.org> <20180425064335.GB28100@infradead.org> <20180425074151.GA2271@ulmo> <20180425085439.GA29996@infradead.org> <20180425100429.GR25142@phenom.ffwll.local> <20180425153312.GD27076@infradead.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Daniel Vetter Cc: "moderated list:DMA BUFFER SHARING FRAMEWORK" , Linux Kernel Mailing List , amd-gfx list , Christoph Hellwig , Jerome Glisse , iommu@lists.linux-foundation.org, dri-devel , Dan Williams , Thierry Reding , Logan Gunthorpe , Christian =?iso-8859-1?Q?K=F6nig?= , Linux ARM , "open list:DMA BUFFER SHARING FRAMEWORK" List-Id: iommu@lists.linux-foundation.org T24gV2VkLCBBcHIgMjUsIDIwMTggYXQgMTE6MzU6MTNQTSArMDIwMCwgRGFuaWVsIFZldHRlciB3 cm90ZToKPiA+IGdldF9yZXF1aXJlZF9tYXNrKCkgaXMgc3VwcG9zZWQgdG8gdGVsbCB5b3UgaWYg eW91IGFyZSBzYWZlLiAgSG93ZXZlcgo+ID4gd2UgYXJlIG1pc3NpbmcgbG90cyBvZiBpbXBsZW1l bnRhdGlvbnMgb2YgaXQgZm9yIGlvbW11cyBzbyB5b3UgbWlnaHQgZ2V0Cj4gPiBzb21lIGZhbHNl IG5lZ2F0aXZlcywgaW1wcm92ZW1lbnRzIHdlbGNvbWUuICBJdCdzIGJlZW4gb24gbXkgbGlzdCBv Zgo+ID4gdGhpbmdzIHRvIGZpeCBpbiB0aGUgRE1BIEFQSSwgYnV0IGl0IGlzIG5vd2hlcmUgbmVh ciB0aGUgdG9wLgo+IAo+IEkgaGFzbid0IGNvbWUgdXAgaW4gYSB3aGlsZSBpbiBzb21lIGZpcmV3 b3Jrcywgc28gSSBob25lc3RseSBkb24ndAo+IHJlbWVtYmVyIGV4YWN0bHkgd2hhdCB0aGUgaXNz dWVzIGhhdmUgYmVlbi4gQnV0Cj4gCj4gY29tbWl0IGQ3NjZlZjUzMDA2YzJjMzhhN2ZlMmJlZjA5 MDQxMDVhNzkzMzgzZjIKPiBBdXRob3I6IENocmlzIFdpbHNvbiA8Y2hyaXNAY2hyaXMtd2lsc29u LmNvLnVrPgo+IERhdGU6ICAgTW9uIERlYyAxOSAxMjo0Mzo0NSAyMDE2ICswMDAwCj4gCj4gICAg IGRybS9pOTE1OiBGYWxsYmFjayB0byBzaW5nbGUgUEFHRV9TSVpFIHNlZ21lbnRzIGZvciBETUEg cmVtYXBwaW5nCj4gCj4gYW5kIHRoZSB2YXJpb3VzIGJpdHMgb2YgY29kZSB0aGF0IGEKPiAKPiAk IGdpdCBncmVwIFNXSU9UTEIgLS0gZHJpdmVycy9ncHUKPiAKPiB0dXJucyB1cCBpcyB3aGF0IHdl J3JlIGRvaW5nIHRvIGhhY2sgYXJvdW5kIHRoYXQgc3R1ZmYuIEFuZCBpbiBnZW5lcmFsCj4gKHRo ZXJlJ3Mgc29tZSBleGNlcHRpb25zKSBncHVzIHNob3VsZCBiZSBhYmxlIHRvIGFkZHJlc3MgZXZl cnl0aGluZywKPiBzbyBJIG5ldmVyIGZ1bGx5IHVuZGVyc3Rvb2Qgd2hlcmUgdGhhdCdzIGV2ZW4g Y29taW5nIGZyb20uCgpJJ20gcHJldHR5IHN1cmUgSSd2ZSBzZWVuIHNvbWUgb2RkbHkgbG93IGRt YSBtYXNrcyBpbiBHUFUgZHJpdmVycy4gIEUuZy4KZHVwbGljYXRlZCBpbiB2YXJpb3VzIEFNRCBm aWxlczoKCglhZGV2LT5uZWVkX2RtYTMyID0gZmFsc2U7CiAgICAgICAgZG1hX2JpdHMgPSBhZGV2 LT5uZWVkX2RtYTMyID8gMzIgOiA0MDsKICAgICAgICByID0gcGNpX3NldF9kbWFfbWFzayhhZGV2 LT5wZGV2LCBETUFfQklUX01BU0soZG1hX2JpdHMpKTsKICAgICAgICBpZiAocikgewogICAgICAg ICAgICAgICAgYWRldi0+bmVlZF9kbWEzMiA9IHRydWU7CiAgICAgICAgICAgICAgICBkbWFfYml0 cyA9IDMyOwogICAgICAgICAgICAgICAgZGV2X3dhcm4oYWRldi0+ZGV2LCAiYW1kZ3B1OiBObyBz dWl0YWJsZSBETUEgYXZhaWxhYmxlLlxuIik7CiAgICAgICAgfQoKc3lub3BzaXM6Cgpkcml2ZXJz L2dwdS9kcm0vYnJpZGdlL3N5bm9wc3lzL2R3LWhkbWktaTJzLWF1ZGlvLmM6ICAgIHBkZXZpbmZv LmRtYV9tYXNrICAgICAgID0gRE1BX0JJVF9NQVNLKDMyKTsKZHJpdmVycy9ncHUvZHJtL2JyaWRn ZS9zeW5vcHN5cy9kdy1oZG1pLmM6ICAgICAgICAgICAgICBwZGV2aW5mby5kbWFfbWFzayA9IERN QV9CSVRfTUFTSygzMik7CmRyaXZlcnMvZ3B1L2RybS9icmlkZ2Uvc3lub3BzeXMvZHctaGRtaS5j OiAgICAgICAgICAgICAgcGRldmluZm8uZG1hX21hc2sgPSBETUFfQklUX01BU0soMzIpOwoKZXRu YXZpdiBnZXRzIGl0IHJpZ2h0OgoKZHJpdmVycy9ncHUvZHJtL2V0bmF2aXYvZXRuYXZpdl9ncHUu YzogICAgICAgICAgdTMyIGRtYV9tYXNrID0gKHUzMilkbWFfZ2V0X3JlcXVpcmVkX21hc2soZ3B1 LT5kZXYpOwoKCkJ1dCB5ZXMsIHRoZSBzd2lvdGxiIGhhY2tlcnkgcmVhbGx5IGlya3MgbWUuICBJ IGp1c3QgaGF2ZSBzb21lIG1vcmUKaW1wb3J0YW50IGFuZCBiaWdnZXIgZmlyZXMgdG8gZmlnaHQg Zmlyc3QsIGJ1dCBJIHBsYW4gdG8gZ2V0IGJhY2sgdG8gdGhlCnJvb3QgY2F1c2Ugb2YgdGhhdCBl dmVudHVhbGx5LgoKPiAKPiA+PiAtIGRtYSBhcGkgaGlkZXMgdGhlIGNhY2hlIGZsdXNoaW5nIHJl cXVpcmVtZW50cyBmcm9tIHVzLiBHUFVzIGxvdmUKPiA+PiAgIG5vbi1zbm9vcGVkIGFjY2Vzcywg YW5kIHdvcnNlIGdpdmUgdXNlcnNwYWNlIGNvbnRyb2wgb3ZlciB0aGF0LiBXZSB3YW50Cj4gPj4g ICBhIHN0cmljdCBzZXBhcmF0aW9uIGJldHdlZW4gbWFwcGluZyBzdHVmZiBhbmQgZmx1c2hpbmcg c3R1ZmYuIFdpdGggdGhlCj4gPj4gICBJT01NVSBhcGkgd2UgbW9zdGx5IGhhdmUgdGhlIGZvcm1l ciwgYnV0IGZvciB0aGUgbGF0ZXIgYXJjaCBtYWludGFpbmVycwo+ID4+ICAgcmVndWxhcmx5IHRl bGxzIHRoZXkgd29uJ3QgYWxsb3cgdGhhdC4gU28gd2UgaGF2ZSBkcm1fY2xmbHVzaC5jLgo+ID4K PiA+IFRoZSBwcm9ibGVtIGlzIHRoYXQgYSBjYWNoZSBmbHVzaGluZyBBUEkgZW50aXJlbHkgc2Vw YXJhdGUgaXMgaGFyZC4gVGhhdAo+ID4gYmVpbmcgc2FpZCBpZiB5b3UgbG9vayBhdCBteSBnZW5l cmljIGRtYS1ub25jb2hlcmVudCBBUEkgc2VyaWVzIGl0IHRyaWVzCj4gPiB0byBtb3ZlIHRoYXQg d2F5LiAgU28gZmFyIGl0IGlzIGluIGVhcmx5IHN0YWdlcyBhbmQgYXBwYXJlbnRseSByYXRoZXIK PiA+IGJ1Z2d5IHVuZm9ydHVuYXRlbHkuCj4gCj4gSSdtIGFzc3VtaW5nIHRoaXMgc3R1ZmYgaGVy ZT8KPiAKPiBodHRwczovL2xrbWwub3JnL2xrbWwvMjAxOC80LzIwLzE0Ngo+IAo+IEFueXdheSBn b3QgbG9zdCBpbiBhbGwgdGhhdCB3b3JrIGEgYml0LCBsb29rcyByZWFsbHkgbmljZS4KClRoYXQg dXJsIGRvZXNuJ3Qgc2VlbSB0byB3b3JrIGN1cnJlbnRseS4gIEJ1dCBJIGFtIHRhbGtpbmcgYWJv dXQgdGhlCnRocmVhZCB0aXRsZWQgJ1tSRkNdIGNvbW1vbiBub24tY2FjaGUgY29oZXJlbnQgZGly ZWN0IGRtYSBtYXBwaW5nIG9wcycKCj4gWWVhaCB0aGUgYWJvdmUgaXMgcHJldHR5IG11Y2ggd2hh dCB3ZSBkbyBvbiB4ODYuIGRtYS1hcGkgYmVsaWV2ZXMKPiBldmVyeXRoaW5nIGlzIGNvaGVyZW50 LCBzbyBkbWFfbWFwX3NnIGRvZXMgdGhlIG1hcHBpbmcgd2Ugd2FudCBhbmQKPiBub3RoaW5nIGVs c2UgKG1pbnVzIHN3aW90bGIgZnVuKS4gQ2FjaGUgZmx1c2hpbmcsIGFsbG9jYXRpb25zLCBhbGwK PiBkb25lIGJ5IHRoZSBkcml2ZXIuCgpXaGljaCBzb3VuZHMgbGlrZSB0aGUgcmlnaHQgdGhpbmcg dG8gZG8gdG8gbWUuCgo+IE9uIGFybSB0aGF0IGRvZXNuJ3Qgd29yay4gVGhlIGlvbW11IGFwaSBz ZWVtcyBsaWtlIGEgZ29vZCBmaXQsIGV4Y2VwdAo+IHRoZSBkbWEtYXBpIHRlbmRzIHRvIGdldCBp biB0aGUgd2F5IGEgYml0IChkcm0vbXNtIGFwcGFyZW50bHkgaGFzCj4gc2ltaWxhciBwcm9ibGVt cyBsaWtlIHRlZ3JhKSwgYW5kIGlmIHlvdSBuZWVkIGNvbnRpZ3VvdXMgbWVtb3J5Cj4gZG1hX2Fs bG9jX2NvaGVyZW50IGlzIHRoZSBvbmx5IHdheSB0byBnZXQgYXQgY29udGlndW91cyBtZW1vcnku IFRoZXJlCj4gd2FzIGEgaHVnZSBkaXNjdXNzaW9uIHllYXJzIGFnbyBhYm91dCB0aGF0LCBhbmQg ZGlyZWN0IGNtYSBhY2Nlc3Mgd2FzCj4gc2hvdCBkb3duIGJlY2F1c2UgaXQgd291bGQgaGF2ZSBl eHBvc2VkIHRvbyBtdWNoIG9mIHRoZSBjYWNoaW5nCj4gYXR0cmlidXRlIG1hbmdsaW5nIHJlcXVp cmVkIChtb3N0IGFybSBwbGF0Zm9ybXMgbmVlZCB3Yy1wYWdlcyB0byBub3QKPiBiZSBpbiB0aGUg a2VybmVsJ3MgbGluZWFyIG1hcCBhcHBhcmVudGx5KS4KClNpbXBsZSBjbWFfYWxsb2MoKSBkb2Vz bid0IGRvIGFueXRoaW5nIGFib3V0IGNhY2hlIGhhbmRsaW5nLCBpdApqdXN0IGlzIGEgdmVyeSBk dW1iIGFsbG9jYXRvciBmb3IgbGFyZ2UgY29udGlndW91cyByZWdpb25zIGluc2lkZQphIGJpZyBw b29sLgoKSSdtIG5vdCB0aGUgQ01BIG1haW50YWluZXIsIGJ1dCBpbiBnZW5lcmFsIEknZCBsb3Zl IHRvIHNlZSBhbgpFWFBPUlRfU1lNQk9MX0dQTCBzbGFwcGVkIG9udG8gY21hX2FsbG9jL3JlbGVh c2UgYW5kIGRyaXZlcnMgdXNlCnRoYXQgd2VyZSBuZWVkZWQuICBVc2luZyB0aGF0IHBsdXMgZG1h X21hcCovZG1hX3VubWFwKiBzb3VuZHMgbGlrZQphIG11Y2ggc2FuZXIgaW50ZXJmYWNlIHRoYW4g ZG1hX2FsbG9jX2F0dHJzICsgRE1BX0FUVFJfTk9OX0NPTlNJU1RFTlQKb3IgRE1BX0FUVFJfTk9f S0VSTkVMX01BUFBJTkcuCgpZb3UgZG9uJ3QgaGFwcGVuIHRvIGhhdmUgYSBwb2ludGVyIHRvIHRo YXQgcHJldmlvdXMgZGlzY3Vzc2lvbj8KCj4gQW55dGhpbmcgdGhhdCBzZXBhcmF0ZSB0aGVzZSAz IHRoaW5ncyBtb3JlIChhbGxvY2F0aW9uIHBvb2xzLCBtYXBwaW5nCj4gdGhyb3VnaCBJT01NVXMg YW5kIGZsdXNoaW5nIGNwdSBjYWNoZXMpIHNvdW5kcyBsaWtlIHRoZSByaWdodAo+IGRpcmVjdGlv biB0byBtZS4gRXZlbiBpZiB0aGF0IHRocm93cyBzb21lIHBvcnRhYmlsaXR5IGFjcm9zcyBwbGF0 Zm9ybXMKPiBhd2F5IC0gZHJpdmVycyB3aG8gd2FudCB0byBjb250cm9sIHRoaW5ncyBpbiB0aGlz IG11Y2ggZGV0YWlsIGFyZW4ndAo+IHJlYWxseSBwb3J0YWJsZSAod2l0aG91dCBzb21lIHNlcmlv dXMgd29yaykgYW55d2F5LgoKQXMgbG9uZyBhcyB3ZSBzdGF5IGF3YXkgZnJvbSB0aGUgZG1hIGNv aGVyZW50IGFsbG9jYXRpb25zIHNlcGFyYXRpbmcKdGhlbSBpcyBmaW5lLCBhbmQgSSdtIHdvcmtp bmcgdG93YXJkcyBpdC4gIGRtYSBjb2hlcmVudCBhbGxvY2F0aW9ucyBvbgp0aGUgb3RoZXIgaGFu ZCBhcmUgdmVyeSBoYWlyeSBiZWFzdHMsIGFuZCBwcm92aWRlIGJ5IHZlcnkgZGlmZmVyZW50Cmlt cGxlbWVudGF0aW9ucyBkZXBlbmRpbmcgb24gdGhlIGFyY2hpdGVjdHVyZSwgb3Igb2Z0ZW4gZXZl biBkZXBlbmRpbmcKb24gdGhlIHNwZWNpZmljcyBvZiBpbmRpdmlkdWFsIGltcGxlbWVudGF0aW9u cyBpbnNpZGUgdGhlIGFyY2hpdGVjdHVyZS4KClNvIGZvciB5b3VyIEdQVS9tZWRpYSBjYXNlIGl0 IHNlZW1zIGxpa2UgeW91J2QgYmV0dGVyIHN0YXkgYXdheSBmcm9tCnRoZW0gYXMgbXVjaCBhcyB5 b3UgY2FuLgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpk cmktZGV2ZWwgbWFpbGluZyBsaXN0CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0 cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 From: hch@infradead.org (Christoph Hellwig) Date: Thu, 26 Apr 2018 02:09:42 -0700 Subject: noveau vs arm dma ops In-Reply-To: References: <20180425054855.GA17038@infradead.org> <20180425064335.GB28100@infradead.org> <20180425074151.GA2271@ulmo> <20180425085439.GA29996@infradead.org> <20180425100429.GR25142@phenom.ffwll.local> <20180425153312.GD27076@infradead.org> Message-ID: <20180426090942.GA18811@infradead.org> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On Wed, Apr 25, 2018 at 11:35:13PM +0200, Daniel Vetter wrote: > > get_required_mask() is supposed to tell you if you are safe. However > > we are missing lots of implementations of it for iommus so you might get > > some false negatives, improvements welcome. It's been on my list of > > things to fix in the DMA API, but it is nowhere near the top. > > I hasn't come up in a while in some fireworks, so I honestly don't > remember exactly what the issues have been. But > > commit d766ef53006c2c38a7fe2bef0904105a793383f2 > Author: Chris Wilson > Date: Mon Dec 19 12:43:45 2016 +0000 > > drm/i915: Fallback to single PAGE_SIZE segments for DMA remapping > > and the various bits of code that a > > $ git grep SWIOTLB -- drivers/gpu > > turns up is what we're doing to hack around that stuff. And in general > (there's some exceptions) gpus should be able to address everything, > so I never fully understood where that's even coming from. I'm pretty sure I've seen some oddly low dma masks in GPU drivers. E.g. duplicated in various AMD files: adev->need_dma32 = false; dma_bits = adev->need_dma32 ? 32 : 40; r = pci_set_dma_mask(adev->pdev, DMA_BIT_MASK(dma_bits)); if (r) { adev->need_dma32 = true; dma_bits = 32; dev_warn(adev->dev, "amdgpu: No suitable DMA available.\n"); } synopsis: drivers/gpu/drm/bridge/synopsys/dw-hdmi-i2s-audio.c: pdevinfo.dma_mask = DMA_BIT_MASK(32); drivers/gpu/drm/bridge/synopsys/dw-hdmi.c: pdevinfo.dma_mask = DMA_BIT_MASK(32); drivers/gpu/drm/bridge/synopsys/dw-hdmi.c: pdevinfo.dma_mask = DMA_BIT_MASK(32); etnaviv gets it right: drivers/gpu/drm/etnaviv/etnaviv_gpu.c: u32 dma_mask = (u32)dma_get_required_mask(gpu->dev); But yes, the swiotlb hackery really irks me. I just have some more important and bigger fires to fight first, but I plan to get back to the root cause of that eventually. > > >> - dma api hides the cache flushing requirements from us. GPUs love > >> non-snooped access, and worse give userspace control over that. We want > >> a strict separation between mapping stuff and flushing stuff. With the > >> IOMMU api we mostly have the former, but for the later arch maintainers > >> regularly tells they won't allow that. So we have drm_clflush.c. > > > > The problem is that a cache flushing API entirely separate is hard. That > > being said if you look at my generic dma-noncoherent API series it tries > > to move that way. So far it is in early stages and apparently rather > > buggy unfortunately. > > I'm assuming this stuff here? > > https://lkml.org/lkml/2018/4/20/146 > > Anyway got lost in all that work a bit, looks really nice. That url doesn't seem to work currently. But I am talking about the thread titled '[RFC] common non-cache coherent direct dma mapping ops' > Yeah the above is pretty much what we do on x86. dma-api believes > everything is coherent, so dma_map_sg does the mapping we want and > nothing else (minus swiotlb fun). Cache flushing, allocations, all > done by the driver. Which sounds like the right thing to do to me. > On arm that doesn't work. The iommu api seems like a good fit, except > the dma-api tends to get in the way a bit (drm/msm apparently has > similar problems like tegra), and if you need contiguous memory > dma_alloc_coherent is the only way to get at contiguous memory. There > was a huge discussion years ago about that, and direct cma access was > shot down because it would have exposed too much of the caching > attribute mangling required (most arm platforms need wc-pages to not > be in the kernel's linear map apparently). Simple cma_alloc() doesn't do anything about cache handling, it just is a very dumb allocator for large contiguous regions inside a big pool. I'm not the CMA maintainer, but in general I'd love to see an EXPORT_SYMBOL_GPL slapped onto cma_alloc/release and drivers use that were needed. Using that plus dma_map*/dma_unmap* sounds like a much saner interface than dma_alloc_attrs + DMA_ATTR_NON_CONSISTENT or DMA_ATTR_NO_KERNEL_MAPPING. You don't happen to have a pointer to that previous discussion? > Anything that separate these 3 things more (allocation pools, mapping > through IOMMUs and flushing cpu caches) sounds like the right > direction to me. Even if that throws some portability across platforms > away - drivers who want to control things in this much detail aren't > really portable (without some serious work) anyway. As long as we stay away from the dma coherent allocations separating them is fine, and I'm working towards it. dma coherent allocations on the other hand are very hairy beasts, and provide by very different implementations depending on the architecture, or often even depending on the specifics of individual implementations inside the architecture. So for your GPU/media case it seems like you'd better stay away from them as much as you can.