From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752485AbeDXTcZ (ORCPT ); Tue, 24 Apr 2018 15:32:25 -0400 Received: from mail-io0-f169.google.com ([209.85.223.169]:39669 "EHLO mail-io0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751095AbeDXTcV (ORCPT ); Tue, 24 Apr 2018 15:32:21 -0400 X-Google-Smtp-Source: AIpwx49cyOyjAD4Rf5rQH/swc2BQSFpP7LDRYawBJWtf+OA3QRwflksLjJBwUIlA56QPKcD3tHeFayynVnngrHnvra8= MIME-Version: 1.0 X-Originating-IP: [2a02:168:5635:0:39d2:f87e:2033:9f6] In-Reply-To: <20180424184847.GA3247@infradead.org> References: <20180403180832.GZ3881@phenom.ffwll.local> <20180416123937.GA9073@infradead.org> <20180419081657.GA16735@infradead.org> <20180420071312.GF31310@phenom.ffwll.local> <3e17afc5-7d6c-5795-07bd-f23e34cf8d4b@gmail.com> <20180420101755.GA11400@infradead.org> <20180420124625.GA31078@infradead.org> <20180420152111.GR31310@phenom.ffwll.local> <20180424184847.GA3247@infradead.org> From: Daniel Vetter Date: Tue, 24 Apr 2018 21:32:20 +0200 X-Google-Sender-Auth: Weph1xNq63TwgBmoR0fV0oTioX4 Message-ID: Subject: Re: [Linaro-mm-sig] [PATCH 4/8] dma-buf: add peer2peer flag To: Christoph Hellwig Cc: =?UTF-8?Q?Christian_K=C3=B6nig?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Linux Kernel Mailing List , amd-gfx list , Jerome Glisse , dri-devel , Dan Williams , Logan Gunthorpe , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Apr 24, 2018 at 8:48 PM, Christoph Hellwig wrote: > On Fri, Apr 20, 2018 at 05:21:11PM +0200, Daniel Vetter wrote: >> > At the very lowest level they will need to be handled differently for >> > many architectures, the questions is at what point we'll do the >> > branching out. >> >> Having at least struct page also in that list with (dma_addr_t, lenght) >> pairs has a bunch of benefits for drivers in unifying buffer handling >> code. You just pass that one single list around, use the dma_addr_t side >> for gpu access (generally bashing it into gpu ptes). And the struct page >> (if present) for cpu access, using kmap or vm_insert_*. We generally >> ignore virt, if we do need a full mapping then we construct a vmap for >> that buffer of our own. > > Well, for mapping a resource (which gets back to the start of the > discussion) you will need an explicit virt pointer. You also need > an explicit virt pointer and not just page_address/kmap for users of > dma_get_sgtable, because for many architectures you will need to flush > the virtual address used to access the data, which might be a > vmap/ioremap style mapping retourned from dma_alloc_address, and not > the directly mapped kernel address. Out of curiosity, how much virtual flushing stuff is there still out there? At least in drm we've pretty much ignore this, and seem to be getting away without a huge uproar (at least from driver developers and users, core folks are less amused about that). And at least for gpus that seems to have been the case since forever, or at least since AGP was a thing 20 years ago: AGP isn't coherent, so needs explicit cache flushing, and we have our own implementations of that in drivers/char/agp. Luckily AGP died 10 years ago, so no one yet proposed to port it all over to the iommu framework and hide it behind the dma api (which really would be the "clean" way to do this, AGP is simply an IOMMU + special socket dedicated for the add-on gpu). > Here is another idea at the low-level dma API level: > > - dma_get_sgtable goes away. The replacement is a new > dma_alloc_remap helper that takes the virtual address returned > from dma_alloc_attrs/coherent and creates a dma_addr_t for the > given new device. If the original allocation was a coherent > one no cache flushing is required either (because the arch > made sure it is coherent), if the original allocation used > DMA_ATTR_NON_CONSISTENT the new allocation will need > dma_cache_sync calls as well. Yeah I think that should work. dma_get_sgtable is a pretty nasty layering violation. > - you never even try to share a mapping retourned from > dma_map_resource - instead each device using it creates a new > mapping, which makes sense as no virtual addresses are involved > at all. Yeah the dma-buf exporter always knows what kind of backing storage it is dealing with, and for which struct device it should set up a new view. Hence can make sure that it calls the right functions to establish a new mapping, whether that's dma_map_sg, dma_map_resource or the new dma_alloc_remap (instead of the dma_get_sgtable layering mixup). The importer doesn't know. >> So maybe a list of (struct page *, dma_addr_t, num_pages) would suit best, >> with struct page * being optional (if it's a resource, or something else >> that the kernel core mm isn't aware of). But that only has benefits if we >> really roll it out everywhere, in all the subsystems and drivers, since if >> we don't we've made the struct pages ** <-> sgt conversion fun only worse >> by adding a 3 representation of gpu buffer object backing storage. > > I think the most important thing about such a buffer object is that > it can distinguish the underlying mapping types. While > dma_alloc_coherent, dma_alloc_attrs with DMA_ATTR_NON_CONSISTENT, > dma_map_page/dma_map_single/dma_map_sg and dma_map_resource all give > back a dma_addr_t they are in now way interchangable. And trying to > stuff them all into a structure like struct scatterlist that has > no indication what kind of mapping you are dealing with is just > asking for trouble. Well the idea was to have 1 interface to allow all drivers to share buffers with anything else, no matter how exactly they're allocated. dma-buf has all the functions for flushing, so you can have coherent mappings, non-coherent mappings and pretty much anything else. Or well could, because in practice people hack up layering violations until it works for the 2-3 drivers they care about. On top of that there's the small issue that x86 insists that dma is coherent (and that's true for most devices, including v4l drivers you might want to share stuff with), and gpus really, really really do want to make almost everything incoherent. The end result is pretty epic :-) -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch From mboxrd@z Thu Jan 1 00:00:00 1970 From: Daniel Vetter Subject: Re: [Linaro-mm-sig] [PATCH 4/8] dma-buf: add peer2peer flag Date: Tue, 24 Apr 2018 21:32:20 +0200 Message-ID: References: <20180403180832.GZ3881@phenom.ffwll.local> <20180416123937.GA9073@infradead.org> <20180419081657.GA16735@infradead.org> <20180420071312.GF31310@phenom.ffwll.local> <3e17afc5-7d6c-5795-07bd-f23e34cf8d4b@gmail.com> <20180420101755.GA11400@infradead.org> <20180420124625.GA31078@infradead.org> <20180420152111.GR31310@phenom.ffwll.local> <20180424184847.GA3247@infradead.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from mail-io0-x22a.google.com (mail-io0-x22a.google.com [IPv6:2607:f8b0:4001:c06::22a]) by gabe.freedesktop.org (Postfix) with ESMTPS id A80A76E49C for ; Tue, 24 Apr 2018 19:32:21 +0000 (UTC) Received: by mail-io0-x22a.google.com with SMTP id t123-v6so24091503iof.7 for ; Tue, 24 Apr 2018 12:32:21 -0700 (PDT) In-Reply-To: <20180424184847.GA3247@infradead.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Christoph Hellwig Cc: Linux Kernel Mailing List , dri-devel , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Jerome Glisse , amd-gfx list , Dan Williams , Logan Gunthorpe , =?UTF-8?Q?Christian_K=C3=B6nig?= , "open list:DMA BUFFER SHARING FRAMEWORK" List-Id: dri-devel@lists.freedesktop.org T24gVHVlLCBBcHIgMjQsIDIwMTggYXQgODo0OCBQTSwgQ2hyaXN0b3BoIEhlbGx3aWcgPGhjaEBp bmZyYWRlYWQub3JnPiB3cm90ZToKPiBPbiBGcmksIEFwciAyMCwgMjAxOCBhdCAwNToyMToxMVBN ICswMjAwLCBEYW5pZWwgVmV0dGVyIHdyb3RlOgo+PiA+IEF0IHRoZSB2ZXJ5IGxvd2VzdCBsZXZl bCB0aGV5IHdpbGwgbmVlZCB0byBiZSBoYW5kbGVkIGRpZmZlcmVudGx5IGZvcgo+PiA+IG1hbnkg YXJjaGl0ZWN0dXJlcywgdGhlIHF1ZXN0aW9ucyBpcyBhdCB3aGF0IHBvaW50IHdlJ2xsIGRvIHRo ZQo+PiA+IGJyYW5jaGluZyBvdXQuCj4+Cj4+IEhhdmluZyBhdCBsZWFzdCBzdHJ1Y3QgcGFnZSBh bHNvIGluIHRoYXQgbGlzdCB3aXRoIChkbWFfYWRkcl90LCBsZW5naHQpCj4+IHBhaXJzIGhhcyBh IGJ1bmNoIG9mIGJlbmVmaXRzIGZvciBkcml2ZXJzIGluIHVuaWZ5aW5nIGJ1ZmZlciBoYW5kbGlu Zwo+PiBjb2RlLiBZb3UganVzdCBwYXNzIHRoYXQgb25lIHNpbmdsZSBsaXN0IGFyb3VuZCwgdXNl IHRoZSBkbWFfYWRkcl90IHNpZGUKPj4gZm9yIGdwdSBhY2Nlc3MgKGdlbmVyYWxseSBiYXNoaW5n IGl0IGludG8gZ3B1IHB0ZXMpLiBBbmQgdGhlIHN0cnVjdCBwYWdlCj4+IChpZiBwcmVzZW50KSBm b3IgY3B1IGFjY2VzcywgdXNpbmcga21hcCBvciB2bV9pbnNlcnRfKi4gV2UgZ2VuZXJhbGx5Cj4+ IGlnbm9yZSB2aXJ0LCBpZiB3ZSBkbyBuZWVkIGEgZnVsbCBtYXBwaW5nIHRoZW4gd2UgY29uc3Ry dWN0IGEgdm1hcCBmb3IKPj4gdGhhdCBidWZmZXIgb2Ygb3VyIG93bi4KPgo+IFdlbGwsIGZvciBt YXBwaW5nIGEgcmVzb3VyY2UgKHdoaWNoIGdldHMgYmFjayB0byB0aGUgc3RhcnQgb2YgdGhlCj4g ZGlzY3Vzc2lvbikgeW91IHdpbGwgbmVlZCBhbiBleHBsaWNpdCB2aXJ0IHBvaW50ZXIuICBZb3Ug YWxzbyBuZWVkCj4gYW4gZXhwbGljaXQgdmlydCBwb2ludGVyIGFuZCBub3QganVzdCBwYWdlX2Fk ZHJlc3Mva21hcCBmb3IgdXNlcnMgb2YKPiBkbWFfZ2V0X3NndGFibGUsIGJlY2F1c2UgZm9yIG1h bnkgYXJjaGl0ZWN0dXJlcyB5b3Ugd2lsbCBuZWVkIHRvIGZsdXNoCj4gdGhlIHZpcnR1YWwgYWRk cmVzcyB1c2VkIHRvIGFjY2VzcyB0aGUgZGF0YSwgd2hpY2ggbWlnaHQgYmUgYQo+IHZtYXAvaW9y ZW1hcCBzdHlsZSBtYXBwaW5nIHJldG91cm5lZCBmcm9tIGRtYV9hbGxvY19hZGRyZXNzLCBhbmQg bm90Cj4gdGhlIGRpcmVjdGx5IG1hcHBlZCBrZXJuZWwgYWRkcmVzcy4KCk91dCBvZiBjdXJpb3Np dHksIGhvdyBtdWNoIHZpcnR1YWwgZmx1c2hpbmcgc3R1ZmYgaXMgdGhlcmUgc3RpbGwgb3V0CnRo ZXJlPyBBdCBsZWFzdCBpbiBkcm0gd2UndmUgcHJldHR5IG11Y2ggaWdub3JlIHRoaXMsIGFuZCBz ZWVtIHRvIGJlCmdldHRpbmcgYXdheSB3aXRob3V0IGEgaHVnZSB1cHJvYXIgKGF0IGxlYXN0IGZy b20gZHJpdmVyIGRldmVsb3BlcnMKYW5kIHVzZXJzLCBjb3JlIGZvbGtzIGFyZSBsZXNzIGFtdXNl ZCBhYm91dCB0aGF0KS4KCkFuZCBhdCBsZWFzdCBmb3IgZ3B1cyB0aGF0IHNlZW1zIHRvIGhhdmUg YmVlbiB0aGUgY2FzZSBzaW5jZSBmb3JldmVyLApvciBhdCBsZWFzdCBzaW5jZSBBR1Agd2FzIGEg dGhpbmcgMjAgeWVhcnMgYWdvOiBBR1AgaXNuJ3QgY29oZXJlbnQsIHNvCm5lZWRzIGV4cGxpY2l0 IGNhY2hlIGZsdXNoaW5nLCBhbmQgd2UgaGF2ZSBvdXIgb3duIGltcGxlbWVudGF0aW9ucyBvZgp0 aGF0IGluIGRyaXZlcnMvY2hhci9hZ3AuIEx1Y2tpbHkgQUdQIGRpZWQgMTAgeWVhcnMgYWdvLCBz byBubyBvbmUgeWV0CnByb3Bvc2VkIHRvIHBvcnQgaXQgYWxsIG92ZXIgdG8gdGhlIGlvbW11IGZy YW1ld29yayBhbmQgaGlkZSBpdCBiZWhpbmQKdGhlIGRtYSBhcGkgKHdoaWNoIHJlYWxseSB3b3Vs ZCBiZSB0aGUgImNsZWFuIiB3YXkgdG8gZG8gdGhpcywgQUdQIGlzCnNpbXBseSBhbiBJT01NVSAr IHNwZWNpYWwgc29ja2V0IGRlZGljYXRlZCBmb3IgdGhlIGFkZC1vbiBncHUpLgoKPiBIZXJlIGlz IGFub3RoZXIgaWRlYSBhdCB0aGUgbG93LWxldmVsIGRtYSBBUEkgbGV2ZWw6Cj4KPiAgLSBkbWFf Z2V0X3NndGFibGUgZ29lcyBhd2F5LiAgVGhlIHJlcGxhY2VtZW50IGlzIGEgbmV3Cj4gICAgZG1h X2FsbG9jX3JlbWFwIGhlbHBlciB0aGF0IHRha2VzIHRoZSB2aXJ0dWFsIGFkZHJlc3MgcmV0dXJu ZWQKPiAgICBmcm9tIGRtYV9hbGxvY19hdHRycy9jb2hlcmVudCBhbmQgY3JlYXRlcyBhIGRtYV9h ZGRyX3QgZm9yIHRoZQo+ICAgIGdpdmVuIG5ldyBkZXZpY2UuICBJZiB0aGUgb3JpZ2luYWwgYWxs b2NhdGlvbiB3YXMgYSBjb2hlcmVudAo+ICAgIG9uZSBubyBjYWNoZSBmbHVzaGluZyBpcyByZXF1 aXJlZCBlaXRoZXIgKGJlY2F1c2UgdGhlIGFyY2gKPiAgICBtYWRlIHN1cmUgaXQgaXMgY29oZXJl bnQpLCBpZiB0aGUgb3JpZ2luYWwgYWxsb2NhdGlvbiB1c2VkCj4gICAgRE1BX0FUVFJfTk9OX0NP TlNJU1RFTlQgdGhlIG5ldyBhbGxvY2F0aW9uIHdpbGwgbmVlZAo+ICAgIGRtYV9jYWNoZV9zeW5j IGNhbGxzIGFzIHdlbGwuCgpZZWFoIEkgdGhpbmsgdGhhdCBzaG91bGQgd29yay4gZG1hX2dldF9z Z3RhYmxlIGlzIGEgcHJldHR5IG5hc3R5CmxheWVyaW5nIHZpb2xhdGlvbi4KCj4gIC0geW91IG5l dmVyIGV2ZW4gdHJ5IHRvIHNoYXJlIGEgbWFwcGluZyByZXRvdXJuZWQgZnJvbQo+ICAgIGRtYV9t YXBfcmVzb3VyY2UgLSBpbnN0ZWFkIGVhY2ggZGV2aWNlIHVzaW5nIGl0IGNyZWF0ZXMgYSBuZXcK PiAgICBtYXBwaW5nLCB3aGljaCBtYWtlcyBzZW5zZSBhcyBubyB2aXJ0dWFsIGFkZHJlc3NlcyBh cmUgaW52b2x2ZWQKPiAgICBhdCBhbGwuCgpZZWFoIHRoZSBkbWEtYnVmIGV4cG9ydGVyIGFsd2F5 cyBrbm93cyB3aGF0IGtpbmQgb2YgYmFja2luZyBzdG9yYWdlIGl0CmlzIGRlYWxpbmcgd2l0aCwg YW5kIGZvciB3aGljaCBzdHJ1Y3QgZGV2aWNlIGl0IHNob3VsZCBzZXQgdXAgYSBuZXcKdmlldy4g SGVuY2UgY2FuIG1ha2Ugc3VyZSB0aGF0IGl0IGNhbGxzIHRoZSByaWdodCBmdW5jdGlvbnMgdG8K ZXN0YWJsaXNoIGEgbmV3IG1hcHBpbmcsIHdoZXRoZXIgdGhhdCdzIGRtYV9tYXBfc2csIGRtYV9t YXBfcmVzb3VyY2UKb3IgdGhlIG5ldyBkbWFfYWxsb2NfcmVtYXAgKGluc3RlYWQgb2YgdGhlIGRt YV9nZXRfc2d0YWJsZSBsYXllcmluZwptaXh1cCkuIFRoZSBpbXBvcnRlciBkb2Vzbid0IGtub3cu Cgo+PiBTbyBtYXliZSBhIGxpc3Qgb2YgKHN0cnVjdCBwYWdlICosIGRtYV9hZGRyX3QsIG51bV9w YWdlcykgd291bGQgc3VpdCBiZXN0LAo+PiB3aXRoIHN0cnVjdCBwYWdlICogYmVpbmcgb3B0aW9u YWwgKGlmIGl0J3MgYSByZXNvdXJjZSwgb3Igc29tZXRoaW5nIGVsc2UKPj4gdGhhdCB0aGUga2Vy bmVsIGNvcmUgbW0gaXNuJ3QgYXdhcmUgb2YpLiBCdXQgdGhhdCBvbmx5IGhhcyBiZW5lZml0cyBp ZiB3ZQo+PiByZWFsbHkgcm9sbCBpdCBvdXQgZXZlcnl3aGVyZSwgaW4gYWxsIHRoZSBzdWJzeXN0 ZW1zIGFuZCBkcml2ZXJzLCBzaW5jZSBpZgo+PiB3ZSBkb24ndCB3ZSd2ZSBtYWRlIHRoZSBzdHJ1 Y3QgcGFnZXMgKiogPC0+IHNndCBjb252ZXJzaW9uIGZ1biBvbmx5IHdvcnNlCj4+IGJ5IGFkZGlu ZyBhIDMgcmVwcmVzZW50YXRpb24gb2YgZ3B1IGJ1ZmZlciBvYmplY3QgYmFja2luZyBzdG9yYWdl Lgo+Cj4gSSB0aGluayB0aGUgbW9zdCBpbXBvcnRhbnQgdGhpbmcgYWJvdXQgc3VjaCBhIGJ1ZmZl ciBvYmplY3QgaXMgdGhhdAo+IGl0IGNhbiBkaXN0aW5ndWlzaCB0aGUgdW5kZXJseWluZyBtYXBw aW5nIHR5cGVzLiAgV2hpbGUKPiBkbWFfYWxsb2NfY29oZXJlbnQsIGRtYV9hbGxvY19hdHRycyB3 aXRoIERNQV9BVFRSX05PTl9DT05TSVNURU5ULAo+IGRtYV9tYXBfcGFnZS9kbWFfbWFwX3Npbmds ZS9kbWFfbWFwX3NnIGFuZCBkbWFfbWFwX3Jlc291cmNlIGFsbCBnaXZlCj4gYmFjayBhIGRtYV9h ZGRyX3QgdGhleSBhcmUgaW4gbm93IHdheSBpbnRlcmNoYW5nYWJsZS4gIEFuZCB0cnlpbmcgdG8K PiBzdHVmZiB0aGVtIGFsbCBpbnRvIGEgc3RydWN0dXJlIGxpa2Ugc3RydWN0IHNjYXR0ZXJsaXN0 IHRoYXQgaGFzCj4gbm8gaW5kaWNhdGlvbiB3aGF0IGtpbmQgb2YgbWFwcGluZyB5b3UgYXJlIGRl YWxpbmcgd2l0aCBpcyBqdXN0Cj4gYXNraW5nIGZvciB0cm91YmxlLgoKV2VsbCB0aGUgaWRlYSB3 YXMgdG8gaGF2ZSAxIGludGVyZmFjZSB0byBhbGxvdyBhbGwgZHJpdmVycyB0byBzaGFyZQpidWZm ZXJzIHdpdGggYW55dGhpbmcgZWxzZSwgbm8gbWF0dGVyIGhvdyBleGFjdGx5IHRoZXkncmUgYWxs b2NhdGVkLgpkbWEtYnVmIGhhcyBhbGwgdGhlIGZ1bmN0aW9ucyBmb3IgZmx1c2hpbmcsIHNvIHlv dSBjYW4gaGF2ZSBjb2hlcmVudAptYXBwaW5ncywgbm9uLWNvaGVyZW50IG1hcHBpbmdzIGFuZCBw cmV0dHkgbXVjaCBhbnl0aGluZyBlbHNlLiBPciB3ZWxsCmNvdWxkLCBiZWNhdXNlIGluIHByYWN0 aWNlIHBlb3BsZSBoYWNrIHVwIGxheWVyaW5nIHZpb2xhdGlvbnMgdW50aWwgaXQKd29ya3MgZm9y IHRoZSAyLTMgZHJpdmVycyB0aGV5IGNhcmUgYWJvdXQuIE9uIHRvcCBvZiB0aGF0IHRoZXJlJ3Mg dGhlCnNtYWxsIGlzc3VlIHRoYXQgeDg2IGluc2lzdHMgdGhhdCBkbWEgaXMgY29oZXJlbnQgKGFu ZCB0aGF0J3MgdHJ1ZSBmb3IKbW9zdCBkZXZpY2VzLCBpbmNsdWRpbmcgdjRsIGRyaXZlcnMgeW91 IG1pZ2h0IHdhbnQgdG8gc2hhcmUgc3R1ZmYKd2l0aCksIGFuZCBncHVzIHJlYWxseSwgcmVhbGx5 IHJlYWxseSBkbyB3YW50IHRvIG1ha2UgYWxtb3N0CmV2ZXJ5dGhpbmcgaW5jb2hlcmVudC4KClRo ZSBlbmQgcmVzdWx0IGlzIHByZXR0eSBlcGljIDotKQotRGFuaWVsCi0tIApEYW5pZWwgVmV0dGVy ClNvZnR3YXJlIEVuZ2luZWVyLCBJbnRlbCBDb3Jwb3JhdGlvbgorNDEgKDApIDc5IDM2NSA1NyA0 OCAtIGh0dHA6Ly9ibG9nLmZmd2xsLmNoCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fCmRyaS1kZXZlbCBtYWlsaW5nIGxpc3QKZHJpLWRldmVsQGxpc3RzLmZy ZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3Rp bmZvL2RyaS1kZXZlbAo=