From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0C22C10F03 for ; Tue, 23 Apr 2019 16:39:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CA47220645 for ; Tue, 23 Apr 2019 16:39:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729583AbfDWQiy (ORCPT ); Tue, 23 Apr 2019 12:38:54 -0400 Received: from relay6-d.mail.gandi.net ([217.70.183.198]:58409 "EHLO relay6-d.mail.gandi.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729557AbfDWQiv (ORCPT ); Tue, 23 Apr 2019 12:38:51 -0400 X-Originating-IP: 93.29.109.196 Received: from collins (196.109.29.93.rev.sfr.net [93.29.109.196]) (Authenticated sender: paul.kocialkowski@bootlin.com) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id AEFF9C0007; Tue, 23 Apr 2019 16:38:46 +0000 (UTC) Message-ID: Subject: Re: [PATCH 00/20] drm: Split out the formats API and move it to a common place From: Paul Kocialkowski To: Daniel Stone , Laurent Pinchart Cc: Maxime Ripard , Daniel Vetter , David Airlie , Maarten Lankhorst , Sean Paul , Mauro Carvalho Chehab , Sakari Ailus , Linux Kernel Mailing List , dri-devel , Hans Verkuil , Thomas Petazzoni , "open list:DMA BUFFER SHARING FRAMEWORK" Date: Tue, 23 Apr 2019 18:38:46 +0200 In-Reply-To: References: <20190417154121.GJ13337@phenom.ffwll.local> <20190418062229.eyog4i62eg4pr6uf@flea> <20190418090221.e57dogn4yx5nwdni@flea> <20190420225904.GZ4964@pendragon.ideasonboard.com> <20190423072554.GW13337@phenom.ffwll.local> <20190423155416.GI16111@pendragon.ideasonboard.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.32.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, Le mardi 23 avril 2019 à 17:02 +0100, Daniel Stone a écrit : > Hi Laurent, > > On Tue, 23 Apr 2019 at 16:54, Laurent Pinchart > wrote: > > On Tue, Apr 23, 2019 at 09:59:37AM +0100, Daniel Stone wrote: > > > On Tue, 23 Apr 2019 at 08:26, Daniel Vetter wrote: > > > Totally. Let's take DRM_FORMAT_XRGB8888 + I915_FORMAT_MOD_Y_TILED as > > > an example. [... details ...] > > > > Looks like we have different kinds of metadata to consider. On the V4L2 > > side metadata usually refers to the context in which a frame was > > captured, it doesn't tell how to interpret the contents of the pixels. > > In the case you just described, the metadata is part of the frame > > contents. I agree that this is a proper use case for storing such > > metadata in a plane. What I wouldn't like to see being stored in a plane > > is for instance gamma tables or similar data that configures the > > processing pipeline in the display engine (I know we have an API for > > gamma tables, this is just an example). > > > > > It would be good to understand what you had in mind when you said that > > > using multiple planes created a mess. I haven't touched media > > > encode/decode units at a low level for quite a while (hooray for > > > gst-v4l2!), but I remember that they often used padding areas around > > > the buffer for scratch space - maybe motion vectors or similar? That > > > case is quite different to something like CCS, since the data is only > > > meaningful to the media engine and must be ignored (but preserved) by > > > everyone else. Using multiple planes in that case isn't appropriate, > > > since it's very specific to how that hardware unit deals with that > > > buffer, instead of something that every consumer needs to understand > > > in order to use it. > > > > With metadata unrelated to the pixel content, using a separate plane in > > the same buffer resulted in an explosion of the number of combinations > > that we would need to support, and ultimately led to a very ill-defined > > API. We decided to convey metadata related to the frame capture context > > (e.g. what exposure time was used for the frame) and processing pipeline > > configuration data in different buffers than the frame itself. > > Yeah, that makes sense. It's not really that different from what > happens with GPUs either: the final colour buffer the display > controller gets from a game is the product of a _lot_ of other work > which is invisible to the display controller, including things like > depth and stencil buffers, command buffers, etc etc. Those are closely > related to the frame production, but totally irrelevant for exchanging > the colour buffer with other subsystems. > > I think we should look at the metadata buffers you're describing in > the same way. Perhaps each V4L2 buffer could have driver-private > auxiliary buffer storage, or perhaps it's something you need to > separately expose to userspace as auxiliary data which must be > preserved but ignored. But modifiers are really only about what you > need when exchanging image colour buffers between subsystems, not > anything else. > > You're pretty close with gamma tables as well; for HDR and other kinds > of complex colour management, we need to carry a fair bit of auxiliary > information in order to display the image correctly. These have quite > different uses though: normally the colour buffer is produced by the > hardware and the primaries/whitepoints/etc are produced by software, > with the colour-management details remaining static across the life of > a swapchain even as you flip between multiple buffers. Given that, it > doesn't really make sense to try to stuff them into the same storage. I agree that we need to keep things minimal and clearly distinguish between what constitutes the description of the pixel buffer in terms of how it's laid out in memory and information about how the data should be interpreted. And there's indeed a fair share of things to consider there. Adding to the list above, I'm also thinking of the YUV colorspace information which must be passed along when displaying a buffer. But that's essentially not required to have a common description of buffers unified accross subsystems. Thinking about that, it would be interesting to have a common base structure for buffers used in v4l2 and drm. Ideally, that could be shared when doing dma-buf to avoid having userspace describe buffers and memory each time a buffer is imported into another subsystem. This could also help us solve the ambiguity related to the YUV M-suffixed formats. Another idea could be having common ioctls to get a unified buffer description from userspace and e.g. mmap on a per-plane basis (with virtual mappings like DRM does). What do you think? Cheers, Paul From mboxrd@z Thu Jan 1 00:00:00 1970 From: Paul Kocialkowski Subject: Re: [PATCH 00/20] drm: Split out the formats API and move it to a common place Date: Tue, 23 Apr 2019 18:38:46 +0200 Message-ID: References: <20190417154121.GJ13337@phenom.ffwll.local> <20190418062229.eyog4i62eg4pr6uf@flea> <20190418090221.e57dogn4yx5nwdni@flea> <20190420225904.GZ4964@pendragon.ideasonboard.com> <20190423072554.GW13337@phenom.ffwll.local> <20190423155416.GI16111@pendragon.ideasonboard.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by gabe.freedesktop.org (Postfix) with ESMTPS id 90111891D2 for ; Tue, 23 Apr 2019 16:38:50 +0000 (UTC) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Daniel Stone , Laurent Pinchart Cc: Thomas Petazzoni , Maxime Ripard , Linux Kernel Mailing List , dri-devel , David Airlie , Hans Verkuil , Sean Paul , Sakari Ailus , Daniel Vetter , Mauro Carvalho Chehab , "open list:DMA BUFFER SHARING FRAMEWORK" List-Id: dri-devel@lists.freedesktop.org SGksCgpMZSBtYXJkaSAyMyBhdnJpbCAyMDE5IMOgIDE3OjAyICswMTAwLCBEYW5pZWwgU3RvbmUg YSDDqWNyaXQgOgo+IEhpIExhdXJlbnQsCj4gCj4gT24gVHVlLCAyMyBBcHIgMjAxOSBhdCAxNjo1 NCwgTGF1cmVudCBQaW5jaGFydAo+IDxsYXVyZW50LnBpbmNoYXJ0QGlkZWFzb25ib2FyZC5jb20+ IHdyb3RlOgo+ID4gT24gVHVlLCBBcHIgMjMsIDIwMTkgYXQgMDk6NTk6MzdBTSArMDEwMCwgRGFu aWVsIFN0b25lIHdyb3RlOgo+ID4gPiBPbiBUdWUsIDIzIEFwciAyMDE5IGF0IDA4OjI2LCBEYW5p ZWwgVmV0dGVyIDxkYW5pZWxAZmZ3bGwuY2g+IHdyb3RlOgo+ID4gPiBUb3RhbGx5LiBMZXQncyB0 YWtlIERSTV9GT1JNQVRfWFJHQjg4ODggKyBJOTE1X0ZPUk1BVF9NT0RfWV9USUxFRCBhcwo+ID4g PiBhbiBleGFtcGxlLiBbLi4uIGRldGFpbHMgLi4uXQo+ID4gCj4gPiBMb29rcyBsaWtlIHdlIGhh dmUgZGlmZmVyZW50IGtpbmRzIG9mIG1ldGFkYXRhIHRvIGNvbnNpZGVyLiBPbiB0aGUgVjRMMgo+ ID4gc2lkZSBtZXRhZGF0YSB1c3VhbGx5IHJlZmVycyB0byB0aGUgY29udGV4dCBpbiB3aGljaCBh IGZyYW1lIHdhcwo+ID4gY2FwdHVyZWQsIGl0IGRvZXNuJ3QgdGVsbCBob3cgdG8gaW50ZXJwcmV0 IHRoZSBjb250ZW50cyBvZiB0aGUgcGl4ZWxzLgo+ID4gSW4gdGhlIGNhc2UgeW91IGp1c3QgZGVz Y3JpYmVkLCB0aGUgbWV0YWRhdGEgaXMgcGFydCBvZiB0aGUgZnJhbWUKPiA+IGNvbnRlbnRzLiBJ IGFncmVlIHRoYXQgdGhpcyBpcyBhIHByb3BlciB1c2UgY2FzZSBmb3Igc3RvcmluZyBzdWNoCj4g PiBtZXRhZGF0YSBpbiBhIHBsYW5lLiBXaGF0IEkgd291bGRuJ3QgbGlrZSB0byBzZWUgYmVpbmcg c3RvcmVkIGluIGEgcGxhbmUKPiA+IGlzIGZvciBpbnN0YW5jZSBnYW1tYSB0YWJsZXMgb3Igc2lt aWxhciBkYXRhIHRoYXQgY29uZmlndXJlcyB0aGUKPiA+IHByb2Nlc3NpbmcgcGlwZWxpbmUgaW4g dGhlIGRpc3BsYXkgZW5naW5lIChJIGtub3cgd2UgaGF2ZSBhbiBBUEkgZm9yCj4gPiBnYW1tYSB0 YWJsZXMsIHRoaXMgaXMganVzdCBhbiBleGFtcGxlKS4KPiA+IAo+ID4gPiBJdCB3b3VsZCBiZSBn b29kIHRvIHVuZGVyc3RhbmQgd2hhdCB5b3UgaGFkIGluIG1pbmQgd2hlbiB5b3Ugc2FpZCB0aGF0 Cj4gPiA+IHVzaW5nIG11bHRpcGxlIHBsYW5lcyBjcmVhdGVkIGEgbWVzcy4gSSBoYXZlbid0IHRv dWNoZWQgbWVkaWEKPiA+ID4gZW5jb2RlL2RlY29kZSB1bml0cyBhdCBhIGxvdyBsZXZlbCBmb3Ig cXVpdGUgYSB3aGlsZSAoaG9vcmF5IGZvcgo+ID4gPiBnc3QtdjRsMiEpLCBidXQgSSByZW1lbWJl ciB0aGF0IHRoZXkgb2Z0ZW4gdXNlZCBwYWRkaW5nIGFyZWFzIGFyb3VuZAo+ID4gPiB0aGUgYnVm ZmVyIGZvciBzY3JhdGNoIHNwYWNlIC0gbWF5YmUgbW90aW9uIHZlY3RvcnMgb3Igc2ltaWxhcj8g VGhhdAo+ID4gPiBjYXNlIGlzIHF1aXRlIGRpZmZlcmVudCB0byBzb21ldGhpbmcgbGlrZSBDQ1Ms IHNpbmNlIHRoZSBkYXRhIGlzIG9ubHkKPiA+ID4gbWVhbmluZ2Z1bCB0byB0aGUgbWVkaWEgZW5n aW5lIGFuZCBtdXN0IGJlIGlnbm9yZWQgKGJ1dCBwcmVzZXJ2ZWQpIGJ5Cj4gPiA+IGV2ZXJ5b25l IGVsc2UuIFVzaW5nIG11bHRpcGxlIHBsYW5lcyBpbiB0aGF0IGNhc2UgaXNuJ3QgYXBwcm9wcmlh dGUsCj4gPiA+IHNpbmNlIGl0J3MgdmVyeSBzcGVjaWZpYyB0byBob3cgdGhhdCBoYXJkd2FyZSB1 bml0IGRlYWxzIHdpdGggdGhhdAo+ID4gPiBidWZmZXIsIGluc3RlYWQgb2Ygc29tZXRoaW5nIHRo YXQgZXZlcnkgY29uc3VtZXIgbmVlZHMgdG8gdW5kZXJzdGFuZAo+ID4gPiBpbiBvcmRlciB0byB1 c2UgaXQuCj4gPiAKPiA+IFdpdGggbWV0YWRhdGEgdW5yZWxhdGVkIHRvIHRoZSBwaXhlbCBjb250 ZW50LCB1c2luZyBhIHNlcGFyYXRlIHBsYW5lIGluCj4gPiB0aGUgc2FtZSBidWZmZXIgcmVzdWx0 ZWQgaW4gYW4gZXhwbG9zaW9uIG9mIHRoZSBudW1iZXIgb2YgY29tYmluYXRpb25zCj4gPiB0aGF0 IHdlIHdvdWxkIG5lZWQgdG8gc3VwcG9ydCwgYW5kIHVsdGltYXRlbHkgbGVkIHRvIGEgdmVyeSBp bGwtZGVmaW5lZAo+ID4gQVBJLiBXZSBkZWNpZGVkIHRvIGNvbnZleSBtZXRhZGF0YSByZWxhdGVk IHRvIHRoZSBmcmFtZSBjYXB0dXJlIGNvbnRleHQKPiA+IChlLmcuIHdoYXQgZXhwb3N1cmUgdGlt ZSB3YXMgdXNlZCBmb3IgdGhlIGZyYW1lKSBhbmQgcHJvY2Vzc2luZyBwaXBlbGluZQo+ID4gY29u ZmlndXJhdGlvbiBkYXRhIGluIGRpZmZlcmVudCBidWZmZXJzIHRoYW4gdGhlIGZyYW1lIGl0c2Vs Zi4KPiAKPiBZZWFoLCB0aGF0IG1ha2VzIHNlbnNlLiBJdCdzIG5vdCByZWFsbHkgdGhhdCBkaWZm ZXJlbnQgZnJvbSB3aGF0Cj4gaGFwcGVucyB3aXRoIEdQVXMgZWl0aGVyOiB0aGUgZmluYWwgY29s b3VyIGJ1ZmZlciB0aGUgZGlzcGxheQo+IGNvbnRyb2xsZXIgZ2V0cyBmcm9tIGEgZ2FtZSBpcyB0 aGUgcHJvZHVjdCBvZiBhIF9sb3RfIG9mIG90aGVyIHdvcmsKPiB3aGljaCBpcyBpbnZpc2libGUg dG8gdGhlIGRpc3BsYXkgY29udHJvbGxlciwgaW5jbHVkaW5nIHRoaW5ncyBsaWtlCj4gZGVwdGgg YW5kIHN0ZW5jaWwgYnVmZmVycywgY29tbWFuZCBidWZmZXJzLCBldGMgZXRjLiBUaG9zZSBhcmUg Y2xvc2VseQo+IHJlbGF0ZWQgdG8gdGhlIGZyYW1lIHByb2R1Y3Rpb24sIGJ1dCB0b3RhbGx5IGly cmVsZXZhbnQgZm9yIGV4Y2hhbmdpbmcKPiB0aGUgY29sb3VyIGJ1ZmZlciB3aXRoIG90aGVyIHN1 YnN5c3RlbXMuCj4gCj4gSSB0aGluayB3ZSBzaG91bGQgbG9vayBhdCB0aGUgbWV0YWRhdGEgYnVm ZmVycyB5b3UncmUgZGVzY3JpYmluZyBpbgo+IHRoZSBzYW1lIHdheS4gUGVyaGFwcyBlYWNoIFY0 TDIgYnVmZmVyIGNvdWxkIGhhdmUgZHJpdmVyLXByaXZhdGUKPiBhdXhpbGlhcnkgYnVmZmVyIHN0 b3JhZ2UsIG9yIHBlcmhhcHMgaXQncyBzb21ldGhpbmcgeW91IG5lZWQgdG8KPiBzZXBhcmF0ZWx5 IGV4cG9zZSB0byB1c2Vyc3BhY2UgYXMgYXV4aWxpYXJ5IGRhdGEgd2hpY2ggbXVzdCBiZQo+IHBy ZXNlcnZlZCBidXQgaWdub3JlZC4gQnV0IG1vZGlmaWVycyBhcmUgcmVhbGx5IG9ubHkgYWJvdXQg d2hhdCB5b3UKPiBuZWVkIHdoZW4gZXhjaGFuZ2luZyBpbWFnZSBjb2xvdXIgYnVmZmVycyBiZXR3 ZWVuIHN1YnN5c3RlbXMsIG5vdAo+IGFueXRoaW5nIGVsc2UuCj4gCj4gWW91J3JlIHByZXR0eSBj bG9zZSB3aXRoIGdhbW1hIHRhYmxlcyBhcyB3ZWxsOyBmb3IgSERSIGFuZCBvdGhlciBraW5kcwo+ IG9mIGNvbXBsZXggY29sb3VyIG1hbmFnZW1lbnQsIHdlIG5lZWQgdG8gY2FycnkgYSBmYWlyIGJp dCBvZiBhdXhpbGlhcnkKPiBpbmZvcm1hdGlvbiBpbiBvcmRlciB0byBkaXNwbGF5IHRoZSBpbWFn ZSBjb3JyZWN0bHkuIFRoZXNlIGhhdmUgcXVpdGUKPiBkaWZmZXJlbnQgdXNlcyB0aG91Z2g6IG5v cm1hbGx5IHRoZSBjb2xvdXIgYnVmZmVyIGlzIHByb2R1Y2VkIGJ5IHRoZQo+IGhhcmR3YXJlIGFu ZCB0aGUgcHJpbWFyaWVzL3doaXRlcG9pbnRzL2V0YyBhcmUgcHJvZHVjZWQgYnkgc29mdHdhcmUs Cj4gd2l0aCB0aGUgY29sb3VyLW1hbmFnZW1lbnQgZGV0YWlscyByZW1haW5pbmcgc3RhdGljIGFj cm9zcyB0aGUgbGlmZSBvZgo+IGEgc3dhcGNoYWluIGV2ZW4gYXMgeW91IGZsaXAgYmV0d2VlbiBt dWx0aXBsZSBidWZmZXJzLiBHaXZlbiB0aGF0LCBpdAo+IGRvZXNuJ3QgcmVhbGx5IG1ha2Ugc2Vu c2UgdG8gdHJ5IHRvIHN0dWZmIHRoZW0gaW50byB0aGUgc2FtZSBzdG9yYWdlLgoKSSBhZ3JlZSB0 aGF0IHdlIG5lZWQgdG8ga2VlcCB0aGluZ3MgbWluaW1hbCBhbmQgY2xlYXJseSBkaXN0aW5ndWlz aApiZXR3ZWVuIHdoYXQgY29uc3RpdHV0ZXMgdGhlIGRlc2NyaXB0aW9uIG9mIHRoZSBwaXhlbCBi dWZmZXIgaW4gdGVybXMKb2YgaG93IGl0J3MgbGFpZCBvdXQgaW4gbWVtb3J5IGFuZCBpbmZvcm1h dGlvbiBhYm91dCBob3cgdGhlIGRhdGEKc2hvdWxkIGJlIGludGVycHJldGVkLgoKQW5kIHRoZXJl J3MgaW5kZWVkIGEgZmFpciBzaGFyZSBvZiB0aGluZ3MgdG8gY29uc2lkZXIgdGhlcmUuIEFkZGlu ZyB0bwp0aGUgbGlzdCBhYm92ZSwgSSdtIGFsc28gdGhpbmtpbmcgb2YgdGhlIFlVViBjb2xvcnNw YWNlIGluZm9ybWF0aW9uCndoaWNoIG11c3QgYmUgcGFzc2VkIGFsb25nIHdoZW4gZGlzcGxheWlu ZyBhIGJ1ZmZlci4KCkJ1dCB0aGF0J3MgZXNzZW50aWFsbHkgbm90IHJlcXVpcmVkIHRvIGhhdmUg YSBjb21tb24gZGVzY3JpcHRpb24gb2YKYnVmZmVycyB1bmlmaWVkIGFjY3Jvc3Mgc3Vic3lzdGVt cy4gVGhpbmtpbmcgYWJvdXQgdGhhdCwgaXQgd291bGQgYmUKaW50ZXJlc3RpbmcgdG8gaGF2ZSBh IGNvbW1vbiBiYXNlIHN0cnVjdHVyZSBmb3IgYnVmZmVycyB1c2VkIGluIHY0bDIKYW5kIGRybS4g SWRlYWxseSwgdGhhdCBjb3VsZCBiZSBzaGFyZWQgd2hlbiBkb2luZyBkbWEtYnVmIHRvIGF2b2lk CmhhdmluZyB1c2Vyc3BhY2UgZGVzY3JpYmUgYnVmZmVycyBhbmQgbWVtb3J5IGVhY2ggdGltZSBh IGJ1ZmZlciBpcwppbXBvcnRlZCBpbnRvIGFub3RoZXIgc3Vic3lzdGVtLiBUaGlzIGNvdWxkIGFs c28gaGVscCB1cyBzb2x2ZSB0aGUKYW1iaWd1aXR5IHJlbGF0ZWQgdG8gdGhlIFlVViBNLXN1ZmZp eGVkIGZvcm1hdHMuIEFub3RoZXIgaWRlYSBjb3VsZCBiZQpoYXZpbmcgY29tbW9uIGlvY3RscyB0 byBnZXQgYSB1bmlmaWVkIGJ1ZmZlciBkZXNjcmlwdGlvbiBmcm9tIHVzZXJzcGFjZQphbmQgZS5n LiBtbWFwIG9uIGEgcGVyLXBsYW5lIGJhc2lzICh3aXRoIHZpcnR1YWwgbWFwcGluZ3MgbGlrZSBE Uk0KZG9lcykuCgpXaGF0IGRvIHlvdSB0aGluaz8KCkNoZWVycywKClBhdWwKCl9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmRyaS1kZXZlbCBtYWlsaW5nIGxp c3QKZHJpLWRldmVsQGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3RzLmZyZWVkZXNr dG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RyaS1kZXZlbA==