From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from lb3-smtp-cloud8.xs4all.net ([194.109.24.29]:41247 "EHLO lb3-smtp-cloud8.xs4all.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751300AbdHXLhm (ORCPT ); Thu, 24 Aug 2017 07:37:42 -0400 Subject: Re: DRM Format Modifiers in v4l2 To: Brian Starkey Cc: Daniel Vetter , "linux-media@vger.kernel.org" , jonathan.chai@arm.com, Laurent Pinchart , dri-devel References: <20170821155203.GB38943@e107564-lin.cambridge.arm.com> <47128f36-2990-bd45-ead9-06a31ed8cde0@xs4all.nl> <20170824111430.GB25711@e107564-lin.cambridge.arm.com> From: Hans Verkuil Message-ID: Date: Thu, 24 Aug 2017 13:37:35 +0200 MIME-Version: 1.0 In-Reply-To: <20170824111430.GB25711@e107564-lin.cambridge.arm.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-media-owner@vger.kernel.org List-ID: On 08/24/17 13:14, Brian Starkey wrote: > Hi Hans, > > On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote: >> On 08/21/2017 06:01 PM, Daniel Vetter wrote: >>> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey wrote: >>>> Hi all, >>>> >>>> I couldn't find this topic talked about elsewhere, but apologies if >>>> it's a duplicate - I'll be glad to be steered in the direction of a >>>> thread. >>>> >>>> We'd like to support DRM format modifiers in v4l2 in order to share >>>> the description of different (mostly proprietary) buffer formats >>>> between e.g. a v4l2 device and a DRM device. >>>> >>>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and >>>> are a vendor-namespaced 64-bit value used to describe various >>>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC >>>> code to give a complete description of the data contained in a buffer. >>>> >>>> The same modifier definition is used in the Khronos EGL extension >>>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the >>>> Wayland linux-dmabuf protocol. >>>> >>>> >>>> This buffer information could of course be described in the >>>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the >>>> information already defined in drm_fourcc.h. Additionally, there >>>> would be quite a format explosion where a device supports a dozen or >>>> more formats, all of which can use one or more different >>>> layouts/compression schemes. >>>> >>>> So, I'm wondering if anyone has views on how/whether this could be >>>> incorporated? >>>> >>>> I spoke briefly about this to Laurent at LPC last year, and he >>>> suggested v4l2_control as one approach. >>>> >>>> I also wondered if could be added in v4l2_pix_format_mplane - looks >>>> like there's 8 bytes left before it exceeds the 200 bytes, or could go >>>> in the reserved portion of v4l2_plane_pix_format. >>>> >>>> Thanks for any thoughts, >>> >>> One problem is that the modifers sometimes reference the DRM fourcc >>> codes. v4l has a different (and incompatible set) of fourcc codes, >>> whereas all the protocols and specs (you can add DRI3.1 for Xorg to >>> that list btw) use both drm fourcc and drm modifiers. >>> >>> This might or might not make this proposal unworkable, but it's >>> something I'd at least review carefully. >>> >>> Otherwise I think it'd be great if we could have one namespace for all >>> modifiers, that's pretty much why we have them. Please also note that >>> for drm_fourcc.h we don't require an in-kernel user for a new modifier >>> since a bunch of them might need to be allocated just for >>> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example >>> for this would be compressed surfaces with fast-clearing, which is >>> planned for i915 (but current hw can't scan it out). And we really >>> want to have one namespace for everything. >> >> Who sets these modifiers? Kernel or userspace? Or can it be set by both? >> I assume any userspace code that sets/reads this is code specific for that >> hardware? > > I think normally the modifier would be set by userspace. However it > might not necessarily be device-specific code. In DRM the intention is > for userspace to query the set of modifiers which are supported, and > then use them without necessarily knowing exactly what they mean > (insofar as that is possible). > > e.g. if I have two devices which support MODIFIER_FOO, I could attempt > to share a buffer between them which uses MODIFIER_FOO without > necessarily knowing exactly what it is/does. > >> >> I think Laurent's suggestion of using a 64 bit V4L2 control for this makes >> the most sense. >> >> Especially if you can assume that whoever sets this knows the hardware. >> >> I think this only makes sense if you pass buffers from one HW device to another. >> >> Because you cannot expect generic video capture code to be able to interpret >> all the zillion different combinations of modifiers. > > I don't quite follow this last bit. The control could report the set > of supported modifiers. What I mean was: an application can use the modifier to give buffers from one device to another without needing to understand it. But a generic video capture application that processes the video itself cannot be expected to know about the modifiers. It's a custom HW specific format that you only use between two HW devices or with software written for that hardware. > > However, in DRM the API lets you get the supported formats for each > modifier as-well-as the modifier list itself. I'm not sure how exactly > to provide that in a control. We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the available modifiers. So enumerating these modifiers would work out-of-the-box. Regards, Hans From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hans Verkuil Subject: Re: DRM Format Modifiers in v4l2 Date: Thu, 24 Aug 2017 13:37:35 +0200 Message-ID: References: <20170821155203.GB38943@e107564-lin.cambridge.arm.com> <47128f36-2990-bd45-ead9-06a31ed8cde0@xs4all.nl> <20170824111430.GB25711@e107564-lin.cambridge.arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from lb2-smtp-cloud8.xs4all.net (lb2-smtp-cloud8.xs4all.net [194.109.24.25]) by gabe.freedesktop.org (Postfix) with ESMTPS id DDE7B6E0ED for ; Thu, 24 Aug 2017 11:37:42 +0000 (UTC) In-Reply-To: <20170824111430.GB25711@e107564-lin.cambridge.arm.com> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Brian Starkey Cc: dri-devel , jonathan.chai@arm.com, Laurent Pinchart , "linux-media@vger.kernel.org" List-Id: dri-devel@lists.freedesktop.org T24gMDgvMjQvMTcgMTM6MTQsIEJyaWFuIFN0YXJrZXkgd3JvdGU6Cj4gSGkgSGFucywKPiAKPiBP biBNb24sIEF1ZyAyMSwgMjAxNyBhdCAwNjozNjoyOVBNICswMjAwLCBIYW5zIFZlcmt1aWwgd3Jv dGU6Cj4+IE9uIDA4LzIxLzIwMTcgMDY6MDEgUE0sIERhbmllbCBWZXR0ZXIgd3JvdGU6Cj4+PiBP biBNb24sIEF1ZyAyMSwgMjAxNyBhdCA1OjUyIFBNLCBCcmlhbiBTdGFya2V5IDxicmlhbi5zdGFy a2V5QGFybS5jb20+IHdyb3RlOgo+Pj4+IEhpIGFsbCwKPj4+Pgo+Pj4+IEkgY291bGRuJ3QgZmlu ZCB0aGlzIHRvcGljIHRhbGtlZCBhYm91dCBlbHNld2hlcmUsIGJ1dCBhcG9sb2dpZXMgaWYKPj4+ PiBpdCdzIGEgZHVwbGljYXRlIC0gSSdsbCBiZSBnbGFkIHRvIGJlIHN0ZWVyZWQgaW4gdGhlIGRp cmVjdGlvbiBvZiBhCj4+Pj4gdGhyZWFkLgo+Pj4+Cj4+Pj4gV2UnZCBsaWtlIHRvIHN1cHBvcnQg RFJNIGZvcm1hdCBtb2RpZmllcnMgaW4gdjRsMiBpbiBvcmRlciB0byBzaGFyZQo+Pj4+IHRoZSBk ZXNjcmlwdGlvbiBvZiBkaWZmZXJlbnQgKG1vc3RseSBwcm9wcmlldGFyeSkgYnVmZmVyIGZvcm1h dHMKPj4+PiBiZXR3ZWVuIGUuZy4gYSB2NGwyIGRldmljZSBhbmQgYSBEUk0gZGV2aWNlLgo+Pj4+ Cj4+Pj4gRFJNIGZvcm1hdCBtb2RpZmllcnMgYXJlIGRlZmluZWQgaW4gaW5jbHVkZS91YXBpL2Ry bS9kcm1fZm91cmNjLmggYW5kCj4+Pj4gYXJlIGEgdmVuZG9yLW5hbWVzcGFjZWQgNjQtYml0IHZh bHVlIHVzZWQgdG8gZGVzY3JpYmUgdmFyaW91cwo+Pj4+IHZlbmRvci1zcGVjaWZpYyBidWZmZXIg bGF5b3V0cy4gVGhleSBhcmUgY29tYmluZWQgd2l0aCBhIChEUk0pIEZvdXJDQwo+Pj4+IGNvZGUg dG8gZ2l2ZSBhIGNvbXBsZXRlIGRlc2NyaXB0aW9uIG9mIHRoZSBkYXRhIGNvbnRhaW5lZCBpbiBh IGJ1ZmZlci4KPj4+Pgo+Pj4+IFRoZSBzYW1lIG1vZGlmaWVyIGRlZmluaXRpb24gaXMgdXNlZCBp biB0aGUgS2hyb25vcyBFR0wgZXh0ZW5zaW9uCj4+Pj4gRUdMX0VYVF9pbWFnZV9kbWFfYnVmX2lt cG9ydF9tb2RpZmllcnMsIGFuZCBpcyBzdXBwb3J0ZWQgaW4gdGhlCj4+Pj4gV2F5bGFuZCBsaW51 eC1kbWFidWYgcHJvdG9jb2wuCj4+Pj4KPj4+Pgo+Pj4+IFRoaXMgYnVmZmVyIGluZm9ybWF0aW9u IGNvdWxkIG9mIGNvdXJzZSBiZSBkZXNjcmliZWQgaW4gdGhlCj4+Pj4gdmVuZG9yLXNwZWNpZmlj IHBhcnQgb2YgVjRMMl9QSVhfRk1UXyosIGJ1dCB0aGlzIHdvdWxkIGR1cGxpY2F0ZSB0aGUKPj4+ PiBpbmZvcm1hdGlvbiBhbHJlYWR5IGRlZmluZWQgaW4gZHJtX2ZvdXJjYy5oLiBBZGRpdGlvbmFs bHksIHRoZXJlCj4+Pj4gd291bGQgYmUgcXVpdGUgYSBmb3JtYXQgZXhwbG9zaW9uIHdoZXJlIGEg ZGV2aWNlIHN1cHBvcnRzIGEgZG96ZW4gb3IKPj4+PiBtb3JlIGZvcm1hdHMsIGFsbCBvZiB3aGlj aCBjYW4gdXNlIG9uZSBvciBtb3JlIGRpZmZlcmVudAo+Pj4+IGxheW91dHMvY29tcHJlc3Npb24g c2NoZW1lcy4KPj4+Pgo+Pj4+IFNvLCBJJ20gd29uZGVyaW5nIGlmIGFueW9uZSBoYXMgdmlld3Mg b24gaG93L3doZXRoZXIgdGhpcyBjb3VsZCBiZQo+Pj4+IGluY29ycG9yYXRlZD8KPj4+Pgo+Pj4+ IEkgc3Bva2UgYnJpZWZseSBhYm91dCB0aGlzIHRvIExhdXJlbnQgYXQgTFBDIGxhc3QgeWVhciwg YW5kIGhlCj4+Pj4gc3VnZ2VzdGVkIHY0bDJfY29udHJvbCBhcyBvbmUgYXBwcm9hY2guCj4+Pj4K Pj4+PiBJIGFsc28gd29uZGVyZWQgaWYgY291bGQgYmUgYWRkZWQgaW4gdjRsMl9waXhfZm9ybWF0 X21wbGFuZSAtIGxvb2tzCj4+Pj4gbGlrZSB0aGVyZSdzIDggYnl0ZXMgbGVmdCBiZWZvcmUgaXQg ZXhjZWVkcyB0aGUgMjAwIGJ5dGVzLCBvciBjb3VsZCBnbwo+Pj4+IGluIHRoZSByZXNlcnZlZCBw b3J0aW9uIG9mIHY0bDJfcGxhbmVfcGl4X2Zvcm1hdC4KPj4+Pgo+Pj4+IFRoYW5rcyBmb3IgYW55 IHRob3VnaHRzLAo+Pj4KPj4+IE9uZSBwcm9ibGVtIGlzIHRoYXQgdGhlIG1vZGlmZXJzIHNvbWV0 aW1lcyByZWZlcmVuY2UgdGhlIERSTSBmb3VyY2MKPj4+IGNvZGVzLiB2NGwgaGFzIGEgZGlmZmVy ZW50IChhbmQgaW5jb21wYXRpYmxlIHNldCkgb2YgZm91cmNjIGNvZGVzLAo+Pj4gd2hlcmVhcyBh bGwgdGhlIHByb3RvY29scyBhbmQgc3BlY3MgKHlvdSBjYW4gYWRkIERSSTMuMSBmb3IgWG9yZyB0 bwo+Pj4gdGhhdCBsaXN0IGJ0dykgdXNlIGJvdGggZHJtIGZvdXJjYyBhbmQgZHJtIG1vZGlmaWVy cy4KPj4+Cj4+PiBUaGlzIG1pZ2h0IG9yIG1pZ2h0IG5vdCBtYWtlIHRoaXMgcHJvcG9zYWwgdW53 b3JrYWJsZSwgYnV0IGl0J3MKPj4+IHNvbWV0aGluZyBJJ2QgYXQgbGVhc3QgcmV2aWV3IGNhcmVm dWxseS4KPj4+Cj4+PiBPdGhlcndpc2UgSSB0aGluayBpdCdkIGJlIGdyZWF0IGlmIHdlIGNvdWxk IGhhdmUgb25lIG5hbWVzcGFjZSBmb3IgYWxsCj4+PiBtb2RpZmllcnMsIHRoYXQncyBwcmV0dHkg bXVjaCB3aHkgd2UgaGF2ZSB0aGVtLiBQbGVhc2UgYWxzbyBub3RlIHRoYXQKPj4+IGZvciBkcm1f Zm91cmNjLmggd2UgZG9uJ3QgcmVxdWlyZSBhbiBpbi1rZXJuZWwgdXNlciBmb3IgYSBuZXcgbW9k aWZpZXIKPj4+IHNpbmNlIGEgYnVuY2ggb2YgdGhlbSBtaWdodCBuZWVkIHRvIGJlIGFsbG9jYXRl ZCBqdXN0IGZvcgo+Pj4gdXNlcnNwYWNlLXRvLXVzZXJzcGFjZSBidWZmZXIgc2hhcmluZyAoZS5n LiBpbiBFR0wvdmspLiBPbmUgZXhhbXBsZQo+Pj4gZm9yIHRoaXMgd291bGQgYmUgY29tcHJlc3Nl ZCBzdXJmYWNlcyB3aXRoIGZhc3QtY2xlYXJpbmcsIHdoaWNoIGlzCj4+PiBwbGFubmVkIGZvciBp OTE1IChidXQgY3VycmVudCBodyBjYW4ndCBzY2FuIGl0IG91dCkuIEFuZCB3ZSByZWFsbHkKPj4+ IHdhbnQgdG8gaGF2ZSBvbmUgbmFtZXNwYWNlIGZvciBldmVyeXRoaW5nLgo+Pgo+PiBXaG8gc2V0 cyB0aGVzZSBtb2RpZmllcnM/IEtlcm5lbCBvciB1c2Vyc3BhY2U/IE9yIGNhbiBpdCBiZSBzZXQg YnkgYm90aD8KPj4gSSBhc3N1bWUgYW55IHVzZXJzcGFjZSBjb2RlIHRoYXQgc2V0cy9yZWFkcyB0 aGlzIGlzIGNvZGUgc3BlY2lmaWMgZm9yIHRoYXQKPj4gaGFyZHdhcmU/Cj4gCj4gSSB0aGluayBu b3JtYWxseSB0aGUgbW9kaWZpZXIgd291bGQgYmUgc2V0IGJ5IHVzZXJzcGFjZS4gSG93ZXZlciBp dAo+IG1pZ2h0IG5vdCBuZWNlc3NhcmlseSBiZSBkZXZpY2Utc3BlY2lmaWMgY29kZS4gSW4gRFJN IHRoZSBpbnRlbnRpb24gaXMKPiBmb3IgdXNlcnNwYWNlIHRvIHF1ZXJ5IHRoZSBzZXQgb2YgbW9k aWZpZXJzIHdoaWNoIGFyZSBzdXBwb3J0ZWQsIGFuZAo+IHRoZW4gdXNlIHRoZW0gd2l0aG91dCBu ZWNlc3NhcmlseSBrbm93aW5nIGV4YWN0bHkgd2hhdCB0aGV5IG1lYW4KPiAoaW5zb2ZhciBhcyB0 aGF0IGlzIHBvc3NpYmxlKS4KPiAKPiBlLmcuIGlmIEkgaGF2ZSB0d28gZGV2aWNlcyB3aGljaCBz dXBwb3J0IE1PRElGSUVSX0ZPTywgSSBjb3VsZCBhdHRlbXB0Cj4gdG8gc2hhcmUgYSBidWZmZXIg YmV0d2VlbiB0aGVtIHdoaWNoIHVzZXMgTU9ESUZJRVJfRk9PIHdpdGhvdXQKPiBuZWNlc3Nhcmls eSBrbm93aW5nIGV4YWN0bHkgd2hhdCBpdCBpcy9kb2VzLgo+IAo+Pgo+PiBJIHRoaW5rIExhdXJl bnQncyBzdWdnZXN0aW9uIG9mIHVzaW5nIGEgNjQgYml0IFY0TDIgY29udHJvbCBmb3IgdGhpcyBt YWtlcwo+PiB0aGUgbW9zdCBzZW5zZS4KPj4KPj4gRXNwZWNpYWxseSBpZiB5b3UgY2FuIGFzc3Vt ZSB0aGF0IHdob2V2ZXIgc2V0cyB0aGlzIGtub3dzIHRoZSBoYXJkd2FyZS4KPj4KPj4gSSB0aGlu ayB0aGlzIG9ubHkgbWFrZXMgc2Vuc2UgaWYgeW91IHBhc3MgYnVmZmVycyBmcm9tIG9uZSBIVyBk ZXZpY2UgdG8gYW5vdGhlci4KPj4KPj4gQmVjYXVzZSB5b3UgY2Fubm90IGV4cGVjdCBnZW5lcmlj IHZpZGVvIGNhcHR1cmUgY29kZSB0byBiZSBhYmxlIHRvIGludGVycHJldAo+PiBhbGwgdGhlIHpp bGxpb24gZGlmZmVyZW50IGNvbWJpbmF0aW9ucyBvZiBtb2RpZmllcnMuCj4gCj4gSSBkb24ndCBx dWl0ZSBmb2xsb3cgdGhpcyBsYXN0IGJpdC4gVGhlIGNvbnRyb2wgY291bGQgcmVwb3J0IHRoZSBz ZXQKPiBvZiBzdXBwb3J0ZWQgbW9kaWZpZXJzLgoKV2hhdCBJIG1lYW4gd2FzOiBhbiBhcHBsaWNh dGlvbiBjYW4gdXNlIHRoZSBtb2RpZmllciB0byBnaXZlIGJ1ZmZlcnMgZnJvbQpvbmUgZGV2aWNl IHRvIGFub3RoZXIgd2l0aG91dCBuZWVkaW5nIHRvIHVuZGVyc3RhbmQgaXQuCgpCdXQgYSBnZW5l cmljIHZpZGVvIGNhcHR1cmUgYXBwbGljYXRpb24gdGhhdCBwcm9jZXNzZXMgdGhlIHZpZGVvIGl0 c2VsZgpjYW5ub3QgYmUgZXhwZWN0ZWQgdG8ga25vdyBhYm91dCB0aGUgbW9kaWZpZXJzLiBJdCdz IGEgY3VzdG9tIEhXIHNwZWNpZmljCmZvcm1hdCB0aGF0IHlvdSBvbmx5IHVzZSBiZXR3ZWVuIHR3 byBIVyBkZXZpY2VzIG9yIHdpdGggc29mdHdhcmUgd3JpdHRlbgpmb3IgdGhhdCBoYXJkd2FyZS4K Cj4gCj4gSG93ZXZlciwgaW4gRFJNIHRoZSBBUEkgbGV0cyB5b3UgZ2V0IHRoZSBzdXBwb3J0ZWQg Zm9ybWF0cyBmb3IgZWFjaAo+IG1vZGlmaWVyIGFzLXdlbGwtYXMgdGhlIG1vZGlmaWVyIGxpc3Qg aXRzZWxmLiBJJ20gbm90IHN1cmUgaG93IGV4YWN0bHkKPiB0byBwcm92aWRlIHRoYXQgaW4gYSBj b250cm9sLgoKV2UgaGF2ZSBzdXBwb3J0IGZvciBhICdtZW51JyBvZiA2NCBiaXQgaW50ZWdlcnM6 IFY0TDJfQ1RSTF9UWVBFX0lOVEVHRVJfTUVOVS4KWW91IHVzZSBWSURJT0NfUVVFUllNRU5VIHRv IGVudW1lcmF0ZSB0aGUgYXZhaWxhYmxlIG1vZGlmaWVycy4KClNvIGVudW1lcmF0aW5nIHRoZXNl IG1vZGlmaWVycyB3b3VsZCB3b3JrIG91dC1vZi10aGUtYm94LgoKUmVnYXJkcywKCglIYW5zCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmRyaS1kZXZlbCBt YWlsaW5nIGxpc3QKZHJpLWRldmVsQGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3Rz LmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RyaS1kZXZlbAo=