DRM Format Modifiers in v4l2

DRM Format Modifiers in v4l2

[Brian Starkey]

Hi all,

I couldn't find this topic talked about elsewhere, but apologies if
it's a duplicate - I'll be glad to be steered in the direction of a
thread.

We'd like to support DRM format modifiers in v4l2 in order to share
the description of different (mostly proprietary) buffer formats
between e.g. a v4l2 device and a DRM device.

DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
are a vendor-namespaced 64-bit value used to describe various
vendor-specific buffer layouts. They are combined with a (DRM) FourCC
code to give a complete description of the data contained in a buffer.

The same modifier definition is used in the Khronos EGL extension
EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
Wayland linux-dmabuf protocol.


This buffer information could of course be described in the
vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
information already defined in drm_fourcc.h. Additionally, there
would be quite a format explosion where a device supports a dozen or
more formats, all of which can use one or more different
layouts/compression schemes.

So, I'm wondering if anyone has views on how/whether this could be
incorporated?

I spoke briefly about this to Laurent at LPC last year, and he
suggested v4l2_control as one approach.

I also wondered if could be added in v4l2_pix_format_mplane - looks
like there's 8 bytes left before it exceeds the 200 bytes, or could go
in the reserved portion of v4l2_plane_pix_format.

Thanks for any thoughts,
-Brian

Re: DRM Format Modifiers in v4l2

[Daniel Vetter]

On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey <brian.starkey@arm.com> wrote:
> Hi all,
>
> I couldn't find this topic talked about elsewhere, but apologies if
> it's a duplicate - I'll be glad to be steered in the direction of a
> thread.
>
> We'd like to support DRM format modifiers in v4l2 in order to share
> the description of different (mostly proprietary) buffer formats
> between e.g. a v4l2 device and a DRM device.
>
> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
> are a vendor-namespaced 64-bit value used to describe various
> vendor-specific buffer layouts. They are combined with a (DRM) FourCC
> code to give a complete description of the data contained in a buffer.
>
> The same modifier definition is used in the Khronos EGL extension
> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
> Wayland linux-dmabuf protocol.
>
>
> This buffer information could of course be described in the
> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
> information already defined in drm_fourcc.h. Additionally, there
> would be quite a format explosion where a device supports a dozen or
> more formats, all of which can use one or more different
> layouts/compression schemes.
>
> So, I'm wondering if anyone has views on how/whether this could be
> incorporated?
>
> I spoke briefly about this to Laurent at LPC last year, and he
> suggested v4l2_control as one approach.
>
> I also wondered if could be added in v4l2_pix_format_mplane - looks
> like there's 8 bytes left before it exceeds the 200 bytes, or could go
> in the reserved portion of v4l2_plane_pix_format.
>
> Thanks for any thoughts,

One problem is that the modifers sometimes reference the DRM fourcc
codes. v4l has a different (and incompatible set) of fourcc codes,
whereas all the protocols and specs (you can add DRI3.1 for Xorg to
that list btw) use both drm fourcc and drm modifiers.

This might or might not make this proposal unworkable, but it's
something I'd at least review carefully.

Otherwise I think it'd be great if we could have one namespace for all
modifiers, that's pretty much why we have them. Please also note that
for drm_fourcc.h we don't require an in-kernel user for a new modifier
since a bunch of them might need to be allocated just for
userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
for this would be compressed surfaces with fast-clearing, which is
planned for i915 (but current hw can't scan it out). And we really
want to have one namespace for everything.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: DRM Format Modifiers in v4l2

[Brian Starkey]

On Mon, Aug 21, 2017 at 06:01:24PM +0200, Daniel Vetter wrote:
>On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey <brian.starkey@arm.com> wrote:
>> Hi all,
>>
>> I couldn't find this topic talked about elsewhere, but apologies if
>> it's a duplicate - I'll be glad to be steered in the direction of a
>> thread.
>>
>> We'd like to support DRM format modifiers in v4l2 in order to share
>> the description of different (mostly proprietary) buffer formats
>> between e.g. a v4l2 device and a DRM device.
>>
>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
>> are a vendor-namespaced 64-bit value used to describe various
>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC
>> code to give a complete description of the data contained in a buffer.
>>
>> The same modifier definition is used in the Khronos EGL extension
>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
>> Wayland linux-dmabuf protocol.
>>
>>
>> This buffer information could of course be described in the
>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
>> information already defined in drm_fourcc.h. Additionally, there
>> would be quite a format explosion where a device supports a dozen or
>> more formats, all of which can use one or more different
>> layouts/compression schemes.
>>
>> So, I'm wondering if anyone has views on how/whether this could be
>> incorporated?
>>
>> I spoke briefly about this to Laurent at LPC last year, and he
>> suggested v4l2_control as one approach.
>>
>> I also wondered if could be added in v4l2_pix_format_mplane - looks
>> like there's 8 bytes left before it exceeds the 200 bytes, or could go
>> in the reserved portion of v4l2_plane_pix_format.
>>
>> Thanks for any thoughts,
>
>One problem is that the modifers sometimes reference the DRM fourcc
>codes. v4l has a different (and incompatible set) of fourcc codes,
>whereas all the protocols and specs (you can add DRI3.1 for Xorg to
>that list btw) use both drm fourcc and drm modifiers.
>

This problem already exists (ignoring modifiers) in the case of any
v4l2 <-> DRM buffer sharing (direct video scanout, for instance).

I was hoping it would be possible to draw enough equivalency between
the different definitions to manage a useful subset through a 1:1
lookup table. If that's not possible then this gets a whole lot more
tricky. Are you already aware of incompatibilities which would prevent
it?

-Brian

>This might or might not make this proposal unworkable, but it's
>something I'd at least review carefully.
>
>Otherwise I think it'd be great if we could have one namespace for all
>modifiers, that's pretty much why we have them. Please also note that
>for drm_fourcc.h we don't require an in-kernel user for a new modifier
>since a bunch of them might need to be allocated just for
>userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
>for this would be compressed surfaces with fast-clearing, which is
>planned for i915 (but current hw can't scan it out). And we really
>want to have one namespace for everything.
>-Daniel
>-- 
>Daniel Vetter
>Software Engineer, Intel Corporation
>+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: DRM Format Modifiers in v4l2

[Hans Verkuil]

On 08/21/2017 06:01 PM, Daniel Vetter wrote:
> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey <brian.starkey@arm.com> wrote:
>> Hi all,
>>
>> I couldn't find this topic talked about elsewhere, but apologies if
>> it's a duplicate - I'll be glad to be steered in the direction of a
>> thread.
>>
>> We'd like to support DRM format modifiers in v4l2 in order to share
>> the description of different (mostly proprietary) buffer formats
>> between e.g. a v4l2 device and a DRM device.
>>
>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
>> are a vendor-namespaced 64-bit value used to describe various
>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC
>> code to give a complete description of the data contained in a buffer.
>>
>> The same modifier definition is used in the Khronos EGL extension
>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
>> Wayland linux-dmabuf protocol.
>>
>>
>> This buffer information could of course be described in the
>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
>> information already defined in drm_fourcc.h. Additionally, there
>> would be quite a format explosion where a device supports a dozen or
>> more formats, all of which can use one or more different
>> layouts/compression schemes.
>>
>> So, I'm wondering if anyone has views on how/whether this could be
>> incorporated?
>>
>> I spoke briefly about this to Laurent at LPC last year, and he
>> suggested v4l2_control as one approach.
>>
>> I also wondered if could be added in v4l2_pix_format_mplane - looks
>> like there's 8 bytes left before it exceeds the 200 bytes, or could go
>> in the reserved portion of v4l2_plane_pix_format.
>>
>> Thanks for any thoughts,
> 
> One problem is that the modifers sometimes reference the DRM fourcc
> codes. v4l has a different (and incompatible set) of fourcc codes,
> whereas all the protocols and specs (you can add DRI3.1 for Xorg to
> that list btw) use both drm fourcc and drm modifiers.
> 
> This might or might not make this proposal unworkable, but it's
> something I'd at least review carefully.
> 
> Otherwise I think it'd be great if we could have one namespace for all
> modifiers, that's pretty much why we have them. Please also note that
> for drm_fourcc.h we don't require an in-kernel user for a new modifier
> since a bunch of them might need to be allocated just for
> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
> for this would be compressed surfaces with fast-clearing, which is
> planned for i915 (but current hw can't scan it out). And we really
> want to have one namespace for everything.

Who sets these modifiers? Kernel or userspace? Or can it be set by both?
I assume any userspace code that sets/reads this is code specific for that
hardware?

I think Laurent's suggestion of using a 64 bit V4L2 control for this makes
the most sense.

Especially if you can assume that whoever sets this knows the hardware.

I think this only makes sense if you pass buffers from one HW device to another.

Because you cannot expect generic video capture code to be able to interpret
all the zillion different combinations of modifiers.

Regards,

	Hans

Re: DRM Format Modifiers in v4l2

[Brian Starkey]

Hi Hans,

On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
>On 08/21/2017 06:01 PM, Daniel Vetter wrote:
>> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey <brian.starkey@arm.com> wrote:
>>> Hi all,
>>>
>>> I couldn't find this topic talked about elsewhere, but apologies if
>>> it's a duplicate - I'll be glad to be steered in the direction of a
>>> thread.
>>>
>>> We'd like to support DRM format modifiers in v4l2 in order to share
>>> the description of different (mostly proprietary) buffer formats
>>> between e.g. a v4l2 device and a DRM device.
>>>
>>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
>>> are a vendor-namespaced 64-bit value used to describe various
>>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC
>>> code to give a complete description of the data contained in a buffer.
>>>
>>> The same modifier definition is used in the Khronos EGL extension
>>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
>>> Wayland linux-dmabuf protocol.
>>>
>>>
>>> This buffer information could of course be described in the
>>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
>>> information already defined in drm_fourcc.h. Additionally, there
>>> would be quite a format explosion where a device supports a dozen or
>>> more formats, all of which can use one or more different
>>> layouts/compression schemes.
>>>
>>> So, I'm wondering if anyone has views on how/whether this could be
>>> incorporated?
>>>
>>> I spoke briefly about this to Laurent at LPC last year, and he
>>> suggested v4l2_control as one approach.
>>>
>>> I also wondered if could be added in v4l2_pix_format_mplane - looks
>>> like there's 8 bytes left before it exceeds the 200 bytes, or could go
>>> in the reserved portion of v4l2_plane_pix_format.
>>>
>>> Thanks for any thoughts,
>>
>> One problem is that the modifers sometimes reference the DRM fourcc
>> codes. v4l has a different (and incompatible set) of fourcc codes,
>> whereas all the protocols and specs (you can add DRI3.1 for Xorg to
>> that list btw) use both drm fourcc and drm modifiers.
>>
>> This might or might not make this proposal unworkable, but it's
>> something I'd at least review carefully.
>>
>> Otherwise I think it'd be great if we could have one namespace for all
>> modifiers, that's pretty much why we have them. Please also note that
>> for drm_fourcc.h we don't require an in-kernel user for a new modifier
>> since a bunch of them might need to be allocated just for
>> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
>> for this would be compressed surfaces with fast-clearing, which is
>> planned for i915 (but current hw can't scan it out). And we really
>> want to have one namespace for everything.
>
>Who sets these modifiers? Kernel or userspace? Or can it be set by both?
>I assume any userspace code that sets/reads this is code specific for that
>hardware?

I think normally the modifier would be set by userspace. However it
might not necessarily be device-specific code. In DRM the intention is
for userspace to query the set of modifiers which are supported, and
then use them without necessarily knowing exactly what they mean
(insofar as that is possible).

e.g. if I have two devices which support MODIFIER_FOO, I could attempt
to share a buffer between them which uses MODIFIER_FOO without
necessarily knowing exactly what it is/does.

>
>I think Laurent's suggestion of using a 64 bit V4L2 control for this makes
>the most sense.
>
>Especially if you can assume that whoever sets this knows the hardware.
>
>I think this only makes sense if you pass buffers from one HW device to another.
>
>Because you cannot expect generic video capture code to be able to interpret
>all the zillion different combinations of modifiers.

I don't quite follow this last bit. The control could report the set
of supported modifiers.

However, in DRM the API lets you get the supported formats for each
modifier as-well-as the modifier list itself. I'm not sure how exactly
to provide that in a control.

Thanks,
-Brian

>
>Regards,
>
>	Hans

Re: DRM Format Modifiers in v4l2

[Hans Verkuil]

On 08/24/17 13:14, Brian Starkey wrote:
> Hi Hans,
> 
> On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
>> On 08/21/2017 06:01 PM, Daniel Vetter wrote:
>>> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey <brian.starkey@arm.com> wrote:
>>>> Hi all,
>>>>
>>>> I couldn't find this topic talked about elsewhere, but apologies if
>>>> it's a duplicate - I'll be glad to be steered in the direction of a
>>>> thread.
>>>>
>>>> We'd like to support DRM format modifiers in v4l2 in order to share
>>>> the description of different (mostly proprietary) buffer formats
>>>> between e.g. a v4l2 device and a DRM device.
>>>>
>>>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
>>>> are a vendor-namespaced 64-bit value used to describe various
>>>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC
>>>> code to give a complete description of the data contained in a buffer.
>>>>
>>>> The same modifier definition is used in the Khronos EGL extension
>>>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
>>>> Wayland linux-dmabuf protocol.
>>>>
>>>>
>>>> This buffer information could of course be described in the
>>>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
>>>> information already defined in drm_fourcc.h. Additionally, there
>>>> would be quite a format explosion where a device supports a dozen or
>>>> more formats, all of which can use one or more different
>>>> layouts/compression schemes.
>>>>
>>>> So, I'm wondering if anyone has views on how/whether this could be
>>>> incorporated?
>>>>
>>>> I spoke briefly about this to Laurent at LPC last year, and he
>>>> suggested v4l2_control as one approach.
>>>>
>>>> I also wondered if could be added in v4l2_pix_format_mplane - looks
>>>> like there's 8 bytes left before it exceeds the 200 bytes, or could go
>>>> in the reserved portion of v4l2_plane_pix_format.
>>>>
>>>> Thanks for any thoughts,
>>>
>>> One problem is that the modifers sometimes reference the DRM fourcc
>>> codes. v4l has a different (and incompatible set) of fourcc codes,
>>> whereas all the protocols and specs (you can add DRI3.1 for Xorg to
>>> that list btw) use both drm fourcc and drm modifiers.
>>>
>>> This might or might not make this proposal unworkable, but it's
>>> something I'd at least review carefully.
>>>
>>> Otherwise I think it'd be great if we could have one namespace for all
>>> modifiers, that's pretty much why we have them. Please also note that
>>> for drm_fourcc.h we don't require an in-kernel user for a new modifier
>>> since a bunch of them might need to be allocated just for
>>> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
>>> for this would be compressed surfaces with fast-clearing, which is
>>> planned for i915 (but current hw can't scan it out). And we really
>>> want to have one namespace for everything.
>>
>> Who sets these modifiers? Kernel or userspace? Or can it be set by both?
>> I assume any userspace code that sets/reads this is code specific for that
>> hardware?
> 
> I think normally the modifier would be set by userspace. However it
> might not necessarily be device-specific code. In DRM the intention is
> for userspace to query the set of modifiers which are supported, and
> then use them without necessarily knowing exactly what they mean
> (insofar as that is possible).
> 
> e.g. if I have two devices which support MODIFIER_FOO, I could attempt
> to share a buffer between them which uses MODIFIER_FOO without
> necessarily knowing exactly what it is/does.
> 
>>
>> I think Laurent's suggestion of using a 64 bit V4L2 control for this makes
>> the most sense.
>>
>> Especially if you can assume that whoever sets this knows the hardware.
>>
>> I think this only makes sense if you pass buffers from one HW device to another.
>>
>> Because you cannot expect generic video capture code to be able to interpret
>> all the zillion different combinations of modifiers.
> 
> I don't quite follow this last bit. The control could report the set
> of supported modifiers.

What I mean was: an application can use the modifier to give buffers from
one device to another without needing to understand it.

But a generic video capture application that processes the video itself
cannot be expected to know about the modifiers. It's a custom HW specific
format that you only use between two HW devices or with software written
for that hardware.

> 
> However, in DRM the API lets you get the supported formats for each
> modifier as-well-as the modifier list itself. I'm not sure how exactly
> to provide that in a control.

We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU.
You use VIDIOC_QUERYMENU to enumerate the available modifiers.

So enumerating these modifiers would work out-of-the-box.

Regards,

	Hans

Re: DRM Format Modifiers in v4l2

[Brian Starkey]

On Thu, Aug 24, 2017 at 01:37:35PM +0200, Hans Verkuil wrote:
>On 08/24/17 13:14, Brian Starkey wrote:
>> Hi Hans,
>>
>> On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
>>> On 08/21/2017 06:01 PM, Daniel Vetter wrote:
>>>> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey <brian.starkey@arm.com> wrote:
>>>>> Hi all,
>>>>>
>>>>> I couldn't find this topic talked about elsewhere, but apologies if
>>>>> it's a duplicate - I'll be glad to be steered in the direction of a
>>>>> thread.
>>>>>
>>>>> We'd like to support DRM format modifiers in v4l2 in order to share
>>>>> the description of different (mostly proprietary) buffer formats
>>>>> between e.g. a v4l2 device and a DRM device.
>>>>>
>>>>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
>>>>> are a vendor-namespaced 64-bit value used to describe various
>>>>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC
>>>>> code to give a complete description of the data contained in a buffer.
>>>>>
>>>>> The same modifier definition is used in the Khronos EGL extension
>>>>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
>>>>> Wayland linux-dmabuf protocol.
>>>>>
>>>>>
>>>>> This buffer information could of course be described in the
>>>>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
>>>>> information already defined in drm_fourcc.h. Additionally, there
>>>>> would be quite a format explosion where a device supports a dozen or
>>>>> more formats, all of which can use one or more different
>>>>> layouts/compression schemes.
>>>>>
>>>>> So, I'm wondering if anyone has views on how/whether this could be
>>>>> incorporated?
>>>>>
>>>>> I spoke briefly about this to Laurent at LPC last year, and he
>>>>> suggested v4l2_control as one approach.
>>>>>
>>>>> I also wondered if could be added in v4l2_pix_format_mplane - looks
>>>>> like there's 8 bytes left before it exceeds the 200 bytes, or could go
>>>>> in the reserved portion of v4l2_plane_pix_format.
>>>>>
>>>>> Thanks for any thoughts,
>>>>
>>>> One problem is that the modifers sometimes reference the DRM fourcc
>>>> codes. v4l has a different (and incompatible set) of fourcc codes,
>>>> whereas all the protocols and specs (you can add DRI3.1 for Xorg to
>>>> that list btw) use both drm fourcc and drm modifiers.
>>>>
>>>> This might or might not make this proposal unworkable, but it's
>>>> something I'd at least review carefully.
>>>>
>>>> Otherwise I think it'd be great if we could have one namespace for all
>>>> modifiers, that's pretty much why we have them. Please also note that
>>>> for drm_fourcc.h we don't require an in-kernel user for a new modifier
>>>> since a bunch of them might need to be allocated just for
>>>> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
>>>> for this would be compressed surfaces with fast-clearing, which is
>>>> planned for i915 (but current hw can't scan it out). And we really
>>>> want to have one namespace for everything.
>>>
>>> Who sets these modifiers? Kernel or userspace? Or can it be set by both?
>>> I assume any userspace code that sets/reads this is code specific for that
>>> hardware?
>>
>> I think normally the modifier would be set by userspace. However it
>> might not necessarily be device-specific code. In DRM the intention is
>> for userspace to query the set of modifiers which are supported, and
>> then use them without necessarily knowing exactly what they mean
>> (insofar as that is possible).
>>
>> e.g. if I have two devices which support MODIFIER_FOO, I could attempt
>> to share a buffer between them which uses MODIFIER_FOO without
>> necessarily knowing exactly what it is/does.
>>
>>>
>>> I think Laurent's suggestion of using a 64 bit V4L2 control for this makes
>>> the most sense.
>>>
>>> Especially if you can assume that whoever sets this knows the hardware.
>>>
>>> I think this only makes sense if you pass buffers from one HW device to another.
>>>
>>> Because you cannot expect generic video capture code to be able to interpret
>>> all the zillion different combinations of modifiers.
>>
>> I don't quite follow this last bit. The control could report the set
>> of supported modifiers.
>
>What I mean was: an application can use the modifier to give buffers from
>one device to another without needing to understand it.
>
>But a generic video capture application that processes the video itself
>cannot be expected to know about the modifiers. It's a custom HW specific
>format that you only use between two HW devices or with software written
>for that hardware.
>

Yes, makes sense.

>>
>> However, in DRM the API lets you get the supported formats for each
>> modifier as-well-as the modifier list itself. I'm not sure how exactly
>> to provide that in a control.
>
>We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU.
>You use VIDIOC_QUERYMENU to enumerate the available modifiers.
>
>So enumerating these modifiers would work out-of-the-box.

Right. So I guess the supported set of formats could be somehow
enumerated in the menu item string. In DRM the pairs are (modifier +
bitmask) where bits represent formats in the supported formats list
(commit db1689aa61bd in drm-next). Printing a hex representation of
the bitmask would be functional but I concede not very pretty.

Cheers,
-Brian

>
>Regards,
>
>	Hans

Re: DRM Format Modifiers in v4l2

[Hans Verkuil]

On 24/08/17 14:26, Brian Starkey wrote:
> On Thu, Aug 24, 2017 at 01:37:35PM +0200, Hans Verkuil wrote:
>> On 08/24/17 13:14, Brian Starkey wrote:
>>> Hi Hans,
>>>
>>> On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
>>>> On 08/21/2017 06:01 PM, Daniel Vetter wrote:
>>>>> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey <brian.starkey@arm.com> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I couldn't find this topic talked about elsewhere, but apologies if
>>>>>> it's a duplicate - I'll be glad to be steered in the direction of a
>>>>>> thread.
>>>>>>
>>>>>> We'd like to support DRM format modifiers in v4l2 in order to share
>>>>>> the description of different (mostly proprietary) buffer formats
>>>>>> between e.g. a v4l2 device and a DRM device.
>>>>>>
>>>>>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
>>>>>> are a vendor-namespaced 64-bit value used to describe various
>>>>>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC
>>>>>> code to give a complete description of the data contained in a buffer.
>>>>>>
>>>>>> The same modifier definition is used in the Khronos EGL extension
>>>>>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
>>>>>> Wayland linux-dmabuf protocol.
>>>>>>
>>>>>>
>>>>>> This buffer information could of course be described in the
>>>>>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
>>>>>> information already defined in drm_fourcc.h. Additionally, there
>>>>>> would be quite a format explosion where a device supports a dozen or
>>>>>> more formats, all of which can use one or more different
>>>>>> layouts/compression schemes.
>>>>>>
>>>>>> So, I'm wondering if anyone has views on how/whether this could be
>>>>>> incorporated?
>>>>>>
>>>>>> I spoke briefly about this to Laurent at LPC last year, and he
>>>>>> suggested v4l2_control as one approach.
>>>>>>
>>>>>> I also wondered if could be added in v4l2_pix_format_mplane - looks
>>>>>> like there's 8 bytes left before it exceeds the 200 bytes, or could go
>>>>>> in the reserved portion of v4l2_plane_pix_format.
>>>>>>
>>>>>> Thanks for any thoughts,
>>>>>
>>>>> One problem is that the modifers sometimes reference the DRM fourcc
>>>>> codes. v4l has a different (and incompatible set) of fourcc codes,
>>>>> whereas all the protocols and specs (you can add DRI3.1 for Xorg to
>>>>> that list btw) use both drm fourcc and drm modifiers.
>>>>>
>>>>> This might or might not make this proposal unworkable, but it's
>>>>> something I'd at least review carefully.
>>>>>
>>>>> Otherwise I think it'd be great if we could have one namespace for all
>>>>> modifiers, that's pretty much why we have them. Please also note that
>>>>> for drm_fourcc.h we don't require an in-kernel user for a new modifier
>>>>> since a bunch of them might need to be allocated just for
>>>>> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
>>>>> for this would be compressed surfaces with fast-clearing, which is
>>>>> planned for i915 (but current hw can't scan it out). And we really
>>>>> want to have one namespace for everything.
>>>>
>>>> Who sets these modifiers? Kernel or userspace? Or can it be set by both?
>>>> I assume any userspace code that sets/reads this is code specific for that
>>>> hardware?
>>>
>>> I think normally the modifier would be set by userspace. However it
>>> might not necessarily be device-specific code. In DRM the intention is
>>> for userspace to query the set of modifiers which are supported, and
>>> then use them without necessarily knowing exactly what they mean
>>> (insofar as that is possible).
>>>
>>> e.g. if I have two devices which support MODIFIER_FOO, I could attempt
>>> to share a buffer between them which uses MODIFIER_FOO without
>>> necessarily knowing exactly what it is/does.
>>>
>>>>
>>>> I think Laurent's suggestion of using a 64 bit V4L2 control for this makes
>>>> the most sense.
>>>>
>>>> Especially if you can assume that whoever sets this knows the hardware.
>>>>
>>>> I think this only makes sense if you pass buffers from one HW device to another.
>>>>
>>>> Because you cannot expect generic video capture code to be able to interpret
>>>> all the zillion different combinations of modifiers.
>>>
>>> I don't quite follow this last bit. The control could report the set
>>> of supported modifiers.
>>
>> What I mean was: an application can use the modifier to give buffers from
>> one device to another without needing to understand it.
>>
>> But a generic video capture application that processes the video itself
>> cannot be expected to know about the modifiers. It's a custom HW specific
>> format that you only use between two HW devices or with software written
>> for that hardware.
>>
> 
> Yes, makes sense.
> 
>>>
>>> However, in DRM the API lets you get the supported formats for each
>>> modifier as-well-as the modifier list itself. I'm not sure how exactly
>>> to provide that in a control.
>>
>> We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU.
>> You use VIDIOC_QUERYMENU to enumerate the available modifiers.
>>
>> So enumerating these modifiers would work out-of-the-box.
> 
> Right. So I guess the supported set of formats could be somehow
> enumerated in the menu item string. In DRM the pairs are (modifier +
> bitmask) where bits represent formats in the supported formats list
> (commit db1689aa61bd in drm-next). Printing a hex representation of
> the bitmask would be functional but I concede not very pretty.

So this patch limits the number of formats to 64 (being the size of
the bit mask). I was hoping these modifiers applied to all formats,
but unfortunately that isn't the case apparently.

How it would work with my proposal is that the integer menu control
would reflect the list of supported modifiers for the currently selected
format. If you change format, then the available modifier list changes
as well. The advantage is that there is no '64 formats' limitation,
something that I feel very uncomfortable about since some devices support
a *lot* of formats. The disadvantage is that it is harder to get a quick
overview of all combinations for formats and modifiers.

This has more to do with limitations in the V4L2 API than with supporting
modifiers in general. We need something better to give userspace a quick
overview of all combinations of pixelformats, framesizes, frameintervals
and now modifiers. However, that's our problem :-)

Regards,

	Hans

> 
> Cheers,
> -Brian
> 
>>
>> Regards,
>>
>>     Hans
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: DRM Format Modifiers in v4l2

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 1791 bytes --]

Le jeudi 24 août 2017 à 13:26 +0100, Brian Starkey a écrit :
> > What I mean was: an application can use the modifier to give buffers from
> > one device to another without needing to understand it.
> > 
> > But a generic video capture application that processes the video itself
> > cannot be expected to know about the modifiers. It's a custom HW specific
> > format that you only use between two HW devices or with software written
> > for that hardware.
> > 
> 
> Yes, makes sense.
> 
> > > 
> > > However, in DRM the API lets you get the supported formats for each
> > > modifier as-well-as the modifier list itself. I'm not sure how exactly
> > > to provide that in a control.
> > 
> > We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU.
> > You use VIDIOC_QUERYMENU to enumerate the available modifiers.
> > 
> > So enumerating these modifiers would work out-of-the-box.
> 
> Right. So I guess the supported set of formats could be somehow
> enumerated in the menu item string. In DRM the pairs are (modifier +
> bitmask) where bits represent formats in the supported formats list
> (commit db1689aa61bd in drm-next). Printing a hex representation of
> the bitmask would be functional but I concede not very pretty.

The problem is that the list of modifiers depends on the format
selected. Having to call S_FMT to obtain this list is quite
inefficient.

Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been
implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think
an other one made it the same way recently, something from Mediatek if
I remember. Though, unlike the Intel one, the same modifier does not
have various result depending on the hardware revision.

regards,
Nicolas


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: DRM Format Modifiers in v4l2

[Daniel Vetter]

On Mon, Aug 28, 2017 at 8:07 PM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> Le jeudi 24 août 2017 à 13:26 +0100, Brian Starkey a écrit :
>> > What I mean was: an application can use the modifier to give buffers from
>> > one device to another without needing to understand it.
>> >
>> > But a generic video capture application that processes the video itself
>> > cannot be expected to know about the modifiers. It's a custom HW specific
>> > format that you only use between two HW devices or with software written
>> > for that hardware.
>> >
>>
>> Yes, makes sense.
>>
>> > >
>> > > However, in DRM the API lets you get the supported formats for each
>> > > modifier as-well-as the modifier list itself. I'm not sure how exactly
>> > > to provide that in a control.
>> >
>> > We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU.
>> > You use VIDIOC_QUERYMENU to enumerate the available modifiers.
>> >
>> > So enumerating these modifiers would work out-of-the-box.
>>
>> Right. So I guess the supported set of formats could be somehow
>> enumerated in the menu item string. In DRM the pairs are (modifier +
>> bitmask) where bits represent formats in the supported formats list
>> (commit db1689aa61bd in drm-next). Printing a hex representation of
>> the bitmask would be functional but I concede not very pretty.
>
> The problem is that the list of modifiers depends on the format
> selected. Having to call S_FMT to obtain this list is quite
> inefficient.
>
> Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been
> implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think
> an other one made it the same way recently, something from Mediatek if
> I remember. Though, unlike the Intel one, the same modifier does not
> have various result depending on the hardware revision.

Note on the intel modifers: On most recent platforms (iirc gen9) the
modifier is well defined and always describes the same byte layout. We
simply didn't want to rewrite our entire software stack for all the
old gunk platforms, hence the language. I guess we could/should
describe the layout in detail, but atm we're the only ones using it.

On your topic of v4l2 encoding the drm fourcc+modifier combo into a
special v4l fourcc: That's exactly the mismatch I was thinking of.
There's other examples of v4l2 fourcc being more specific than their
drm counters (e.g. specific way the different planes are laid out).
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: DRM Format Modifiers in v4l2

[Brian Starkey]

On Fri, Aug 25, 2017 at 10:14:03AM +0200, Hans Verkuil wrote:
>On 24/08/17 14:26, Brian Starkey wrote:
>> On Thu, Aug 24, 2017 at 01:37:35PM +0200, Hans Verkuil wrote:
>>> On 08/24/17 13:14, Brian Starkey wrote:
>>>> Hi Hans,
>>>>
>>>> On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
>>>>> On 08/21/2017 06:01 PM, Daniel Vetter wrote:
>>>>>> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey <brian.starkey@arm.com> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I couldn't find this topic talked about elsewhere, but apologies if
>>>>>>> it's a duplicate - I'll be glad to be steered in the direction of a
>>>>>>> thread.
>>>>>>>
>>>>>>> We'd like to support DRM format modifiers in v4l2 in order to share
>>>>>>> the description of different (mostly proprietary) buffer formats
>>>>>>> between e.g. a v4l2 device and a DRM device.
>>>>>>>
>>>>>>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
>>>>>>> are a vendor-namespaced 64-bit value used to describe various
>>>>>>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC
>>>>>>> code to give a complete description of the data contained in a buffer.
>>>>>>>
>>>>>>> The same modifier definition is used in the Khronos EGL extension
>>>>>>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
>>>>>>> Wayland linux-dmabuf protocol.
>>>>>>>
>>>>>>>
>>>>>>> This buffer information could of course be described in the
>>>>>>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
>>>>>>> information already defined in drm_fourcc.h. Additionally, there
>>>>>>> would be quite a format explosion where a device supports a dozen or
>>>>>>> more formats, all of which can use one or more different
>>>>>>> layouts/compression schemes.
>>>>>>>
>>>>>>> So, I'm wondering if anyone has views on how/whether this could be
>>>>>>> incorporated?
>>>>>>>
>>>>>>> I spoke briefly about this to Laurent at LPC last year, and he
>>>>>>> suggested v4l2_control as one approach.
>>>>>>>
>>>>>>> I also wondered if could be added in v4l2_pix_format_mplane - looks
>>>>>>> like there's 8 bytes left before it exceeds the 200 bytes, or could go
>>>>>>> in the reserved portion of v4l2_plane_pix_format.
>>>>>>>
>>>>>>> Thanks for any thoughts,
>>>>>>
>>>>>> One problem is that the modifers sometimes reference the DRM fourcc
>>>>>> codes. v4l has a different (and incompatible set) of fourcc codes,
>>>>>> whereas all the protocols and specs (you can add DRI3.1 for Xorg to
>>>>>> that list btw) use both drm fourcc and drm modifiers.
>>>>>>
>>>>>> This might or might not make this proposal unworkable, but it's
>>>>>> something I'd at least review carefully.
>>>>>>
>>>>>> Otherwise I think it'd be great if we could have one namespace for all
>>>>>> modifiers, that's pretty much why we have them. Please also note that
>>>>>> for drm_fourcc.h we don't require an in-kernel user for a new modifier
>>>>>> since a bunch of them might need to be allocated just for
>>>>>> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
>>>>>> for this would be compressed surfaces with fast-clearing, which is
>>>>>> planned for i915 (but current hw can't scan it out). And we really
>>>>>> want to have one namespace for everything.
>>>>>
>>>>> Who sets these modifiers? Kernel or userspace? Or can it be set by both?
>>>>> I assume any userspace code that sets/reads this is code specific for that
>>>>> hardware?
>>>>
>>>> I think normally the modifier would be set by userspace. However it
>>>> might not necessarily be device-specific code. In DRM the intention is
>>>> for userspace to query the set of modifiers which are supported, and
>>>> then use them without necessarily knowing exactly what they mean
>>>> (insofar as that is possible).
>>>>
>>>> e.g. if I have two devices which support MODIFIER_FOO, I could attempt
>>>> to share a buffer between them which uses MODIFIER_FOO without
>>>> necessarily knowing exactly what it is/does.
>>>>
>>>>>
>>>>> I think Laurent's suggestion of using a 64 bit V4L2 control for this makes
>>>>> the most sense.
>>>>>
>>>>> Especially if you can assume that whoever sets this knows the hardware.
>>>>>
>>>>> I think this only makes sense if you pass buffers from one HW device to another.
>>>>>
>>>>> Because you cannot expect generic video capture code to be able to interpret
>>>>> all the zillion different combinations of modifiers.
>>>>
>>>> I don't quite follow this last bit. The control could report the set
>>>> of supported modifiers.
>>>
>>> What I mean was: an application can use the modifier to give buffers from
>>> one device to another without needing to understand it.
>>>
>>> But a generic video capture application that processes the video itself
>>> cannot be expected to know about the modifiers. It's a custom HW specific
>>> format that you only use between two HW devices or with software written
>>> for that hardware.
>>>
>>
>> Yes, makes sense.
>>
>>>>
>>>> However, in DRM the API lets you get the supported formats for each
>>>> modifier as-well-as the modifier list itself. I'm not sure how exactly
>>>> to provide that in a control.
>>>
>>> We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU.
>>> You use VIDIOC_QUERYMENU to enumerate the available modifiers.
>>>
>>> So enumerating these modifiers would work out-of-the-box.
>>
>> Right. So I guess the supported set of formats could be somehow
>> enumerated in the menu item string. In DRM the pairs are (modifier +
>> bitmask) where bits represent formats in the supported formats list
>> (commit db1689aa61bd in drm-next). Printing a hex representation of
>> the bitmask would be functional but I concede not very pretty.
>
>So this patch limits the number of formats to 64 (being the size of
>the bit mask).

It's not limited to 64 formats. Right now no DRM drivers support more
than 64 formats, but when they do, the "offset" field in struct
drm_format_modifier can be set to 64 and then the bit mask represents
formats 65 through 128 (see the comment on that struct):

  * If the number formats grew to 128, and formats 98-102 are
  * supported with the modifier:
  *
  * 0x0000003c00000000 0000000000000000
  *                  ^
  *                  |__offset = 64, formats = 0x3c00000000

> I was hoping these modifiers applied to all formats,
>but unfortunately that isn't the case apparently.
>

Yeah, if only it were so simple :-)

>How it would work with my proposal is that the integer menu control
>would reflect the list of supported modifiers for the currently selected
>format. If you change format, then the available modifier list changes
>as well.

Ah yes, I need to get used to thinking about stateful APIs - that
works.

>The advantage is that there is no '64 formats' limitation,
>something that I feel very uncomfortable about since some devices support
>a *lot* of formats. The disadvantage is that it is harder to get a quick
>overview of all combinations for formats and modifiers.
>
>This has more to do with limitations in the V4L2 API than with supporting
>modifiers in general. We need something better to give userspace a quick
>overview of all combinations of pixelformats, framesizes, frameintervals
>and now modifiers. However, that's our problem :-)
>
>Regards,
>
>	Hans
>

Thanks for the inputs,

-Brian

>>
>> Cheers,
>> -Brian
>>
>>>
>>> Regards,
>>>
>>>     Hans
>> _______________________________________________
>> dri-devel mailing list
>> dri-devel@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
>

Re: DRM Format Modifiers in v4l2

[Brian Starkey]

On Mon, Aug 28, 2017 at 10:49:07PM +0200, Daniel Vetter wrote:
>On Mon, Aug 28, 2017 at 8:07 PM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>> Le jeudi 24 ao??t 2017 ?? 13:26 +0100, Brian Starkey a ??crit :
>>> > What I mean was: an application can use the modifier to give buffers from
>>> > one device to another without needing to understand it.
>>> >
>>> > But a generic video capture application that processes the video itself
>>> > cannot be expected to know about the modifiers. It's a custom HW specific
>>> > format that you only use between two HW devices or with software written
>>> > for that hardware.
>>> >
>>>
>>> Yes, makes sense.
>>>
>>> > >
>>> > > However, in DRM the API lets you get the supported formats for each
>>> > > modifier as-well-as the modifier list itself. I'm not sure how exactly
>>> > > to provide that in a control.
>>> >
>>> > We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU.
>>> > You use VIDIOC_QUERYMENU to enumerate the available modifiers.
>>> >
>>> > So enumerating these modifiers would work out-of-the-box.
>>>
>>> Right. So I guess the supported set of formats could be somehow
>>> enumerated in the menu item string. In DRM the pairs are (modifier +
>>> bitmask) where bits represent formats in the supported formats list
>>> (commit db1689aa61bd in drm-next). Printing a hex representation of
>>> the bitmask would be functional but I concede not very pretty.
>>
>> The problem is that the list of modifiers depends on the format
>> selected. Having to call S_FMT to obtain this list is quite
>> inefficient.
>>
>> Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been
>> implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think
>> an other one made it the same way recently, something from Mediatek if
>> I remember. Though, unlike the Intel one, the same modifier does not
>> have various result depending on the hardware revision.
>
>Note on the intel modifers: On most recent platforms (iirc gen9) the
>modifier is well defined and always describes the same byte layout. We
>simply didn't want to rewrite our entire software stack for all the
>old gunk platforms, hence the language. I guess we could/should
>describe the layout in detail, but atm we're the only ones using it.
>
>On your topic of v4l2 encoding the drm fourcc+modifier combo into a
>special v4l fourcc: That's exactly the mismatch I was thinking of.
>There's other examples of v4l2 fourcc being more specific than their
>drm counters (e.g. specific way the different planes are laid out).

I'm not entirely clear on the v4l2 fourccs being more specific than
DRM ones - do you mean e.g. NV12 vs NV12M? Specifically in the case of
multi-planar formats I think it's a non-issue because modifiers are
allowed to alter the number of planes and the meanings of them. Also
V4L2 NV12M is a superset of NV12 - so NV12M would always be able to
describe a DRM NV12 buffer.

I don't see the "special v4l2 format already exists" case as a problem
either. It would be up to any drivers that already have special
formats to decide if they want to also support it via a more generic
modifiers API or not.

The fact is, adding special formats for each combination is
unmanageable - we're talking dozens in the case of our hardware.

Cheers,
-Brian

>-Daniel
>-- 
>Daniel Vetter
>Software Engineer, Intel Corporation
>+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: DRM Format Modifiers in v4l2

[Daniel Vetter]

On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
> On Mon, Aug 28, 2017 at 10:49:07PM +0200, Daniel Vetter wrote:
> > On Mon, Aug 28, 2017 at 8:07 PM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> > > Le jeudi 24 ao??t 2017 ?? 13:26 +0100, Brian Starkey a ??crit :
> > > > > What I mean was: an application can use the modifier to give buffers from
> > > > > one device to another without needing to understand it.
> > > > >
> > > > > But a generic video capture application that processes the video itself
> > > > > cannot be expected to know about the modifiers. It's a custom HW specific
> > > > > format that you only use between two HW devices or with software written
> > > > > for that hardware.
> > > > >
> > > > 
> > > > Yes, makes sense.
> > > > 
> > > > > >
> > > > > > However, in DRM the API lets you get the supported formats for each
> > > > > > modifier as-well-as the modifier list itself. I'm not sure how exactly
> > > > > > to provide that in a control.
> > > > >
> > > > > We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU.
> > > > > You use VIDIOC_QUERYMENU to enumerate the available modifiers.
> > > > >
> > > > > So enumerating these modifiers would work out-of-the-box.
> > > > 
> > > > Right. So I guess the supported set of formats could be somehow
> > > > enumerated in the menu item string. In DRM the pairs are (modifier +
> > > > bitmask) where bits represent formats in the supported formats list
> > > > (commit db1689aa61bd in drm-next). Printing a hex representation of
> > > > the bitmask would be functional but I concede not very pretty.
> > > 
> > > The problem is that the list of modifiers depends on the format
> > > selected. Having to call S_FMT to obtain this list is quite
> > > inefficient.
> > > 
> > > Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been
> > > implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think
> > > an other one made it the same way recently, something from Mediatek if
> > > I remember. Though, unlike the Intel one, the same modifier does not
> > > have various result depending on the hardware revision.
> > 
> > Note on the intel modifers: On most recent platforms (iirc gen9) the
> > modifier is well defined and always describes the same byte layout. We
> > simply didn't want to rewrite our entire software stack for all the
> > old gunk platforms, hence the language. I guess we could/should
> > describe the layout in detail, but atm we're the only ones using it.
> > 
> > On your topic of v4l2 encoding the drm fourcc+modifier combo into a
> > special v4l fourcc: That's exactly the mismatch I was thinking of.
> > There's other examples of v4l2 fourcc being more specific than their
> > drm counters (e.g. specific way the different planes are laid out).
> 
> I'm not entirely clear on the v4l2 fourccs being more specific than
> DRM ones - do you mean e.g. NV12 vs NV12M? Specifically in the case of
> multi-planar formats I think it's a non-issue because modifiers are
> allowed to alter the number of planes and the meanings of them. Also
> V4L2 NV12M is a superset of NV12 - so NV12M would always be able to
> describe a DRM NV12 buffer.
> 
> I don't see the "special v4l2 format already exists" case as a problem
> either. It would be up to any drivers that already have special
> formats to decide if they want to also support it via a more generic
> modifiers API or not.
> 
> The fact is, adding special formats for each combination is
> unmanageable - we're talking dozens in the case of our hardware.

Hm right, we can just remap the special combos to the drm-fourcc +
modifier style. Bonus point if v4l does that in the core so not everyone
has to reinvent that wheel :-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: DRM Format Modifiers in v4l2

[Hans Verkuil]

On 30/08/17 09:50, Daniel Vetter wrote:
> On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
>> On Mon, Aug 28, 2017 at 10:49:07PM +0200, Daniel Vetter wrote:
>>> On Mon, Aug 28, 2017 at 8:07 PM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>>>> Le jeudi 24 ao??t 2017 ?? 13:26 +0100, Brian Starkey a ??crit :
>>>>>> What I mean was: an application can use the modifier to give buffers from
>>>>>> one device to another without needing to understand it.
>>>>>>
>>>>>> But a generic video capture application that processes the video itself
>>>>>> cannot be expected to know about the modifiers. It's a custom HW specific
>>>>>> format that you only use between two HW devices or with software written
>>>>>> for that hardware.
>>>>>>
>>>>>
>>>>> Yes, makes sense.
>>>>>
>>>>>>>
>>>>>>> However, in DRM the API lets you get the supported formats for each
>>>>>>> modifier as-well-as the modifier list itself. I'm not sure how exactly
>>>>>>> to provide that in a control.
>>>>>>
>>>>>> We have support for a 'menu' of 64 bit integers: V4L2_CTRL_TYPE_INTEGER_MENU.
>>>>>> You use VIDIOC_QUERYMENU to enumerate the available modifiers.
>>>>>>
>>>>>> So enumerating these modifiers would work out-of-the-box.
>>>>>
>>>>> Right. So I guess the supported set of formats could be somehow
>>>>> enumerated in the menu item string. In DRM the pairs are (modifier +
>>>>> bitmask) where bits represent formats in the supported formats list
>>>>> (commit db1689aa61bd in drm-next). Printing a hex representation of
>>>>> the bitmask would be functional but I concede not very pretty.
>>>>
>>>> The problem is that the list of modifiers depends on the format
>>>> selected. Having to call S_FMT to obtain this list is quite
>>>> inefficient.
>>>>
>>>> Also, be aware that DRM_FORMAT_MOD_SAMSUNG_64_32_TILE modifier has been
>>>> implemented in V4L2 with a direct format (V4L2_PIX_FMT_NV12MT). I think
>>>> an other one made it the same way recently, something from Mediatek if
>>>> I remember. Though, unlike the Intel one, the same modifier does not
>>>> have various result depending on the hardware revision.
>>>
>>> Note on the intel modifers: On most recent platforms (iirc gen9) the
>>> modifier is well defined and always describes the same byte layout. We
>>> simply didn't want to rewrite our entire software stack for all the
>>> old gunk platforms, hence the language. I guess we could/should
>>> describe the layout in detail, but atm we're the only ones using it.
>>>
>>> On your topic of v4l2 encoding the drm fourcc+modifier combo into a
>>> special v4l fourcc: That's exactly the mismatch I was thinking of.
>>> There's other examples of v4l2 fourcc being more specific than their
>>> drm counters (e.g. specific way the different planes are laid out).
>>
>> I'm not entirely clear on the v4l2 fourccs being more specific than
>> DRM ones - do you mean e.g. NV12 vs NV12M? Specifically in the case of
>> multi-planar formats I think it's a non-issue because modifiers are
>> allowed to alter the number of planes and the meanings of them. Also
>> V4L2 NV12M is a superset of NV12 - so NV12M would always be able to
>> describe a DRM NV12 buffer.
>>
>> I don't see the "special v4l2 format already exists" case as a problem
>> either. It would be up to any drivers that already have special
>> formats to decide if they want to also support it via a more generic
>> modifiers API or not.
>>
>> The fact is, adding special formats for each combination is
>> unmanageable - we're talking dozens in the case of our hardware.
> 
> Hm right, we can just remap the special combos to the drm-fourcc +
> modifier style. Bonus point if v4l does that in the core so not everyone
> has to reinvent that wheel :-)

Probably not something we'll do: there are I believe only two drivers that
are affected (exynos & mediatek), so they can do that in their driver.

Question: how many modifiers will typically apply to a format? I ask
because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link
between a fourcc and modifiers:

https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html

The __u32 reserved[4] array can be used to provide a bitmask to modifier
indices (for the integer menu control). It's similar to what drm does,
except instead of modifiers mapping to fourccs it is the other way around.

This would avoid having to change the modifiers control whenever a new
format is set and it makes it easy to enumerate all combinations.

But this only works if the total number of modifiers used by a single driver
is expected to remain small (let's say no more than 64).

Regards,

	Hans

Re: DRM Format Modifiers in v4l2

[Brian Starkey]

On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
>On 30/08/17 09:50, Daniel Vetter wrote:
>> On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
>>> The fact is, adding special formats for each combination is
>>> unmanageable - we're talking dozens in the case of our hardware.
>>
>> Hm right, we can just remap the special combos to the drm-fourcc +
>> modifier style. Bonus point if v4l does that in the core so not everyone
>> has to reinvent that wheel :-)
>
>Probably not something we'll do: there are I believe only two drivers that
>are affected (exynos & mediatek), so they can do that in their driver.
>
>Question: how many modifiers will typically apply to a format? I ask
>because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link
>between a fourcc and modifiers:
>
>https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
>
>The __u32 reserved[4] array can be used to provide a bitmask to modifier
>indices (for the integer menu control). It's similar to what drm does,
>except instead of modifiers mapping to fourccs it is the other way around.
>
>This would avoid having to change the modifiers control whenever a new
>format is set and it makes it easy to enumerate all combinations.
>
>But this only works if the total number of modifiers used by a single driver
>is expected to remain small (let's say no more than 64).

In our current (yet to be submitted) description, we've got around a
dozen modifiers for any one format to describe our compression
variants. We have a lot of on/off toggles which leads to combinatorial
expansion, so it can grow pretty quickly (though I am trying to limit
the valid combinations as much as possible).

How about if the mask fills up then VIDIOC_ENUM_FMT can return another
fmtdsc with the same FourCC and different modifier bitmask, where the
second one's modifier bitmask is for the next "N" modifiers?

-Brian
>
>Regards,
>
>	Hans

Re: DRM Format Modifiers in v4l2

[Hans Verkuil]

On 30/08/17 11:36, Brian Starkey wrote:
> On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
>> On 30/08/17 09:50, Daniel Vetter wrote:
>>> On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
>>>> The fact is, adding special formats for each combination is
>>>> unmanageable - we're talking dozens in the case of our hardware.
>>>
>>> Hm right, we can just remap the special combos to the drm-fourcc +
>>> modifier style. Bonus point if v4l does that in the core so not everyone
>>> has to reinvent that wheel :-)
>>
>> Probably not something we'll do: there are I believe only two drivers that
>> are affected (exynos & mediatek), so they can do that in their driver.
>>
>> Question: how many modifiers will typically apply to a format? I ask
>> because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link
>> between a fourcc and modifiers:
>>
>> https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
>>
>> The __u32 reserved[4] array can be used to provide a bitmask to modifier
>> indices (for the integer menu control). It's similar to what drm does,
>> except instead of modifiers mapping to fourccs it is the other way around.
>>
>> This would avoid having to change the modifiers control whenever a new
>> format is set and it makes it easy to enumerate all combinations.
>>
>> But this only works if the total number of modifiers used by a single driver
>> is expected to remain small (let's say no more than 64).
> 
> In our current (yet to be submitted) description, we've got around a
> dozen modifiers for any one format to describe our compression
> variants. We have a lot of on/off toggles which leads to combinatorial
> expansion, so it can grow pretty quickly (though I am trying to limit
> the valid combinations as much as possible).
> 
> How about if the mask fills up then VIDIOC_ENUM_FMT can return another
> fmtdsc with the same FourCC and different modifier bitmask, where the
> second one's modifier bitmask is for the next "N" modifiers?

I was thinking along similar lines, but it could cause some problems with
the ABI since applications currently assume that no fourcc will appear
twice when enumerating formats. Admittedly, we never explicitly said in
the spec that that can't happen, but it is kind of expected.

There are ways around that, but if possible I'd like to avoid that.

In theory there are up to 128 bits available but I can't help thinking
that if you create more than, say, 64 modifiers for a HW platform you
have a big mess anyway.

If I am wrong, then I need to know because then I can prepare for it
(or whoever is going to actually implement this...)

If the number of modifiers is expected to be limited then making 64 bits
available would be good enough, at least for now.

BTW, is a modifier always optional? I.e. for all fourccs, is the unmodified
format always available? Or are there fourccs that require the use of a
modifier?

Regards,

	Hans

Re: DRM Format Modifiers in v4l2

[Brian Starkey]

On Wed, Aug 30, 2017 at 11:53:58AM +0200, Hans Verkuil wrote:
>On 30/08/17 11:36, Brian Starkey wrote:
>> On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
>>> On 30/08/17 09:50, Daniel Vetter wrote:
>>>> On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
>>>>> The fact is, adding special formats for each combination is
>>>>> unmanageable - we're talking dozens in the case of our hardware.
>>>>
>>>> Hm right, we can just remap the special combos to the drm-fourcc +
>>>> modifier style. Bonus point if v4l does that in the core so not everyone
>>>> has to reinvent that wheel :-)
>>>
>>> Probably not something we'll do: there are I believe only two drivers that
>>> are affected (exynos & mediatek), so they can do that in their driver.
>>>
>>> Question: how many modifiers will typically apply to a format? I ask
>>> because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link
>>> between a fourcc and modifiers:
>>>
>>> https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
>>>
>>> The __u32 reserved[4] array can be used to provide a bitmask to modifier
>>> indices (for the integer menu control). It's similar to what drm does,
>>> except instead of modifiers mapping to fourccs it is the other way around.
>>>
>>> This would avoid having to change the modifiers control whenever a new
>>> format is set and it makes it easy to enumerate all combinations.
>>>
>>> But this only works if the total number of modifiers used by a single driver
>>> is expected to remain small (let's say no more than 64).
>>
>> In our current (yet to be submitted) description, we've got around a
>> dozen modifiers for any one format to describe our compression
>> variants. We have a lot of on/off toggles which leads to combinatorial
>> expansion, so it can grow pretty quickly (though I am trying to limit
>> the valid combinations as much as possible).
>>
>> How about if the mask fills up then VIDIOC_ENUM_FMT can return another
>> fmtdsc with the same FourCC and different modifier bitmask, where the
>> second one's modifier bitmask is for the next "N" modifiers?
>
>I was thinking along similar lines, but it could cause some problems with
>the ABI since applications currently assume that no fourcc will appear
>twice when enumerating formats. Admittedly, we never explicitly said in
>the spec that that can't happen, but it is kind of expected.
>
>There are ways around that, but if possible I'd like to avoid that.
>
>In theory there are up to 128 bits available but I can't help thinking
>that if you create more than, say, 64 modifiers for a HW platform you
>have a big mess anyway.
>
>If I am wrong, then I need to know because then I can prepare for it
>(or whoever is going to actually implement this...)

You're probably right, but I can't speak for everyone. From the
current state of drm_fourcc.h it looks like 64 would be plenty (there
aren't anywhere near 64 modifiers even defined right now). Adding in
the Arm compression formats will expand it a lot, but still not to 64
(yet).

>
>If the number of modifiers is expected to be limited then making 64 bits
>available would be good enough, at least for now.
>
>BTW, is a modifier always optional? I.e. for all fourccs, is the unmodified
>format always available? Or are there fourccs that require the use of a
>modifier?

We do actually have one or two formats which are only supported with a
modifier (on our HW).

-Brian

>
>Regards,
>
>	Hans

Re: DRM Format Modifiers in v4l2

[Laurent Pinchart]

Hi Brian,

On Thursday, 24 August 2017 14:14:31 EEST Brian Starkey wrote:
> On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
> > On 08/21/2017 06:01 PM, Daniel Vetter wrote:
> >> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey wrote:
> >>> Hi all,
> >>> 
> >>> I couldn't find this topic talked about elsewhere, but apologies if
> >>> it's a duplicate - I'll be glad to be steered in the direction of a
> >>> thread.
> >>> 
> >>> We'd like to support DRM format modifiers in v4l2 in order to share
> >>> the description of different (mostly proprietary) buffer formats
> >>> between e.g. a v4l2 device and a DRM device.
> >>> 
> >>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and
> >>> are a vendor-namespaced 64-bit value used to describe various
> >>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC
> >>> code to give a complete description of the data contained in a buffer.
> >>> 
> >>> The same modifier definition is used in the Khronos EGL extension
> >>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
> >>> Wayland linux-dmabuf protocol.
> >>> 
> >>> 
> >>> This buffer information could of course be described in the
> >>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
> >>> information already defined in drm_fourcc.h. Additionally, there
> >>> would be quite a format explosion where a device supports a dozen or
> >>> more formats, all of which can use one or more different
> >>> layouts/compression schemes.
> >>> 
> >>> So, I'm wondering if anyone has views on how/whether this could be
> >>> incorporated?
> >>> 
> >>> I spoke briefly about this to Laurent at LPC last year, and he
> >>> suggested v4l2_control as one approach.
> >>> 
> >>> I also wondered if could be added in v4l2_pix_format_mplane - looks
> >>> like there's 8 bytes left before it exceeds the 200 bytes, or could go
> >>> in the reserved portion of v4l2_plane_pix_format.

We're considering reworking the format ioctls at some point. We don't 
necessarily need to wait until then to implement support for modifiers, but we 
will have an opportunity to integrate them with formats at that point.

> >>> Thanks for any thoughts,
> >> 
> >> One problem is that the modifers sometimes reference the DRM fourcc
> >> codes. v4l has a different (and incompatible set) of fourcc codes,
> >> whereas all the protocols and specs (you can add DRI3.1 for Xorg to
> >> that list btw) use both drm fourcc and drm modifiers.
> >> 
> >> This might or might not make this proposal unworkable, but it's
> >> something I'd at least review carefully.
> >> 
> >> Otherwise I think it'd be great if we could have one namespace for all
> >> modifiers, that's pretty much why we have them. Please also note that
> >> for drm_fourcc.h we don't require an in-kernel user for a new modifier
> >> since a bunch of them might need to be allocated just for
> >> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
> >> for this would be compressed surfaces with fast-clearing, which is
> >> planned for i915 (but current hw can't scan it out). And we really
> >> want to have one namespace for everything.
> >
> > Who sets these modifiers? Kernel or userspace? Or can it be set by both?
> > I assume any userspace code that sets/reads this is code specific for that
> > hardware?
> 
> I think normally the modifier would be set by userspace. However it
> might not necessarily be device-specific code. In DRM the intention is
> for userspace to query the set of modifiers which are supported, and
> then use them without necessarily knowing exactly what they mean
> (insofar as that is possible).
> 
> e.g. if I have two devices which support MODIFIER_FOO, I could attempt
> to share a buffer between them which uses MODIFIER_FOO without
> necessarily knowing exactly what it is/does.

Userspace could certainly set modifiers blindly, but the point of modifiers is 
to generate side effects benefitial to the use case at hand (for instance by 
optimizing the memory access pattern). To use them meaningfully userspace 
would need to have at least an idea of the side effects they generate.

> > I think Laurent's suggestion of using a 64 bit V4L2 control for this makes
> > the most sense.
> >
> > Especially if you can assume that whoever sets this knows the hardware.
> >
> > I think this only makes sense if you pass buffers from one HW device to
> > another.
> >
> > Because you cannot expect generic video capture code to be able to
> > interpret all the zillion different combinations of modifiers.
> 
> I don't quite follow this last bit. The control could report the set
> of supported modifiers.
> 
> However, in DRM the API lets you get the supported formats for each
> modifier as-well-as the modifier list itself. I'm not sure how exactly
> to provide that in a control.

-- 
Regards,

Laurent Pinchart

Re: DRM Format Modifiers in v4l2

[Laurent Pinchart]

Hi Brian,

On Tuesday, 29 August 2017 12:19:43 EEST Brian Starkey wrote:
> On Fri, Aug 25, 2017 at 10:14:03AM +0200, Hans Verkuil wrote:
> >On 24/08/17 14:26, Brian Starkey wrote:
> >> On Thu, Aug 24, 2017 at 01:37:35PM +0200, Hans Verkuil wrote:
> >>> On 08/24/17 13:14, Brian Starkey wrote:
> >>>> On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote:
> >>>>> On 08/21/2017 06:01 PM, Daniel Vetter wrote:
> >>>>>> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey wrote:
> >>>>>>> Hi all,
> >>>>>>> 
> >>>>>>> I couldn't find this topic talked about elsewhere, but apologies if
> >>>>>>> it's a duplicate - I'll be glad to be steered in the direction of a
> >>>>>>> thread.
> >>>>>>> 
> >>>>>>> We'd like to support DRM format modifiers in v4l2 in order to share
> >>>>>>> the description of different (mostly proprietary) buffer formats
> >>>>>>> between e.g. a v4l2 device and a DRM device.
> >>>>>>> 
> >>>>>>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h
> >>>>>>> and are a vendor-namespaced 64-bit value used to describe various
> >>>>>>> vendor-specific buffer layouts. They are combined with a (DRM)
> >>>>>>> FourCC code to give a complete description of the data contained in
> >>>>>>> a buffer.
> >>>>>>> 
> >>>>>>> The same modifier definition is used in the Khronos EGL extension
> >>>>>>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the
> >>>>>>> Wayland linux-dmabuf protocol.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> This buffer information could of course be described in the
> >>>>>>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the
> >>>>>>> information already defined in drm_fourcc.h. Additionally, there
> >>>>>>> would be quite a format explosion where a device supports a dozen or
> >>>>>>> more formats, all of which can use one or more different
> >>>>>>> layouts/compression schemes.
> >>>>>>> 
> >>>>>>> So, I'm wondering if anyone has views on how/whether this could be
> >>>>>>> incorporated?
> >>>>>>> 
> >>>>>>> I spoke briefly about this to Laurent at LPC last year, and he
> >>>>>>> suggested v4l2_control as one approach.
> >>>>>>> 
> >>>>>>> I also wondered if could be added in v4l2_pix_format_mplane - looks
> >>>>>>> like there's 8 bytes left before it exceeds the 200 bytes, or could
> >>>>>>> go in the reserved portion of v4l2_plane_pix_format.
> >>>>>>> 
> >>>>>>> Thanks for any thoughts,
> >>>>>> 
> >>>>>> One problem is that the modifers sometimes reference the DRM fourcc
> >>>>>> codes. v4l has a different (and incompatible set) of fourcc codes,
> >>>>>> whereas all the protocols and specs (you can add DRI3.1 for Xorg to
> >>>>>> that list btw) use both drm fourcc and drm modifiers.
> >>>>>> 
> >>>>>> This might or might not make this proposal unworkable, but it's
> >>>>>> something I'd at least review carefully.
> >>>>>> 
> >>>>>> Otherwise I think it'd be great if we could have one namespace for
> >>>>>> all modifiers, that's pretty much why we have them. Please also note
> >>>>>> that for drm_fourcc.h we don't require an in-kernel user for a new
> >>>>>> modifier since a bunch of them might need to be allocated just for
> >>>>>> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example
> >>>>>> for this would be compressed surfaces with fast-clearing, which is
> >>>>>> planned for i915 (but current hw can't scan it out). And we really
> >>>>>> want to have one namespace for everything.
> >>>>> 
> >>>>> Who sets these modifiers? Kernel or userspace? Or can it be set by
> >>>>> both? I assume any userspace code that sets/reads this is code
> >>>>> specific for that hardware?
> >>>> 
> >>>> I think normally the modifier would be set by userspace. However it
> >>>> might not necessarily be device-specific code. In DRM the intention is
> >>>> for userspace to query the set of modifiers which are supported, and
> >>>> then use them without necessarily knowing exactly what they mean
> >>>> (insofar as that is possible).
> >>>> 
> >>>> e.g. if I have two devices which support MODIFIER_FOO, I could attempt
> >>>> to share a buffer between them which uses MODIFIER_FOO without
> >>>> necessarily knowing exactly what it is/does.
> >>>> 
> >>>>> I think Laurent's suggestion of using a 64 bit V4L2 control for this
> >>>>> makes the most sense.
> >>>>> 
> >>>>> Especially if you can assume that whoever sets this knows the
> >>>>> hardware.
> >>>>> 
> >>>>> I think this only makes sense if you pass buffers from one HW device
> >>>>> to another.
> >>>>> 
> >>>>> Because you cannot expect generic video capture code to be able to
> >>>>> interpret all the zillion different combinations of modifiers.
> >>>> 
> >>>> I don't quite follow this last bit. The control could report the set
> >>>> of supported modifiers.
> >>> 
> >>> What I mean was: an application can use the modifier to give buffers
> >>> from one device to another without needing to understand it.
> >>> 
> >>> But a generic video capture application that processes the video itself
> >>> cannot be expected to know about the modifiers. It's a custom HW
> >>> specific format that you only use between two HW devices or with
> >>> software written for that hardware.
> >> 
> >> Yes, makes sense.
> >> 
> >>>> However, in DRM the API lets you get the supported formats for each
> >>>> modifier as-well-as the modifier list itself. I'm not sure how exactly
> >>>> to provide that in a control.
> >>> 
> >>> We have support for a 'menu' of 64 bit integers:
> >>> V4L2_CTRL_TYPE_INTEGER_MENU. You use VIDIOC_QUERYMENU to enumerate the
> >>> available modifiers.
> >>> 
> >>> So enumerating these modifiers would work out-of-the-box.
> >> 
> >> Right. So I guess the supported set of formats could be somehow
> >> enumerated in the menu item string. In DRM the pairs are (modifier +
> >> bitmask) where bits represent formats in the supported formats list
> >> (commit db1689aa61bd in drm-next). Printing a hex representation of
> >> the bitmask would be functional but I concede not very pretty.
> >
> > So this patch limits the number of formats to 64 (being the size of
> > the bit mask).
> 
> It's not limited to 64 formats. Right now no DRM drivers support more
> than 64 formats, but when they do, the "offset" field in struct
> drm_format_modifier can be set to 64 and then the bit mask represents
> formats 65 through 128 (see the comment on that struct):
> 
>   * If the number formats grew to 128, and formats 98-102 are
>   * supported with the modifier:
>   *
>   * 0x0000003c00000000 0000000000000000
>   *                  ^
>   *                  |__offset = 64, formats = 0x3c00000000
> 
> > I was hoping these modifiers applied to all formats,
> > but unfortunately that isn't the case apparently.
> 
> Yeah, if only it were so simple :-)
> 
> > How it would work with my proposal is that the integer menu control
> > would reflect the list of supported modifiers for the currently selected
> > format. If you change format, then the available modifier list changes
> > as well.
> 
> Ah yes, I need to get used to thinking about stateful APIs - that
> works.

No, you don't, we need to make enumeration stateless :-)

> > The advantage is that there is no '64 formats' limitation,
> > something that I feel very uncomfortable about since some devices support
> > a *lot* of formats. The disadvantage is that it is harder to get a quick
> > overview of all combinations for formats and modifiers.
> >
> > This has more to do with limitations in the V4L2 API than with supporting
> > modifiers in general. We need something better to give userspace a quick
> > overview of all combinations of pixelformats, framesizes, frameintervals
> > and now modifiers. However, that's our problem :-)

-- 
Regards,

Laurent Pinchart

Re: DRM Format Modifiers in v4l2

[Laurent Pinchart]

Hi Brian,

On Wednesday, 30 August 2017 13:32:01 EEST Brian Starkey wrote:
> On Wed, Aug 30, 2017 at 11:53:58AM +0200, Hans Verkuil wrote:
> > On 30/08/17 11:36, Brian Starkey wrote:
> >> On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
> >>> On 30/08/17 09:50, Daniel Vetter wrote:
> >>>> On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
> >>>>> The fact is, adding special formats for each combination is
> >>>>> unmanageable - we're talking dozens in the case of our hardware.
> >>>> 
> >>>> Hm right, we can just remap the special combos to the drm-fourcc +
> >>>> modifier style. Bonus point if v4l does that in the core so not
> >>>> everyone has to reinvent that wheel :-)
> >>> 
> >>> Probably not something we'll do: there are I believe only two drivers
> >>> that are affected (exynos & mediatek), so they can do that in their
> >>> driver.
> >>> 
> >>> Question: how many modifiers will typically apply to a format? I ask
> >>> because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link
> >>> between a fourcc and modifiers:
> >>> 
> >>> https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
> >>> 
> >>> The __u32 reserved[4] array can be used to provide a bitmask to modifier
> >>> indices (for the integer menu control). It's similar to what drm does,
> >>> except instead of modifiers mapping to fourccs it is the other way
> >>> around.
> >>> 
> >>> This would avoid having to change the modifiers control whenever a new
> >>> format is set and it makes it easy to enumerate all combinations.
> >>> 
> >>> But this only works if the total number of modifiers used by a single
> >>> driver is expected to remain small (let's say no more than 64).
> >> 
> >> In our current (yet to be submitted) description, we've got around a
> >> dozen modifiers for any one format to describe our compression
> >> variants. We have a lot of on/off toggles which leads to combinatorial
> >> expansion, so it can grow pretty quickly (though I am trying to limit
> >> the valid combinations as much as possible).
> >> 
> >> How about if the mask fills up then VIDIOC_ENUM_FMT can return another
> >> fmtdsc with the same FourCC and different modifier bitmask, where the
> >> second one's modifier bitmask is for the next "N" modifiers?
> >
> > I was thinking along similar lines, but it could cause some problems with
> > the ABI since applications currently assume that no fourcc will appear
> > twice when enumerating formats. Admittedly, we never explicitly said in
> > the spec that that can't happen, but it is kind of expected.
> >
> > There are ways around that, but if possible I'd like to avoid that.
> >
> > In theory there are up to 128 bits available but I can't help thinking
> > that if you create more than, say, 64 modifiers for a HW platform you
> > have a big mess anyway.
> >
> > If I am wrong, then I need to know because then I can prepare for it
> > (or whoever is going to actually implement this...)
> 
> You're probably right, but I can't speak for everyone. From the
> current state of drm_fourcc.h it looks like 64 would be plenty (there
> aren't anywhere near 64 modifiers even defined right now). Adding in
> the Arm compression formats will expand it a lot, but still not to 64
> (yet).

Do all those modifiers make sense on the V4L2 side ? I expect that some 
modifiers will mostly be used for buffers shared between the GPU and the 
display engine, while others will be used by codecs. The sets will likely 
overlap, but might not be identical.

> > If the number of modifiers is expected to be limited then making 64 bits
> > available would be good enough, at least for now.
> >
> > BTW, is a modifier always optional? I.e. for all fourccs, is the
> > unmodified format always available? Or are there fourccs that require the
> > use of a modifier?
> 
> We do actually have one or two formats which are only supported with a
> modifier (on our HW).

-- 
Regards,

Laurent Pinchart

Re: DRM Format Modifiers in v4l2

[Brian Starkey]

Hi Laurent,

On Thu, Aug 31, 2017 at 05:51:33PM +0300, Laurent Pinchart wrote:
>Hi Brian,
>
>On Wednesday, 30 August 2017 13:32:01 EEST Brian Starkey wrote:
>> On Wed, Aug 30, 2017 at 11:53:58AM +0200, Hans Verkuil wrote:
>> > On 30/08/17 11:36, Brian Starkey wrote:
>> >> On Wed, Aug 30, 2017 at 10:10:01AM +0200, Hans Verkuil wrote:
>> >>> On 30/08/17 09:50, Daniel Vetter wrote:
>> >>>> On Tue, Aug 29, 2017 at 10:47:01AM +0100, Brian Starkey wrote:
>> >>>>> The fact is, adding special formats for each combination is
>> >>>>> unmanageable - we're talking dozens in the case of our hardware.
>> >>>>
>> >>>> Hm right, we can just remap the special combos to the drm-fourcc +
>> >>>> modifier style. Bonus point if v4l does that in the core so not
>> >>>> everyone has to reinvent that wheel :-)
>> >>>
>> >>> Probably not something we'll do: there are I believe only two drivers
>> >>> that are affected (exynos & mediatek), so they can do that in their
>> >>> driver.
>> >>>
>> >>> Question: how many modifiers will typically apply to a format? I ask
>> >>> because I realized that V4L2 could use VIDIOC_ENUMFMT to make the link
>> >>> between a fourcc and modifiers:
>> >>>
>> >>> https://hverkuil.home.xs4all.nl/spec/uapi/v4l/vidioc-enum-fmt.html
>> >>>
>> >>> The __u32 reserved[4] array can be used to provide a bitmask to modifier
>> >>> indices (for the integer menu control). It's similar to what drm does,
>> >>> except instead of modifiers mapping to fourccs it is the other way
>> >>> around.
>> >>>
>> >>> This would avoid having to change the modifiers control whenever a new
>> >>> format is set and it makes it easy to enumerate all combinations.
>> >>>
>> >>> But this only works if the total number of modifiers used by a single
>> >>> driver is expected to remain small (let's say no more than 64).
>> >>
>> >> In our current (yet to be submitted) description, we've got around a
>> >> dozen modifiers for any one format to describe our compression
>> >> variants. We have a lot of on/off toggles which leads to combinatorial
>> >> expansion, so it can grow pretty quickly (though I am trying to limit
>> >> the valid combinations as much as possible).
>> >>
>> >> How about if the mask fills up then VIDIOC_ENUM_FMT can return another
>> >> fmtdsc with the same FourCC and different modifier bitmask, where the
>> >> second one's modifier bitmask is for the next "N" modifiers?
>> >
>> > I was thinking along similar lines, but it could cause some problems with
>> > the ABI since applications currently assume that no fourcc will appear
>> > twice when enumerating formats. Admittedly, we never explicitly said in
>> > the spec that that can't happen, but it is kind of expected.
>> >
>> > There are ways around that, but if possible I'd like to avoid that.
>> >
>> > In theory there are up to 128 bits available but I can't help thinking
>> > that if you create more than, say, 64 modifiers for a HW platform you
>> > have a big mess anyway.
>> >
>> > If I am wrong, then I need to know because then I can prepare for it
>> > (or whoever is going to actually implement this...)
>>
>> You're probably right, but I can't speak for everyone. From the
>> current state of drm_fourcc.h it looks like 64 would be plenty (there
>> aren't anywhere near 64 modifiers even defined right now). Adding in
>> the Arm compression formats will expand it a lot, but still not to 64
>> (yet).
>
>Do all those modifiers make sense on the V4L2 side ? I expect that some
>modifiers will mostly be used for buffers shared between the GPU and the
>display engine, while others will be used by codecs. The sets will likely
>overlap, but might not be identical.
>

All of the ones from drm_fourcc.h - I expect not. In the case of Arm's
framebuffer compression though, it's used in all our media IPs;
Display, GPU and Video, and ideally on all data exchanged between
them. For the most part the modifiers describing it could apply to
all three.

There's also the question of what you're calling a codec - mem2mem
rotation blocks/scalers/etc. exposed via V4L2 would likely want the
same set as any display device which consumes their output (which I
think puts them firmly in the shared between the XXX and display
engine camp).

Cheers,
-Brian

>> > If the number of modifiers is expected to be limited then making 64 bits
>> > available would be good enough, at least for now.
>> >
>> > BTW, is a modifier always optional? I.e. for all fourccs, is the
>> > unmodified format always available? Or are there fourccs that require the
>> > use of a modifier?
>>
>> We do actually have one or two formats which are only supported with a
>> modifier (on our HW).
>
>-- 
>Regards,
>
>Laurent Pinchart
>

Re: DRM Format Modifiers in v4l2

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

Le jeudi 31 août 2017 à 17:28 +0300, Laurent Pinchart a écrit :
> > e.g. if I have two devices which support MODIFIER_FOO, I could attempt
> > to share a buffer between them which uses MODIFIER_FOO without
> > necessarily knowing exactly what it is/does.
> 
> Userspace could certainly set modifiers blindly, but the point of modifiers is 
> to generate side effects benefitial to the use case at hand (for instance by 
> optimizing the memory access pattern). To use them meaningfully userspace 
> would need to have at least an idea of the side effects they generate.

Generic userspace will basically pick some random combination. To allow
generically picking the optimal configuration we could indeed rely on
the application knowledge, but we could also enhance the spec so that
the order in the enumeration becomes meaningful.

regards,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: DRM Format Modifiers in v4l2

[Laurent Pinchart]

Hi Nicolas,

On Thursday, 31 August 2017 19:12:58 EEST Nicolas Dufresne wrote:
> Le jeudi 31 août 2017 à 17:28 +0300, Laurent Pinchart a écrit :
> >> e.g. if I have two devices which support MODIFIER_FOO, I could attempt
> >> to share a buffer between them which uses MODIFIER_FOO without
> >> necessarily knowing exactly what it is/does.
> > 
> > Userspace could certainly set modifiers blindly, but the point of
> > modifiers is to generate side effects benefitial to the use case at hand
> > (for instance by optimizing the memory access pattern). To use them
> > meaningfully userspace would need to have at least an idea of the side
> > effects they generate.
> 
> Generic userspace will basically pick some random combination.

In that case userspace could set no modifier at all by default (except in the 
case where unmodified formats are not supported by the hardware, but I don't 
expect that to be the most common case).

> To allow generically picking the optimal configuration we could indeed rely
> on the application knowledge, but we could also enhance the spec so that
> the order in the enumeration becomes meaningful.

I'm not sure how far we should go. I could imagine a system where the API 
would report capabilities for modifiers (e.g. this modifier lowers the 
bandwidth, this one enhances the quality, ...), but going in that direction, 
where do we stop ? In practice I expect userspace to know some information 
about the hardware, so I'd rather avoid over-engineering the API.

-- 
Regards,

Laurent Pinchart

Re: DRM Format Modifiers in v4l2

[Rob Clark]

On Fri, Sep 1, 2017 at 3:13 AM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> Hi Nicolas,
>
> On Thursday, 31 August 2017 19:12:58 EEST Nicolas Dufresne wrote:
>> Le jeudi 31 août 2017 à 17:28 +0300, Laurent Pinchart a écrit :
>> >> e.g. if I have two devices which support MODIFIER_FOO, I could attempt
>> >> to share a buffer between them which uses MODIFIER_FOO without
>> >> necessarily knowing exactly what it is/does.
>> >
>> > Userspace could certainly set modifiers blindly, but the point of
>> > modifiers is to generate side effects benefitial to the use case at hand
>> > (for instance by optimizing the memory access pattern). To use them
>> > meaningfully userspace would need to have at least an idea of the side
>> > effects they generate.
>>
>> Generic userspace will basically pick some random combination.
>
> In that case userspace could set no modifier at all by default (except in the
> case where unmodified formats are not supported by the hardware, but I don't
> expect that to be the most common case).
>
>> To allow generically picking the optimal configuration we could indeed rely
>> on the application knowledge, but we could also enhance the spec so that
>> the order in the enumeration becomes meaningful.
>
> I'm not sure how far we should go. I could imagine a system where the API
> would report capabilities for modifiers (e.g. this modifier lowers the
> bandwidth, this one enhances the quality, ...), but going in that direction,
> where do we stop ? In practice I expect userspace to know some information
> about the hardware, so I'd rather avoid over-engineering the API.
>

I think in the (hopefully not too) long term, something like
https://github.com/cubanismo/allocator/ is the way forward.  That
doesn't quite solve how v4l2 kernel part sorts out w/ corresponding
userspace .so what is preferable, but at least that is
compartmentalized to v4l2.. on the gl/vk side of things there will ofc
be a hardware specific userspace part that knows what it prefers.  For
v4l2, it probably makes sense to sort out what the userspace level API
is and work backwards from there, rather than risk trying to design a
kernel uapi that might turn out to be the wrong thing.

BR,
-R

Re: DRM Format Modifiers in v4l2

[Daniel Vetter]

On Fri, Sep 1, 2017 at 2:43 PM, Rob Clark <robdclark@gmail.com> wrote:
> On Fri, Sep 1, 2017 at 3:13 AM, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
>> Hi Nicolas,
>>
>> On Thursday, 31 August 2017 19:12:58 EEST Nicolas Dufresne wrote:
>>> Le jeudi 31 août 2017 à 17:28 +0300, Laurent Pinchart a écrit :
>>> >> e.g. if I have two devices which support MODIFIER_FOO, I could attempt
>>> >> to share a buffer between them which uses MODIFIER_FOO without
>>> >> necessarily knowing exactly what it is/does.
>>> >
>>> > Userspace could certainly set modifiers blindly, but the point of
>>> > modifiers is to generate side effects benefitial to the use case at hand
>>> > (for instance by optimizing the memory access pattern). To use them
>>> > meaningfully userspace would need to have at least an idea of the side
>>> > effects they generate.
>>>
>>> Generic userspace will basically pick some random combination.
>>
>> In that case userspace could set no modifier at all by default (except in the
>> case where unmodified formats are not supported by the hardware, but I don't
>> expect that to be the most common case).
>>
>>> To allow generically picking the optimal configuration we could indeed rely
>>> on the application knowledge, but we could also enhance the spec so that
>>> the order in the enumeration becomes meaningful.
>>
>> I'm not sure how far we should go. I could imagine a system where the API
>> would report capabilities for modifiers (e.g. this modifier lowers the
>> bandwidth, this one enhances the quality, ...), but going in that direction,
>> where do we stop ? In practice I expect userspace to know some information
>> about the hardware, so I'd rather avoid over-engineering the API.
>>
>
> I think in the (hopefully not too) long term, something like
> https://github.com/cubanismo/allocator/ is the way forward.  That
> doesn't quite solve how v4l2 kernel part sorts out w/ corresponding
> userspace .so what is preferable, but at least that is
> compartmentalized to v4l2.. on the gl/vk side of things there will ofc
> be a hardware specific userspace part that knows what it prefers.  For
> v4l2, it probably makes sense to sort out what the userspace level API
> is and work backwards from there, rather than risk trying to design a
> kernel uapi that might turn out to be the wrong thing.

I thought for kms the plan is to make the ordering meaningful, because
it doesn't necessarily match the gl/vk one. E.g. on intel gl would
prefer Y compressed, Y, X, untiled. Whereas display would be Y
compressed, X (much easier to scan out, in many cases allows more
planes to be used), Y (is necessary for 90° rotation), untiled. So if
drm_hwc really wants to use all the planes, it could prioritize the
display over rendering and request X instead of Y tiled.

I think the same would go for v4l.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch