[PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Ming Qian]

By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
user-space should be able to hint decoder
the vb2 only contains codec config header,
but does not contain any frame data.
It's only used for parsing header, and can't be decoded.

Current, it's usually used by android.

Signed-off-by: Ming Qian <ming.qian@nxp.com>
---
 Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
 include/uapi/linux/videodev2.h                   | 2 ++
 2 files changed, 11 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
index 4638ec64db00..acdc4556f4f4 100644
--- a/Documentation/userspace-api/media/v4l/buffer.rst
+++ b/Documentation/userspace-api/media/v4l/buffer.rst
@@ -607,6 +607,15 @@ Buffer Flags
 	the format. Any subsequent call to the
 	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
 	but return an ``EPIPE`` error code.
+    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
+
+      - ``V4L2_BUF_FLAG_CODECCONFIG``
+      - 0x00200000
+      - This flag may be set when the buffer only contains codec config
+    header, but does not contain any frame data. Usually the codec config
+    header is merged to the next idr frame, with the flag
+    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
+    split the header and queue it separately.
     * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
 
       - ``V4L2_BUF_FLAG_REQUEST_FD``
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 5311ac4fde35..8708ef257710 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
 /* mem2mem encoder/decoder */
 #define V4L2_BUF_FLAG_LAST			0x00100000
+/* Buffer only contains codec header */
+#define V4L2_BUF_FLAG_CODECCONFIG		0x00200000
 /* request_fd is valid */
 #define V4L2_BUF_FLAG_REQUEST_FD		0x00800000
 
-- 
2.36.1

[PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Ming Qian]

By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
user-space should be able to hint decoder
the vb2 only contains codec config header,
but does not contain any frame data.
It's only used for parsing header, and can't be decoded.

Current, it's usually used by android.

Signed-off-by: Ming Qian <ming.qian@nxp.com>
---
 Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
 include/uapi/linux/videodev2.h                   | 2 ++
 2 files changed, 11 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
index 4638ec64db00..acdc4556f4f4 100644
--- a/Documentation/userspace-api/media/v4l/buffer.rst
+++ b/Documentation/userspace-api/media/v4l/buffer.rst
@@ -607,6 +607,15 @@ Buffer Flags
 	the format. Any subsequent call to the
 	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
 	but return an ``EPIPE`` error code.
+    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
+
+      - ``V4L2_BUF_FLAG_CODECCONFIG``
+      - 0x00200000
+      - This flag may be set when the buffer only contains codec config
+    header, but does not contain any frame data. Usually the codec config
+    header is merged to the next idr frame, with the flag
+    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
+    split the header and queue it separately.
     * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
 
       - ``V4L2_BUF_FLAG_REQUEST_FD``
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 5311ac4fde35..8708ef257710 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
 /* mem2mem encoder/decoder */
 #define V4L2_BUF_FLAG_LAST			0x00100000
+/* Buffer only contains codec header */
+#define V4L2_BUF_FLAG_CODECCONFIG		0x00200000
 /* request_fd is valid */
 #define V4L2_BUF_FLAG_REQUEST_FD		0x00800000
 
-- 
2.36.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Nicolas Dufresne]

Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> user-space should be able to hint decoder
> the vb2 only contains codec config header,
> but does not contain any frame data.
> It's only used for parsing header, and can't be decoded.

This is copied from OMX specification. I think we we import this, we should at
least refer to the original.

> 
> Current, it's usually used by android.
> 
> Signed-off-by: Ming Qian <ming.qian@nxp.com>
> ---
>  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
>  include/uapi/linux/videodev2.h                   | 2 ++
>  2 files changed, 11 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> index 4638ec64db00..acdc4556f4f4 100644
> --- a/Documentation/userspace-api/media/v4l/buffer.rst
> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> @@ -607,6 +607,15 @@ Buffer Flags
>  	the format. Any subsequent call to the
>  	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
>  	but return an ``EPIPE`` error code.
> +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> +
> +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> +      - 0x00200000
> +      - This flag may be set when the buffer only contains codec config
> +    header, but does not contain any frame data. Usually the codec config
> +    header is merged to the next idr frame, with the flag
> +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> +    split the header and queue it separately.

I think the documentation is clear. Now, if a driver uses this, will existing
userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
So we need existing driver to do this when flagged to, and just copy/append when
the userland didn't opt-in that feature ?

>      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>  
>        - ``V4L2_BUF_FLAG_REQUEST_FD``
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 5311ac4fde35..8708ef257710 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
>  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
>  /* mem2mem encoder/decoder */
>  #define V4L2_BUF_FLAG_LAST			0x00100000
> +/* Buffer only contains codec header */
> +#define V4L2_BUF_FLAG_CODECCONFIG		0x00200000
>  /* request_fd is valid */
>  #define V4L2_BUF_FLAG_REQUEST_FD		0x00800000
>  

Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Nicolas Dufresne]

Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> user-space should be able to hint decoder
> the vb2 only contains codec config header,
> but does not contain any frame data.
> It's only used for parsing header, and can't be decoded.

This is copied from OMX specification. I think we we import this, we should at
least refer to the original.

> 
> Current, it's usually used by android.
> 
> Signed-off-by: Ming Qian <ming.qian@nxp.com>
> ---
>  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
>  include/uapi/linux/videodev2.h                   | 2 ++
>  2 files changed, 11 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> index 4638ec64db00..acdc4556f4f4 100644
> --- a/Documentation/userspace-api/media/v4l/buffer.rst
> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> @@ -607,6 +607,15 @@ Buffer Flags
>  	the format. Any subsequent call to the
>  	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
>  	but return an ``EPIPE`` error code.
> +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> +
> +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> +      - 0x00200000
> +      - This flag may be set when the buffer only contains codec config
> +    header, but does not contain any frame data. Usually the codec config
> +    header is merged to the next idr frame, with the flag
> +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> +    split the header and queue it separately.

I think the documentation is clear. Now, if a driver uses this, will existing
userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
So we need existing driver to do this when flagged to, and just copy/append when
the userland didn't opt-in that feature ?

>      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>  
>        - ``V4L2_BUF_FLAG_REQUEST_FD``
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 5311ac4fde35..8708ef257710 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
>  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
>  /* mem2mem encoder/decoder */
>  #define V4L2_BUF_FLAG_LAST			0x00100000
> +/* Buffer only contains codec header */
> +#define V4L2_BUF_FLAG_CODECCONFIG		0x00200000
>  /* request_fd is valid */
>  #define V4L2_BUF_FLAG_REQUEST_FD		0x00800000
>  


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [EXT] Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Ming Qian]

> From: Nicolas Dufresne <nicolas@ndufresne.ca>
> Sent: 2022年7月4日 23:53
> To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
> hverkuil-cisco@xs4all.nl
> Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
> <linux-imx@nxp.com>; linux-media@vger.kernel.org;
> linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: [EXT] Re: [PATCH] media: videobuf2: add
> V4L2_BUF_FLAG_CODECCONFIG flag
> 
> Caution: EXT Email
> 
> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
> > able to hint decoder the vb2 only contains codec config header, but
> > does not contain any frame data.
> > It's only used for parsing header, and can't be decoded.
> 
> This is copied from OMX specification. I think we we import this, we should at
> least refer to the original.
> 

Hi Nicolas,
    Do you mean OMX_BUFFERFLAG_CODECCONFIG?
    I'm sorry that I didn't notice it before.
    Currently we only encounter this requirement on Android, I'm not sure if it has a reference to omx.
    And thank you very much for pointing out it.

Ming

> >
> > Current, it's usually used by android.
> >
> > Signed-off-by: Ming Qian <ming.qian@nxp.com>
> > ---
> >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> >  include/uapi/linux/videodev2.h                   | 2 ++
> >  2 files changed, 11 insertions(+)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
> > b/Documentation/userspace-api/media/v4l/buffer.rst
> > index 4638ec64db00..acdc4556f4f4 100644
> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > @@ -607,6 +607,15 @@ Buffer Flags
> >       the format. Any subsequent call to the
> >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> >       but return an ``EPIPE`` error code.
> > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > +
> > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> > +      - 0x00200000
> > +      - This flag may be set when the buffer only contains codec config
> > +    header, but does not contain any frame data. Usually the codec config
> > +    header is merged to the next idr frame, with the flag
> > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > +    split the header and queue it separately.
> 
> I think the documentation is clear. Now, if a driver uses this, will existing
> userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will
> break ?
> So we need existing driver to do this when flagged to, and just copy/append
> when the userland didn't opt-in that feature ?
> 
> >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> >
> >        - ``V4L2_BUF_FLAG_REQUEST_FD``
> > diff --git a/include/uapi/linux/videodev2.h
> > b/include/uapi/linux/videodev2.h index 5311ac4fde35..8708ef257710
> > 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
> struct timeval *tv)
> >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
> >  /* mem2mem encoder/decoder */
> >  #define V4L2_BUF_FLAG_LAST                   0x00100000
> > +/* Buffer only contains codec header */
> > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
> >  /* request_fd is valid */
> >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
> >

RE: [EXT] Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Ming Qian]

> From: Nicolas Dufresne <nicolas@ndufresne.ca>
> Sent: 2022年7月4日 23:53
> To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
> hverkuil-cisco@xs4all.nl
> Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
> <linux-imx@nxp.com>; linux-media@vger.kernel.org;
> linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: [EXT] Re: [PATCH] media: videobuf2: add
> V4L2_BUF_FLAG_CODECCONFIG flag
> 
> Caution: EXT Email
> 
> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
> > able to hint decoder the vb2 only contains codec config header, but
> > does not contain any frame data.
> > It's only used for parsing header, and can't be decoded.
> 
> This is copied from OMX specification. I think we we import this, we should at
> least refer to the original.
> 

Hi Nicolas,
    Do you mean OMX_BUFFERFLAG_CODECCONFIG?
    I'm sorry that I didn't notice it before.
    Currently we only encounter this requirement on Android, I'm not sure if it has a reference to omx.
    And thank you very much for pointing out it.

Ming

> >
> > Current, it's usually used by android.
> >
> > Signed-off-by: Ming Qian <ming.qian@nxp.com>
> > ---
> >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> >  include/uapi/linux/videodev2.h                   | 2 ++
> >  2 files changed, 11 insertions(+)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
> > b/Documentation/userspace-api/media/v4l/buffer.rst
> > index 4638ec64db00..acdc4556f4f4 100644
> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > @@ -607,6 +607,15 @@ Buffer Flags
> >       the format. Any subsequent call to the
> >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> >       but return an ``EPIPE`` error code.
> > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > +
> > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> > +      - 0x00200000
> > +      - This flag may be set when the buffer only contains codec config
> > +    header, but does not contain any frame data. Usually the codec config
> > +    header is merged to the next idr frame, with the flag
> > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > +    split the header and queue it separately.
> 
> I think the documentation is clear. Now, if a driver uses this, will existing
> userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will
> break ?
> So we need existing driver to do this when flagged to, and just copy/append
> when the userland didn't opt-in that feature ?
> 
> >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> >
> >        - ``V4L2_BUF_FLAG_REQUEST_FD``
> > diff --git a/include/uapi/linux/videodev2.h
> > b/include/uapi/linux/videodev2.h index 5311ac4fde35..8708ef257710
> > 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
> struct timeval *tv)
> >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
> >  /* mem2mem encoder/decoder */
> >  #define V4L2_BUF_FLAG_LAST                   0x00100000
> > +/* Buffer only contains codec header */
> > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
> >  /* request_fd is valid */
> >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
> >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [EXT] Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Ming Qian]

>>From: Ming Qian
>>Sent: 2022年7月5日 9:52
>>To: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org;
>>hverkuil-cisco@xs4all.nl
>>Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
>>kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
>><linux-imx@nxp.com>; linux-media@vger.kernel.org;
>>linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
>>Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
>>V4L2_BUF_FLAG_CODECCONFIG flag
>>
>>> From: Nicolas Dufresne <nicolas@ndufresne.ca>
>>> Sent: 2022年7月4日 23:53
>>> To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
>>> hverkuil-cisco@xs4all.nl
>>> Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
>>> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
>>> <linux-imx@nxp.com>; linux-media@vger.kernel.org;
>>> linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
>>> Subject: [EXT] Re: [PATCH] media: videobuf2: add
>>> V4L2_BUF_FLAG_CODECCONFIG flag
>>>
>>> Caution: EXT Email
>>>
>>> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
>>> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
>>> > able to hint decoder the vb2 only contains codec config header, but
>>> > does not contain any frame data.
>>> > It's only used for parsing header, and can't be decoded.
>>>
>>> This is copied from OMX specification. I think we we import this, we
>>> should at least refer to the original.
>>>
>>
>>Hi Nicolas,
>>    Do you mean OMX_BUFFERFLAG_CODECCONFIG?
>>    I'm sorry that I didn't notice it before.
>>    Currently we only encounter this requirement on Android, I'm not sure if
>>it has a reference to omx.
>>    And thank you very much for pointing out it.
>>
>
> Android media stack has been based on OMX for the last decade. They are slowly moving to CODEC2 which more or less is a similar abstraction with similar ideas. Let's research prior art, so we don't screw compatibility.
>

I got it, I'll try to study the android codec2, 
and do you agree that we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like OMX_BUFFERFLAG_CODECCONFIG?
Or is there any other solution that can handle this case?

>>Ming
>>
>>> >
>>> > Current, it's usually used by android.
>>> >
>>> > Signed-off-by: Ming Qian <ming.qian@nxp.com>
>>> > ---
>>> >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
>>> >  include/uapi/linux/videodev2.h                   | 2 ++
>>> >  2 files changed, 11 insertions(+)
>>> >
>>> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
>>> > b/Documentation/userspace-api/media/v4l/buffer.rst
>>> > index 4638ec64db00..acdc4556f4f4 100644
>>> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
>>> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>>> > @@ -607,6 +607,15 @@ Buffer Flags
>>> >       the format. Any subsequent call to the
>>> >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
>>> >       but return an ``EPIPE`` error code.
>>> > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
>>> > +
>>> > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
>>> > +      - 0x00200000
>>> > +      - This flag may be set when the buffer only contains codec config
>>> > +    header, but does not contain any frame data. Usually the codec
>>config
>>> > +    header is merged to the next idr frame, with the flag
>>> > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that
>>will
>>> > +    split the header and queue it separately.
>>>
>>> I think the documentation is clear. Now, if a driver uses this, will
>>> existing userland (perhaps good to check GStreamer, FFMPEG and
>>> Chromium ?) will break ?
>>> So we need existing driver to do this when flagged to, and just
>>> copy/append when the userland didn't opt-in that feature ?
>>>
>>> >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>>> >
>>> >        - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
>>> > a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>> > index 5311ac4fde35..8708ef257710
>>> > 100644
>>> > --- a/include/uapi/linux/videodev2.h
>>> > +++ b/include/uapi/linux/videodev2.h
>>> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
>>> struct timeval *tv)
>>> >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
>>> >  /* mem2mem encoder/decoder */
>>> >  #define V4L2_BUF_FLAG_LAST                   0x00100000
>>> > +/* Buffer only contains codec header */
>>> > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
>>> >  /* request_fd is valid */
>>> >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
>>> >

RE: [EXT] Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Ming Qian]

>>From: Ming Qian
>>Sent: 2022年7月5日 9:52
>>To: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org;
>>hverkuil-cisco@xs4all.nl
>>Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
>>kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
>><linux-imx@nxp.com>; linux-media@vger.kernel.org;
>>linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
>>Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
>>V4L2_BUF_FLAG_CODECCONFIG flag
>>
>>> From: Nicolas Dufresne <nicolas@ndufresne.ca>
>>> Sent: 2022年7月4日 23:53
>>> To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
>>> hverkuil-cisco@xs4all.nl
>>> Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
>>> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
>>> <linux-imx@nxp.com>; linux-media@vger.kernel.org;
>>> linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
>>> Subject: [EXT] Re: [PATCH] media: videobuf2: add
>>> V4L2_BUF_FLAG_CODECCONFIG flag
>>>
>>> Caution: EXT Email
>>>
>>> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
>>> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
>>> > able to hint decoder the vb2 only contains codec config header, but
>>> > does not contain any frame data.
>>> > It's only used for parsing header, and can't be decoded.
>>>
>>> This is copied from OMX specification. I think we we import this, we
>>> should at least refer to the original.
>>>
>>
>>Hi Nicolas,
>>    Do you mean OMX_BUFFERFLAG_CODECCONFIG?
>>    I'm sorry that I didn't notice it before.
>>    Currently we only encounter this requirement on Android, I'm not sure if
>>it has a reference to omx.
>>    And thank you very much for pointing out it.
>>
>
> Android media stack has been based on OMX for the last decade. They are slowly moving to CODEC2 which more or less is a similar abstraction with similar ideas. Let's research prior art, so we don't screw compatibility.
>

I got it, I'll try to study the android codec2, 
and do you agree that we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like OMX_BUFFERFLAG_CODECCONFIG?
Or is there any other solution that can handle this case?

>>Ming
>>
>>> >
>>> > Current, it's usually used by android.
>>> >
>>> > Signed-off-by: Ming Qian <ming.qian@nxp.com>
>>> > ---
>>> >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
>>> >  include/uapi/linux/videodev2.h                   | 2 ++
>>> >  2 files changed, 11 insertions(+)
>>> >
>>> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
>>> > b/Documentation/userspace-api/media/v4l/buffer.rst
>>> > index 4638ec64db00..acdc4556f4f4 100644
>>> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
>>> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>>> > @@ -607,6 +607,15 @@ Buffer Flags
>>> >       the format. Any subsequent call to the
>>> >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
>>> >       but return an ``EPIPE`` error code.
>>> > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
>>> > +
>>> > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
>>> > +      - 0x00200000
>>> > +      - This flag may be set when the buffer only contains codec config
>>> > +    header, but does not contain any frame data. Usually the codec
>>config
>>> > +    header is merged to the next idr frame, with the flag
>>> > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that
>>will
>>> > +    split the header and queue it separately.
>>>
>>> I think the documentation is clear. Now, if a driver uses this, will
>>> existing userland (perhaps good to check GStreamer, FFMPEG and
>>> Chromium ?) will break ?
>>> So we need existing driver to do this when flagged to, and just
>>> copy/append when the userland didn't opt-in that feature ?
>>>
>>> >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>>> >
>>> >        - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
>>> > a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>> > index 5311ac4fde35..8708ef257710
>>> > 100644
>>> > --- a/include/uapi/linux/videodev2.h
>>> > +++ b/include/uapi/linux/videodev2.h
>>> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
>>> struct timeval *tv)
>>> >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
>>> >  /* mem2mem encoder/decoder */
>>> >  #define V4L2_BUF_FLAG_LAST                   0x00100000
>>> > +/* Buffer only contains codec header */
>>> > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
>>> >  /* request_fd is valid */
>>> >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
>>> >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [EXT] Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Nicolas Dufresne]

Le mardi 05 juillet 2022 à 11:34 +0000, Ming Qian a écrit :
> > > From: Ming Qian
> > > Sent: 2022年7月5日 9:52
> > > To: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org;
> > > hverkuil-cisco@xs4all.nl
> > > Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
> > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
> > > <linux-imx@nxp.com>; linux-media@vger.kernel.org;
> > > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > > Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
> > > V4L2_BUF_FLAG_CODECCONFIG flag
> > > 
> > > > From: Nicolas Dufresne <nicolas@ndufresne.ca>
> > > > Sent: 2022年7月4日 23:53
> > > > To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
> > > > hverkuil-cisco@xs4all.nl
> > > > Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
> > > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
> > > > <linux-imx@nxp.com>; linux-media@vger.kernel.org;
> > > > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > > > Subject: [EXT] Re: [PATCH] media: videobuf2: add
> > > > V4L2_BUF_FLAG_CODECCONFIG flag
> > > > 
> > > > Caution: EXT Email
> > > > 
> > > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > > > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
> > > > > able to hint decoder the vb2 only contains codec config header, but
> > > > > does not contain any frame data.
> > > > > It's only used for parsing header, and can't be decoded.
> > > > 
> > > > This is copied from OMX specification. I think we we import this, we
> > > > should at least refer to the original.
> > > > 
> > > 
> > > Hi Nicolas,
> > >    Do you mean OMX_BUFFERFLAG_CODECCONFIG?
> > >    I'm sorry that I didn't notice it before.
> > >    Currently we only encounter this requirement on Android, I'm not sure
> > > if
> > > it has a reference to omx.
> > >    And thank you very much for pointing out it.
> > > 
> > 
> > Android media stack has been based on OMX for the last decade. They are
> > slowly moving to CODEC2 which more or less is a similar abstraction with
> > similar ideas. Let's research prior art, so we don't screw compatibility.
> > 
> 
> I got it, I'll try to study the android codec2, 
> and do you agree that we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like
> OMX_BUFFERFLAG_CODECCONFIG?
> Or is there any other solution that can handle this case?

We can probably discuss the name. CODECCONFIG is a bit strange, could be
CODEC_HEADER, HEADER_ONLY, CONFIG_ONLY, something along these lines. I'm just
wondering what is the best rule, since more specification is needed here.
Current userland expect full frames into each encoded buffer. If we start
splitting these up, we'll break non-android userland (and Android userland does
not seems to be very upstream thing, every vendor forks it).

I also think that what the CODECCONFIG contains and its format need to be
strictly documented for every CODEC that would allow it. In H.264 notably, the
headers could be packed in Annex B. or AVCc NAL headers. If we look at FFMPEG,
which uses codec_data name, they only requires this when the header are not
"inline", which means only for AVCc. Also, many codec_data is for other codecs
get wrapped into ISOMP4 or Matroska (webm) envelope.

On the other hand, I don't remember if the 1 frame per buffer is an actual rule
or simply what existing userland expect. Also, I'll be fair, there is no reason
this must come from the driver. Android OMX or CODEC2 COMPONENT is a userland
component, it could do bitstream scanning (using traditional boyer-more) to find
and split appart the config to satisfy its internal API. This can be done with
low overhead and zero-copy.

> 
> > > Ming
> > > 
> > > > > 
> > > > > Current, it's usually used by android.
> > > > > 
> > > > > Signed-off-by: Ming Qian <ming.qian@nxp.com>
> > > > > ---
> > > > >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> > > > >  include/uapi/linux/videodev2.h                   | 2 ++
> > > > >  2 files changed, 11 insertions(+)
> > > > > 
> > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > b/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > index 4638ec64db00..acdc4556f4f4 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > @@ -607,6 +607,15 @@ Buffer Flags
> > > > >       the format. Any subsequent call to the
> > > > >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> > > > >       but return an ``EPIPE`` error code.
> > > > > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > > > > +
> > > > > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> > > > > +      - 0x00200000
> > > > > +      - This flag may be set when the buffer only contains codec
> > > > > config
> > > > > +    header, but does not contain any frame data. Usually the codec
> > > config
> > > > > +    header is merged to the next idr frame, with the flag
> > > > > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that
> > > will
> > > > > +    split the header and queue it separately.
> > > > 
> > > > I think the documentation is clear. Now, if a driver uses this, will
> > > > existing userland (perhaps good to check GStreamer, FFMPEG and
> > > > Chromium ?) will break ?
> > > > So we need existing driver to do this when flagged to, and just
> > > > copy/append when the userland didn't opt-in that feature ?
> > > > 
> > > > >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> > > > > 
> > > > >        - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
> > > > > a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > > > index 5311ac4fde35..8708ef257710
> > > > > 100644
> > > > > --- a/include/uapi/linux/videodev2.h
> > > > > +++ b/include/uapi/linux/videodev2.h
> > > > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
> > > > struct timeval *tv)
> > > > >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
> > > > >  /* mem2mem encoder/decoder */
> > > > >  #define V4L2_BUF_FLAG_LAST                   0x00100000
> > > > > +/* Buffer only contains codec header */
> > > > > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
> > > > >  /* request_fd is valid */
> > > > >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
> > > > > 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [EXT] Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Nicolas Dufresne]

Le mardi 05 juillet 2022 à 11:34 +0000, Ming Qian a écrit :
> > > From: Ming Qian
> > > Sent: 2022年7月5日 9:52
> > > To: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org;
> > > hverkuil-cisco@xs4all.nl
> > > Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
> > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
> > > <linux-imx@nxp.com>; linux-media@vger.kernel.org;
> > > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > > Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
> > > V4L2_BUF_FLAG_CODECCONFIG flag
> > > 
> > > > From: Nicolas Dufresne <nicolas@ndufresne.ca>
> > > > Sent: 2022年7月4日 23:53
> > > > To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
> > > > hverkuil-cisco@xs4all.nl
> > > > Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
> > > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
> > > > <linux-imx@nxp.com>; linux-media@vger.kernel.org;
> > > > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> > > > Subject: [EXT] Re: [PATCH] media: videobuf2: add
> > > > V4L2_BUF_FLAG_CODECCONFIG flag
> > > > 
> > > > Caution: EXT Email
> > > > 
> > > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > > > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
> > > > > able to hint decoder the vb2 only contains codec config header, but
> > > > > does not contain any frame data.
> > > > > It's only used for parsing header, and can't be decoded.
> > > > 
> > > > This is copied from OMX specification. I think we we import this, we
> > > > should at least refer to the original.
> > > > 
> > > 
> > > Hi Nicolas,
> > >    Do you mean OMX_BUFFERFLAG_CODECCONFIG?
> > >    I'm sorry that I didn't notice it before.
> > >    Currently we only encounter this requirement on Android, I'm not sure
> > > if
> > > it has a reference to omx.
> > >    And thank you very much for pointing out it.
> > > 
> > 
> > Android media stack has been based on OMX for the last decade. They are
> > slowly moving to CODEC2 which more or less is a similar abstraction with
> > similar ideas. Let's research prior art, so we don't screw compatibility.
> > 
> 
> I got it, I'll try to study the android codec2, 
> and do you agree that we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like
> OMX_BUFFERFLAG_CODECCONFIG?
> Or is there any other solution that can handle this case?

We can probably discuss the name. CODECCONFIG is a bit strange, could be
CODEC_HEADER, HEADER_ONLY, CONFIG_ONLY, something along these lines. I'm just
wondering what is the best rule, since more specification is needed here.
Current userland expect full frames into each encoded buffer. If we start
splitting these up, we'll break non-android userland (and Android userland does
not seems to be very upstream thing, every vendor forks it).

I also think that what the CODECCONFIG contains and its format need to be
strictly documented for every CODEC that would allow it. In H.264 notably, the
headers could be packed in Annex B. or AVCc NAL headers. If we look at FFMPEG,
which uses codec_data name, they only requires this when the header are not
"inline", which means only for AVCc. Also, many codec_data is for other codecs
get wrapped into ISOMP4 or Matroska (webm) envelope.

On the other hand, I don't remember if the 1 frame per buffer is an actual rule
or simply what existing userland expect. Also, I'll be fair, there is no reason
this must come from the driver. Android OMX or CODEC2 COMPONENT is a userland
component, it could do bitstream scanning (using traditional boyer-more) to find
and split appart the config to satisfy its internal API. This can be done with
low overhead and zero-copy.

> 
> > > Ming
> > > 
> > > > > 
> > > > > Current, it's usually used by android.
> > > > > 
> > > > > Signed-off-by: Ming Qian <ming.qian@nxp.com>
> > > > > ---
> > > > >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> > > > >  include/uapi/linux/videodev2.h                   | 2 ++
> > > > >  2 files changed, 11 insertions(+)
> > > > > 
> > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > b/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > index 4638ec64db00..acdc4556f4f4 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > @@ -607,6 +607,15 @@ Buffer Flags
> > > > >       the format. Any subsequent call to the
> > > > >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> > > > >       but return an ``EPIPE`` error code.
> > > > > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > > > > +
> > > > > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> > > > > +      - 0x00200000
> > > > > +      - This flag may be set when the buffer only contains codec
> > > > > config
> > > > > +    header, but does not contain any frame data. Usually the codec
> > > config
> > > > > +    header is merged to the next idr frame, with the flag
> > > > > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that
> > > will
> > > > > +    split the header and queue it separately.
> > > > 
> > > > I think the documentation is clear. Now, if a driver uses this, will
> > > > existing userland (perhaps good to check GStreamer, FFMPEG and
> > > > Chromium ?) will break ?
> > > > So we need existing driver to do this when flagged to, and just
> > > > copy/append when the userland didn't opt-in that feature ?
> > > > 
> > > > >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> > > > > 
> > > > >        - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
> > > > > a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > > > index 5311ac4fde35..8708ef257710
> > > > > 100644
> > > > > --- a/include/uapi/linux/videodev2.h
> > > > > +++ b/include/uapi/linux/videodev2.h
> > > > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
> > > > struct timeval *tv)
> > > > >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
> > > > >  /* mem2mem encoder/decoder */
> > > > >  #define V4L2_BUF_FLAG_LAST                   0x00100000
> > > > > +/* Buffer only contains codec header */
> > > > > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
> > > > >  /* request_fd is valid */
> > > > >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
> > > > > 
> 

RE: [EXT] Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Ming Qian]

>From: Nicolas Dufresne <nicolas@ndufresne.ca>
>Sent: 2022年7月5日 21:13
>To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
>hverkuil-cisco@xs4all.nl
>Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
>kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
><linux-imx@nxp.com>; linux-media@vger.kernel.org;
>linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
>Subject: Re: [EXT] Re: [PATCH] media: videobuf2: add
>V4L2_BUF_FLAG_CODECCONFIG flag
>
>Caution: EXT Email
>
>Le mardi 05 juillet 2022 à 11:34 +0000, Ming Qian a écrit :
>> > > From: Ming Qian
>> > > Sent: 2022年7月5日 9:52
>> > > To: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org;
>> > > hverkuil-cisco@xs4all.nl
>> > > Cc: shawnguo@kernel.org; robh+dt@kernel.org;
>> > > s.hauer@pengutronix.de; kernel@pengutronix.de; festevam@gmail.com;
>> > > dl-linux-imx <linux-imx@nxp.com>; linux-media@vger.kernel.org;
>> > > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
>> > > Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
>> > > V4L2_BUF_FLAG_CODECCONFIG flag
>> > >
>> > > > From: Nicolas Dufresne <nicolas@ndufresne.ca>
>> > > > Sent: 2022年7月4日 23:53
>> > > > To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
>> > > > hverkuil-cisco@xs4all.nl
>> > > > Cc: shawnguo@kernel.org; robh+dt@kernel.org;
>> > > > s.hauer@pengutronix.de; kernel@pengutronix.de;
>> > > > festevam@gmail.com; dl-linux-imx <linux-imx@nxp.com>;
>> > > > linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
>> > > > linux-arm-kernel@lists.infradead.org
>> > > > Subject: [EXT] Re: [PATCH] media: videobuf2: add
>> > > > V4L2_BUF_FLAG_CODECCONFIG flag
>> > > >
>> > > > Caution: EXT Email
>> > > >
>> > > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
>> > > > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space
>> > > > > should be able to hint decoder the vb2 only contains codec
>> > > > > config header, but does not contain any frame data.
>> > > > > It's only used for parsing header, and can't be decoded.
>> > > >
>> > > > This is copied from OMX specification. I think we we import
>> > > > this, we should at least refer to the original.
>> > > >
>> > >
>> > > Hi Nicolas,
>> > >    Do you mean OMX_BUFFERFLAG_CODECCONFIG?
>> > >    I'm sorry that I didn't notice it before.
>> > >    Currently we only encounter this requirement on Android, I'm
>> > > not sure if it has a reference to omx.
>> > >    And thank you very much for pointing out it.
>> > >
>> >
>> > Android media stack has been based on OMX for the last decade. They
>> > are slowly moving to CODEC2 which more or less is a similar
>> > abstraction with similar ideas. Let's research prior art, so we don't screw
>compatibility.
>> >
>>
>> I got it, I'll try to study the android codec2, and do you agree that
>> we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like
>> OMX_BUFFERFLAG_CODECCONFIG?
>> Or is there any other solution that can handle this case?
>
>We can probably discuss the name. CODECCONFIG is a bit strange, could be
>CODEC_HEADER, HEADER_ONLY, CONFIG_ONLY, something along these lines.
>I'm just wondering what is the best rule, since more specification is needed
>here.
>Current userland expect full frames into each encoded buffer. If we start
>splitting these up, we'll break non-android userland (and Android userland does
>not seems to be very upstream thing, every vendor forks it).
>
>I also think that what the CODECCONFIG contains and its format need to be
>strictly documented for every CODEC that would allow it. In H.264 notably, the
>headers could be packed in Annex B. or AVCc NAL headers. If we look at
>FFMPEG, which uses codec_data name, they only requires this when the header
>are not "inline", which means only for AVCc. Also, many codec_data is for other
>codecs get wrapped into ISOMP4 or Matroska (webm) envelope.
>
>On the other hand, I don't remember if the 1 frame per buffer is an actual rule
>or simply what existing userland expect. Also, I'll be fair, there is no reason this
>must come from the driver. Android OMX or CODEC2 COMPONENT is a userland
>component, it could do bitstream scanning (using traditional boyer-more) to
>find and split appart the config to satisfy its internal API. This can be done with
>low overhead and zero-copy.
>

OK, I'll try to change the name, maybe CODEC_HEADER is better.
And I think we add the flag just to support more scenarios, it won't break the existing userland flow.

It's not easy to change the android codec2, and we indeed want to support it.

>>
>> > > Ming
>> > >
>> > > > >
>> > > > > Current, it's usually used by android.
>> > > > >
>> > > > > Signed-off-by: Ming Qian <ming.qian@nxp.com>
>> > > > > ---
>> > > > >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
>> > > > >  include/uapi/linux/videodev2.h                   | 2 ++
>> > > > >  2 files changed, 11 insertions(+)
>> > > > >
>> > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > b/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > index 4638ec64db00..acdc4556f4f4 100644
>> > > > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > @@ -607,6 +607,15 @@ Buffer Flags
>> > > > >       the format. Any subsequent call to the
>> > > > >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block
>anymore,
>> > > > >       but return an ``EPIPE`` error code.
>> > > > > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
>> > > > > +
>> > > > > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
>> > > > > +      - 0x00200000
>> > > > > +      - This flag may be set when the buffer only contains
>> > > > > + codec
>> > > > > config
>> > > > > +    header, but does not contain any frame data. Usually the
>> > > > > + codec
>> > > config
>> > > > > +    header is merged to the next idr frame, with the flag
>> > > > > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some
>> > > > > + scenes that
>> > > will
>> > > > > +    split the header and queue it separately.
>> > > >
>> > > > I think the documentation is clear. Now, if a driver uses this,
>> > > > will existing userland (perhaps good to check GStreamer, FFMPEG
>> > > > and Chromium ?) will break ?
>> > > > So we need existing driver to do this when flagged to, and just
>> > > > copy/append when the userland didn't opt-in that feature ?
>> > > >
>> > > > >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>> > > > >
>> > > > >        - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
>> > > > > a/include/uapi/linux/videodev2.h
>> > > > > b/include/uapi/linux/videodev2.h index
>> > > > > 5311ac4fde35..8708ef257710
>> > > > > 100644
>> > > > > --- a/include/uapi/linux/videodev2.h
>> > > > > +++ b/include/uapi/linux/videodev2.h
>> > > > > @@ -1131,6 +1131,8 @@ static inline __u64
>> > > > > v4l2_timeval_to_ns(const
>> > > > struct timeval *tv)
>> > > > >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
>> > > > >  /* mem2mem encoder/decoder */
>> > > > >  #define V4L2_BUF_FLAG_LAST                   0x00100000
>> > > > > +/* Buffer only contains codec header */
>> > > > > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
>> > > > >  /* request_fd is valid */
>> > > > >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
>> > > > >
>>

RE: [EXT] Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Ming Qian]

>From: Nicolas Dufresne <nicolas@ndufresne.ca>
>Sent: 2022年7月5日 21:13
>To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
>hverkuil-cisco@xs4all.nl
>Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de;
>kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
><linux-imx@nxp.com>; linux-media@vger.kernel.org;
>linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
>Subject: Re: [EXT] Re: [PATCH] media: videobuf2: add
>V4L2_BUF_FLAG_CODECCONFIG flag
>
>Caution: EXT Email
>
>Le mardi 05 juillet 2022 à 11:34 +0000, Ming Qian a écrit :
>> > > From: Ming Qian
>> > > Sent: 2022年7月5日 9:52
>> > > To: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org;
>> > > hverkuil-cisco@xs4all.nl
>> > > Cc: shawnguo@kernel.org; robh+dt@kernel.org;
>> > > s.hauer@pengutronix.de; kernel@pengutronix.de; festevam@gmail.com;
>> > > dl-linux-imx <linux-imx@nxp.com>; linux-media@vger.kernel.org;
>> > > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
>> > > Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
>> > > V4L2_BUF_FLAG_CODECCONFIG flag
>> > >
>> > > > From: Nicolas Dufresne <nicolas@ndufresne.ca>
>> > > > Sent: 2022年7月4日 23:53
>> > > > To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
>> > > > hverkuil-cisco@xs4all.nl
>> > > > Cc: shawnguo@kernel.org; robh+dt@kernel.org;
>> > > > s.hauer@pengutronix.de; kernel@pengutronix.de;
>> > > > festevam@gmail.com; dl-linux-imx <linux-imx@nxp.com>;
>> > > > linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
>> > > > linux-arm-kernel@lists.infradead.org
>> > > > Subject: [EXT] Re: [PATCH] media: videobuf2: add
>> > > > V4L2_BUF_FLAG_CODECCONFIG flag
>> > > >
>> > > > Caution: EXT Email
>> > > >
>> > > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
>> > > > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space
>> > > > > should be able to hint decoder the vb2 only contains codec
>> > > > > config header, but does not contain any frame data.
>> > > > > It's only used for parsing header, and can't be decoded.
>> > > >
>> > > > This is copied from OMX specification. I think we we import
>> > > > this, we should at least refer to the original.
>> > > >
>> > >
>> > > Hi Nicolas,
>> > >    Do you mean OMX_BUFFERFLAG_CODECCONFIG?
>> > >    I'm sorry that I didn't notice it before.
>> > >    Currently we only encounter this requirement on Android, I'm
>> > > not sure if it has a reference to omx.
>> > >    And thank you very much for pointing out it.
>> > >
>> >
>> > Android media stack has been based on OMX for the last decade. They
>> > are slowly moving to CODEC2 which more or less is a similar
>> > abstraction with similar ideas. Let's research prior art, so we don't screw
>compatibility.
>> >
>>
>> I got it, I'll try to study the android codec2, and do you agree that
>> we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like
>> OMX_BUFFERFLAG_CODECCONFIG?
>> Or is there any other solution that can handle this case?
>
>We can probably discuss the name. CODECCONFIG is a bit strange, could be
>CODEC_HEADER, HEADER_ONLY, CONFIG_ONLY, something along these lines.
>I'm just wondering what is the best rule, since more specification is needed
>here.
>Current userland expect full frames into each encoded buffer. If we start
>splitting these up, we'll break non-android userland (and Android userland does
>not seems to be very upstream thing, every vendor forks it).
>
>I also think that what the CODECCONFIG contains and its format need to be
>strictly documented for every CODEC that would allow it. In H.264 notably, the
>headers could be packed in Annex B. or AVCc NAL headers. If we look at
>FFMPEG, which uses codec_data name, they only requires this when the header
>are not "inline", which means only for AVCc. Also, many codec_data is for other
>codecs get wrapped into ISOMP4 or Matroska (webm) envelope.
>
>On the other hand, I don't remember if the 1 frame per buffer is an actual rule
>or simply what existing userland expect. Also, I'll be fair, there is no reason this
>must come from the driver. Android OMX or CODEC2 COMPONENT is a userland
>component, it could do bitstream scanning (using traditional boyer-more) to
>find and split appart the config to satisfy its internal API. This can be done with
>low overhead and zero-copy.
>

OK, I'll try to change the name, maybe CODEC_HEADER is better.
And I think we add the flag just to support more scenarios, it won't break the existing userland flow.

It's not easy to change the android codec2, and we indeed want to support it.

>>
>> > > Ming
>> > >
>> > > > >
>> > > > > Current, it's usually used by android.
>> > > > >
>> > > > > Signed-off-by: Ming Qian <ming.qian@nxp.com>
>> > > > > ---
>> > > > >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
>> > > > >  include/uapi/linux/videodev2.h                   | 2 ++
>> > > > >  2 files changed, 11 insertions(+)
>> > > > >
>> > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > b/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > index 4638ec64db00..acdc4556f4f4 100644
>> > > > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > @@ -607,6 +607,15 @@ Buffer Flags
>> > > > >       the format. Any subsequent call to the
>> > > > >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block
>anymore,
>> > > > >       but return an ``EPIPE`` error code.
>> > > > > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
>> > > > > +
>> > > > > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
>> > > > > +      - 0x00200000
>> > > > > +      - This flag may be set when the buffer only contains
>> > > > > + codec
>> > > > > config
>> > > > > +    header, but does not contain any frame data. Usually the
>> > > > > + codec
>> > > config
>> > > > > +    header is merged to the next idr frame, with the flag
>> > > > > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some
>> > > > > + scenes that
>> > > will
>> > > > > +    split the header and queue it separately.
>> > > >
>> > > > I think the documentation is clear. Now, if a driver uses this,
>> > > > will existing userland (perhaps good to check GStreamer, FFMPEG
>> > > > and Chromium ?) will break ?
>> > > > So we need existing driver to do this when flagged to, and just
>> > > > copy/append when the userland didn't opt-in that feature ?
>> > > >
>> > > > >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>> > > > >
>> > > > >        - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
>> > > > > a/include/uapi/linux/videodev2.h
>> > > > > b/include/uapi/linux/videodev2.h index
>> > > > > 5311ac4fde35..8708ef257710
>> > > > > 100644
>> > > > > --- a/include/uapi/linux/videodev2.h
>> > > > > +++ b/include/uapi/linux/videodev2.h
>> > > > > @@ -1131,6 +1131,8 @@ static inline __u64
>> > > > > v4l2_timeval_to_ns(const
>> > > > struct timeval *tv)
>> > > > >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
>> > > > >  /* mem2mem encoder/decoder */
>> > > > >  #define V4L2_BUF_FLAG_LAST                   0x00100000
>> > > > > +/* Buffer only contains codec header */
>> > > > > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
>> > > > >  /* request_fd is valid */
>> > > > >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
>> > > > >
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Dave Stevenson]

Hi Ming and Nicolas

On Mon, 4 Jul 2022 at 16:53, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>
> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> > user-space should be able to hint decoder
> > the vb2 only contains codec config header,
> > but does not contain any frame data.
> > It's only used for parsing header, and can't be decoded.
>
> This is copied from OMX specification. I think we we import this, we should at
> least refer to the original.
>
> >
> > Current, it's usually used by android.
> >
> > Signed-off-by: Ming Qian <ming.qian@nxp.com>
> > ---
> >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> >  include/uapi/linux/videodev2.h                   | 2 ++
> >  2 files changed, 11 insertions(+)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> > index 4638ec64db00..acdc4556f4f4 100644
> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > @@ -607,6 +607,15 @@ Buffer Flags
> >       the format. Any subsequent call to the
> >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> >       but return an ``EPIPE`` error code.
> > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > +
> > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> > +      - 0x00200000
> > +      - This flag may be set when the buffer only contains codec config
> > +    header, but does not contain any frame data. Usually the codec config
> > +    header is merged to the next idr frame, with the flag
> > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > +    split the header and queue it separately.
>
> I think the documentation is clear. Now, if a driver uses this, will existing
> userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
> So we need existing driver to do this when flagged to, and just copy/append when
> the userland didn't opt-in that feature ?

The commit text says it is for userspace feeding data into a video
decoder, so it's a userspace choice instead of driver.

For encoders there is already V4L2_CID_MPEG_VIDEO_HEADER_MODE [1]
which allows for V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE or
V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. FFmpeg selects
_SEPARATE by default [2], whilst the default is normally
_JOINED_WITH_1ST_FRAME.

It does raise the question as to whether all decoders will support
header byte only buffers, and does there need to be a capabilities
flag to denote that it is supported.
And should encoders in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE mode set
it on the headers only buffers?

A number of undefined elements of how this should be implemented/used :-(

  Dave

[1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html
[2] https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c#L196

> >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> >
> >        - ``V4L2_BUF_FLAG_REQUEST_FD``
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index 5311ac4fde35..8708ef257710 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
> >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
> >  /* mem2mem encoder/decoder */
> >  #define V4L2_BUF_FLAG_LAST                   0x00100000
> > +/* Buffer only contains codec header */
> > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
> >  /* request_fd is valid */
> >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
> >
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Dave Stevenson]

Hi Ming and Nicolas

On Mon, 4 Jul 2022 at 16:53, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>
> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> > user-space should be able to hint decoder
> > the vb2 only contains codec config header,
> > but does not contain any frame data.
> > It's only used for parsing header, and can't be decoded.
>
> This is copied from OMX specification. I think we we import this, we should at
> least refer to the original.
>
> >
> > Current, it's usually used by android.
> >
> > Signed-off-by: Ming Qian <ming.qian@nxp.com>
> > ---
> >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> >  include/uapi/linux/videodev2.h                   | 2 ++
> >  2 files changed, 11 insertions(+)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> > index 4638ec64db00..acdc4556f4f4 100644
> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > @@ -607,6 +607,15 @@ Buffer Flags
> >       the format. Any subsequent call to the
> >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> >       but return an ``EPIPE`` error code.
> > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > +
> > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> > +      - 0x00200000
> > +      - This flag may be set when the buffer only contains codec config
> > +    header, but does not contain any frame data. Usually the codec config
> > +    header is merged to the next idr frame, with the flag
> > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > +    split the header and queue it separately.
>
> I think the documentation is clear. Now, if a driver uses this, will existing
> userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
> So we need existing driver to do this when flagged to, and just copy/append when
> the userland didn't opt-in that feature ?

The commit text says it is for userspace feeding data into a video
decoder, so it's a userspace choice instead of driver.

For encoders there is already V4L2_CID_MPEG_VIDEO_HEADER_MODE [1]
which allows for V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE or
V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. FFmpeg selects
_SEPARATE by default [2], whilst the default is normally
_JOINED_WITH_1ST_FRAME.

It does raise the question as to whether all decoders will support
header byte only buffers, and does there need to be a capabilities
flag to denote that it is supported.
And should encoders in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE mode set
it on the headers only buffers?

A number of undefined elements of how this should be implemented/used :-(

  Dave

[1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html
[2] https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c#L196

> >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> >
> >        - ``V4L2_BUF_FLAG_REQUEST_FD``
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index 5311ac4fde35..8708ef257710 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
> >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
> >  /* mem2mem encoder/decoder */
> >  #define V4L2_BUF_FLAG_LAST                   0x00100000
> > +/* Buffer only contains codec header */
> > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
> >  /* request_fd is valid */
> >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
> >
>

Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Nicolas Dufresne]

Le mardi 05 juillet 2022 à 13:34 +0100, Dave Stevenson a écrit :
> Hi Ming and Nicolas
> 
> On Mon, 4 Jul 2022 at 16:53, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> > 
> > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> > > user-space should be able to hint decoder
> > > the vb2 only contains codec config header,
> > > but does not contain any frame data.
> > > It's only used for parsing header, and can't be decoded.
> > 
> > This is copied from OMX specification. I think we we import this, we should at
> > least refer to the original.
> > 
> > > 
> > > Current, it's usually used by android.
> > > 
> > > Signed-off-by: Ming Qian <ming.qian@nxp.com>
> > > ---
> > >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> > >  include/uapi/linux/videodev2.h                   | 2 ++
> > >  2 files changed, 11 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> > > index 4638ec64db00..acdc4556f4f4 100644
> > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > > @@ -607,6 +607,15 @@ Buffer Flags
> > >       the format. Any subsequent call to the
> > >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> > >       but return an ``EPIPE`` error code.
> > > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > > +
> > > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> > > +      - 0x00200000
> > > +      - This flag may be set when the buffer only contains codec config
> > > +    header, but does not contain any frame data. Usually the codec config
> > > +    header is merged to the next idr frame, with the flag
> > > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > > +    split the header and queue it separately.
> > 
> > I think the documentation is clear. Now, if a driver uses this, will existing
> > userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
> > So we need existing driver to do this when flagged to, and just copy/append when
> > the userland didn't opt-in that feature ?
> 
> The commit text says it is for userspace feeding data into a video
> decoder, so it's a userspace choice instead of driver.

I see, the spec needs to be more clear then.

> 
> For encoders there is already V4L2_CID_MPEG_VIDEO_HEADER_MODE [1]
> which allows for V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE or
> V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. FFmpeg selects
> _SEPARATE by default [2], whilst the default is normally
> _JOINED_WITH_1ST_FRAME.

I did miss the addition of this API, thanks for the reminder. The problem right
now is that things are being added with the needed cross-reference.

> 
> It does raise the question as to whether all decoders will support
> header byte only buffers, and does there need to be a capabilities
> flag to denote that it is supported.
> And should encoders in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE mode set
> it on the headers only buffers?

What about:
- Document better that in absence of V4L2_CID_MPEG_VIDEO_HEADER_MODE in a
driver, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME is to be assumed
- Document that in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE, driver should signal
the header buffer with the newly added buffer flag (I would suggest
V4L2_BUF_FLAG_HEADERS_ONLY)
- Document for each support CODECs if V4L2_CID_MPEG_VIDEO_HEADER_MODE can be
supported, and the expected headers and the packing of these 
- Document that decoders that didn't implement V4L2_CID_MPEG_VIDEO_HEADER_MODE
should be assumed to only support 
V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME decoding (even though it
allowed to support any alignment, even random).
- Cross-reference V4L2_CID_MPEG_VIDEO_HEADER_MODE into the new flag
documentation (whatever this new flag will be called).

> 
> A number of undefined elements of how this should be implemented/used :-(
> 
>   Dave
> 
> [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html
> [2] https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c#L196

> 
> > >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> > > 
> > >        - ``V4L2_BUF_FLAG_REQUEST_FD``
> > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > index 5311ac4fde35..8708ef257710 100644
> > > --- a/include/uapi/linux/videodev2.h
> > > +++ b/include/uapi/linux/videodev2.h
> > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
> > >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
> > >  /* mem2mem encoder/decoder */
> > >  #define V4L2_BUF_FLAG_LAST                   0x00100000
> > > +/* Buffer only contains codec header */
> > > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
> > >  /* request_fd is valid */
> > >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
> > > 
> > 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Nicolas Dufresne]

Le mardi 05 juillet 2022 à 13:34 +0100, Dave Stevenson a écrit :
> Hi Ming and Nicolas
> 
> On Mon, 4 Jul 2022 at 16:53, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> > 
> > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> > > user-space should be able to hint decoder
> > > the vb2 only contains codec config header,
> > > but does not contain any frame data.
> > > It's only used for parsing header, and can't be decoded.
> > 
> > This is copied from OMX specification. I think we we import this, we should at
> > least refer to the original.
> > 
> > > 
> > > Current, it's usually used by android.
> > > 
> > > Signed-off-by: Ming Qian <ming.qian@nxp.com>
> > > ---
> > >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> > >  include/uapi/linux/videodev2.h                   | 2 ++
> > >  2 files changed, 11 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> > > index 4638ec64db00..acdc4556f4f4 100644
> > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > > @@ -607,6 +607,15 @@ Buffer Flags
> > >       the format. Any subsequent call to the
> > >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> > >       but return an ``EPIPE`` error code.
> > > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > > +
> > > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
> > > +      - 0x00200000
> > > +      - This flag may be set when the buffer only contains codec config
> > > +    header, but does not contain any frame data. Usually the codec config
> > > +    header is merged to the next idr frame, with the flag
> > > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > > +    split the header and queue it separately.
> > 
> > I think the documentation is clear. Now, if a driver uses this, will existing
> > userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
> > So we need existing driver to do this when flagged to, and just copy/append when
> > the userland didn't opt-in that feature ?
> 
> The commit text says it is for userspace feeding data into a video
> decoder, so it's a userspace choice instead of driver.

I see, the spec needs to be more clear then.

> 
> For encoders there is already V4L2_CID_MPEG_VIDEO_HEADER_MODE [1]
> which allows for V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE or
> V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. FFmpeg selects
> _SEPARATE by default [2], whilst the default is normally
> _JOINED_WITH_1ST_FRAME.

I did miss the addition of this API, thanks for the reminder. The problem right
now is that things are being added with the needed cross-reference.

> 
> It does raise the question as to whether all decoders will support
> header byte only buffers, and does there need to be a capabilities
> flag to denote that it is supported.
> And should encoders in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE mode set
> it on the headers only buffers?

What about:
- Document better that in absence of V4L2_CID_MPEG_VIDEO_HEADER_MODE in a
driver, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME is to be assumed
- Document that in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE, driver should signal
the header buffer with the newly added buffer flag (I would suggest
V4L2_BUF_FLAG_HEADERS_ONLY)
- Document for each support CODECs if V4L2_CID_MPEG_VIDEO_HEADER_MODE can be
supported, and the expected headers and the packing of these 
- Document that decoders that didn't implement V4L2_CID_MPEG_VIDEO_HEADER_MODE
should be assumed to only support 
V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME decoding (even though it
allowed to support any alignment, even random).
- Cross-reference V4L2_CID_MPEG_VIDEO_HEADER_MODE into the new flag
documentation (whatever this new flag will be called).

> 
> A number of undefined elements of how this should be implemented/used :-(
> 
>   Dave
> 
> [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html
> [2] https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c#L196

> 
> > >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> > > 
> > >        - ``V4L2_BUF_FLAG_REQUEST_FD``
> > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > index 5311ac4fde35..8708ef257710 100644
> > > --- a/include/uapi/linux/videodev2.h
> > > +++ b/include/uapi/linux/videodev2.h
> > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
> > >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
> > >  /* mem2mem encoder/decoder */
> > >  #define V4L2_BUF_FLAG_LAST                   0x00100000
> > > +/* Buffer only contains codec header */
> > > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
> > >  /* request_fd is valid */
> > >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
> > > 
> > 

Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[kernel test robot]

Hi Ming,

I love your patch! Perhaps something to improve:

[auto build test WARNING on media-tree/master]
[also build test WARNING on linus/master v5.19-rc5 next-20220705]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/intel-lab-lkp/linux/commits/Ming-Qian/media-videobuf2-add-V4L2_BUF_FLAG_CODECCONFIG-flag/20220628-103906
base:   git://linuxtv.org/media_tree.git master
reproduce: make htmldocs

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> Documentation/userspace-api/media/v4l/buffer.rst:614: WARNING: Bullet list ends without a blank line; unexpected unindent.
>> Documentation/userspace-api/media/v4l/buffer.rst:623: WARNING: Block quote ends without a blank line; unexpected unindent.
>> Documentation/userspace-api/media/v4l/buffer.rst:460: WARNING: Error parsing content block for the "flat-table" directive: exactly one bullet list expected.

vim +614 Documentation/userspace-api/media/v4l/buffer.rst

 > 460	
   461	.. flat-table::
   462	    :header-rows:  0
   463	    :stub-columns: 0
   464	    :widths:       65 18 70
   465	
   466	    * .. _`V4L2-BUF-FLAG-MAPPED`:
   467	
   468	      - ``V4L2_BUF_FLAG_MAPPED``
   469	      - 0x00000001
   470	      - The buffer resides in device memory and has been mapped into the
   471		application's address space, see :ref:`mmap` for details.
   472		Drivers set or clear this flag when the
   473		:ref:`VIDIOC_QUERYBUF`,
   474		:ref:`VIDIOC_QBUF` or
   475		:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. Set by the
   476		driver.
   477	    * .. _`V4L2-BUF-FLAG-QUEUED`:
   478	
   479	      - ``V4L2_BUF_FLAG_QUEUED``
   480	      - 0x00000002
   481	      - Internally drivers maintain two buffer queues, an incoming and
   482		outgoing queue. When this flag is set, the buffer is currently on
   483		the incoming queue. It automatically moves to the outgoing queue
   484		after the buffer has been filled (capture devices) or displayed
   485		(output devices). Drivers set or clear this flag when the
   486		``VIDIOC_QUERYBUF`` ioctl is called. After (successful) calling
   487		the ``VIDIOC_QBUF``\ ioctl it is always set and after
   488		``VIDIOC_DQBUF`` always cleared.
   489	    * .. _`V4L2-BUF-FLAG-DONE`:
   490	
   491	      - ``V4L2_BUF_FLAG_DONE``
   492	      - 0x00000004
   493	      - When this flag is set, the buffer is currently on the outgoing
   494		queue, ready to be dequeued from the driver. Drivers set or clear
   495		this flag when the ``VIDIOC_QUERYBUF`` ioctl is called. After
   496		calling the ``VIDIOC_QBUF`` or ``VIDIOC_DQBUF`` it is always
   497		cleared. Of course a buffer cannot be on both queues at the same
   498		time, the ``V4L2_BUF_FLAG_QUEUED`` and ``V4L2_BUF_FLAG_DONE`` flag
   499		are mutually exclusive. They can be both cleared however, then the
   500		buffer is in "dequeued" state, in the application domain so to
   501		say.
   502	    * .. _`V4L2-BUF-FLAG-ERROR`:
   503	
   504	      - ``V4L2_BUF_FLAG_ERROR``
   505	      - 0x00000040
   506	      - When this flag is set, the buffer has been dequeued successfully,
   507		although the data might have been corrupted. This is recoverable,
   508		streaming may continue as normal and the buffer may be reused
   509		normally. Drivers set this flag when the ``VIDIOC_DQBUF`` ioctl is
   510		called.
   511	    * .. _`V4L2-BUF-FLAG-IN-REQUEST`:
   512	
   513	      - ``V4L2_BUF_FLAG_IN_REQUEST``
   514	      - 0x00000080
   515	      - This buffer is part of a request that hasn't been queued yet.
   516	    * .. _`V4L2-BUF-FLAG-KEYFRAME`:
   517	
   518	      - ``V4L2_BUF_FLAG_KEYFRAME``
   519	      - 0x00000008
   520	      - Drivers set or clear this flag when calling the ``VIDIOC_DQBUF``
   521		ioctl. It may be set by video capture devices when the buffer
   522		contains a compressed image which is a key frame (or field), i. e.
   523		can be decompressed on its own. Also known as an I-frame.
   524		Applications can set this bit when ``type`` refers to an output
   525		stream.
   526	    * .. _`V4L2-BUF-FLAG-PFRAME`:
   527	
   528	      - ``V4L2_BUF_FLAG_PFRAME``
   529	      - 0x00000010
   530	      - Similar to ``V4L2_BUF_FLAG_KEYFRAME`` this flags predicted frames
   531		or fields which contain only differences to a previous key frame.
   532		Applications can set this bit when ``type`` refers to an output
   533		stream.
   534	    * .. _`V4L2-BUF-FLAG-BFRAME`:
   535	
   536	      - ``V4L2_BUF_FLAG_BFRAME``
   537	      - 0x00000020
   538	      - Similar to ``V4L2_BUF_FLAG_KEYFRAME`` this flags a bi-directional
   539		predicted frame or field which contains only the differences
   540		between the current frame and both the preceding and following key
   541		frames to specify its content. Applications can set this bit when
   542		``type`` refers to an output stream.
   543	    * .. _`V4L2-BUF-FLAG-TIMECODE`:
   544	
   545	      - ``V4L2_BUF_FLAG_TIMECODE``
   546	      - 0x00000100
   547	      - The ``timecode`` field is valid. Drivers set or clear this flag
   548		when the ``VIDIOC_DQBUF`` ioctl is called. Applications can set
   549		this bit and the corresponding ``timecode`` structure when
   550		``type`` refers to an output stream.
   551	    * .. _`V4L2-BUF-FLAG-PREPARED`:
   552	
   553	      - ``V4L2_BUF_FLAG_PREPARED``
   554	      - 0x00000400
   555	      - The buffer has been prepared for I/O and can be queued by the
   556		application. Drivers set or clear this flag when the
   557		:ref:`VIDIOC_QUERYBUF`,
   558		:ref:`VIDIOC_PREPARE_BUF <VIDIOC_QBUF>`,
   559		:ref:`VIDIOC_QBUF` or
   560		:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called.
   561	    * .. _`V4L2-BUF-FLAG-NO-CACHE-INVALIDATE`:
   562	
   563	      - ``V4L2_BUF_FLAG_NO_CACHE_INVALIDATE``
   564	      - 0x00000800
   565	      - Caches do not have to be invalidated for this buffer. Typically
   566		applications shall use this flag if the data captured in the
   567		buffer is not going to be touched by the CPU, instead the buffer
   568		will, probably, be passed on to a DMA-capable hardware unit for
   569		further processing or output. This flag is ignored unless the
   570		queue is used for :ref:`memory mapping <mmap>` streaming I/O and
   571		reports :ref:`V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS
   572		<V4L2-BUF-CAP-SUPPORTS-MMAP-CACHE-HINTS>` capability.
   573	    * .. _`V4L2-BUF-FLAG-NO-CACHE-CLEAN`:
   574	
   575	      - ``V4L2_BUF_FLAG_NO_CACHE_CLEAN``
   576	      - 0x00001000
   577	      - Caches do not have to be cleaned for this buffer. Typically
   578		applications shall use this flag for output buffers if the data in
   579		this buffer has not been created by the CPU but by some
   580		DMA-capable unit, in which case caches have not been used. This flag
   581		is ignored unless the queue is used for :ref:`memory mapping <mmap>`
   582		streaming I/O and reports :ref:`V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS
   583		<V4L2-BUF-CAP-SUPPORTS-MMAP-CACHE-HINTS>` capability.
   584	    * .. _`V4L2-BUF-FLAG-M2M-HOLD-CAPTURE-BUF`:
   585	
   586	      - ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF``
   587	      - 0x00000200
   588	      - Only valid if struct :c:type:`v4l2_requestbuffers` flag ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` is
   589		set. It is typically used with stateless decoders where multiple
   590		output buffers each decode to a slice of the decoded frame.
   591		Applications can set this flag when queueing the output buffer
   592		to prevent the driver from dequeueing the capture buffer after
   593		the output buffer has been decoded (i.e. the capture buffer is
   594		'held'). If the timestamp of this output buffer differs from that
   595		of the previous output buffer, then that indicates the start of a
   596		new frame and the previously held capture buffer is dequeued.
   597	    * .. _`V4L2-BUF-FLAG-LAST`:
   598	
   599	      - ``V4L2_BUF_FLAG_LAST``
   600	      - 0x00100000
   601	      - Last buffer produced by the hardware. mem2mem codec drivers set
   602		this flag on the capture queue for the last buffer when the
   603		:ref:`VIDIOC_QUERYBUF` or
   604		:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. Due to
   605		hardware limitations, the last buffer may be empty. In this case
   606		the driver will set the ``bytesused`` field to 0, regardless of
   607		the format. Any subsequent call to the
   608		:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
   609		but return an ``EPIPE`` error code.
   610	    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
   611	
   612	      - ``V4L2_BUF_FLAG_CODECCONFIG``
   613	      - 0x00200000
 > 614	      - This flag may be set when the buffer only contains codec config
   615	    header, but does not contain any frame data. Usually the codec config
   616	    header is merged to the next idr frame, with the flag
   617	    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
   618	    split the header and queue it separately.
   619	    * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
   620	
   621	      - ``V4L2_BUF_FLAG_REQUEST_FD``
   622	      - 0x00800000
 > 623	      - The ``request_fd`` field contains a valid file descriptor.
   624	    * .. _`V4L2-BUF-FLAG-TIMESTAMP-MASK`:
   625	
   626	      - ``V4L2_BUF_FLAG_TIMESTAMP_MASK``
   627	      - 0x0000e000
   628	      - Mask for timestamp types below. To test the timestamp type, mask
   629		out bits not belonging to timestamp type by performing a logical
   630		and operation with buffer flags and timestamp mask.
   631	    * .. _`V4L2-BUF-FLAG-TIMESTAMP-UNKNOWN`:
   632	
   633	      - ``V4L2_BUF_FLAG_TIMESTAMP_UNKNOWN``
   634	      - 0x00000000
   635	      - Unknown timestamp type. This type is used by drivers before Linux
   636		3.9 and may be either monotonic (see below) or realtime (wall
   637		clock). Monotonic clock has been favoured in embedded systems
   638		whereas most of the drivers use the realtime clock. Either kinds
   639		of timestamps are available in user space via
   640		:c:func:`clock_gettime` using clock IDs ``CLOCK_MONOTONIC``
   641		and ``CLOCK_REALTIME``, respectively.
   642	    * .. _`V4L2-BUF-FLAG-TIMESTAMP-MONOTONIC`:
   643	
   644	      - ``V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC``
   645	      - 0x00002000
   646	      - The buffer timestamp has been taken from the ``CLOCK_MONOTONIC``
   647		clock. To access the same clock outside V4L2, use
   648		:c:func:`clock_gettime`.
   649	    * .. _`V4L2-BUF-FLAG-TIMESTAMP-COPY`:
   650	
   651	      - ``V4L2_BUF_FLAG_TIMESTAMP_COPY``
   652	      - 0x00004000
   653	      - The CAPTURE buffer timestamp has been taken from the corresponding
   654		OUTPUT buffer. This flag applies only to mem2mem devices.
   655	    * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-MASK`:
   656	
   657	      - ``V4L2_BUF_FLAG_TSTAMP_SRC_MASK``
   658	      - 0x00070000
   659	      - Mask for timestamp sources below. The timestamp source defines the
   660		point of time the timestamp is taken in relation to the frame.
   661		Logical 'and' operation between the ``flags`` field and
   662		``V4L2_BUF_FLAG_TSTAMP_SRC_MASK`` produces the value of the
   663		timestamp source. Applications must set the timestamp source when
   664		``type`` refers to an output stream and
   665		``V4L2_BUF_FLAG_TIMESTAMP_COPY`` is set.
   666	    * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-EOF`:
   667	
   668	      - ``V4L2_BUF_FLAG_TSTAMP_SRC_EOF``
   669	      - 0x00000000
   670	      - End Of Frame. The buffer timestamp has been taken when the last
   671		pixel of the frame has been received or the last pixel of the
   672		frame has been transmitted. In practice, software generated
   673		timestamps will typically be read from the clock a small amount of
   674		time after the last pixel has been received or transmitten,
   675		depending on the system and other activity in it.
   676	    * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-SOE`:
   677	
   678	      - ``V4L2_BUF_FLAG_TSTAMP_SRC_SOE``
   679	      - 0x00010000
   680	      - Start Of Exposure. The buffer timestamp has been taken when the
   681		exposure of the frame has begun. This is only valid for the
   682		``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
   683	

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[kernel test robot]

Hi Ming,

I love your patch! Perhaps something to improve:

[auto build test WARNING on media-tree/master]
[also build test WARNING on linus/master v5.19-rc5 next-20220705]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/intel-lab-lkp/linux/commits/Ming-Qian/media-videobuf2-add-V4L2_BUF_FLAG_CODECCONFIG-flag/20220628-103906
base:   git://linuxtv.org/media_tree.git master
reproduce: make htmldocs

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> Documentation/userspace-api/media/v4l/buffer.rst:614: WARNING: Bullet list ends without a blank line; unexpected unindent.
>> Documentation/userspace-api/media/v4l/buffer.rst:623: WARNING: Block quote ends without a blank line; unexpected unindent.
>> Documentation/userspace-api/media/v4l/buffer.rst:460: WARNING: Error parsing content block for the "flat-table" directive: exactly one bullet list expected.

vim +614 Documentation/userspace-api/media/v4l/buffer.rst

 > 460	
   461	.. flat-table::
   462	    :header-rows:  0
   463	    :stub-columns: 0
   464	    :widths:       65 18 70
   465	
   466	    * .. _`V4L2-BUF-FLAG-MAPPED`:
   467	
   468	      - ``V4L2_BUF_FLAG_MAPPED``
   469	      - 0x00000001
   470	      - The buffer resides in device memory and has been mapped into the
   471		application's address space, see :ref:`mmap` for details.
   472		Drivers set or clear this flag when the
   473		:ref:`VIDIOC_QUERYBUF`,
   474		:ref:`VIDIOC_QBUF` or
   475		:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. Set by the
   476		driver.
   477	    * .. _`V4L2-BUF-FLAG-QUEUED`:
   478	
   479	      - ``V4L2_BUF_FLAG_QUEUED``
   480	      - 0x00000002
   481	      - Internally drivers maintain two buffer queues, an incoming and
   482		outgoing queue. When this flag is set, the buffer is currently on
   483		the incoming queue. It automatically moves to the outgoing queue
   484		after the buffer has been filled (capture devices) or displayed
   485		(output devices). Drivers set or clear this flag when the
   486		``VIDIOC_QUERYBUF`` ioctl is called. After (successful) calling
   487		the ``VIDIOC_QBUF``\ ioctl it is always set and after
   488		``VIDIOC_DQBUF`` always cleared.
   489	    * .. _`V4L2-BUF-FLAG-DONE`:
   490	
   491	      - ``V4L2_BUF_FLAG_DONE``
   492	      - 0x00000004
   493	      - When this flag is set, the buffer is currently on the outgoing
   494		queue, ready to be dequeued from the driver. Drivers set or clear
   495		this flag when the ``VIDIOC_QUERYBUF`` ioctl is called. After
   496		calling the ``VIDIOC_QBUF`` or ``VIDIOC_DQBUF`` it is always
   497		cleared. Of course a buffer cannot be on both queues at the same
   498		time, the ``V4L2_BUF_FLAG_QUEUED`` and ``V4L2_BUF_FLAG_DONE`` flag
   499		are mutually exclusive. They can be both cleared however, then the
   500		buffer is in "dequeued" state, in the application domain so to
   501		say.
   502	    * .. _`V4L2-BUF-FLAG-ERROR`:
   503	
   504	      - ``V4L2_BUF_FLAG_ERROR``
   505	      - 0x00000040
   506	      - When this flag is set, the buffer has been dequeued successfully,
   507		although the data might have been corrupted. This is recoverable,
   508		streaming may continue as normal and the buffer may be reused
   509		normally. Drivers set this flag when the ``VIDIOC_DQBUF`` ioctl is
   510		called.
   511	    * .. _`V4L2-BUF-FLAG-IN-REQUEST`:
   512	
   513	      - ``V4L2_BUF_FLAG_IN_REQUEST``
   514	      - 0x00000080
   515	      - This buffer is part of a request that hasn't been queued yet.
   516	    * .. _`V4L2-BUF-FLAG-KEYFRAME`:
   517	
   518	      - ``V4L2_BUF_FLAG_KEYFRAME``
   519	      - 0x00000008
   520	      - Drivers set or clear this flag when calling the ``VIDIOC_DQBUF``
   521		ioctl. It may be set by video capture devices when the buffer
   522		contains a compressed image which is a key frame (or field), i. e.
   523		can be decompressed on its own. Also known as an I-frame.
   524		Applications can set this bit when ``type`` refers to an output
   525		stream.
   526	    * .. _`V4L2-BUF-FLAG-PFRAME`:
   527	
   528	      - ``V4L2_BUF_FLAG_PFRAME``
   529	      - 0x00000010
   530	      - Similar to ``V4L2_BUF_FLAG_KEYFRAME`` this flags predicted frames
   531		or fields which contain only differences to a previous key frame.
   532		Applications can set this bit when ``type`` refers to an output
   533		stream.
   534	    * .. _`V4L2-BUF-FLAG-BFRAME`:
   535	
   536	      - ``V4L2_BUF_FLAG_BFRAME``
   537	      - 0x00000020
   538	      - Similar to ``V4L2_BUF_FLAG_KEYFRAME`` this flags a bi-directional
   539		predicted frame or field which contains only the differences
   540		between the current frame and both the preceding and following key
   541		frames to specify its content. Applications can set this bit when
   542		``type`` refers to an output stream.
   543	    * .. _`V4L2-BUF-FLAG-TIMECODE`:
   544	
   545	      - ``V4L2_BUF_FLAG_TIMECODE``
   546	      - 0x00000100
   547	      - The ``timecode`` field is valid. Drivers set or clear this flag
   548		when the ``VIDIOC_DQBUF`` ioctl is called. Applications can set
   549		this bit and the corresponding ``timecode`` structure when
   550		``type`` refers to an output stream.
   551	    * .. _`V4L2-BUF-FLAG-PREPARED`:
   552	
   553	      - ``V4L2_BUF_FLAG_PREPARED``
   554	      - 0x00000400
   555	      - The buffer has been prepared for I/O and can be queued by the
   556		application. Drivers set or clear this flag when the
   557		:ref:`VIDIOC_QUERYBUF`,
   558		:ref:`VIDIOC_PREPARE_BUF <VIDIOC_QBUF>`,
   559		:ref:`VIDIOC_QBUF` or
   560		:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called.
   561	    * .. _`V4L2-BUF-FLAG-NO-CACHE-INVALIDATE`:
   562	
   563	      - ``V4L2_BUF_FLAG_NO_CACHE_INVALIDATE``
   564	      - 0x00000800
   565	      - Caches do not have to be invalidated for this buffer. Typically
   566		applications shall use this flag if the data captured in the
   567		buffer is not going to be touched by the CPU, instead the buffer
   568		will, probably, be passed on to a DMA-capable hardware unit for
   569		further processing or output. This flag is ignored unless the
   570		queue is used for :ref:`memory mapping <mmap>` streaming I/O and
   571		reports :ref:`V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS
   572		<V4L2-BUF-CAP-SUPPORTS-MMAP-CACHE-HINTS>` capability.
   573	    * .. _`V4L2-BUF-FLAG-NO-CACHE-CLEAN`:
   574	
   575	      - ``V4L2_BUF_FLAG_NO_CACHE_CLEAN``
   576	      - 0x00001000
   577	      - Caches do not have to be cleaned for this buffer. Typically
   578		applications shall use this flag for output buffers if the data in
   579		this buffer has not been created by the CPU but by some
   580		DMA-capable unit, in which case caches have not been used. This flag
   581		is ignored unless the queue is used for :ref:`memory mapping <mmap>`
   582		streaming I/O and reports :ref:`V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS
   583		<V4L2-BUF-CAP-SUPPORTS-MMAP-CACHE-HINTS>` capability.
   584	    * .. _`V4L2-BUF-FLAG-M2M-HOLD-CAPTURE-BUF`:
   585	
   586	      - ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF``
   587	      - 0x00000200
   588	      - Only valid if struct :c:type:`v4l2_requestbuffers` flag ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` is
   589		set. It is typically used with stateless decoders where multiple
   590		output buffers each decode to a slice of the decoded frame.
   591		Applications can set this flag when queueing the output buffer
   592		to prevent the driver from dequeueing the capture buffer after
   593		the output buffer has been decoded (i.e. the capture buffer is
   594		'held'). If the timestamp of this output buffer differs from that
   595		of the previous output buffer, then that indicates the start of a
   596		new frame and the previously held capture buffer is dequeued.
   597	    * .. _`V4L2-BUF-FLAG-LAST`:
   598	
   599	      - ``V4L2_BUF_FLAG_LAST``
   600	      - 0x00100000
   601	      - Last buffer produced by the hardware. mem2mem codec drivers set
   602		this flag on the capture queue for the last buffer when the
   603		:ref:`VIDIOC_QUERYBUF` or
   604		:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. Due to
   605		hardware limitations, the last buffer may be empty. In this case
   606		the driver will set the ``bytesused`` field to 0, regardless of
   607		the format. Any subsequent call to the
   608		:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
   609		but return an ``EPIPE`` error code.
   610	    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
   611	
   612	      - ``V4L2_BUF_FLAG_CODECCONFIG``
   613	      - 0x00200000
 > 614	      - This flag may be set when the buffer only contains codec config
   615	    header, but does not contain any frame data. Usually the codec config
   616	    header is merged to the next idr frame, with the flag
   617	    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
   618	    split the header and queue it separately.
   619	    * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
   620	
   621	      - ``V4L2_BUF_FLAG_REQUEST_FD``
   622	      - 0x00800000
 > 623	      - The ``request_fd`` field contains a valid file descriptor.
   624	    * .. _`V4L2-BUF-FLAG-TIMESTAMP-MASK`:
   625	
   626	      - ``V4L2_BUF_FLAG_TIMESTAMP_MASK``
   627	      - 0x0000e000
   628	      - Mask for timestamp types below. To test the timestamp type, mask
   629		out bits not belonging to timestamp type by performing a logical
   630		and operation with buffer flags and timestamp mask.
   631	    * .. _`V4L2-BUF-FLAG-TIMESTAMP-UNKNOWN`:
   632	
   633	      - ``V4L2_BUF_FLAG_TIMESTAMP_UNKNOWN``
   634	      - 0x00000000
   635	      - Unknown timestamp type. This type is used by drivers before Linux
   636		3.9 and may be either monotonic (see below) or realtime (wall
   637		clock). Monotonic clock has been favoured in embedded systems
   638		whereas most of the drivers use the realtime clock. Either kinds
   639		of timestamps are available in user space via
   640		:c:func:`clock_gettime` using clock IDs ``CLOCK_MONOTONIC``
   641		and ``CLOCK_REALTIME``, respectively.
   642	    * .. _`V4L2-BUF-FLAG-TIMESTAMP-MONOTONIC`:
   643	
   644	      - ``V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC``
   645	      - 0x00002000
   646	      - The buffer timestamp has been taken from the ``CLOCK_MONOTONIC``
   647		clock. To access the same clock outside V4L2, use
   648		:c:func:`clock_gettime`.
   649	    * .. _`V4L2-BUF-FLAG-TIMESTAMP-COPY`:
   650	
   651	      - ``V4L2_BUF_FLAG_TIMESTAMP_COPY``
   652	      - 0x00004000
   653	      - The CAPTURE buffer timestamp has been taken from the corresponding
   654		OUTPUT buffer. This flag applies only to mem2mem devices.
   655	    * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-MASK`:
   656	
   657	      - ``V4L2_BUF_FLAG_TSTAMP_SRC_MASK``
   658	      - 0x00070000
   659	      - Mask for timestamp sources below. The timestamp source defines the
   660		point of time the timestamp is taken in relation to the frame.
   661		Logical 'and' operation between the ``flags`` field and
   662		``V4L2_BUF_FLAG_TSTAMP_SRC_MASK`` produces the value of the
   663		timestamp source. Applications must set the timestamp source when
   664		``type`` refers to an output stream and
   665		``V4L2_BUF_FLAG_TIMESTAMP_COPY`` is set.
   666	    * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-EOF`:
   667	
   668	      - ``V4L2_BUF_FLAG_TSTAMP_SRC_EOF``
   669	      - 0x00000000
   670	      - End Of Frame. The buffer timestamp has been taken when the last
   671		pixel of the frame has been received or the last pixel of the
   672		frame has been transmitted. In practice, software generated
   673		timestamps will typically be read from the clock a small amount of
   674		time after the last pixel has been received or transmitten,
   675		depending on the system and other activity in it.
   676	    * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-SOE`:
   677	
   678	      - ``V4L2_BUF_FLAG_TSTAMP_SRC_SOE``
   679	      - 0x00010000
   680	      - Start Of Exposure. The buffer timestamp has been taken when the
   681		exposure of the frame has begun. This is only valid for the
   682		``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
   683	

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel