[ANN] Report of Media Summit: V4L2 Future Work

[ANN] Report of Media Summit: V4L2 Future Work

[Hans Verkuil]

Media Summit: V4L2 Future Work
October 30, 2019 - Lyon, France

Many thanks to the Linux Foundation for hosting this meeting. Much appreciated!

Please reply to this report with any comments/corrections. Especially if I missed
any action items or attendees!


Original announcement:

https://lore.kernel.org/linux-media/2ef17f32-5957-7b52-fea3-19913ec1c7b3@xs4all.nl/T/


Attendees:

Michael Ashton <mpa@fb.com> (Facebook/Oculus)
Boris Brezillon <boris.brezillon@collabora.com>
Alexandre Courbot <acourbot@chromium.org> (Google, Chrome OS)
Nicolas Dufresne <nicolas@collabora.com>
Tomasz Figa <tfiga@chromium.org> (Google, Chrome OS)
Ezequiel Garcia <ezequiel@collabora.com>
Daniel Gomez <daniel@qtec.com>
Peter Griffin <peter.griffin@linaro.org>
Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
Eugen Hristev <eugen.hristev@microchip.com>
Shuah Khan <skhan@linuxfoundation.org>
Helen Koike <helen.koike@collabora.com>
Jacopo Mondi <jacopo@jmondi.org>
Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Niklas Söderlund <niklas.soderlund@ragnatech.se>
Stanimir Varbanov <stanimir.varbanov@linaro.org>
Hans Verkuil <hverkuil@xs4all.nl> (Cisco Systems Norway)


V4L2 Future Work
----------------

This discussion was intentionally not very structured. The goal was
to see what the main pain points are and how to fix those.

Boris Brezillon made an RFC v3 with proposed new ioctls:

https://patchwork.linuxtv.org/cover/59345/


The main blockers for drivers/userspace today with our current
API are:

- Missing format (DRM) modifier support
- one dmabuf fd with per-plane offsets (mplane API today requires
  separate dmabuf fds for each plane)
- userspace cannot specify offsets of planes

Note that the DRM modifiers are per frame and not per plane.
Typically the modifiers describe tiled and/or compressed formats.
Compressed formats are typically lossless HW compression schemes
(often proprietary) used to reduce memory bandwidth. Compressed
formats are considered opaque formats by userspace.

EGL has a dma-buf extension that supports modifiers (see
EXT_image_dma_buf_import_modifiers). Also see Wayland protocol:
https://gitlab.freedesktop.org/wayland/wayland-protocols/blob/master/unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml


While not strictly a blocker feature, it is desirable to get
rid of the distinction between the single and multiplane APIs
and always describe formats as multiplane. The advantage of
the multiplane API is that bytesperline and sizeimage are
described per plane, which is much easier to understand, and it
does not require userspace to calculate bytesperline for the chroma
planes.


The RFCv3 replaces v4l2_format with a new struct, but the consensus
was that it is much better to create a new ioctl just to replace the
v4l2_pix_format struct and leave the other structs in the v4l2_format
union alone. Two reasons: there is no need to change anything for the
other VBI etc. buffer types, it's only v4l2_pix_format that no longer
scales; and it is easy to extend a struct v4l2_pix_format_ext in the
future. You can't really do that if it is in a union.


If a new ENUM_FMT_EXT ioctl is created, then let it return all
pixelformats + modifiers combinations rather than having to enumerate
over all combinations. This reduces the number of ioctl calls that
userspace has to do.

To indicate if a format supports supports putting all planes in one
buffer, or supports putting each plane in a separate buffer, new format
flags are needed. Userspace needs to know this.

The vb2 core should check if all planes belong to the same buffer or not,
and validate it based on the driver requirements. I.e. in the case of
dmabuf import it should check that all dmabuf FDs refer to the same
memory and that the per-plane offsets are valid.

The current RFCv3 drops the v4l2_timecode field. Nicolas mentioned that
while this is not used by any mainline drivers (except virtual drivers)
this is in fact used by out-of-tree drivers. While they don't use the
same struct, they do return this information since it is used by broadcast
systems. So this field needs to return.


In the past there was a discussion of creating new fourcc codes to
describe pixelformats for both DRM and V4L2. This didn't work out, but
it was suggested that we just document the mapping between DRM and V4L2
pixelformats as a reference to end-users. We can maintain that as part of
our V4L2 documentation.


The suggestion was made to add a channel number to the new format struct
and to the streaming I/O ioctls to indicate support for streaming
multiple channels. This would allow the M2M framework to support multiple
capture and output streams per instance, something that is not possible
today. One use-case would be support for H.264 Scalable Video Coding.
Another is support for compositors.

This would need more research, but the new API should prepare for this.


A known outstanding issue is syncing buffers between the CPU and the
device. Drivers that want to update the buffer before returning it
to userspace do not make the right dmabuf calls. A patch series fixing
this was never finished. Google offered to pick this up and give it
another try.


The general consensus was that these proposed new ioctls only address
part of the problems. The complexity of modern camera pipelines creates
too many video and v4l-subdev devices, making it very hard for userspace
to control all of this.

Ideally it should be possible to control a camera ISP through a single
/dev/mediaX only, rather than though a myriad of other devices. This
is however a huge change, requiring a lot of work. Quite possible a
completely new API would be required for this. There is real interest
in solving this problem, which includes assigning resources to work
on this.

Google will collect some requirements and report on that later this year.
Based on that we might decide on a three day brainstorm session in
Brussels before or after FOSDEM (Feb 1-2, 2020).


Action Items
------------

Nicolas Dufresne:

- provide more info about timecode use in userspace for Hans Verkuil
  to verify if struct v4l2_timecode provides sufficient information.

Tomasz Figa:

- Check if we can drop support for using two memory banks for luma and
  chroma planes? It's only used by the s5p architecture, and dropping
  support for that would simplify some of the code. Note that s5p doesn't
  support dmabuf.

- Continue work on my old patch series dealing with dma_buf_begin/end_cpu_access():

  https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=vb2-cpu-access

  This series converts most drivers to use these dmabuf functions.

  See also this thread: https://patchwork.kernel.org/patch/7159871/
  As mentioned in my last reply, the outstanding drivers that are not
  converted are netup_unidvb, coda, exynos4-is, rcar_jpu and au0828:
  as far as I remember I did not know how to convert coda and exynos4-is
  and never found the time to figure that out. The others are new and need
  to be looked at again. There are probably more drivers that need work
  since my patch series is from September 2015.

- Collect API requirements for a camera ISP API and report on that later
  this year.

Boris Brezillon:

- Post an RFCv4 that takes more of these issues into accounts. Research
  multistream support a bit more to see what would be involved adding support
  for that.

Re: [ANN] Report of Media Summit: V4L2 Future Work

[Sakari Ailus]

Hi Hans,

On Sat, Nov 02, 2019 at 03:06:07PM +0100, Hans Verkuil wrote:
> - Continue work on my old patch series dealing with dma_buf_begin/end_cpu_access():
> 
>   https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=vb2-cpu-access
> 
>   This series converts most drivers to use these dmabuf functions.
> 
>   See also this thread: https://patchwork.kernel.org/patch/7159871/
>   As mentioned in my last reply, the outstanding drivers that are not
>   converted are netup_unidvb, coda, exynos4-is, rcar_jpu and au0828:
>   as far as I remember I did not know how to convert coda and exynos4-is
>   and never found the time to figure that out. The others are new and need
>   to be looked at again. There are probably more drivers that need work
>   since my patch series is from September 2015.

I've worked on the set since; it's here:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=vb2-dc-noncoherent>

It does address other issues as well. There are likely topics that will
need to be discussed. I remember I got somewhat stuck converting drivers
(coda?) that accessed the buffer data whilst the hardware was operating on
it.

-- 
Regards,

Sakari Ailus

Re: [ANN] Report of Media Summit: V4L2 Future Work

[Nicolas Dufresne]

Le samedi 02 novembre 2019 à 15:06 +0100, Hans Verkuil a écrit :
> Action Items
> ------------
> 
> Nicolas Dufresne:
> 
> - provide more info about timecode use in userspace for Hans Verkuil
>   to verify if struct v4l2_timecode provides sufficient information.

Initial context we have:

   struct v4l2_timecode {
   	__u32	type; /* This is the FPS */
   	__u32	flags;
   	__u8	frames;
   	__u8	seconds;
   	__u8	minutes;
   	__u8	hours;
   	__u8	userbits[4];
   };

GStreamer on it's side exposes:

   struct _GstVideoTimeCode {
     GstVideoTimeCodeConfig config;

     guint hours;
     guint minutes;
     guint seconds;
     guint frames;
     guint field_count;
   };

   struct _GstVideoTimeCodeConfig {
     guint fps_n;
     guint fps_d;
     GstVideoTimeCodeFlags flags;
     GDateTime *latest_daily_jam;
   };
See https://gstreamer.freedesktop.org/documentation/video/gstvideotimecode.html?gi-language=c#GstVideoTimeCode

So the main difference is that GST allow arbitrary framerate, where
V4L2 uses a type for this purpose. I'm not sure how things like 29.97
is really handled in V4L2, is it exposed as 30fps and then handled with
drop frames ? That isn't very clear, but basically it is more of less
the same thing as Gst, and there is likely possible glue to make this
work.

In GStreamer, time code have been implemented when muxing into ISOMP4
format. For H264 and HEVC, they can be encoded into the bitstream and
extracted by the parser. We also have the ability to combine these
timecode with the close caption data. In therm of CAPTURE and OUTPUT,
we have support for decklink PCI card, which are SDI capture/output
card with currently proprietary drivers and userspace on Linux [1].

Real usage started around GStreamer 1.12 (so it's not that old) and was
needed for live television broadcast authoring software. The timecode
are (to the best of my knowledge) mostly used to synchronized the
streams together, as the streaming timestamp (e.g. RTP) might not
correlate with the moment the recording have happened.

It is also used when storing the live recording to disk, so that later
non-linear editor can align the tracks based on these timecode. There
is likely loads of other use case I'm not aware of, it's basically
information upon when this live recording have happened, could be few
seconds in the past, or years for recording.

regards,
Nicolas


[1] https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/blob/master/sys/decklink/linux/DeckLinkAPI.h#L794

Re: [ANN] Report of Media Summit: V4L2 Future Work

[Tomasz Figa]

Hi Hans,

On Sat, Nov 2, 2019 at 11:06 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> Media Summit: V4L2 Future Work
> October 30, 2019 - Lyon, France
>
> Many thanks to the Linux Foundation for hosting this meeting. Much appreciated!
>
> Please reply to this report with any comments/corrections. Especially if I missed
> any action items or attendees!
>

Thanks for the notes. They look good to me.

I'll want to discuss some things about the vb2 begin/end CPU access
changes, but we should probably do it via a different channel.

Best regards,
Tomasz

>
> Original announcement:
>
> https://lore.kernel.org/linux-media/2ef17f32-5957-7b52-fea3-19913ec1c7b3@xs4all.nl/T/
>
>
> Attendees:
>
> Michael Ashton <mpa@fb.com> (Facebook/Oculus)
> Boris Brezillon <boris.brezillon@collabora.com>
> Alexandre Courbot <acourbot@chromium.org> (Google, Chrome OS)
> Nicolas Dufresne <nicolas@collabora.com>
> Tomasz Figa <tfiga@chromium.org> (Google, Chrome OS)
> Ezequiel Garcia <ezequiel@collabora.com>
> Daniel Gomez <daniel@qtec.com>
> Peter Griffin <peter.griffin@linaro.org>
> Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> Eugen Hristev <eugen.hristev@microchip.com>
> Shuah Khan <skhan@linuxfoundation.org>
> Helen Koike <helen.koike@collabora.com>
> Jacopo Mondi <jacopo@jmondi.org>
> Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Niklas Söderlund <niklas.soderlund@ragnatech.se>
> Stanimir Varbanov <stanimir.varbanov@linaro.org>
> Hans Verkuil <hverkuil@xs4all.nl> (Cisco Systems Norway)
>
>
> V4L2 Future Work
> ----------------
>
> This discussion was intentionally not very structured. The goal was
> to see what the main pain points are and how to fix those.
>
> Boris Brezillon made an RFC v3 with proposed new ioctls:
>
> https://patchwork.linuxtv.org/cover/59345/
>
>
> The main blockers for drivers/userspace today with our current
> API are:
>
> - Missing format (DRM) modifier support
> - one dmabuf fd with per-plane offsets (mplane API today requires
>   separate dmabuf fds for each plane)
> - userspace cannot specify offsets of planes
>
> Note that the DRM modifiers are per frame and not per plane.
> Typically the modifiers describe tiled and/or compressed formats.
> Compressed formats are typically lossless HW compression schemes
> (often proprietary) used to reduce memory bandwidth. Compressed
> formats are considered opaque formats by userspace.
>
> EGL has a dma-buf extension that supports modifiers (see
> EXT_image_dma_buf_import_modifiers). Also see Wayland protocol:
> https://gitlab.freedesktop.org/wayland/wayland-protocols/blob/master/unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml
>
>
> While not strictly a blocker feature, it is desirable to get
> rid of the distinction between the single and multiplane APIs
> and always describe formats as multiplane. The advantage of
> the multiplane API is that bytesperline and sizeimage are
> described per plane, which is much easier to understand, and it
> does not require userspace to calculate bytesperline for the chroma
> planes.
>
>
> The RFCv3 replaces v4l2_format with a new struct, but the consensus
> was that it is much better to create a new ioctl just to replace the
> v4l2_pix_format struct and leave the other structs in the v4l2_format
> union alone. Two reasons: there is no need to change anything for the
> other VBI etc. buffer types, it's only v4l2_pix_format that no longer
> scales; and it is easy to extend a struct v4l2_pix_format_ext in the
> future. You can't really do that if it is in a union.
>
>
> If a new ENUM_FMT_EXT ioctl is created, then let it return all
> pixelformats + modifiers combinations rather than having to enumerate
> over all combinations. This reduces the number of ioctl calls that
> userspace has to do.
>
> To indicate if a format supports supports putting all planes in one
> buffer, or supports putting each plane in a separate buffer, new format
> flags are needed. Userspace needs to know this.
>
> The vb2 core should check if all planes belong to the same buffer or not,
> and validate it based on the driver requirements. I.e. in the case of
> dmabuf import it should check that all dmabuf FDs refer to the same
> memory and that the per-plane offsets are valid.
>
> The current RFCv3 drops the v4l2_timecode field. Nicolas mentioned that
> while this is not used by any mainline drivers (except virtual drivers)
> this is in fact used by out-of-tree drivers. While they don't use the
> same struct, they do return this information since it is used by broadcast
> systems. So this field needs to return.
>
>
> In the past there was a discussion of creating new fourcc codes to
> describe pixelformats for both DRM and V4L2. This didn't work out, but
> it was suggested that we just document the mapping between DRM and V4L2
> pixelformats as a reference to end-users. We can maintain that as part of
> our V4L2 documentation.
>
>
> The suggestion was made to add a channel number to the new format struct
> and to the streaming I/O ioctls to indicate support for streaming
> multiple channels. This would allow the M2M framework to support multiple
> capture and output streams per instance, something that is not possible
> today. One use-case would be support for H.264 Scalable Video Coding.
> Another is support for compositors.
>
> This would need more research, but the new API should prepare for this.
>
>
> A known outstanding issue is syncing buffers between the CPU and the
> device. Drivers that want to update the buffer before returning it
> to userspace do not make the right dmabuf calls. A patch series fixing
> this was never finished. Google offered to pick this up and give it
> another try.
>
>
> The general consensus was that these proposed new ioctls only address
> part of the problems. The complexity of modern camera pipelines creates
> too many video and v4l-subdev devices, making it very hard for userspace
> to control all of this.
>
> Ideally it should be possible to control a camera ISP through a single
> /dev/mediaX only, rather than though a myriad of other devices. This
> is however a huge change, requiring a lot of work. Quite possible a
> completely new API would be required for this. There is real interest
> in solving this problem, which includes assigning resources to work
> on this.
>
> Google will collect some requirements and report on that later this year.
> Based on that we might decide on a three day brainstorm session in
> Brussels before or after FOSDEM (Feb 1-2, 2020).
>
>
> Action Items
> ------------
>
> Nicolas Dufresne:
>
> - provide more info about timecode use in userspace for Hans Verkuil
>   to verify if struct v4l2_timecode provides sufficient information.
>
> Tomasz Figa:
>
> - Check if we can drop support for using two memory banks for luma and
>   chroma planes? It's only used by the s5p architecture, and dropping
>   support for that would simplify some of the code. Note that s5p doesn't
>   support dmabuf.
>
> - Continue work on my old patch series dealing with dma_buf_begin/end_cpu_access():
>
>   https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=vb2-cpu-access
>
>   This series converts most drivers to use these dmabuf functions.
>
>   See also this thread: https://patchwork.kernel.org/patch/7159871/
>   As mentioned in my last reply, the outstanding drivers that are not
>   converted are netup_unidvb, coda, exynos4-is, rcar_jpu and au0828:
>   as far as I remember I did not know how to convert coda and exynos4-is
>   and never found the time to figure that out. The others are new and need
>   to be looked at again. There are probably more drivers that need work
>   since my patch series is from September 2015.
>
> - Collect API requirements for a camera ISP API and report on that later
>   this year.
>
> Boris Brezillon:
>
> - Post an RFCv4 that takes more of these issues into accounts. Research
>   multistream support a bit more to see what would be involved adding support
>   for that.