Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Tomasz Figa <tfiga@chromium.org>]

+Hu, Jerry W +Mani, Rajmohan +Sakari Ailus

FYI
On Fri, May 18, 2018 at 4:07 AM Mauro Carvalho Chehab <
mchehab+samsung@kernel.org> wrote:

> Hi all,

> The goal of this e-mail is to schedule a meeting in order to discuss
> improvements at the media subsystem in order to support complex camera
> hardware by usual apps.

> The main focus here is to allow supporting devices with MC-based
> hardware connected to a camera.

> In short, my proposal is to meet with the interested parties on solving
> this issue during the Open Source Summit in Japan, e. g. between
> June, 19-22, in Tokyo.

> I'd like to know who is interested on joining us for such meeting,
> and to hear a proposal of themes for discussions.

> I'm enclosing a detailed description of the problem, in order to
> allow the interested parties to be at the same page.

> Regards,
> Mauro

> ---


> 1. Introduction
> ===============

> 1.1 V4L2 Kernel aspects
> -----------------------

> The media subsystem supports two types of devices:

> - "traditional" media hardware, supported via V4L2 API. On such hardware,
>    opening a single device node (usually /dev/video0) is enough to control
>    the entire device. We call it as devnode-based devices.

> - Media-controller based devices. On those devices, there are several
>    /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus a media
>    controller device node (usually /dev/media0).
>    We call it as mc-based devices. Controlling the hardware require
>    opening the media device (/dev/media0), setup the pipeline and adjust
>    the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
>    by /dev/video?.

> All "standard" media applications, including open source ones (Camorama,
> Cheese, Xawtv, Firefox, Chromium, ...) and closed source ones (Skype,
> Chrome, ...) supports devnode-based devices.

> Support for mc-based devices currently require an specialized application
> in order to prepare the device for its usage (setup pipelines, adjust
> hardware controls, etc). Once pipeline is set, the streaming goes via
> /dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
> be opened, in order to implement algorithms designed to make video quality
> reasonable. On such devices, it is not uncommon that the device used by
the
> application to be a random number (on OMAP3 driver, typically, is either
> /dev/video4 or /dev/video6).

> One example of such hardware is at the OMAP3-based hardware:


http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png

> On the picture, there's a graph with the hardware blocks in blue/dark/blue
> and the corresponding devnode interfaces in yellow.

> The mc-based approach was taken when support for Nokia N9/N900 cameras
> was added (with has OMAP3 SoC). It is required because the camera hardware
> on SoC comes with a media processor (ISP), with does a lot more than just
> capturing, allowing complex algorithms to enhance image quality in
runtime.
> Those algorithms are known as 3A - an acronym for 3 other acronyms:

>          - AE (Auto Exposure);
>          - AF (Auto Focus);
>          - AWB (Auto White Balance).

> Setting a camera with such ISPs are harder because the pipelines to be
> set actually depends the requirements for those 3A algorithms to run.
> Also, usually, the 3A algorithms use some chipset-specific userspace API,
> that exports some image properties, calculated by the ISP, to speed up
> the convergence of those algorithms.

> Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> as binary only blobs, although there are a few OSS implementations.

> 1.2 V4L2 userspace aspects
> --------------------------

> Back when USB cameras were introduced, the hardware were really simple:
> they had a CCD camera sensor and a chip that bridges the data though
> USB. CCD camera sensors typically provide data using a bayer format,
> but they usually have their own proprietary ways to pack the data,
> in order to reduce the USB bandwidth (original cameras were USB 1.1).

> So, V4L2 has a myriad of different formats, in order to match each
> CCD camera sensor format. At the end of the day, applications were
> able to use only a subset of the available hardware, since they need
> to come with format converters for all formats the developer uses
> (usually a very small subset of the available ones).

> To end with this mess, an userspace library was written, called libv4l.
> It supports all those proprietary formats. So, applications can use
> a RGB or YUV format, without needing to concern about conversions.

> The way it works is by adding wrappers to system calls: open, close,
> ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> at the source code of the apps, all it was needed is to prepend the
> existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.

> All open source apps we know now supports libv4l. On a few (like
> gstreamer), support for it is optional.

> In order to support closed source, another wrapper was added, allowing
> to call any closed source application to use it, by using LD_PRELOAD.
> For example, using skype with it is as simple as calling it with:

>          $ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so /usr/bin/skypeforlinux

> 2. Current problems
> ===================

> 2.1 Libv4l can slow image handling
> ----------------------------------

> Nowadays, almost all new "simple" cameras are connected via USB using
> the UVC class (USB Video Class). UVC standardized the allowed formats,
> and most apps just implement them. The UVC hardware is more complex,
> having format converters inside it. So, for most usages, format
> conversion isn't needed anymore.

> The need of doing format conversion in software makes libv4l slow,
> requiring lots of CPU usage in order to convert a 4K or 8K format,
> being even worse with 3D cameras.

> Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> DMA_BUFFER currently doesn't work with libv4l.

> Right now, gstreamer defaults to not enable libv4l2, mainly due to
> those performance issues.


> 2.2 Modern hardware is starting to come with "complex" camera ISP
> -----------------------------------------------------------------

> While mc-based devices were limited to SoC, it was easy to
> "delegate" the task of talking with the hardware to the
> embedded hardware designers.

> However, this is changing. Dell Latitude 5285 laptop is a standard
> PC with an i3-core, i5-core or i7-core CPU, with comes with the
> Intel IMU3 ISP hardware[1]

> [1] https://www.spinics.net/lists/linux-usb/msg167478.html

> There, instead of an USB camera, the hardware is equipped with a
> MC-based ISP, connected to its camera. Currently, despite having
> a Kernel driver for it, the camera doesn't work with any
> userspace application.

> I'm also aware of other projects that are considering the usage of
> mc-based devices for non-dedicated hardware.

> 3. How to solve it?
> ===================

> That's the main focus of the meeting :-)

>  From a previous discussion I had with media sub-maintainers, there are
> at least two actions that seem required. I'm listing them below as
> an starting point for the discussions, but we can eventually come up
> with some different approach after the meeting.

> 3.1 libv4l2 support for mc-based hardware
> =========================================

> In order to support those hardware, we'll need to do some redesign
> mainly at libv4l2[2].

> The idea is to work on a new API for libv4l2 that will allow to
> split the format conversion on a separate part of it, add support
> for DMA Buffer and come up with a way for the library to work
> transparently with both devnode-based and mc-based hardware.

> That envolves adding capacity at libv4l to setup hardware pipelines
> and to propagate controls among their sub-devices. Eventually, part
> of it will be done in Kernel.

> That should give performance increase at the library and would allow
> gstreamer to use it by default, without compromising performance.

> [2] I don't discard that some Kernel changes could also be part of the
> solution, like, for example, doing control propagation along the pipeline
> on simple use case scenarios.

> 3.2 libv4l2 support for 3A algorithms
> =====================================

> The 3A algorithm handing is highly dependent on the hardware. The
> idea here is to allow libv4l to have a set of 3A algorithms that
> will be specific to certain mc-based hardware. Ideally, this should
> be added in a way that it will allow external closed-source
> algorithms to run as well.

> Thanks,
> Mauro