[ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Mauro Carvalho Chehab]

Hi all,

The goal of this e-mail is to schedule a meeting in order to discuss
improvements at the media subsystem in order to support complex camera
hardware by usual apps.

The main focus here is to allow supporting devices with MC-based
hardware connected to a camera.

In short, my proposal is to meet with the interested parties on solving
this issue during the Open Source Summit in Japan, e. g. between
June, 19-22, in Tokyo.

I'd like to know who is interested on joining us for such meeting,
and to hear a proposal of themes for discussions.

I'm enclosing a detailed description of the problem, in order to
allow the interested parties to be at the same page.

Regards,
Mauro

---


1. Introduction
===============

1.1 V4L2 Kernel aspects
-----------------------

The media subsystem supports two types of devices:

- "traditional" media hardware, supported via V4L2 API. On such hardware, 
  opening a single device node (usually /dev/video0) is enough to control
  the entire device. We call it as devnode-based devices.

- Media-controller based devices. On those devices, there are several
  /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus a media
  controller device node (usually /dev/media0).
  We call it as mc-based devices. Controlling the hardware require
  opening the media device (/dev/media0), setup the pipeline and adjust
  the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
  by /dev/video?.

All "standard" media applications, including open source ones (Camorama,
Cheese, Xawtv, Firefox, Chromium, ...) and closed source ones (Skype, 
Chrome, ...) supports devnode-based devices.

Support for mc-based devices currently require an specialized application 
in order to prepare the device for its usage (setup pipelines, adjust
hardware controls, etc). Once pipeline is set, the streaming goes via
/dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
be opened, in order to implement algorithms designed to make video quality
reasonable. On such devices, it is not uncommon that the device used by the
application to be a random number (on OMAP3 driver, typically, is either
/dev/video4 or /dev/video6).

One example of such hardware is at the OMAP3-based hardware:

	http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png

On the picture, there's a graph with the hardware blocks in blue/dark/blue
and the corresponding devnode interfaces in yellow.

The mc-based approach was taken when support for Nokia N9/N900 cameras 
was added (with has OMAP3 SoC). It is required because the camera hardware
on SoC comes with a media processor (ISP), with does a lot more than just
capturing, allowing complex algorithms to enhance image quality in runtime.
Those algorithms are known as 3A - an acronym for 3 other acronyms:

	- AE (Auto Exposure);
	- AF (Auto Focus);
	- AWB (Auto White Balance).

Setting a camera with such ISPs are harder because the pipelines to be
set actually depends the requirements for those 3A algorithms to run.
Also, usually, the 3A algorithms use some chipset-specific userspace API,
that exports some image properties, calculated by the ISP, to speed up
the convergence of those algorithms.

Btw, usually, the 3A algorithms are IP-protected, provided by vendors
as binary only blobs, although there are a few OSS implementations.

1.2 V4L2 userspace aspects
--------------------------

Back when USB cameras were introduced, the hardware were really simple:
they had a CCD camera sensor and a chip that bridges the data though
USB. CCD camera sensors typically provide data using a bayer format,
but they usually have their own proprietary ways to pack the data,
in order to reduce the USB bandwidth (original cameras were USB 1.1).

So, V4L2 has a myriad of different formats, in order to match each
CCD camera sensor format. At the end of the day, applications were
able to use only a subset of the available hardware, since they need
to come with format converters for all formats the developer uses
(usually a very small subset of the available ones).

To end with this mess, an userspace library was written, called libv4l.
It supports all those proprietary formats. So, applications can use
a RGB or YUV format, without needing to concern about conversions.

The way it works is by adding wrappers to system calls: open, close,
ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
at the source code of the apps, all it was needed is to prepend the
existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.

All open source apps we know now supports libv4l. On a few (like
gstreamer), support for it is optional.

In order to support closed source, another wrapper was added, allowing
to call any closed source application to use it, by using LD_PRELOAD.
For example, using skype with it is as simple as calling it with:

	$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so /usr/bin/skypeforlinux

2. Current problems
===================

2.1 Libv4l can slow image handling
----------------------------------

Nowadays, almost all new "simple" cameras are connected via USB using
the UVC class (USB Video Class). UVC standardized the allowed formats,
and most apps just implement them. The UVC hardware is more complex,
having format converters inside it. So, for most usages, format
conversion isn't needed anymore.

The need of doing format conversion in software makes libv4l slow,
requiring lots of CPU usage in order to convert a 4K or 8K format,
being even worse with 3D cameras.

Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
DMA_BUFFER currently doesn't work with libv4l.

Right now, gstreamer defaults to not enable libv4l2, mainly due to 
those performance issues.


2.2 Modern hardware is starting to come with "complex" camera ISP
-----------------------------------------------------------------

While mc-based devices were limited to SoC, it was easy to
"delegate" the task of talking with the hardware to the
embedded hardware designers.

However, this is changing. Dell Latitude 5285 laptop is a standard
PC with an i3-core, i5-core or i7-core CPU, with comes with the
Intel IMU3 ISP hardware[1]

[1] https://www.spinics.net/lists/linux-usb/msg167478.html

There, instead of an USB camera, the hardware is equipped with a
MC-based ISP, connected to its camera. Currently, despite having
a Kernel driver for it, the camera doesn't work with any
userspace application.

I'm also aware of other projects that are considering the usage of
mc-based devices for non-dedicated hardware.

3. How to solve it?
===================

That's the main focus of the meeting :-)

From a previous discussion I had with media sub-maintainers, there are
at least two actions that seem required. I'm listing them below as
an starting point for the discussions, but we can eventually come up
with some different approach after the meeting.

3.1 libv4l2 support for mc-based hardware
=========================================

In order to support those hardware, we'll need to do some redesign
mainly at libv4l2[2].

The idea is to work on a new API for libv4l2 that will allow to
split the format conversion on a separate part of it, add support
for DMA Buffer and come up with a way for the library to work
transparently with both devnode-based and mc-based hardware.

That envolves adding capacity at libv4l to setup hardware pipelines
and to propagate controls among their sub-devices. Eventually, part
of it will be done in Kernel.

That should give performance increase at the library and would allow
gstreamer to use it by default, without compromising performance.

[2] I don't discard that some Kernel changes could also be part of the
solution, like, for example, doing control propagation along the pipeline
on simple use case scenarios.

3.2 libv4l2 support for 3A algorithms
=====================================

The 3A algorithm handing is highly dependent on the hardware. The
idea here is to allow libv4l to have a set of 3A algorithms that
will be specific to certain mc-based hardware. Ideally, this should
be added in a way that it will allow external closed-source
algorithms to run as well.

Thanks,
Mauro

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 11871 bytes --]

Le jeudi 17 mai 2018 à 16:07 -0300, Mauro Carvalho Chehab a écrit :
> Hi all,
> 
> The goal of this e-mail is to schedule a meeting in order to discuss
> improvements at the media subsystem in order to support complex
> camera
> hardware by usual apps.
> 
> The main focus here is to allow supporting devices with MC-based
> hardware connected to a camera.
> 
> In short, my proposal is to meet with the interested parties on
> solving
> this issue during the Open Source Summit in Japan, e. g. between
> June, 19-22, in Tokyo.
> 
> I'd like to know who is interested on joining us for such meeting,
> and to hear a proposal of themes for discussions.
> 
> I'm enclosing a detailed description of the problem, in order to
> allow the interested parties to be at the same page.

It's unlikely I'll be able to attend this meeting, but I'd like to
provide some initial input on this. Find inline some clarification on
why libv4l2 is disabled by default in Gst, as it's not just
performance.

A major aspect that is totally absent of this mail is PipeWire. With
the venue of sandboxed application, there is a need to control access
to cameras through a daemon. The same daemon is also used to control
access to screen capture on Wayland (instead of letting any random
application capture your screen, like on X11). The effort is lead by
the desktop team at RedHat (folks CCed). PipeWire already have V4L2
native support and is integrated in GStreamer already in a way that it
can totally replace the V4L2 capture component there. PipeWire is
plugin base, so more type of camera support (including proprietary
ones) can be added. Remote daemon can also provide streams, as this is
the case for compositors and screen casting. An extra benefit is that
you can have multiple application reading frames from the same camera.
It also allow sandboxed application (the do not have access to /dev) to
use the cameras. PipeWire is much more then that, but let's focus on
that.

This is the direction we are heading on the "generic" / Desktop Linux.
Porting Firefox and Chrome is obviously planed, as these beast are
clear candidate for being sand-boxed and requires screen share feature
for WebRTC.

In this context, proprietary or HW specific algorithm could be
implemented in userspace as PipeWire plugins, and then application will
automatically be enable to enumerate and use these. I'm not saying the
libv4l2 stuff is not needed short term, but it's just a short term
thing in my opinion.

> 
> Regards,
> Mauro
> 
> ---
> 
> 
> 1. Introduction
> ===============
> 
> 1.1 V4L2 Kernel aspects
> -----------------------
> 
> The media subsystem supports two types of devices:
> 
> - "traditional" media hardware, supported via V4L2 API. On such
> hardware, 
>   opening a single device node (usually /dev/video0) is enough to
> control
>   the entire device. We call it as devnode-based devices.
> 
> - Media-controller based devices. On those devices, there are several
>   /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus a media
>   controller device node (usually /dev/media0).
>   We call it as mc-based devices. Controlling the hardware require
>   opening the media device (/dev/media0), setup the pipeline and
> adjust
>   the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
>   by /dev/video?.
> 
> All "standard" media applications, including open source ones
> (Camorama,
> Cheese, Xawtv, Firefox, Chromium, ...) and closed source ones
> (Skype, 
> Chrome, ...) supports devnode-based devices.
> 
> Support for mc-based devices currently require an specialized
> application 
> in order to prepare the device for its usage (setup pipelines, adjust
> hardware controls, etc). Once pipeline is set, the streaming goes via
> /dev/video?, although usually some /dev/v4l2-subdev? devnodes should
> also
> be opened, in order to implement algorithms designed to make video
> quality
> reasonable. On such devices, it is not uncommon that the device used
> by the
> application to be a random number (on OMAP3 driver, typically, is
> either
> /dev/video4 or /dev/video6).
> 
> One example of such hardware is at the OMAP3-based hardware:
> 
> 	http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with
> -tvp5150.png
> 
> On the picture, there's a graph with the hardware blocks in
> blue/dark/blue
> and the corresponding devnode interfaces in yellow.
> 
> The mc-based approach was taken when support for Nokia N9/N900
> cameras 
> was added (with has OMAP3 SoC). It is required because the camera
> hardware
> on SoC comes with a media processor (ISP), with does a lot more than
> just
> capturing, allowing complex algorithms to enhance image quality in
> runtime.
> Those algorithms are known as 3A - an acronym for 3 other acronyms:
> 
> 	- AE (Auto Exposure);
> 	- AF (Auto Focus);
> 	- AWB (Auto White Balance).
> 
> Setting a camera with such ISPs are harder because the pipelines to
> be
> set actually depends the requirements for those 3A algorithms to run.
> Also, usually, the 3A algorithms use some chipset-specific userspace
> API,
> that exports some image properties, calculated by the ISP, to speed
> up
> the convergence of those algorithms.
> 
> Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> as binary only blobs, although there are a few OSS implementations.
> 
> 1.2 V4L2 userspace aspects
> --------------------------
> 
> Back when USB cameras were introduced, the hardware were really
> simple:
> they had a CCD camera sensor and a chip that bridges the data though
> USB. CCD camera sensors typically provide data using a bayer format,
> but they usually have their own proprietary ways to pack the data,
> in order to reduce the USB bandwidth (original cameras were USB 1.1).
> 
> So, V4L2 has a myriad of different formats, in order to match each
> CCD camera sensor format. At the end of the day, applications were
> able to use only a subset of the available hardware, since they need
> to come with format converters for all formats the developer uses
> (usually a very small subset of the available ones).
> 
> To end with this mess, an userspace library was written, called
> libv4l.
> It supports all those proprietary formats. So, applications can use
> a RGB or YUV format, without needing to concern about conversions.
> 
> The way it works is by adding wrappers to system calls: open, close,
> ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> at the source code of the apps, all it was needed is to prepend the
> existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.
> 
> All open source apps we know now supports libv4l. On a few (like
> gstreamer), support for it is optional.
> 
> In order to support closed source, another wrapper was added,
> allowing
> to call any closed source application to use it, by using LD_PRELOAD.
> For example, using skype with it is as simple as calling it with:
> 
> 	$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so
> /usr/bin/skypeforlinux
> 
> 2. Current problems
> ===================
> 
> 2.1 Libv4l can slow image handling
> ----------------------------------
> 
> Nowadays, almost all new "simple" cameras are connected via USB using
> the UVC class (USB Video Class). UVC standardized the allowed
> formats,
> and most apps just implement them. The UVC hardware is more complex,
> having format converters inside it. So, for most usages, format
> conversion isn't needed anymore.
> 
> The need of doing format conversion in software makes libv4l slow,
> requiring lots of CPU usage in order to convert a 4K or 8K format,
> being even worse with 3D cameras.
> 
> Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> DMA_BUFFER currently doesn't work with libv4l.
> 
> Right now, gstreamer defaults to not enable libv4l2, mainly due to 
> those performance issues.

I need to clarify a little bit on why we disabled libv4l2 in GStreamer,
as it's not only for performance reason, there is couple of major
issues in the libv4l2 implementation that get's in way. Just a short
list:

  - Crash when CREATE_BUFS is being used
  - Crash in the jpeg decoder (when frames are corrupted)
  - App exporting DMABuf need to be aware of emulation, otherwise the
    DMABuf exported are in the orignal format
  - RW emulation only initialize the queue on first read (causing 
    userspace poll() to fail)
  - Signature of v4l2_mmap does not match mmap() (minor)
  - The colorimetry does not seem emulated when conversion
  - Sub-optimal locking (at least deadlocks were fixed)

Except for the colorimetry (which causes negotiation failure, as it
causes invalid colorimetry / format matches), these issues are already
worked around in GStreamer, but with the lost of features of course.
There is other cases were something worked without libv4l2, but didn't
work with libv4l2, but we haven't tracked down the cause.

For people working on this venue, since 1.14, you can enable libv4l2 at
run-time using env GST_V4L2_USE_LIBV4L2=1.

> 
> 
> 2.2 Modern hardware is starting to come with "complex" camera ISP
> -----------------------------------------------------------------
> 
> While mc-based devices were limited to SoC, it was easy to
> "delegate" the task of talking with the hardware to the
> embedded hardware designers.
> 
> However, this is changing. Dell Latitude 5285 laptop is a standard
> PC with an i3-core, i5-core or i7-core CPU, with comes with the
> Intel IMU3 ISP hardware[1]
> 
> [1] https://www.spinics.net/lists/linux-usb/msg167478.html
> 
> There, instead of an USB camera, the hardware is equipped with a
> MC-based ISP, connected to its camera. Currently, despite having
> a Kernel driver for it, the camera doesn't work with any
> userspace application.
> 
> I'm also aware of other projects that are considering the usage of
> mc-based devices for non-dedicated hardware.
> 
> 3. How to solve it?
> ===================
> 
> That's the main focus of the meeting :-)
> 
> From a previous discussion I had with media sub-maintainers, there
> are
> at least two actions that seem required. I'm listing them below as
> an starting point for the discussions, but we can eventually come up
> with some different approach after the meeting.
> 
> 3.1 libv4l2 support for mc-based hardware
> =========================================
> 
> In order to support those hardware, we'll need to do some redesign
> mainly at libv4l2[2].
> 
> The idea is to work on a new API for libv4l2 that will allow to
> split the format conversion on a separate part of it, add support
> for DMA Buffer and come up with a way for the library to work
> transparently with both devnode-based and mc-based hardware.
> 
> That envolves adding capacity at libv4l to setup hardware pipelines
> and to propagate controls among their sub-devices. Eventually, part
> of it will be done in Kernel.
> 
> That should give performance increase at the library and would allow
> gstreamer to use it by default, without compromising performance.
> 
> [2] I don't discard that some Kernel changes could also be part of
> the
> solution, like, for example, doing control propagation along the
> pipeline
> on simple use case scenarios.
> 
> 3.2 libv4l2 support for 3A algorithms
> =====================================
> 
> The 3A algorithm handing is highly dependent on the hardware. The
> idea here is to allow libv4l to have a set of 3A algorithms that
> will be specific to certain mc-based hardware. Ideally, this should
> be added in a way that it will allow external closed-source
> algorithms to run as well.
> 
> Thanks,
> Mauro

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Nicolas,

On Friday, 18 May 2018 00:38:53 EEST Nicolas Dufresne wrote:
> Le jeudi 17 mai 2018 à 16:07 -0300, Mauro Carvalho Chehab a écrit :
> > Hi all,
> > 
> > The goal of this e-mail is to schedule a meeting in order to discuss
> > improvements at the media subsystem in order to support complex
> > camera hardware by usual apps.
> > 
> > The main focus here is to allow supporting devices with MC-based
> > hardware connected to a camera.
> > 
> > In short, my proposal is to meet with the interested parties on
> > solving this issue during the Open Source Summit in Japan, e. g. between
> > June, 19-22, in Tokyo.
> > 
> > I'd like to know who is interested on joining us for such meeting,
> > and to hear a proposal of themes for discussions.
> > 
> > I'm enclosing a detailed description of the problem, in order to
> > allow the interested parties to be at the same page.
> 
> It's unlikely I'll be able to attend this meeting, but I'd like to
> provide some initial input on this. Find inline some clarification on
> why libv4l2 is disabled by default in Gst, as it's not just
> performance.
> 
> A major aspect that is totally absent of this mail is PipeWire. With
> the venue of sandboxed application, there is a need to control access
> to cameras through a daemon. The same daemon is also used to control
> access to screen capture on Wayland (instead of letting any random
> application capture your screen, like on X11). The effort is lead by
> the desktop team at RedHat (folks CCed). PipeWire already have V4L2
> native support and is integrated in GStreamer already in a way that it
> can totally replace the V4L2 capture component there. PipeWire is
> plugin base, so more type of camera support (including proprietary
> ones) can be added.

One issue that has been worrying me for the past five years or so is how to 
ensure that we will continue having open-source camera support in the future. 
Pipewire is just a technology and as such can be used in good or evil ways, 
but as a community we need to care about availability of open solutions.

So far, by pushing the V4L2 API as the proper way to support cameras, we have 
tried to resist the natural inclination of vendors to close everything, as 
implementing a closed-source kernel driver isn't an option that most would 
consider. Of course, the drawback is that some vendors have simply decided not 
to care about upstream camera support.

If we move the camera API one level up to userspace (and whether the API will 
be defined by Pipewire, by libv4l or by something else), we'll make it easier 
for vendors not to play along. My big question is how to prevent that. I think 
there's still value there in mandating V4L2 as the only API for cameras, and 
in ensuring that we support multiple userspace multimedia stacks, not just 
Pipewire (this is already done in a way, as I don't foresee Android moving 
away from their camera HAL in the near future). That will likely not be 
enough, and I'd like to hear other people's opinions on this topic.

I would like to emphasize that I don't expect vendors to open the 
implementation of their 3A algorithms, and I'm not actually concerned about 
that part. If that's the only part shipped as closed-source, and if the 
hardware operation is documented (ideally in public datasheet, but at a 
minimum with proper documentation of custom ioctls used to configure the 
hardware), then the community will have the opportunity to implement an open-
source 3A library. My main concern is thus about all component other than the 
3A library.

> Remote daemon can also provide streams, as this is
> the case for compositors and screen casting. An extra benefit is that
> you can have multiple application reading frames from the same camera.
> It also allow sandboxed application (the do not have access to /dev) to
> use the cameras. PipeWire is much more then that, but let's focus on
> that.
> 
> This is the direction we are heading on the "generic" / Desktop Linux.
> Porting Firefox and Chrome is obviously planed, as these beast are
> clear candidate for being sand-boxed and requires screen share feature
> for WebRTC.
> 
> In this context, proprietary or HW specific algorithm could be
> implemented in userspace as PipeWire plugins, and then application will
> automatically be enable to enumerate and use these. I'm not saying the
> libv4l2 stuff is not needed short term, but it's just a short term
> thing in my opinion.
> 
> > 1. Introduction
> > ===============
> > 
> > 1.1 V4L2 Kernel aspects
> > -----------------------
> > 
> > The media subsystem supports two types of devices:
> > 
> > - "traditional" media hardware, supported via V4L2 API. On such
> >   hardware, opening a single device node (usually /dev/video0) is enough
> >   to control the entire device. We call it as devnode-based devices.
> > 
> > - Media-controller based devices. On those devices, there are several
> >   /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus a media
> >   controller device node (usually /dev/media0).
> >   We call it as mc-based devices. Controlling the hardware require
> >   opening the media device (/dev/media0), setup the pipeline and
> >   adjust the sub-devices via /dev/v4l2-subdev?. Only streaming is
> >   controlled by /dev/video?.
> > 
> > All "standard" media applications, including open source ones
> > (Camorama, Cheese, Xawtv, Firefox, Chromium, ...) and closed source ones
> > (Skype, Chrome, ...) supports devnode-based devices.
> > 
> > Support for mc-based devices currently require an specialized
> > application in order to prepare the device for its usage (setup pipelines,
> > adjust hardware controls, etc). Once pipeline is set, the streaming goes
> > via /dev/video?, although usually some /dev/v4l2-subdev? devnodes should
> > also be opened, in order to implement algorithms designed to make video
> > quality reasonable. On such devices, it is not uncommon that the device
> > used by the application to be a random number (on OMAP3 driver, typically,
> > is either /dev/video4 or /dev/video6).
> > 
> > One example of such hardware is at the OMAP3-based hardware:
> > 	http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with
> > 
> > -tvp5150.png
> > 
> > On the picture, there's a graph with the hardware blocks in
> > blue/dark/blue and the corresponding devnode interfaces in yellow.
> > 
> > The mc-based approach was taken when support for Nokia N9/N900
> > cameras was added (with has OMAP3 SoC). It is required because the camera
> > hardware on SoC comes with a media processor (ISP), with does a lot more
> > than just capturing, allowing complex algorithms to enhance image quality
> > in runtime.
> > 
> > Those algorithms are known as 3A - an acronym for 3 other acronyms:
> > 	- AE (Auto Exposure);
> > 	- AF (Auto Focus);
> > 	- AWB (Auto White Balance).
> > 
> > Setting a camera with such ISPs are harder because the pipelines to
> > be set actually depends the requirements for those 3A algorithms to run.
> > Also, usually, the 3A algorithms use some chipset-specific userspace
> > API, that exports some image properties, calculated by the ISP, to speed
> > up the convergence of those algorithms.
> > 
> > Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> > as binary only blobs, although there are a few OSS implementations.
> > 
> > 1.2 V4L2 userspace aspects
> > --------------------------
> > 
> > Back when USB cameras were introduced, the hardware were really
> > simple: they had a CCD camera sensor and a chip that bridges the data
> > though USB. CCD camera sensors typically provide data using a bayer
> > format, but they usually have their own proprietary ways to pack the data,
> > in order to reduce the USB bandwidth (original cameras were USB 1.1).
> > 
> > So, V4L2 has a myriad of different formats, in order to match each
> > CCD camera sensor format. At the end of the day, applications were
> > able to use only a subset of the available hardware, since they need
> > to come with format converters for all formats the developer uses
> > (usually a very small subset of the available ones).
> > 
> > To end with this mess, an userspace library was written, called
> > libv4l. It supports all those proprietary formats. So, applications can
> > use a RGB or YUV format, without needing to concern about conversions.
> > 
> > The way it works is by adding wrappers to system calls: open, close,
> > ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> > at the source code of the apps, all it was needed is to prepend the
> > existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.
> > 
> > All open source apps we know now supports libv4l. On a few (like
> > gstreamer), support for it is optional.
> > 
> > In order to support closed source, another wrapper was added,
> > allowing to call any closed source application to use it, by using
> > LD_PRELOAD.
> > 
> > For example, using skype with it is as simple as calling it with:
> > 	$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so
> > 
> > /usr/bin/skypeforlinux
> > 
> > 2. Current problems
> > ===================
> > 
> > 2.1 Libv4l can slow image handling
> > ----------------------------------
> > 
> > Nowadays, almost all new "simple" cameras are connected via USB using
> > the UVC class (USB Video Class). UVC standardized the allowed
> > formats, and most apps just implement them. The UVC hardware is more
> > complex, having format converters inside it. So, for most usages, format
> > conversion isn't needed anymore.
> > 
> > The need of doing format conversion in software makes libv4l slow,
> > requiring lots of CPU usage in order to convert a 4K or 8K format,
> > being even worse with 3D cameras.
> > 
> > Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> > DMA_BUFFER currently doesn't work with libv4l.
> > 
> > Right now, gstreamer defaults to not enable libv4l2, mainly due to
> > those performance issues.
> 
> I need to clarify a little bit on why we disabled libv4l2 in GStreamer,
> as it's not only for performance reason, there is couple of major
> issues in the libv4l2 implementation that get's in way. Just a short
> list:
> 
>   - Crash when CREATE_BUFS is being used
>   - Crash in the jpeg decoder (when frames are corrupted)
>   - App exporting DMABuf need to be aware of emulation, otherwise the
>     DMABuf exported are in the orignal format
>   - RW emulation only initialize the queue on first read (causing
>     userspace poll() to fail)
>   - Signature of v4l2_mmap does not match mmap() (minor)
>   - The colorimetry does not seem emulated when conversion
>   - Sub-optimal locking (at least deadlocks were fixed)

Do you see any point in that list that couldn't be fixed in libv4l ?

> Except for the colorimetry (which causes negotiation failure, as it
> causes invalid colorimetry / format matches), these issues are already
> worked around in GStreamer, but with the lost of features of course.
> There is other cases were something worked without libv4l2, but didn't
> work with libv4l2, but we haven't tracked down the cause.
> 
> For people working on this venue, since 1.14, you can enable libv4l2 at
> run-time using env GST_V4L2_USE_LIBV4L2=1.
> 
> > 2.2 Modern hardware is starting to come with "complex" camera ISP
> > -----------------------------------------------------------------
> > 
> > While mc-based devices were limited to SoC, it was easy to
> > "delegate" the task of talking with the hardware to the
> > embedded hardware designers.
> > 
> > However, this is changing. Dell Latitude 5285 laptop is a standard
> > PC with an i3-core, i5-core or i7-core CPU, with comes with the
> > Intel IMU3 ISP hardware[1]
> > 
> > [1] https://www.spinics.net/lists/linux-usb/msg167478.html
> > 
> > There, instead of an USB camera, the hardware is equipped with a
> > MC-based ISP, connected to its camera. Currently, despite having
> > a Kernel driver for it, the camera doesn't work with any
> > userspace application.
> > 
> > I'm also aware of other projects that are considering the usage of
> > mc-based devices for non-dedicated hardware.
> > 
> > 3. How to solve it?
> > ===================
> > 
> > That's the main focus of the meeting :-)
> > 
> > From a previous discussion I had with media sub-maintainers, there
> > are at least two actions that seem required. I'm listing them below as
> > an starting point for the discussions, but we can eventually come up
> > with some different approach after the meeting.
> > 
> > 3.1 libv4l2 support for mc-based hardware
> > =========================================
> > 
> > In order to support those hardware, we'll need to do some redesign
> > mainly at libv4l2[2].
> > 
> > The idea is to work on a new API for libv4l2 that will allow to
> > split the format conversion on a separate part of it, add support
> > for DMA Buffer and come up with a way for the library to work
> > transparently with both devnode-based and mc-based hardware.
> > 
> > That envolves adding capacity at libv4l to setup hardware pipelines
> > and to propagate controls among their sub-devices. Eventually, part
> > of it will be done in Kernel.
> > 
> > That should give performance increase at the library and would allow
> > gstreamer to use it by default, without compromising performance.
> > 
> > [2] I don't discard that some Kernel changes could also be part of
> > the solution, like, for example, doing control propagation along the
> > pipeline on simple use case scenarios.
> > 
> > 3.2 libv4l2 support for 3A algorithms
> > =====================================
> > 
> > The 3A algorithm handing is highly dependent on the hardware. The
> > idea here is to allow libv4l to have a set of 3A algorithms that
> > will be specific to certain mc-based hardware. Ideally, this should
> > be added in a way that it will allow external closed-source
> > algorithms to run as well.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Mauro Carvalho Chehab]

Em Fri, 18 May 2018 11:15:39 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> escreveu:

> Hi Nicolas,
> 
> On Friday, 18 May 2018 00:38:53 EEST Nicolas Dufresne wrote:
> > Le jeudi 17 mai 2018 à 16:07 -0300, Mauro Carvalho Chehab a écrit :  
> > > Hi all,
> > > 
> > > The goal of this e-mail is to schedule a meeting in order to discuss
> > > improvements at the media subsystem in order to support complex
> > > camera hardware by usual apps.
> > > 
> > > The main focus here is to allow supporting devices with MC-based
> > > hardware connected to a camera.
> > > 
> > > In short, my proposal is to meet with the interested parties on
> > > solving this issue during the Open Source Summit in Japan, e. g. between
> > > June, 19-22, in Tokyo.
> > > 
> > > I'd like to know who is interested on joining us for such meeting,
> > > and to hear a proposal of themes for discussions.
> > > 
> > > I'm enclosing a detailed description of the problem, in order to
> > > allow the interested parties to be at the same page.  
> > 
> > It's unlikely I'll be able to attend this meeting, but I'd like to
> > provide some initial input on this. Find inline some clarification on
> > why libv4l2 is disabled by default in Gst, as it's not just
> > performance.

Thanks for complementing it!

> > 
> > A major aspect that is totally absent of this mail is PipeWire. With
> > the venue of sandboxed application, there is a need to control access
> > to cameras through a daemon. The same daemon is also used to control
> > access to screen capture on Wayland (instead of letting any random
> > application capture your screen, like on X11). The effort is lead by
> > the desktop team at RedHat (folks CCed). PipeWire already have V4L2
> > native support and is integrated in GStreamer already in a way that it
> > can totally replace the V4L2 capture component there. PipeWire is
> > plugin base, so more type of camera support (including proprietary
> > ones) can be added.  
> 
> One issue that has been worrying me for the past five years or so is how to 
> ensure that we will continue having open-source camera support in the future. 
> Pipewire is just a technology and as such can be used in good or evil ways, 
> but as a community we need to care about availability of open solutions.
> 
> So far, by pushing the V4L2 API as the proper way to support cameras, we have 
> tried to resist the natural inclination of vendors to close everything, as 
> implementing a closed-source kernel driver isn't an option that most would 
> consider. Of course, the drawback is that some vendors have simply decided not 
> to care about upstream camera support.
> 
> If we move the camera API one level up to userspace (and whether the API will 
> be defined by Pipewire, by libv4l or by something else), we'll make it easier 
> for vendors not to play along. My big question is how to prevent that. I think 
> there's still value there in mandating V4L2 as the only API for cameras, and 
> in ensuring that we support multiple userspace multimedia stacks, not just 
> Pipewire (this is already done in a way, as I don't foresee Android moving 
> away from their camera HAL in the near future). That will likely not be 
> enough, and I'd like to hear other people's opinions on this topic.
> 
> I would like to emphasize that I don't expect vendors to open the 
> implementation of their 3A algorithms, and I'm not actually concerned about 
> that part. If that's the only part shipped as closed-source, and if the 
> hardware operation is documented (ideally in public datasheet, but at a 
> minimum with proper documentation of custom ioctls used to configure the 
> hardware), then the community will have the opportunity to implement an open-
> source 3A library. My main concern is thus about all component other than the 
> 3A library.

Yeah, I share the same concern. Whatever solution we take, we should
do our best to ensure that the camera driver will be open and it would
be possible to have open-sourced 3A libraries as alternatives.

One of the biggest reasons why we decided to start libv4l project,
in the past, was to ensure an open source solution. The problem we
faced on that time is to ensure that, when a new media driver were
added with some proprietary output format, an open source decoding
software were also added at libv4l.

This approach ensured that all non-MC cameras are supported by all
V4L2 applications.

Before libv4l, media support for a given device were limited to a few 
apps that knew how to decode the format. There were even cases were a
proprietary app were required, as no open source decoders were available.

From my PoV, the biggest gain with libv4l is that the same group of
maintainers can ensure that the entire solution (Kernel driver and
low level userspace support) will provide everything required for an
open source app to work with it.

I'm not sure how we would keep enforcing it if the pipeline setting
and control propagation logic for an specific hardware will be
delegated to PipeWire. It seems easier to keep doing it on a libv4l
(version 2) and let PipeWire to use it.

> 
> > Remote daemon can also provide streams, as this is
> > the case for compositors and screen casting. An extra benefit is that
> > you can have multiple application reading frames from the same camera.
> > It also allow sandboxed application (the do not have access to /dev) to
> > use the cameras. PipeWire is much more then that, but let's focus on
> > that.
> > 
> > This is the direction we are heading on the "generic" / Desktop Linux.
> > Porting Firefox and Chrome is obviously planed, as these beast are
> > clear candidate for being sand-boxed and requires screen share feature
> > for WebRTC.
> > 
> > In this context, proprietary or HW specific algorithm could be
> > implemented in userspace as PipeWire plugins, and then application will
> > automatically be enable to enumerate and use these. I'm not saying the
> > libv4l2 stuff is not needed short term, but it's just a short term
> > thing in my opinion.
> >   
> > > 1. Introduction
> > > ===============
> > > 
> > > 1.1 V4L2 Kernel aspects
> > > -----------------------
> > > 
> > > The media subsystem supports two types of devices:
> > > 
> > > - "traditional" media hardware, supported via V4L2 API. On such
> > >   hardware, opening a single device node (usually /dev/video0) is enough
> > >   to control the entire device. We call it as devnode-based devices.
> > > 
> > > - Media-controller based devices. On those devices, there are several
> > >   /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus a media
> > >   controller device node (usually /dev/media0).
> > >   We call it as mc-based devices. Controlling the hardware require
> > >   opening the media device (/dev/media0), setup the pipeline and
> > >   adjust the sub-devices via /dev/v4l2-subdev?. Only streaming is
> > >   controlled by /dev/video?.
> > > 
> > > All "standard" media applications, including open source ones
> > > (Camorama, Cheese, Xawtv, Firefox, Chromium, ...) and closed source ones
> > > (Skype, Chrome, ...) supports devnode-based devices.
> > > 
> > > Support for mc-based devices currently require an specialized
> > > application in order to prepare the device for its usage (setup pipelines,
> > > adjust hardware controls, etc). Once pipeline is set, the streaming goes
> > > via /dev/video?, although usually some /dev/v4l2-subdev? devnodes should
> > > also be opened, in order to implement algorithms designed to make video
> > > quality reasonable. On such devices, it is not uncommon that the device
> > > used by the application to be a random number (on OMAP3 driver, typically,
> > > is either /dev/video4 or /dev/video6).
> > > 
> > > One example of such hardware is at the OMAP3-based hardware:
> > > 	http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with
> > > 
> > > -tvp5150.png
> > > 
> > > On the picture, there's a graph with the hardware blocks in
> > > blue/dark/blue and the corresponding devnode interfaces in yellow.
> > > 
> > > The mc-based approach was taken when support for Nokia N9/N900
> > > cameras was added (with has OMAP3 SoC). It is required because the camera
> > > hardware on SoC comes with a media processor (ISP), with does a lot more
> > > than just capturing, allowing complex algorithms to enhance image quality
> > > in runtime.
> > > 
> > > Those algorithms are known as 3A - an acronym for 3 other acronyms:
> > > 	- AE (Auto Exposure);
> > > 	- AF (Auto Focus);
> > > 	- AWB (Auto White Balance).
> > > 
> > > Setting a camera with such ISPs are harder because the pipelines to
> > > be set actually depends the requirements for those 3A algorithms to run.
> > > Also, usually, the 3A algorithms use some chipset-specific userspace
> > > API, that exports some image properties, calculated by the ISP, to speed
> > > up the convergence of those algorithms.
> > > 
> > > Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> > > as binary only blobs, although there are a few OSS implementations.
> > > 
> > > 1.2 V4L2 userspace aspects
> > > --------------------------
> > > 
> > > Back when USB cameras were introduced, the hardware were really
> > > simple: they had a CCD camera sensor and a chip that bridges the data
> > > though USB. CCD camera sensors typically provide data using a bayer
> > > format, but they usually have their own proprietary ways to pack the data,
> > > in order to reduce the USB bandwidth (original cameras were USB 1.1).
> > > 
> > > So, V4L2 has a myriad of different formats, in order to match each
> > > CCD camera sensor format. At the end of the day, applications were
> > > able to use only a subset of the available hardware, since they need
> > > to come with format converters for all formats the developer uses
> > > (usually a very small subset of the available ones).
> > > 
> > > To end with this mess, an userspace library was written, called
> > > libv4l. It supports all those proprietary formats. So, applications can
> > > use a RGB or YUV format, without needing to concern about conversions.
> > > 
> > > The way it works is by adding wrappers to system calls: open, close,
> > > ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> > > at the source code of the apps, all it was needed is to prepend the
> > > existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.
> > > 
> > > All open source apps we know now supports libv4l. On a few (like
> > > gstreamer), support for it is optional.
> > > 
> > > In order to support closed source, another wrapper was added,
> > > allowing to call any closed source application to use it, by using
> > > LD_PRELOAD.
> > > 
> > > For example, using skype with it is as simple as calling it with:
> > > 	$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so
> > > 
> > > /usr/bin/skypeforlinux
> > > 
> > > 2. Current problems
> > > ===================
> > > 
> > > 2.1 Libv4l can slow image handling
> > > ----------------------------------
> > > 
> > > Nowadays, almost all new "simple" cameras are connected via USB using
> > > the UVC class (USB Video Class). UVC standardized the allowed
> > > formats, and most apps just implement them. The UVC hardware is more
> > > complex, having format converters inside it. So, for most usages, format
> > > conversion isn't needed anymore.
> > > 
> > > The need of doing format conversion in software makes libv4l slow,
> > > requiring lots of CPU usage in order to convert a 4K or 8K format,
> > > being even worse with 3D cameras.
> > > 
> > > Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> > > DMA_BUFFER currently doesn't work with libv4l.
> > > 
> > > Right now, gstreamer defaults to not enable libv4l2, mainly due to
> > > those performance issues.  
> > 
> > I need to clarify a little bit on why we disabled libv4l2 in GStreamer,
> > as it's not only for performance reason, there is couple of major
> > issues in the libv4l2 implementation that get's in way. Just a short
> > list:
> > 
> >   - Crash when CREATE_BUFS is being used
> >   - Crash in the jpeg decoder (when frames are corrupted)
> >   - App exporting DMABuf need to be aware of emulation, otherwise the
> >     DMABuf exported are in the orignal format
> >   - RW emulation only initialize the queue on first read (causing
> >     userspace poll() to fail)
> >   - Signature of v4l2_mmap does not match mmap() (minor)
> >   - The colorimetry does not seem emulated when conversion
> >   - Sub-optimal locking (at least deadlocks were fixed)  
> 
> Do you see any point in that list that couldn't be fixed in libv4l ?
> 
> > Except for the colorimetry (which causes negotiation failure, as it
> > causes invalid colorimetry / format matches), these issues are already
> > worked around in GStreamer, but with the lost of features of course.
> > There is other cases were something worked without libv4l2, but didn't
> > work with libv4l2, but we haven't tracked down the cause.
> > 
> > For people working on this venue, since 1.14, you can enable libv4l2 at
> > run-time using env GST_V4L2_USE_LIBV4L2=1.
> >   
> > > 2.2 Modern hardware is starting to come with "complex" camera ISP
> > > -----------------------------------------------------------------
> > > 
> > > While mc-based devices were limited to SoC, it was easy to
> > > "delegate" the task of talking with the hardware to the
> > > embedded hardware designers.
> > > 
> > > However, this is changing. Dell Latitude 5285 laptop is a standard
> > > PC with an i3-core, i5-core or i7-core CPU, with comes with the
> > > Intel IMU3 ISP hardware[1]
> > > 
> > > [1] https://www.spinics.net/lists/linux-usb/msg167478.html
> > > 
> > > There, instead of an USB camera, the hardware is equipped with a
> > > MC-based ISP, connected to its camera. Currently, despite having
> > > a Kernel driver for it, the camera doesn't work with any
> > > userspace application.
> > > 
> > > I'm also aware of other projects that are considering the usage of
> > > mc-based devices for non-dedicated hardware.
> > > 
> > > 3. How to solve it?
> > > ===================
> > > 
> > > That's the main focus of the meeting :-)
> > > 
> > > From a previous discussion I had with media sub-maintainers, there
> > > are at least two actions that seem required. I'm listing them below as
> > > an starting point for the discussions, but we can eventually come up
> > > with some different approach after the meeting.
> > > 
> > > 3.1 libv4l2 support for mc-based hardware
> > > =========================================
> > > 
> > > In order to support those hardware, we'll need to do some redesign
> > > mainly at libv4l2[2].
> > > 
> > > The idea is to work on a new API for libv4l2 that will allow to
> > > split the format conversion on a separate part of it, add support
> > > for DMA Buffer and come up with a way for the library to work
> > > transparently with both devnode-based and mc-based hardware.
> > > 
> > > That envolves adding capacity at libv4l to setup hardware pipelines
> > > and to propagate controls among their sub-devices. Eventually, part
> > > of it will be done in Kernel.
> > > 
> > > That should give performance increase at the library and would allow
> > > gstreamer to use it by default, without compromising performance.
> > > 
> > > [2] I don't discard that some Kernel changes could also be part of
> > > the solution, like, for example, doing control propagation along the
> > > pipeline on simple use case scenarios.
> > > 
> > > 3.2 libv4l2 support for 3A algorithms
> > > =====================================
> > > 
> > > The 3A algorithm handing is highly dependent on the hardware. The
> > > idea here is to allow libv4l to have a set of 3A algorithms that
> > > will be specific to certain mc-based hardware. Ideally, this should
> > > be added in a way that it will allow external closed-source
> > > algorithms to run as well.  
> 

Thanks,
Mauro

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Mauro,

On Thursday, 17 May 2018 22:07:08 EEST Mauro Carvalho Chehab wrote:
> Hi all,
> 
> The goal of this e-mail is to schedule a meeting in order to discuss
> improvements at the media subsystem in order to support complex camera
> hardware by usual apps.
> 
> The main focus here is to allow supporting devices with MC-based
> hardware connected to a camera.
> 
> In short, my proposal is to meet with the interested parties on solving
> this issue during the Open Source Summit in Japan, e. g. between
> June, 19-22, in Tokyo.
> 
> I'd like to know who is interested on joining us for such meeting,
> and to hear a proposal of themes for discussions.

I want to attend the meeting, and will be in Tokyo during that period, so it 
will be easy for me. Please note that I will give a talk at the Automotive 
Linux Summit on Friday the 22nd at 15:15, so I won't be available during that 
time and would prefer the meeting to happen earlier during the week (which 
would also give us an opportunity for more discussions if one day turns out to 
be too short).

> I'm enclosing a detailed description of the problem, in order to
> allow the interested parties to be at the same page.
> 
> Regards,
> Mauro
> 
> ---
> 
> 
> 1. Introduction
> ===============
> 
> 1.1 V4L2 Kernel aspects
> -----------------------
> 
> The media subsystem supports two types of devices:
> 
> - "traditional" media hardware, supported via V4L2 API. On such hardware,
>   opening a single device node (usually /dev/video0) is enough to control
>   the entire device. We call it as devnode-based devices.

To be precise, an application sometimes need to use multiple video nodes with 
devnode-based drivers to capture multiple streams in parallel (when the 
hardware allows it of course).

> - Media-controller based devices. On those devices, there are several
>   /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus a media
>   controller device node (usually /dev/media0).

To be precise again, there could be a single video node and a single subdev 
node, although in practice there will likely be multiple subdevs.

>   We call it as mc-based devices. Controlling the hardware require
>   opening the media device (/dev/media0), setup the pipeline and adjust
>   the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
>   by /dev/video?.

That's the key difference, yes. Both configuration and streaming go through 
the video device node on devnode-based drivers, while video device nodes are 
only used for streaming on mc-based drivers.

> All "standard" media applications, including open source ones (Camorama,
> Cheese, Xawtv, Firefox, Chromium, ...) and closed source ones (Skype,
> Chrome, ...) supports devnode-based devices.

To some extend at least. The closed-source applications tend to have various 
bugs that prevent them from working properly on many devnode-based devices. 
Skype is a good example, we had to include a software scaler in libv4l to make 
it happy. So in practice not everything works smoothly with closed-source 
applications (or even open-source ones) with devnode-based drivers. Where I 
agree is that mc-based drivers are not supported at all by any of the 
"regular" applications.

> Support for mc-based devices currently require an specialized application
> in order to prepare the device for its usage (setup pipelines, adjust
> hardware controls, etc). Once pipeline is set, the streaming goes via
> /dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
> be opened, in order to implement algorithms designed to make video quality
> reasonable. On such devices, it is not uncommon that the device used by the
> application to be a random number (on OMAP3 driver, typically, is either
> /dev/video4 or /dev/video6).

The randomness is not unique to mc-based drivers, there's no guarantee that a 
particular device will have a particular device node index even with devnode-
based drivers.

> One example of such hardware is at the OMAP3-based hardware:
> 
> 	http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png
> 
> On the picture, there's a graph with the hardware blocks in blue/dark/blue
> and the corresponding devnode interfaces in yellow.
> 
> The mc-based approach was taken when support for Nokia N9/N900 cameras
> was added (with has OMAP3 SoC). It is required because the camera hardware
> on SoC comes with a media processor (ISP), with does a lot more than just
> capturing, allowing complex algorithms to enhance image quality in runtime.
> Those algorithms are known as 3A - an acronym for 3 other acronyms:
> 
> 	- AE (Auto Exposure);
> 	- AF (Auto Focus);
> 	- AWB (Auto White Balance).
> 
> Setting a camera with such ISPs are harder because the pipelines to be
> set actually depends the requirements for those 3A algorithms to run.
> Also, usually, the 3A algorithms use some chipset-specific userspace API,
> that exports some image properties, calculated by the ISP, to speed up
> the convergence of those algorithms.

The main reason that drove the MC design is that the 3A algorithms (that is 
the 3A control loop, and sometimes part of the image processing itself) often 
need to run on the CPU. As a kernel-space implementation wasn't possible, we 
needed a lower-level UAPI.

> Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> as binary only blobs, although there are a few OSS implementations.

That's the very unfortunate current state. To the defense of vendors, we so 
far have no proper userspace API to implement 3A libraries. The open-source 
implementations suffer from the same problem. I hope for a positive evolution 
in this domain once we provide a userspace camera stack.

> 1.2 V4L2 userspace aspects
> --------------------------
> 
> Back when USB cameras were introduced, the hardware were really simple:
> they had a CCD camera sensor and a chip that bridges the data though
> USB. CCD camera sensors typically provide data using a bayer format,
> but they usually have their own proprietary ways to pack the data,
> in order to reduce the USB bandwidth (original cameras were USB 1.1).

Do you mean CMOS instead of CCD ?

> So, V4L2 has a myriad of different formats, in order to match each
> CCD camera sensor format.

While several webcams use custom formats, those are often related to custom 
compression performed by the USB bridge, so it's not really related to the 
sensor itself.

> At the end of the day, applications were able to use only a subset of the
> available hardware, since they need to come with format converters for all
> formats the developer uses (usually a very small subset of the available
> ones).

Note that this was only transitory, as webcams then converged towards a small 
set of standard formats (even if recent webcams also implement custom formats, 
but that's usually related to secondary data streams such as depth maps, or to 
non-consumer cameras such as industrial cameras).

> To end with this mess, an userspace library was written, called libv4l.
> It supports all those proprietary formats. So, applications can use
> a RGB or YUV format, without needing to concern about conversions.
> 
> The way it works is by adding wrappers to system calls: open, close,
> ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> at the source code of the apps, all it was needed is to prepend the
> existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.
> 
> All open source apps we know now supports libv4l. On a few (like
> gstreamer), support for it is optional.
> 
> In order to support closed source, another wrapper was added, allowing
> to call any closed source application to use it, by using LD_PRELOAD.
> For example, using skype with it is as simple as calling it with:
> 
> 	$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so /usr/bin/skypeforlinux

It should be noted that libv4l also handles scaling (Skype requires 640x480 if 
I remember correctly, or at least used to, and would crash if given another 
resolution) and rough auto-gain and auto-white balance that are required by 
some cameras to achieve a usable image.

> 2. Current problems
> ===================
> 
> 2.1 Libv4l can slow image handling
> ----------------------------------
> 
> Nowadays, almost all new "simple" cameras are connected via USB using
> the UVC class (USB Video Class). UVC standardized the allowed formats,
> and most apps just implement them. The UVC hardware is more complex,
> having format converters inside it. So, for most usages, format
> conversion isn't needed anymore.
> 
> The need of doing format conversion in software makes libv4l slow,
> requiring lots of CPU usage in order to convert a 4K or 8K format,
> being even worse with 3D cameras.

I would hope that libv4l doesn't perform conversion unnecessarily if not 
needed. The main issue, as I understand it, is that application may fail to 
notice that some formats require software conversion, and prefer them over 
formats provided directly by the device even when such hardware formats are 
supported by the application. We have introduced the V4L2_FMT_FLAG_EMULATED 
flag in the kernel API in order for libv4l to flag emulated formats, but 
applications may fail to see it.

(On a personal note, I think that extending the kernel API with features that 
are not used by the kernel might not be the best API design decision we've 
ever made, it questions whether the libv4l API design was right.)

> Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> DMA_BUFFER currently doesn't work with libv4l.
> 
> Right now, gstreamer defaults to not enable libv4l2, mainly due to
> those performance issues.
> 
> 
> 2.2 Modern hardware is starting to come with "complex" camera ISP
> -----------------------------------------------------------------
> 
> While mc-based devices were limited to SoC, it was easy to
> "delegate" the task of talking with the hardware to the
> embedded hardware designers.
> 
> However, this is changing. Dell Latitude 5285 laptop is a standard
> PC with an i3-core, i5-core or i7-core CPU, with comes with the
> Intel IMU3 ISP hardware[1]
> 
> [1] https://www.spinics.net/lists/linux-usb/msg167478.html

Do you think it's changing, or could that be an exception rather than a future 
rule ?

> There, instead of an USB camera, the hardware is equipped with a
> MC-based ISP, connected to its camera. Currently, despite having
> a Kernel driver for it, the camera doesn't work with any
> userspace application.
> 
> I'm also aware of other projects that are considering the usage of
> mc-based devices for non-dedicated hardware.

What are those projects ?

> 3. How to solve it?
> ===================
> 
> That's the main focus of the meeting :-)
> 
> From a previous discussion I had with media sub-maintainers, there are
> at least two actions that seem required. I'm listing them below as
> an starting point for the discussions, but we can eventually come up
> with some different approach after the meeting.
> 
> 3.1 libv4l2 support for mc-based hardware
> =========================================
> 
> In order to support those hardware, we'll need to do some redesign
> mainly at libv4l2[2].

I wouldn't necessarily focus on libv4l2.

What I believe we need is to design a userspace camera stack that will support 
the features needed by more modern cameras. Let's keep the dogmas aside, if 
libv4l2 turns out to be the right option we can certainly go for it, but if we 
find out that another option is better (Nicolas mentioned Pipewire, I have no 
personal opinion about that yet, and there could be other options too), then 
so be it.

Then, once we get a proper camera stack, we need to support traditional 
applications that use libv4l2 and/or the V4L2 API directly. To that end we 
will need to expose the libv4l2 API on top of the camera stack (through 
libv4l2 if it makes sense, or through a separate implementation) and implement 
a transparent LD_PRELOAD-based library (again either through libv4l2 or 
through a separate implementation).

I'm certainly not calling for libv4l2 to be thrown away, but I think the 
problem is complex enough without adding an artificial requirement of basing 
the camera stack on a particular existing implementation. As long as we 
achieve compatibility with the existing libv4l2 API and the V4L2 API I believe 
we don't need to add any specific implementation restriction. What matters the 
most to me is achieving a maintainable camera stack with a good API that 
supports all our needs, and providing a compatibility layer on top of it.

(On that note, even if we end up using libv4l2 as the base, we will likely nd 
up with a modified and incompatible main API anyway, with the need to provide 
support for the existing API on top. That's why I think it doesn't matter much 
if we decide to refactor libv4l2 or start from scratch and possibly reuse part 
of the libv4l2 code.)

> The idea is to work on a new API for libv4l2 that will allow to
> split the format conversion on a separate part of it, add support
> for DMA Buffer and come up with a way for the library to work
> transparently with both devnode-based and mc-based hardware.
> 
> That envolves adding capacity at libv4l to setup hardware pipelines
> and to propagate controls among their sub-devices. Eventually, part
> of it will be done in Kernel.

Which part do you mean exactly ?

> That should give performance increase at the library and would allow
> gstreamer to use it by default, without compromising performance.
> 
> [2] I don't discard that some Kernel changes could also be part of the
> solution, like, for example, doing control propagation along the pipeline
> on simple use case scenarios.

We've tried that in the past and it was both cumbersome and not very useful. I 
won't rule out kernel changes for the sake of it, but I'd rather focus on use 
cases, and find for each of them the best possible implementation. In my 
opinion we will find that most, if not all, of the new features and changes 
would be better implemented in userspace.

> 3.2 libv4l2 support for 3A algorithms
> =====================================
> 
> The 3A algorithm handing is highly dependent on the hardware. The
> idea here is to allow libv4l to have a set of 3A algorithms that
> will be specific to certain mc-based hardware. Ideally, this should
> be added in a way that it will allow external closed-source
> algorithms to run as well.

As much as I dislike closed-source components, I don't think we "ideally" need 
to support them, I believe they're mandatory if we want any vendor to use our 
camera stack (unless someone can find enough resources to implement a product-
quality 3A library that will work with all ISPs available out there, but I 
don't think that's a reasonable assumption to start with).

My personal requirements for a 3A library are:

- The 3A library API must be standardized, to allow the closed-source vendor 
implementation to be replaced by an open-source implementation should someone 
have the time and energy (and qualifications) to write one.

- Sandboxed execution of the 3A library must be possible as closed-source 
vendor code can't always be blindly trusted (that's an understatement). This 
includes the ability to wrap the library in a daemon should the platform's 
multimedia stack wishes, and to avoid any direct access to the kernel devices 
by the 3A library itself (all accesses should be marshaled by the camera 
stack).

Please note that this daemon is *not* a camera daemon as used by Qualcomm or 
other vendors that communicates with the V4L2 driver through a custom back 
channel. I want the kernel driver to implement the V4L2 API and to be usable 
without any daemon. The decision to run the 3A library in a sandboxed process 
or to call it directly from the camera stack should be left to the camera 
stack and to the platform integrator, and should not be visible by the 3A 
library.

- The 3A library must be usable on major Linux-based camera stacks (the 
Android and Chrome OS camera HALs are certainly important targets, more can be 
added) unmodified, which will allow usage of the vendor binary provided for 
Chrome OS or Android on regular Linux systems.

On that note, I believe it would make sense to design a modular camera stack, 
and try to make most components as platform-independent as possible. This 
should include the kernel drivers (V4L2-compliant and usable without any 
closed-source userspace component), the 3A library, and any other component 
that could be shared (for instance a possible request API library). The rest 
of the code will mostly be glue around those components to integrate them in a 
particular camera stack, and should be as platform-agnostic as possible. 
Ideally I would like to see one Android camera HAL glue that could be used 
with different camera vendors (probably with some kind of vendor-specific 
configuration, or possibly with a separate vendor-specific component to handle 
pipeline configuration).

-- 
Regards,

Laurent Pinchart

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Mauro,

On Friday, 18 May 2018 14:24:47 EEST Mauro Carvalho Chehab wrote:
> Em Fri, 18 May 2018 11:15:39 +0300 Laurent Pinchart escreveu:
> > On Friday, 18 May 2018 00:38:53 EEST Nicolas Dufresne wrote:
> >> Le jeudi 17 mai 2018 à 16:07 -0300, Mauro Carvalho Chehab a écrit :
> >>> Hi all,
> >>> 
> >>> The goal of this e-mail is to schedule a meeting in order to discuss
> >>> improvements at the media subsystem in order to support complex
> >>> camera hardware by usual apps.
> >>> 
> >>> The main focus here is to allow supporting devices with MC-based
> >>> hardware connected to a camera.
> >>> 
> >>> In short, my proposal is to meet with the interested parties on
> >>> solving this issue during the Open Source Summit in Japan, e. g.
> >>> between June, 19-22, in Tokyo.
> >>> 
> >>> I'd like to know who is interested on joining us for such meeting,
> >> and to hear a proposal of themes for discussions.
> >>> 
> >>> I'm enclosing a detailed description of the problem, in order to
> >>> allow the interested parties to be at the same page.
> >> 
> >> It's unlikely I'll be able to attend this meeting, but I'd like to
> >> provide some initial input on this. Find inline some clarification on
> >> why libv4l2 is disabled by default in Gst, as it's not just
> >> performance.
> 
> Thanks for complementing it!
> 
> >> A major aspect that is totally absent of this mail is PipeWire. With
> >> the venue of sandboxed application, there is a need to control access
> >> to cameras through a daemon. The same daemon is also used to control
> >> access to screen capture on Wayland (instead of letting any random
> >> application capture your screen, like on X11). The effort is lead by
> >> the desktop team at RedHat (folks CCed). PipeWire already have V4L2
> >> native support and is integrated in GStreamer already in a way that it
> >> can totally replace the V4L2 capture component there. PipeWire is
> >> plugin base, so more type of camera support (including proprietary
> >> ones) can be added.
> >
> > One issue that has been worrying me for the past five years or so is how
> > to ensure that we will continue having open-source camera support in the
> > future. Pipewire is just a technology and as such can be used in good or
> > evil ways, but as a community we need to care about availability of open
> > solutions.
> > 
> > So far, by pushing the V4L2 API as the proper way to support cameras, we
> > have tried to resist the natural inclination of vendors to close
> > everything, as implementing a closed-source kernel driver isn't an option
> > that most would consider. Of course, the drawback is that some vendors
> > have simply decided not to care about upstream camera support.
> > 
> > If we move the camera API one level up to userspace (and whether the API
> > will be defined by Pipewire, by libv4l or by something else), we'll make
> > it easier for vendors not to play along. My big question is how to
> > prevent that. I think there's still value there in mandating V4L2 as the
> > only API for cameras, and in ensuring that we support multiple userspace
> > multimedia stacks, not just Pipewire (this is already done in a way, as I
> > don't foresee Android moving away from their camera HAL in the near
> > future). That will likely not be enough, and I'd like to hear other
> > people's opinions on this topic.
> > 
> > I would like to emphasize that I don't expect vendors to open the
> > implementation of their 3A algorithms, and I'm not actually concerned
> > about that part. If that's the only part shipped as closed-source, and if
> > the hardware operation is documented (ideally in public datasheet, but at
> > a minimum with proper documentation of custom ioctls used to configure the
> > hardware), then the community will have the opportunity to implement an
> > open- source 3A library. My main concern is thus about all component
> > other than the 3A library.
> 
> Yeah, I share the same concern. Whatever solution we take, we should
> do our best to ensure that the camera driver will be open and it would
> be possible to have open-sourced 3A libraries as alternatives.
> 
> One of the biggest reasons why we decided to start libv4l project,
> in the past, was to ensure an open source solution. The problem we
> faced on that time is to ensure that, when a new media driver were
> added with some proprietary output format, an open source decoding
> software were also added at libv4l.
> 
> This approach ensured that all non-MC cameras are supported by all
> V4L2 applications.
> 
> Before libv4l, media support for a given device were limited to a few
> apps that knew how to decode the format. There were even cases were a
> proprietary app were required, as no open source decoders were available.
> 
> From my PoV, the biggest gain with libv4l is that the same group of
> maintainers can ensure that the entire solution (Kernel driver and
> low level userspace support) will provide everything required for an
> open source app to work with it.
> 
> I'm not sure how we would keep enforcing it if the pipeline setting
> and control propagation logic for an specific hardware will be
> delegated to PipeWire. It seems easier to keep doing it on a libv4l
> (version 2) and let PipeWire to use it.

I believe we need to first study pipewire in more details. I have no personal 
opinion yet as I haven't had time to investigate it. That being said, I don't 
think that libv4l with closed-source plugins would be much better than a 
closed-source pipewire plugin. What main concern once we provide a userspace 
camera stack API is that vendors might implement that API in a closed-source 
component that calls to a kernel driver implementing a custom API, with all 
knowledge about the camera located in the closed-source component. I'm not 
sure how to prevent that, my best proposal would be to make V4L2 so useful 
that vendors wouldn't even think about a different solution (possibly coupled 
by the pressure put by platform vendors such as Google who mandate upstream 
kernel drivers for Chrome OS, but that's still limited as even when it comes 
to Google there's no such pressure on the Android side).

> >> Remote daemon can also provide streams, as this is the case for
> >> compositors and screen casting. An extra benefit is that you can have
> >> multiple application reading frames from the same camera. It also allow
> >> sandboxed application (the do not have access to /dev) to use the
> >> cameras. PipeWire is much more then that, but let's focus on that.
> >> 
> >> This is the direction we are heading on the "generic" / Desktop Linux.
> >> Porting Firefox and Chrome is obviously planed, as these beast are
> >> clear candidate for being sand-boxed and requires screen share feature
> >> for WebRTC.
> >> 
> >> In this context, proprietary or HW specific algorithm could be
> >> implemented in userspace as PipeWire plugins, and then application will
> >> automatically be enable to enumerate and use these. I'm not saying the
> >> libv4l2 stuff is not needed short term, but it's just a short term
> >> thing in my opinion.
> >> 
> >>> 1. Introduction
> >>> ===============
> >>> 
> >>> 1.1 V4L2 Kernel aspects
> >>> -----------------------
> >>> 
> >>> The media subsystem supports two types of devices:
> >>> 
> >>> - "traditional" media hardware, supported via V4L2 API. On such
> >>>   hardware, opening a single device node (usually /dev/video0) is
> >>>   enough to control the entire device. We call it as devnode-based
> >>>   devices.
> >>> 
> >>> - Media-controller based devices. On those devices, there are several
> >>>   /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus a media
> >>>   controller device node (usually /dev/media0).
> >>>   We call it as mc-based devices. Controlling the hardware require
> >>>   opening the media device (/dev/media0), setup the pipeline and
> >>>   adjust the sub-devices via /dev/v4l2-subdev?. Only streaming is
> >>>   controlled by /dev/video?.
> >>> 
> >>> All "standard" media applications, including open source ones
> >>> (Camorama, Cheese, Xawtv, Firefox, Chromium, ...) and closed source
> >>> ones (Skype, Chrome, ...) supports devnode-based devices.
> >>> 
> >>> Support for mc-based devices currently require an specialized
> >>> application in order to prepare the device for its usage (setup
> >>> pipelines, adjust hardware controls, etc). Once pipeline is set, the
> >>> streaming goes via /dev/video?, although usually some /dev/v4l2-subdev?
> >>> devnodes should also be opened, in order to implement algorithms
> >>> designed to make video quality reasonable. On such devices, it is not
> >>> uncommon that the device used by the application to be a random number
> >>> (on OMAP3 driver, typically, is either /dev/video4 or /dev/video6).
> >>> 
> >>> One example of such hardware is at the OMAP3-based hardware:
> >>> 	http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with
> >>> 
> >>> -tvp5150.png
> >>> 
> >>> On the picture, there's a graph with the hardware blocks in
> >>> blue/dark/blue and the corresponding devnode interfaces in yellow.
> >>> 
> >>> The mc-based approach was taken when support for Nokia N9/N900
> >>> cameras was added (with has OMAP3 SoC). It is required because the
> >>> camera hardware on SoC comes with a media processor (ISP), with does a
> >>> lot more than just capturing, allowing complex algorithms to enhance
> >>> image quality in runtime.
> >>> 
> >>> Those algorithms are known as 3A - an acronym for 3 other acronyms:
> >>> 	- AE (Auto Exposure);
> >>> 	- AF (Auto Focus);
> >>> 	- AWB (Auto White Balance).
> >>> 
> >>> Setting a camera with such ISPs are harder because the pipelines to
> >>> be set actually depends the requirements for those 3A algorithms to
> >>> run. Also, usually, the 3A algorithms use some chipset-specific
> >>> userspace API, that exports some image properties, calculated by the
> >>> ISP, to speed up the convergence of those algorithms.
> >>> 
> >>> Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> >>> as binary only blobs, although there are a few OSS implementations.
> >>> 
> >>> 1.2 V4L2 userspace aspects
> >>> --------------------------
> >>> 
> >>> Back when USB cameras were introduced, the hardware were really
> >>> simple: they had a CCD camera sensor and a chip that bridges the data
> >>> though USB. CCD camera sensors typically provide data using a bayer
> >>> format, but they usually have their own proprietary ways to pack the
> >>> data, in order to reduce the USB bandwidth (original cameras were USB
> >>> 1.1).
> >>> 
> >>> So, V4L2 has a myriad of different formats, in order to match each
> >>> CCD camera sensor format. At the end of the day, applications were
> >>> able to use only a subset of the available hardware, since they need
> >>> to come with format converters for all formats the developer uses
> >>> (usually a very small subset of the available ones).
> >>> 
> >>> To end with this mess, an userspace library was written, called
> >>> libv4l. It supports all those proprietary formats. So, applications
> >>> can use a RGB or YUV format, without needing to concern about
> >>> conversions.
> >>> 
> >>> The way it works is by adding wrappers to system calls: open, close,
> >>> ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> >>> at the source code of the apps, all it was needed is to prepend the
> >>> existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.
> >>> 
> >>> All open source apps we know now supports libv4l. On a few (like
> >>> gstreamer), support for it is optional.
> >>> 
> >>> In order to support closed source, another wrapper was added,
> >>> allowing to call any closed source application to use it, by using
> >>> LD_PRELOAD.
> >>> 
> >>> For example, using skype with it is as simple as calling it with:
> >>> 	$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so
> >>> 
> >>> /usr/bin/skypeforlinux
> >>> 
> >>> 2. Current problems
> >>> ===================
> >>> 
> >>> 2.1 Libv4l can slow image handling
> >>> ----------------------------------
> >>> 
> >>> Nowadays, almost all new "simple" cameras are connected via USB using
> >>> the UVC class (USB Video Class). UVC standardized the allowed
> >>> formats, and most apps just implement them. The UVC hardware is more
> >>> complex, having format converters inside it. So, for most usages,
> >>> format conversion isn't needed anymore.
> >>> 
> >>> The need of doing format conversion in software makes libv4l slow,
> >>> requiring lots of CPU usage in order to convert a 4K or 8K format,
> >>> being even worse with 3D cameras.
> >>> 
> >>> Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> >>> DMA_BUFFER currently doesn't work with libv4l.
> >>> 
> >>> Right now, gstreamer defaults to not enable libv4l2, mainly due to
> >>> those performance issues.
> >> 
> >> I need to clarify a little bit on why we disabled libv4l2 in GStreamer,
> >> as it's not only for performance reason, there is couple of major
> >> issues in the libv4l2 implementation that get's in way. Just a short
> >> list:
> >> 
> >>   - Crash when CREATE_BUFS is being used
> >>   - Crash in the jpeg decoder (when frames are corrupted)
> >>   - App exporting DMABuf need to be aware of emulation, otherwise the
> >>     DMABuf exported are in the orignal format
> >>   - RW emulation only initialize the queue on first read (causing
> >>     userspace poll() to fail)
> >>   - Signature of v4l2_mmap does not match mmap() (minor)
> >>   - The colorimetry does not seem emulated when conversion
> >>   - Sub-optimal locking (at least deadlocks were fixed)
> > 
> > Do you see any point in that list that couldn't be fixed in libv4l ?
> > 
> >> Except for the colorimetry (which causes negotiation failure, as it
> >> causes invalid colorimetry / format matches), these issues are already
> >> worked around in GStreamer, but with the lost of features of course.
> >> There is other cases were something worked without libv4l2, but didn't
> >> work with libv4l2, but we haven't tracked down the cause.
> >> 
> >> For people working on this venue, since 1.14, you can enable libv4l2 at
> >> run-time using env GST_V4L2_USE_LIBV4L2=1.
> >> 
> >>> 2.2 Modern hardware is starting to come with "complex" camera ISP
> >>> -----------------------------------------------------------------
> >>> 
> >>> While mc-based devices were limited to SoC, it was easy to
> >>> "delegate" the task of talking with the hardware to the
> >>> embedded hardware designers.
> >>> 
> >>> However, this is changing. Dell Latitude 5285 laptop is a standard
> >>> PC with an i3-core, i5-core or i7-core CPU, with comes with the
> >>> Intel IMU3 ISP hardware[1]
> >>> 
> >>> [1] https://www.spinics.net/lists/linux-usb/msg167478.html
> >>> 
> >>> There, instead of an USB camera, the hardware is equipped with a
> >>> MC-based ISP, connected to its camera. Currently, despite having
> >>> a Kernel driver for it, the camera doesn't work with any
> >>> userspace application.
> >>> 
> >>> I'm also aware of other projects that are considering the usage of
> >>> mc-based devices for non-dedicated hardware.
> >>> 
> >>> 3. How to solve it?
> >>> ===================
> >>> 
> >>> That's the main focus of the meeting :-)
> >>> 
> >>> From a previous discussion I had with media sub-maintainers, there
> >>> are at least two actions that seem required. I'm listing them below as
> >>> an starting point for the discussions, but we can eventually come up
> >>> with some different approach after the meeting.
> >>> 
> >>> 3.1 libv4l2 support for mc-based hardware
> >>> =========================================
> >>> 
> >>> In order to support those hardware, we'll need to do some redesign
> >>> mainly at libv4l2[2].
> >>> 
> >>> The idea is to work on a new API for libv4l2 that will allow to
> >>> split the format conversion on a separate part of it, add support
> >>> for DMA Buffer and come up with a way for the library to work
> >>> transparently with both devnode-based and mc-based hardware.
> >>> 
> >>> That envolves adding capacity at libv4l to setup hardware pipelines
> >>> and to propagate controls among their sub-devices. Eventually, part
> >>> of it will be done in Kernel.
> >>> 
> >>> That should give performance increase at the library and would allow
> >>> gstreamer to use it by default, without compromising performance.
> >>> 
> >>> [2] I don't discard that some Kernel changes could also be part of
> >>> the solution, like, for example, doing control propagation along the
> >>> pipeline on simple use case scenarios.
> >>> 
> >>> 3.2 libv4l2 support for 3A algorithms
> >>> =====================================
> >>> 
> >>> The 3A algorithm handing is highly dependent on the hardware. The
> >>> idea here is to allow libv4l to have a set of 3A algorithms that
> >>> will be specific to certain mc-based hardware. Ideally, this should
> >>> be added in a way that it will allow external closed-source
> >>> algorithms to run as well.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 2303 bytes --]

Le vendredi 18 mai 2018 à 11:15 +0300, Laurent Pinchart a écrit :
> > I need to clarify a little bit on why we disabled libv4l2 in
> > GStreamer,
> > as it's not only for performance reason, there is couple of major
> > issues in the libv4l2 implementation that get's in way. Just a
> > short
> > list:
> > 
> > 
> 
> Do you see any point in that list that couldn't be fixed in libv4l ?

Sure, most of it is features being added into the kernel uAPI but not
added to the emulation layer. But appart from that, libv4l will only
offer legacy use case, we need to think how generic userspace will be
able to access these camera, and leverage the per request controls,
multi-stream, etc. features. This is mostly what Android Camera HAL2
does (and it does it well), but it won't try and ensure this stays Open
Source in any ways. I would not mind if Android Camera HAL2 leads the
way, and a facilitator (something that does 90% of the work if you have
a proper Open Source driver) would lead the way in getting more OSS
drivers submitted.

> >    - Crash when CREATE_BUFS is being used

This is a side effect of CREATE_BUFS being passed-through, implementing
emulation for this should be straightforward.

> >    - Crash in the jpeg decoder (when frames are corrupted)

A minimalist framing parser would detect just enough of this, and would
fix it.

> >    - App exporting DMABuf need to be aware of emulation, otherwise the
> >      DMABuf exported are in the orignal format

libv4l2 can return ENOTTY to expbufs calls in 

> >    - RW emulation only initialize the queue on first read (causing
> >      userspace poll() to fail)

This is not fixable, only place it would be fixed is by moving this
emulation into VideoBuf2. That would assume someone do care about RW
(even though it could be nicer uAPI when dealing with muxed or byte-
stream type of data).

> >    - Signature of v4l2_mmap does not match mmap() (minor)
> >    - The colorimetry does not seem emulated when conversion

This one is probably tricky, special is the converter plugin API is
considered stable. Maybe just resetting everything to DEFAULT would
work ?

> >    - Sub-optimal locking (at least deadlocks were fixed)

Need more investigation really, and proper measurement.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Mauro Carvalho Chehab]

Em Fri, 18 May 2018 15:27:24 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> escreveu:

> Hi Mauro,
> 
> On Thursday, 17 May 2018 22:07:08 EEST Mauro Carvalho Chehab wrote:
> > Hi all,
> > 
> > The goal of this e-mail is to schedule a meeting in order to discuss
> > improvements at the media subsystem in order to support complex camera
> > hardware by usual apps.
> > 
> > The main focus here is to allow supporting devices with MC-based
> > hardware connected to a camera.
> > 
> > In short, my proposal is to meet with the interested parties on solving
> > this issue during the Open Source Summit in Japan, e. g. between
> > June, 19-22, in Tokyo.
> > 
> > I'd like to know who is interested on joining us for such meeting,
> > and to hear a proposal of themes for discussions.  
> 
> I want to attend the meeting, and will be in Tokyo during that period, so it 
> will be easy for me. Please note that I will give a talk at the Automotive 
> Linux Summit on Friday the 22nd at 15:15, so I won't be available during that 
> time and would prefer the meeting to happen earlier during the week (which 
> would also give us an opportunity for more discussions if one day turns out to 
> be too short).
> 
> > I'm enclosing a detailed description of the problem, in order to
> > allow the interested parties to be at the same page.
> > 
> > Regards,
> > Mauro
> > 
> > ---
> > 
> > 
> > 1. Introduction
> > ===============
> > 
> > 1.1 V4L2 Kernel aspects
> > -----------------------
> > 
> > The media subsystem supports two types of devices:
> > 
> > - "traditional" media hardware, supported via V4L2 API. On such hardware,
> >   opening a single device node (usually /dev/video0) is enough to control
> >   the entire device. We call it as devnode-based devices.  
> 
> To be precise, an application sometimes need to use multiple video nodes with 
> devnode-based drivers to capture multiple streams in parallel (when the 
> hardware allows it of course).
> 
> > - Media-controller based devices. On those devices, there are several
> >   /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus a media
> >   controller device node (usually /dev/media0).  
> 
> To be precise again, there could be a single video node and a single subdev 
> node, although in practice there will likely be multiple subdevs.
> 
> >   We call it as mc-based devices. Controlling the hardware require
> >   opening the media device (/dev/media0), setup the pipeline and adjust
> >   the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
> >   by /dev/video?.  
> 
> That's the key difference, yes. Both configuration and streaming go through 
> the video device node on devnode-based drivers, while video device nodes are 
> only used for streaming on mc-based drivers.
> 
> > All "standard" media applications, including open source ones (Camorama,
> > Cheese, Xawtv, Firefox, Chromium, ...) and closed source ones (Skype,
> > Chrome, ...) supports devnode-based devices.  
> 
> To some extend at least. The closed-source applications tend to have various 
> bugs that prevent them from working properly on many devnode-based devices. 
> Skype is a good example, we had to include a software scaler in libv4l to make 
> it happy. So in practice not everything works smoothly with closed-source 
> applications (or even open-source ones) with devnode-based drivers. Where I 
> agree is that mc-based drivers are not supported at all by any of the 
> "regular" applications.
> 
> > Support for mc-based devices currently require an specialized application
> > in order to prepare the device for its usage (setup pipelines, adjust
> > hardware controls, etc). Once pipeline is set, the streaming goes via
> > /dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
> > be opened, in order to implement algorithms designed to make video quality
> > reasonable. On such devices, it is not uncommon that the device used by the
> > application to be a random number (on OMAP3 driver, typically, is either
> > /dev/video4 or /dev/video6).  
> 
> The randomness is not unique to mc-based drivers, there's no guarantee that a 
> particular device will have a particular device node index even with devnode-
> based drivers.
> 
> > One example of such hardware is at the OMAP3-based hardware:
> > 
> > 	http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png
> > 
> > On the picture, there's a graph with the hardware blocks in blue/dark/blue
> > and the corresponding devnode interfaces in yellow.
> > 
> > The mc-based approach was taken when support for Nokia N9/N900 cameras
> > was added (with has OMAP3 SoC). It is required because the camera hardware
> > on SoC comes with a media processor (ISP), with does a lot more than just
> > capturing, allowing complex algorithms to enhance image quality in runtime.
> > Those algorithms are known as 3A - an acronym for 3 other acronyms:
> > 
> > 	- AE (Auto Exposure);
> > 	- AF (Auto Focus);
> > 	- AWB (Auto White Balance).
> > 
> > Setting a camera with such ISPs are harder because the pipelines to be
> > set actually depends the requirements for those 3A algorithms to run.
> > Also, usually, the 3A algorithms use some chipset-specific userspace API,
> > that exports some image properties, calculated by the ISP, to speed up
> > the convergence of those algorithms.  
> 
> The main reason that drove the MC design is that the 3A algorithms (that is 
> the 3A control loop, and sometimes part of the image processing itself) often 
> need to run on the CPU. As a kernel-space implementation wasn't possible, we 
> needed a lower-level UAPI.
> 
> > Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> > as binary only blobs, although there are a few OSS implementations.  
> 
> That's the very unfortunate current state. To the defense of vendors, we so 
> far have no proper userspace API to implement 3A libraries. The open-source 
> implementations suffer from the same problem. I hope for a positive evolution 
> in this domain once we provide a userspace camera stack.
> 
> > 1.2 V4L2 userspace aspects
> > --------------------------
> > 
> > Back when USB cameras were introduced, the hardware were really simple:
> > they had a CCD camera sensor and a chip that bridges the data though
> > USB. CCD camera sensors typically provide data using a bayer format,
> > but they usually have their own proprietary ways to pack the data,
> > in order to reduce the USB bandwidth (original cameras were USB 1.1).  
> 
> Do you mean CMOS instead of CCD ?
> 
> > So, V4L2 has a myriad of different formats, in order to match each
> > CCD camera sensor format.  
> 
> While several webcams use custom formats, those are often related to custom 
> compression performed by the USB bridge, so it's not really related to the 
> sensor itself.

Yes, on several cases, the custom format is at the bridge, and not at
the sensor itself. Anyway, the point is that format conversion used to
be a key feature in order for video apps to work with different cameras.

> > At the end of the day, applications were able to use only a subset of the
> > available hardware, since they need to come with format converters for all
> > formats the developer uses (usually a very small subset of the available
> > ones).  
> 
> Note that this was only transitory, as webcams then converged towards a small 
> set of standard formats (even if recent webcams also implement custom formats, 
> but that's usually related to secondary data streams such as depth maps, or to 
> non-consumer cameras such as industrial cameras).

We are still adding new fourcc formats nowadays. IMHO, there's now a 
certain stability/convergence on formats, although newer cameras with 
advanced features (like 3D and light-field) will very likely require new
formats.

> > To end with this mess, an userspace library was written, called libv4l.
> > It supports all those proprietary formats. So, applications can use
> > a RGB or YUV format, without needing to concern about conversions.
> > 
> > The way it works is by adding wrappers to system calls: open, close,
> > ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> > at the source code of the apps, all it was needed is to prepend the
> > existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.
> > 
> > All open source apps we know now supports libv4l. On a few (like
> > gstreamer), support for it is optional.
> > 
> > In order to support closed source, another wrapper was added, allowing
> > to call any closed source application to use it, by using LD_PRELOAD.
> > For example, using skype with it is as simple as calling it with:
> > 
> > 	$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so /usr/bin/skypeforlinux  
> 
> It should be noted that libv4l also handles scaling (Skype requires 640x480 if 
> I remember correctly, or at least used to, and would crash if given another 
> resolution) and rough auto-gain and auto-white balance that are required by 
> some cameras to achieve a usable image.
> 
> > 2. Current problems
> > ===================
> > 
> > 2.1 Libv4l can slow image handling
> > ----------------------------------
> > 
> > Nowadays, almost all new "simple" cameras are connected via USB using
> > the UVC class (USB Video Class). UVC standardized the allowed formats,
> > and most apps just implement them. The UVC hardware is more complex,
> > having format converters inside it. So, for most usages, format
> > conversion isn't needed anymore.
> > 
> > The need of doing format conversion in software makes libv4l slow,
> > requiring lots of CPU usage in order to convert a 4K or 8K format,
> > being even worse with 3D cameras.  
> 
> I would hope that libv4l doesn't perform conversion unnecessarily if not 
> needed. The main issue, as I understand it, is that application may fail to 
> notice that some formats require software conversion, and prefer them over 
> formats provided directly by the device even when such hardware formats are 
> supported by the application. We have introduced the V4L2_FMT_FLAG_EMULATED 
> flag in the kernel API in order for libv4l to flag emulated formats, but 
> applications may fail to see it.

I don't see how an application may fail to see it. If libv4l is used, the
flag will be there. This is not a new flag. Apps can easily look into it
and opt to use non-emulated formats. On non-emulated formats, there won't
be unnecessarily format conversions.

Granted, applications may opt to not use it, and just select an specific
format, but it is an application choice to either use it or not.

> (On a personal note, I think that extending the kernel API with features that 
> are not used by the kernel might not be the best API design decision we've 
> ever made, it questions whether the libv4l API design was right.)
> 
> > Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> > DMA_BUFFER currently doesn't work with libv4l.
> > 
> > Right now, gstreamer defaults to not enable libv4l2, mainly due to
> > those performance issues.
> > 
> > 
> > 2.2 Modern hardware is starting to come with "complex" camera ISP
> > -----------------------------------------------------------------
> > 
> > While mc-based devices were limited to SoC, it was easy to
> > "delegate" the task of talking with the hardware to the
> > embedded hardware designers.
> > 
> > However, this is changing. Dell Latitude 5285 laptop is a standard
> > PC with an i3-core, i5-core or i7-core CPU, with comes with the
> > Intel IMU3 ISP hardware[1]

Typo:
	IMU3 -> IPU3

> > 
> > [1] https://www.spinics.net/lists/linux-usb/msg167478.html  
> 
> Do you think it's changing, or could that be an exception rather than a future 
> rule ?

Well, on ARM, all SoC chipsets I'm aware have an ISP IP block. As Intel is 
now adding IPU3 on some of their chipsets, I think that we'll see more and
more hardware with it.

I don't have a crystal ball, but I guess that the trend would be to keep using
USB with UVC for external cameras, but some ISP-based hardware for the
internal ones.

> 
> > There, instead of an USB camera, the hardware is equipped with a
> > MC-based ISP, connected to its camera. Currently, despite having
> > a Kernel driver for it, the camera doesn't work with any
> > userspace application.
> > 
> > I'm also aware of other projects that are considering the usage of
> > mc-based devices for non-dedicated hardware.  
> 
> What are those projects ?

Well, cheap ARM-based hardware like RPi3 already has this issue: they
have an ISP (or some GPU firmware meant to emulate an ISP). While
those hardware could have multiple sensors, typically they have just
one.

> 
> > 3. How to solve it?
> > ===================
> > 
> > That's the main focus of the meeting :-)
> > 
> > From a previous discussion I had with media sub-maintainers, there are
> > at least two actions that seem required. I'm listing them below as
> > an starting point for the discussions, but we can eventually come up
> > with some different approach after the meeting.
> > 
> > 3.1 libv4l2 support for mc-based hardware
> > =========================================
> > 
> > In order to support those hardware, we'll need to do some redesign
> > mainly at libv4l2[2].  
> 
> I wouldn't necessarily focus on libv4l2.
> 
> What I believe we need is to design a userspace camera stack that will support 
> the features needed by more modern cameras. Let's keep the dogmas aside, if 
> libv4l2 turns out to be the right option we can certainly go for it, but if we 
> find out that another option is better (Nicolas mentioned Pipewire, I have no 
> personal opinion about that yet, and there could be other options too), then 
> so be it.
> Then, once we get a proper camera stack, we need to support traditional 
> applications that use libv4l2 and/or the V4L2 API directly. To that end we 
> will need to expose the libv4l2 API on top of the camera stack (through 
> libv4l2 if it makes sense, or through a separate implementation) and implement 
> a transparent LD_PRELOAD-based library (again either through libv4l2 or 
> through a separate implementation).
> 
> I'm certainly not calling for libv4l2 to be thrown away, but I think the 
> problem is complex enough without adding an artificial requirement of basing 
> the camera stack on a particular existing implementation. As long as we 
> achieve compatibility with the existing libv4l2 API and the V4L2 API I believe 
> we don't need to add any specific implementation restriction. What matters the 
> most to me is achieving a maintainable camera stack with a good API that 
> supports all our needs, and providing a compatibility layer on top of it.
> 
> (On that note, even if we end up using libv4l2 as the base, we will likely nd 
> up with a modified and incompatible main API anyway, with the need to provide 
> support for the existing API on top. That's why I think it doesn't matter much 
> if we decide to refactor libv4l2 or start from scratch and possibly reuse part 
> of the libv4l2 code.)
> 

Yeah, agreed. On this stage, we shouldn't be limited to libv4l2 or its
current implementation. The hole idea of the discussions is to understand
the alternatives and chose a clear path for us to fix the issue.

> > The idea is to work on a new API for libv4l2 that will allow to
> > split the format conversion on a separate part of it, add support
> > for DMA Buffer and come up with a way for the library to work
> > transparently with both devnode-based and mc-based hardware.
> > 
> > That envolves adding capacity at libv4l to setup hardware pipelines
> > and to propagate controls among their sub-devices. Eventually, part
> > of it will be done in Kernel.  
> 
> Which part do you mean exactly ?

I don't mean any specific part. Just saying that we shouldn't discard that
some Kernel support could be implemented in order to make easier to 
implement/maintain it.

> > That should give performance increase at the library and would allow
> > gstreamer to use it by default, without compromising performance.
> > 
> > [2] I don't discard that some Kernel changes could also be part of the
> > solution, like, for example, doing control propagation along the pipeline
> > on simple use case scenarios.  
> 
> We've tried that in the past and it was both cumbersome and not very useful. I 
> won't rule out kernel changes for the sake of it, but I'd rather focus on use 
> cases, and find for each of them the best possible implementation. In my 
> opinion we will find that most, if not all, of the new features and changes 
> would be better implemented in userspace.
> 
> > 3.2 libv4l2 support for 3A algorithms
> > =====================================
> > 
> > The 3A algorithm handing is highly dependent on the hardware. The
> > idea here is to allow libv4l to have a set of 3A algorithms that
> > will be specific to certain mc-based hardware. Ideally, this should
> > be added in a way that it will allow external closed-source
> > algorithms to run as well.  
> 
> As much as I dislike closed-source components, I don't think we "ideally" need 
> to support them, I believe they're mandatory if we want any vendor to use our 
> camera stack (unless someone can find enough resources to implement a product-
> quality 3A library that will work with all ISPs available out there, but I 
> don't think that's a reasonable assumption to start with).
> 
> My personal requirements for a 3A library are:
> 
> - The 3A library API must be standardized, to allow the closed-source vendor 
> implementation to be replaced by an open-source implementation should someone 
> have the time and energy (and qualifications) to write one.
> 
> - Sandboxed execution of the 3A library must be possible as closed-source 
> vendor code can't always be blindly trusted (that's an understatement). This 
> includes the ability to wrap the library in a daemon should the platform's 
> multimedia stack wishes, and to avoid any direct access to the kernel devices 
> by the 3A library itself (all accesses should be marshaled by the camera 
> stack).
> 
> Please note that this daemon is *not* a camera daemon as used by Qualcomm or 
> other vendors that communicates with the V4L2 driver through a custom back 
> channel. I want the kernel driver to implement the V4L2 API and to be usable 
> without any daemon. The decision to run the 3A library in a sandboxed process 
> or to call it directly from the camera stack should be left to the camera 
> stack and to the platform integrator, and should not be visible by the 3A 
> library.
> 
> - The 3A library must be usable on major Linux-based camera stacks (the 
> Android and Chrome OS camera HALs are certainly important targets, more can be 
> added) unmodified, which will allow usage of the vendor binary provided for 
> Chrome OS or Android on regular Linux systems.

One of the issues with closed-source components is that they usually
have dependencies that end being distro-dependent and arch-dependent [1].

[1] on a side note, I have here one example of a hardware that depends on
a closed-source plugin: a HP Laserjet USB printer/scanner. I would love
to use it connected to an Exynos hardware here, but the closed-source
plugin only works on x86, forcing me to connect to one desktop. The side
effect is that, every time I upgrade CUPS, the printer stops working, 
as it waits for me to download the newest plugin that is compatible with
such version. That's specially evil when I travel, as other people here
won't be able to print anything, until I remotely connect to it and
upgrade the plugin. Really nasty.

> On that note, I believe it would make sense to design a modular camera stack, 
> and try to make most components as platform-independent as possible. This 
> should include the kernel drivers (V4L2-compliant and usable without any 
> closed-source userspace component), the 3A library, and any other component 
> that could be shared (for instance a possible request API library). The rest 
> of the code will mostly be glue around those components to integrate them in a 
> particular camera stack, and should be as platform-agnostic as possible. 
> Ideally I would like to see one Android camera HAL glue that could be used 
> with different camera vendors (probably with some kind of vendor-specific 
> configuration, or possibly with a separate vendor-specific component to handle 
> pipeline configuration).

Makes sense.

Thanks,
Mauro

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 2731 bytes --]

Le vendredi 18 mai 2018 à 15:38 +0300, Laurent Pinchart a écrit :
> > Before libv4l, media support for a given device were limited to a few
> > apps that knew how to decode the format. There were even cases were a
> > proprietary app were required, as no open source decoders were available.
> > 
> > From my PoV, the biggest gain with libv4l is that the same group of
> > maintainers can ensure that the entire solution (Kernel driver and
> > low level userspace support) will provide everything required for an
> > open source app to work with it.
> > 
> > I'm not sure how we would keep enforcing it if the pipeline setting
> > and control propagation logic for an specific hardware will be
> > delegated to PipeWire. It seems easier to keep doing it on a libv4l
> > (version 2) and let PipeWire to use it.
> 
> I believe we need to first study pipewire in more details. I have no personal 
> opinion yet as I haven't had time to investigate it. That being said, I don't 
> think that libv4l with closed-source plugins would be much better than a 
> closed-source pipewire plugin. What main concern once we provide a userspace 
> camera stack API is that vendors might implement that API in a closed-source 
> component that calls to a kernel driver implementing a custom API, with all 
> knowledge about the camera located in the closed-source component. I'm not 
> sure how to prevent that, my best proposal would be to make V4L2 so useful 
> that vendors wouldn't even think about a different solution (possibly coupled 
> by the pressure put by platform vendors such as Google who mandate upstream 
> kernel drivers for Chrome OS, but that's still limited as even when it comes 
> to Google there's no such pressure on the Android side).

If there is proprietary plugins, then I don't think it will make any
difference were this is implemented. The difference is the feature set
we expose. 3A is per device, but multiple streams, with per request
controls is also possible. PipeWire gives central place to manage this,
while giving multiple process access to the camera streams. I think in
the end, what fits better would be something like or the Android Camera
HAL2. But we could encourage OSS by maintaining a base implementation
that covers all the V4L2 aspect, leaving only the 3A aspect of the work
to be done. Maybe we need to come up with an abstraction that does not
prevent multi-streams, but only requires 3A per vendors (saying per
vendors, as some of this could be Open Source by third parties).

just thinking out loud now ;-P
Nicolas

p.s. Do we have the Intel / IPU3 folks in in the loop ? This is likely
the most pressing HW as it's shipping on many laptops now.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Nicolas,

On Friday, 18 May 2018 17:22:47 EEST Nicolas Dufresne wrote:
> Le vendredi 18 mai 2018 à 11:15 +0300, Laurent Pinchart a écrit :
> >> I need to clarify a little bit on why we disabled libv4l2 in GStreamer,
> >> as it's not only for performance reason, there is couple of major issues
> >> in the libv4l2 implementation that get's in way. Just a short
> >> list:
> > 
> > Do you see any point in that list that couldn't be fixed in libv4l ?
> 
> Sure, most of it is features being added into the kernel uAPI but not
> added to the emulation layer. But appart from that, libv4l will only
> offer legacy use case, we need to think how generic userspace will be
> able to access these camera, and leverage the per request controls,
> multi-stream, etc. features. This is mostly what Android Camera HAL2
> does (and it does it well), but it won't try and ensure this stays Open
> Source in any ways. I would not mind if Android Camera HAL2 leads the
> way, and a facilitator (something that does 90% of the work if you have
> a proper Open Source driver) would lead the way in getting more OSS
> drivers submitted.

There are a few issues with the Android camera HAL that make implementations 
very painful. If we were to model a camera stack on such an API, we should at 
least fix those. The biggest issue in my opinion is that the HAL mandates that 
a request captures to a specific buffer, making implementations very complex, 
and requiring memcpy() from time to time when losing race conditions. It would 
be much simpler to instead require capture to any buffer from a given pool.

> >>    - Crash when CREATE_BUFS is being used
> 
> This is a side effect of CREATE_BUFS being passed-through, implementing
> emulation for this should be straightforward.
> 
> >>    - Crash in the jpeg decoder (when frames are corrupted)
> 
> A minimalist framing parser would detect just enough of this, and would
> fix it.
> 
> >>    - App exporting DMABuf need to be aware of emulation, otherwise the
> >>      DMABuf exported are in the orignal format
> 
> libv4l2 can return ENOTTY to expbufs calls in
> 
> >>    - RW emulation only initialize the queue on first read (causing
> >>      userspace poll() to fail)
> 
> This is not fixable, only place it would be fixed is by moving this
> emulation into VideoBuf2. That would assume someone do care about RW
> (even though it could be nicer uAPI when dealing with muxed or byte-
> stream type of data).

I personally don't care much about R/W. I have heard that the recent 
implementation of mmap'ed buffer support in DVB showed impressive performance 
improvements, so a R/W API isn't something I'd prioritize.

> > >    - Signature of v4l2_mmap does not match mmap() (minor)
> > >    - The colorimetry does not seem emulated when conversion
> 
> This one is probably tricky, special is the converter plugin API is
> considered stable. Maybe just resetting everything to DEFAULT would
> work ?

I'd actually like to rework that API. The conversion code should be moved to a 
separate library that would allow its usage without a V4L2 device.

> > >    - Sub-optimal locking (at least deadlocks were fixed)
> 
> Need more investigation really, and proper measurement.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Mauro,

On Friday, 18 May 2018 18:05:22 EEST Mauro Carvalho Chehab wrote:
> Em Fri, 18 May 2018 15:27:24 +0300 Laurent Pinchart escreveu:
> > On Thursday, 17 May 2018 22:07:08 EEST Mauro Carvalho Chehab wrote:

[snip]

> >> 1.2 V4L2 userspace aspects
> >> --------------------------
> >> 
> >> Back when USB cameras were introduced, the hardware were really simple:
> >> they had a CCD camera sensor and a chip that bridges the data though
> >> USB. CCD camera sensors typically provide data using a bayer format,
> >> but they usually have their own proprietary ways to pack the data,
> >> in order to reduce the USB bandwidth (original cameras were USB 1.1).
> > 
> > Do you mean CMOS instead of CCD ?
> > 
> >> So, V4L2 has a myriad of different formats, in order to match each
> >> CCD camera sensor format.
> > 
> > While several webcams use custom formats, those are often related to
> > custom compression performed by the USB bridge, so it's not really related
> > to the sensor itself.
> 
> Yes, on several cases, the custom format is at the bridge, and not at
> the sensor itself. Anyway, the point is that format conversion used to
> be a key feature in order for video apps to work with different cameras.

That's not disputed, I just wanted to clarify the introduction for people who 
might not be familiar with the history of V4L development.

> >> At the end of the day, applications were able to use only a subset of
> >> the available hardware, since they need to come with format converters
> >> for all formats the developer uses (usually a very small subset of the
> >> available ones).
> > 
> > Note that this was only transitory, as webcams then converged towards a
> > small set of standard formats (even if recent webcams also implement
> > custom formats, but that's usually related to secondary data streams such
> > as depth maps, or to non-consumer cameras such as industrial cameras).
> 
> We are still adding new fourcc formats nowadays. IMHO, there's now a
> certain stability/convergence on formats, although newer cameras with
> advanced features (like 3D and light-field) will very likely require new
> formats.

As usual formats for old features are stabilizing, but new features require 
new formats. From a libv4l point of view those new formats are less of an 
issue than they used to be in the past, as nowadays they're mostly about 
features that libv4l and the applications using it don't really care about. I 
don't think we'll need to transparently convert a 3D format to 2D YUYV for 
Skype :-)

> >> To end with this mess, an userspace library was written, called libv4l.
> >> It supports all those proprietary formats. So, applications can use
> >> a RGB or YUV format, without needing to concern about conversions.
> >> 
> >> The way it works is by adding wrappers to system calls: open, close,
> >> ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> >> at the source code of the apps, all it was needed is to prepend the
> >> existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.
> >> 
> >> All open source apps we know now supports libv4l. On a few (like
> >> gstreamer), support for it is optional.
> >> 
> >> In order to support closed source, another wrapper was added, allowing
> >> to call any closed source application to use it, by using LD_PRELOAD.
> >> 
> >> For example, using skype with it is as simple as calling it with:
> >> 	$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so /usr/bin/skypeforlinux
> > 
> > It should be noted that libv4l also handles scaling (Skype requires
> > 640x480 if I remember correctly, or at least used to, and would crash if
> > given another resolution) and rough auto-gain and auto-white balance that
> > are required by some cameras to achieve a usable image.
> > 
> >> 2. Current problems
> >> ===================
> >> 
> >> 2.1 Libv4l can slow image handling
> >> ----------------------------------
> >> 
> >> Nowadays, almost all new "simple" cameras are connected via USB using
> >> the UVC class (USB Video Class). UVC standardized the allowed formats,
> >> and most apps just implement them. The UVC hardware is more complex,
> >> having format converters inside it. So, for most usages, format
> >> conversion isn't needed anymore.
> >> 
> >> The need of doing format conversion in software makes libv4l slow,
> >> requiring lots of CPU usage in order to convert a 4K or 8K format,
> >> being even worse with 3D cameras.
> > 
> > I would hope that libv4l doesn't perform conversion unnecessarily if not
> > needed. The main issue, as I understand it, is that application may fail
> > to notice that some formats require software conversion, and prefer them
> > over formats provided directly by the device even when such hardware
> > formats are supported by the application. We have introduced the
> > V4L2_FMT_FLAG_EMULATED flag in the kernel API in order for libv4l to flag
> > emulated formats, but applications may fail to see it.
> 
> I don't see how an application may fail to see it. If libv4l is used, the
> flag will be there. This is not a new flag. Apps can easily look into it
> and opt to use non-emulated formats. On non-emulated formats, there won't
> be unnecessarily format conversions.

My point was that some applications simply don't check the flag, especially 
applications that don't explicitly link against libv4l for which the 
LD_PRELOAD method is used.

> Granted, applications may opt to not use it, and just select an specific
> format, but it is an application choice to either use it or not.

Sure. It could possibly be useful to change the order of the format in the 
emulated ENUMFMT implementation to have all emulated formats at the end, for 
applications that pick the first format they support from the list.

> > (On a personal note, I think that extending the kernel API with features
> > that are not used by the kernel might not be the best API design decision
> > we've ever made, it questions whether the libv4l API design was right.)
> > 
> >> Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> >> DMA_BUFFER currently doesn't work with libv4l.
> >> 
> >> Right now, gstreamer defaults to not enable libv4l2, mainly due to
> >> those performance issues.
> >> 
> >> 
> >> 2.2 Modern hardware is starting to come with "complex" camera ISP
> >> -----------------------------------------------------------------
> >> 
> >> While mc-based devices were limited to SoC, it was easy to
> >> "delegate" the task of talking with the hardware to the
> >> embedded hardware designers.
> >> 
> >> However, this is changing. Dell Latitude 5285 laptop is a standard
> >> PC with an i3-core, i5-core or i7-core CPU, with comes with the
> >> Intel IMU3 ISP hardware[1]
> 
> Typo:
> 	IMU3 -> IPU3
> 
> >> [1] https://www.spinics.net/lists/linux-usb/msg167478.html
> > 
> > Do you think it's changing, or could that be an exception rather than a
> > future rule ?
> 
> Well, on ARM, all SoC chipsets I'm aware have an ISP IP block. As Intel is
> now adding IPU3 on some of their chipsets, I think that we'll see more and
> more hardware with it.
> 
> I don't have a crystal ball, but I guess that the trend would be to keep
> using USB with UVC for external cameras, but some ISP-based hardware for
> the internal ones.

I suppose we'll see what happens then. In any case, even if that laptop 
remains the only one with an ISP-based camera, we need a proper camera stack 
for ARM systems anyway, so it will come for free should it become more useful 
on x86 in the future.

> >> There, instead of an USB camera, the hardware is equipped with a
> >> MC-based ISP, connected to its camera. Currently, despite having
> >> a Kernel driver for it, the camera doesn't work with any
> >> userspace application.
> >> 
> >> I'm also aware of other projects that are considering the usage of
> >> mc-based devices for non-dedicated hardware.
> > 
> > What are those projects ?
> 
> Well, cheap ARM-based hardware like RPi3 already has this issue: they
> have an ISP (or some GPU firmware meant to emulate an ISP). While
> those hardware could have multiple sensors, typically they have just
> one.

GPU emulating the ISP. Ouch :-/ I wasn't aware of that.

> >> 3. How to solve it?
> >> ===================
> >> 
> >> That's the main focus of the meeting :-)
> >> 
> >> From a previous discussion I had with media sub-maintainers, there are
> >> at least two actions that seem required. I'm listing them below as
> >> an starting point for the discussions, but we can eventually come up
> >> with some different approach after the meeting.
> >> 
> >> 3.1 libv4l2 support for mc-based hardware
> >> =========================================
> >> 
> >> In order to support those hardware, we'll need to do some redesign
> >> mainly at libv4l2[2].
> > 
> > I wouldn't necessarily focus on libv4l2.
> > 
> > What I believe we need is to design a userspace camera stack that will
> > support the features needed by more modern cameras. Let's keep the dogmas
> > aside, if libv4l2 turns out to be the right option we can certainly go
> > for it, but if we find out that another option is better (Nicolas
> > mentioned Pipewire, I have no personal opinion about that yet, and there
> > could be other options too), then so be it.
> > Then, once we get a proper camera stack, we need to support traditional
> > applications that use libv4l2 and/or the V4L2 API directly. To that end we
> > will need to expose the libv4l2 API on top of the camera stack (through
> > libv4l2 if it makes sense, or through a separate implementation) and
> > implement a transparent LD_PRELOAD-based library (again either through
> > libv4l2 or through a separate implementation).
> > 
> > I'm certainly not calling for libv4l2 to be thrown away, but I think the
> > problem is complex enough without adding an artificial requirement of
> > basing the camera stack on a particular existing implementation. As long
> > as we achieve compatibility with the existing libv4l2 API and the V4L2
> > API I believe we don't need to add any specific implementation
> > restriction. What matters the most to me is achieving a maintainable
> > camera stack with a good API that supports all our needs, and providing a
> > compatibility layer on top of it.
> > 
> > (On that note, even if we end up using libv4l2 as the base, we will likely
> > nd up with a modified and incompatible main API anyway, with the need to
> > provide support for the existing API on top. That's why I think it
> > doesn't matter much if we decide to refactor libv4l2 or start from
> > scratch and possibly reuse part of the libv4l2 code.)
> 
> Yeah, agreed. On this stage, we shouldn't be limited to libv4l2 or its
> current implementation. The hole idea of the discussions is to understand
> the alternatives and chose a clear path for us to fix the issue.

I'm glad we agree :-) Let's then first focus on the needs of the camera stack 
for this meeting, and as a second step see how to implement it (which could 
also be discussed during the meeting if time permits).

> >> The idea is to work on a new API for libv4l2 that will allow to
> >> split the format conversion on a separate part of it, add support
> >> for DMA Buffer and come up with a way for the library to work
> >> transparently with both devnode-based and mc-based hardware.
> >> 
> >> That envolves adding capacity at libv4l to setup hardware pipelines
> >> and to propagate controls among their sub-devices. Eventually, part
> >> of it will be done in Kernel.
> > 
> > Which part do you mean exactly ?
> 
> I don't mean any specific part. Just saying that we shouldn't discard that
> some Kernel support could be implemented in order to make easier to
> implement/maintain it.

OK.

> >> That should give performance increase at the library and would allow
> >> gstreamer to use it by default, without compromising performance.
> >> 
> >> [2] I don't discard that some Kernel changes could also be part of the
> >> solution, like, for example, doing control propagation along the
> >> pipeline on simple use case scenarios.
> > 
> > We've tried that in the past and it was both cumbersome and not very
> > useful. I won't rule out kernel changes for the sake of it, but I'd
> > rather focus on use cases, and find for each of them the best possible
> > implementation. In my opinion we will find that most, if not all, of the
> > new features and changes would be better implemented in userspace.
> > 
> >> 3.2 libv4l2 support for 3A algorithms
> >> =====================================
> >> 
> >> The 3A algorithm handing is highly dependent on the hardware. The
> >> idea here is to allow libv4l to have a set of 3A algorithms that
> >> will be specific to certain mc-based hardware. Ideally, this should
> >> be added in a way that it will allow external closed-source
> >> algorithms to run as well.
> > 
> > As much as I dislike closed-source components, I don't think we "ideally"
> > need to support them, I believe they're mandatory if we want any vendor
> > to use our camera stack (unless someone can find enough resources to
> > implement a product- quality 3A library that will work with all ISPs
> > available out there, but I don't think that's a reasonable assumption to
> > start with).
> > 
> > My personal requirements for a 3A library are:
> > 
> > - The 3A library API must be standardized, to allow the closed-source
> > vendor implementation to be replaced by an open-source implementation
> > should someone have the time and energy (and qualifications) to write
> > one.
> > 
> > - Sandboxed execution of the 3A library must be possible as closed-source
> > vendor code can't always be blindly trusted (that's an understatement).
> > This includes the ability to wrap the library in a daemon should the
> > platform's multimedia stack wishes, and to avoid any direct access to the
> > kernel devices by the 3A library itself (all accesses should be marshaled
> > by the camera stack).
> > 
> > Please note that this daemon is *not* a camera daemon as used by Qualcomm
> > or other vendors that communicates with the V4L2 driver through a custom
> > back channel. I want the kernel driver to implement the V4L2 API and to
> > be usable without any daemon. The decision to run the 3A library in a
> > sandboxed process or to call it directly from the camera stack should be
> > left to the camera stack and to the platform integrator, and should not
> > be visible by the 3A library.
> > 
> > - The 3A library must be usable on major Linux-based camera stacks (the
> > Android and Chrome OS camera HALs are certainly important targets, more
> > can be added) unmodified, which will allow usage of the vendor binary
> > provided for Chrome OS or Android on regular Linux systems.
> 
> One of the issues with closed-source components is that they usually
> have dependencies that end being distro-dependent and arch-dependent [1].

There are lots of issues with closed-source components, the main one being 
that they're closed-source :-) I would really like to consider how to push 
vendors to open-source most (if not all) of the camera-related code.

> [1] on a side note, I have here one example of a hardware that depends on
> a closed-source plugin: a HP Laserjet USB printer/scanner. I would love
> to use it connected to an Exynos hardware here, but the closed-source
> plugin only works on x86, forcing me to connect to one desktop. The side
> effect is that, every time I upgrade CUPS, the printer stops working,
> as it waits for me to download the newest plugin that is compatible with
> such version. That's specially evil when I travel, as other people here
> won't be able to print anything, until I remotely connect to it and
> upgrade the plugin. Really nasty.
> 
> > On that note, I believe it would make sense to design a modular camera
> > stack, and try to make most components as platform-independent as
> > possible. This should include the kernel drivers (V4L2-compliant and
> > usable without any closed-source userspace component), the 3A library,
> > and any other component that could be shared (for instance a possible
> > request API library). The rest of the code will mostly be glue around
> > those components to integrate them in a particular camera stack, and
> > should be as platform-agnostic as possible. Ideally I would like to see
> > one Android camera HAL glue that could be used with different camera
> > vendors (probably with some kind of vendor-specific configuration, or
> > possibly with a separate vendor-specific component to handle pipeline
> > configuration).
> 
> Makes sense.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Dave Stevenson]

On 18 May 2018 at 16:05, Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
> Em Fri, 18 May 2018 15:27:24 +0300
<snip>
>>
>> > There, instead of an USB camera, the hardware is equipped with a
>> > MC-based ISP, connected to its camera. Currently, despite having
>> > a Kernel driver for it, the camera doesn't work with any
>> > userspace application.
>> >
>> > I'm also aware of other projects that are considering the usage of
>> > mc-based devices for non-dedicated hardware.
>>
>> What are those projects ?
>
> Well, cheap ARM-based hardware like RPi3 already has this issue: they
> have an ISP (or some GPU firmware meant to emulate an ISP). While
> those hardware could have multiple sensors, typically they have just
> one.

Slight hijack, but a closely linked issue for the Pi.
The way I understand the issue of V4L2 / MC on Pi is a more
fundamental mismatch in architecture. Please correct me if I'm wrong
here.

The Pi CSI2 receiver peripheral always writes the incoming data to
SDRAM, and the ISP is then a memory to memory device.

V4L2 subdevices are not dma controllers and therefore have no buffers
allocated to them. So to support the full complexity of the pipeline
in V4L2 requires that something somewhere would have to be dequeuing
the buffers from the CSI receiver V4L2 device and queuing them to the
input of a (theoretical) ISP M2M V4L2 device, and returning them once
processed. The application only cares about the output of the ISP M2M
device.

So I guess my question is whether there is a sane mechanism to remove
that buffer allocation and handling from the app? Without it we are
pretty much forced to hide bigger blobs of functionality to even
vaguely fit in with V4L2.

I'm at the point where it shouldn't be a huge amount of work to create
at least a basic ISP V4L2 M2M device, but I'm not planning on doing it
if it pushes the above buffer handling onto the app because it simply
won't get used beyond demo apps. The likes of Cheese, Scratch, etc,
just won't do it.


To avoid ambiguity, the Pi has a hardware ISP block. There are other
SoCs that use either GPU code or a DSP to implement their ISP.

  Dave

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Tomasz Figa]

+Hu, Jerry W +Mani, Rajmohan +Sakari Ailus

FYI
On Fri, May 18, 2018 at 4:07 AM Mauro Carvalho Chehab <
mchehab+samsung@kernel.org> wrote:

> Hi all,

> The goal of this e-mail is to schedule a meeting in order to discuss
> improvements at the media subsystem in order to support complex camera
> hardware by usual apps.

> The main focus here is to allow supporting devices with MC-based
> hardware connected to a camera.

> In short, my proposal is to meet with the interested parties on solving
> this issue during the Open Source Summit in Japan, e. g. between
> June, 19-22, in Tokyo.

> I'd like to know who is interested on joining us for such meeting,
> and to hear a proposal of themes for discussions.

> I'm enclosing a detailed description of the problem, in order to
> allow the interested parties to be at the same page.

> Regards,
> Mauro

> ---


> 1. Introduction
> ===============

> 1.1 V4L2 Kernel aspects
> -----------------------

> The media subsystem supports two types of devices:

> - "traditional" media hardware, supported via V4L2 API. On such hardware,
>    opening a single device node (usually /dev/video0) is enough to control
>    the entire device. We call it as devnode-based devices.

> - Media-controller based devices. On those devices, there are several
>    /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus a media
>    controller device node (usually /dev/media0).
>    We call it as mc-based devices. Controlling the hardware require
>    opening the media device (/dev/media0), setup the pipeline and adjust
>    the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
>    by /dev/video?.

> All "standard" media applications, including open source ones (Camorama,
> Cheese, Xawtv, Firefox, Chromium, ...) and closed source ones (Skype,
> Chrome, ...) supports devnode-based devices.

> Support for mc-based devices currently require an specialized application
> in order to prepare the device for its usage (setup pipelines, adjust
> hardware controls, etc). Once pipeline is set, the streaming goes via
> /dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
> be opened, in order to implement algorithms designed to make video quality
> reasonable. On such devices, it is not uncommon that the device used by
the
> application to be a random number (on OMAP3 driver, typically, is either
> /dev/video4 or /dev/video6).

> One example of such hardware is at the OMAP3-based hardware:


http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png

> On the picture, there's a graph with the hardware blocks in blue/dark/blue
> and the corresponding devnode interfaces in yellow.

> The mc-based approach was taken when support for Nokia N9/N900 cameras
> was added (with has OMAP3 SoC). It is required because the camera hardware
> on SoC comes with a media processor (ISP), with does a lot more than just
> capturing, allowing complex algorithms to enhance image quality in
runtime.
> Those algorithms are known as 3A - an acronym for 3 other acronyms:

>          - AE (Auto Exposure);
>          - AF (Auto Focus);
>          - AWB (Auto White Balance).

> Setting a camera with such ISPs are harder because the pipelines to be
> set actually depends the requirements for those 3A algorithms to run.
> Also, usually, the 3A algorithms use some chipset-specific userspace API,
> that exports some image properties, calculated by the ISP, to speed up
> the convergence of those algorithms.

> Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> as binary only blobs, although there are a few OSS implementations.

> 1.2 V4L2 userspace aspects
> --------------------------

> Back when USB cameras were introduced, the hardware were really simple:
> they had a CCD camera sensor and a chip that bridges the data though
> USB. CCD camera sensors typically provide data using a bayer format,
> but they usually have their own proprietary ways to pack the data,
> in order to reduce the USB bandwidth (original cameras were USB 1.1).

> So, V4L2 has a myriad of different formats, in order to match each
> CCD camera sensor format. At the end of the day, applications were
> able to use only a subset of the available hardware, since they need
> to come with format converters for all formats the developer uses
> (usually a very small subset of the available ones).

> To end with this mess, an userspace library was written, called libv4l.
> It supports all those proprietary formats. So, applications can use
> a RGB or YUV format, without needing to concern about conversions.

> The way it works is by adding wrappers to system calls: open, close,
> ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> at the source code of the apps, all it was needed is to prepend the
> existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.

> All open source apps we know now supports libv4l. On a few (like
> gstreamer), support for it is optional.

> In order to support closed source, another wrapper was added, allowing
> to call any closed source application to use it, by using LD_PRELOAD.
> For example, using skype with it is as simple as calling it with:

>          $ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so /usr/bin/skypeforlinux

> 2. Current problems
> ===================

> 2.1 Libv4l can slow image handling
> ----------------------------------

> Nowadays, almost all new "simple" cameras are connected via USB using
> the UVC class (USB Video Class). UVC standardized the allowed formats,
> and most apps just implement them. The UVC hardware is more complex,
> having format converters inside it. So, for most usages, format
> conversion isn't needed anymore.

> The need of doing format conversion in software makes libv4l slow,
> requiring lots of CPU usage in order to convert a 4K or 8K format,
> being even worse with 3D cameras.

> Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> DMA_BUFFER currently doesn't work with libv4l.

> Right now, gstreamer defaults to not enable libv4l2, mainly due to
> those performance issues.


> 2.2 Modern hardware is starting to come with "complex" camera ISP
> -----------------------------------------------------------------

> While mc-based devices were limited to SoC, it was easy to
> "delegate" the task of talking with the hardware to the
> embedded hardware designers.

> However, this is changing. Dell Latitude 5285 laptop is a standard
> PC with an i3-core, i5-core or i7-core CPU, with comes with the
> Intel IMU3 ISP hardware[1]

> [1] https://www.spinics.net/lists/linux-usb/msg167478.html

> There, instead of an USB camera, the hardware is equipped with a
> MC-based ISP, connected to its camera. Currently, despite having
> a Kernel driver for it, the camera doesn't work with any
> userspace application.

> I'm also aware of other projects that are considering the usage of
> mc-based devices for non-dedicated hardware.

> 3. How to solve it?
> ===================

> That's the main focus of the meeting :-)

>  From a previous discussion I had with media sub-maintainers, there are
> at least two actions that seem required. I'm listing them below as
> an starting point for the discussions, but we can eventually come up
> with some different approach after the meeting.

> 3.1 libv4l2 support for mc-based hardware
> =========================================

> In order to support those hardware, we'll need to do some redesign
> mainly at libv4l2[2].

> The idea is to work on a new API for libv4l2 that will allow to
> split the format conversion on a separate part of it, add support
> for DMA Buffer and come up with a way for the library to work
> transparently with both devnode-based and mc-based hardware.

> That envolves adding capacity at libv4l to setup hardware pipelines
> and to propagate controls among their sub-devices. Eventually, part
> of it will be done in Kernel.

> That should give performance increase at the library and would allow
> gstreamer to use it by default, without compromising performance.

> [2] I don't discard that some Kernel changes could also be part of the
> solution, like, for example, doing control propagation along the pipeline
> on simple use case scenarios.

> 3.2 libv4l2 support for 3A algorithms
> =====================================

> The 3A algorithm handing is highly dependent on the hardware. The
> idea here is to allow libv4l to have a set of 3A algorithms that
> will be specific to certain mc-based hardware. Ideally, this should
> be added in a way that it will allow external closed-source
> algorithms to run as well.

> Thanks,
> Mauro

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Tomasz Figa]

On Sat, May 19, 2018 at 12:15 AM Nicolas Dufresne <nicolas@ndufresne.ca>
wrote:

> Le vendredi 18 mai 2018 à 15:38 +0300, Laurent Pinchart a écrit :
> > > Before libv4l, media support for a given device were limited to a few
> > > apps that knew how to decode the format. There were even cases were a
> > > proprietary app were required, as no open source decoders were
available.
> > >
> > > From my PoV, the biggest gain with libv4l is that the same group of
> > > maintainers can ensure that the entire solution (Kernel driver and
> > > low level userspace support) will provide everything required for an
> > > open source app to work with it.
> > >
> > > I'm not sure how we would keep enforcing it if the pipeline setting
> > > and control propagation logic for an specific hardware will be
> > > delegated to PipeWire. It seems easier to keep doing it on a libv4l
> > > (version 2) and let PipeWire to use it.
> >
> > I believe we need to first study pipewire in more details. I have no
personal
> > opinion yet as I haven't had time to investigate it. That being said, I
don't
> > think that libv4l with closed-source plugins would be much better than a
> > closed-source pipewire plugin. What main concern once we provide a
userspace
> > camera stack API is that vendors might implement that API in a
closed-source
> > component that calls to a kernel driver implementing a custom API, with
all
> > knowledge about the camera located in the closed-source component. I'm
not
> > sure how to prevent that, my best proposal would be to make V4L2 so
useful
> > that vendors wouldn't even think about a different solution (possibly
coupled
> > by the pressure put by platform vendors such as Google who mandate
upstream
> > kernel drivers for Chrome OS, but that's still limited as even when it
comes
> > to Google there's no such pressure on the Android side).

> If there is proprietary plugins, then I don't think it will make any
> difference were this is implemented. The difference is the feature set
> we expose. 3A is per device, but multiple streams, with per request
> controls is also possible. PipeWire gives central place to manage this,
> while giving multiple process access to the camera streams. I think in
> the end, what fits better would be something like or the Android Camera
> HAL2. But we could encourage OSS by maintaining a base implementation
> that covers all the V4L2 aspect, leaving only the 3A aspect of the work
> to be done. Maybe we need to come up with an abstraction that does not
> prevent multi-streams, but only requires 3A per vendors (saying per
> vendors, as some of this could be Open Source by third parties).

> just thinking out loud now ;-P
> Nicolas

> p.s. Do we have the Intel / IPU3 folks in in the loop ? This is likely
> the most pressing HW as it's shipping on many laptops now.

Yes, I added Jerry, Raj and Sakari to the first post and also this one.

Best regards,
Tomasz

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 2929 bytes --]

Le vendredi 18 mai 2018 à 16:37 +0100, Dave Stevenson a écrit :
> On 18 May 2018 at 16:05, Mauro Carvalho Chehab
> <mchehab+samsung@kernel.org> wrote:
> > Em Fri, 18 May 2018 15:27:24 +0300
> 
> <snip>
> > > 
> > > > There, instead of an USB camera, the hardware is equipped with a
> > > > MC-based ISP, connected to its camera. Currently, despite having
> > > > a Kernel driver for it, the camera doesn't work with any
> > > > userspace application.
> > > > 
> > > > I'm also aware of other projects that are considering the usage of
> > > > mc-based devices for non-dedicated hardware.
> > > 
> > > What are those projects ?
> > 
> > Well, cheap ARM-based hardware like RPi3 already has this issue: they
> > have an ISP (or some GPU firmware meant to emulate an ISP). While
> > those hardware could have multiple sensors, typically they have just
> > one.
> 
> Slight hijack, but a closely linked issue for the Pi.
> The way I understand the issue of V4L2 / MC on Pi is a more
> fundamental mismatch in architecture. Please correct me if I'm wrong
> here.
> 
> The Pi CSI2 receiver peripheral always writes the incoming data to
> SDRAM, and the ISP is then a memory to memory device.

This is the same for IPU3 and some new can ARM ISP works like this too.
Though, IPU3 is fixed, you simply enable / disable / configure the ISP
base on the stats/metadata. Basically, it's not a single device, but
really two separate thing, were the ISP could be used without a sensor.
(Hope this make sense, need to be taken in consideration).

> 
> V4L2 subdevices are not dma controllers and therefore have no buffers
> allocated to them. So to support the full complexity of the pipeline
> in V4L2 requires that something somewhere would have to be dequeuing
> the buffers from the CSI receiver V4L2 device and queuing them to the
> input of a (theoretical) ISP M2M V4L2 device, and returning them once
> processed. The application only cares about the output of the ISP M2M
> device.
> 
> So I guess my question is whether there is a sane mechanism to remove
> that buffer allocation and handling from the app? Without it we are
> pretty much forced to hide bigger blobs of functionality to even
> vaguely fit in with V4L2.
> 
> I'm at the point where it shouldn't be a huge amount of work to create
> at least a basic ISP V4L2 M2M device, but I'm not planning on doing it
> if it pushes the above buffer handling onto the app because it simply
> won't get used beyond demo apps. The likes of Cheese, Scratch, etc,
> just won't do it.

Well, would have to be a media controller running in M2M as it is not
1:1 in term of number of input : number of output.

> 
> To avoid ambiguity, the Pi has a hardware ISP block. There are other
> SoCs that use either GPU code or a DSP to implement their ISP.

That's a good point, something that need to be kept in mind.

> 
>   Dave

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Nicolas,

On Friday, 18 May 2018 18:15:20 EEST Nicolas Dufresne wrote:
> Le vendredi 18 mai 2018 à 15:38 +0300, Laurent Pinchart a écrit :
> >> Before libv4l, media support for a given device were limited to a few
> >> apps that knew how to decode the format. There were even cases were a
> >> proprietary app were required, as no open source decoders were
> >> available.
> >> 
> >> From my PoV, the biggest gain with libv4l is that the same group of
> >> maintainers can ensure that the entire solution (Kernel driver and
> >> low level userspace support) will provide everything required for an
> >> open source app to work with it.
> >> 
> >> I'm not sure how we would keep enforcing it if the pipeline setting
> >> and control propagation logic for an specific hardware will be
> >> delegated to PipeWire. It seems easier to keep doing it on a libv4l
> >> (version 2) and let PipeWire to use it.
> > 
> > I believe we need to first study pipewire in more details. I have no
> > personal opinion yet as I haven't had time to investigate it. That being
> > said, I don't think that libv4l with closed-source plugins would be much
> > better than a closed-source pipewire plugin. What main concern once we
> > provide a userspace camera stack API is that vendors might implement that
> > API in a closed-source component that calls to a kernel driver
> > implementing a custom API, with all knowledge about the camera located in
> > the closed-source component. I'm not sure how to prevent that, my best
> > proposal would be to make V4L2 so useful that vendors wouldn't even think
> > about a different solution (possibly coupled by the pressure put by
> > platform vendors such as Google who mandate upstream kernel drivers for
> > Chrome OS, but that's still limited as even when it comes to Google
> > there's no such pressure on the Android side).
> 
> If there is proprietary plugins, then I don't think it will make any
> difference were this is implemented.

I tend to agree overall, although the community that develops the framework we 
will end up using can make a difference in that area.

> The difference is the feature set we expose. 3A is per device, but multiple
> streams, with per request controls is also possible.

Could you detail what you mean exactly by multiple streams ? Are you talking 
about multiple independent streams coming from the same device (such as video 
+ depth map, 3D video, ...) or streams created from a single source (sensor) 
to serve different purposes (viewfinder, video capture, still image capture, 
...) ?

> PipeWire gives central place to manage this, while giving multiple process
> access to the camera streams. I think in the end, what fits better would be
> something like or the Android Camera HAL2. But we could encourage OSS by
> maintaining a base implementation that covers all the V4L2 aspect, leaving
> only the 3A aspect of the work to be done.

Ideally that's the goal I'd like to reach, regardless of which multimedia 
stack we go for.

> Maybe we need to come up with an abstraction that does not prevent multi-
> streams, but only requires 3A per vendors

That would be tricky to achieve, as it's usually very use-case- and ISP-
dependent. Maybe we could come up with an interface for another vendor-
specific component to handle this, but I fear it will be so intertwined with 
the 3A implementation that it wouldn't be possible to isolate those two 
components.

> (saying per vendors, as some of this could be Open Source by third parties).

Note that in practice 3A is often tuned per-device, starting from a per-vendor 
implementation.

> just thinking out loud now ;-P

That's exactly what we need to do to start with :-)

> p.s. Do we have the Intel / IPU3 folks in in the loop ? This is likely
> the most pressing HW as it's shipping on many laptops now.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Dave,

On Friday, 18 May 2018 18:37:01 EEST Dave Stevenson wrote:
> On 18 May 2018 at 16:05, Mauro Carvalho Chehab wrote:
> > Em Fri, 18 May 2018 15:27:24 +0300
> 
> <snip>
> 
> >>> There, instead of an USB camera, the hardware is equipped with a
> >>> MC-based ISP, connected to its camera. Currently, despite having
> >>> a Kernel driver for it, the camera doesn't work with any
> >>> userspace application.
> >>> 
> >>> I'm also aware of other projects that are considering the usage of
> >>> mc-based devices for non-dedicated hardware.
> >> 
> >> What are those projects ?
> > 
> > Well, cheap ARM-based hardware like RPi3 already has this issue: they
> > have an ISP (or some GPU firmware meant to emulate an ISP). While
> > those hardware could have multiple sensors, typically they have just
> > one.
> 
> Slight hijack, but a closely linked issue for the Pi.
> The way I understand the issue of V4L2 / MC on Pi is a more
> fundamental mismatch in architecture. Please correct me if I'm wrong
> here.
> 
> The Pi CSI2 receiver peripheral always writes the incoming data to
> SDRAM, and the ISP is then a memory to memory device.
> 
> V4L2 subdevices are not dma controllers and therefore have no buffers
> allocated to them. So to support the full complexity of the pipeline
> in V4L2 requires that something somewhere would have to be dequeuing
> the buffers from the CSI receiver V4L2 device and queuing them to the
> input of a (theoretical) ISP M2M V4L2 device, and returning them once
> processed. The application only cares about the output of the ISP M2M
> device.

Regardless of the software stack architecture, something running on the CPU 
has to perform that job. We have decided that that "something" needs to run in 
userspace, to avoid pushing use-case-dependent code to the kernel.

Note that this isn't specific to the RPi. The OMAP3 ISP, while integrating the 
CSI-2 receiver and being able to process data on the fly, can also write the 
raw images to memory and then process them in memory-to-memory mode. This 
feature is used mostly for still image capture to perform pre-processing with 
the CPU (or possibly GPU) on the raw images before processing them in the ISP. 
There's no way we could implement this fully in the kernel.

> So I guess my question is whether there is a sane mechanism to remove
> that buffer allocation and handling from the app? Without it we are
> pretty much forced to hide bigger blobs of functionality to even
> vaguely fit in with V4L2.

We need a way to remove that from the application, but it won't be pushed down 
to the kernel. These tasks should be handled by a userspace framework, 
transparently for the application. The purpose of this discussion is to decide 
on the design of the framework.

> I'm at the point where it shouldn't be a huge amount of work to create
> at least a basic ISP V4L2 M2M device, but I'm not planning on doing it
> if it pushes the above buffer handling onto the app because it simply
> won't get used beyond demo apps. The likes of Cheese, Scratch, etc,
> just won't do it.
> 
> 
> To avoid ambiguity, the Pi has a hardware ISP block. There are other
> SoCs that use either GPU code or a DSP to implement their ISP.

Is that ISP documented publicly ?

-- 
Regards,

Laurent Pinchart

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Dave Stevenson]

Hi Laurent

On 19 May 2018 at 08:04, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> Hi Dave,
>
> On Friday, 18 May 2018 18:37:01 EEST Dave Stevenson wrote:
>> On 18 May 2018 at 16:05, Mauro Carvalho Chehab wrote:
>> > Em Fri, 18 May 2018 15:27:24 +0300
>>
>> <snip>
>>
>> >>> There, instead of an USB camera, the hardware is equipped with a
>> >>> MC-based ISP, connected to its camera. Currently, despite having
>> >>> a Kernel driver for it, the camera doesn't work with any
>> >>> userspace application.
>> >>>
>> >>> I'm also aware of other projects that are considering the usage of
>> >>> mc-based devices for non-dedicated hardware.
>> >>
>> >> What are those projects ?
>> >
>> > Well, cheap ARM-based hardware like RPi3 already has this issue: they
>> > have an ISP (or some GPU firmware meant to emulate an ISP). While
>> > those hardware could have multiple sensors, typically they have just
>> > one.
>>
>> Slight hijack, but a closely linked issue for the Pi.
>> The way I understand the issue of V4L2 / MC on Pi is a more
>> fundamental mismatch in architecture. Please correct me if I'm wrong
>> here.
>>
>> The Pi CSI2 receiver peripheral always writes the incoming data to
>> SDRAM, and the ISP is then a memory to memory device.
>>
>> V4L2 subdevices are not dma controllers and therefore have no buffers
>> allocated to them. So to support the full complexity of the pipeline
>> in V4L2 requires that something somewhere would have to be dequeuing
>> the buffers from the CSI receiver V4L2 device and queuing them to the
>> input of a (theoretical) ISP M2M V4L2 device, and returning them once
>> processed. The application only cares about the output of the ISP M2M
>> device.
>
> Regardless of the software stack architecture, something running on the CPU
> has to perform that job. We have decided that that "something" needs to run in
> userspace, to avoid pushing use-case-dependent code to the kernel.
>
> Note that this isn't specific to the RPi. The OMAP3 ISP, while integrating the
> CSI-2 receiver and being able to process data on the fly, can also write the
> raw images to memory and then process them in memory-to-memory mode. This
> feature is used mostly for still image capture to perform pre-processing with
> the CPU (or possibly GPU) on the raw images before processing them in the ISP.
> There's no way we could implement this fully in the kernel.

Sure. I was mainly flagging that having to manage buffers also needs
to be considered in order to make a usable system. Just configuring an
MC pipeline won't solve all the issues.

>> So I guess my question is whether there is a sane mechanism to remove
>> that buffer allocation and handling from the app? Without it we are
>> pretty much forced to hide bigger blobs of functionality to even
>> vaguely fit in with V4L2.
>
> We need a way to remove that from the application, but it won't be pushed down
> to the kernel. These tasks should be handled by a userspace framework,
> transparently for the application. The purpose of this discussion is to decide
> on the design of the framework.

I'm in agreement there, but hadn't seen discussion on buffer
management, only MC configuration.

>> I'm at the point where it shouldn't be a huge amount of work to create
>> at least a basic ISP V4L2 M2M device, but I'm not planning on doing it
>> if it pushes the above buffer handling onto the app because it simply
>> won't get used beyond demo apps. The likes of Cheese, Scratch, etc,
>> just won't do it.
>>
>>
>> To avoid ambiguity, the Pi has a hardware ISP block. There are other
>> SoCs that use either GPU code or a DSP to implement their ISP.
>
> Is that ISP documented publicly ?

Not publicly, and as it's Broadcom's IP we can't release it :-(

What I have working is using the Broadcom MMAL API (very similar to
OpenMAX IL) to wrap the ISP hardware block via the VideoCore firmware.
Currently it has the major controls exposed (black level, digital
gain, white balance, CCMs, lens shading tables) and I'll add
additional controls as time permits or use cases require. Resizing and
format conversion are done based on input and output formats. Defining
stats regions and extracting the resulting stats is still to be done.
Overall it keeps all the implementation details hidden so we don't
break NDAs, but should allow efficient processing. It supports
dma-bufs in and out, so no extra copies of the data should be
required.

I'm currently finishing off a V4L2 M2M wrapper around the MMAL
video_encode and video_decode components, so modifying it to support
the ISP component shouldn't be difficult if there is value in doing
so.
I know it's not the ideal, but our hands are tied.

  Dave

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Dave,

On Monday, 21 May 2018 15:16:19 EEST Dave Stevenson wrote:
> On 19 May 2018 at 08:04, Laurent Pinchart wrote:
> > On Friday, 18 May 2018 18:37:01 EEST Dave Stevenson wrote:
> >> On 18 May 2018 at 16:05, Mauro Carvalho Chehab wrote:
> >>> Em Fri, 18 May 2018 15:27:24 +0300
> >> 
> >> <snip>
> >> 
> >>>>> There, instead of an USB camera, the hardware is equipped with a
> >>>>> MC-based ISP, connected to its camera. Currently, despite having
> >>>>> a Kernel driver for it, the camera doesn't work with any
> >>>>> userspace application.
> >>>>> 
> >>>>> I'm also aware of other projects that are considering the usage of
> >>>>> mc-based devices for non-dedicated hardware.
> >>>> 
> >>>> What are those projects ?
> >>> 
> >>> Well, cheap ARM-based hardware like RPi3 already has this issue: they
> >>> have an ISP (or some GPU firmware meant to emulate an ISP). While
> >>> those hardware could have multiple sensors, typically they have just
> >>> one.
> >> 
> >> Slight hijack, but a closely linked issue for the Pi.
> >> The way I understand the issue of V4L2 / MC on Pi is a more
> >> fundamental mismatch in architecture. Please correct me if I'm wrong
> >> here.
> >> 
> >> The Pi CSI2 receiver peripheral always writes the incoming data to
> >> SDRAM, and the ISP is then a memory to memory device.
> >> 
> >> V4L2 subdevices are not dma controllers and therefore have no buffers
> >> allocated to them. So to support the full complexity of the pipeline
> >> in V4L2 requires that something somewhere would have to be dequeuing
> >> the buffers from the CSI receiver V4L2 device and queuing them to the
> >> input of a (theoretical) ISP M2M V4L2 device, and returning them once
> >> processed. The application only cares about the output of the ISP M2M
> >> device.
> > 
> > Regardless of the software stack architecture, something running on the
> > CPU has to perform that job. We have decided that that "something" needs
> > to run in userspace, to avoid pushing use-case-dependent code to the
> > kernel.
> > 
> > Note that this isn't specific to the RPi. The OMAP3 ISP, while integrating
> > the CSI-2 receiver and being able to process data on the fly, can also
> > write the raw images to memory and then process them in memory-to-memory
> > mode. This feature is used mostly for still image capture to perform
> > pre-processing with the CPU (or possibly GPU) on the raw images before
> > processing them in the ISP. There's no way we could implement this fully
> > in the kernel.
> 
> Sure. I was mainly flagging that having to manage buffers also needs
> to be considered in order to make a usable system. Just configuring an
> MC pipeline won't solve all the issues.
> 
> >> So I guess my question is whether there is a sane mechanism to remove
> >> that buffer allocation and handling from the app? Without it we are
> >> pretty much forced to hide bigger blobs of functionality to even
> >> vaguely fit in with V4L2.
> > 
> > We need a way to remove that from the application, but it won't be pushed
> > down to the kernel. These tasks should be handled by a userspace
> > framework, transparently for the application. The purpose of this
> > discussion is to decide on the design of the framework.
> 
> I'm in agreement there, but hadn't seen discussion on buffer
> management, only MC configuration.

I'll take the blame for not having been clear enough. We certainly need to 
support more than static pipeline configuration and 3A algorithms, there's a 
need to support interactions with the device at runtime even for simple 
capture without 3A. That's a use case I certainly want to see addressed.

> >> I'm at the point where it shouldn't be a huge amount of work to create
> >> at least a basic ISP V4L2 M2M device, but I'm not planning on doing it
> >> if it pushes the above buffer handling onto the app because it simply
> >> won't get used beyond demo apps. The likes of Cheese, Scratch, etc,
> >> just won't do it.
> >> 
> >> 
> >> To avoid ambiguity, the Pi has a hardware ISP block. There are other
> >> SoCs that use either GPU code or a DSP to implement their ISP.
> > 
> > Is that ISP documented publicly ?
> 
> Not publicly, and as it's Broadcom's IP we can't release it :-(
> 
> What I have working is using the Broadcom MMAL API (very similar to
> OpenMAX IL) to wrap the ISP hardware block via the VideoCore firmware.
> Currently it has the major controls exposed (black level, digital
> gain, white balance, CCMs, lens shading tables) and I'll add
> additional controls as time permits or use cases require. Resizing and
> format conversion are done based on input and output formats. Defining
> stats regions and extracting the resulting stats is still to be done.
> Overall it keeps all the implementation details hidden so we don't
> break NDAs, but should allow efficient processing. It supports
> dma-bufs in and out, so no extra copies of the data should be
> required.
> 
> I'm currently finishing off a V4L2 M2M wrapper around the MMAL
> video_encode and video_decode components, so modifying it to support
> the ISP component shouldn't be difficult if there is value in doing
> so. I know it's not the ideal, but our hands are tied.

I won't blame you for that. My personal goal with the new framework is however 
to create the foundation of an open-source ecosystem.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Hans Verkuil]

On 18/05/18 13:24, Mauro Carvalho Chehab wrote:
> One of the biggest reasons why we decided to start libv4l project,
> in the past, was to ensure an open source solution. The problem we
> faced on that time is to ensure that, when a new media driver were
> added with some proprietary output format, an open source decoding
> software were also added at libv4l.
> 
> This approach ensured that all non-MC cameras are supported by all
> V4L2 applications.
> 
> Before libv4l, media support for a given device were limited to a few 
> apps that knew how to decode the format. There were even cases were a
> proprietary app were required, as no open source decoders were available.
> 
> From my PoV, the biggest gain with libv4l is that the same group of
> maintainers can ensure that the entire solution (Kernel driver and
> low level userspace support) will provide everything required for an
> open source app to work with it.
> 
> I'm not sure how we would keep enforcing it if the pipeline setting
> and control propagation logic for an specific hardware will be
> delegated to PipeWire. It seems easier to keep doing it on a libv4l
> (version 2) and let PipeWire to use it.

I've decided not to attend this meeting. It is not quite my core expertise
and it is a bit too far to make it worth my time. If there are good reasons
for me being there that I missed, then please let me know asap and I might
reconsider this.

What I would like to say though it that I think libv4l is a bit of a dead
end and probably not suitable for adding support for this.

Currently libv4l2 is too intertwined with libv4lconvert and too messy.
Its original motivation was for converting custom formats and that is
mostly obsolete now that UVC has standardized formats to just a few.

I think a core library is needed that provides the basic functionality
and that can be used directly by applications if they don't want to use
v4l2_open() and friends.

I.e. it should be possible for e.g. gstreamer to use this core library
to easily configure and use the MC instead of having to call v4l2_open() etc.
and rely on magic code to do this for them. It's simply ugly to overload
mmap with v4l2_mmap or to emulate read() if the driver doesn't support it.

We might still have a libv4l2-like library sitting on top of this, but
perhaps with limited functionality. For example, I think it would be
reasonable to no longer support custom formats. If an application wants
to support that, then it should call conversion functions for the core
library explicitly. This has the big advantage of solving the dmabuf
and mmap issues in today's libv4l2.

Regards,

	Hans

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Hans,

On Wednesday, 23 May 2018 19:19:37 EEST Hans Verkuil wrote:
> On 18/05/18 13:24, Mauro Carvalho Chehab wrote:
> > One of the biggest reasons why we decided to start libv4l project,
> > in the past, was to ensure an open source solution. The problem we
> > faced on that time is to ensure that, when a new media driver were
> > added with some proprietary output format, an open source decoding
> > software were also added at libv4l.
> > 
> > This approach ensured that all non-MC cameras are supported by all
> > V4L2 applications.
> > 
> > Before libv4l, media support for a given device were limited to a few
> > apps that knew how to decode the format. There were even cases were a
> > proprietary app were required, as no open source decoders were available.
> > 
> > From my PoV, the biggest gain with libv4l is that the same group of
> > maintainers can ensure that the entire solution (Kernel driver and
> > low level userspace support) will provide everything required for an
> > open source app to work with it.
> > 
> > I'm not sure how we would keep enforcing it if the pipeline setting
> > and control propagation logic for an specific hardware will be
> > delegated to PipeWire. It seems easier to keep doing it on a libv4l
> > (version 2) and let PipeWire to use it.
> 
> I've decided not to attend this meeting. It is not quite my core expertise
> and it is a bit too far to make it worth my time. If there are good reasons
> for me being there that I missed, then please let me know asap and I might
> reconsider this.
> 
> What I would like to say though it that I think libv4l is a bit of a dead
> end and probably not suitable for adding support for this.
> 
> Currently libv4l2 is too intertwined with libv4lconvert and too messy.
> Its original motivation was for converting custom formats and that is
> mostly obsolete now that UVC has standardized formats to just a few.
> 
> I think a core library is needed that provides the basic functionality
> and that can be used directly by applications if they don't want to use
> v4l2_open() and friends.
> 
> I.e. it should be possible for e.g. gstreamer to use this core library
> to easily configure and use the MC instead of having to call v4l2_open()
> etc. and rely on magic code to do this for them. It's simply ugly to
> overload mmap with v4l2_mmap or to emulate read() if the driver doesn't
> support it.
> 
> We might still have a libv4l2-like library sitting on top of this, but
> perhaps with limited functionality. For example, I think it would be
> reasonable to no longer support custom formats. If an application wants
> to support that, then it should call conversion functions for the core
> library explicitly. This has the big advantage of solving the dmabuf
> and mmap issues in today's libv4l2.

I agree with all that. I also believe we need to design a clean solution 
without caring about the existing libv4l2 API, and then implement a 
compatibility layer on top of our new library. The way to move away from the 
stone age is to design a new technology for the future, and then help the past 
climb on the bandwagon.

-- 
Regards,

Laurent Pinchart

RE: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Zheng, Jian Xu]

Hi Tomasz & Mauro,

> -----Original Message-----
> From: linux-media-owner@vger.kernel.org [mailto:linux-media-
> owner@vger.kernel.org] On Behalf Of Tomasz Figa
> Sent: Friday, May 18, 2018 11:38 PM
> To: mchehab+samsung@kernel.org
> Cc: Linux Media Mailing List <linux-media@vger.kernel.org>; Hu, Jerry W
> <jerry.w.hu@intel.com>; Mani, Rajmohan <rajmohan.mani@intel.com>;
> Sakari Ailus <sakari.ailus@linux.intel.com>
> Subject: Re: [ANN] Meeting to discuss improvements to support MC-based
> cameras on generic apps
> 
> +Hu, Jerry W +Mani, Rajmohan +Sakari Ailus
> 
> FYI
> On Fri, May 18, 2018 at 4:07 AM Mauro Carvalho Chehab <
> mchehab+samsung@kernel.org> wrote:
> 
> > Hi all,
> 
> > The goal of this e-mail is to schedule a meeting in order to discuss
> > improvements at the media subsystem in order to support complex camera
> > hardware by usual apps.
> 
> > The main focus here is to allow supporting devices with MC-based
> > hardware connected to a camera.
> 
> > In short, my proposal is to meet with the interested parties on
> > solving this issue during the Open Source Summit in Japan, e. g.
> > between June, 19-22, in Tokyo.
> 
> > I'd like to know who is interested on joining us for such meeting, and
> > to hear a proposal of themes for discussions.

I should be able to join the meeting in Tokyo from Intel camera team. 
Looking forward to this discussion and will send out topic from my end later.

Best Regards,
Jianxu(Clive) Zheng

> > I'm enclosing a detailed description of the problem, in order to allow
> > the interested parties to be at the same page.
> 

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Mauro Carvalho Chehab]

Em Thu, 17 May 2018 16:07:08 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:

> Hi all,
> 
> The goal of this e-mail is to schedule a meeting in order to discuss
> improvements at the media subsystem in order to support complex camera
> hardware by usual apps.
> 
> The main focus here is to allow supporting devices with MC-based
> hardware connected to a camera.
> 
> In short, my proposal is to meet with the interested parties on solving
> this issue during the Open Source Summit in Japan, e. g. between
> June, 19-22, in Tokyo.

Let's schedule the meeting to happen in Tokyo, Japan at June, 19.

Location yet to be defined, but it will either be together with
OSS Japan or at Google. I'll confirm the address tomorrow.

Regards,
Mauro



Thanks,
Mauro

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Niklas Söderlund]

Hi Mauro,

On 2018-05-28 10:43:51 -0300, Mauro Carvalho Chehab wrote:
> Em Thu, 17 May 2018 16:07:08 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> 
> > Hi all,
> > 
> > The goal of this e-mail is to schedule a meeting in order to discuss
> > improvements at the media subsystem in order to support complex camera
> > hardware by usual apps.
> > 
> > The main focus here is to allow supporting devices with MC-based
> > hardware connected to a camera.
> > 
> > In short, my proposal is to meet with the interested parties on solving
> > this issue during the Open Source Summit in Japan, e. g. between
> > June, 19-22, in Tokyo.
> 
> Let's schedule the meeting to happen in Tokyo, Japan at June, 19.

I will be in Tokyo at that time and would be happy to join.

> 
> Location yet to be defined, but it will either be together with
> OSS Japan or at Google. I'll confirm the address tomorrow.
> 
> Regards,
> Mauro
> 
> 
> 
> Thanks,
> Mauro

-- 
Regards,
Niklas Söderlund

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Mauro Carvalho Chehab]

Em Mon, 28 May 2018 10:43:51 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:

> Em Thu, 17 May 2018 16:07:08 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> 
> > Hi all,
> > 
> > The goal of this e-mail is to schedule a meeting in order to discuss
> > improvements at the media subsystem in order to support complex camera
> > hardware by usual apps.
> > 
> > The main focus here is to allow supporting devices with MC-based
> > hardware connected to a camera.
> > 
> > In short, my proposal is to meet with the interested parties on solving
> > this issue during the Open Source Summit in Japan, e. g. between
> > June, 19-22, in Tokyo.  
> 
> Let's schedule the meeting to happen in Tokyo, Japan at June, 19.
> 
> Location yet to be defined, but it will either be together with
> OSS Japan or at Google. I'll confirm the address tomorrow.

More details about the meeting:

Date: June, 19
Site: Google
Address: 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F

Please confirm who will be attending the meeting.

Hoping to see you there.

Thanks and regards!
Mauro

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Kieran Bingham]

[-- Attachment #1.1: Type: text/plain, Size: 424 bytes --]

Hi Mauro

On 31/05/18 14:22, Mauro Carvalho Chehab wrote:
> Em Mon, 28 May 2018 10:43:51 -0300

<snip>

> More details about the meeting:
> 
> Date: June, 19
> Site: Google
> Address: 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
> 
> Please confirm who will be attending the meeting.

I'll be in Japan, so I will attend the meeting if that's OK.

Regards

Kieran


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Laurent Pinchart]

Hi Mauro,

On Thursday, 31 May 2018 16:22:12 EEST Mauro Carvalho Chehab wrote:
> Em Mon, 28 May 2018 10:43:51 -0300 Mauro Carvalho Chehab escreveu:
> > Em Thu, 17 May 2018 16:07:08 -0300 Mauro Carvalho Chehab escreveu:
> >> Hi all,
> >> 
> >> The goal of this e-mail is to schedule a meeting in order to discuss
> >> improvements at the media subsystem in order to support complex camera
> >> hardware by usual apps.
> >> 
> >> The main focus here is to allow supporting devices with MC-based
> >> hardware connected to a camera.
> >> 
> >> In short, my proposal is to meet with the interested parties on solving
> >> this issue during the Open Source Summit in Japan, e. g. between
> >> June, 19-22, in Tokyo.
> > 
> > Let's schedule the meeting to happen in Tokyo, Japan at June, 19.
> > 
> > Location yet to be defined, but it will either be together with
> > OSS Japan or at Google. I'll confirm the address tomorrow.
> 
> More details about the meeting:
> 
> Date: June, 19
> Site: Google
> Address: 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori
> Tower 44F
> 
> Please confirm who will be attending the meeting.
> 
> Hoping to see you there.

I will be there.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Hans Verkuil]

On 05/31/18 15:22, Mauro Carvalho Chehab wrote:
> Em Mon, 28 May 2018 10:43:51 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> 
>> Em Thu, 17 May 2018 16:07:08 -0300
>> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>>
>>> Hi all,
>>>
>>> The goal of this e-mail is to schedule a meeting in order to discuss
>>> improvements at the media subsystem in order to support complex camera
>>> hardware by usual apps.
>>>
>>> The main focus here is to allow supporting devices with MC-based
>>> hardware connected to a camera.
>>>
>>> In short, my proposal is to meet with the interested parties on solving
>>> this issue during the Open Source Summit in Japan, e. g. between
>>> June, 19-22, in Tokyo.  
>>
>> Let's schedule the meeting to happen in Tokyo, Japan at June, 19.
>>
>> Location yet to be defined, but it will either be together with
>> OSS Japan or at Google. I'll confirm the address tomorrow.
> 
> More details about the meeting:
> 
> Date: June, 19
> Site: Google
> Address: 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
> 
> Please confirm who will be attending the meeting.

I plan to attend the meeting via Google Hangouts.

Regards,

	Hans

> 
> Hoping to see you there.
> 
> Thanks and regards!
> Mauro
> 

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Tomasz Figa]

Hi Hans and everyone,

On Thu, May 31, 2018 at 10:58 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> On 05/31/18 15:22, Mauro Carvalho Chehab wrote:
> > Em Mon, 28 May 2018 10:43:51 -0300
> > Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> >
> >> Em Thu, 17 May 2018 16:07:08 -0300
> >> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> >>
> >>> Hi all,
> >>>
> >>> The goal of this e-mail is to schedule a meeting in order to discuss
> >>> improvements at the media subsystem in order to support complex camera
> >>> hardware by usual apps.
> >>>
> >>> The main focus here is to allow supporting devices with MC-based
> >>> hardware connected to a camera.
> >>>
> >>> In short, my proposal is to meet with the interested parties on solving
> >>> this issue during the Open Source Summit in Japan, e. g. between
> >>> June, 19-22, in Tokyo.
> >>
> >> Let's schedule the meeting to happen in Tokyo, Japan at June, 19.
> >>
> >> Location yet to be defined, but it will either be together with
> >> OSS Japan or at Google. I'll confirm the address tomorrow.
> >
> > More details about the meeting:
> >
> > Date: June, 19
> > Site: Google
> > Address: 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
> >
> > Please confirm who will be attending the meeting.
>
> I plan to attend the meeting via Google Hangouts.

I'll be there as an organizer and participant. We will also have Ricky
from our camera team in Taipei join via Hangouts.

To avoid wasting time on the day of the event, I'd like to have a full
list of participants with emails, so that I can arrange building
access. Replying "Yes, I'll be there" will work for me. ;)

Please let me know if you need any further details or help.

Best regards,
Tomasz

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Hans Verkuil]

On 05/31/18 15:58, Hans Verkuil wrote:
> On 05/31/18 15:22, Mauro Carvalho Chehab wrote:
>> Em Mon, 28 May 2018 10:43:51 -0300
>> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>>
>>> Em Thu, 17 May 2018 16:07:08 -0300
>>> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>>>
>>>> Hi all,
>>>>
>>>> The goal of this e-mail is to schedule a meeting in order to discuss
>>>> improvements at the media subsystem in order to support complex camera
>>>> hardware by usual apps.
>>>>
>>>> The main focus here is to allow supporting devices with MC-based
>>>> hardware connected to a camera.
>>>>
>>>> In short, my proposal is to meet with the interested parties on solving
>>>> this issue during the Open Source Summit in Japan, e. g. between
>>>> June, 19-22, in Tokyo.  
>>>
>>> Let's schedule the meeting to happen in Tokyo, Japan at June, 19.
>>>
>>> Location yet to be defined, but it will either be together with
>>> OSS Japan or at Google. I'll confirm the address tomorrow.
>>
>> More details about the meeting:
>>
>> Date: June, 19
>> Site: Google
>> Address: 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
>>
>> Please confirm who will be attending the meeting.
> 
> I plan to attend the meeting via Google Hangouts.

Well, the afternoon part of the meeting at least :-)

Regards,

	Hans

> 
> Regards,
> 
> 	Hans
> 
>>
>> Hoping to see you there.
>>
>> Thanks and regards!
>> Mauro
>>
> 

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Mauro Carvalho Chehab]

Em Thu, 31 May 2018 23:18:38 +0900
Tomasz Figa <tfiga@google.com> escreveu:

> Hi Hans and everyone,
> 
> On Thu, May 31, 2018 at 10:58 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
> >
> > On 05/31/18 15:22, Mauro Carvalho Chehab wrote:  
> > > Em Mon, 28 May 2018 10:43:51 -0300
> > > Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> > >  
> > >> Em Thu, 17 May 2018 16:07:08 -0300
> > >> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> > >>  
> > >>> Hi all,
> > >>>
> > >>> The goal of this e-mail is to schedule a meeting in order to discuss
> > >>> improvements at the media subsystem in order to support complex camera
> > >>> hardware by usual apps.
> > >>>
> > >>> The main focus here is to allow supporting devices with MC-based
> > >>> hardware connected to a camera.
> > >>>
> > >>> In short, my proposal is to meet with the interested parties on solving
> > >>> this issue during the Open Source Summit in Japan, e. g. between
> > >>> June, 19-22, in Tokyo.  
> > >>
> > >> Let's schedule the meeting to happen in Tokyo, Japan at June, 19.
> > >>
> > >> Location yet to be defined, but it will either be together with
> > >> OSS Japan or at Google. I'll confirm the address tomorrow.  
> > >
> > > More details about the meeting:
> > >
> > > Date: June, 19
> > > Site: Google
> > > Address: 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
> > >
> > > Please confirm who will be attending the meeting.  
> >
> > I plan to attend the meeting via Google Hangouts.  
> 
> I'll be there as an organizer and participant. We will also have Ricky
> from our camera team in Taipei join via Hangouts.
> 
> To avoid wasting time on the day of the event, I'd like to have a full
> list of participants with emails, so that I can arrange building
> access. Replying "Yes, I'll be there" will work for me. ;)

Please add Mr. Seung-Woo Kim also from Samsung. He's also intending
to be there.

Thanks,
Mauro

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Javier Martinez Canillas]

On Thu, May 31, 2018 at 4:19 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> On 05/31/18 15:58, Hans Verkuil wrote:
>> On 05/31/18 15:22, Mauro Carvalho Chehab wrote:
>>> Em Mon, 28 May 2018 10:43:51 -0300
>>> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>>>
>>>> Em Thu, 17 May 2018 16:07:08 -0300
>>>> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>>>>
>>>>> Hi all,
>>>>>
>>>>> The goal of this e-mail is to schedule a meeting in order to discuss
>>>>> improvements at the media subsystem in order to support complex camera
>>>>> hardware by usual apps.
>>>>>
>>>>> The main focus here is to allow supporting devices with MC-based
>>>>> hardware connected to a camera.
>>>>>
>>>>> In short, my proposal is to meet with the interested parties on solving
>>>>> this issue during the Open Source Summit in Japan, e. g. between
>>>>> June, 19-22, in Tokyo.
>>>>
>>>> Let's schedule the meeting to happen in Tokyo, Japan at June, 19.
>>>>
>>>> Location yet to be defined, but it will either be together with
>>>> OSS Japan or at Google. I'll confirm the address tomorrow.
>>>
>>> More details about the meeting:
>>>
>>> Date: June, 19
>>> Site: Google
>>> Address: 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
>>>
>>> Please confirm who will be attending the meeting.
>>
>> I plan to attend the meeting via Google Hangouts.
>
> Well, the afternoon part of the meeting at least :-)
>

I also plan to attend via Google Hangouts. At least the hours that are
reasonable on my timezone.

Best regards,
Javier

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Sakari Ailus]

Hi Tomasz,

On Sat, May 19, 2018 at 12:37:34AM +0900, Tomasz Figa wrote:
> +Hu, Jerry W +Mani, Rajmohan +Sakari Ailus
> 
> FYI
> On Fri, May 18, 2018 at 4:07 AM Mauro Carvalho Chehab <
> mchehab+samsung@kernel.org> wrote:
> 
> > Hi all,
> 
> > The goal of this e-mail is to schedule a meeting in order to discuss
> > improvements at the media subsystem in order to support complex camera
> > hardware by usual apps.
> 
> > The main focus here is to allow supporting devices with MC-based
> > hardware connected to a camera.
> 
> > In short, my proposal is to meet with the interested parties on solving
> > this issue during the Open Source Summit in Japan, e. g. between
> > June, 19-22, in Tokyo.
> 
> > I'd like to know who is interested on joining us for such meeting,
> > and to hear a proposal of themes for discussions.
> 
> > I'm enclosing a detailed description of the problem, in order to
> > allow the interested parties to be at the same page.

At the moment it looks very much like that I won't be able to attend in
person. Would you be able to set up a Google Hangouts meeting for this? I
will attempt to attend remotely instead.

-- 
Kind regards,

Sakari Ailus
sakari.ailus@linux.intel.com

Re: [ANN] Meeting to discuss improvements to support MC-based cameras on generic apps

[Tomasz Figa]

Hi Sakari,

On Wed, Jun 13, 2018 at 11:36 PM Sakari Ailus
<sakari.ailus@linux.intel.com> wrote:
>
> Hi Tomasz,
>
> On Sat, May 19, 2018 at 12:37:34AM +0900, Tomasz Figa wrote:
> > +Hu, Jerry W +Mani, Rajmohan +Sakari Ailus
> >
> > FYI
> > On Fri, May 18, 2018 at 4:07 AM Mauro Carvalho Chehab <
> > mchehab+samsung@kernel.org> wrote:
> >
> > > Hi all,
> >
> > > The goal of this e-mail is to schedule a meeting in order to discuss
> > > improvements at the media subsystem in order to support complex camera
> > > hardware by usual apps.
> >
> > > The main focus here is to allow supporting devices with MC-based
> > > hardware connected to a camera.
> >
> > > In short, my proposal is to meet with the interested parties on solving
> > > this issue during the Open Source Summit in Japan, e. g. between
> > > June, 19-22, in Tokyo.
> >
> > > I'd like to know who is interested on joining us for such meeting,
> > > and to hear a proposal of themes for discussions.
> >
> > > I'm enclosing a detailed description of the problem, in order to
> > > allow the interested parties to be at the same page.
>
> At the moment it looks very much like that I won't be able to attend in
> person. Would you be able to set up a Google Hangouts meeting for this? I
> will attempt to attend remotely instead.

That's the plan. I'll share the details once we set it up.

Best regards,
Tomasz