[ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Mauro Carvalho Chehab]

Hi all,

I consolidated hopefully all comments I receive on the past announcement
with regards to the complex camera workshop we're planning to happen in
Tokyo, just before the Open Source Summit in Japan.

The main focus of the workshop is to allow supporting devices with MC-based
hardware connected to a camera.

I'm enclosing a detailed description of the problem, in order to
allow the interested parties to be at the same page.

We need to work towards an agenda for the meeting.

From my side, I think we should have at least the following topics at
the agenda:

- a quick review about what's currently at libv4l2;
- a presentation about PipeWire solution;
- a discussion with the requirements for the new solution;
- a discussion about how we'll address - who will do what.

Comments? Suggestions?

Are there anyone else planning to either be there physically or via
Google Hangouts?

Tomaz,

Do you have any limit about the number of people that could join us
via Google Hangouts?


Regards,
Mauro

---

1. Introduction
===============

1.1 V4L2 Kernel aspects
-----------------------

The media subsystem supports two types of devices:

- "traditional" media hardware, supported via V4L2 API. On such hardware, 
  opening a single device node (usually /dev/video0) is enough to control
  the entire device. We call it as devnode-based devices.
  An application sometimes may need to use multiple video nodes with 
  devnode-based drivers to capture multiple streams in parallel
  (when the hardware allows it of course). That's quite common for
  Analog TV devices, where both /dev/video0 and /dev/vbi0 are opened
  at the same time.

- Media-controller based devices. On those devices, there are typically
  several /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus
  a media controller device node (usually /dev/media0).
  We call it as mc-based devices. Controlling the hardware require
  opening the media device (/dev/media0), setup the pipeline and adjust
  the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
  by /dev/video?.

In other words, both configuration and streaming go through the video
device node on devnode-based drivers, while video device nodes are used
used for streaming on mc-based drivers.

With devnode-based drivers, "standard" media applications, including open 
source ones (Camorama, Cheese, Xawtv, Firefox, Chromium, ...) and closed
source ones (Skype, Chrome, ...) support devnode-based devices[1]. Also,
when just one media device is connected, the streaming/control device
is typically /dev/video0.

[1] It should be noticed that closed-source applications tend to have 
various bugs that prevent them from working properly on many devnode-based
devices. Due to that, some additional blocks were requred at libv4l to
support some of them. Skype is a good example, as we had to include a
software scaler in libv4l to make it happy. So in practice not everything
works smoothly with closed-source applications with devnode-based drivers.
A few such adjustments were also made on some drivers and/or libv4l, in
order to fulfill some open-source app requirements.

Support for mc-based devices currently require an specialized application 
in order to prepare the device for its usage (setup pipelines, adjust
hardware controls, etc). Once pipeline is set, the streaming goes via
/dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
be opened, in order to implement algorithms designed to make video quality
reasonable. On such devices, it is not uncommon that the device used by the
application to be a random number (on OMAP3 driver, typically, is either
/dev/video4 or /dev/video6).

One example of such hardware is at the OMAP3-based hardware:

	http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png

On the picture, there's a graph with the hardware blocks in blue/dark/blue
and the corresponding devnode interfaces in yellow.

The mc-based approach was taken when support for Nokia N9/N900 cameras 
was added (with has OMAP3 SoC). It is required because the camera hardware
on SoC comes with a media processor (ISP), with does a lot more than just
capturing, allowing complex algorithms to enhance image quality in runtime.
Those algorithms are known as 3A - an acronym for 3 other acronyms:

	- AE (Auto Exposure);
	- AF (Auto Focus);
	- AWB (Auto White Balance).

The main reason that drove the MC design is that the 3A algorithms (that is 
the 3A control loop, and sometimes part of the image processing itself) often 
need to run, at least partially, on the CPU. As a kernel-space implementation
wasn't possible, we needed a lower-level UAPI.

Setting a camera with such ISPs are harder because the pipelines to be
set actually depends the requirements for those 3A algorithms to run.
Also, usually, the 3A algorithms use some chipset-specific userspace API,
that exports some image properties, calculated by the ISP, to speed up
the convergence of those algorithms.

Btw, usually, the 3A algorithms are IP-protected, provided by vendors
as binary only blobs, although there are a few OSS implementations.

Part of the problem is that, so far, there isn't a proper userspace API 
to implement 3A libraries. Once we have an userspace camera stack, we
hope that we'll gradually increase the number and quality of open-source
3A stacks.


1.2 V4L2 userspace aspects
--------------------------

Back when USB cameras were introduced, the hardware were really simple:
they had a CMOS or CCD camera sensor and a chip that bridges the data 
though USB. Camera sensors typically provide data using a bayer
format, but they usually have their own proprietary ways to pack the data
at the USB bridges, in order to reduce the bandwidth (the first
implementations were using USB version 1.1).

So, V4L2 has a myriad of different formats, in order to match each
camera sensor format. At the end of the day, applications were
able to use only a subset of the available hardware, since they need
to come with format converters for all formats the developer uses
(usually a very small subset of the available ones).

That's said, newer cameras are converged towards a small set of
standard formats - except for secondary data streams (like depth
maps). Yet, industrial cameras, and newer technologies, like
3D, light-field, etc may still bring newer formats.

To end with this mess, an userspace library was written, called libv4l.
It supports all those proprietary formats. So, applications can use
a RGB or YUV format, without needing to concern about conversions.

The way it works is by adding wrappers to system calls: open, close,
ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
at the source code of the apps, all it was needed is to prepend the
existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.

All open source apps we know now supports libv4l. On a few (like
gstreamer), support for it is optional.

It should be noted that libv4l also handles scaling and rough auto-gain 
and auto-white balance that are required by  some cameras to achieve
a usable image.

In order to support closed source, another wrapper was added, allowing
to call any closed source application to use it, by using LD_PRELOAD.
For example, using skype with it is as simple as calling it with:

	$ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so /usr/bin/skypeforlinux


2. Current problems
===================

2.1 Libv4l can slow image handling
----------------------------------

Nowadays, almost all new "simple" cameras are connected via USB using
the UVC class (USB Video Class). UVC standardized the allowed formats,
and most apps just implement them. The UVC hardware is more complex,
having format converters inside it. So, for most usages, format
conversion isn't needed anymore.

The need of doing format conversion in software makes libv4l slow,
requiring lots of CPU usage in order to convert a 4K or 8K format,
being even worse with 3D cameras.

Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
DMA_BUFFER currently doesn't work with libv4l.

Right now, gstreamer defaults to not enable libv4l2, due to several
reasons:
  - Crash when CREATE_BUFS is being used;
  - Crash in the jpeg decoder (when frames are corrupted);
  - App exporting DMABuf need to be aware of emulation, otherwise the
    DMABuf exported are in the orignal format;
  - RW emulation only initialize the queue on first read (causing 
    userspace poll() to fail);
  - Signature of v4l2_mmap does not match mmap() (minor);
  - The colorimetry does not seem emulated when conversion;
  - Sub-optimal locking (at least deadlocks were fixed).

Most of the above are due to new features added to Kernel uAPI, but
not added to libv4l2.

These issues are already worked around in GStreamer, but with the lost
of features of course. There is other cases were something worked 
without libv4l2, but didn't work with libv4l2, but Gstreamer developers
haven't tracked down the cause yet. Since 1.14, libv4l2 can be enabled
at run-time using env GST_V4L2_USE_LIBV4L2=1.

2.2 Modern hardware is starting to come with "complex" camera ISP
-----------------------------------------------------------------

While mc-based devices were limited to SoC, it was easy to
"delegate" the task of talking with the hardware to the
embedded hardware designers.

However, this is changing. Dell Latitude 5285 laptop is a standard
PC with an i3-core, i5-core or i7-core CPU, with comes with the
Intel IMU3 ISP hardware[2].

[2] https://www.spinics.net/lists/linux-usb/msg167478.html

There, instead of an USB camera, the hardware is equipped with a
MC-based ISP, connected to its camera. Currently, despite having
a Kernel driver for it, the camera doesn't work with any
userspace application.

I'm also aware of other projects that are considering the usage of
mc-based devices for non-dedicated hardware.

3. How to solve it?
===================

That's the main focus of the meeting :-)

From a previous discussion I had with media sub-maintainers, there are
at least two actions that seem required. I'm listing them below as
an starting point for the discussions, but we can eventually come up
with some different approach after the meeting.

3.1 Library support for mc-based hardware
=========================================

In order to support those hardware, we'll need to do some redesign,
mainly at userspace, via a library[3] that will replace/compliment
libv4l2.

The idea is to work on a library API that will allow to split the 
format conversion on a separate part of it, add support
for DMA Buffer and come up with a way for the library to work
transparently with both devnode-based and mc-based hardware.

The library should be capable of doing dynamic pipeline setups,
including the functionality of allow tranfering buffers between
different media devices, as, on several cases, a a camera input
device node should be connected to a m2m device that would be
doing some image processing, and providing output data on a
standard video output format.

One possibility is to redesign libv4l2. It should be noticed that
Videolan people is working on a daemon that could also provide
another way to implement it[4].

Once we get a proper camera stack, we need to support traditional 
applications that use libv4l2 and/or the new library directly. To 
that end we  will need to expose the libv4l2 API on top of the 
camera stack (through libv4l2 if it makes sense, or through a separate
implementation) and implement a transparent LD_PRELOAD-based library.

That envolves adding capacity to the library to setup hardware pipelines
and to propagate controls among their sub-devices. Eventually, part
of it will be done in Kernel.

That should give performance increase at the library and would allow
gstreamer to use it by default, without compromising performance.

[3] I don't discard that some Kernel changes could also be part of the
solution, like, for example, doing control propagation along the pipeline
on simple use case scenarios.

[4] With the venue of sandboxed application, there is a need to control
access to cameras through a daemon. The same daemon is also used to
control access to screen capture on Wayland (instead of letting any
random application capture your screen, like on X11). The effort is 
lead by the desktop team at RedHat. PipeWire already have V4L2 native
support and is integrated in GStreamer already in a way that it can
totally replace the V4L2 capture component there. PipeWire is plugin based,
so more type of camera support can be added. Remote daemon can also provide 
streams, as this is the case for compositors and screen casting. An extra 
benefit is that you can have multiple application reading frames from the
same camera. It also allow sandboxed  application (the do not have access 
to /dev) to use the cameras. In this context, proprietary or HW specific
algorithm could be implemented in userspace as PipeWire plugins, and then
application will automatically be enable to enumerate and use these. 

3.2 libv4l2 support for 3A algorithms
=====================================

The 3A algorithm handing is highly dependent on the hardware. The
idea here is to allow libv4l to have a set of 3A algorithms that
will be specific to certain mc-based hardware.

One requirement, if we want vendor stacks to use our solution, is that 
it should allow allow external closed-source algorithms to run as well.

The 3A library API must be standardized, to allow the closed-source 
vendor implementation to be replaced by an open-source implementation
should someone have the time and energy (and qualifications) to write
one.

Sandboxed execution of the 3A library must be possible as closed-source 
can't always be blindly trusted. This includes the ability to wrap the
library in a daemon should the platform's multimedia stack wishes
and to avoid any direct access to the kernel devices by the 3A library
itself (all accesses should be marshaled by the camera stack).

Please note that this daemon is *not* a camera daemon that would
communicates with the V4L2 driver through a custom back channel.

The decision to run the 3A library in a sandboxed process or to call
it directly from the camera stack should be left to the camera stack
and to the platform integrator, and should not be visible by the 3A
library.

The 3A library must be usable on major Linux-based camera stacks (the 
Android and Chrome OS camera HALs are certainly important targets, 
more can be added) unmodified, which will allow usage of the vendor
binary provided for Chrome OS or Android on regular Linux systems.

It would make sense to design a modular camera stack, and try to make 
most components as platform-independent as possible. This should include:

- the kernel drivers (V4L2-compliant and usable without any closed-source 
  userspace component);
- the 3A library
- any other component that could be shared (for instance a possible
  request API library). 

The rest of the code will mostly be glue around those components to
integrate them in a particular camera stack, and should be as 
platform-agnostic as possible. 

In the case of the Android camera HAL, ideally it would be a glue that
could be used with different camera vendors (probably with some kind of
vendor-specific configuration, or possibly with a separate vendor-specific
component to handle pipeline configuration).

4 Complex camera workshop
=========================

The workshop will be happening in Tokyo, Japan, at Jun, 19, at the
google offices. The location is:

〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F

4.1 Physical Attendees
======================

Tomasz Figa <tfiga@google.com>
Mauro Carvalho Chehab <Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
Kieran Bingham <kieran.bingham@ideasonboard.com>
Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Niklas Söderlund <niklas.soderlund@ragnatech.se>
Zheng, Jian Xu Zheng <jian.xu.zheng@intel.com>

Anywone else?

4.2. Attendees Via Google Hangouts
==================================

Hans Verkuil <hverkuil@xs4all.nl> - Via Google Hangouts - maybe only on afternoon
Javier Martinez Canillas <javier@dowhile0.org> - Via Google Hangouts - only on reasonable TZ-compatible-hours
Ricky - Google camera team in Taipei - Via Google Hangouts

Anywone else?

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[jacopo mondi]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

Hi Mauro,

On Mon, Jun 04, 2018 at 10:33:03AM -0300, Mauro Carvalho Chehab wrote:
> Hi all,


[snip]

> 4.1 Physical Attendees
> ======================
>
> Tomasz Figa <tfiga@google.com>
> Mauro Carvalho Chehab <Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> Kieran Bingham <kieran.bingham@ideasonboard.com>
> Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Niklas Söderlund <niklas.soderlund@ragnatech.se>
> Zheng, Jian Xu Zheng <jian.xu.zheng@intel.com>
>
> Anywone else?

Sorry, I've not listed myself in reply to the previous email.

As I'll be in Tokyo for OSS, I would like to join you for the
meeting.

Thanks
   j

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Tomasz Figa]

On Mon, Jun 4, 2018 at 10:33 PM Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
>
> Hi all,
>
> I consolidated hopefully all comments I receive on the past announcement
> with regards to the complex camera workshop we're planning to happen in
> Tokyo, just before the Open Source Summit in Japan.
>
> The main focus of the workshop is to allow supporting devices with MC-based
> hardware connected to a camera.
>
> I'm enclosing a detailed description of the problem, in order to
> allow the interested parties to be at the same page.
>
> We need to work towards an agenda for the meeting.
>
> From my side, I think we should have at least the following topics at
> the agenda:
>
> - a quick review about what's currently at libv4l2;
> - a presentation about PipeWire solution;
> - a discussion with the requirements for the new solution;
> - a discussion about how we'll address - who will do what.

I believe Intel's Jian Xu would be able to give us some brief
introduction to IPU3 hardware architecture and possibly also upcoming
hardware generations as well.

My experience with existing generations of ISPs from other vendors is
that the main principles of operation are very similar to the model
represented by IPU3 and very much different to the OMAP3 example
mentioned by Mauro below. I further commented on it below.

>
> Comments? Suggestions?
>
> Are there anyone else planning to either be there physically or via
> Google Hangouts?
>
> Tomaz,
>
> Do you have any limit about the number of people that could join us
> via Google Hangouts?
>

Technically, Hangouts should be able to work with really huge
multi-party conferences. There is obviously some limitation on client
side, since thumbnails of participants need to be decoded at real
time, so even if the resolution is low, if the client is very slow,
there might be some really bad frame drop happening on client side.

However, I often have meetings with around 8 parties and it tends to
work fine. We can also disable video of all participants, who don't
need to present anything at the moment and the problem would go away
completely.

>
> Regards,
> Mauro
>
> ---
>
> 1. Introduction
> ===============
>
> 1.1 V4L2 Kernel aspects
> -----------------------
>
> The media subsystem supports two types of devices:
>
> - "traditional" media hardware, supported via V4L2 API. On such hardware,
>   opening a single device node (usually /dev/video0) is enough to control
>   the entire device. We call it as devnode-based devices.
>   An application sometimes may need to use multiple video nodes with
>   devnode-based drivers to capture multiple streams in parallel
>   (when the hardware allows it of course). That's quite common for
>   Analog TV devices, where both /dev/video0 and /dev/vbi0 are opened
>   at the same time.
>
> - Media-controller based devices. On those devices, there are typically
>   several /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus
>   a media controller device node (usually /dev/media0).
>   We call it as mc-based devices. Controlling the hardware require
>   opening the media device (/dev/media0), setup the pipeline and adjust
>   the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
>   by /dev/video?.
>
> In other words, both configuration and streaming go through the video
> device node on devnode-based drivers, while video device nodes are used
> used for streaming on mc-based drivers.
>
> With devnode-based drivers, "standard" media applications, including open
> source ones (Camorama, Cheese, Xawtv, Firefox, Chromium, ...) and closed
> source ones (Skype, Chrome, ...) support devnode-based devices[1]. Also,
> when just one media device is connected, the streaming/control device
> is typically /dev/video0.
>
> [1] It should be noticed that closed-source applications tend to have
> various bugs that prevent them from working properly on many devnode-based
> devices. Due to that, some additional blocks were requred at libv4l to
> support some of them. Skype is a good example, as we had to include a
> software scaler in libv4l to make it happy. So in practice not everything
> works smoothly with closed-source applications with devnode-based drivers.
> A few such adjustments were also made on some drivers and/or libv4l, in
> order to fulfill some open-source app requirements.
>
> Support for mc-based devices currently require an specialized application
> in order to prepare the device for its usage (setup pipelines, adjust
> hardware controls, etc). Once pipeline is set, the streaming goes via
> /dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
> be opened, in order to implement algorithms designed to make video quality
> reasonable.

To further complicate the problem, on many modern imaging subsystems
(Intel IPU3, Rockchip RKISP1), there is more than 1 video output
(CAPTURE device), for example:
1) full resolution capture stream and
2) downscaled preview stream.

Moreover, many ISPs also produce per-frame metadata (statistics) for
3A algorithms, which then produces per-frame metadata (parameters) for
processing of next frame. These would be also exposed as /dev/video?
nodes with respective V4L2_BUF_TYPE_META_* queues.

It is complicated even more on systems with separate input (e.g. CSI2)
and processing (ISP) hardware, such as Intel IPU3. In such case, the
raw frames captured from the CSI2 interface directly are not usable
for end-user applications. This means that some component in userspace
needs to forward the raw frames to the ISP and only the output of the
ISP can be passed to the application.

> On such devices, it is not uncommon that the device used by the
> application to be a random number (on OMAP3 driver, typically, is either
> /dev/video4 or /dev/video6).
>
> One example of such hardware is at the OMAP3-based hardware:
>
>         http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png
>
> On the picture, there's a graph with the hardware blocks in blue/dark/blue
> and the corresponding devnode interfaces in yellow.
>
> The mc-based approach was taken when support for Nokia N9/N900 cameras
> was added (with has OMAP3 SoC). It is required because the camera hardware
> on SoC comes with a media processor (ISP), with does a lot more than just
> capturing, allowing complex algorithms to enhance image quality in runtime.
> Those algorithms are known as 3A - an acronym for 3 other acronyms:
>
>         - AE (Auto Exposure);
>         - AF (Auto Focus);
>         - AWB (Auto White Balance).
>
> The main reason that drove the MC design is that the 3A algorithms (that is
> the 3A control loop, and sometimes part of the image processing itself) often
> need to run, at least partially, on the CPU. As a kernel-space implementation
> wasn't possible, we needed a lower-level UAPI.
>
> Setting a camera with such ISPs are harder because the pipelines to be
> set actually depends the requirements for those 3A algorithms to run.
> Also, usually, the 3A algorithms use some chipset-specific userspace API,
> that exports some image properties, calculated by the ISP, to speed up
> the convergence of those algorithms.
>
> Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> as binary only blobs, although there are a few OSS implementations.
>
> Part of the problem is that, so far, there isn't a proper userspace API
> to implement 3A libraries. Once we have an userspace camera stack, we
> hope that we'll gradually increase the number and quality of open-source
> 3A stacks.
>
[snip]
>
> 2.2 Modern hardware is starting to come with "complex" camera ISP
> -----------------------------------------------------------------
>
> While mc-based devices were limited to SoC, it was easy to
> "delegate" the task of talking with the hardware to the
> embedded hardware designers.
>
> However, this is changing. Dell Latitude 5285 laptop is a standard
> PC with an i3-core, i5-core or i7-core CPU, with comes with the
> Intel IMU3 ISP hardware[2].

IPU3 :)

>
> [2] https://www.spinics.net/lists/linux-usb/msg167478.html
>
> There, instead of an USB camera, the hardware is equipped with a
> MC-based ISP, connected to its camera. Currently, despite having
> a Kernel driver for it, the camera doesn't work with any
> userspace application.
>
> I'm also aware of other projects that are considering the usage of
> mc-based devices for non-dedicated hardware.
>
[snip]
>
> 3.2 libv4l2 support for 3A algorithms
> =====================================
>
> The 3A algorithm handing is highly dependent on the hardware. The
> idea here is to allow libv4l to have a set of 3A algorithms that
> will be specific to certain mc-based hardware.
>
> One requirement, if we want vendor stacks to use our solution, is that
> it should allow allow external closed-source algorithms to run as well.
>
> The 3A library API must be standardized, to allow the closed-source
> vendor implementation to be replaced by an open-source implementation
> should someone have the time and energy (and qualifications) to write
> one.
>
> Sandboxed execution of the 3A library must be possible as closed-source
> can't always be blindly trusted. This includes the ability to wrap the
> library in a daemon should the platform's multimedia stack wishes
> and to avoid any direct access to the kernel devices by the 3A library
> itself (all accesses should be marshaled by the camera stack).
>
> Please note that this daemon is *not* a camera daemon that would
> communicates with the V4L2 driver through a custom back channel.
>
> The decision to run the 3A library in a sandboxed process or to call
> it directly from the camera stack should be left to the camera stack
> and to the platform integrator, and should not be visible by the 3A
> library.
>
> The 3A library must be usable on major Linux-based camera stacks (the
> Android and Chrome OS camera HALs are certainly important targets,
> more can be added) unmodified, which will allow usage of the vendor
> binary provided for Chrome OS or Android on regular Linux systems.

This is quite an interesting idea and it would be really useful if it
could be done. I'm kind of worried, though, about Android in
particular, since the execution environment in Android differs
significantly from a regular Linux distributions (including Chrome OS,
which is not so far from such), namely:
- different libc (bionic) and dynamic linker - I guess this could be
solved by static linking?
- dedicated toolchains - perhaps not much of a problem if the per-arch
ABI is the same?

>
> It would make sense to design a modular camera stack, and try to make
> most components as platform-independent as possible. This should include:
>
> - the kernel drivers (V4L2-compliant and usable without any closed-source
>   userspace component);
> - the 3A library
> - any other component that could be shared (for instance a possible
>   request API library).
>
> The rest of the code will mostly be glue around those components to
> integrate them in a particular camera stack, and should be as
> platform-agnostic as possible.
>
> In the case of the Android camera HAL, ideally it would be a glue that
> could be used with different camera vendors (probably with some kind of
> vendor-specific configuration, or possibly with a separate vendor-specific
> component to handle pipeline configuration).
>
> 4 Complex camera workshop
> =========================
>
> The workshop will be happening in Tokyo, Japan, at Jun, 19, at the
> google offices. The location is:
>
> 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F

Nearest station exits:
- Hibiya line Roppongi station exit 1c (recommended)
- Oedo line Roppongi station exit 3 (and few minutes walk)

>
> 4.1 Physical Attendees
> ======================
>
> Tomasz Figa <tfiga@google.com>
> Mauro Carvalho Chehab <Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> Kieran Bingham <kieran.bingham@ideasonboard.com>
> Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Niklas Söderlund <niklas.soderlund@ragnatech.se>
> Zheng, Jian Xu Zheng <jian.xu.zheng@intel.com>
>
> Anywone else?

Looking at latest reply in this thread:

jacopo mondi <jacopo@jmondi.org>

Anyone else, please tell me beforehand (at least 1-2 days before), as
I need to take care of building access, since it's a multi-tenant
office building. I'll contact each attendee separately with further
details by email.

>
> 4.2. Attendees Via Google Hangouts
> ==================================
>
> Hans Verkuil <hverkuil@xs4all.nl> - Via Google Hangouts - maybe only on afternoon
> Javier Martinez Canillas <javier@dowhile0.org> - Via Google Hangouts - only on reasonable TZ-compatible-hours

What time zone would that be? I guess we could try to tweak the agenda
to take this into account.

Best regards.
Tomasz

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Javier Martinez Canillas]

[adding Wim Taymans and Mario Limonciello to CC who said that they may
also join via Hangous]

On Wed, Jun 6, 2018 at 6:19 AM, Tomasz Figa <tfiga@chromium.org> wrote:
> On Mon, Jun 4, 2018 at 10:33 PM Mauro Carvalho Chehab
> <mchehab+samsung@kernel.org> wrote:
>>
>> Hi all,
>>
>> I consolidated hopefully all comments I receive on the past announcement
>> with regards to the complex camera workshop we're planning to happen in
>> Tokyo, just before the Open Source Summit in Japan.
>>
>> The main focus of the workshop is to allow supporting devices with MC-based
>> hardware connected to a camera.
>>
>> I'm enclosing a detailed description of the problem, in order to
>> allow the interested parties to be at the same page.
>>
>> We need to work towards an agenda for the meeting.
>>
>> From my side, I think we should have at least the following topics at
>> the agenda:
>>
>> - a quick review about what's currently at libv4l2;
>> - a presentation about PipeWire solution;

Wim mentioned that he could do this.

>> - a discussion with the requirements for the new solution;
>> - a discussion about how we'll address - who will do what.
>
> I believe Intel's Jian Xu would be able to give us some brief
> introduction to IPU3 hardware architecture and possibly also upcoming
> hardware generations as well.
>
> My experience with existing generations of ISPs from other vendors is
> that the main principles of operation are very similar to the model
> represented by IPU3 and very much different to the OMAP3 example
> mentioned by Mauro below. I further commented on it below.
>
>>
>> Comments? Suggestions?
>>
>> Are there anyone else planning to either be there physically or via
>> Google Hangouts?
>>
>> Tomaz,
>>
>> Do you have any limit about the number of people that could join us
>> via Google Hangouts?
>>
>
> Technically, Hangouts should be able to work with really huge
> multi-party conferences. There is obviously some limitation on client
> side, since thumbnails of participants need to be decoded at real
> time, so even if the resolution is low, if the client is very slow,
> there might be some really bad frame drop happening on client side.
>
> However, I often have meetings with around 8 parties and it tends to
> work fine. We can also disable video of all participants, who don't
> need to present anything at the moment and the problem would go away
> completely.
>
>>
>> Regards,
>> Mauro
>>
>> ---
>>
>> 1. Introduction
>> ===============
>>
>> 1.1 V4L2 Kernel aspects
>> -----------------------
>>
>> The media subsystem supports two types of devices:
>>
>> - "traditional" media hardware, supported via V4L2 API. On such hardware,
>>   opening a single device node (usually /dev/video0) is enough to control
>>   the entire device. We call it as devnode-based devices.
>>   An application sometimes may need to use multiple video nodes with
>>   devnode-based drivers to capture multiple streams in parallel
>>   (when the hardware allows it of course). That's quite common for
>>   Analog TV devices, where both /dev/video0 and /dev/vbi0 are opened
>>   at the same time.
>>
>> - Media-controller based devices. On those devices, there are typically
>>   several /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus
>>   a media controller device node (usually /dev/media0).
>>   We call it as mc-based devices. Controlling the hardware require
>>   opening the media device (/dev/media0), setup the pipeline and adjust
>>   the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
>>   by /dev/video?.
>>
>> In other words, both configuration and streaming go through the video
>> device node on devnode-based drivers, while video device nodes are used
>> used for streaming on mc-based drivers.
>>
>> With devnode-based drivers, "standard" media applications, including open
>> source ones (Camorama, Cheese, Xawtv, Firefox, Chromium, ...) and closed
>> source ones (Skype, Chrome, ...) support devnode-based devices[1]. Also,
>> when just one media device is connected, the streaming/control device
>> is typically /dev/video0.
>>
>> [1] It should be noticed that closed-source applications tend to have
>> various bugs that prevent them from working properly on many devnode-based
>> devices. Due to that, some additional blocks were requred at libv4l to
>> support some of them. Skype is a good example, as we had to include a
>> software scaler in libv4l to make it happy. So in practice not everything
>> works smoothly with closed-source applications with devnode-based drivers.
>> A few such adjustments were also made on some drivers and/or libv4l, in
>> order to fulfill some open-source app requirements.
>>
>> Support for mc-based devices currently require an specialized application
>> in order to prepare the device for its usage (setup pipelines, adjust
>> hardware controls, etc). Once pipeline is set, the streaming goes via
>> /dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
>> be opened, in order to implement algorithms designed to make video quality
>> reasonable.
>
> To further complicate the problem, on many modern imaging subsystems
> (Intel IPU3, Rockchip RKISP1), there is more than 1 video output
> (CAPTURE device), for example:
> 1) full resolution capture stream and
> 2) downscaled preview stream.
>
> Moreover, many ISPs also produce per-frame metadata (statistics) for
> 3A algorithms, which then produces per-frame metadata (parameters) for
> processing of next frame. These would be also exposed as /dev/video?
> nodes with respective V4L2_BUF_TYPE_META_* queues.
>
> It is complicated even more on systems with separate input (e.g. CSI2)
> and processing (ISP) hardware, such as Intel IPU3. In such case, the
> raw frames captured from the CSI2 interface directly are not usable
> for end-user applications. This means that some component in userspace
> needs to forward the raw frames to the ISP and only the output of the
> ISP can be passed to the application.
>
>> On such devices, it is not uncommon that the device used by the
>> application to be a random number (on OMAP3 driver, typically, is either
>> /dev/video4 or /dev/video6).
>>
>> One example of such hardware is at the OMAP3-based hardware:
>>
>>         http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png
>>
>> On the picture, there's a graph with the hardware blocks in blue/dark/blue
>> and the corresponding devnode interfaces in yellow.
>>
>> The mc-based approach was taken when support for Nokia N9/N900 cameras
>> was added (with has OMAP3 SoC). It is required because the camera hardware
>> on SoC comes with a media processor (ISP), with does a lot more than just
>> capturing, allowing complex algorithms to enhance image quality in runtime.
>> Those algorithms are known as 3A - an acronym for 3 other acronyms:
>>
>>         - AE (Auto Exposure);
>>         - AF (Auto Focus);
>>         - AWB (Auto White Balance).
>>
>> The main reason that drove the MC design is that the 3A algorithms (that is
>> the 3A control loop, and sometimes part of the image processing itself) often
>> need to run, at least partially, on the CPU. As a kernel-space implementation
>> wasn't possible, we needed a lower-level UAPI.
>>
>> Setting a camera with such ISPs are harder because the pipelines to be
>> set actually depends the requirements for those 3A algorithms to run.
>> Also, usually, the 3A algorithms use some chipset-specific userspace API,
>> that exports some image properties, calculated by the ISP, to speed up
>> the convergence of those algorithms.
>>
>> Btw, usually, the 3A algorithms are IP-protected, provided by vendors
>> as binary only blobs, although there are a few OSS implementations.
>>
>> Part of the problem is that, so far, there isn't a proper userspace API
>> to implement 3A libraries. Once we have an userspace camera stack, we
>> hope that we'll gradually increase the number and quality of open-source
>> 3A stacks.
>>
> [snip]
>>
>> 2.2 Modern hardware is starting to come with "complex" camera ISP
>> -----------------------------------------------------------------
>>
>> While mc-based devices were limited to SoC, it was easy to
>> "delegate" the task of talking with the hardware to the
>> embedded hardware designers.
>>
>> However, this is changing. Dell Latitude 5285 laptop is a standard
>> PC with an i3-core, i5-core or i7-core CPU, with comes with the
>> Intel IMU3 ISP hardware[2].
>
> IPU3 :)
>
>>
>> [2] https://www.spinics.net/lists/linux-usb/msg167478.html
>>
>> There, instead of an USB camera, the hardware is equipped with a
>> MC-based ISP, connected to its camera. Currently, despite having
>> a Kernel driver for it, the camera doesn't work with any
>> userspace application.
>>
>> I'm also aware of other projects that are considering the usage of
>> mc-based devices for non-dedicated hardware.
>>
> [snip]
>>
>> 3.2 libv4l2 support for 3A algorithms
>> =====================================
>>
>> The 3A algorithm handing is highly dependent on the hardware. The
>> idea here is to allow libv4l to have a set of 3A algorithms that
>> will be specific to certain mc-based hardware.
>>
>> One requirement, if we want vendor stacks to use our solution, is that
>> it should allow allow external closed-source algorithms to run as well.
>>
>> The 3A library API must be standardized, to allow the closed-source
>> vendor implementation to be replaced by an open-source implementation
>> should someone have the time and energy (and qualifications) to write
>> one.
>>
>> Sandboxed execution of the 3A library must be possible as closed-source
>> can't always be blindly trusted. This includes the ability to wrap the
>> library in a daemon should the platform's multimedia stack wishes
>> and to avoid any direct access to the kernel devices by the 3A library
>> itself (all accesses should be marshaled by the camera stack).
>>
>> Please note that this daemon is *not* a camera daemon that would
>> communicates with the V4L2 driver through a custom back channel.
>>
>> The decision to run the 3A library in a sandboxed process or to call
>> it directly from the camera stack should be left to the camera stack
>> and to the platform integrator, and should not be visible by the 3A
>> library.
>>
>> The 3A library must be usable on major Linux-based camera stacks (the
>> Android and Chrome OS camera HALs are certainly important targets,
>> more can be added) unmodified, which will allow usage of the vendor
>> binary provided for Chrome OS or Android on regular Linux systems.
>
> This is quite an interesting idea and it would be really useful if it
> could be done. I'm kind of worried, though, about Android in
> particular, since the execution environment in Android differs
> significantly from a regular Linux distributions (including Chrome OS,
> which is not so far from such), namely:
> - different libc (bionic) and dynamic linker - I guess this could be
> solved by static linking?
> - dedicated toolchains - perhaps not much of a problem if the per-arch
> ABI is the same?
>
>>
>> It would make sense to design a modular camera stack, and try to make
>> most components as platform-independent as possible. This should include:
>>
>> - the kernel drivers (V4L2-compliant and usable without any closed-source
>>   userspace component);
>> - the 3A library
>> - any other component that could be shared (for instance a possible
>>   request API library).
>>
>> The rest of the code will mostly be glue around those components to
>> integrate them in a particular camera stack, and should be as
>> platform-agnostic as possible.
>>
>> In the case of the Android camera HAL, ideally it would be a glue that
>> could be used with different camera vendors (probably with some kind of
>> vendor-specific configuration, or possibly with a separate vendor-specific
>> component to handle pipeline configuration).
>>
>> 4 Complex camera workshop
>> =========================
>>
>> The workshop will be happening in Tokyo, Japan, at Jun, 19, at the
>> google offices. The location is:
>>
>> 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
>
> Nearest station exits:
> - Hibiya line Roppongi station exit 1c (recommended)
> - Oedo line Roppongi station exit 3 (and few minutes walk)
>
>>
>> 4.1 Physical Attendees
>> ======================
>>
>> Tomasz Figa <tfiga@google.com>
>> Mauro Carvalho Chehab <Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
>> Kieran Bingham <kieran.bingham@ideasonboard.com>
>> Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> Niklas Söderlund <niklas.soderlund@ragnatech.se>
>> Zheng, Jian Xu Zheng <jian.xu.zheng@intel.com>
>>
>> Anywone else?
>
> Looking at latest reply in this thread:
>
> jacopo mondi <jacopo@jmondi.org>
>
> Anyone else, please tell me beforehand (at least 1-2 days before), as
> I need to take care of building access, since it's a multi-tenant
> office building. I'll contact each attendee separately with further
> details by email.
>
>>
>> 4.2. Attendees Via Google Hangouts
>> ==================================
>>
>> Hans Verkuil <hverkuil@xs4all.nl> - Via Google Hangouts - maybe only on afternoon
>> Javier Martinez Canillas <javier@dowhile0.org> - Via Google Hangouts - only on reasonable TZ-compatible-hours
>
> What time zone would that be? I guess we could try to tweak the agenda
> to take this into account.
>

Wim, Nicolas and myself are in CEST (UTC +2). The best time for Wim to
do the PipeWire presentation would be 10:30 am CEST.

Best regards,
Javier

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Mauro Carvalho Chehab]

Em Wed, 6 Jun 2018 13:19:39 +0900
Tomasz Figa <tfiga@chromium.org> escreveu:

> On Mon, Jun 4, 2018 at 10:33 PM Mauro Carvalho Chehab
> <mchehab+samsung@kernel.org> wrote:
> >
> > Hi all,
> >
> > I consolidated hopefully all comments I receive on the past announcement
> > with regards to the complex camera workshop we're planning to happen in
> > Tokyo, just before the Open Source Summit in Japan.
> >
> > The main focus of the workshop is to allow supporting devices with MC-based
> > hardware connected to a camera.
> >
> > I'm enclosing a detailed description of the problem, in order to
> > allow the interested parties to be at the same page.
> >
> > We need to work towards an agenda for the meeting.
> >
> > From my side, I think we should have at least the following topics at
> > the agenda:
> >
> > - a quick review about what's currently at libv4l2;
> > - a presentation about PipeWire solution;
> > - a discussion with the requirements for the new solution;
> > - a discussion about how we'll address - who will do what.  
> 
> I believe Intel's Jian Xu would be able to give us some brief
> introduction to IPU3 hardware architecture and possibly also upcoming
> hardware generations as well.

That would be great!

> My experience with existing generations of ISPs from other vendors is
> that the main principles of operation are very similar to the model
> represented by IPU3 and very much different to the OMAP3 example
> mentioned by Mauro below. I further commented on it below.
> 
> >
> > Comments? Suggestions?
> >
> > Are there anyone else planning to either be there physically or via
> > Google Hangouts?
> >
> > Tomaz,
> >
> > Do you have any limit about the number of people that could join us
> > via Google Hangouts?
> >  
> 
> Technically, Hangouts should be able to work with really huge
> multi-party conferences. There is obviously some limitation on client
> side, since thumbnails of participants need to be decoded at real
> time, so even if the resolution is low, if the client is very slow,
> there might be some really bad frame drop happening on client side.
> 
> However, I often have meetings with around 8 parties and it tends to
> work fine. We can also disable video of all participants, who don't
> need to present anything at the moment and the problem would go away
> completely.

Ok, good!

> > Regards,
> > Mauro
> >
> > ---
> >
> > 1. Introduction
> > ===============
> >
> > 1.1 V4L2 Kernel aspects
> > -----------------------
> >
> > The media subsystem supports two types of devices:
> >
> > - "traditional" media hardware, supported via V4L2 API. On such hardware,
> >   opening a single device node (usually /dev/video0) is enough to control
> >   the entire device. We call it as devnode-based devices.
> >   An application sometimes may need to use multiple video nodes with
> >   devnode-based drivers to capture multiple streams in parallel
> >   (when the hardware allows it of course). That's quite common for
> >   Analog TV devices, where both /dev/video0 and /dev/vbi0 are opened
> >   at the same time.
> >
> > - Media-controller based devices. On those devices, there are typically
> >   several /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus
> >   a media controller device node (usually /dev/media0).
> >   We call it as mc-based devices. Controlling the hardware require
> >   opening the media device (/dev/media0), setup the pipeline and adjust
> >   the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
> >   by /dev/video?.
> >
> > In other words, both configuration and streaming go through the video
> > device node on devnode-based drivers, while video device nodes are used
> > used for streaming on mc-based drivers.
> >
> > With devnode-based drivers, "standard" media applications, including open
> > source ones (Camorama, Cheese, Xawtv, Firefox, Chromium, ...) and closed
> > source ones (Skype, Chrome, ...) support devnode-based devices[1]. Also,
> > when just one media device is connected, the streaming/control device
> > is typically /dev/video0.
> >
> > [1] It should be noticed that closed-source applications tend to have
> > various bugs that prevent them from working properly on many devnode-based
> > devices. Due to that, some additional blocks were requred at libv4l to
> > support some of them. Skype is a good example, as we had to include a
> > software scaler in libv4l to make it happy. So in practice not everything
> > works smoothly with closed-source applications with devnode-based drivers.
> > A few such adjustments were also made on some drivers and/or libv4l, in
> > order to fulfill some open-source app requirements.
> >
> > Support for mc-based devices currently require an specialized application
> > in order to prepare the device for its usage (setup pipelines, adjust
> > hardware controls, etc). Once pipeline is set, the streaming goes via
> > /dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
> > be opened, in order to implement algorithms designed to make video quality
> > reasonable.  
> 
> To further complicate the problem, on many modern imaging subsystems
> (Intel IPU3, Rockchip RKISP1), there is more than 1 video output
> (CAPTURE device), for example:
> 1) full resolution capture stream and
> 2) downscaled preview stream.

OMAP3 also has both full-res and downscaled for previews, on separate
/dev/video.

On a "simple" usecase (like PC apps and videoconference), just one
/dev/video is enough. For devices used to take static pictures,
both streams are important.

> Moreover, many ISPs also produce per-frame metadata (statistics) for
> 3A algorithms, which then produces per-frame metadata (parameters) for
> processing of next frame. These would be also exposed as /dev/video?
> nodes with respective V4L2_BUF_TYPE_META_* queues.

True, but the metadata frames don't need to go the the application,
as it will be consumed by the 3A logic.

> 
> It is complicated even more on systems with separate input (e.g. CSI2)
> and processing (ISP) hardware, such as Intel IPU3. In such case, the
> raw frames captured from the CSI2 interface directly are not usable
> for end-user applications. This means that some component in userspace
> needs to forward the raw frames to the ISP and only the output of the
> ISP can be passed to the application.

Yes. Nowadays, several devices do the same.

> > On such devices, it is not uncommon that the device used by the
> > application to be a random number (on OMAP3 driver, typically, is either
> > /dev/video4 or /dev/video6).
> >
> > One example of such hardware is at the OMAP3-based hardware:
> >
> >         http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png
> >
> > On the picture, there's a graph with the hardware blocks in blue/dark/blue
> > and the corresponding devnode interfaces in yellow.
> >
> > The mc-based approach was taken when support for Nokia N9/N900 cameras
> > was added (with has OMAP3 SoC). It is required because the camera hardware
> > on SoC comes with a media processor (ISP), with does a lot more than just
> > capturing, allowing complex algorithms to enhance image quality in runtime.
> > Those algorithms are known as 3A - an acronym for 3 other acronyms:
> >
> >         - AE (Auto Exposure);
> >         - AF (Auto Focus);
> >         - AWB (Auto White Balance).
> >
> > The main reason that drove the MC design is that the 3A algorithms (that is
> > the 3A control loop, and sometimes part of the image processing itself) often
> > need to run, at least partially, on the CPU. As a kernel-space implementation
> > wasn't possible, we needed a lower-level UAPI.
> >
> > Setting a camera with such ISPs are harder because the pipelines to be
> > set actually depends the requirements for those 3A algorithms to run.
> > Also, usually, the 3A algorithms use some chipset-specific userspace API,
> > that exports some image properties, calculated by the ISP, to speed up
> > the convergence of those algorithms.
> >
> > Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> > as binary only blobs, although there are a few OSS implementations.
> >
> > Part of the problem is that, so far, there isn't a proper userspace API
> > to implement 3A libraries. Once we have an userspace camera stack, we
> > hope that we'll gradually increase the number and quality of open-source
> > 3A stacks.
> >  
> [snip]
> >
> > 2.2 Modern hardware is starting to come with "complex" camera ISP
> > -----------------------------------------------------------------
> >
> > While mc-based devices were limited to SoC, it was easy to
> > "delegate" the task of talking with the hardware to the
> > embedded hardware designers.
> >
> > However, this is changing. Dell Latitude 5285 laptop is a standard
> > PC with an i3-core, i5-core or i7-core CPU, with comes with the
> > Intel IMU3 ISP hardware[2].  
> 
> IPU3 :)

Yeah, I noticed the typo too late. I actually fixed it at the announcement
at linuxtv.org.

> >
> > [2] https://www.spinics.net/lists/linux-usb/msg167478.html
> >
> > There, instead of an USB camera, the hardware is equipped with a
> > MC-based ISP, connected to its camera. Currently, despite having
> > a Kernel driver for it, the camera doesn't work with any
> > userspace application.
> >
> > I'm also aware of other projects that are considering the usage of
> > mc-based devices for non-dedicated hardware.
> >  
> [snip]
> >
> > 3.2 libv4l2 support for 3A algorithms
> > =====================================
> >
> > The 3A algorithm handing is highly dependent on the hardware. The
> > idea here is to allow libv4l to have a set of 3A algorithms that
> > will be specific to certain mc-based hardware.
> >
> > One requirement, if we want vendor stacks to use our solution, is that
> > it should allow allow external closed-source algorithms to run as well.
> >
> > The 3A library API must be standardized, to allow the closed-source
> > vendor implementation to be replaced by an open-source implementation
> > should someone have the time and energy (and qualifications) to write
> > one.
> >
> > Sandboxed execution of the 3A library must be possible as closed-source
> > can't always be blindly trusted. This includes the ability to wrap the
> > library in a daemon should the platform's multimedia stack wishes
> > and to avoid any direct access to the kernel devices by the 3A library
> > itself (all accesses should be marshaled by the camera stack).
> >
> > Please note that this daemon is *not* a camera daemon that would
> > communicates with the V4L2 driver through a custom back channel.
> >
> > The decision to run the 3A library in a sandboxed process or to call
> > it directly from the camera stack should be left to the camera stack
> > and to the platform integrator, and should not be visible by the 3A
> > library.
> >
> > The 3A library must be usable on major Linux-based camera stacks (the
> > Android and Chrome OS camera HALs are certainly important targets,
> > more can be added) unmodified, which will allow usage of the vendor
> > binary provided for Chrome OS or Android on regular Linux systems.  
> 
> This is quite an interesting idea and it would be really useful if it
> could be done. I'm kind of worried, though, about Android in
> particular, since the execution environment in Android differs
> significantly from a regular Linux distributions (including Chrome OS,
> which is not so far from such), namely:
> - different libc (bionic) and dynamic linker - I guess this could be
> solved by static linking?

Static link is one possible solution. IMHO, we should try to make it
use just a C library (if possible) and be sure that it will also compile
with bionic/ulibc in order to make it easier to be used by Android and
other embedded distros.

> - dedicated toolchains - perhaps not much of a problem if the per-arch
> ABI is the same?

Depending on library dependency, we could likely make it work with more
than one toolchain. I guess acconfig works with Android, right?
If so, it could auto-adjust to the different toolchains everywhere.

> > It would make sense to design a modular camera stack, and try to make
> > most components as platform-independent as possible. This should include:
> >
> > - the kernel drivers (V4L2-compliant and usable without any closed-source
> >   userspace component);
> > - the 3A library
> > - any other component that could be shared (for instance a possible
> >   request API library).
> >
> > The rest of the code will mostly be glue around those components to
> > integrate them in a particular camera stack, and should be as
> > platform-agnostic as possible.
> >
> > In the case of the Android camera HAL, ideally it would be a glue that
> > could be used with different camera vendors (probably with some kind of
> > vendor-specific configuration, or possibly with a separate vendor-specific
> > component to handle pipeline configuration).
> >
> > 4 Complex camera workshop
> > =========================
> >
> > The workshop will be happening in Tokyo, Japan, at Jun, 19, at the
> > google offices. The location is:
> >
> > 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F  
> 
> Nearest station exits:
> - Hibiya line Roppongi station exit 1c (recommended)
> - Oedo line Roppongi station exit 3 (and few minutes walk)
> 
> >
> > 4.1 Physical Attendees
> > ======================
> >
> > Tomasz Figa <tfiga@google.com>
> > Mauro Carvalho Chehab <Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> > Kieran Bingham <kieran.bingham@ideasonboard.com>
> > Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Niklas Söderlund <niklas.soderlund@ragnatech.se>
> > Zheng, Jian Xu Zheng <jian.xu.zheng@intel.com>
> >
> > Anywone else?  
> 
> Looking at latest reply in this thread:
> 
> jacopo mondi <jacopo@jmondi.org>
> 
> Anyone else, please tell me beforehand (at least 1-2 days before), as
> I need to take care of building access, since it's a multi-tenant
> office building. I'll contact each attendee separately with further
> details by email.
> 
> >
> > 4.2. Attendees Via Google Hangouts
> > ==================================
> >
> > Hans Verkuil <hverkuil@xs4all.nl> - Via Google Hangouts - maybe only on afternoon
> > Javier Martinez Canillas <javier@dowhile0.org> - Via Google Hangouts - only on reasonable TZ-compatible-hours  
> 
> What time zone would that be? I guess we could try to tweak the agenda
> to take this into account.

Thanks,
Mauro

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Tomasz Figa]

On Thu, Jun 7, 2018 at 1:26 AM Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
>
> Em Wed, 6 Jun 2018 13:19:39 +0900
> Tomasz Figa <tfiga@chromium.org> escreveu:
>
> > On Mon, Jun 4, 2018 at 10:33 PM Mauro Carvalho Chehab
> > <mchehab+samsung@kernel.org> wrote:
[snip]
> > > 3.2 libv4l2 support for 3A algorithms
> > > =====================================
> > >
> > > The 3A algorithm handing is highly dependent on the hardware. The
> > > idea here is to allow libv4l to have a set of 3A algorithms that
> > > will be specific to certain mc-based hardware.
> > >
> > > One requirement, if we want vendor stacks to use our solution, is that
> > > it should allow allow external closed-source algorithms to run as well.
> > >
> > > The 3A library API must be standardized, to allow the closed-source
> > > vendor implementation to be replaced by an open-source implementation
> > > should someone have the time and energy (and qualifications) to write
> > > one.
> > >
> > > Sandboxed execution of the 3A library must be possible as closed-source
> > > can't always be blindly trusted. This includes the ability to wrap the
> > > library in a daemon should the platform's multimedia stack wishes
> > > and to avoid any direct access to the kernel devices by the 3A library
> > > itself (all accesses should be marshaled by the camera stack).
> > >
> > > Please note that this daemon is *not* a camera daemon that would
> > > communicates with the V4L2 driver through a custom back channel.
> > >
> > > The decision to run the 3A library in a sandboxed process or to call
> > > it directly from the camera stack should be left to the camera stack
> > > and to the platform integrator, and should not be visible by the 3A
> > > library.
> > >
> > > The 3A library must be usable on major Linux-based camera stacks (the
> > > Android and Chrome OS camera HALs are certainly important targets,
> > > more can be added) unmodified, which will allow usage of the vendor
> > > binary provided for Chrome OS or Android on regular Linux systems.
> >
> > This is quite an interesting idea and it would be really useful if it
> > could be done. I'm kind of worried, though, about Android in
> > particular, since the execution environment in Android differs
> > significantly from a regular Linux distributions (including Chrome OS,
> > which is not so far from such), namely:
> > - different libc (bionic) and dynamic linker - I guess this could be
> > solved by static linking?
>
> Static link is one possible solution. IMHO, we should try to make it
> use just a C library (if possible) and be sure that it will also compile
> with bionic/ulibc in order to make it easier to be used by Android and
> other embedded distros.
>
> > - dedicated toolchains - perhaps not much of a problem if the per-arch
> > ABI is the same?
>
> Depending on library dependency, we could likely make it work with more
> than one toolchain. I guess acconfig works with Android, right?
> If so, it could auto-adjust to the different toolchains everywhere.

That works for open source libraries obviously. I was thinking more
about the closed source 3A libraries coming from Android, since we
can't recompile them.

Best regards,
Tomasz

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Mauro Carvalho Chehab]

Em Thu, 7 Jun 2018 16:47:50 +0900
Tomasz Figa <tfiga@chromium.org> escreveu:

> On Thu, Jun 7, 2018 at 1:26 AM Mauro Carvalho Chehab
> <mchehab+samsung@kernel.org> wrote:
> >
> > Em Wed, 6 Jun 2018 13:19:39 +0900
> > Tomasz Figa <tfiga@chromium.org> escreveu:
> >  
> > > On Mon, Jun 4, 2018 at 10:33 PM Mauro Carvalho Chehab
> > > <mchehab+samsung@kernel.org> wrote:  
> [snip]
> > > > 3.2 libv4l2 support for 3A algorithms
> > > > =====================================
> > > >
> > > > The 3A algorithm handing is highly dependent on the hardware. The
> > > > idea here is to allow libv4l to have a set of 3A algorithms that
> > > > will be specific to certain mc-based hardware.
> > > >
> > > > One requirement, if we want vendor stacks to use our solution, is that
> > > > it should allow allow external closed-source algorithms to run as well.
> > > >
> > > > The 3A library API must be standardized, to allow the closed-source
> > > > vendor implementation to be replaced by an open-source implementation
> > > > should someone have the time and energy (and qualifications) to write
> > > > one.
> > > >
> > > > Sandboxed execution of the 3A library must be possible as closed-source
> > > > can't always be blindly trusted. This includes the ability to wrap the
> > > > library in a daemon should the platform's multimedia stack wishes
> > > > and to avoid any direct access to the kernel devices by the 3A library
> > > > itself (all accesses should be marshaled by the camera stack).
> > > >
> > > > Please note that this daemon is *not* a camera daemon that would
> > > > communicates with the V4L2 driver through a custom back channel.
> > > >
> > > > The decision to run the 3A library in a sandboxed process or to call
> > > > it directly from the camera stack should be left to the camera stack
> > > > and to the platform integrator, and should not be visible by the 3A
> > > > library.
> > > >
> > > > The 3A library must be usable on major Linux-based camera stacks (the
> > > > Android and Chrome OS camera HALs are certainly important targets,
> > > > more can be added) unmodified, which will allow usage of the vendor
> > > > binary provided for Chrome OS or Android on regular Linux systems.  
> > >
> > > This is quite an interesting idea and it would be really useful if it
> > > could be done. I'm kind of worried, though, about Android in
> > > particular, since the execution environment in Android differs
> > > significantly from a regular Linux distributions (including Chrome OS,
> > > which is not so far from such), namely:
> > > - different libc (bionic) and dynamic linker - I guess this could be
> > > solved by static linking?  
> >
> > Static link is one possible solution. IMHO, we should try to make it
> > use just a C library (if possible) and be sure that it will also compile
> > with bionic/ulibc in order to make it easier to be used by Android and
> > other embedded distros.
> >  
> > > - dedicated toolchains - perhaps not much of a problem if the per-arch
> > > ABI is the same?  
> >
> > Depending on library dependency, we could likely make it work with more
> > than one toolchain. I guess acconfig works with Android, right?
> > If so, it could auto-adjust to the different toolchains everywhere.  
> 
> That works for open source libraries obviously. I was thinking more
> about the closed source 3A libraries coming from Android, since we
> can't recompile them.

Ah! It probably makes sense to place them on some sandboxed environment.
If we're using that, it probably makes sense to have them running
on a sort of daemon with a sockets-based API.

If we're willing to do that, it doesn't really matter how the 3A
was implemented. It can even be in Java. All it matters is to have
a way to plug the library to it. A config file could provide such
link, telling what 3A library should be used (and, eventually, what
commands should be used to start/stop the daemon).

Thanks,
Mauro

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Alexandre Courbot]

On Mon, Jun 4, 2018 at 10:33 PM Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
>
> Hi all,
>
> I consolidated hopefully all comments I receive on the past announcement
> with regards to the complex camera workshop we're planning to happen in
> Tokyo, just before the Open Source Summit in Japan.
>
> The main focus of the workshop is to allow supporting devices with MC-based
> hardware connected to a camera.
>
> I'm enclosing a detailed description of the problem, in order to
> allow the interested parties to be at the same page.
>
> We need to work towards an agenda for the meeting.
>
> From my side, I think we should have at least the following topics at
> the agenda:
>
> - a quick review about what's currently at libv4l2;
> - a presentation about PipeWire solution;
> - a discussion with the requirements for the new solution;
> - a discussion about how we'll address - who will do what.
>
> Comments? Suggestions?
>
> Are there anyone else planning to either be there physically or via
> Google Hangouts?
>
> Tomaz,
>
> Do you have any limit about the number of people that could join us
> via Google Hangouts?
>
>
> Regards,
> Mauro
>
> ---
>
> 1. Introduction
> ===============
>
> 1.1 V4L2 Kernel aspects
> -----------------------
>
> The media subsystem supports two types of devices:
>
> - "traditional" media hardware, supported via V4L2 API. On such hardware,
>   opening a single device node (usually /dev/video0) is enough to control
>   the entire device. We call it as devnode-based devices.
>   An application sometimes may need to use multiple video nodes with
>   devnode-based drivers to capture multiple streams in parallel
>   (when the hardware allows it of course). That's quite common for
>   Analog TV devices, where both /dev/video0 and /dev/vbi0 are opened
>   at the same time.
>
> - Media-controller based devices. On those devices, there are typically
>   several /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus
>   a media controller device node (usually /dev/media0).
>   We call it as mc-based devices. Controlling the hardware require
>   opening the media device (/dev/media0), setup the pipeline and adjust
>   the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
>   by /dev/video?.
>
> In other words, both configuration and streaming go through the video
> device node on devnode-based drivers, while video device nodes are used
> used for streaming on mc-based drivers.
>
> With devnode-based drivers, "standard" media applications, including open
> source ones (Camorama, Cheese, Xawtv, Firefox, Chromium, ...) and closed
> source ones (Skype, Chrome, ...) support devnode-based devices[1]. Also,
> when just one media device is connected, the streaming/control device
> is typically /dev/video0.
>
> [1] It should be noticed that closed-source applications tend to have
> various bugs that prevent them from working properly on many devnode-based
> devices. Due to that, some additional blocks were requred at libv4l to
> support some of them. Skype is a good example, as we had to include a
> software scaler in libv4l to make it happy. So in practice not everything
> works smoothly with closed-source applications with devnode-based drivers.
> A few such adjustments were also made on some drivers and/or libv4l, in
> order to fulfill some open-source app requirements.
>
> Support for mc-based devices currently require an specialized application
> in order to prepare the device for its usage (setup pipelines, adjust
> hardware controls, etc). Once pipeline is set, the streaming goes via
> /dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
> be opened, in order to implement algorithms designed to make video quality
> reasonable. On such devices, it is not uncommon that the device used by the
> application to be a random number (on OMAP3 driver, typically, is either
> /dev/video4 or /dev/video6).
>
> One example of such hardware is at the OMAP3-based hardware:
>
>         http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png
>
> On the picture, there's a graph with the hardware blocks in blue/dark/blue
> and the corresponding devnode interfaces in yellow.
>
> The mc-based approach was taken when support for Nokia N9/N900 cameras
> was added (with has OMAP3 SoC). It is required because the camera hardware
> on SoC comes with a media processor (ISP), with does a lot more than just
> capturing, allowing complex algorithms to enhance image quality in runtime.
> Those algorithms are known as 3A - an acronym for 3 other acronyms:
>
>         - AE (Auto Exposure);
>         - AF (Auto Focus);
>         - AWB (Auto White Balance).
>
> The main reason that drove the MC design is that the 3A algorithms (that is
> the 3A control loop, and sometimes part of the image processing itself) often
> need to run, at least partially, on the CPU. As a kernel-space implementation
> wasn't possible, we needed a lower-level UAPI.
>
> Setting a camera with such ISPs are harder because the pipelines to be
> set actually depends the requirements for those 3A algorithms to run.
> Also, usually, the 3A algorithms use some chipset-specific userspace API,
> that exports some image properties, calculated by the ISP, to speed up
> the convergence of those algorithms.
>
> Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> as binary only blobs, although there are a few OSS implementations.
>
> Part of the problem is that, so far, there isn't a proper userspace API
> to implement 3A libraries. Once we have an userspace camera stack, we
> hope that we'll gradually increase the number and quality of open-source
> 3A stacks.
>
>
> 1.2 V4L2 userspace aspects
> --------------------------
>
> Back when USB cameras were introduced, the hardware were really simple:
> they had a CMOS or CCD camera sensor and a chip that bridges the data
> though USB. Camera sensors typically provide data using a bayer
> format, but they usually have their own proprietary ways to pack the data
> at the USB bridges, in order to reduce the bandwidth (the first
> implementations were using USB version 1.1).
>
> So, V4L2 has a myriad of different formats, in order to match each
> camera sensor format. At the end of the day, applications were
> able to use only a subset of the available hardware, since they need
> to come with format converters for all formats the developer uses
> (usually a very small subset of the available ones).
>
> That's said, newer cameras are converged towards a small set of
> standard formats - except for secondary data streams (like depth
> maps). Yet, industrial cameras, and newer technologies, like
> 3D, light-field, etc may still bring newer formats.
>
> To end with this mess, an userspace library was written, called libv4l.
> It supports all those proprietary formats. So, applications can use
> a RGB or YUV format, without needing to concern about conversions.
>
> The way it works is by adding wrappers to system calls: open, close,
> ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> at the source code of the apps, all it was needed is to prepend the
> existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.
>
> All open source apps we know now supports libv4l. On a few (like
> gstreamer), support for it is optional.
>
> It should be noted that libv4l also handles scaling and rough auto-gain
> and auto-white balance that are required by  some cameras to achieve
> a usable image.
>
> In order to support closed source, another wrapper was added, allowing
> to call any closed source application to use it, by using LD_PRELOAD.
> For example, using skype with it is as simple as calling it with:
>
>         $ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so /usr/bin/skypeforlinux
>
>
> 2. Current problems
> ===================
>
> 2.1 Libv4l can slow image handling
> ----------------------------------
>
> Nowadays, almost all new "simple" cameras are connected via USB using
> the UVC class (USB Video Class). UVC standardized the allowed formats,
> and most apps just implement them. The UVC hardware is more complex,
> having format converters inside it. So, for most usages, format
> conversion isn't needed anymore.
>
> The need of doing format conversion in software makes libv4l slow,
> requiring lots of CPU usage in order to convert a 4K or 8K format,
> being even worse with 3D cameras.
>
> Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> DMA_BUFFER currently doesn't work with libv4l.
>
> Right now, gstreamer defaults to not enable libv4l2, due to several
> reasons:
>   - Crash when CREATE_BUFS is being used;
>   - Crash in the jpeg decoder (when frames are corrupted);
>   - App exporting DMABuf need to be aware of emulation, otherwise the
>     DMABuf exported are in the orignal format;
>   - RW emulation only initialize the queue on first read (causing
>     userspace poll() to fail);
>   - Signature of v4l2_mmap does not match mmap() (minor);
>   - The colorimetry does not seem emulated when conversion;
>   - Sub-optimal locking (at least deadlocks were fixed).
>
> Most of the above are due to new features added to Kernel uAPI, but
> not added to libv4l2.
>
> These issues are already worked around in GStreamer, but with the lost
> of features of course. There is other cases were something worked
> without libv4l2, but didn't work with libv4l2, but Gstreamer developers
> haven't tracked down the cause yet. Since 1.14, libv4l2 can be enabled
> at run-time using env GST_V4L2_USE_LIBV4L2=1.
>
> 2.2 Modern hardware is starting to come with "complex" camera ISP
> -----------------------------------------------------------------
>
> While mc-based devices were limited to SoC, it was easy to
> "delegate" the task of talking with the hardware to the
> embedded hardware designers.
>
> However, this is changing. Dell Latitude 5285 laptop is a standard
> PC with an i3-core, i5-core or i7-core CPU, with comes with the
> Intel IMU3 ISP hardware[2].
>
> [2] https://www.spinics.net/lists/linux-usb/msg167478.html
>
> There, instead of an USB camera, the hardware is equipped with a
> MC-based ISP, connected to its camera. Currently, despite having
> a Kernel driver for it, the camera doesn't work with any
> userspace application.
>
> I'm also aware of other projects that are considering the usage of
> mc-based devices for non-dedicated hardware.
>
> 3. How to solve it?
> ===================
>
> That's the main focus of the meeting :-)
>
> From a previous discussion I had with media sub-maintainers, there are
> at least two actions that seem required. I'm listing them below as
> an starting point for the discussions, but we can eventually come up
> with some different approach after the meeting.
>
> 3.1 Library support for mc-based hardware
> =========================================
>
> In order to support those hardware, we'll need to do some redesign,
> mainly at userspace, via a library[3] that will replace/compliment
> libv4l2.
>
> The idea is to work on a library API that will allow to split the
> format conversion on a separate part of it, add support
> for DMA Buffer and come up with a way for the library to work
> transparently with both devnode-based and mc-based hardware.
>
> The library should be capable of doing dynamic pipeline setups,
> including the functionality of allow tranfering buffers between
> different media devices, as, on several cases, a a camera input
> device node should be connected to a m2m device that would be
> doing some image processing, and providing output data on a
> standard video output format.
>
> One possibility is to redesign libv4l2. It should be noticed that
> Videolan people is working on a daemon that could also provide
> another way to implement it[4].
>
> Once we get a proper camera stack, we need to support traditional
> applications that use libv4l2 and/or the new library directly. To
> that end we  will need to expose the libv4l2 API on top of the
> camera stack (through libv4l2 if it makes sense, or through a separate
> implementation) and implement a transparent LD_PRELOAD-based library.
>
> That envolves adding capacity to the library to setup hardware pipelines
> and to propagate controls among their sub-devices. Eventually, part
> of it will be done in Kernel.
>
> That should give performance increase at the library and would allow
> gstreamer to use it by default, without compromising performance.
>
> [3] I don't discard that some Kernel changes could also be part of the
> solution, like, for example, doing control propagation along the pipeline
> on simple use case scenarios.
>
> [4] With the venue of sandboxed application, there is a need to control
> access to cameras through a daemon. The same daemon is also used to
> control access to screen capture on Wayland (instead of letting any
> random application capture your screen, like on X11). The effort is
> lead by the desktop team at RedHat. PipeWire already have V4L2 native
> support and is integrated in GStreamer already in a way that it can
> totally replace the V4L2 capture component there. PipeWire is plugin based,
> so more type of camera support can be added. Remote daemon can also provide
> streams, as this is the case for compositors and screen casting. An extra
> benefit is that you can have multiple application reading frames from the
> same camera. It also allow sandboxed  application (the do not have access
> to /dev) to use the cameras. In this context, proprietary or HW specific
> algorithm could be implemented in userspace as PipeWire plugins, and then
> application will automatically be enable to enumerate and use these.
>
> 3.2 libv4l2 support for 3A algorithms
> =====================================
>
> The 3A algorithm handing is highly dependent on the hardware. The
> idea here is to allow libv4l to have a set of 3A algorithms that
> will be specific to certain mc-based hardware.
>
> One requirement, if we want vendor stacks to use our solution, is that
> it should allow allow external closed-source algorithms to run as well.
>
> The 3A library API must be standardized, to allow the closed-source
> vendor implementation to be replaced by an open-source implementation
> should someone have the time and energy (and qualifications) to write
> one.
>
> Sandboxed execution of the 3A library must be possible as closed-source
> can't always be blindly trusted. This includes the ability to wrap the
> library in a daemon should the platform's multimedia stack wishes
> and to avoid any direct access to the kernel devices by the 3A library
> itself (all accesses should be marshaled by the camera stack).
>
> Please note that this daemon is *not* a camera daemon that would
> communicates with the V4L2 driver through a custom back channel.
>
> The decision to run the 3A library in a sandboxed process or to call
> it directly from the camera stack should be left to the camera stack
> and to the platform integrator, and should not be visible by the 3A
> library.
>
> The 3A library must be usable on major Linux-based camera stacks (the
> Android and Chrome OS camera HALs are certainly important targets,
> more can be added) unmodified, which will allow usage of the vendor
> binary provided for Chrome OS or Android on regular Linux systems.
>
> It would make sense to design a modular camera stack, and try to make
> most components as platform-independent as possible. This should include:
>
> - the kernel drivers (V4L2-compliant and usable without any closed-source
>   userspace component);
> - the 3A library
> - any other component that could be shared (for instance a possible
>   request API library).
>
> The rest of the code will mostly be glue around those components to
> integrate them in a particular camera stack, and should be as
> platform-agnostic as possible.
>
> In the case of the Android camera HAL, ideally it would be a glue that
> could be used with different camera vendors (probably with some kind of
> vendor-specific configuration, or possibly with a separate vendor-specific
> component to handle pipeline configuration).
>
> 4 Complex camera workshop
> =========================
>
> The workshop will be happening in Tokyo, Japan, at Jun, 19, at the
> google offices. The location is:
>
> 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
>
> 4.1 Physical Attendees
> ======================
>
> Tomasz Figa <tfiga@google.com>
> Mauro Carvalho Chehab <Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> Kieran Bingham <kieran.bingham@ideasonboard.com>
> Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Niklas Söderlund <niklas.soderlund@ragnatech.se>
> Zheng, Jian Xu Zheng <jian.xu.zheng@intel.com>
>
> Anywone else?

I will probably be there as well.

Alexandre Courbot <acourbot@google.com>

Cheers,
Alex.

>
> 4.2. Attendees Via Google Hangouts
> ==================================
>
> Hans Verkuil <hverkuil@xs4all.nl> - Via Google Hangouts - maybe only on afternoon
> Javier Martinez Canillas <javier@dowhile0.org> - Via Google Hangouts - only on reasonable TZ-compatible-hours
> Ricky - Google camera team in Taipei - Via Google Hangouts
>
> Anywone else?

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Tomasz Figa]

[+CC Ricky]

On Mon, Jun 4, 2018 at 10:33 PM Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
>
> Hi all,
>
> I consolidated hopefully all comments I receive on the past announcement
> with regards to the complex camera workshop we're planning to happen in
> Tokyo, just before the Open Source Summit in Japan.
>
> The main focus of the workshop is to allow supporting devices with MC-based
> hardware connected to a camera.
>
> I'm enclosing a detailed description of the problem, in order to
> allow the interested parties to be at the same page.
>
> We need to work towards an agenda for the meeting.
>
> From my side, I think we should have at least the following topics at
> the agenda:
>
> - a quick review about what's currently at libv4l2;
> - a presentation about PipeWire solution;
> - a discussion with the requirements for the new solution;
> - a discussion about how we'll address - who will do what.
>
> Comments? Suggestions?
>
> Are there anyone else planning to either be there physically or via
> Google Hangouts?
>
> Tomaz,
>
> Do you have any limit about the number of people that could join us
> via Google Hangouts?
>
>
> Regards,
> Mauro
>
> ---
>
> 1. Introduction
> ===============
>
> 1.1 V4L2 Kernel aspects
> -----------------------
>
> The media subsystem supports two types of devices:
>
> - "traditional" media hardware, supported via V4L2 API. On such hardware,
>   opening a single device node (usually /dev/video0) is enough to control
>   the entire device. We call it as devnode-based devices.
>   An application sometimes may need to use multiple video nodes with
>   devnode-based drivers to capture multiple streams in parallel
>   (when the hardware allows it of course). That's quite common for
>   Analog TV devices, where both /dev/video0 and /dev/vbi0 are opened
>   at the same time.
>
> - Media-controller based devices. On those devices, there are typically
>   several /dev/video? nodes and several /dev/v4l2-subdev? nodes, plus
>   a media controller device node (usually /dev/media0).
>   We call it as mc-based devices. Controlling the hardware require
>   opening the media device (/dev/media0), setup the pipeline and adjust
>   the sub-devices via /dev/v4l2-subdev?. Only streaming is controlled
>   by /dev/video?.
>
> In other words, both configuration and streaming go through the video
> device node on devnode-based drivers, while video device nodes are used
> used for streaming on mc-based drivers.
>
> With devnode-based drivers, "standard" media applications, including open
> source ones (Camorama, Cheese, Xawtv, Firefox, Chromium, ...) and closed
> source ones (Skype, Chrome, ...) support devnode-based devices[1]. Also,
> when just one media device is connected, the streaming/control device
> is typically /dev/video0.
>
> [1] It should be noticed that closed-source applications tend to have
> various bugs that prevent them from working properly on many devnode-based
> devices. Due to that, some additional blocks were requred at libv4l to
> support some of them. Skype is a good example, as we had to include a
> software scaler in libv4l to make it happy. So in practice not everything
> works smoothly with closed-source applications with devnode-based drivers.
> A few such adjustments were also made on some drivers and/or libv4l, in
> order to fulfill some open-source app requirements.
>
> Support for mc-based devices currently require an specialized application
> in order to prepare the device for its usage (setup pipelines, adjust
> hardware controls, etc). Once pipeline is set, the streaming goes via
> /dev/video?, although usually some /dev/v4l2-subdev? devnodes should also
> be opened, in order to implement algorithms designed to make video quality
> reasonable. On such devices, it is not uncommon that the device used by the
> application to be a random number (on OMAP3 driver, typically, is either
> /dev/video4 or /dev/video6).
>
> One example of such hardware is at the OMAP3-based hardware:
>
>         http://www.infradead.org/~mchehab/mc-next-gen/omap3-igepv2-with-tvp5150.png
>
> On the picture, there's a graph with the hardware blocks in blue/dark/blue
> and the corresponding devnode interfaces in yellow.
>
> The mc-based approach was taken when support for Nokia N9/N900 cameras
> was added (with has OMAP3 SoC). It is required because the camera hardware
> on SoC comes with a media processor (ISP), with does a lot more than just
> capturing, allowing complex algorithms to enhance image quality in runtime.
> Those algorithms are known as 3A - an acronym for 3 other acronyms:
>
>         - AE (Auto Exposure);
>         - AF (Auto Focus);
>         - AWB (Auto White Balance).
>
> The main reason that drove the MC design is that the 3A algorithms (that is
> the 3A control loop, and sometimes part of the image processing itself) often
> need to run, at least partially, on the CPU. As a kernel-space implementation
> wasn't possible, we needed a lower-level UAPI.
>
> Setting a camera with such ISPs are harder because the pipelines to be
> set actually depends the requirements for those 3A algorithms to run.
> Also, usually, the 3A algorithms use some chipset-specific userspace API,
> that exports some image properties, calculated by the ISP, to speed up
> the convergence of those algorithms.
>
> Btw, usually, the 3A algorithms are IP-protected, provided by vendors
> as binary only blobs, although there are a few OSS implementations.
>
> Part of the problem is that, so far, there isn't a proper userspace API
> to implement 3A libraries. Once we have an userspace camera stack, we
> hope that we'll gradually increase the number and quality of open-source
> 3A stacks.
>
>
> 1.2 V4L2 userspace aspects
> --------------------------
>
> Back when USB cameras were introduced, the hardware were really simple:
> they had a CMOS or CCD camera sensor and a chip that bridges the data
> though USB. Camera sensors typically provide data using a bayer
> format, but they usually have their own proprietary ways to pack the data
> at the USB bridges, in order to reduce the bandwidth (the first
> implementations were using USB version 1.1).
>
> So, V4L2 has a myriad of different formats, in order to match each
> camera sensor format. At the end of the day, applications were
> able to use only a subset of the available hardware, since they need
> to come with format converters for all formats the developer uses
> (usually a very small subset of the available ones).
>
> That's said, newer cameras are converged towards a small set of
> standard formats - except for secondary data streams (like depth
> maps). Yet, industrial cameras, and newer technologies, like
> 3D, light-field, etc may still bring newer formats.
>
> To end with this mess, an userspace library was written, called libv4l.
> It supports all those proprietary formats. So, applications can use
> a RGB or YUV format, without needing to concern about conversions.
>
> The way it works is by adding wrappers to system calls: open, close,
> ioctl, mmap, mmunmap. So, a conversion to use it is really simple:
> at the source code of the apps, all it was needed is to prepend the
> existing calls with "v4l2_", e. g. v4l2_open, v4l2_close, etc.
>
> All open source apps we know now supports libv4l. On a few (like
> gstreamer), support for it is optional.
>
> It should be noted that libv4l also handles scaling and rough auto-gain
> and auto-white balance that are required by  some cameras to achieve
> a usable image.
>
> In order to support closed source, another wrapper was added, allowing
> to call any closed source application to use it, by using LD_PRELOAD.
> For example, using skype with it is as simple as calling it with:
>
>         $ LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so /usr/bin/skypeforlinux
>
>
> 2. Current problems
> ===================
>
> 2.1 Libv4l can slow image handling
> ----------------------------------
>
> Nowadays, almost all new "simple" cameras are connected via USB using
> the UVC class (USB Video Class). UVC standardized the allowed formats,
> and most apps just implement them. The UVC hardware is more complex,
> having format converters inside it. So, for most usages, format
> conversion isn't needed anymore.
>
> The need of doing format conversion in software makes libv4l slow,
> requiring lots of CPU usage in order to convert a 4K or 8K format,
> being even worse with 3D cameras.
>
> Also, due to the need of supporting LD_PRELOAD, zero-buffer copy via
> DMA_BUFFER currently doesn't work with libv4l.
>
> Right now, gstreamer defaults to not enable libv4l2, due to several
> reasons:
>   - Crash when CREATE_BUFS is being used;
>   - Crash in the jpeg decoder (when frames are corrupted);
>   - App exporting DMABuf need to be aware of emulation, otherwise the
>     DMABuf exported are in the orignal format;
>   - RW emulation only initialize the queue on first read (causing
>     userspace poll() to fail);
>   - Signature of v4l2_mmap does not match mmap() (minor);
>   - The colorimetry does not seem emulated when conversion;
>   - Sub-optimal locking (at least deadlocks were fixed).
>
> Most of the above are due to new features added to Kernel uAPI, but
> not added to libv4l2.
>
> These issues are already worked around in GStreamer, but with the lost
> of features of course. There is other cases were something worked
> without libv4l2, but didn't work with libv4l2, but Gstreamer developers
> haven't tracked down the cause yet. Since 1.14, libv4l2 can be enabled
> at run-time using env GST_V4L2_USE_LIBV4L2=1.
>
> 2.2 Modern hardware is starting to come with "complex" camera ISP
> -----------------------------------------------------------------
>
> While mc-based devices were limited to SoC, it was easy to
> "delegate" the task of talking with the hardware to the
> embedded hardware designers.
>
> However, this is changing. Dell Latitude 5285 laptop is a standard
> PC with an i3-core, i5-core or i7-core CPU, with comes with the
> Intel IMU3 ISP hardware[2].
>
> [2] https://www.spinics.net/lists/linux-usb/msg167478.html
>
> There, instead of an USB camera, the hardware is equipped with a
> MC-based ISP, connected to its camera. Currently, despite having
> a Kernel driver for it, the camera doesn't work with any
> userspace application.
>
> I'm also aware of other projects that are considering the usage of
> mc-based devices for non-dedicated hardware.
>
> 3. How to solve it?
> ===================
>
> That's the main focus of the meeting :-)
>
> From a previous discussion I had with media sub-maintainers, there are
> at least two actions that seem required. I'm listing them below as
> an starting point for the discussions, but we can eventually come up
> with some different approach after the meeting.
>
> 3.1 Library support for mc-based hardware
> =========================================
>
> In order to support those hardware, we'll need to do some redesign,
> mainly at userspace, via a library[3] that will replace/compliment
> libv4l2.
>
> The idea is to work on a library API that will allow to split the
> format conversion on a separate part of it, add support
> for DMA Buffer and come up with a way for the library to work
> transparently with both devnode-based and mc-based hardware.
>
> The library should be capable of doing dynamic pipeline setups,
> including the functionality of allow tranfering buffers between
> different media devices, as, on several cases, a a camera input
> device node should be connected to a m2m device that would be
> doing some image processing, and providing output data on a
> standard video output format.
>
> One possibility is to redesign libv4l2. It should be noticed that
> Videolan people is working on a daemon that could also provide
> another way to implement it[4].
>
> Once we get a proper camera stack, we need to support traditional
> applications that use libv4l2 and/or the new library directly. To
> that end we  will need to expose the libv4l2 API on top of the
> camera stack (through libv4l2 if it makes sense, or through a separate
> implementation) and implement a transparent LD_PRELOAD-based library.
>
> That envolves adding capacity to the library to setup hardware pipelines
> and to propagate controls among their sub-devices. Eventually, part
> of it will be done in Kernel.
>
> That should give performance increase at the library and would allow
> gstreamer to use it by default, without compromising performance.
>
> [3] I don't discard that some Kernel changes could also be part of the
> solution, like, for example, doing control propagation along the pipeline
> on simple use case scenarios.
>
> [4] With the venue of sandboxed application, there is a need to control
> access to cameras through a daemon. The same daemon is also used to
> control access to screen capture on Wayland (instead of letting any
> random application capture your screen, like on X11). The effort is
> lead by the desktop team at RedHat. PipeWire already have V4L2 native
> support and is integrated in GStreamer already in a way that it can
> totally replace the V4L2 capture component there. PipeWire is plugin based,
> so more type of camera support can be added. Remote daemon can also provide
> streams, as this is the case for compositors and screen casting. An extra
> benefit is that you can have multiple application reading frames from the
> same camera. It also allow sandboxed  application (the do not have access
> to /dev) to use the cameras. In this context, proprietary or HW specific
> algorithm could be implemented in userspace as PipeWire plugins, and then
> application will automatically be enable to enumerate and use these.
>
> 3.2 libv4l2 support for 3A algorithms
> =====================================
>
> The 3A algorithm handing is highly dependent on the hardware. The
> idea here is to allow libv4l to have a set of 3A algorithms that
> will be specific to certain mc-based hardware.
>
> One requirement, if we want vendor stacks to use our solution, is that
> it should allow allow external closed-source algorithms to run as well.
>
> The 3A library API must be standardized, to allow the closed-source
> vendor implementation to be replaced by an open-source implementation
> should someone have the time and energy (and qualifications) to write
> one.
>
> Sandboxed execution of the 3A library must be possible as closed-source
> can't always be blindly trusted. This includes the ability to wrap the
> library in a daemon should the platform's multimedia stack wishes
> and to avoid any direct access to the kernel devices by the 3A library
> itself (all accesses should be marshaled by the camera stack).
>
> Please note that this daemon is *not* a camera daemon that would
> communicates with the V4L2 driver through a custom back channel.
>
> The decision to run the 3A library in a sandboxed process or to call
> it directly from the camera stack should be left to the camera stack
> and to the platform integrator, and should not be visible by the 3A
> library.
>
> The 3A library must be usable on major Linux-based camera stacks (the
> Android and Chrome OS camera HALs are certainly important targets,
> more can be added) unmodified, which will allow usage of the vendor
> binary provided for Chrome OS or Android on regular Linux systems.
>
> It would make sense to design a modular camera stack, and try to make
> most components as platform-independent as possible. This should include:
>
> - the kernel drivers (V4L2-compliant and usable without any closed-source
>   userspace component);
> - the 3A library
> - any other component that could be shared (for instance a possible
>   request API library).
>
> The rest of the code will mostly be glue around those components to
> integrate them in a particular camera stack, and should be as
> platform-agnostic as possible.
>
> In the case of the Android camera HAL, ideally it would be a glue that
> could be used with different camera vendors (probably with some kind of
> vendor-specific configuration, or possibly with a separate vendor-specific
> component to handle pipeline configuration).
>
> 4 Complex camera workshop
> =========================
>
> The workshop will be happening in Tokyo, Japan, at Jun, 19, at the
> google offices. The location is:
>
> 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
>
> 4.1 Physical Attendees
> ======================
>
> Tomasz Figa <tfiga@google.com>
> Mauro Carvalho Chehab <Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> Kieran Bingham <kieran.bingham@ideasonboard.com>
> Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Niklas Söderlund <niklas.soderlund@ragnatech.se>
> Zheng, Jian Xu Zheng <jian.xu.zheng@intel.com>
>
> Anywone else?
>
> 4.2. Attendees Via Google Hangouts
> ==================================
>
> Hans Verkuil <hverkuil@xs4all.nl> - Via Google Hangouts - maybe only on afternoon
> Javier Martinez Canillas <javier@dowhile0.org> - Via Google Hangouts - only on reasonable TZ-compatible-hours
> Ricky - Google camera team in Taipei - Via Google Hangouts
>
> Anywone else?

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Laurent Pinchart]

Hello,

On Monday, 18 June 2018 11:42:37 EEST Paul Elder wrote:
> On June 4, 2018 10:33:03 PM GMT+09:00, Mauro Carvalho Chehab  wrote:
> > Hi all,
> >
> > I consolidated hopefully all comments I receive on the past announcement
> > with regards to the complex camera workshop we're planning to happen in
> > Tokyo, just before the Open Source Summit in Japan.
> >
> > The main focus of the workshop is to allow supporting devices with
> > MC-based hardware connected to a camera.
> >
> > I'm enclosing a detailed description of the problem, in order to
> > allow the interested parties to be at the same page.
> >
> > We need to work towards an agenda for the meeting.
> >
> > From my side, I think we should have at least the following topics at
> > the agenda:
> >
> > - a quick review about what's currently at libv4l2;
> > - a presentation about PipeWire solution;
> > - a discussion with the requirements for the new solution;
> > - a discussion about how we'll address - who will do what.
> >
> > Comments? Suggestions?
> >
> > Are there anyone else planning to either be there physically or via
> > Google Hangouts?
> 
> My name is Paul Elder. I am a university student studying computer science,
> and I am interested in complex camera support in Linux.
> 
> If it's not too late, could I join this meeting as well please, as I am in
> Tokyo?

For the record, Paul is working with Kieran and me on V4L2 (and UVC in 
particular).

-- 
Regards,

Laurent Pinchart

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Tomasz Figa]

Hi Paul,

On Mon, Jun 18, 2018 at 5:42 PM Paul Elder <paul.elder@ideasonboard.com> wrote:
>
>
>
> Hello all,
>
> On June 4, 2018 10:33:03 PM GMT+09:00, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> >Hi all,
> >
> >I consolidated hopefully all comments I receive on the past
> >announcement
> >with regards to the complex camera workshop we're planning to happen in
> >Tokyo, just before the Open Source Summit in Japan.
> >
> >The main focus of the workshop is to allow supporting devices with
> >MC-based
> >hardware connected to a camera.
> >
> >I'm enclosing a detailed description of the problem, in order to
> >allow the interested parties to be at the same page.
> >
> >We need to work towards an agenda for the meeting.
> >
> >From my side, I think we should have at least the following topics at
> >the agenda:
> >
> >- a quick review about what's currently at libv4l2;
> >- a presentation about PipeWire solution;
> >- a discussion with the requirements for the new solution;
> >- a discussion about how we'll address - who will do what.
> >
> >Comments? Suggestions?
> >
> >Are there anyone else planning to either be there physically or via
> >Google Hangouts?
> >
> My name is Paul Elder. I am a university student studying computer science, and I am interested in complex camera support in Linux.
>
> If it's not too late, could I join this meeting as well please, as I am in Tokyo?

Done. You should have received 3 further emails with necessary invitations.

Best regards,
Tomasz

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Paul Elder]

Hi Tomasz,

On June 18, 2018 6:00:47 PM GMT+09:00, Tomasz Figa <tfiga@google.com> wrote:
>Hi Paul,
>
>On Mon, Jun 18, 2018 at 5:42 PM Paul Elder
><paul.elder@ideasonboard.com> wrote:
>>
>>
>>
>> Hello all,
>>
>> On June 4, 2018 10:33:03 PM GMT+09:00, Mauro Carvalho Chehab
><mchehab+samsung@kernel.org> wrote:
>> >Hi all,
>> >
>> >I consolidated hopefully all comments I receive on the past
>> >announcement
>> >with regards to the complex camera workshop we're planning to happen
>in
>> >Tokyo, just before the Open Source Summit in Japan.
>> >
>> >The main focus of the workshop is to allow supporting devices with
>> >MC-based
>> >hardware connected to a camera.
>> >
>> >I'm enclosing a detailed description of the problem, in order to
>> >allow the interested parties to be at the same page.
>> >
>> >We need to work towards an agenda for the meeting.
>> >
>> >From my side, I think we should have at least the following topics
>at
>> >the agenda:
>> >
>> >- a quick review about what's currently at libv4l2;
>> >- a presentation about PipeWire solution;
>> >- a discussion with the requirements for the new solution;
>> >- a discussion about how we'll address - who will do what.
>> >
>> >Comments? Suggestions?
>> >
>> >Are there anyone else planning to either be there physically or via
>> >Google Hangouts?
>> >
>> My name is Paul Elder. I am a university student studying computer
>science, and I am interested in complex camera support in Linux.
>>
>> If it's not too late, could I join this meeting as well please, as I
>am in Tokyo?
>
>Done. You should have received 3 further emails with necessary
>invitations.

Thank you.
I have only received two: 来訪者事前登録のご案内, and the Google invitation.

Paul

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.