Re: [PATCH 00/20] Add multiplexed media pads to support CSI-2 virtual channels

[Maxime Ripard <maxime.ripard@free-electrons.com>]

[-- Attachment #1: Type: text/plain, Size: 4769 bytes --]

Hi Niklas,

On Fri, Aug 11, 2017 at 11:56:43AM +0200, Niklas Söderlund wrote:
> This series is a RFC for how I think one could add CSI-2 virtual channel 
> support to the V4L2 framework. The problem is that there is no way to in 
> the media framework describe and control links between subdevices which 
> carry more then one video stream, for example a CSI-2 bus which can have 
> 4 virtual channels carrying different video streams.
> 
> This series adds a new pad flag which would indicate that a pad carries 
> multiplexed streams, adds a new s_stream() operation to the pad 
> operations structure which takes a new argument 'stream'. This new 
> s_stream() operation then is both pad and stream aware. It also extends 
> struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how 
> a CSI-2 link multiplexes virtual channels. I also include one 
> implementation based on Renesas R-Car which makes use of these patches 
> as I think they help with understanding but they have no impact on the 
> RFC feature itself.
> 
> The idea is that on both sides of the multiplexed media link there are 
> one multiplexer subdevice and one demultiplexer subdevice. These two 
> subdevices can't do any format conversions, there sole purpose is to 
> (de)multiplex the CSI-2 link. If there is hardware which can do both 
> CSI-2 multiplexing and format conversions they can be modeled as two 
> subdevices from the same device driver and using the still pending 
> incremental async mechanism to connect the external pads. The reason 
> there is no format conversion is important as the multiplexed pads can't 
> have a format in the current V4L2 model, get/set_fmt are not aware of 
> streams.
> 
>         +------------------+              +------------------+
>      +-------+  subdev 1   |              |  subdev 2   +-------+
>   +--+ Pad 1 |             |              |             | Pad 3 +---+
>      +--+----+   +---------+---+      +---+---------+   +----+--+
>         |        | Muxed pad A +------+ Muxed pad B |        |
>      +--+----+   +---------+---+      +---+---------+   +----+--+
>   +--+ Pad 2 |             |              |             | Pad 4 +---+
>      +-------+             |              |             +-------+
>         +------------------+              +------------------+
> 
> In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, 
> and the video data for both of them travels the link between pad A and 
> B. One shortcoming of this RFC is that there currently are no way to 
> express to user-space which pad is routed to which stream of the 
> multiplexed link. But inside the kernel this is known and format 
> validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 
> to Pad 4 by the V4L2 framework. But it would be nice for the user to 
> also be able to get this information while setting up the MC graph in 
> user-space.

Thanks for your patches.

I guess I already have a different use-case for this, one you might
not have envisionned (on top of the Cadence CSI2-RX driver I've been
posting and that would need it at some point).

I have a CSI2-TX controller that I'm currently writing a driver
for. That controller takes 4 video streams in input, and aggregates
them on a single CSI-2 link. Those video streams use some kind of
parallel interfaces. So far, so good.

However, those parallel interfaces come with additional signals that
set the virtual channel and data types of the associated video
signal. Those signals are under control of the upstream video output
device, and can change at runtime.

The virtual channel are direcly mapped to the CSI-2 Virtual Channels,
so that part is completely transparent to the transmitter. However,
the video input datatype is a 0-7 value. Each data type value has a
separate set of registers where you need to set up the width and
height of the video data, and the corresponding CSI-2 Data Type.

You can use this to do some interleaving of either the virtual
channels or the data types, or both.

To make things (slightly) more complicated, the FIFO registers are
per-video input. Which means that while most of the (input)
configuration will be done per-datatype, we still have to know which
input stream is generating it, for example to optimise the FIFO fill
levels to the resolution...

Since the stream is multiplexed both using the virtual channels and
the data-types, I'm not sure representing it using only muxed pads
like you did would work.

And I don't really know what a good stop gap measure would be either.

Thanks!
Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]