Re: [PATCH v7 22/27] v4l: subdev: add stream based configuration

[Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>]

Hi,

On 06/06/2021 02:42, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Mon, May 24, 2021 at 01:44:03PM +0300, Tomi Valkeinen wrote:
>> Add support to manage configurations (format, crop, compose) per stream,
>> instead of per pad. This is accomplished with data structures that hold
>> an array of all subdev's stream configurations.
>>
>> The number of streams can vary at runtime based on routing. Every time
>> the routing is changed, the stream configurations need to be
>> re-initialized.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 62 +++++++++++++++++++++++++++
>>   include/media/v4l2-subdev.h           | 55 ++++++++++++++++++++++++
>>   2 files changed, 117 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index ef18682dbc6f..da6ea9b14631 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1112,3 +1112,65 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>> +
>> +int v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
>> +			     const struct v4l2_subdev_krouting *routing)
>> +{
>> +	u32 num_configs = 0;
>> +	unsigned int i;
>> +	u32 format_idx = 0;
>> +
>> +	v4l2_uninit_stream_configs(stream_configs);
>> +
>> +	/* Count number of formats needed */
>> +	for (i = 0; i < routing->num_routes; ++i) {
>> +		struct v4l2_subdev_route *route = &routing->routes[i];
>> +
>> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
>> +			continue;
>> +
>> +		/* Each route needs a format on both ends of the route */
>> +		num_configs += 2;
>> +	}
>> +
>> +	if (num_configs) {
>> +		stream_configs->configs =
>> +			kvcalloc(num_configs, sizeof(*stream_configs->configs),
>> +				 GFP_KERNEL);
>> +
>> +		if (!stream_configs->configs)
>> +			return -ENOMEM;
>> +
>> +		stream_configs->num_configs = num_configs;
>> +	}
>> +
>> +	/* Fill in the 'pad' and stream' value for each item in the array from the routing table */
>> +	for (i = 0; i < routing->num_routes; ++i) {
>> +		struct v4l2_subdev_route *route = &routing->routes[i];
>> +		u32 idx;
>> +
>> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
>> +			continue;
>> +
>> +		idx = format_idx++;
>> +
>> +		stream_configs->configs[idx].pad = route->sink_pad;
>> +		stream_configs->configs[idx].stream = route->sink_stream;
>> +
>> +		idx = format_idx++;
>> +
>> +		stream_configs->configs[idx].pad = route->source_pad;
>> +		stream_configs->configs[idx].stream = route->source_stream;
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_init_stream_configs);
>> +
>> +void v4l2_uninit_stream_configs(struct v4l2_subdev_stream_configs *stream_configs)
>> +{
>> +	kvfree(stream_configs->configs);
>> +	stream_configs->configs = NULL;
>> +	stream_configs->num_configs = 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_uninit_stream_configs);
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index 87f6bb78bbe9..39c6b811463a 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -661,6 +661,37 @@ struct v4l2_subdev_pad_config {
>>   	struct v4l2_rect try_compose;
>>   };
>>   
>> +/**
>> + * struct v4l2_subdev_stream_config - Used for storing stream configuration.
>> + *
>> + * @pad: pad number
>> + * @stream: stream number
>> + * @fmt: &struct v4l2_mbus_framefmt
>> + * @crop: &struct v4l2_rect to be used for crop
>> + * @compose: &struct v4l2_rect to be used for compose
>> + *
>> + * This structure stores configuration for a stream.
>> + */
>> +struct v4l2_subdev_stream_config {
>> +	u32 pad;
>> +	u32 stream;
>> +
>> +	struct v4l2_mbus_framefmt fmt;
>> +	struct v4l2_rect crop;
>> +	struct v4l2_rect compose;
>> +};
>> +
>> +/**
>> + * struct v4l2_subdev_stream_configs - A collection of stream configs.
>> + *
>> + * @num_configs: number of entries in @config.
>> + * @config: an array of &struct v4l2_subdev_stream_configs.
>> + */
>> +struct v4l2_subdev_stream_configs {
>> +	u32 num_configs;
>> +	struct v4l2_subdev_stream_config *configs;
>> +};
> 
> Honestly, this feels over-complicated and under-specified. What happens

Under-specified I agree, but why is it over-complicated? It's just a 
dynamic map of (pad+stream) -> stream-config. Or do you mean the whole 
approach to multiplexed streams?

> to formats that have been previously configured when routing is changed

Changing routing is a drastic operation, so all formats are reset to 
default.

> ? What happens when changing routing while streaming is in progress ?

It's not allowed (probably my code doesn't check this at the moment).

> All these use cases need to be clearly specified.
> 
> If you want to go in this direction, this needs to be integrated with
> v4l2_subdev_state (which I think you do in a later patch), but also
> handled completely in the core. There's no way drivers will get this
> right, I don't want them to see v4l2_subdev_stream_configs. On the
> upside, embedding v4l2_subdev_state in subdevs would help, but I'm not
> sure if it will be enough.
> 
> Making subdev pad operations handle streams explicitly opens the door to
> thousands of questions, and each of them needs to be answered. I fear
> that's a daunting task.

Yes, that's multiplexed streams for you =).

The previous approach I tried, where pads were either single-stream and 
configurable or multistream and non-configurable, also raised lots of 
questions and the implementation complexity was getting quite horrible. 
E.g. a simple sensor driver that also has metadata had to be split into 
three subdevs, and the need to walk the media graph back and forth lots 
of times was very ugly.

My opinion is that this version is much better. It's quite 
understandable and easy to use from the userspace (taking into account 
that the issue of multiplexed streams is complex, and that the APIs are 
perhaps not yet optimal), and the code on the kernel side resembles the 
non-multiplexed operations, and is more manageable in the drivers than 
with the previous version.

Now that the subdev state has been merged, I can work towards handling 
many of these operations in the v4l2 core. But that's essentially "just" 
kernel cleanup, I don't think that affects much whether this multiplexed 
streams approach in general is ok or not.

  Tomi