Question regarding optimizing pipeline in Vimc

Question regarding optimizing pipeline in Vimc

[Helen Koike]

Hello,

One of the discussions we had when developing Vimc, was regarding
optimizing image generation.
The ideia was to generate the images directly in the capture instead
of propagating through the pipeline (to make things faster).
But my question is: if this optimization is on, and if there is a
greyscaler filter in the middle of the pipeline, do you expect to see
a grey image with this optimization? Or if we just generate a dummy
image (with the right size format) at the end of the pipeline, would
it be ok? (I am asking because it doesn't sound that simple to
propagate the image transformation made by each entity in the pipe)
Or do you have any other thing in mind?

Thanks
Helen

Re: Question regarding optimizing pipeline in Vimc

[Hans Verkuil]

On 08/22/2018 05:35 AM, Helen Koike wrote:
> Hello,
> 
> One of the discussions we had when developing Vimc, was regarding
> optimizing image generation.
> The ideia was to generate the images directly in the capture instead
> of propagating through the pipeline (to make things faster).
> But my question is: if this optimization is on, and if there is a
> greyscaler filter in the middle of the pipeline, do you expect to see
> a grey image with this optimization?

Yes.

> Or if we just generate a dummy
> image (with the right size format) at the end of the pipeline, would
> it be ok? (I am asking because it doesn't sound that simple to
> propagate the image transformation made by each entity in the pipe)

No, that would not be OK.

My basic idea was that you use a TPG state structure that contains the
desired output: the sensor starts with e.g. 720p using some bayer pixelformat,
the debayer module replaces the pixelformat with e.g. PIX_FMT_RGB32, a
grayscale filter replaces it with PI_FMT_GREY, and that's what the TPG for the
video device eventually will use to generate the video.

This assumes of course that all the vimc blocks only do operations that can
be handled by the TPG. Depending on what the blocks will do the TPG might need
to be extended if a feature is missing.

Regards,

	Hans

> Or do you have any other thing in mind?
> 
> Thanks
> Helen
> 

Re: Question regarding optimizing pipeline in Vimc

[Helen Koike]

On 8/22/18 3:49 AM, Hans Verkuil wrote:
> On 08/22/2018 05:35 AM, Helen Koike wrote:
>> Hello,
>>
>> One of the discussions we had when developing Vimc, was regarding
>> optimizing image generation.
>> The ideia was to generate the images directly in the capture instead
>> of propagating through the pipeline (to make things faster).
>> But my question is: if this optimization is on, and if there is a
>> greyscaler filter in the middle of the pipeline, do you expect to see
>> a grey image with this optimization?
> 
> Yes.
> 
>> Or if we just generate a dummy
>> image (with the right size format) at the end of the pipeline, would
>> it be ok? (I am asking because it doesn't sound that simple to
>> propagate the image transformation made by each entity in the pipe)
> 
> No, that would not be OK.
> 
> My basic idea was that you use a TPG state structure that contains the
> desired output: the sensor starts with e.g. 720p using some bayer pixelformat,
> the debayer module replaces the pixelformat with e.g. PIX_FMT_RGB32, a
> grayscale filter replaces it with PI_FMT_GREY, and that's what the TPG for the
> video device eventually will use to generate the video.
> 
> This assumes of course that all the vimc blocks only do operations that can
> be handled by the TPG. Depending on what the blocks will do the TPG might need
> to be extended if a feature is missing.
> 
> Regards,
> 
> 	Hans
> 
>> Or do you have any other thing in mind?
>>
>> Thanks
>> Helen
>>
> 

Thanks Hans,

We'll be working on that soon.

Helen

Re: Question regarding optimizing pipeline in Vimc

[Lucas Magalhães]

On Wed, Aug 22, 2018 at 3:49 AM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> My basic idea was that you use a TPG state structure that contains the
> desired output: the sensor starts with e.g. 720p using some bayer pixelformat,
> the debayer module replaces the pixelformat with e.g. PIX_FMT_RGB32, a
> grayscale filter replaces it with PI_FMT_GREY, and that's what the TPG for the
> video device eventually will use to generate the video.
>
> This assumes of course that all the vimc blocks only do operations that can
> be handled by the TPG. Depending on what the blocks will do the TPG might need
> to be extended if a feature is missing.
>
Hi Hans,

I start to work on this task but I have another question. I understand that the
final image should have the correct format as if the frame was passing through
the whole topology. But the operations itself doesn't needed to be done on each
entity. For example, a scaled image will have deformations that will not be
present if it is generated on the end of the pipeline with the final size.
You just need the format, size and properties to be correct, do I got it right?

Thanks Lucas.

Re: Question regarding optimizing pipeline in Vimc

[Hans Verkuil]

On 08/31/2018 01:37 AM, Lucas Magalhães wrote:
> On Wed, Aug 22, 2018 at 3:49 AM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>
>> My basic idea was that you use a TPG state structure that contains the
>> desired output: the sensor starts with e.g. 720p using some bayer pixelformat,
>> the debayer module replaces the pixelformat with e.g. PIX_FMT_RGB32, a
>> grayscale filter replaces it with PI_FMT_GREY, and that's what the TPG for the
>> video device eventually will use to generate the video.
>>
>> This assumes of course that all the vimc blocks only do operations that can
>> be handled by the TPG. Depending on what the blocks will do the TPG might need
>> to be extended if a feature is missing.
>>
> Hi Hans,
> 
> I start to work on this task but I have another question. I understand that the
> final image should have the correct format as if the frame was passing through
> the whole topology. But the operations itself doesn't needed to be done on each
> entity. For example, a scaled image will have deformations that will not be
> present if it is generated on the end of the pipeline with the final size.
> You just need the format, size and properties to be correct, do I got it right?

Yes. Although this example is unfortunate since the TPG can actually scale:
with tpg_reset_source you define the width/height of the 'source', and with
tpg_s_crop_compose you can define the crop and compose rectangles, which in
turn translates to scaling. The TPG has a poor man's scaler, so if you scale
up by a factor of 2, you will in fact see those deformations.

But if you have a complex pipeline with e.g. two scalers with additional
processing in between, then that cannot be modeled accurately with the TPG.
So be it. There is a balance between accuracy and performance, and I think
this is a decent compromise.

Regards,

	Hans