Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset

[Bingbu Cao <bingbu.cao@linux.intel.com>]

On 3/23/19 9:02 PM, Jacopo Mondi wrote:
> Hello,
>    sorry for resurrecting the thread.
> 
> The development of libcamera has IPU3 devices as first target, and
> we're now at the point where some clarifications are required.
> I'll re-use this email, as some of points were already stated here
> and, as they've become pressing for libcamera development, I would like
> to have them clarified here on the list.
> 
> On Wed, Jan 02, 2019 at 10:38:33AM +0800, Bingbu Cao wrote:
>>
>> On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
>>> Hello Bingbu,
>>>
> 
> [snip]
> 
>>>>>>>>> 2. ImgU pipeline needs to be configured for image processing as below.
>>>>>>>>>
>>>>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
>>>>>>>>> have the processed image output to the DDR memory.
>>>>>>>>>
>>>>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
>>>>>>>>> Geometric Distortion Correction (GDC) -> DDR
>>>>>>>>>
>>>>>>>>> The ImgU V4L2 subdev has to be configured with the supported
>>>>>>>>> resolutions in all the above HW blocks, for a given input resolution.
>>>>>>>>>
>>>>>>>>> For a given supported resolution for an input frame, the Input Feeder,
>>>>>>>>> Bayer Down Scaling and GDC blocks should be configured with the
>>>>>>>>> supported resolutions. This information can be obtained by looking at
>>>>>>>>> the following IPU3 ISP configuration table for ov5670 sensor.
>>>>>>>>>
>>>>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
>>>>>>>>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
>>>>>>>>> files/gcss/graph_settings_ov5670.xml
>>>>>>>>>
>>>>>>>>> For the ov5670 example, for an input frame with a resolution of
>>>>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
>>>>>>>>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
>>>>>>>>> 2560x1920 respectively.
>>>>>>>> How is the GDC output resolution computed from the input resolution ?
>>>>>>>> Does the GDC always consume 32 columns and 22 lines ?
>>>>>> All the intermediate resolutions in the pipeline are determined by the
>>>>>> actual use case, in other word determined by the IMGU input
>>>>>> resolution(sensor output) and the final output and viewfinder resolution.
>>>>>> BDS mainly do Bayer downscaling, it has limitation that the downscaling
>>>>>> factor must be a value a integer multiple of 1/32.
>>>>>> GDC output depends on the input and width should be x8 and height x4
>>>>>> alignment.
>>>>> Thank you for the information. This will need to be captured in the
>>>>> documentation, along with information related to how each block in the
>>>>> hardware pipeline interacts with the image size. It should be possible for
>>>>> a developer to compute the output and viewfinder resolutions based on the
>>>>> parameters of the image processing algorithms just with the information
>>>>> contained in the driver documentation.
> 
> In libcamera development we're now at the point of having to calculate
> the sizes to apply to all intermediate pipeline stages based on the
> following informations:
> 
> 1) Main output resolution
> 2) Secondary output resolution (optional)
> 3) Image sensor's available resolutions
> 
> Right now that informations are captured in the xml file you linked
> here above, but we need a programmatic way to do the calculation,
> without going through an XML file, that refers to two specific sensors
> only.
> 
> As Laurent said here, this should come as part of the documentation
> for driver users and would unblock libcamera IPU3 support
> development.
> 
> Could you provide documentation on how to calculate each
> intermediate step resolutions?
All the intermediate step resolutions are generated by the specific tool
with sensor input and outputs resolutions.

The tool try to keep maximum fov and has the knowledge of all the
limitations of each intermediate hardware components(mainly BDS and GDC).

Currently, there is not a very simple calculation to get the
intermediate resolutions.
Let's take some effort to try find a programmatic way to do calculation
instead of the tool.

> 
>>>>>
> 
> [snip]
> 
>>>>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
>>>>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
>>>>>>>>> target, using the BDS height and width.
>>>>>>>>>
>>>>>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
>>>>>>>>> ImgU V4L2 subdev for processing.
>>>>>>>> Do I need to capture from both the output and viewfinder nodes ? How
>>>>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
>>>>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in that
>>>>>>>> picture ?
>>>>>> The output capture should be set, the viewfinder can be disabled.
>>>>>> The IF and BDS are seen as crop and compose of the imgu input video
>>>>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
>>>>>> pads.
> 
> This is another point that we would like to have clarified:
> 1) which outputs are mandatory and which one are not
> 2) which operations are mandatory on un-used outputs
> 3) does the 'ipu_pipe_mode' control impact this
> 
> As you mentioned here, "output" seems to be mandatory, while
> "viewfinder" and "stat" are optional. We have tried using the "output"
> video node only but the system hangs to an un-recoverable state.
Yes, main output is mandatory, 'vf' and 'stat' are optional.

> 
> What I have noticed is instead that the viewfinder and stat nodes
> needs to be:
> 1) Linked to the respective "ImgU" subdevice pads
> 2) Format configured
> 3) Memory reserved
> 4) video device nodes started
> 
> It it not required to queue/dequeue buffers from viewfinder and stat,
> but steps 1-4 have to be performed.
> 
> Can you confirm this is intended?

viewfinder and stats are enabled when the link for respective subdev
pads enabled, and then driver can use these input conditions to find the
binary to run.

> Could you please list all the steps that have to be applied to the
> ImgU's capture video nodes, and which ones are mandatory and which ones
> are optional, for the following use cases:
> 1) Main output capture only
> 2) Main + secondary output capture
> 3) Secondary capture only.
I think the 3) is not supported.

The steps are:
1). link necessary the respective subdevices
input --> imgu -->output
            |  -->vf
            |  -->3a stats

2). set all the formats for input, output and intermediate resolutions.
3). start stream

The ipu pipe_mode will not impact the whole pipe behavior. It just ask
firmware to run different processing to generate same format outputs.

> 
> Thanks
>    j
>