[REGRESSION] breakage in sun6i-csi media api

[REGRESSION] breakage in sun6i-csi media api

[Martijn Braam]

It seems like this commit:

media: sun6i-csi: Add bridge v4l2 subdev with port management

Has changed the way the media pipeline on a64 devices, in my case the 
PINE64 PinePhone works. Since this is an API towards userspace and 
there's active applications that use it I think this counts as a regression.

#regzbot introduced: 0d2b746b1bef73de62d2d311e594a7ffed4ca43

Greetings,
Martijn Braam

Re: [REGRESSION] breakage in sun6i-csi media api

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

Hi Martijn,

On Thu 01 Jun 23, 23:19, Martijn Braam wrote:
> It seems like this commit:
> 
> media: sun6i-csi: Add bridge v4l2 subdev with port management
> 
> Has changed the way the media pipeline on a64 devices, in my case the PINE64
> PinePhone works. Since this is an API towards userspace and there's active
> applications that use it I think this counts as a regression.

Do you have more details on what changed specifically?

The commit added a bridge subdev in addition to the video node, which is
generally a better description of the CSI hardware and also a necessity
to support the ISP data flow.

Maybe your userspace application is not configuring the bridge media block with
the right format, which results in a mismatch?

Some work was started to achieve automatic format propagation, perhaps it
would be enough to solve your issue.

Cheers,

Paul

> #regzbot introduced: 0d2b746b1bef73de62d2d311e594a7ffed4ca43
> 
> Greetings,
> Martijn Braam

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [REGRESSION] breakage in sun6i-csi media api

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 2581 bytes --]

Hi,

(Re-adding folks from the original email, adding Laurent and Hans.)

On Fri 02 Jun 23, 11:24, Martijn Braam wrote:
> Hi Paul,
> 
> That's basically it yes. My software doesn't expect the bridge block,
> because it wasn't there. The pipeline always worked "automatically".
> 
> This is the workaround that's been made now it run on newer kernels:
> https://gitlab.com/postmarketOS/megapixels/-/merge_requests/31
> 
> I'm pretty sure format propagation would fix this issue.

Okay that's good to know.

To be honest it's still not very clear to me if in-driver format propagation is
a "nice to have" feature or something that all media pipeline drivers are
supposed to implement.

Anyway I feel like this is not really a regression but a result of the driver
being converted to a newer API.

Also there's a V4L2_CAP_IO_MC flag which should indicate that the video device
must be controlled via the media controller API instead of being
video-device-centric, but I've seen comments asking not to set the flag even
when MC is used so I'm a bit confused here. Perhaps the flag is only required
when there is no automatic format propagation?

If anyone has solid answers on these points I'd be happy to read some
clarification (and act accordingly).

Cheers,

Paul

> Greetings,
> Martijn
> 
> On 6/2/23 11:16, Paul Kocialkowski wrote:
> > Hi Martijn,
> > 
> > On Thu 01 Jun 23, 23:19, Martijn Braam wrote:
> > > It seems like this commit:
> > > 
> > > media: sun6i-csi: Add bridge v4l2 subdev with port management
> > > 
> > > Has changed the way the media pipeline on a64 devices, in my case the PINE64
> > > PinePhone works. Since this is an API towards userspace and there's active
> > > applications that use it I think this counts as a regression.
> > Do you have more details on what changed specifically?
> > 
> > The commit added a bridge subdev in addition to the video node, which is
> > generally a better description of the CSI hardware and also a necessity
> > to support the ISP data flow.
> > 
> > Maybe your userspace application is not configuring the bridge media block with
> > the right format, which results in a mismatch?
> > 
> > Some work was started to achieve automatic format propagation, perhaps it
> > would be enough to solve your issue.
> > 
> > Cheers,
> > 
> > Paul
> > 
> > > #regzbot introduced: 0d2b746b1bef73de62d2d311e594a7ffed4ca43
> > > 
> > > Greetings,
> > > Martijn Braam

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [REGRESSION] breakage in sun6i-csi media api

[Laurent Pinchart]

Hi Paul,

On Fri, Jun 02, 2023 at 11:39:54AM +0200, Paul Kocialkowski wrote:
> Hi,
> 
> (Re-adding folks from the original email, adding Laurent and Hans.)
> 
> On Fri 02 Jun 23, 11:24, Martijn Braam wrote:
> > Hi Paul,
> > 
> > That's basically it yes. My software doesn't expect the bridge block,
> > because it wasn't there. The pipeline always worked "automatically".
> > 
> > This is the workaround that's been made now it run on newer kernels:
> > https://gitlab.com/postmarketOS/megapixels/-/merge_requests/31
> > 
> > I'm pretty sure format propagation would fix this issue.
> 
> Okay that's good to know.
> 
> To be honest it's still not very clear to me if in-driver format propagation is
> a "nice to have" feature or something that all media pipeline drivers are
> supposed to implement.

For MC-based drivers, in-kernel propagation *inside* subdevs is
mandatory, in-kernel propagration *between* subdevs is not allowed. The
latter is the responsibility of userspace.

For traditional (I'd say legacy, but I know not everybody likes that
term :-)) drivers that only expose video device nodes and do not expose
subdev nodes to userspace, the driver is responsible for configuring the
full pipeline internally based on the S_FMT call on the video nodes
only. This isn't applicable to the sun6i-csi driver, as it exposes
subdev nodes to userspace.

> Anyway I feel like this is not really a regression but a result of the driver
> being converted to a newer API.
> 
> Also there's a V4L2_CAP_IO_MC flag which should indicate that the video device
> must be controlled via the media controller API instead of being
> video-device-centric, but I've seen comments asking not to set the flag even
> when MC is used so I'm a bit confused here.

Would you have pointers to those comments ?

> Perhaps the flag is only required when there is no automatic format
> propagation?

The flag is more or less required when you expose subdev nodes to
userspace.

> If anyone has solid answers on these points I'd be happy to read some
> clarification (and act accordingly).
>
> > On 6/2/23 11:16, Paul Kocialkowski wrote:
> > > Hi Martijn,
> > > 
> > > On Thu 01 Jun 23, 23:19, Martijn Braam wrote:
> > > > It seems like this commit:
> > > > 
> > > > media: sun6i-csi: Add bridge v4l2 subdev with port management
> > > > 
> > > > Has changed the way the media pipeline on a64 devices, in my case the PINE64
> > > > PinePhone works. Since this is an API towards userspace and there's active
> > > > applications that use it I think this counts as a regression.
> > > Do you have more details on what changed specifically?
> > > 
> > > The commit added a bridge subdev in addition to the video node, which is
> > > generally a better description of the CSI hardware and also a necessity
> > > to support the ISP data flow.
> > > 
> > > Maybe your userspace application is not configuring the bridge media block with
> > > the right format, which results in a mismatch?
> > > 
> > > Some work was started to achieve automatic format propagation, perhaps it
> > > would be enough to solve your issue.
> > > 
> > > Cheers,
> > > 
> > > Paul
> > > 
> > > > #regzbot introduced: 0d2b746b1bef73de62d2d311e594a7ffed4ca43

-- 
Regards,

Laurent Pinchart

Re: [REGRESSION] breakage in sun6i-csi media api

[Linux regression tracking (Thorsten Leemhuis)]

Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting
for once, to make this easily accessible to everyone.

Paul, what happened to this? It looks like this fall through the cracks,
but maybe I'm missing something, that's why I ask.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

On 02.06.23 12:03, Laurent Pinchart wrote:
> On Fri, Jun 02, 2023 at 11:39:54AM +0200, Paul Kocialkowski wrote:
>> (Re-adding folks from the original email, adding Laurent and Hans.)
>> On Fri 02 Jun 23, 11:24, Martijn Braam wrote:
>>>
>>> That's basically it yes. My software doesn't expect the bridge block,
>>> because it wasn't there. The pipeline always worked "automatically".
>>>
>>> This is the workaround that's been made now it run on newer kernels:
>>> https://gitlab.com/postmarketOS/megapixels/-/merge_requests/31
>>>
>>> I'm pretty sure format propagation would fix this issue.
>>
>> Okay that's good to know.
>>
>> To be honest it's still not very clear to me if in-driver format propagation is
>> a "nice to have" feature or something that all media pipeline drivers are
>> supposed to implement.
> 
> For MC-based drivers, in-kernel propagation *inside* subdevs is
> mandatory, in-kernel propagration *between* subdevs is not allowed. The
> latter is the responsibility of userspace.
> 
> For traditional (I'd say legacy, but I know not everybody likes that
> term :-)) drivers that only expose video device nodes and do not expose
> subdev nodes to userspace, the driver is responsible for configuring the
> full pipeline internally based on the S_FMT call on the video nodes
> only. This isn't applicable to the sun6i-csi driver, as it exposes
> subdev nodes to userspace.
> 
>> Anyway I feel like this is not really a regression but a result of the driver
>> being converted to a newer API.
>>
>> Also there's a V4L2_CAP_IO_MC flag which should indicate that the video device
>> must be controlled via the media controller API instead of being
>> video-device-centric, but I've seen comments asking not to set the flag even
>> when MC is used so I'm a bit confused here.
> 
> Would you have pointers to those comments ?
> 
>> Perhaps the flag is only required when there is no automatic format
>> propagation?
> 
> The flag is more or less required when you expose subdev nodes to
> userspace.
> 
>> If anyone has solid answers on these points I'd be happy to read some
>> clarification (and act accordingly).
>>
>>> On 6/2/23 11:16, Paul Kocialkowski wrote:
>>>> Hi Martijn,
>>>>
>>>> On Thu 01 Jun 23, 23:19, Martijn Braam wrote:
>>>>> It seems like this commit:
>>>>>
>>>>> media: sun6i-csi: Add bridge v4l2 subdev with port management
>>>>>
>>>>> Has changed the way the media pipeline on a64 devices, in my case the PINE64
>>>>> PinePhone works. Since this is an API towards userspace and there's active
>>>>> applications that use it I think this counts as a regression.
>>>> Do you have more details on what changed specifically?
>>>>
>>>> The commit added a bridge subdev in addition to the video node, which is
>>>> generally a better description of the CSI hardware and also a necessity
>>>> to support the ISP data flow.
>>>>
>>>> Maybe your userspace application is not configuring the bridge media block with
>>>> the right format, which results in a mismatch?
>>>>
>>>> Some work was started to achieve automatic format propagation, perhaps it
>>>> would be enough to solve your issue.
>>>>
>>>> Cheers,
>>>>
>>>> Paul
>>>>
>>>>> #regzbot introduced: 0d2b746b1bef73de62d2d311e594a7ffed4ca43
> 

Re: [REGRESSION] breakage in sun6i-csi media api

[Mauro Carvalho Chehab]

Em Mon, 26 Jun 2023 14:23:50 +0200
"Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> escreveu:

> Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting
> for once, to make this easily accessible to everyone.
> 
> Paul, what happened to this? It looks like this fall through the cracks,
> but maybe I'm missing something, that's why I ask.

On a quick look, checking the proposed app fix[1], it sounds to me that
the application is not prepared to actually use the devices that are
enumerated, as it is setting only the values that are known.

See, the media controller has the MEDIA_IOC_G_TOPOLOGY ioctl [2], which
enumerates the entire sub-device topology.

The current application was just expecting two sub-devices, failing
if other devices are added on newer Kernels. This is not how this uAPI is
supposed to work.

[1] https://gitlab.com/postmarketOS/megapixels/-/merge_requests/31/diffs?commit_id=38bbee084126b15d39c1bce5cb5d45e6efea64fa
[2] https://gitlab.com/postmarketOS/megapixels/-/blob/master/src/device.c#L106

Now, I was told during yesterday, during the Media Summit that there are
new apps since Kernel 6.2 that do require setting the bridge with different
configurations. Paul/Sebastian: is this the case? If so, could you provide
more details about it?

If this is true, it seems too late to revert the changes, as this will
break other existing applications.

IMO, the best here would be to modify the application to be smarter,
using the topology actually reported by MEDIA_IOC_G_TOPOLOGY, instead
on relying on some specific hard-coded types.

Regards,
Mauro

> 
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
> 
> #regzbot poke
> 
> On 02.06.23 12:03, Laurent Pinchart wrote:
> > On Fri, Jun 02, 2023 at 11:39:54AM +0200, Paul Kocialkowski wrote:  
> >> (Re-adding folks from the original email, adding Laurent and Hans.)
> >> On Fri 02 Jun 23, 11:24, Martijn Braam wrote:  
> >>>
> >>> That's basically it yes. My software doesn't expect the bridge block,
> >>> because it wasn't there. The pipeline always worked "automatically".
> >>>
> >>> This is the workaround that's been made now it run on newer kernels:
> >>> https://gitlab.com/postmarketOS/megapixels/-/merge_requests/31
> >>>
> >>> I'm pretty sure format propagation would fix this issue.  
> >>
> >> Okay that's good to know.
> >>
> >> To be honest it's still not very clear to me if in-driver format propagation is
> >> a "nice to have" feature or something that all media pipeline drivers are
> >> supposed to implement.  
> > 
> > For MC-based drivers, in-kernel propagation *inside* subdevs is
> > mandatory, in-kernel propagration *between* subdevs is not allowed. The
> > latter is the responsibility of userspace.
> > 
> > For traditional (I'd say legacy, but I know not everybody likes that
> > term :-)) drivers that only expose video device nodes and do not expose
> > subdev nodes to userspace, the driver is responsible for configuring the
> > full pipeline internally based on the S_FMT call on the video nodes
> > only. This isn't applicable to the sun6i-csi driver, as it exposes
> > subdev nodes to userspace.
> >   
> >> Anyway I feel like this is not really a regression but a result of the driver
> >> being converted to a newer API.
> >>
> >> Also there's a V4L2_CAP_IO_MC flag which should indicate that the video device
> >> must be controlled via the media controller API instead of being
> >> video-device-centric, but I've seen comments asking not to set the flag even
> >> when MC is used so I'm a bit confused here.  
> > 
> > Would you have pointers to those comments ?
> >   
> >> Perhaps the flag is only required when there is no automatic format
> >> propagation?  
> > 
> > The flag is more or less required when you expose subdev nodes to
> > userspace.
> >   
> >> If anyone has solid answers on these points I'd be happy to read some
> >> clarification (and act accordingly).
> >>  
> >>> On 6/2/23 11:16, Paul Kocialkowski wrote:  
> >>>> Hi Martijn,
> >>>>
> >>>> On Thu 01 Jun 23, 23:19, Martijn Braam wrote:  
> >>>>> It seems like this commit:
> >>>>>
> >>>>> media: sun6i-csi: Add bridge v4l2 subdev with port management
> >>>>>
> >>>>> Has changed the way the media pipeline on a64 devices, in my case the PINE64
> >>>>> PinePhone works. Since this is an API towards userspace and there's active
> >>>>> applications that use it I think this counts as a regression.  
> >>>> Do you have more details on what changed specifically?
> >>>>
> >>>> The commit added a bridge subdev in addition to the video node, which is
> >>>> generally a better description of the CSI hardware and also a necessity
> >>>> to support the ISP data flow.
> >>>>
> >>>> Maybe your userspace application is not configuring the bridge media block with
> >>>> the right format, which results in a mismatch?
> >>>>
> >>>> Some work was started to achieve automatic format propagation, perhaps it
> >>>> would be enough to solve your issue.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Paul
> >>>>  
> >>>>> #regzbot introduced: 0d2b746b1bef73de62d2d311e594a7ffed4ca43  
> >   

Re: [REGRESSION] breakage in sun6i-csi media api

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 6328 bytes --]

Hi,

Sorry for the delayed reply here.

On Thu 29 Jun 23, 00:08, Mauro Carvalho Chehab wrote:
> Em Mon, 26 Jun 2023 14:23:50 +0200
> "Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> escreveu:
> 
> > Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting
> > for once, to make this easily accessible to everyone.
> > 
> > Paul, what happened to this? It looks like this fall through the cracks,
> > but maybe I'm missing something, that's why I ask.
> 
> On a quick look, checking the proposed app fix[1], it sounds to me that
> the application is not prepared to actually use the devices that are
> enumerated, as it is setting only the values that are known.

I totally agree with this, it seems to me that the application just had
hardcoded expectations about what is exposed and did not implement
dynamic enumeration as it should have (thus not following the uAPI).

Hopefully the fix to the application will make it future-proof and we can avoid
this kind of confusion in the future!

> See, the media controller has the MEDIA_IOC_G_TOPOLOGY ioctl [2], which
> enumerates the entire sub-device topology.
> 
> The current application was just expecting two sub-devices, failing
> if other devices are added on newer Kernels. This is not how this uAPI is
> supposed to work.
> 
> [1] https://gitlab.com/postmarketOS/megapixels/-/merge_requests/31/diffs?commit_id=38bbee084126b15d39c1bce5cb5d45e6efea64fa
> [2] https://gitlab.com/postmarketOS/megapixels/-/blob/master/src/device.c#L106
> 
> Now, I was told during yesterday, during the Media Summit that there are
> new apps since Kernel 6.2 that do require setting the bridge with different
> configurations. Paul/Sebastian: is this the case? If so, could you provide
> more details about it?

I am not aware of this, so also interested in details.

> If this is true, it seems too late to revert the changes, as this will
> break other existing applications.
> 
> IMO, the best here would be to modify the application to be smarter,
> using the topology actually reported by MEDIA_IOC_G_TOPOLOGY, instead
> on relying on some specific hard-coded types.

Absolutely!

Thanks,

Paul

> Regards,
> Mauro
> 
> > 
> > Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> > --
> > Everything you wanna know about Linux kernel regression tracking:
> > https://linux-regtracking.leemhuis.info/about/#tldr
> > If I did something stupid, please tell me, as explained on that page.
> > 
> > #regzbot poke
> > 
> > On 02.06.23 12:03, Laurent Pinchart wrote:
> > > On Fri, Jun 02, 2023 at 11:39:54AM +0200, Paul Kocialkowski wrote:  
> > >> (Re-adding folks from the original email, adding Laurent and Hans.)
> > >> On Fri 02 Jun 23, 11:24, Martijn Braam wrote:  
> > >>>
> > >>> That's basically it yes. My software doesn't expect the bridge block,
> > >>> because it wasn't there. The pipeline always worked "automatically".
> > >>>
> > >>> This is the workaround that's been made now it run on newer kernels:
> > >>> https://gitlab.com/postmarketOS/megapixels/-/merge_requests/31
> > >>>
> > >>> I'm pretty sure format propagation would fix this issue.  
> > >>
> > >> Okay that's good to know.
> > >>
> > >> To be honest it's still not very clear to me if in-driver format propagation is
> > >> a "nice to have" feature or something that all media pipeline drivers are
> > >> supposed to implement.  
> > > 
> > > For MC-based drivers, in-kernel propagation *inside* subdevs is
> > > mandatory, in-kernel propagration *between* subdevs is not allowed. The
> > > latter is the responsibility of userspace.
> > > 
> > > For traditional (I'd say legacy, but I know not everybody likes that
> > > term :-)) drivers that only expose video device nodes and do not expose
> > > subdev nodes to userspace, the driver is responsible for configuring the
> > > full pipeline internally based on the S_FMT call on the video nodes
> > > only. This isn't applicable to the sun6i-csi driver, as it exposes
> > > subdev nodes to userspace.
> > >   
> > >> Anyway I feel like this is not really a regression but a result of the driver
> > >> being converted to a newer API.
> > >>
> > >> Also there's a V4L2_CAP_IO_MC flag which should indicate that the video device
> > >> must be controlled via the media controller API instead of being
> > >> video-device-centric, but I've seen comments asking not to set the flag even
> > >> when MC is used so I'm a bit confused here.  
> > > 
> > > Would you have pointers to those comments ?
> > >   
> > >> Perhaps the flag is only required when there is no automatic format
> > >> propagation?  
> > > 
> > > The flag is more or less required when you expose subdev nodes to
> > > userspace.
> > >   
> > >> If anyone has solid answers on these points I'd be happy to read some
> > >> clarification (and act accordingly).
> > >>  
> > >>> On 6/2/23 11:16, Paul Kocialkowski wrote:  
> > >>>> Hi Martijn,
> > >>>>
> > >>>> On Thu 01 Jun 23, 23:19, Martijn Braam wrote:  
> > >>>>> It seems like this commit:
> > >>>>>
> > >>>>> media: sun6i-csi: Add bridge v4l2 subdev with port management
> > >>>>>
> > >>>>> Has changed the way the media pipeline on a64 devices, in my case the PINE64
> > >>>>> PinePhone works. Since this is an API towards userspace and there's active
> > >>>>> applications that use it I think this counts as a regression.  
> > >>>> Do you have more details on what changed specifically?
> > >>>>
> > >>>> The commit added a bridge subdev in addition to the video node, which is
> > >>>> generally a better description of the CSI hardware and also a necessity
> > >>>> to support the ISP data flow.
> > >>>>
> > >>>> Maybe your userspace application is not configuring the bridge media block with
> > >>>> the right format, which results in a mismatch?
> > >>>>
> > >>>> Some work was started to achieve automatic format propagation, perhaps it
> > >>>> would be enough to solve your issue.
> > >>>>
> > >>>> Cheers,
> > >>>>
> > >>>> Paul
> > >>>>  
> > >>>>> #regzbot introduced: 0d2b746b1bef73de62d2d311e594a7ffed4ca43  
> > >   

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [REGRESSION] breakage in sun6i-csi media api

[Linux regression tracking #update (Thorsten Leemhuis)]

[TLDR: This mail in primarily relevant for Linux kernel regression
tracking. See link in footer if these mails annoy you.]

On 12.07.23 16:47, Paul Kocialkowski wrote:
> On Thu 29 Jun 23, 00:08, Mauro Carvalho Chehab wrote:
>> Em Mon, 26 Jun 2023 14:23:50 +0200
>> "Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> escreveu:
>>
>> IMO, the best here would be to modify the application to be smarter,
>> using the topology actually reported by MEDIA_IOC_G_TOPOLOGY, instead
>> on relying on some specific hard-coded types.
> Absolutely!

#regzbot resolve: reverting or fixing would cause other regression,
fixing the app thus is the better of two bad solutions
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Re: [REGRESSION] breakage in sun6i-csi media api

[Paul Kocialkowski]

[-- Attachment #1: Type: text/plain, Size: 4281 bytes --]

Hi,

On Fri 02 Jun 23, 13:03, Laurent Pinchart wrote:
> Hi Paul,
> 
> On Fri, Jun 02, 2023 at 11:39:54AM +0200, Paul Kocialkowski wrote:
> > Hi,
> > 
> > (Re-adding folks from the original email, adding Laurent and Hans.)
> > 
> > On Fri 02 Jun 23, 11:24, Martijn Braam wrote:
> > > Hi Paul,
> > > 
> > > That's basically it yes. My software doesn't expect the bridge block,
> > > because it wasn't there. The pipeline always worked "automatically".
> > > 
> > > This is the workaround that's been made now it run on newer kernels:
> > > https://gitlab.com/postmarketOS/megapixels/-/merge_requests/31
> > > 
> > > I'm pretty sure format propagation would fix this issue.
> > 
> > Okay that's good to know.
> > 
> > To be honest it's still not very clear to me if in-driver format propagation is
> > a "nice to have" feature or something that all media pipeline drivers are
> > supposed to implement.
> 
> For MC-based drivers, in-kernel propagation *inside* subdevs is
> mandatory, in-kernel propagration *between* subdevs is not allowed. The
> latter is the responsibility of userspace.

Right, makes good sense.

> For traditional (I'd say legacy, but I know not everybody likes that
> term :-)) drivers that only expose video device nodes and do not expose
> subdev nodes to userspace, the driver is responsible for configuring the
> full pipeline internally based on the S_FMT call on the video nodes
> only. This isn't applicable to the sun6i-csi driver, as it exposes
> subdev nodes to userspace.
> 
> > Anyway I feel like this is not really a regression but a result of the driver
> > being converted to a newer API.
> > 
> > Also there's a V4L2_CAP_IO_MC flag which should indicate that the video device
> > must be controlled via the media controller API instead of being
> > video-device-centric, but I've seen comments asking not to set the flag even
> > when MC is used so I'm a bit confused here.
> 
> Would you have pointers to those comments ?

I was thinking of a reply from Hans on the Tegra driver series:
https://patchwork.kernel.org/project/linux-media/patch/20230309144320.2937553-4-luca.ceresoli@bootlin.com/#25285456

From what I understand there's some bridge entity that needs to be explicitly
configured to match the sensor's format via MC, yet the reply says that the
driver "doesn't use the media controller for anything at the moment" and
should not be converted to become a proper MC-driven device.

Well maybe there was a misunderstanding on my side, or Hans', or both.

> > Perhaps the flag is only required when there is no automatic format
> > propagation?
> 
> The flag is more or less required when you expose subdev nodes to
> userspace.

Thanks for the clarifications :)

Cheers,

Paul

> > If anyone has solid answers on these points I'd be happy to read some
> > clarification (and act accordingly).
> >
> > > On 6/2/23 11:16, Paul Kocialkowski wrote:
> > > > Hi Martijn,
> > > > 
> > > > On Thu 01 Jun 23, 23:19, Martijn Braam wrote:
> > > > > It seems like this commit:
> > > > > 
> > > > > media: sun6i-csi: Add bridge v4l2 subdev with port management
> > > > > 
> > > > > Has changed the way the media pipeline on a64 devices, in my case the PINE64
> > > > > PinePhone works. Since this is an API towards userspace and there's active
> > > > > applications that use it I think this counts as a regression.
> > > > Do you have more details on what changed specifically?
> > > > 
> > > > The commit added a bridge subdev in addition to the video node, which is
> > > > generally a better description of the CSI hardware and also a necessity
> > > > to support the ISP data flow.
> > > > 
> > > > Maybe your userspace application is not configuring the bridge media block with
> > > > the right format, which results in a mismatch?
> > > > 
> > > > Some work was started to achieve automatic format propagation, perhaps it
> > > > would be enough to solve your issue.
> > > > 
> > > > Cheers,
> > > > 
> > > > Paul
> > > > 
> > > > > #regzbot introduced: 0d2b746b1bef73de62d2d311e594a7ffed4ca43
> 
> -- 
> Regards,
> 
> Laurent Pinchart

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [REGRESSION] breakage in sun6i-csi media api

[Laurent Pinchart]

On Wed, Jul 12, 2023 at 04:55:21PM +0200, Paul Kocialkowski wrote:
> On Fri 02 Jun 23, 13:03, Laurent Pinchart wrote:
> > On Fri, Jun 02, 2023 at 11:39:54AM +0200, Paul Kocialkowski wrote:
> > > Hi,
> > > 
> > > (Re-adding folks from the original email, adding Laurent and Hans.)
> > > 
> > > On Fri 02 Jun 23, 11:24, Martijn Braam wrote:
> > > > Hi Paul,
> > > > 
> > > > That's basically it yes. My software doesn't expect the bridge block,
> > > > because it wasn't there. The pipeline always worked "automatically".
> > > > 
> > > > This is the workaround that's been made now it run on newer kernels:
> > > > https://gitlab.com/postmarketOS/megapixels/-/merge_requests/31
> > > > 
> > > > I'm pretty sure format propagation would fix this issue.
> > > 
> > > Okay that's good to know.
> > > 
> > > To be honest it's still not very clear to me if in-driver format propagation is
> > > a "nice to have" feature or something that all media pipeline drivers are
> > > supposed to implement.
> > 
> > For MC-based drivers, in-kernel propagation *inside* subdevs is
> > mandatory, in-kernel propagration *between* subdevs is not allowed. The
> > latter is the responsibility of userspace.
> 
> Right, makes good sense.
> 
> > For traditional (I'd say legacy, but I know not everybody likes that
> > term :-)) drivers that only expose video device nodes and do not expose
> > subdev nodes to userspace, the driver is responsible for configuring the
> > full pipeline internally based on the S_FMT call on the video nodes
> > only. This isn't applicable to the sun6i-csi driver, as it exposes
> > subdev nodes to userspace.
> > 
> > > Anyway I feel like this is not really a regression but a result of the driver
> > > being converted to a newer API.
> > > 
> > > Also there's a V4L2_CAP_IO_MC flag which should indicate that the video device
> > > must be controlled via the media controller API instead of being
> > > video-device-centric, but I've seen comments asking not to set the flag even
> > > when MC is used so I'm a bit confused here.
> > 
> > Would you have pointers to those comments ?
> 
> I was thinking of a reply from Hans on the Tegra driver series:
> https://patchwork.kernel.org/project/linux-media/patch/20230309144320.2937553-4-luca.ceresoli@bootlin.com/#25285456
> 
> From what I understand there's some bridge entity that needs to be explicitly
> configured to match the sensor's format via MC, yet the reply says that the
> driver "doesn't use the media controller for anything at the moment" and
> should not be converted to become a proper MC-driven device.
> 
> Well maybe there was a misunderstanding on my side, or Hans', or both.

It makes sense not to set the flag in some cases. For instance, the
uvcvideo driver creates an MC device to expose the camera's internal
topology to userspace, but doesn't create subdevs as the UVC protocol
doesn't allow fine-grained configuration of formats between entities. In
the sun6i-csi case, I don't think this is applicable, the device should
be configured directly at the subdev level by userspace as you can't
know in advance what will be connected to its inputs.

For what it's worth, the same reasoning should apply to the tegra
driver, and I think it should set V4L2_CAP_IO_MC.

> > > Perhaps the flag is only required when there is no automatic format
> > > propagation?
> > 
> > The flag is more or less required when you expose subdev nodes to
> > userspace.
> 
> Thanks for the clarifications :)
> 
> > > If anyone has solid answers on these points I'd be happy to read some
> > > clarification (and act accordingly).
> > >
> > > > On 6/2/23 11:16, Paul Kocialkowski wrote:
> > > > > Hi Martijn,
> > > > > 
> > > > > On Thu 01 Jun 23, 23:19, Martijn Braam wrote:
> > > > > > It seems like this commit:
> > > > > > 
> > > > > > media: sun6i-csi: Add bridge v4l2 subdev with port management
> > > > > > 
> > > > > > Has changed the way the media pipeline on a64 devices, in my case the PINE64
> > > > > > PinePhone works. Since this is an API towards userspace and there's active
> > > > > > applications that use it I think this counts as a regression.
> > > > > Do you have more details on what changed specifically?
> > > > > 
> > > > > The commit added a bridge subdev in addition to the video node, which is
> > > > > generally a better description of the CSI hardware and also a necessity
> > > > > to support the ISP data flow.
> > > > > 
> > > > > Maybe your userspace application is not configuring the bridge media block with
> > > > > the right format, which results in a mismatch?
> > > > > 
> > > > > Some work was started to achieve automatic format propagation, perhaps it
> > > > > would be enough to solve your issue.
> > > > > 
> > > > > Cheers,
> > > > > 
> > > > > Paul
> > > > > 
> > > > > > #regzbot introduced: 0d2b746b1bef73de62d2d311e594a7ffed4ca43

-- 
Regards,

Laurent Pinchart