* camss: camera controls missing on vfe interfaces @ 2017-10-26 16:11 Daniel Mack 2017-11-15 19:31 ` Daniel Mack 0 siblings, 1 reply; 8+ messages in thread From: Daniel Mack @ 2017-10-26 16:11 UTC (permalink / raw) To: Todor Tomov; +Cc: linux-media, linux-arm-msm, laurent.pinchart Hi Todor, When using the camss driver trough one of its /dev/videoX device nodes, applications are currently unable to see the video controls the camera sensor exposes. Same goes for other ioctls such as VIDIOC_ENUM_FMT, so the only valid resolution setting for applications to use is the one that was previously set through the media controller layer. Applications usually query the available formats and then pick one using the standard V4L2 APIs, and many can't easily be forced to use a specific one. If I'm getting this right, could you explain what's the rationale here? Is that simply a missing feature or was that approach chosen on purpose? Thanks, Daniel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: camss: camera controls missing on vfe interfaces 2017-10-26 16:11 camss: camera controls missing on vfe interfaces Daniel Mack @ 2017-11-15 19:31 ` Daniel Mack 2017-11-20 8:32 ` Todor Tomov 0 siblings, 1 reply; 8+ messages in thread From: Daniel Mack @ 2017-11-15 19:31 UTC (permalink / raw) To: Todor Tomov; +Cc: linux-media, linux-arm-msm, laurent.pinchart Todor et all, Any hint on how to tackle this? I can contribute patches, but I'd like to understand what the idea is. Thanks, Daniel On Thursday, October 26, 2017 06:11 PM, Daniel Mack wrote: > Hi Todor, > > When using the camss driver trough one of its /dev/videoX device nodes, > applications are currently unable to see the video controls the camera > sensor exposes. > > Same goes for other ioctls such as VIDIOC_ENUM_FMT, so the only valid > resolution setting for applications to use is the one that was > previously set through the media controller layer. Applications usually > query the available formats and then pick one using the standard V4L2 > APIs, and many can't easily be forced to use a specific one. > > If I'm getting this right, could you explain what's the rationale here? > Is that simply a missing feature or was that approach chosen on purpose? > > > Thanks, > Daniel > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: camss: camera controls missing on vfe interfaces 2017-11-15 19:31 ` Daniel Mack @ 2017-11-20 8:32 ` Todor Tomov 2017-11-20 10:59 ` Daniel Mack 0 siblings, 1 reply; 8+ messages in thread From: Todor Tomov @ 2017-11-20 8:32 UTC (permalink / raw) To: Daniel Mack, laurent.pinchart, Mauro Carvalho Chehab, Hans Verkuil Cc: linux-media, linux-arm-msm Hi Daniel, Sorry for my late reply. This is actually an important question that you ask. On 15.11.2017 21:31, Daniel Mack wrote: > Todor et all, > > Any hint on how to tackle this? > > I can contribute patches, but I'd like to understand what the idea is. > > > Thanks, > Daniel > > > On Thursday, October 26, 2017 06:11 PM, Daniel Mack wrote: >> Hi Todor, >> >> When using the camss driver trough one of its /dev/videoX device nodes, >> applications are currently unable to see the video controls the camera >> sensor exposes. >> >> Same goes for other ioctls such as VIDIOC_ENUM_FMT, so the only valid >> resolution setting for applications to use is the one that was >> previously set through the media controller layer. Applications usually >> query the available formats and then pick one using the standard V4L2 >> APIs, and many can't easily be forced to use a specific one. >> >> If I'm getting this right, could you explain what's the rationale here? >> Is that simply a missing feature or was that approach chosen on purpose? >> It is not a missing feature, it is more of a missing userspace implementation. When working with a media oriented device driver, the userspace has to config the media pipeline too and if controls are exposed by the subdev nodes, the userspace has to configure them on the subdev nodes. As there weren't a lot of media oriented drivers there is no generic implementation/support for this in the userspace (at least I'm not aware of any). There have been discussions about adding such functionality in libv4l so that applications which do not support media configuration can still use these drivers. I'm not sure if decision for this was taken or not or is it just that there was noone to actually do the work. Probably Laurent, Mauro or Hans know more about what were the plans for this. -- Best regards, Todor Tomov ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: camss: camera controls missing on vfe interfaces 2017-11-20 8:32 ` Todor Tomov @ 2017-11-20 10:59 ` Daniel Mack 2017-11-26 23:50 ` Daniel Mack 2017-12-14 16:49 ` Mauro Carvalho Chehab 0 siblings, 2 replies; 8+ messages in thread From: Daniel Mack @ 2017-11-20 10:59 UTC (permalink / raw) To: Todor Tomov, laurent.pinchart, Mauro Carvalho Chehab, Hans Verkuil Cc: linux-media, linux-arm-msm Hi Todor, Thanks for following up! On Monday, November 20, 2017 09:32 AM, Todor Tomov wrote: > On 15.11.2017 21:31, Daniel Mack wrote: >> Todor et all, >> >> Any hint on how to tackle this? >> >> I can contribute patches, but I'd like to understand what the idea is. >> >> >> Thanks, >> Daniel >> >> >> On Thursday, October 26, 2017 06:11 PM, Daniel Mack wrote: >>> Hi Todor, >>> >>> When using the camss driver trough one of its /dev/videoX device nodes, >>> applications are currently unable to see the video controls the camera >>> sensor exposes. >>> >>> Same goes for other ioctls such as VIDIOC_ENUM_FMT, so the only valid >>> resolution setting for applications to use is the one that was >>> previously set through the media controller layer. Applications usually >>> query the available formats and then pick one using the standard V4L2 >>> APIs, and many can't easily be forced to use a specific one. >>> >>> If I'm getting this right, could you explain what's the rationale here? >>> Is that simply a missing feature or was that approach chosen on purpose? >>> > > It is not a missing feature, it is more of a missing userspace implementation. > When working with a media oriented device driver, the userspace has to > config the media pipeline too and if controls are exposed by the subdev nodes, > the userspace has to configure them on the subdev nodes. > > As there weren't a lot of media oriented drivers there is no generic > implementation/support for this in the userspace (at least I'm not aware of > any). There have been discussions about adding such functionality in libv4l > so that applications which do not support media configuration can still > use these drivers. I'm not sure if decision for this was taken or not or > is it just that there was noone to actually do the work. Probably Laurent, > Mauro or Hans know more about what were the plans for this. Hmm, that's not good. Considering the use-case in our application, the pipeline is set up once and considered more or less static, and then applications such as the Chrome browsers make use of the high-level VFE interface. If there are no controls exposed on that interface, they are not available to the application. Patching all userspace applications is an uphill battle that can't be won I'm afraid. Is there any good reason not to expose the sensor controls on the VFE? I guess it would be easy to do, right? Thanks, Daniel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: camss: camera controls missing on vfe interfaces 2017-11-20 10:59 ` Daniel Mack @ 2017-11-26 23:50 ` Daniel Mack 2017-12-05 12:10 ` Daniel Mack 2017-12-14 16:49 ` Mauro Carvalho Chehab 1 sibling, 1 reply; 8+ messages in thread From: Daniel Mack @ 2017-11-26 23:50 UTC (permalink / raw) To: Todor Tomov, laurent.pinchart, Mauro Carvalho Chehab, Hans Verkuil Cc: linux-media, linux-arm-msm Hi Todor, everyone, On Monday, November 20, 2017 11:59 AM, Daniel Mack wrote: > On Monday, November 20, 2017 09:32 AM, Todor Tomov wrote: >> It is not a missing feature, it is more of a missing userspace implementation. >> When working with a media oriented device driver, the userspace has to >> config the media pipeline too and if controls are exposed by the subdev nodes, >> the userspace has to configure them on the subdev nodes. >> >> As there weren't a lot of media oriented drivers there is no generic >> implementation/support for this in the userspace (at least I'm not aware of >> any). There have been discussions about adding such functionality in libv4l >> so that applications which do not support media configuration can still >> use these drivers. I'm not sure if decision for this was taken or not or >> is it just that there was noone to actually do the work. Probably Laurent, >> Mauro or Hans know more about what were the plans for this. > > Hmm, that's not good. > > Considering the use-case in our application, the pipeline is set up once > and considered more or less static, and then applications such as the > Chrome browsers make use of the high-level VFE interface. If there are > no controls exposed on that interface, they are not available to the > application. Patching all userspace applications is an uphill battle > that can't be won I'm afraid. > > Is there any good reason not to expose the sensor controls on the VFE? I > guess it would be easy to do, right? Do you see an alternative to implementing the above in order to support existing v4l-enabled applications? Thanks, Daniel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: camss: camera controls missing on vfe interfaces 2017-11-26 23:50 ` Daniel Mack @ 2017-12-05 12:10 ` Daniel Mack 0 siblings, 0 replies; 8+ messages in thread From: Daniel Mack @ 2017-12-05 12:10 UTC (permalink / raw) To: Todor Tomov, laurent.pinchart, Mauro Carvalho Chehab, Hans Verkuil Cc: linux-media, linux-arm-msm [-- Attachment #1: Type: text/plain, Size: 1924 bytes --] Hi, FWIW, we are using the attached trivial patch in our tree now. Not sure if this is interesting for upstream, but it does solve our problem. Let me know what you think. Thanks, Daniel On Monday, November 27, 2017 12:50 AM, Daniel Mack wrote: > Hi Todor, everyone, > > On Monday, November 20, 2017 11:59 AM, Daniel Mack wrote: >> On Monday, November 20, 2017 09:32 AM, Todor Tomov wrote: >>> It is not a missing feature, it is more of a missing userspace implementation. >>> When working with a media oriented device driver, the userspace has to >>> config the media pipeline too and if controls are exposed by the subdev nodes, >>> the userspace has to configure them on the subdev nodes. >>> >>> As there weren't a lot of media oriented drivers there is no generic >>> implementation/support for this in the userspace (at least I'm not aware of >>> any). There have been discussions about adding such functionality in libv4l >>> so that applications which do not support media configuration can still >>> use these drivers. I'm not sure if decision for this was taken or not or >>> is it just that there was noone to actually do the work. Probably Laurent, >>> Mauro or Hans know more about what were the plans for this. >> >> Hmm, that's not good. >> >> Considering the use-case in our application, the pipeline is set up once >> and considered more or less static, and then applications such as the >> Chrome browsers make use of the high-level VFE interface. If there are >> no controls exposed on that interface, they are not available to the >> application. Patching all userspace applications is an uphill battle >> that can't be won I'm afraid. >> >> Is there any good reason not to expose the sensor controls on the VFE? I >> guess it would be easy to do, right? > > Do you see an alternative to implementing the above in order to support > existing v4l-enabled applications? > > > Thanks, > Daniel > [-- Attachment #2: 0001-camss-expose-sensor-controls-on-video-node.patch --] [-- Type: text/x-patch, Size: 1848 bytes --] >From 5142b5cdf613f6b61e9506eee81e908045b22404 Mon Sep 17 00:00:00 2001 From: Daniel Mack <daniel@zonque.org> Date: Tue, 5 Dec 2017 11:39:42 +0100 Subject: [PATCH] camss: expose sensor controls on video node --- .../media/platform/qcom/camss-8x16/camss-video.c | 32 ++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/drivers/media/platform/qcom/camss-8x16/camss-video.c b/drivers/media/platform/qcom/camss-8x16/camss-video.c index 2998ad677bee..fd9403b988ec 100644 --- a/drivers/media/platform/qcom/camss-8x16/camss-video.c +++ b/drivers/media/platform/qcom/camss-8x16/camss-video.c @@ -640,10 +640,34 @@ static const struct v4l2_ioctl_ops msm_vid_ioctl_ops = { * V4L2 file operations */ +static struct media_entity *video_find_sensor(struct camss_video *video) +{ + struct media_pad *pad = &video->pad; + + while (1) { + struct media_entity *entity; + + if (!(pad->flags & MEDIA_PAD_FL_SINK)) + return NULL; + + pad = media_entity_remote_pad(pad); + if (!pad || !is_media_entity_v4l2_subdev(pad->entity)) + return NULL; + + entity = pad->entity; + + if (entity->function == MEDIA_ENT_F_CAM_SENSOR) + return entity; + + pad = &entity->pads[0]; + } +} + static int video_open(struct file *file) { struct video_device *vdev = video_devdata(file); struct camss_video *video = video_drvdata(file); + struct media_entity *sensor_entity; struct v4l2_fh *vfh; int ret; @@ -660,6 +684,14 @@ static int video_open(struct file *file) file->private_data = vfh; + sensor_entity = video_find_sensor(video); + if (sensor_entity) { + struct v4l2_subdev *sd = + media_entity_to_v4l2_subdev(sensor_entity); + + vdev->ctrl_handler = sd->ctrl_handler; + } + ret = v4l2_pipeline_pm_use(&vdev->entity, 1); if (ret < 0) { dev_err(video->camss->dev, "Failed to power up pipeline: %d\n", -- 2.14.3 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: camss: camera controls missing on vfe interfaces 2017-11-20 10:59 ` Daniel Mack 2017-11-26 23:50 ` Daniel Mack @ 2017-12-14 16:49 ` Mauro Carvalho Chehab 2018-08-25 12:57 ` Sakari Ailus 1 sibling, 1 reply; 8+ messages in thread From: Mauro Carvalho Chehab @ 2017-12-14 16:49 UTC (permalink / raw) To: Daniel Mack Cc: Todor Tomov, laurent.pinchart, Hans Verkuil, linux-media, linux-arm-msm Em Mon, 20 Nov 2017 11:59:59 +0100 Daniel Mack <daniel@zonque.org> escreveu: > Hi Todor, > > Thanks for following up! > > On Monday, November 20, 2017 09:32 AM, Todor Tomov wrote: > > On 15.11.2017 21:31, Daniel Mack wrote: > >> Todor et all, > >> > >> Any hint on how to tackle this? > >> > >> I can contribute patches, but I'd like to understand what the idea is. > >> > >> > >> Thanks, > >> Daniel > >> > >> > >> On Thursday, October 26, 2017 06:11 PM, Daniel Mack wrote: > >>> Hi Todor, > >>> > >>> When using the camss driver trough one of its /dev/videoX device nodes, > >>> applications are currently unable to see the video controls the camera > >>> sensor exposes. > >>> > >>> Same goes for other ioctls such as VIDIOC_ENUM_FMT, so the only valid > >>> resolution setting for applications to use is the one that was > >>> previously set through the media controller layer. Applications usually > >>> query the available formats and then pick one using the standard V4L2 > >>> APIs, and many can't easily be forced to use a specific one. > >>> > >>> If I'm getting this right, could you explain what's the rationale here? > >>> Is that simply a missing feature or was that approach chosen on purpose? > >>> > > > > It is not a missing feature, it is more of a missing userspace implementation. > > When working with a media oriented device driver, the userspace has to > > config the media pipeline too and if controls are exposed by the subdev nodes, > > the userspace has to configure them on the subdev nodes. > > > > As there weren't a lot of media oriented drivers there is no generic > > implementation/support for this in the userspace (at least I'm not aware of > > any). There have been discussions about adding such functionality in libv4l > > so that applications which do not support media configuration can still > > use these drivers. I'm not sure if decision for this was taken or not or > > is it just that there was noone to actually do the work. Probably Laurent, > > Mauro or Hans know more about what were the plans for this. > > Hmm, that's not good. > > Considering the use-case in our application, the pipeline is set up once > and considered more or less static, and then applications such as the > Chrome browsers make use of the high-level VFE interface. If there are > no controls exposed on that interface, they are not available to the > application. Patching all userspace applications is an uphill battle > that can't be won I'm afraid. > > Is there any good reason not to expose the sensor controls on the VFE? I > guess it would be easy to do, right? Sorry for a late answer. I'm usually very busy on 4Q, but this year, it was atypical. A little historic is needed in order to answer this question. Up to very recently, V4L2 drivers that are media-controller centric, e. g. whose sub-devices are controlled directly by subdev devnodes, were used only on specialized hardware, with special V4L2 applications designed for them. In other words, it was designed to be used by generic applications (although we always wanted a solution for it), and this was never a real problem so far. However, with the advent of cheap SoC hardware with complex media processors on it, the scenario changed recently, and we need some discussions upstream about the best way to solve it. The original idea, back when the media controller was introduced, were to add support at libv4l. But this never happened, and it turns to be a way more complex than originally foreseen. As you're pointing, on such scenarios, one alternative is to expose subdev controls also to the /dev/video devnode that is controlling the pipeline streaming. However, depending on the pipeline, this may not be possible, as the same control could be implemented on more than on block inside the pipeline. When such case happens, the proper solution is to pinpoint what sub-device will handle the control via the subdev API. However, on several scenarios (like, for instance, a RPi3 with a single camera sensor), the pipeline is simple enough to either avoid such conflicts, or to have an obvious subdevice that would be handling such control. From my PoV, on cases like RPi3, the best is to just implement control propagation inside the pipelines. However, other media core developers think otherwise. If you can provide us a broader view about what are the issues that you're facing, what's your use case scenario and what are the pipelines, this could be valuable for us to improve our discussions about the best way to solve it. Please notice, however, that this is not the best time for taking such discussions, as several core developers will be taking vacations those days. Thanks, Mauro ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: camss: camera controls missing on vfe interfaces 2017-12-14 16:49 ` Mauro Carvalho Chehab @ 2018-08-25 12:57 ` Sakari Ailus 0 siblings, 0 replies; 8+ messages in thread From: Sakari Ailus @ 2018-08-25 12:57 UTC (permalink / raw) To: Mauro Carvalho Chehab Cc: Daniel Mack, Todor Tomov, laurent.pinchart, Hans Verkuil, linux-media, linux-arm-msm On Thu, Dec 14, 2017 at 02:49:11PM -0200, Mauro Carvalho Chehab wrote: > Em Mon, 20 Nov 2017 11:59:59 +0100 > Daniel Mack <daniel@zonque.org> escreveu: > > > Hi Todor, > > > > Thanks for following up! > > > > On Monday, November 20, 2017 09:32 AM, Todor Tomov wrote: > > > On 15.11.2017 21:31, Daniel Mack wrote: > > >> Todor et all, > > >> > > >> Any hint on how to tackle this? > > >> > > >> I can contribute patches, but I'd like to understand what the idea is. > > >> > > >> > > >> Thanks, > > >> Daniel > > >> > > >> > > >> On Thursday, October 26, 2017 06:11 PM, Daniel Mack wrote: > > >>> Hi Todor, > > >>> > > >>> When using the camss driver trough one of its /dev/videoX device nodes, > > >>> applications are currently unable to see the video controls the camera > > >>> sensor exposes. > > >>> > > >>> Same goes for other ioctls such as VIDIOC_ENUM_FMT, so the only valid > > >>> resolution setting for applications to use is the one that was > > >>> previously set through the media controller layer. Applications usually > > >>> query the available formats and then pick one using the standard V4L2 > > >>> APIs, and many can't easily be forced to use a specific one. > > >>> > > >>> If I'm getting this right, could you explain what's the rationale here? > > >>> Is that simply a missing feature or was that approach chosen on purpose? > > >>> > > > > > > It is not a missing feature, it is more of a missing userspace implementation. > > > When working with a media oriented device driver, the userspace has to > > > config the media pipeline too and if controls are exposed by the subdev nodes, > > > the userspace has to configure them on the subdev nodes. > > > > > > As there weren't a lot of media oriented drivers there is no generic > > > implementation/support for this in the userspace (at least I'm not aware of > > > any). There have been discussions about adding such functionality in libv4l > > > so that applications which do not support media configuration can still > > > use these drivers. I'm not sure if decision for this was taken or not or > > > is it just that there was noone to actually do the work. Probably Laurent, > > > Mauro or Hans know more about what were the plans for this. > > > > Hmm, that's not good. > > > > Considering the use-case in our application, the pipeline is set up once > > and considered more or less static, and then applications such as the > > Chrome browsers make use of the high-level VFE interface. If there are > > no controls exposed on that interface, they are not available to the > > application. Patching all userspace applications is an uphill battle > > that can't be won I'm afraid. > > > > Is there any good reason not to expose the sensor controls on the VFE? I > > guess it would be easy to do, right? > > Sorry for a late answer. I'm usually very busy on 4Q, but this year, it > was atypical. > > A little historic is needed in order to answer this question. > Up to very recently, V4L2 drivers that are media-controller centric, > e. g. whose sub-devices are controlled directly by subdev devnodes, > were used only on specialized hardware, with special V4L2 applications > designed for them. In other words, it was designed to be used by generic > applications (although we always wanted a solution for it), and this > was never a real problem so far. > > However, with the advent of cheap SoC hardware with complex media > processors on it, the scenario changed recently, and we need some > discussions upstream about the best way to solve it. > > The original idea, back when the media controller was introduced, > were to add support at libv4l. But this never happened, and it turns > to be a way more complex than originally foreseen. > > As you're pointing, on such scenarios, one alternative is to expose subdev > controls also to the /dev/video devnode that is controlling the pipeline > streaming. However, depending on the pipeline, this may not be possible, > as the same control could be implemented on more than on block inside > the pipeline. When such case happens, the proper solution is to pinpoint > what sub-device will handle the control via the subdev API. > > However, on several scenarios (like, for instance, a RPi3 with > a single camera sensor), the pipeline is simple enough to either > avoid such conflicts, or to have an obvious subdevice that would > be handling such control. > > From my PoV, on cases like RPi3, the best is to just implement control > propagation inside the pipelines. However, other media core developers > think otherwise. > > If you can provide us a broader view about what are the issues that > you're facing, what's your use case scenario and what are the pipelines, > this could be valuable for us to improve our discussions about the > best way to solve it. > > Please notice, however, that this is not the best time for taking > such discussions, as several core developers will be taking > vacations those days. I marked the patch obsoleted for now --- the controls available are dependent on the pipeline and the pipeline may change, leading the handler pointing to a different device's control handler. I certainly have no problems in doing this in principle, but it cannot currently be done safely nor correctly in the kernel in many cases. As the controls available in the video node, in general case, would comprise of controls from several different devices, and thus several different control handlers, there may well be duplicates and in this case the kernel would have no clue what to do in such a case. In user space you can have policies that could be applied in such a case; the kernel just provides an interface to the hardware, effectively. Regarding this topic --- there's work beginning towards supporting complex cameras in Linux and this particular matter belongs to the same problem area. -- Kind regards, Sakari Ailus e-mail: sakari.ailus@iki.fi ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-08-25 16:36 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-10-26 16:11 camss: camera controls missing on vfe interfaces Daniel Mack 2017-11-15 19:31 ` Daniel Mack 2017-11-20 8:32 ` Todor Tomov 2017-11-20 10:59 ` Daniel Mack 2017-11-26 23:50 ` Daniel Mack 2017-12-05 12:10 ` Daniel Mack 2017-12-14 16:49 ` Mauro Carvalho Chehab 2018-08-25 12:57 ` Sakari Ailus
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).