[RFC] How to pass camera Orientation to userspace

[RFC] How to pass camera Orientation to userspace

[Adam Baker]

(linux-omap included in distribution as lots of omap systems include cameras 
so this could be relevant there.)

Background

A number of the webcams now supported by v4l have sensors that are mounted 
upside down. Up to now this has been handled by having a table in libv4l of 
the USB IDs of affected cameras. This approach however fails to address two 
known cases (and probably more as yet unknown ones) where the USB ID is 
insufficient to determine the sensor orientation.

In one of those cases (SQ-905) USB commands must be issued to the camera  at 
probe time) to determine what sensor is fitted and in the other case (Genesys 
gl860) the camera can be pointed towards or away from the user and it swaps 
orientation when it is changed.

It is possible that there are cameras that can use gravity sensors or similar 
to report how they are being held but such user driven orientation which may 
be intended for creative effect should probably be separated from this 
hardware related issue.

Most if not all of the cameras affected by this problem produce video formats 
that are not widely supported by applications. They are therefore all likely 
to be normally used in conjunction with libv4l to perform format conversion 
and so it makes sense to retain the actual flipping operation in libv4l. 
libv4l provides a capability via LD_PRELOAD to attach itself to unmodified 
binaries.

It is likely that whatever solution is chosen there will end up being cases 
that have to be handled in libv4l as external information (such as laptop 
model ID) is the only mechanism to distinguish otherwise identical cameras.

So far libv4l only supports the case of data needing a 180 degree rotation but 
there is known to exist hardware which requires a VFLIP and it is possible 
some hardware requires just HFLIP so all of those cases should be supported.

There have been a number of inconclusive discussions on this subject on the 
v4l / linux-media mailing lists over the last few months offering many 
options for how to pass this information across which I will list below and 
hopefully a preferred solution can be selected from them

1) Reuse the existing HFLIP and VFLIP controls, marking them as read-only
Pros : No change needed to videodev2.h
Cons: It is confusing to have controls that have a subtly different meaning if 
they are read only. Existing apps that support those controls might get 
confused. Would require polling to support the case of a camera being turned 
toward / away from the user while streaming.

2) Introduce a new orientation control (possibly in a new CAMERA_PROPERTIES 
class)
Pros: libv4l can easily tell if the driver supports the control.
Cons: It is really a property, not a control so calling it a control is wrong. 
Controls add lots of overhead in terms of driver code. Would require polling 
to support the case of a camera being turned toward / away from the user 
while streaming.

3) Use an extra couple of bits in V4L2_BUF_FLAGS
Pros: Simple to implement. Can change per frame without needing polling.
Cons: Doesn't work for non libv4l apps that try to use the read() interface. 
Can't easily identify drivers that don't support it (does 0 mean not rotated 
or just not implemented). Can only be read when streaming (does that matter?)

4) Use some reserved bits from the v4l2_capability structure
Pros: Less overhead than controls.
Cons: Would require polling to support the case of a camera being turned 
toward / away from the user while streaming. Can't easily identify drivers 
that don't support it.

5) Use some reserved bits from the v4l2_input structure (or possibly the 
status word but that is normally only valid for current input)
Pros: Less overhead than controls. Could support multiple sensors in one 
camera if such a beast exists.
Cons: Would require polling to support the case of a camera being turned 
toward / away from the user while streaming. Can't easily identify drivers 
that don't support it.

The interest in detecting if a driver provides this informnation is to allow 
libv4l to know when it should use the driver provided information and when it 
should use its internal table (which needs to be retained for backward 
compatibility). With no detection capability the driver provided info should 
be ignored for USB IDs in the built in table.

Thoughts please

Adam Baker

Re: [RFC] How to pass camera Orientation to userspace

[DongSoo(Nathaniel) Kim]

Hello Adam,

I've been thinking exactly the same issue not usb but SoC based camera.
I have no idea about how usb cameras work but I am quite curious about
is it really possible to make proper orientation with only querying
camera driver.
Because in case of SoC based camera device, many of camera ISPs are
supporting VFLIP, HFLIP register on their own, and we can read current
orientation by reading those registers.

But the problem is ISP's registers are set as not flipped at all but
it physically mounted upside down, because the H/W  vendor has packed
the camera module upside down. (it sounds ridiculous but happens
sometimes)
So in that case when we query orientation of camera, it returns not
flipped vertically or horizontally at all but actually it turns out to
be upside down. Actually we are setting camera device to be flipped
for default in that case.

I guess in USB camera case that kind of situation is not happening,
but for SoC camera it should be a tough job to find out exact way of
determine the orientation of camera automatically.

Cheers,

Nate

On Wed, Feb 18, 2009 at 9:30 AM, Adam Baker <linux@baker-net.org.uk> wrote:
> (linux-omap included in distribution as lots of omap systems include cameras
> so this could be relevant there.)
>
> Background
>
> A number of the webcams now supported by v4l have sensors that are mounted
> upside down. Up to now this has been handled by having a table in libv4l of
> the USB IDs of affected cameras. This approach however fails to address two
> known cases (and probably more as yet unknown ones) where the USB ID is
> insufficient to determine the sensor orientation.
>
> In one of those cases (SQ-905) USB commands must be issued to the camera  at
> probe time) to determine what sensor is fitted and in the other case (Genesys
> gl860) the camera can be pointed towards or away from the user and it swaps
> orientation when it is changed.
>
> It is possible that there are cameras that can use gravity sensors or similar
> to report how they are being held but such user driven orientation which may
> be intended for creative effect should probably be separated from this
> hardware related issue.
>
> Most if not all of the cameras affected by this problem produce video formats
> that are not widely supported by applications. They are therefore all likely
> to be normally used in conjunction with libv4l to perform format conversion
> and so it makes sense to retain the actual flipping operation in libv4l.
> libv4l provides a capability via LD_PRELOAD to attach itself to unmodified
> binaries.
>
> It is likely that whatever solution is chosen there will end up being cases
> that have to be handled in libv4l as external information (such as laptop
> model ID) is the only mechanism to distinguish otherwise identical cameras.
>
> So far libv4l only supports the case of data needing a 180 degree rotation but
> there is known to exist hardware which requires a VFLIP and it is possible
> some hardware requires just HFLIP so all of those cases should be supported.
>
> There have been a number of inconclusive discussions on this subject on the
> v4l / linux-media mailing lists over the last few months offering many
> options for how to pass this information across which I will list below and
> hopefully a preferred solution can be selected from them
>
> 1) Reuse the existing HFLIP and VFLIP controls, marking them as read-only
> Pros : No change needed to videodev2.h
> Cons: It is confusing to have controls that have a subtly different meaning if
> they are read only. Existing apps that support those controls might get
> confused. Would require polling to support the case of a camera being turned
> toward / away from the user while streaming.
>
> 2) Introduce a new orientation control (possibly in a new CAMERA_PROPERTIES
> class)
> Pros: libv4l can easily tell if the driver supports the control.
> Cons: It is really a property, not a control so calling it a control is wrong.
> Controls add lots of overhead in terms of driver code. Would require polling
> to support the case of a camera being turned toward / away from the user
> while streaming.
>
> 3) Use an extra couple of bits in V4L2_BUF_FLAGS
> Pros: Simple to implement. Can change per frame without needing polling.
> Cons: Doesn't work for non libv4l apps that try to use the read() interface.
> Can't easily identify drivers that don't support it (does 0 mean not rotated
> or just not implemented). Can only be read when streaming (does that matter?)
>
> 4) Use some reserved bits from the v4l2_capability structure
> Pros: Less overhead than controls.
> Cons: Would require polling to support the case of a camera being turned
> toward / away from the user while streaming. Can't easily identify drivers
> that don't support it.
>
> 5) Use some reserved bits from the v4l2_input structure (or possibly the
> status word but that is normally only valid for current input)
> Pros: Less overhead than controls. Could support multiple sensors in one
> camera if such a beast exists.
> Cons: Would require polling to support the case of a camera being turned
> toward / away from the user while streaming. Can't easily identify drivers
> that don't support it.
>
> The interest in detecting if a driver provides this informnation is to allow
> libv4l to know when it should use the driver provided information and when it
> should use its internal table (which needs to be retained for backward
> compatibility). With no detection capability the driver provided info should
> be ignored for USB IDs in the built in table.
>
> Thoughts please
>
> Adam Baker
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
========================================================
DongSoo(Nathaniel), Kim
Engineer
Mobile S/W Platform Lab. S/W centre
Telecommunication R&D Centre
Samsung Electronics CO., LTD.
e-mail : dongsoo.kim@gmail.com
          dongsoo45.kim@samsung.com
========================================================

Re: [RFC] How to pass camera Orientation to userspace

[Hans de Goede]

DongSoo(Nathaniel) Kim wrote:
> Hello Adam,
> 
> I've been thinking exactly the same issue not usb but SoC based camera.
> I have no idea about how usb cameras work but I am quite curious about
> is it really possible to make proper orientation with only querying
> camera driver.
> Because in case of SoC based camera device, many of camera ISPs are
> supporting VFLIP, HFLIP register on their own, and we can read current
> orientation by reading those registers.
> 
> But the problem is ISP's registers are set as not flipped at all but
> it physically mounted upside down, because the H/W  vendor has packed
> the camera module upside down. (it sounds ridiculous but happens
> sometimes)

That happens a lot with webcams too. Given that these SoC systems will come 
with some board specific config anyways, all that is needed is to pass some 
boardconfig in to the camera driver (through platform data for example) which 
tells the camera driver that on this board the sensor is mounted upside down.

> So in that case when we query orientation of camera, it returns not
> flipped vertically or horizontally at all but actually it turns out to
> be upside down. Actually we are setting camera device to be flipped
> for default in that case.

Ack, but the right thing to do is not to set the vflip and hflip video4linux2 
controls on by default, but to invert their meaning, so when the sensor is 
upside down, the hflip and vflip controls as seen by the application through 
the v4l2 API will report not flipping, but the hwcontrols will actually be set 
to flipping, and when an app enables flipping at the v4l2 API level it will 
actually gets disables at the HW level, this way the upside downness is 100% 
hidden from userspace. So your problem does not need any of the new API we are 
working on. The new API is for when the hardware cannot flip and we need to 
tell userspace to correct for this in software.

Regards,

Hans

Re: [RFC] How to pass camera Orientation to userspace

[Dongsoo Kim]

Hello Hans.

I went too far.
Just forgot what we can do through porting job.
You are right. That should be board specific item.
Cheers,

Nate

2009. 02. 18, 오후 11:36, Hans de Goede 작성:

>
>
> DongSoo(Nathaniel) Kim wrote:
>> Hello Adam,
>> I've been thinking exactly the same issue not usb but SoC based  
>> camera.
>> I have no idea about how usb cameras work but I am quite curious  
>> about
>> is it really possible to make proper orientation with only querying
>> camera driver.
>> Because in case of SoC based camera device, many of camera ISPs are
>> supporting VFLIP, HFLIP register on their own, and we can read  
>> current
>> orientation by reading those registers.
>> But the problem is ISP's registers are set as not flipped at all but
>> it physically mounted upside down, because the H/W  vendor has packed
>> the camera module upside down. (it sounds ridiculous but happens
>> sometimes)
>
> That happens a lot with webcams too. Given that these SoC systems  
> will come with some board specific config anyways, all that is  
> needed is to pass some boardconfig in to the camera driver (through  
> platform data for example) which tells the camera driver that on  
> this board the sensor is mounted upside down.
>
>> So in that case when we query orientation of camera, it returns not
>> flipped vertically or horizontally at all but actually it turns out  
>> to
>> be upside down. Actually we are setting camera device to be flipped
>> for default in that case.
>
> Ack, but the right thing to do is not to set the vflip and hflip  
> video4linux2 controls on by default, but to invert their meaning, so  
> when the sensor is upside down, the hflip and vflip controls as seen  
> by the application through the v4l2 API will report not flipping,  
> but the hwcontrols will actually be set to flipping, and when an app  
> enables flipping at the v4l2 API level it will actually gets  
> disables at the HW level, this way the upside downness is 100%  
> hidden from userspace. So your problem does not need any of the new  
> API we are working on. The new API is for when the hardware cannot  
> flip and we need to tell userspace to correct for this in software.
>
> Regards,
>
> Hans

Re: [RFC] How to pass camera Orientation to userspace

[Dongsoo Kim]

Hello Hans.

I went too far.
Just forgot what we can do through porting job.
You are right. That should be board specific item.
Cheers,

Nate

2009. 02. 18, 오후 11:36, Hans de Goede 작성:

>
>
> DongSoo(Nathaniel) Kim wrote:
>> Hello Adam,
>> I've been thinking exactly the same issue not usb but SoC based  
>> camera.
>> I have no idea about how usb cameras work but I am quite curious  
>> about
>> is it really possible to make proper orientation with only querying
>> camera driver.
>> Because in case of SoC based camera device, many of camera ISPs are
>> supporting VFLIP, HFLIP register on their own, and we can read  
>> current
>> orientation by reading those registers.
>> But the problem is ISP's registers are set as not flipped at all but
>> it physically mounted upside down, because the H/W  vendor has packed
>> the camera module upside down. (it sounds ridiculous but happens
>> sometimes)
>
> That happens a lot with webcams too. Given that these SoC systems  
> will come with some board specific config anyways, all that is  
> needed is to pass some boardconfig in to the camera driver (through  
> platform data for example) which tells the camera driver that on  
> this board the sensor is mounted upside down.
>
>> So in that case when we query orientation of camera, it returns not
>> flipped vertically or horizontally at all but actually it turns out  
>> to
>> be upside down. Actually we are setting camera device to be flipped
>> for default in that case.
>
> Ack, but the right thing to do is not to set the vflip and hflip  
> video4linux2 controls on by default, but to invert their meaning, so  
> when the sensor is upside down, the hflip and vflip controls as seen  
> by the application through the v4l2 API will report not flipping,  
> but the hwcontrols will actually be set to flipping, and when an app  
> enables flipping at the v4l2 API level it will actually gets  
> disables at the HW level, this way the upside downness is 100%  
> hidden from userspace. So your problem does not need any of the new  
> API we are working on. The new API is for when the hardware cannot  
> flip and we need to tell userspace to correct for this in software.
>
> Regards,
>
> Hans



--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] How to pass camera Orientation to userspace

[Hans Verkuil]

Hi Adam,

Sorry for the late reply, it's been very busy.

On Wednesday 18 February 2009 01:30:52 Adam Baker wrote:
> (linux-omap included in distribution as lots of omap systems include
> cameras so this could be relevant there.)
>
> Background
>
> A number of the webcams now supported by v4l have sensors that are
> mounted upside down. Up to now this has been handled by having a table in
> libv4l of the USB IDs of affected cameras. This approach however fails to
> address two known cases (and probably more as yet unknown ones) where the
> USB ID is insufficient to determine the sensor orientation.
>
> In one of those cases (SQ-905) USB commands must be issued to the camera 
> at probe time) to determine what sensor is fitted and in the other case
> (Genesys gl860) the camera can be pointed towards or away from the user
> and it swaps orientation when it is changed.
>
> It is possible that there are cameras that can use gravity sensors or
> similar to report how they are being held but such user driven
> orientation which may be intended for creative effect should probably be
> separated from this hardware related issue.

Yes, I strongly agree with this.

> Most if not all of the cameras affected by this problem produce video
> formats that are not widely supported by applications. They are therefore
> all likely to be normally used in conjunction with libv4l to perform
> format conversion and so it makes sense to retain the actual flipping
> operation in libv4l. libv4l provides a capability via LD_PRELOAD to
> attach itself to unmodified binaries.
>
> It is likely that whatever solution is chosen there will end up being
> cases that have to be handled in libv4l as external information (such as
> laptop model ID) is the only mechanism to distinguish otherwise identical
> cameras.
>
> So far libv4l only supports the case of data needing a 180 degree
> rotation but there is known to exist hardware which requires a VFLIP and
> it is possible some hardware requires just HFLIP so all of those cases
> should be supported.

While libv4l will be the main user of this information, it should not matter 
for the chosen API whether it is libv4l or another application, possibly a 
custom application running on an embedded system.

> There have been a number of inconclusive discussions on this subject on
> the v4l / linux-media mailing lists over the last few months offering
> many options for how to pass this information across which I will list
> below and hopefully a preferred solution can be selected from them
>
> 1) Reuse the existing HFLIP and VFLIP controls, marking them as read-only
> Pros : No change needed to videodev2.h
> Cons: It is confusing to have controls that have a subtly different
> meaning if they are read only. Existing apps that support those controls
> might get confused. Would require polling to support the case of a camera
> being turned toward / away from the user while streaming.
>
> 2) Introduce a new orientation control (possibly in a new
> CAMERA_PROPERTIES class)
> Pros: libv4l can easily tell if the driver supports the control.
> Cons: It is really a property, not a control so calling it a control is
> wrong. Controls add lots of overhead in terms of driver code. Would
> require polling to support the case of a camera being turned toward /
> away from the user while streaming.
>
> 3) Use an extra couple of bits in V4L2_BUF_FLAGS
> Pros: Simple to implement. Can change per frame without needing polling.
> Cons: Doesn't work for non libv4l apps that try to use the read()
> interface. Can't easily identify drivers that don't support it (does 0
> mean not rotated or just not implemented). Can only be read when
> streaming (does that matter?)

I think that matters, yes.

> 4) Use some reserved bits from the v4l2_capability structure
> Pros: Less overhead than controls.
> Cons: Would require polling to support the case of a camera being turned
> toward / away from the user while streaming. Can't easily identify
> drivers that don't support it.
>
> 5) Use some reserved bits from the v4l2_input structure (or possibly the
> status word but that is normally only valid for current input)
> Pros: Less overhead than controls. Could support multiple sensors in one
> camera if such a beast exists.

What does exist is devices with a video input (e.g. composite) and a camera 
input: each input will have different flags. Since these vflip/hflip 
properties do not change they can be enumerated in advance and you know 
what each input supports.

> Cons: Would require polling to support the case of a camera being turned
> toward / away from the user while streaming.

Polling applies only to the bits that tell the orientation of the camera. 
See below for a discussion of this.

> Can't easily identify drivers that don't support it.

Not too difficult to add through the use of a capability bit. Either in 
v4l2_input or (perhaps) v4l2_capability.

Another Pro is that this approach will also work for v4l2_output in the case 
of, say, rotated LCD displays. Using camera orientation bits in v4l2_buffer 
while capturing will work, but using similar bits for output will fail 
since the data is going in the wrong direction.

> The interest in detecting if a driver provides this informnation is to
> allow libv4l to know when it should use the driver provided information
> and when it should use its internal table (which needs to be retained for
> backward compatibility). With no detection capability the driver provided
> info should be ignored for USB IDs in the built in table.
>
> Thoughts please

Is polling bad in this case? It is not something that needs immediate 
attention IMHO. The overhead for checking once every X seconds is quite 
low. Furthermore, it is only needed on devices that cannot do v/hflipping 
in hardware.

An alternative is to put some effort in a proper event interface. There is 
one implemented in include/linux/dvb/video.h and used by ivtv for video 
decoding. The idea is that the application registers events it wants to 
receive, and whenever such an event arrives the select() call will exit 
with a high-prio event (exception). The application then checks what 
happened.

The video.h implementation is pretty crappy, but we can make something more 
useful. Alternatively, I believe there are other event mechanisms as well 
in the kernel. I know some work was done on that, but I don't know what the 
end result was.

Note that this does not apply to sensor mount information. I think that 
v4l2_input is the perfect place for that. It is after all an input-specific 
property and static property, and it maps 1-on-1 to the output case as 
well, should we need it there as well.

Regards,

	Hans

-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG

Re: [RFC] How to pass camera Orientation to userspace

[Hans de Goede]

Hans Verkuil wrote:
> Hi Adam,
> 
> Sorry for the late reply, it's been very busy.
> 

Same here :)

> On Wednesday 18 February 2009 01:30:52 Adam Baker wrote:
>> (linux-omap included in distribution as lots of omap systems include
>> cameras so this could be relevant there.)
>>
>> Background
>>
>> A number of the webcams now supported by v4l have sensors that are
>> mounted upside down. Up to now this has been handled by having a table in
>> libv4l of the USB IDs of affected cameras. This approach however fails to
>> address two known cases (and probably more as yet unknown ones) where the
>> USB ID is insufficient to determine the sensor orientation.
>>
>> In one of those cases (SQ-905) USB commands must be issued to the camera 
>> at probe time) to determine what sensor is fitted and in the other case
>> (Genesys gl860) the camera can be pointed towards or away from the user
>> and it swaps orientation when it is changed.
>>
>> It is possible that there are cameras that can use gravity sensors or
>> similar to report how they are being held but such user driven
>> orientation which may be intended for creative effect should probably be
>> separated from this hardware related issue.
> 
> Yes, I strongly agree with this.
> 

+1

>> 3) Use an extra couple of bits in V4L2_BUF_FLAGS
>> Pros: Simple to implement. Can change per frame without needing polling.
>> Cons: Doesn't work for non libv4l apps that try to use the read()
>> interface. Can't easily identify drivers that don't support it (does 0
>> mean not rotated or just not implemented). Can only be read when
>> streaming (does that matter?)
> 
> I think that matters, yes.
> 

I too can see it being usefull to get the orientation when not streaming.

>> 4) Use some reserved bits from the v4l2_capability structure
>> Pros: Less overhead than controls.
>> Cons: Would require polling to support the case of a camera being turned
>> toward / away from the user while streaming. Can't easily identify
>> drivers that don't support it.
>>
>> 5) Use some reserved bits from the v4l2_input structure (or possibly the
>> status word but that is normally only valid for current input)
>> Pros: Less overhead than controls. Could support multiple sensors in one
>> camera if such a beast exists.
> 
> What does exist is devices with a video input (e.g. composite) and a camera 
> input: each input will have different flags. Since these vflip/hflip 
> properties do not change they can be enumerated in advance and you know 
> what each input supports.
> 
>> Cons: Would require polling to support the case of a camera being turned
>> toward / away from the user while streaming.
> 
> Polling applies only to the bits that tell the orientation of the camera. 
> See below for a discussion of this.
> 

Didn't we agree to separate orietation (as in sensor mounted upside down) and 
the user being able to rotate the camera (which i believe we called pivotting).

For the orientation case, which is the case with the sq905X, we do not need to 
poll, as for pivotting, here is my proposal:

Have both orientation and pivotting flags in the input flags. The pivotting 
flags will indicate the pivotting state at the moment of doing the ioctl (which 
may change later), so these indeed may need to be polled, unlike the 
orientation flags which will never change.

>> Can't easily identify drivers that don't support it.
> 
> Not too difficult to add through the use of a capability bit. Either in 
> v4l2_input or (perhaps) v4l2_capability.
> 

+1

> Another Pro is that this approach will also work for v4l2_output in the case 
> of, say, rotated LCD displays. Using camera orientation bits in v4l2_buffer 
> while capturing will work, but using similar bits for output will fail 
> since the data is going in the wrong direction.
> 
>> The interest in detecting if a driver provides this informnation is to
>> allow libv4l to know when it should use the driver provided information
>> and when it should use its internal table (which needs to be retained for
>> backward compatibility). With no detection capability the driver provided
>> info should be ignored for USB IDs in the built in table.
>>
>> Thoughts please
> 
> Is polling bad in this case? It is not something that needs immediate 
> attention IMHO.  The overhead for checking once every X seconds is quite
 > low.

Agreed, but it still is something which should be avoided if possible, 
implementing polling also means adding a lot of IMHO unneeded code on the 
userspace side.

I would prefer to make the "realtime" pivotting state available to the app by 
adding flags containing the pivotting state at frame capture to the v4l2_buf flags.

But if people dislike this, libv4l can simple poll the input now and then.

> Furthermore, it is only needed on devices that cannot do v/hflipping 
> in hardware.
> 

Erm, no, since we decided we want to differentiate between sensor orientation 
and camera pivotting, I think the driver should not automagically do 
v/hflipping in the pivot case, this should be left up to userspace. Maybe the 
user actually wants to have an upside down picture ?

> An alternative is to put some effort in a proper event interface. There is 
> one implemented in include/linux/dvb/video.h and used by ivtv for video 
> decoding. The idea is that the application registers events it wants to 
> receive, and whenever such an event arrives the select() call will exit 
> with a high-prio event (exception). The application then checks what 
> happened.
> 

No not another event interface please. Once could argue that we should export 
the pivotting info through the generic linux input system, but not a v4l 
specific event interface please. Actually I think making the pivotting sensor 
available through the generic input system in addition to making it available 
through input flags would be nice to have. Just like we need to make all those 
take a picture now buttons on webcams available through the generic input system.

Regards,

Hans

Re: [RFC] How to pass camera Orientation to userspace

[Hans Verkuil]

On Sunday 22 February 2009 12:17:58 Hans de Goede wrote:
> Hans Verkuil wrote:
> >> Cons: Would require polling to support the case of a camera being
> >> turned toward / away from the user while streaming.
> >
> > Polling applies only to the bits that tell the orientation of the
> > camera. See below for a discussion of this.
>
> Didn't we agree to separate orietation (as in sensor mounted upside down)
> and the user being able to rotate the camera (which i believe we called
> pivotting).

Yes, I wrote this more as a clarification since I thought that the original 
text didn't make this distinction clearly enough.

> For the orientation case, which is the case with the sq905X, we do not
> need to poll, as for pivotting, here is my proposal:
>
> Have both orientation and pivotting flags in the input flags. The
> pivotting flags will indicate the pivotting state at the moment of doing
> the ioctl (which may change later), so these indeed may need to be
> polled, unlike the orientation flags which will never change.

Ack.

I propose to take one of the reserved fields from v4l2_input and either turn 
it into two __u16 fields (capabilities and flags) or one __u32 field 
(flags). I'm not sure whether the alignment of two __u16 fields will be 
done correctly on 32 and 64 bit systems since v4l2_input cannot change in 
size. Something to test first.

> >> Can't easily identify drivers that don't support it.
> >
> > Not too difficult to add through the use of a capability bit. Either in
> > v4l2_input or (perhaps) v4l2_capability.
>
> +1
>
> > Another Pro is that this approach will also work for v4l2_output in the
> > case of, say, rotated LCD displays. Using camera orientation bits in
> > v4l2_buffer while capturing will work, but using similar bits for
> > output will fail since the data is going in the wrong direction.
> >
> >> The interest in detecting if a driver provides this informnation is to
> >> allow libv4l to know when it should use the driver provided
> >> information and when it should use its internal table (which needs to
> >> be retained for backward compatibility). With no detection capability
> >> the driver provided info should be ignored for USB IDs in the built in
> >> table.
> >>
> >> Thoughts please
> >
> > Is polling bad in this case? It is not something that needs immediate
> > attention IMHO.  The overhead for checking once every X seconds is
> > quite
> >
>  > low.
>
> Agreed, but it still is something which should be avoided if possible,
> implementing polling also means adding a lot of IMHO unneeded code on the
> userspace side.
>
> I would prefer to make the "realtime" pivotting state available to the
> app by adding flags containing the pivotting state at frame capture to
> the v4l2_buf flags.
>
> But if people dislike this, libv4l can simple poll the input now and
> then.

I think this should be prototyped. Compare polling vs. putting the state in 
the v4l2_buf flags and see if that makes a lot of difference in the user 
experience compared to polling once a second. If it clearly improves things 
for the user, then I have no objections to adding bits to v4l2_buf.

> > Furthermore, it is only needed on devices that cannot do v/hflipping
> > in hardware.
>
> Erm, no, since we decided we want to differentiate between sensor
> orientation and camera pivotting, I think the driver should not
> automagically do v/hflipping in the pivot case, this should be left up to
> userspace. Maybe the user actually wants to have an upside down picture ?

Sorry, you're right. I'd forgotten that.

> > An alternative is to put some effort in a proper event interface. There
> > is one implemented in include/linux/dvb/video.h and used by ivtv for
> > video decoding. The idea is that the application registers events it
> > wants to receive, and whenever such an event arrives the select() call
> > will exit with a high-prio event (exception). The application then
> > checks what happened.
>
> No not another event interface please. Once could argue that we should
> export the pivotting info through the generic linux input system, but not
> a v4l specific event interface please. Actually I think making the
> pivotting sensor available through the generic input system in addition
> to making it available through input flags would be nice to have. Just
> like we need to make all those take a picture now buttons on webcams
> available through the generic input system.

Agreed. I'm not up to date when it comes to the state of event interfaces in 
linux, and this is a solution for the longer term anyway. I guess someone 
needs to pick this up and research it to see what works best.

Regards,

	Hans

-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG

Re: [RFC] How to pass camera Orientation to userspace

[Hans de Goede]

Hans Verkuil wrote:
> On Sunday 22 February 2009 12:17:58 Hans de Goede wrote:

<snipped a large part we agree upon (hurray!) >

>> Agreed, but it still is something which should be avoided if possible,
>> implementing polling also means adding a lot of IMHO unneeded code on the
>> userspace side.
>>
>> I would prefer to make the "realtime" pivotting state available to the
>> app by adding flags containing the pivotting state at frame capture to
>> the v4l2_buf flags.
>>
>> But if people dislike this, libv4l can simple poll the input now and
>> then.
> 
> I think this should be prototyped. Compare polling vs. putting the state in 
> the v4l2_buf flags and see if that makes a lot of difference in the user 
> experience compared to polling once a second. If it clearly improves things 
> for the user, then I have no objections to adding bits to v4l2_buf.
> 

This is were the difficulties start, doing this 1 per second requires making 
clock() calls and comparing timestamps (always tricky to get right). We could 
do this once every X frames, but framerates can vary wildly with webcams.

My biggest concern is not the user experience, but the fact that needing to 
poll uglyfies the userspace code. However thinking about this more, I guess we 
could just poll every frame. That makes the polling code simple and should give 
a good user experience.

<snip of more things we agree up on>

Regards,

Hans

Re: [RFC] How to pass camera Orientation to userspace

[kilgota]

On Sun, 22 Feb 2009, Hans de Goede wrote:

>
>
> Hans Verkuil wrote:
>> Hi Adam,
>> 
>> Sorry for the late reply, it's been very busy.
>> 
>
> Same here :)
>
>> On Wednesday 18 February 2009 01:30:52 Adam Baker wrote:
>>> (linux-omap included in distribution as lots of omap systems include
>>> cameras so this could be relevant there.)
>>> 
>>> Background
>>> 
>>> A number of the webcams now supported by v4l have sensors that are
>>> mounted upside down. Up to now this has been handled by having a table in
>>> libv4l of the USB IDs of affected cameras. This approach however fails to
>>> address two known cases (and probably more as yet unknown ones) where the
>>> USB ID is insufficient to determine the sensor orientation.
>>> 
>>> In one of those cases (SQ-905) USB commands must be issued to the camera 
>>> at probe time) to determine what sensor is fitted and in the other case
>>> (Genesys gl860) the camera can be pointed towards or away from the user
>>> and it swaps orientation when it is changed.
>>> 
>>> It is possible that there are cameras that can use gravity sensors or
>>> similar to report how they are being held but such user driven
>>> orientation which may be intended for creative effect should probably be
>>> separated from this hardware related issue.
>> 
>> Yes, I strongly agree with this.
>> 
>
> +1
>
>>> 3) Use an extra couple of bits in V4L2_BUF_FLAGS
>>> Pros: Simple to implement. Can change per frame without needing polling.
>>> Cons: Doesn't work for non libv4l apps that try to use the read()
>>> interface. Can't easily identify drivers that don't support it (does 0
>>> mean not rotated or just not implemented). Can only be read when
>>> streaming (does that matter?)
>> 
>> I think that matters, yes.
>> 
>
> I too can see it being usefull to get the orientation when not streaming.
>
>>> 4) Use some reserved bits from the v4l2_capability structure
>>> Pros: Less overhead than controls.
>>> Cons: Would require polling to support the case of a camera being turned
>>> toward / away from the user while streaming. Can't easily identify
>>> drivers that don't support it.
>>> 
>>> 5) Use some reserved bits from the v4l2_input structure (or possibly the
>>> status word but that is normally only valid for current input)
>>> Pros: Less overhead than controls. Could support multiple sensors in one
>>> camera if such a beast exists.
>> 
>> What does exist is devices with a video input (e.g. composite) and a camera 
>> input: each input will have different flags. Since these vflip/hflip 
>> properties do not change they can be enumerated in advance and you know 
>> what each input supports.
>> 
>>> Cons: Would require polling to support the case of a camera being turned
>>> toward / away from the user while streaming.
>> 
>> Polling applies only to the bits that tell the orientation of the camera. 
>> See below for a discussion of this.
>> 
>
> Didn't we agree to separate orietation (as in sensor mounted upside down) and 
> the user being able to rotate the camera (which i believe we called 
> pivotting).
>
> For the orientation case, which is the case with the sq905X, we do not need 
> to poll, as for pivotting, here is my proposal:
>
> Have both orientation and pivotting flags in the input flags. The pivotting 
> flags will indicate the pivotting state at the moment of doing the ioctl 
> (which may change later), so these indeed may need to be polled, unlike the 
> orientation flags which will never change.
>
>>> Can't easily identify drivers that don't support it.
>> 
>> Not too difficult to add through the use of a capability bit. Either in 
>> v4l2_input or (perhaps) v4l2_capability.
>> 
>
> +1
>
>> Another Pro is that this approach will also work for v4l2_output in the 
>> case of, say, rotated LCD displays. Using camera orientation bits in 
>> v4l2_buffer while capturing will work, but using similar bits for output 
>> will fail since the data is going in the wrong direction.
>> 
>>> The interest in detecting if a driver provides this informnation is to
>>> allow libv4l to know when it should use the driver provided information
>>> and when it should use its internal table (which needs to be retained for
>>> backward compatibility). With no detection capability the driver provided
>>> info should be ignored for USB IDs in the built in table.
>>> 
>>> Thoughts please
>> 
>> Is polling bad in this case? It is not something that needs immediate 
>> attention IMHO.  The overhead for checking once every X seconds is quite
>> low.
>
> Agreed, but it still is something which should be avoided if possible, 
> implementing polling also means adding a lot of IMHO unneeded code on the 
> userspace side.
>
> I would prefer to make the "realtime" pivotting state available to the app by 
> adding flags containing the pivotting state at frame capture to the v4l2_buf 
> flags.
>
> But if people dislike this, libv4l can simple poll the input now and then.
>
>> Furthermore, it is only needed on devices that cannot do v/hflipping in 
>> hardware.
>> 
>
> Erm, no, since we decided we want to differentiate between sensor orientation 
> and camera pivotting, I think the driver should not automagically do 
> v/hflipping in the pivot case, this should be left up to userspace. Maybe the 
> user actually wants to have an upside down picture ?
>
>> An alternative is to put some effort in a proper event interface. There is 
>> one implemented in include/linux/dvb/video.h and used by ivtv for video 
>> decoding. The idea is that the application registers events it wants to 
>> receive, and whenever such an event arrives the select() call will exit 
>> with a high-prio event (exception). The application then checks what 
>> happened.
>> 
>
> No not another event interface please. Once could argue that we should export 
> the pivotting info through the generic linux input system, but not a v4l 
> specific event interface please. Actually I think making the pivotting sensor 
> available through the generic input system in addition to making it available 
> through input flags would be nice to have. Just like we need to make all 
> those take a picture now buttons on webcams available through the generic 
> input system.
>
> Regards,
>
> Hans
>

Hans and Adam,

I am not sure how it fits into the above discussion, but perhaps it is 
relevant to point out that flags can be toggled. Here is what I mean:

Suppose that we have two flags 01 and 10 (i.e. 2), and 01 signifies VFLIP 
and 10 signifies HFLIP.

Then for an "ordinary" camera in ordinary position 
these are initialized as 00. If the "ordinary" camera is turned in some 
funny way (and it is possible to know that) then one or both of these 
flags gets turned off.

But if it is a "funny" camera like some of the SQ905s the initial values 
are 1 and 1, because the sensor is in fact mounted upside down. Now, 
suppose that there is some camera in the future which, just like this, has 
the sensor upside down, and suppose that said hypothetical camera also has 
the ability to "know" that it has been turned over so what was upside down 
is now right side up. Well, all  that one has to do is to flip the two 
bits from whatever they were to have instead the opposite values!

Observe that this would take care of the orientation problem both for 
cameras which had the sensor mounted right in the first place, and for 
cameras which had the sensor mounted wrong in the first place. Just use 
the same two bits to describe the sensor orientation, and if there is any 
reason (based upon some ability to know that the camera orientation is now 
different) that the orientation should change, then just flip the bits as 
appropriate.

Then it would be the job of the support module to provide proper initial 
values only for these bits, and everything else could be done later on, in 
userspace.

Theodore Kilgore

Re: [RFC] How to pass camera Orientation to userspace

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 
<snip>


> Hans and Adam,
> 
> I am not sure how it fits into the above discussion, but perhaps it is 
> relevant to point out that flags can be toggled. Here is what I mean:
> 
> Suppose that we have two flags 01 and 10 (i.e. 2), and 01 signifies 
> VFLIP and 10 signifies HFLIP.
> 
> Then for an "ordinary" camera in ordinary position these are initialized 
> as 00. If the "ordinary" camera is turned in some funny way (and it is 
> possible to know that) then one or both of these flags gets turned off.
> 
> But if it is a "funny" camera like some of the SQ905s the initial values 
> are 1 and 1, because the sensor is in fact mounted upside down. Now, 
> suppose that there is some camera in the future which, just like this, 
> has the sensor upside down, and suppose that said hypothetical camera 
> also has the ability to "know" that it has been turned over so what was 
> upside down is now right side up. Well, all  that one has to do is to 
> flip the two bits from whatever they were to have instead the opposite 
> values!
> 
> Observe that this would take care of the orientation problem both for 
> cameras which had the sensor mounted right in the first place, and for 
> cameras which had the sensor mounted wrong in the first place. Just use 
> the same two bits to describe the sensor orientation, and if there is 
> any reason (based upon some ability to know that the camera orientation 
> is now different) that the orientation should change, then just flip the 
> bits as appropriate.
> 
> Then it would be the job of the support module to provide proper initial 
> values only for these bits, and everything else could be done later on, 
> in userspace.
> 
> Theodore Kilgore

Theodore,

We want to be able to differentiate between a cam which has its sensor mounted 
upside down, and a cam which can be pivotted and happens to be upside down at 
the moment, in case of any upside down mounted sensor, we will always want to 
compentsate, in case of a pivotting camera wether we compensate or not could be 
a user preference.

So in you example of an upside down mounted sensor in a pivotting encasing and 
the encasing is pivotted 180 degrees we would have the hflip and vflip bits set 
for sensor orientation and we would have the pivotted 180 degrees bit set. If 
the user has choosen to compensate for pivotting the default, we would do 
nothing. But it is important to be able to differentiate between the 2.

Regards,

Hans

Re: [RFC] How to pass camera Orientation to userspace

[kilgota]

On Sun, 22 Feb 2009, Hans de Goede wrote:

>
>
> kilgota@banach.math.auburn.edu wrote:
>> 
> <snip>
>
>
>> Hans and Adam,
>> 
>> I am not sure how it fits into the above discussion, but perhaps it is 
>> relevant to point out that flags can be toggled. Here is what I mean:
>> 
>> Suppose that we have two flags 01 and 10 (i.e. 2), and 01 signifies VFLIP 
>> and 10 signifies HFLIP.
>> 
>> Then for an "ordinary" camera in ordinary position these are initialized as 
>> 00. If the "ordinary" camera is turned in some funny way (and it is 
>> possible to know that) then one or both of these flags gets turned off.
>> 
>> But if it is a "funny" camera like some of the SQ905s the initial values 
>> are 1 and 1, because the sensor is in fact mounted upside down. Now, 
>> suppose that there is some camera in the future which, just like this, has 
>> the sensor upside down, and suppose that said hypothetical camera also has 
>> the ability to "know" that it has been turned over so what was upside down 
>> is now right side up. Well, all  that one has to do is to flip the two bits 
>> from whatever they were to have instead the opposite values!
>> 
>> Observe that this would take care of the orientation problem both for 
>> cameras which had the sensor mounted right in the first place, and for 
>> cameras which had the sensor mounted wrong in the first place. Just use the 
>> same two bits to describe the sensor orientation, and if there is any 
>> reason (based upon some ability to know that the camera orientation is now 
>> different) that the orientation should change, then just flip the bits as 
>> appropriate.
>> 
>> Then it would be the job of the support module to provide proper initial 
>> values only for these bits, and everything else could be done later on, in 
>> userspace.
>> 
>> Theodore Kilgore
>
> Theodore,
>
> We want to be able to differentiate between a cam which has its sensor 
> mounted upside down, and a cam which can be pivotted and happens to be upside 
> down at the moment, in case of any upside down mounted sensor, we will always 
> want to compentsate, in case of a pivotting camera wether we compensate or 
> not could be a user preference.
>
> So in you example of an upside down mounted sensor in a pivotting encasing 
> and the encasing is pivotted 180 degrees we would have the hflip and vflip 
> bits set for sensor orientation and we would have the pivotted 180 degrees 
> bit set. If the user has choosen to compensate for pivotting the default, we 
> would do nothing. But it is important to be able to differentiate between the 
> 2.

Hans,

I am not sure if we are talking past each other, or what. But I was 
pointing out that the initial values of two bits can indicate the 
"default" orientation of the sensor, and this can be done permanently in 
the module, which transmits the initial setting of those two bits to 
anything up the line which is interested or curious to know those initial 
values. The information in those two bits will definitely tell whether the 
sensor is mounted upside down in the camera. For example, if it is mounted 
upside down, then they are both set in the module to "on" and exported 
therefrom. But if the sensor is mounted correctly, then both of them are 
set to "off" and similarly exported.

Now if any application for any reason (such as "knowing" that the 
camera is upside down or is pointing in the opposite direction, or into a 
mirror) wants to change the defaults, all it has to do is to toggle the 
bits.

But, hmmm. Perhaps there is the question about how the app "knows" that 
the camera is upside down or is pointed in another direction. If the 
camera has a gyroscope inside, for example, then it could be the camera 
which needs to tell to the app about the current orientation, or else the 
app would not have any way to know ... Is this the problem, then? For that 
kind of thing, one might need more than two bits in order to pass the 
needed information.

Incidentally, I obviously agree about the need to pass the initial sensor 
orientation. So anything I say about this should be taken in that context.

Theodore Kilgore

Re: [RFC] How to pass camera Orientation to userspace

[Adam Baker]

On Sunday 22 February 2009, Hans de Goede wrote:
> We want to be able to differentiate between a cam which has its sensor
> mounted upside down, and a cam which can be pivotted and happens to be
> upside down at the moment, in case of any upside down mounted sensor, we
> will always want to compentsate, in case of a pivotting camera wether we
> compensate or not could be a user preference.


If we take Olivier Lorin's gl-860 case though, how do we define what is the 
normal orientation and what is pivoted, it is likely we'd just decide the 
direction where the sensor output is the right way up is normal and the other 
is pivoted and then what info have you got from having multiple flags.

In order to explain what I mean it is probably best to refer to rotations in 
terms of pitch, yaw and roll (as per the definitions at 
http://en.wikipedia.org/wiki/Flight_dynamics) where the forward direction is 
the shooting direction.

When still cameras are fitted with gravity sensors they are normally set up 
with the intention of measuring roll and will often get confused by 90 
degrees of pitch. If a laptop is fitted with a camera that can either record 
the user or the view looking away from the user then the camera needs to be 
able to either pitch or yaw but not roll. If the camera yaws then no 
correction is needed but if it pitches then the resulting image needs 
rotating to be the correct way up (and in that scenario it is improbable that 
the user doesn't want the correction applied). Other than the fact that one 
needs correcting and the other doesn't these options appear identical to the 
user and so manufacturers provide one or the other but not both.

If a video camera had a roll sensor (or even, as a believe some specialist 
tripods can manage, a full set of roll, pitch and yaw measurements) then a 
substantially different mechanism is needed to provide access to that data 
but in the absence of anyone having access to such equipment I don't think we 
can design the interface now.

Adam

Re: [RFC] How to pass camera Orientation to userspace

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 
> 
> On Sun, 22 Feb 2009, Hans de Goede wrote:
> 
>>
>>
>> kilgota@banach.math.auburn.edu wrote:
>>>
>> <snip>
>>
>>
>>> Hans and Adam,
>>>
>>> I am not sure how it fits into the above discussion, but perhaps it 
>>> is relevant to point out that flags can be toggled. Here is what I mean:
>>>
>>> Suppose that we have two flags 01 and 10 (i.e. 2), and 01 signifies 
>>> VFLIP and 10 signifies HFLIP.
>>>
>>> Then for an "ordinary" camera in ordinary position these are 
>>> initialized as 00. If the "ordinary" camera is turned in some funny 
>>> way (and it is possible to know that) then one or both of these flags 
>>> gets turned off.
>>>
>>> But if it is a "funny" camera like some of the SQ905s the initial 
>>> values are 1 and 1, because the sensor is in fact mounted upside 
>>> down. Now, suppose that there is some camera in the future which, 
>>> just like this, has the sensor upside down, and suppose that said 
>>> hypothetical camera also has the ability to "know" that it has been 
>>> turned over so what was upside down is now right side up. Well, all  
>>> that one has to do is to flip the two bits from whatever they were to 
>>> have instead the opposite values!
>>>
>>> Observe that this would take care of the orientation problem both for 
>>> cameras which had the sensor mounted right in the first place, and 
>>> for cameras which had the sensor mounted wrong in the first place. 
>>> Just use the same two bits to describe the sensor orientation, and if 
>>> there is any reason (based upon some ability to know that the camera 
>>> orientation is now different) that the orientation should change, 
>>> then just flip the bits as appropriate.
>>>
>>> Then it would be the job of the support module to provide proper 
>>> initial values only for these bits, and everything else could be done 
>>> later on, in userspace.
>>>
>>> Theodore Kilgore
>>
>> Theodore,
>>
>> We want to be able to differentiate between a cam which has its sensor 
>> mounted upside down, and a cam which can be pivotted and happens to be 
>> upside down at the moment, in case of any upside down mounted sensor, 
>> we will always want to compentsate, in case of a pivotting camera 
>> wether we compensate or not could be a user preference.
>>
>> So in you example of an upside down mounted sensor in a pivotting 
>> encasing and the encasing is pivotted 180 degrees we would have the 
>> hflip and vflip bits set for sensor orientation and we would have the 
>> pivotted 180 degrees bit set. If the user has choosen to compensate 
>> for pivotting the default, we would do nothing. But it is important to 
>> be able to differentiate between the 2.
> 
> Hans,
> 
> I am not sure if we are talking past each other, or what. But I was 
> pointing out that the initial values of two bits can indicate the 
> "default" orientation of the sensor, and this can be done permanently in 
> the module, which transmits the initial setting of those two bits to 
> anything up the line which is interested or curious to know those 
> initial values. The information in those two bits will definitely tell 
> whether the sensor is mounted upside down in the camera. For example, if 
> it is mounted upside down, then they are both set in the module to "on" 
> and exported therefrom. But if the sensor is mounted correctly, then 
> both of them are set to "off" and similarly exported.
> 
> Now if any application for any reason (such as "knowing" that the camera 
> is upside down or is pointing in the opposite direction, or into a 
> mirror) wants to change the defaults, all it has to do is to toggle the 
> bits.
> 
> But, hmmm. Perhaps there is the question about how the app "knows" that 
> the camera is upside down or is pointed in another direction. If the 
> camera has a gyroscope inside, for example, then it could be the camera 
> which needs to tell to the app about the current orientation, or else 
> the app would not have any way to know ... Is this the problem, then? 
> For that kind of thing, one might need more than two bits in order to 
> pass the needed information.
> 

Yes that is what we are talking about, the camera having a gravity switch 
(usually nothing as advanced as a gyroscope). Also the bits we are talking 
about are in a struct which communicates information one way, from the camera 
to userspace, so there is no way to clear the bits to make the camera do something.

Regards,

Hans

Re: [RFC] How to pass camera Orientation to userspace

[kilgota]

On Sun, 22 Feb 2009, Hans de Goede wrote:

<big snip>

> Yes that is what we are talking about, the camera having a gravity switch 
> (usually nothing as advanced as a gyroscope). Also the bits we are talking 
> about are in a struct which communicates information one way, from the camera 
> to userspace, so there is no way to clear the bits to make the camera do 
> something.
>
> Regards,
>
> Hans

Well, of course the bits are in a struct which is communicates information 
one way from the camera to userspace. But userspace can do what it deems 
appropriate with those bits. My point was that if userspace wants to turn 
the camera upside down in software, all it has to do is to negate those 
bits. For that purpose, it does not matter whether the bits were 
originally set "on" (indicating that the sensor is upside down in the 
camera) or whether they were set "off" (indicating the sensor is right 
side up in the camera). Now, if it is a question of passing along a 
changing camera orientation, it is obvious that has to be done with some 
other mechanism. So, unless my not terribly profound observation about 
the ability to negate bits was confusing, we are not in disagreement.

Theodore Kilgore

Re: [RFC] How to pass camera Orientation to userspace

[Trent Piepho]

On Sun, 22 Feb 2009, Hans de Goede wrote:
> Yes that is what we are talking about, the camera having a gravity switch
> (usually nothing as advanced as a gyroscope). Also the bits we are talking
> about are in a struct which communicates information one way, from the camera
> to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of
the camera sensor really is a different concept than the current value of a
gravity sensor.  It's not necessary, and maybe not even desirable, to
handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits.  In some structures there are
only a few left.  They will run out.  Then what?  Packing non-standard
sensor attributes and camera sensor meta-data into a few reserved bits is
not a sustainable policy.

Controls on the other card are not limited and won't run out.

Currently reserved bits and previously reserved but now in use bits
look exactly the same.  How does an app know if the bits are valid
or not?  More reserved bits?

The control API is already designed to allow for enumeration of controls.

Reserved bits have no meta-data to document them.  An app sees that some
bits in the input struct which were reserved when it was written are now
set.  What does this tell the app?  Nothing.  What can the app tell the
user?  Nothing.

The control API provides for control meta-data.  We have a means of
enumeration, name, min, max, step size, and various flags to tell us about
a control that wasn't already handled.  Does this new gravity sensor have
45 degree resolution?  The control's step size can show this even if only
90 degree rotations had been considered when gravity sensor support was
first added to some other driver.  At the very least, the app (and various
existing utilities) can provide information about the controls to the user.

Re: [RFC] How to pass camera Orientation to userspace

[Hans de Goede]

Trent Piepho wrote:
> On Sun, 22 Feb 2009, Hans de Goede wrote:
>> Yes that is what we are talking about, the camera having a gravity switch
>> (usually nothing as advanced as a gyroscope). Also the bits we are talking
>> about are in a struct which communicates information one way, from the camera
>> to userspace, so there is no way to clear the bits to make the camera do something.
> 
> First, I'd like to say I agree with most that the installed orientation of
> the camera sensor really is a different concept than the current value of a
> gravity sensor.  It's not necessary, and maybe not even desirable, to
> handle them in the same way.
> 
> I do not see the advantage of using reserved bits instead of controls.
> 
> The are a limited number of reserved bits.  In some structures there are
> only a few left.  They will run out.  Then what?  Packing non-standard
> sensor attributes and camera sensor meta-data into a few reserved bits is
> not a sustainable policy.
> 
> Controls on the other card are not limited and won't run out.
> 

Yes but these things are *not* controls, end of discussion. The control API is 
for controls, not to stuff all kind of cruft in.

Regards,

Hans

Re: [RFC] How to pass camera Orientation to userspace

[Trent Piepho]

On Sun, 22 Feb 2009, Hans de Goede wrote:
> Trent Piepho wrote:
> > On Sun, 22 Feb 2009, Hans de Goede wrote:
> >> Yes that is what we are talking about, the camera having a gravity switch
> >> (usually nothing as advanced as a gyroscope). Also the bits we are talking
> >> about are in a struct which communicates information one way, from the camera
> >> to userspace, so there is no way to clear the bits to make the camera do something.
> >
> > First, I'd like to say I agree with most that the installed orientation of
> > the camera sensor really is a different concept than the current value of a
> > gravity sensor.  It's not necessary, and maybe not even desirable, to
> > handle them in the same way.
> >
> > I do not see the advantage of using reserved bits instead of controls.
> >
> > The are a limited number of reserved bits.  In some structures there are
> > only a few left.  They will run out.  Then what?  Packing non-standard
> > sensor attributes and camera sensor meta-data into a few reserved bits is
> > not a sustainable policy.
> >
> > Controls on the other card are not limited and won't run out.
> >
>
> Yes but these things are *not* controls, end of discussion. The control API is
> for controls, not to stuff all kind of cruft in.

All kind of cruft belongs in the reserved bits of whatever field it can be
stuffed in?  Until the bits run out and then it belongs where?

What is the difference?  Why does it matter?  Performance?  Maintenance?
Is there something that's not possible?  I do not find "end of discussion"
to be a very convincing argument.

Re: [RFC] How to pass camera Orientation to userspace

[Hans Verkuil]

On Sunday 22 February 2009 23:54:42 Hans de Goede wrote:
> Trent Piepho wrote:
> > On Sun, 22 Feb 2009, Hans de Goede wrote:
> >> Yes that is what we are talking about, the camera having a gravity
> >> switch (usually nothing as advanced as a gyroscope). Also the bits we
> >> are talking about are in a struct which communicates information one
> >> way, from the camera to userspace, so there is no way to clear the
> >> bits to make the camera do something.
> >
> > First, I'd like to say I agree with most that the installed orientation
> > of the camera sensor really is a different concept than the current
> > value of a gravity sensor.  It's not necessary, and maybe not even
> > desirable, to handle them in the same way.
> >
> > I do not see the advantage of using reserved bits instead of controls.
> >
> > The are a limited number of reserved bits.  In some structures there
> > are only a few left.  They will run out.  Then what?  Packing
> > non-standard sensor attributes and camera sensor meta-data into a few
> > reserved bits is not a sustainable policy.
> >
> > Controls on the other card are not limited and won't run out.
>
> Yes but these things are *not* controls, end of discussion. The control
> API is for controls, not to stuff all kind of cruft in.

I agree, these are not controls.

There is an option to use the current status field. There are enough bits 
free, that's not the problem. But the spec is explicit about the fact that 
these bits apply to the current input only, and that's not true for these 
new bits. We can change the spec in this regard of course, but then you 
have to document each bit of the status field whether it is valid for the 
current input only, or also if this isn't the current input. It's all a bit 
messy.

In addition, there are 4 reserved fields here and it is the first time in a 
very long time that we actually need one. And after all, that's why they 
are there in the first place.

I see three options:

1) stuff them into the status field after all.
2) take one of the reserved fields and make it a single 'flags' field.
3) take one of the reserved fields and make it a u16 capabilities and u16 
flags field.

Trent does have a point that we need to be careful not to add fields without 
a good reason. Choosing option 1 fits the bill, and the orientation also 
fits the 'status' name. Only the sensor mount orientation is not really a 
status. Although with some creative naming we might come close :-)

Hmm, let's see:

V4L2_IN_ST_HAS_SENSOR_INFO 	0x00000010
V4L2_IN_ST_SENSOR_HFLIPPED 	0x00000020
V4L2_IN_ST_SENSOR_VFLIPPED 	0x00000040

V4L2_IN_ST_HAS_PIVOT_INFO	0x00001000
V4L2_IN_ST_PIVOT_0 		0x00000000
V4L2_IN_ST_PIVOT_90 		0x00002000
V4L2_IN_ST_PIVOT_180 		0x00004000
V4L2_IN_ST_PIVOT_270 		0x00006000
V4L2_IN_ST_PIVOT_MSK 		0x00006000

Actually, that's not too bad.

Regards,

	Hans

-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG

Re: [RFC] How to pass camera Orientation to userspace

[Hans de Goede]

Trent Piepho wrote:
> On Sun, 22 Feb 2009, Hans de Goede wrote:
>> Trent Piepho wrote:
>>> On Sun, 22 Feb 2009, Hans de Goede wrote:
>>>> Yes that is what we are talking about, the camera having a gravity switch
>>>> (usually nothing as advanced as a gyroscope). Also the bits we are talking
>>>> about are in a struct which communicates information one way, from the camera
>>>> to userspace, so there is no way to clear the bits to make the camera do something.
>>> First, I'd like to say I agree with most that the installed orientation of
>>> the camera sensor really is a different concept than the current value of a
>>> gravity sensor.  It's not necessary, and maybe not even desirable, to
>>> handle them in the same way.
>>>
>>> I do not see the advantage of using reserved bits instead of controls.
>>>
>>> The are a limited number of reserved bits.  In some structures there are
>>> only a few left.  They will run out.  Then what?  Packing non-standard
>>> sensor attributes and camera sensor meta-data into a few reserved bits is
>>> not a sustainable policy.
>>>
>>> Controls on the other card are not limited and won't run out.
>>>
>> Yes but these things are *not* controls, end of discussion. The control API is
>> for controls, not to stuff all kind of cruft in.
> 
> All kind of cruft belongs in the reserved bits of whatever field it can be
> stuffed in?

Not whatever field, these are input properties which happen to also be pretty 
binary so putting them in the input flags field makes plenty of sense.

> What is the difference?  Why does it matter?  Performance?  Maintenance?
> Is there something that's not possible?  I do not find "end of discussion"
> to be a very convincing argument.

Well they are not controls, that is the difference, the control interface is 
for controls (and only for controls, end of discussion if you ask me). These 
are not controls but properties, they do not have a default min and max value, 
they have only one *unchanging* value, there  is nothing the application can 
control / change. It has been suggested to make them readonly, but that does 
not fix the ugliness. A proper written v4l2 application will enumerate all the 
controls, and then the user will see a grayed out control saying: "your cam is 
upside down" what is there to control ? will this be a grayed out slider? or a 
grayed out checkbox "your cam is upside down", or maybe a not grayed out 
dropdown: where the user can select: "my sensor is upside down", "I deny my 
sensor is upside down", "I don't care my sensor is upside down", "WTF is this 
doing in my webcam control panel?", "nwod edispu si rosnes yM"

Do you know I have an idea, lets get rid of the S2 API for DVB and put all that 
in controls too. Oh, and think like standards for video formats, surely that 
can be a control too, and ... and, ...

Yes we can stuff almost anything in a control, that does not make it a good idea.

Regards,

Hans

Re: [RFC] How to pass camera Orientation to userspace

[Trent Piepho]

On Mon, 23 Feb 2009, Hans Verkuil wrote:
> On Sunday 22 February 2009 23:54:42 Hans de Goede wrote:
> > Trent Piepho wrote:
> > > On Sun, 22 Feb 2009, Hans de Goede wrote:
> > >> Yes that is what we are talking about, the camera having a gravity
> > >> switch (usually nothing as advanced as a gyroscope). Also the bits we
> > >> are talking about are in a struct which communicates information one
> > >> way, from the camera to userspace, so there is no way to clear the
> > >> bits to make the camera do something.
> > >
> > > First, I'd like to say I agree with most that the installed orientation
> > > of the camera sensor really is a different concept than the current
> > > value of a gravity sensor.  It's not necessary, and maybe not even
> > > desirable, to handle them in the same way.
> > >
> > > I do not see the advantage of using reserved bits instead of controls.
> > >
> > > The are a limited number of reserved bits.  In some structures there
> > > are only a few left.  They will run out.  Then what?  Packing
> > > non-standard sensor attributes and camera sensor meta-data into a few
> > > reserved bits is not a sustainable policy.
> > >
> > > Controls on the other card are not limited and won't run out.
> >
> > Yes but these things are *not* controls, end of discussion. The control
> > API is for controls, not to stuff all kind of cruft in.
>
> I agree, these are not controls.
>
> There is an option to use the current status field. There are enough bits
> free, that's not the problem. But the spec is explicit about the fact that
> these bits apply to the current input only, and that's not true for these
> new bits. We can change the spec in this regard of course, but then you
> have to document each bit of the status field whether it is valid for the
> current input only, or also if this isn't the current input. It's all a bit
> messy.
>
> In addition, there are 4 reserved fields here and it is the first time in a
> very long time that we actually need one. And after all, that's why they
> are there in the first place.

v4l2_capability: 5 of 32 cap bits left, 4 reserved words
v4l2_fmtdesc: 31 flag bits, 4 reserved words
v4l2_buffer: 22 flags bits, 1 reserved word
v4l2_framebuffer: 25 cap bits, 26 flag bits, no reserved words
v4l2_input: 4 reserved words
v4l2_output: 4 reserved words
v4l2_tuner: 27 cap bits, 4 reserved words

> Trent does have a point that we need to be careful not to add fields without
> a good reason. Choosing option 1 fits the bill, and the orientation also
> fits the 'status' name. Only the sensor mount orientation is not really a
> status. Although with some creative naming we might come close :-)
>
> Hmm, let's see:
>
> V4L2_IN_ST_HAS_SENSOR_INFO 	0x00000010
> V4L2_IN_ST_SENSOR_HFLIPPED 	0x00000020
> V4L2_IN_ST_SENSOR_VFLIPPED 	0x00000040
>
> V4L2_IN_ST_HAS_PIVOT_INFO	0x00001000
> V4L2_IN_ST_PIVOT_0 		0x00000000
> V4L2_IN_ST_PIVOT_90 		0x00002000
> V4L2_IN_ST_PIVOT_180 		0x00004000
> V4L2_IN_ST_PIVOT_270 		0x00006000
> V4L2_IN_ST_PIVOT_MSK 		0x00006000

One of my other points what the controls include meta-data.  What happens
when someone has an orientation sensor with 45 degree resolution?  A
control would just change the step size from 90 to 45.  Existing software
would already know what it means.  With this method you have to add:

V4L2_IN_ST_HAS_PIVOT_45_INFO	0x00008000
V4L2_IN_ST_PIVOT_45		0x00010000
V4L2_IN_ST_PIVOT_135		0x00012000
V4L2_IN_ST_PIVOT_225		0x00014000
V4L2_IN_ST_PIVOT_315		0x00016000
V4L2_IN_ST_PIVOT_45_MSK		0x00016000

Interpreting the pivot bits becomes more fun now.  Existing software
can do nothing with these extra bits.  It can tell the user nothing.

Next, add camera aperture.  f/0.7 to f/64 in both half stop and third stop
increments.  Getting low on bits already.

Suppose I have just three or four sensors on my camera.  That's not all
that much.  A typical digital camera has more than that.  They're probably
connected by something slow, like I2C.  Maybe non-trivial calculations are
needed to use the data, like most demod's SNR registers.  Each time I query
the input the driver must query and calculate the value of ALL the sensors.
Even though I only can about one of them.  Controls can be queried individually.

Re: [RFC] How to pass camera Orientation to userspace

[Trent Piepho]

On Mon, 23 Feb 2009, Hans de Goede wrote:
> Trent Piepho wrote:
> > On Sun, 22 Feb 2009, Hans de Goede wrote:
> >> Trent Piepho wrote:
> >>> On Sun, 22 Feb 2009, Hans de Goede wrote:
> >>>> Yes that is what we are talking about, the camera having a gravity switch
> >>>> (usually nothing as advanced as a gyroscope). Also the bits we are talking
> >>>> about are in a struct which communicates information one way, from the camera
> >>>> to userspace, so there is no way to clear the bits to make the camera do something.
> >>> First, I'd like to say I agree with most that the installed orientation of
> >>> the camera sensor really is a different concept than the current value of a
> >>> gravity sensor.  It's not necessary, and maybe not even desirable, to
> >>> handle them in the same way.
> >>>
> >>> I do not see the advantage of using reserved bits instead of controls.
> >>>
> >>> The are a limited number of reserved bits.  In some structures there are
> >>> only a few left.  They will run out.  Then what?  Packing non-standard
> >>> sensor attributes and camera sensor meta-data into a few reserved bits is
> >>> not a sustainable policy.
> >>>
> >>> Controls on the other card are not limited and won't run out.
> >>>
> >> Yes but these things are *not* controls, end of discussion. The control API is
> >> for controls, not to stuff all kind of cruft in.
> >
> > All kind of cruft belongs in the reserved bits of whatever field it can be
> > stuffed in?
>
> Not whatever field, these are input properties which happen to also be pretty
> binary so putting them in the input flags field makes plenty of sense.
>
> > What is the difference?  Why does it matter?  Performance?  Maintenance?
> > Is there something that's not possible?  I do not find "end of discussion"
> > to be a very convincing argument.
>
> Well they are not controls, that is the difference, the control interface is
> for controls (and only for controls, end of discussion if you ask me). These
> are not controls but properties, they do not have a default min and max value,

Camera pivot sensor ranges from 0 to 270.  How is that not a min and max?

> they have only one *unchanging* value, there  is nothing the application can

Camera sensors don't have an unchanging value.

And who says scan order can't change?  Suppose the camera returns raw bayer
format data top to bottom, but if you request yuv then an image processing
section needs to kick in and that returns the data bottom to top.

> control / change. It has been suggested to make them readonly, but that does
> not fix the ugliness. A proper written v4l2 application will enumerate all the
> controls, and then the user will see a grayed out control saying: "your cam is
> upside down" what is there to control ? will this be a grayed out slider? or a
> grayed out checkbox "your cam is upside down", or maybe a not grayed out
> dropdown: where the user can select: "my sensor is upside down", "I deny my
> sensor is upside down", "I don't care my sensor is upside down", "WTF is this
> doing in my webcam control panel?", "nwod edispu si rosnes yM"

Why is there a read-only flag for controls if the concept is so mind
blowing to users?  Have there been complaints about it?

> Do you know I have an idea, lets get rid of the S2 API for DVB and put all that
> in controls too. Oh, and think like standards for video formats, surely that
> can be a control too, and ... and, ...

Good point.  The S2 API is much more like the control interface than the
previous API.  Enumerated attributes which can be set one at a time or in
groups.  More can be added.  There is some meta data about them.  The old
API used the a limited number of fixed structs, a few reserved bits, no
meta-data, and a query/set everything at once API.

I think the camera meta-data and camera sensor API should look more like
S2.

Re: [RFC] How to pass camera Orientation to userspace

[Hans Verkuil]

On Monday 23 February 2009 00:56:40 Trent Piepho wrote:
> On Mon, 23 Feb 2009, Hans Verkuil wrote:
> > On Sunday 22 February 2009 23:54:42 Hans de Goede wrote:
> > > Trent Piepho wrote:
> > > > On Sun, 22 Feb 2009, Hans de Goede wrote:
> > > >> Yes that is what we are talking about, the camera having a gravity
> > > >> switch (usually nothing as advanced as a gyroscope). Also the bits
> > > >> we are talking about are in a struct which communicates
> > > >> information one way, from the camera to userspace, so there is no
> > > >> way to clear the bits to make the camera do something.
> > > >
> > > > First, I'd like to say I agree with most that the installed
> > > > orientation of the camera sensor really is a different concept than
> > > > the current value of a gravity sensor.  It's not necessary, and
> > > > maybe not even desirable, to handle them in the same way.
> > > >
> > > > I do not see the advantage of using reserved bits instead of
> > > > controls.
> > > >
> > > > The are a limited number of reserved bits.  In some structures
> > > > there are only a few left.  They will run out.  Then what?  Packing
> > > > non-standard sensor attributes and camera sensor meta-data into a
> > > > few reserved bits is not a sustainable policy.
> > > >
> > > > Controls on the other card are not limited and won't run out.
> > >
> > > Yes but these things are *not* controls, end of discussion. The
> > > control API is for controls, not to stuff all kind of cruft in.
> >
> > I agree, these are not controls.
> >
> > There is an option to use the current status field. There are enough
> > bits free, that's not the problem. But the spec is explicit about the
> > fact that these bits apply to the current input only, and that's not
> > true for these new bits. We can change the spec in this regard of
> > course, but then you have to document each bit of the status field
> > whether it is valid for the current input only, or also if this isn't
> > the current input. It's all a bit messy.
> >
> > In addition, there are 4 reserved fields here and it is the first time
> > in a very long time that we actually need one. And after all, that's
> > why they are there in the first place.
>
> v4l2_capability: 5 of 32 cap bits left, 4 reserved words
> v4l2_fmtdesc: 31 flag bits, 4 reserved words
> v4l2_buffer: 22 flags bits, 1 reserved word
> v4l2_framebuffer: 25 cap bits, 26 flag bits, no reserved words
> v4l2_input: 4 reserved words
> v4l2_output: 4 reserved words
> v4l2_tuner: 27 cap bits, 4 reserved words
>
> > Trent does have a point that we need to be careful not to add fields
> > without a good reason. Choosing option 1 fits the bill, and the
> > orientation also fits the 'status' name. Only the sensor mount
> > orientation is not really a status. Although with some creative naming
> > we might come close :-)
> >
> > Hmm, let's see:
> >
> > V4L2_IN_ST_HAS_SENSOR_INFO 	0x00000010
> > V4L2_IN_ST_SENSOR_HFLIPPED 	0x00000020
> > V4L2_IN_ST_SENSOR_VFLIPPED 	0x00000040
> >
> > V4L2_IN_ST_HAS_PIVOT_INFO	0x00001000
> > V4L2_IN_ST_PIVOT_0 		0x00000000
> > V4L2_IN_ST_PIVOT_90 		0x00002000
> > V4L2_IN_ST_PIVOT_180 		0x00004000
> > V4L2_IN_ST_PIVOT_270 		0x00006000
> > V4L2_IN_ST_PIVOT_MSK 		0x00006000
>
> One of my other points what the controls include meta-data.  What happens
> when someone has an orientation sensor with 45 degree resolution?  A
> control would just change the step size from 90 to 45.  Existing software
> would already know what it means.  With this method you have to add:
>
> V4L2_IN_ST_HAS_PIVOT_45_INFO	0x00008000
> V4L2_IN_ST_PIVOT_45		0x00010000
> V4L2_IN_ST_PIVOT_135		0x00012000
> V4L2_IN_ST_PIVOT_225		0x00014000
> V4L2_IN_ST_PIVOT_315		0x00016000
> V4L2_IN_ST_PIVOT_45_MSK		0x00016000
>
> Interpreting the pivot bits becomes more fun now.  Existing software
> can do nothing with these extra bits.  It can tell the user nothing.

These bits deal exclusively with the sensor position and for the sole 
purpose of deciding whether the image has to be rotated in hard/software to 
get it the right-way up. In most cases this is limited to rotating 180 
degrees, but omap has some memory management tricks to do the 90/270 degree 
cases as well.

This is not the same as having a full positioning 2D or 3D system in a 
camera that can give you exact angles. I do not think that should be part 
of v4l2. There isn't anything that v4l2 can do with that. An application 
might use this information to setup special transformations to do such 
rotates, and if the hardware can do that then that would be part of an 
effects API or something like that. With non-90 degree angle you just run 
into a totally different set of problems that are definitely out-of-scope.

> Next, add camera aperture.  f/0.7 to f/64 in both half stop and third
> stop increments.  Getting low on bits already.
>
> Suppose I have just three or four sensors on my camera.  That's not all
> that much.  A typical digital camera has more than that.  They're
> probably connected by something slow, like I2C.  Maybe non-trivial
> calculations are needed to use the data, like most demod's SNR registers.
>  Each time I query the input the driver must query and calculate the
> value of ALL the sensors. Even though I only can about one of them. 
> Controls can be queried individually.

Note that I am not averse to this idea in general. It would be easy to setup 
a separate class of 'controls' containing meta data that work the same as 
controls, but are just called differently. It has always been in the back 
of my mind that it would be a pretty good match, functionality-wise.

But in this case it just fits very well in the existing v4l2_input struct. 
These bits are specific to each input, so VIDIOC_ENUMINPUTS is the perfect 
match for that. If you want to do this with controls, then you would end up 
with a control for each input. Hardly elegant.

Regards,

	Hans

-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG

Re: [RFC] How to pass camera Orientation to userspace

[Hans de Goede]

Trent Piepho wrote:
> On Mon, 23 Feb 2009, Hans de Goede wrote:
>> Trent Piepho wrote:
>>> On Sun, 22 Feb 2009, Hans de Goede wrote:
>>>> Trent Piepho wrote:
>>>>> On Sun, 22 Feb 2009, Hans de Goede wrote:
>>>>>> Yes that is what we are talking about, the camera having a gravity switch
>>>>>> (usually nothing as advanced as a gyroscope). Also the bits we are talking
>>>>>> about are in a struct which communicates information one way, from the camera
>>>>>> to userspace, so there is no way to clear the bits to make the camera do something.
>>>>> First, I'd like to say I agree with most that the installed orientation of
>>>>> the camera sensor really is a different concept than the current value of a
>>>>> gravity sensor.  It's not necessary, and maybe not even desirable, to
>>>>> handle them in the same way.
>>>>>
>>>>> I do not see the advantage of using reserved bits instead of controls.
>>>>>
>>>>> The are a limited number of reserved bits.  In some structures there are
>>>>> only a few left.  They will run out.  Then what?  Packing non-standard
>>>>> sensor attributes and camera sensor meta-data into a few reserved bits is
>>>>> not a sustainable policy.
>>>>>
>>>>> Controls on the other card are not limited and won't run out.
>>>>>
>>>> Yes but these things are *not* controls, end of discussion. The control API is
>>>> for controls, not to stuff all kind of cruft in.
>>> All kind of cruft belongs in the reserved bits of whatever field it can be
>>> stuffed in?
>> Not whatever field, these are input properties which happen to also be pretty
>> binary so putting them in the input flags field makes plenty of sense.
>>
>>> What is the difference?  Why does it matter?  Performance?  Maintenance?
>>> Is there something that's not possible?  I do not find "end of discussion"
>>> to be a very convincing argument.
>> Well they are not controls, that is the difference, the control interface is
>> for controls (and only for controls, end of discussion if you ask me). These
>> are not controls but properties, they do not have a default min and max value,
> 
> Camera pivot sensor ranges from 0 to 270.  How is that not a min and max?
> 
>> they have only one *unchanging* value, there  is nothing the application can
> 
> Camera sensors don't have an unchanging value.
> 
> And who says scan order can't change?  Suppose the camera returns raw bayer
> format data top to bottom, but if you request yuv then an image processing
> section needs to kick in and that returns the data bottom to top.
> 

Yes, because hardware designers like throwing away lots of transistors to 
memory so they are going to put memory in the controller to buffer an entire 
frame and then scan out the memory buffer in different order then the sensor 
gave them the data, so they cannot do FIFO, so they will actually need 2 frames 
of memory.

If the sensor is soldered upside down on the PCB that is a very much unchanging 
value, and an input property if you ask me.

So new proposal: use 2 bits in the input flags to indicate if the input is 
hardwired vflipped and/or hflipped.

Create a new class of controls for querying possible changing camera properties 
like pivoting and aperture.

Regards,

Hans

Re: [RFC] How to pass camera Orientation to userspace

[Mauro Carvalho Chehab]

On Sat, 21 Feb 2009 12:53:57 +0100
Hans Verkuil <hverkuil@xs4all.nl> wrote:

> Hi Adam,
> 
> Sorry for the late reply, it's been very busy.

Me too.

> > 1) Reuse the existing HFLIP and VFLIP controls, marking them as read-only
> > Pros : No change needed to videodev2.h
> > Cons: It is confusing to have controls that have a subtly different
> > meaning if they are read only. Existing apps that support those controls
> > might get confused. Would require polling to support the case of a camera
> > being turned toward / away from the user while streaming.

Reusing an existing control for a different meaning seems wrong. What happens
when some cam has the capability of doing hardware flipping, and have the cam
flipped?

> > 2) Introduce a new orientation control (possibly in a new
> > CAMERA_PROPERTIES class)
> > Pros: libv4l can easily tell if the driver supports the control.
> > Cons: It is really a property, not a control so calling it a control is
> > wrong. Controls add lots of overhead in terms of driver code. Would
> > require polling to support the case of a camera being turned toward /
> > away from the user while streaming.

I think this could be a good idea, but I agree with Hans Verkuil comments: since
this is a characteristics for a given input, using a control here would mean
that the driver will return it based on the selected input. Seems a little
messy. Also, a mounted characteristics of the device, is not really a sensor,
as Hans de Goede pointed.

> > 3) Use an extra couple of bits in V4L2_BUF_FLAGS
> > Pros: Simple to implement. Can change per frame without needing polling.
> > Cons: Doesn't work for non libv4l apps that try to use the read()
> > interface. Can't easily identify drivers that don't support it (does 0
> > mean not rotated or just not implemented). Can only be read when
> > streaming (does that matter?)
> 
> I think that matters, yes.

I don't think this is a good idea. The metadata at the frame polling is meant
to return the stream info. We shouldn't mix sensor position here.

> > 4) Use some reserved bits from the v4l2_capability structure
> > Pros: Less overhead than controls.
> > Cons: Would require polling to support the case of a camera being turned
> > toward / away from the user while streaming. Can't easily identify
> > drivers that don't support it.
> > 5) Use some reserved bits from the v4l2_input structure (or possibly the
> > status word but that is normally only valid for current input)
> > Pros: Less overhead than controls. Could support multiple sensors in one
> > camera if such a beast exists.
> What does exist is devices with a video input (e.g. composite) and a camera 
> input: each input will have different flags. Since these vflip/hflip 
> properties do not change they can be enumerated in advance and you know 
> what each input supports.
> 
> > Cons: Would require polling to support the case of a camera being turned
> > toward / away from the user while streaming.
> 
> Polling applies only to the bits that tell the orientation of the camera. 
> See below for a discussion of this.

Analog tv does polling for signal strength, since userspace apps do mute and
stops presenting video, if the signal is too weak. IMO, a similar mechanism
should be used by pivoting.

IMO, this would be better addressed as a property of v4l2_input. So, I think
that (5) is better than (4).

> > Can't easily identify drivers that don't support it.
> 
> Not too difficult to add through the use of a capability bit. Either in 
> v4l2_input or (perhaps) v4l2_capability.
> 
> Another Pro is that this approach will also work for v4l2_output in the case 
> of, say, rotated LCD displays. Using camera orientation bits in v4l2_buffer 
> while capturing will work, but using similar bits for output will fail 
> since the data is going in the wrong direction.
> 
> > The interest in detecting if a driver provides this informnation is to
> > allow libv4l to know when it should use the driver provided information
> > and when it should use its internal table (which needs to be retained for
> > backward compatibility). With no detection capability the driver provided
> > info should be ignored for USB IDs in the built in table.
> >
> > Thoughts please

There is a case that we should think: at libv4l, we may need to override the
"default" orientation, by a custom one. For example: Surveillance systems have
cameras mounted on a fixed position. Depending on the camera, and the desired
position, some cameras may needed to be mounted rotated (the same case also
applies to some embedded hardware like ATM machines, where a webcam maybe
mounted with 180 degrees, due to hardware constraints).

Ok, this is nothing that kernel needs to handle, but, at userspace, we need to
have a file where the user could edit and store the camera position, to
override whatever we have in kernel.

> 
> Is polling bad in this case? It is not something that needs immediate 
> attention IMHO. The overhead for checking once every X seconds is quite 
> low. Furthermore, it is only needed on devices that cannot do v/hflipping 
> in hardware.
> 
> An alternative is to put some effort in a proper event interface. There is 
> one implemented in include/linux/dvb/video.h and used by ivtv for video 
> decoding. The idea is that the application registers events it wants to 
> receive, and whenever such an event arrives the select() call will exit 
> with a high-prio event (exception). The application then checks what 
> happened.
> 
> The video.h implementation is pretty crappy, but we can make something more 
> useful. Alternatively, I believe there are other event mechanisms as well 
> in the kernel. I know some work was done on that, but I don't know what the 
> end result was.
> 
> Note that this does not apply to sensor mount information. I think that 
> v4l2_input is the perfect place for that. It is after all an input-specific 
> property and static property, and it maps 1-on-1 to the output case as 
> well, should we need it there as well.

I think we will need an event interface sooner or later. I had some
discussions in the past about servo mechanisms to control the webcam position.
Let's consider a webcam with movement servos at vertical and horizontal, and
that we want to follow an object, like a person face. After sending a command
to the servo, it will start moving, but it will take some time until it reaches
the point. An event interface seems interesting to advice userspace that the
movement already happened.

Maybe we could use this approach for the gravity sensors.

Cheers,
Mauro

Re: [RFC] How to pass camera Orientation to userspace

[Mauro Carvalho Chehab]

On Mon, 23 Feb 2009 08:34:08 +0100
Hans Verkuil <hverkuil@xs4all.nl> wrote:

> On Monday 23 February 2009 00:56:40 Trent Piepho wrote:
> > On Mon, 23 Feb 2009, Hans Verkuil wrote:
> > > On Sunday 22 February 2009 23:54:42 Hans de Goede wrote:
> > > > Trent Piepho wrote:
> > > > > On Sun, 22 Feb 2009, Hans de Goede wrote:
> > > > >> Yes that is what we are talking about, the camera having a gravity
> > > > >> switch (usually nothing as advanced as a gyroscope). Also the bits
> > > > >> we are talking about are in a struct which communicates
> > > > >> information one way, from the camera to userspace, so there is no
> > > > >> way to clear the bits to make the camera do something.
> > > > >
> > > > > First, I'd like to say I agree with most that the installed
> > > > > orientation of the camera sensor really is a different concept than
> > > > > the current value of a gravity sensor.  It's not necessary, and
> > > > > maybe not even desirable, to handle them in the same way.
> > > > >
> > > > > I do not see the advantage of using reserved bits instead of
> > > > > controls.
> > > > >
> > > > > The are a limited number of reserved bits.  In some structures
> > > > > there are only a few left.  They will run out.  Then what?  Packing
> > > > > non-standard sensor attributes and camera sensor meta-data into a
> > > > > few reserved bits is not a sustainable policy.
> > > > >
> > > > > Controls on the other card are not limited and won't run out.
> > > >
> > > > Yes but these things are *not* controls, end of discussion. The
> > > > control API is for controls, not to stuff all kind of cruft in.
> > >
> > > I agree, these are not controls.
> > >
> > > There is an option to use the current status field. There are enough
> > > bits free, that's not the problem. But the spec is explicit about the
> > > fact that these bits apply to the current input only, and that's not
> > > true for these new bits. We can change the spec in this regard of
> > > course, but then you have to document each bit of the status field
> > > whether it is valid for the current input only, or also if this isn't
> > > the current input. It's all a bit messy.
> > >
> > > In addition, there are 4 reserved fields here and it is the first time
> > > in a very long time that we actually need one. And after all, that's
> > > why they are there in the first place.
> >
> > v4l2_capability: 5 of 32 cap bits left, 4 reserved words
> > v4l2_fmtdesc: 31 flag bits, 4 reserved words
> > v4l2_buffer: 22 flags bits, 1 reserved word
> > v4l2_framebuffer: 25 cap bits, 26 flag bits, no reserved words
> > v4l2_input: 4 reserved words
> > v4l2_output: 4 reserved words
> > v4l2_tuner: 27 cap bits, 4 reserved words
> >
> > > Trent does have a point that we need to be careful not to add fields
> > > without a good reason. Choosing option 1 fits the bill, and the
> > > orientation also fits the 'status' name. Only the sensor mount
> > > orientation is not really a status. Although with some creative naming
> > > we might come close :-)
> > >
> > > Hmm, let's see:
> > >
> > > V4L2_IN_ST_HAS_SENSOR_INFO 	0x00000010
> > > V4L2_IN_ST_SENSOR_HFLIPPED 	0x00000020
> > > V4L2_IN_ST_SENSOR_VFLIPPED 	0x00000040
> > >
> > > V4L2_IN_ST_HAS_PIVOT_INFO	0x00001000
> > > V4L2_IN_ST_PIVOT_0 		0x00000000
> > > V4L2_IN_ST_PIVOT_90 		0x00002000
> > > V4L2_IN_ST_PIVOT_180 		0x00004000
> > > V4L2_IN_ST_PIVOT_270 		0x00006000
> > > V4L2_IN_ST_PIVOT_MSK 		0x00006000
> >
> > One of my other points what the controls include meta-data.  What happens
> > when someone has an orientation sensor with 45 degree resolution?  A
> > control would just change the step size from 90 to 45.  Existing software
> > would already know what it means.  With this method you have to add:
> >
> > V4L2_IN_ST_HAS_PIVOT_45_INFO	0x00008000
> > V4L2_IN_ST_PIVOT_45		0x00010000
> > V4L2_IN_ST_PIVOT_135		0x00012000
> > V4L2_IN_ST_PIVOT_225		0x00014000
> > V4L2_IN_ST_PIVOT_315		0x00016000
> > V4L2_IN_ST_PIVOT_45_MSK		0x00016000
> >
> > Interpreting the pivot bits becomes more fun now.  Existing software
> > can do nothing with these extra bits.  It can tell the user nothing.
> 
> These bits deal exclusively with the sensor position and for the sole 
> purpose of deciding whether the image has to be rotated in hard/software to 
> get it the right-way up. In most cases this is limited to rotating 180 
> degrees, but omap has some memory management tricks to do the 90/270 degree 
> cases as well.

> This is not the same as having a full positioning 2D or 3D system in a 
> camera that can give you exact angles. I do not think that should be part 
> of v4l2. There isn't anything that v4l2 can do with that. An application 
> might use this information to setup special transformations to do such 
> rotates, and if the hardware can do that then that would be part of an 
> effects API or something like that. With non-90 degree angle you just run 
> into a totally different set of problems that are definitely out-of-scope.

We are trying to address 2 different situations using the same approach. After
thinking more about it, I suspect that we need two separate approaches. 

For the static info that a sensor is mounted on a different position on a
notebook case, for example. The same camera could be mounted on different
position for different hardwares and kernel will never know for sure, if the
vendor didn't care to change the USB ID for the rotated camera.

All kernel could provide is a HINT info, for some known cases, but it should be
possible to override this in userspace. For this case, the better userspace api
seems to be by using some flags at v4l2_input.

However, I'm starting to agree with one of the Hans de Goede opinions that this
information should be stored at userspace, since, on a large amount of cases,
like surveillance mounted cameras, or some hardware with a fixed commercial
webcam, like an ATM machine, this info is not stored anyware. The only way is
to allow users to edit a file pointing the webcam orientation for that device.

The second case is something like a gravity sensor. This can provide us 2D/3D
angles with 90 degrees stepping, but I don't doubt that more sophisticated
cameras will start to appear, having a more precise mechanism, to be used by
applications that could compensate a 2D/3D rotation on software.

In the trivial case of a 90 degrees step, IMO, the event interface seems to be
the better approach. However, if we really want to do motion compensation in
software, the only alternative is to have this information together with the
streaming meta-data.

Cheers,
Mauro

Re: [RFC] How to pass camera Orientation to userspace

[Adam Baker]

On Monday 23 February 2009, Mauro Carvalho Chehab wrote:
> On Sat, 21 Feb 2009 12:53:57 +0100
>
> Hans Verkuil <hverkuil@xs4all.nl> wrote:
> > Hi Adam,
> >
> > Sorry for the late reply, it's been very busy.
>
> Me too.
>
> > > 1) Reuse the existing HFLIP and VFLIP controls, marking them as
> > > read-only Pros : No change needed to videodev2.h
> > > Cons: It is confusing to have controls that have a subtly different
> > > meaning if they are read only. Existing apps that support those
> > > controls might get confused. Would require polling to support the case
> > > of a camera being turned toward / away from the user while streaming.
>
> Reusing an existing control for a different meaning seems wrong. What
> happens when some cam has the capability of doing hardware flipping, and
> have the cam flipped?

I thought that case had already been agreed, implement the flip controls but
set the flip flag in the hardware to the opposite of what the control says.


>
> > > 4) Use some reserved bits from the v4l2_capability structure
> > > Pros: Less overhead than controls.
> > > Cons: Would require polling to support the case of a camera being
> > > turned toward / away from the user while streaming. Can't easily
> > > identify drivers that don't support it.
> > > 5) Use some reserved bits from the v4l2_input structure (or possibly
> > > the status word but that is normally only valid for current input)
> > > Pros: Less overhead than controls. Could support multiple sensors in
> > > one camera if such a beast exists.
> >
> > What does exist is devices with a video input (e.g. composite) and a
> > camera input: each input will have different flags. Since these
> > vflip/hflip properties do not change they can be enumerated in advance
> > and you know what each input supports.
> >
> > > Cons: Would require polling to support the case of a camera being
> > > turned toward / away from the user while streaming.
> >
> > Polling applies only to the bits that tell the orientation of the camera.
> > See below for a discussion of this.
>
> Analog tv does polling for signal strength, since userspace apps do mute
> and stops presenting video, if the signal is too weak. IMO, a similar
> mechanism should be used by pivoting.
>
> IMO, this would be better addressed as a property of v4l2_input. So, I
> think that (5) is better than (4).
>
> > > Can't easily identify drivers that don't support it.
> >
> > Not too difficult to add through the use of a capability bit. Either in
> > v4l2_input or (perhaps) v4l2_capability.
> >
> > Another Pro is that this approach will also work for v4l2_output in the
> > case of, say, rotated LCD displays. Using camera orientation bits in
> > v4l2_buffer while capturing will work, but using similar bits for output
> > will fail since the data is going in the wrong direction.
> >
> > > The interest in detecting if a driver provides this informnation is to
> > > allow libv4l to know when it should use the driver provided information
> > > and when it should use its internal table (which needs to be retained
> > > for backward compatibility). With no detection capability the driver
> > > provided info should be ignored for USB IDs in the built in table.
> > >
> > > Thoughts please
>
> There is a case that we should think: at libv4l, we may need to override
> the "default" orientation, by a custom one. For example: Surveillance
> systems have cameras mounted on a fixed position. Depending on the camera,
> and the desired position, some cameras may needed to be mounted rotated
> (the same case also applies to some embedded hardware like ATM machines,
> where a webcam maybe mounted with 180 degrees, due to hardware
> constraints).

Agreed, Hans de Geode pointed out the similar case that 2 laptops may use the 
same camera but one mount it upside down so hardware info unrelated to the 
camera indicates the orientation.

>
> Ok, this is nothing that kernel needs to handle, but, at userspace, we need
> to have a file where the user could edit and store the camera position, to
> override whatever we have in kernel.

Unfortunately what that doesn't address is the problem that first started this 
discussion. A camera where the orientation information is contained in the 
USB messages from the camera so the driver is the only thing that can 
reasonably access it.

Note also that for sensor orientation I doubt that 90 or 270 degrees rotation 
will be seen but I do know that the case of data being flipped on just one 
axis does exist.

>
> > Is polling bad in this case? It is not something that needs immediate
> > attention IMHO. The overhead for checking once every X seconds is quite
> > low. Furthermore, it is only needed on devices that cannot do v/hflipping
> > in hardware.
> >
> > An alternative is to put some effort in a proper event interface. There
> > is one implemented in include/linux/dvb/video.h and used by ivtv for
> > video decoding. The idea is that the application registers events it
> > wants to receive, and whenever such an event arrives the select() call
> > will exit with a high-prio event (exception). The application then checks
> > what happened.
> >
> > The video.h implementation is pretty crappy, but we can make something
> > more useful. Alternatively, I believe there are other event mechanisms as
> > well in the kernel. I know some work was done on that, but I don't know
> > what the end result was.
> >
> > Note that this does not apply to sensor mount information. I think that
> > v4l2_input is the perfect place for that. It is after all an
> > input-specific property and static property, and it maps 1-on-1 to the
> > output case as well, should we need it there as well.
>
> I think we will need an event interface sooner or later. I had some
> discussions in the past about servo mechanisms to control the webcam
> position. Let's consider a webcam with movement servos at vertical and
> horizontal, and that we want to follow an object, like a person face. After
> sending a command to the servo, it will start moving, but it will take some
> time until it reaches the point. An event interface seems interesting to
> advice userspace that the movement already happened.

I suspect it would make most sense in that sort of scenario to consider the 
servos as a separate device - they might easily be part of a mount that is 
usable independent of the camera fitted to it.

Adam

Re: [RFC] How to pass camera Orientation to userspace

[kilgota]

On Mon, 23 Feb 2009, Adam Baker wrote:

> On Monday 23 February 2009, Mauro Carvalho Chehab wrote:
>> On Sat, 21 Feb 2009 12:53:57 +0100
>>
>> Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>> Hi Adam,
>>>
>>> Sorry for the late reply, it's been very busy.
>>
>> Me too.

<big snip>

>>>> The interest in detecting if a driver provides this informnation is to
>>>> allow libv4l to know when it should use the driver provided information
>>>> and when it should use its internal table (which needs to be retained
>>>> for backward compatibility). With no detection capability the driver
>>>> provided info should be ignored for USB IDs in the built in table.
>>>>

<snip>

>
>>
>> Ok, this is nothing that kernel needs to handle, but, at userspace, we need
>> to have a file where the user could edit and store the camera position, to
>> override whatever we have in kernel.
>
> Unfortunately what that doesn't address is the problem that first started this
> discussion. A camera where the orientation information is contained in the
> USB messages from the camera so the driver is the only thing that can
> reasonably access it.

Alas, this is so true. What started the entire discussion about passing 
the info about sensor orientation is a set of cameras all of which have 
the same Vendor:Product ID but require different handling of the data 
(vflip and hflip, or just vflip) depending upon information which can only 
be obtained by communication with the camera, *not* by just knowing its 
Vendor:Product ID.

Therefore, it is a little bit disheartening to see discussions -- again -- 
which come back to some kind of "internal table" in V4L, or which come 
back to things like "have a file where the user could edit and store the 
camera position, to override whatever we have in kernel."

Repeating the obvious, which apparently still needs to be repeated because 
not all of the participants in the discussion "get it":

1. The "internal table" in V4L could not handle this problem, because the 
internal table would be based upon what information? The USB 
Vendor:Product number? For that matter, no other table could work, 
either. Well, actually, a table could work. It would have to be inside the 
module supporting the camera, and the matter of which table entry 
corresponds to what camera would have to be settled by passing a USB 
command to the camera and then parsing the camera's response. So now the 
question is how to get the information out of the module, which can only 
be collected and analysed inside the module.

2. The "have a file where the user could edit" idea may seem attractive to 
some, because it shoves the whole problem of agreeing on the appropriate 
way to get needed information out of a kernel module and into userspace 
onto someone not present during the current discussion, the user. However, 
this is not a solution, and thinking about it just a little bit ought to 
make that totally obvious. This is a strongly worded statement, so I will 
proceed to explain why the matter is so obvious.

Let us assume the very best, and assume that every app which is V4L 
conformant has a "one time" initialization step and creates a directory 
$HOME/.app containing stored settings. So we might have a file called 
$HOME/.app/sq905. This file gets automatically written when the hapless 
user hooks up his sq905 camera the first time, and has to go through a 
choice routine to decide which side of the frame is up and which side is 
to the left and to the right. One could even take serious steps to make 
this otherwise unnecessary and silly sequence to be as really nice and 
"user friendly" as it could possibly be, and set this all up to be done 
with a sequence of mouse clicks and then the file, very kindly, gets 
written automatically. Those of you who are thinking that this "file a 
user could edit" is the way to go are, presumably, thinking along these 
lines. Well there are at least four things which are obviously wrong with 
this "solution":

2.1. The user is forced to deal with something which the user should not 
even have to confront. The user is called upon to remedy an omission and a 
deficiency which was ignored at a lower level, because a bunch of 
developers could not come together on a reasonable course of action. Well, 
some don't like to see this one, so there are three more reasons.

2.2 Every camera is going to require a file for itself in $HOME/.app even 
if there is nothing that the user needs to do. Many of the supported 
cameras need nothing of the kind, so this would be kind of silly.

2.3 The user has two apps for dealing with webcams. So now the user needs 
to have another directory called $HOME/.app2 with similar files in it?

2.4 (and this one is the worst of all) The user has two cameras which are 
both powered by the same kernel module, and the two cameras need two 
different things done. Now what??? Both cameras can not simultaneously 
have a valid entry in $HOME/.app/sq905 which tells what to do with the 
data out of the camera. Because what is right for one of them is wrong for 
the other. Further to add to this, one of the tests which was run on this 
very module was to see if it will run two of the same kind of camera 
simultaneously. It passed the test. Now someone wants the data to be 
right-side up from both cameras, and correctly oriented left-to 
right. Is that an unreasonable desire on the part of a user? Of 
course not. Again, now what????

There is no way out of this series of dilemmas and absurdities with the 
"file a user could edit" approach. None.

So, please, could we get serious and actually come up with a reasonable 
solution? To put the matter completely in perspective, the support for the 
affected set of devices is otherwise quite complete, and the only thing 
remaining is to come up with a reasonable solution, which would provide a 
pleasing experience to users.


Theodore Kilgore

Re: [RFC] How to pass camera Orientation to userspace

[Mauro Carvalho Chehab]

On Mon, 23 Feb 2009, kilgota@banach.math.auburn.edu wrote:

>
>
> On Mon, 23 Feb 2009, Adam Baker wrote:
>
>>  On Monday 23 February 2009, Mauro Carvalho Chehab wrote:
>> >  On Sat, 21 Feb 2009 12:53:57 +0100
>> > 
>> >  Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> > >  Hi Adam,
>> > > 
>> > >  Sorry for the late reply, it's been very busy.
>> > 
>> >  Me too.
>
> <big snip>
>
>> > > >  The interest in detecting if a driver provides this informnation is 
>> > > >  to
>> > > >  allow libv4l to know when it should use the driver provided 
>> > > >  information
>> > > >  and when it should use its internal table (which needs to be 
>> > > >  retained
>> > > >  for backward compatibility). With no detection capability the 
>> > > >  driver
>> > > >  provided info should be ignored for USB IDs in the built in table.
>> > > > 
>
> <snip>
>
>> 
>> > 
>> >  Ok, this is nothing that kernel needs to handle, but, at userspace, we 
>> >  need
>> >  to have a file where the user could edit and store the camera position, 
>> >  to
>> >  override whatever we have in kernel.
>>
>>  Unfortunately what that doesn't address is the problem that first started
>>  this
>>  discussion. A camera where the orientation information is contained in
>>  the
>>  USB messages from the camera so the driver is the only thing that can
>>  reasonably access it.
>
> Alas, this is so true. What started the entire discussion about passing the 
> info about sensor orientation is a set of cameras all of which have the same 
> Vendor:Product ID but require different handling of the data (vflip and 
> hflip, or just vflip) depending upon information which can only be obtained 
> by communication with the camera, *not* by just knowing its Vendor:Product 
> ID.
>
> Therefore, it is a little bit disheartening to see discussions -- again -- 
> which come back to some kind of "internal table" in V4L, or which come back 
> to things like "have a file where the user could edit and store the camera 
> position, to override whatever we have in kernel."
>
> Repeating the obvious, which apparently still needs to be repeated because 
> not all of the participants in the discussion "get it":
>
> 1. The "internal table" in V4L could not handle this problem, because the 
> internal table would be based upon what information? The USB Vendor:Product 
> number? For that matter, no other table could work, either. Well, actually, 
> a table could work. It would have to be inside the module supporting the 
> camera, and the matter of which table entry corresponds to what camera would 
> have to be settled by passing a USB command to the camera and then parsing 
> the camera's response. So now the question is how to get the information out 
> of the module, which can only be collected and analysed inside the module.
>
> 2. The "have a file where the user could edit" idea may seem attractive to 
> some, because it shoves the whole problem of agreeing on the appropriate way 
> to get needed information out of a kernel module and into userspace onto 
> someone not present during the current discussion, the user. However, this 
> is not a solution, and thinking about it just a little bit ought to make 
> that totally obvious. This is a strongly worded statement, so I will proceed 
> to explain why the matter is so obvious.
>
> Let us assume the very best, and assume that every app which is V4L 
> conformant has a "one time" initialization step and creates a directory 
> $HOME/.app containing stored settings. So we might have a file called 
> $HOME/.app/sq905. This file gets automatically written when the hapless user 
> hooks up his sq905 camera the first time, and has to go through a choice 
> routine to decide which side of the frame is up and which side is to the 
> left and to the right. One could even take serious steps to make this 
> otherwise unnecessary and silly sequence to be as really nice and "user 
> friendly" as it could possibly be, and set this all up to be done with a 
> sequence of mouse clicks and then the file, very kindly, gets written 
> automatically. Those of you who are thinking that this "file a user could 
> edit" is the way to go are, presumably, thinking along these lines. Well 
> there are at least four things which are obviously wrong with this 
> "solution":
>
> 2.1. The user is forced to deal with something which the user should not 
> even have to confront. The user is called upon to remedy an omission and a 
> deficiency which was ignored at a lower level, because a bunch of developers 
> could not come together on a reasonable course of action. Well, some don't 
> like to see this one, so there are three more reasons.
>
> 2.2 Every camera is going to require a file for itself in $HOME/.app even if 
> there is nothing that the user needs to do. Many of the supported cameras 
> need nothing of the kind, so this would be kind of silly.
>
> 2.3 The user has two apps for dealing with webcams. So now the user needs to 
> have another directory called $HOME/.app2 with similar files in it?
>
> 2.4 (and this one is the worst of all) The user has two cameras which are 
> both powered by the same kernel module, and the two cameras need two 
> different things done. Now what??? Both cameras can not simultaneously have 
> a valid entry in $HOME/.app/sq905 which tells what to do with the data out 
> of the camera. Because what is right for one of them is wrong for the other. 
> Further to add to this, one of the tests which was run on this very module 
> was to see if it will run two of the same kind of camera simultaneously. It 
> passed the test. Now someone wants the data to be right-side up from both 
> cameras, and correctly oriented left-to right. Is that an unreasonable 
> desire on the part of a user? Of course not. Again, now what????
>
> There is no way out of this series of dilemmas and absurdities with the 
> "file a user could edit" approach. None.
>
> So, please, could we get serious and actually come up with a reasonable 
> solution? To put the matter completely in perspective, the support for the 
> affected set of devices is otherwise quite complete, and the only thing 
> remaining is to come up with a reasonable solution, which would provide a 
> pleasing experience to users.

Theodore,

You're considering just one subset of the V4L usages: notebook webcams. If 
you think about non-notebook webcams [1] or about security cams, you'll 
see that the only way for you to have the flipping information is inside 
some userspace file.

For example, I have here one video input bttv board with 16 cameras,
connected on 4 bttv chips. I have also a Pelco camera, that has just one 
support socket. Depending on the place you mount it, the camera has to be 
rotated by 180 degrees. It can also be useful to mount it rotated by 90 
degrees.

After mounting the cameras, no matter what apps you are using (a streaming 
video app to broadcast it via internet, a security app, etc), the rotation
information for that input, on that particular PCI, bus won't change.

As we've standardized the VIDIOC_QUERYCAP bus_info, now, the information 
for the camera at input 3 of bttv hardware at PCI addres 03:00.3 is 
unique. It is also unique the same information for a notebook mounted 
webcam (since the USB bus ID won't change for that devices).

So, if we standardize where this information is stored, for example, at 
/etc/libv4l and/or at ~/.libv4lrc, and let libv4l handle such info, this 
will be something consistent for all applications using libv4l. Other apps 
that might not want to trust on libv4l can also read the info at the same 
file.

So, I really think that this should be part of the approach.

I agree that we should have a way to get a hint about a camera rotation,
if this is somehow provided by reading at the hardware.

[1] Normal webcams can be mounted on some hardware and have some 
orientations that would be different than expected. I've seen this before 
on some PC-based harware where the camera is enclosed inside the clause, 
like on Automatic Transfer Machines. Also, the user may want to use the 
camera on an inverted position, to make easy to fix it somewhere.

  -- 
Cheers,
Mauro Carvalho Chehab
http://linuxtv.org
mchehab@infradead.org

Re: [RFC] How to pass camera Orientation to userspace

[kilgota]

On Tue, 24 Feb 2009, Mauro Carvalho Chehab wrote:

> On Mon, 23 Feb 2009, kilgota@banach.math.auburn.edu wrote:


<big snip>


> Theodore,
>
> You're considering just one subset of the V4L usages: notebook webcams.

Actually, the sq905 cameras are not "notebook webcams." They are cheap, 
consumer entry level dual-mode cameras. They can be used as hand-held 
still cameras, to shoot still photos, and they can also be used as 
webcams. When sold, they usually came with some kind of mounting device 
that could hold them rigidly for webcam use. There are lots of similar 
cameras. Mercifully, not all of those others have the problems of the 
sq905, which have led to the present impasse.


If 
> you think about non-notebook webcams [1] or about security cams, you'll see 
> that the only way for you to have the flipping information is inside some 
> userspace file.

However, there are obvious differences. For those cameras the question 
might well come up about how to control the movement of the camera, or, at 
least, to be aware of which way the camera is pointing. For these, the 
topic is an inherent property of the particular model of the camera -- or 
a defect, if someone wants to say so. Since the property is not determined 
by USB number, it is inherently impossible outside of the module to create 
a table which contains the needed information.

My intention here was to re-focus attention on the original problem which 
brought up the current discussion, and to that end the problem must be 
clearly understood. To have the flipping information inside some userspace 
file might solve some other problem and may be a generally very good idea. 
But it will not, can not, and never will be able to solve this problem.

>
> For example, I have here one video input bttv board with 16 cameras,
> connected on 4 bttv chips. I have also a Pelco camera, that has just one 
> support socket. Depending on the place you mount it, the camera has to be 
> rotated by 180 degrees. It can also be useful to mount it rotated by 90 
> degrees.

Good. So one needs external controls and userspace tools. Did I ever say 
that such things should never be done? No. All I said was that there is a 
problem, presently on the table, and those kinds of things are not, can 
not be, never were, and never will be solutions for _this_ problem.

>
> After mounting the cameras, no matter what apps you are using (a streaming 
> video app to broadcast it via internet, a security app, etc), the rotation
> information for that input, on that particular PCI, bus won't change.
>
> As we've standardized the VIDIOC_QUERYCAP bus_info, now, the information for 
> the camera at input 3 of bttv hardware at PCI addres 03:00.3 is unique. It is 
> also unique the same information for a notebook mounted webcam (since the USB 
> bus ID won't change for that devices).

Errrm... Again, the cameras in question here are not notebook mounted 
webcams.

>
> So, if we standardize where this information is stored, for example, at 
> /etc/libv4l and/or at ~/.libv4lrc, and let libv4l handle such info, this will 
> be something consistent for all applications using libv4l. Other apps that 
> might not want to trust on libv4l can also read the info at the same file.

Sorry, this will not work here. It may solve some other problem, but not 
this problem. Or, if one wants to "store" the information there, I don't 
care, really, but then there needs to be a way to get the information from 
the module, where it is, and get written into said table, which is where 
you want it, and this needs to happen every time an sq905 camera gets 
plugged in -- without pestering the user about the matter every time that 
such a camera gets hooked up.

Comparison: I have tossed a coin. Is it going to come up heads? Or tails? 
It is possible to know which, because the coin has been tossed. It would 
not even be cheating to look at it, or allow someone who did look at it, 
to pass to us the information. But we are not going to look, and if 
someone tells us we will not listen because we have not agreed on what 
language to use for communication. Instead, we will put a guess about the 
outcome into a table. We will make it a nice table, which can be revised 
using nice GUI tools, so it is easy for the user. So if our guess is wrong 
let the user fix it. Then next time we toss the coin the table entry will 
be right because either it was right before, or now someone fixed it???

>
> So, I really think that this should be part of the approach.

I was not even addressing what should or should not be part of the 
approach to some other problems. My point was that such discussion is not 
germane to the problem of how to pass on the correct orientation of the 
sensor, for the sq905 cameras. There are lots of other problems out there 
to solve. No denying that.

Also an overview is often very helpful. Also trying to visualize what 
might be needed in the future is helpful. All of this can be extremely 
helpful. But not everyone can see or imagine every possible thing. For 
example, it seems that some of the best minds in the business are stunned 
when confronted with the fact that some manufacturer of cheap electronics 
in Taiwan has produced a lot of mass-market cameras with the sensors 
turned upside down, along with some other cameras having the same USB ID 
with different sensors, which act a bit differently. Clearly, if such a 
thing happened once it can happen again. So how to deal with such a
problem? Something similar or worse will surely come up again.

>
> I agree that we should have a way to get a hint about a camera rotation,
> if this is somehow provided by reading at the hardware.
>
> [1] Normal webcams can be mounted on some hardware and have some orientations 
> that would be different than expected. I've seen this before on some PC-based 
> harware where the camera is enclosed inside the clause, like on Automatic 
> Transfer Machines. Also, the user may want to use the camera on an inverted 
> position, to make easy to fix it somewhere.

Again, these are in fact separate issues. The fundamental issue required 
for supporting the sq905 cameras is that an agreed-upon method must exist 
by which precisely two pieces of information can be passed along by the 
module. Which of the two pieces of information is relevant and needs to be 
sent along is, alas, only known from within the module. That is the 
backdrop for the entire discussion. As far as I know, this thread 
would not exist if that need had not come up. These two pieces 
of information are, precisely, "frame data requires flipping across a 
horizontal axis unless camera is upside down" and "frame data requires 
flipping across a vertical axis unless camera is held up to a mirror"

So, what do these two deep questions, which confound the assembled wisdom 
of an entire list of Linux video developers, have to do with tables in 
userspace? None that I can see, unless someone wants to provide a 
mechanism for the information, having been collected in the 
module, to be available to the table in userspace.

Theodore Kilgore

Re: [RFC] How to pass camera Orientation to userspace

[Mauro Carvalho Chehab]

On Tue, 24 Feb 2009, kilgota@banach.math.auburn.edu wrote:

>
>
> On Tue, 24 Feb 2009, Mauro Carvalho Chehab wrote:
>
>>  On Mon, 23 Feb 2009, kilgota@banach.math.auburn.edu wrote:
>
>
> <big snip>
>
>
>>  Theodore,
>>
>>  You're considering just one subset of the V4L usages: notebook webcams.
>
> Actually, the sq905 cameras are not "notebook webcams." They are cheap, 
> consumer entry level dual-mode cameras. They can be used as hand-held still 
> cameras, to shoot still photos, and they can also be used as webcams. When 
> sold, they usually came with some kind of mounting device that could hold 
> them rigidly for webcam use. There are lots of similar cameras. Mercifully, 
> not all of those others have the problems of the sq905, which have led to 
> the present impasse.
>
>
> If
>>  you think about non-notebook webcams [1] or about security cams, you'll
>>  see that the only way for you to have the flipping information is inside
>>  some userspace file.
>
> However, there are obvious differences. For those cameras the question might 
> well come up about how to control the movement of the camera, or, at least, 
> to be aware of which way the camera is pointing. For these, the topic is an 
> inherent property of the particular model of the camera -- or a defect, if 
> someone wants to say so. Since the property is not determined by USB number, 
> it is inherently impossible outside of the module to create a table which 
> contains the needed information.
>
> My intention here was to re-focus attention on the original problem which 
> brought up the current discussion, and to that end the problem must be 
> clearly understood. To have the flipping information inside some userspace 
> file might solve some other problem and may be a generally very good idea. 
> But it will not, can not, and never will be able to solve this problem.
>
>>
>>  For example, I have here one video input bttv board with 16 cameras,
>>  connected on 4 bttv chips. I have also a Pelco camera, that has just one
>>  support socket. Depending on the place you mount it, the camera has to be
>>  rotated by 180 degrees. It can also be useful to mount it rotated by 90
>>  degrees.
>
> Good. So one needs external controls and userspace tools. Did I ever say 
> that such things should never be done? No. All I said was that there is a 
> problem, presently on the table, and those kinds of things are not, can not 
> be, never were, and never will be solutions for _this_ problem.
>
>>
>>  After mounting the cameras, no matter what apps you are using (a
>>  streaming video app to broadcast it via internet, a security app, etc),
>>  the rotation
>>  information for that input, on that particular PCI, bus won't change.
>>
>>  As we've standardized the VIDIOC_QUERYCAP bus_info, now, the information
>>  for the camera at input 3 of bttv hardware at PCI addres 03:00.3 is
>>  unique. It is also unique the same information for a notebook mounted
>>  webcam (since the USB bus ID won't change for that devices).
>
> Errrm... Again, the cameras in question here are not notebook mounted 
> webcams.
>
>>
>>  So, if we standardize where this information is stored, for example, at
>>  /etc/libv4l and/or at ~/.libv4lrc, and let libv4l handle such info, this
>>  will be something consistent for all applications using libv4l. Other
>>  apps that might not want to trust on libv4l can also read the info at the
>>  same file.
>
> Sorry, this will not work here. It may solve some other problem, but not 
> this problem. Or, if one wants to "store" the information there, I don't 
> care, really, but then there needs to be a way to get the information from 
> the module, where it is, and get written into said table, which is where you 
> want it, and this needs to happen every time an sq905 camera gets plugged in 
> -- without pestering the user about the matter every time that such a camera 
> gets hooked up.
>
> Comparison: I have tossed a coin. Is it going to come up heads? Or tails? It 
> is possible to know which, because the coin has been tossed. It would not 
> even be cheating to look at it, or allow someone who did look at it, to pass 
> to us the information. But we are not going to look, and if someone tells us 
> we will not listen because we have not agreed on what language to use for 
> communication. Instead, we will put a guess about the outcome into a table. 
> We will make it a nice table, which can be revised using nice GUI tools, so 
> it is easy for the user. So if our guess is wrong let the user fix it. Then 
> next time we toss the coin the table entry will be right because either it 
> was right before, or now someone fixed it???
>
>>
>>  So, I really think that this should be part of the approach.
>
> I was not even addressing what should or should not be part of the approach 
> to some other problems. My point was that such discussion is not germane to 
> the problem of how to pass on the correct orientation of the sensor, for the 
> sq905 cameras. There are lots of other problems out there to solve. No 
> denying that.
>
> Also an overview is often very helpful. Also trying to visualize what might 
> be needed in the future is helpful. All of this can be extremely helpful. 
> But not everyone can see or imagine every possible thing. For example, it 
> seems that some of the best minds in the business are stunned when 
> confronted with the fact that some manufacturer of cheap electronics in 
> Taiwan has produced a lot of mass-market cameras with the sensors turned 
> upside down, along with some other cameras having the same USB ID with 
> different sensors, which act a bit differently. Clearly, if such a thing 
> happened once it can happen again. So how to deal with such a
> problem? Something similar or worse will surely come up again.
>
>>
>>  I agree that we should have a way to get a hint about a camera rotation,
>>  if this is somehow provided by reading at the hardware.
>>
>>  [1] Normal webcams can be mounted on some hardware and have some
>>  orientations that would be different than expected. I've seen this before
>>  on some PC-based harware where the camera is enclosed inside the clause,
>>  like on Automatic Transfer Machines. Also, the user may want to use the
>>  camera on an inverted position, to make easy to fix it somewhere.
>
> Again, these are in fact separate issues. The fundamental issue required for 
> supporting the sq905 cameras is that an agreed-upon method must exist by 
> which precisely two pieces of information can be passed along by the module. 
> Which of the two pieces of information is relevant and needs to be sent 
> along is, alas, only known from within the module. That is the backdrop for 
> the entire discussion. As far as I know, this thread would not exist if that 
> need had not come up. These two pieces of information are, precisely, "frame 
> data requires flipping across a horizontal axis unless camera is upside 
> down" and "frame data requires flipping across a vertical axis unless camera 
> is held up to a mirror"
>
> So, what do these two deep questions, which confound the assembled wisdom of 
> an entire list of Linux video developers, have to do with tables in 
> userspace? None that I can see, unless someone wants to provide a mechanism 
> for the information, having been collected in the module, to be available to 
> the table in userspace.

I'm not saying that userspace tables would solve all problems. I'm just 
saying that this should be part of the solution.

For sure we need to have a way for retrieving this information for devices 
like the sq905 cameras, where the information can't be currently be 
determined by userspace.

In the case of sq905, this information is static, right? If so, IMO, the 
better approach is to use a flag at the v4l2_input, as already discussed 
in this thread.

-- 
Cheers,
Mauro Carvalho Chehab
http://linuxtv.org
mchehab@infradead.org

Re: [RFC] How to pass camera Orientation to userspace

[kilgota]

On Tue, 24 Feb 2009, Mauro Carvalho Chehab wrote:

> On Tue, 24 Feb 2009, kilgota@banach.math.auburn.edu wrote:
>
>> 
>> 
>> On Tue, 24 Feb 2009, Mauro Carvalho Chehab wrote:
>>
>>>  On Mon, 23 Feb 2009, kilgota@banach.math.auburn.edu wrote:
>> 
>> 
>> <big snip>
>> 
>>
>>>  Theodore,
>>>
>>>  You're considering just one subset of the V4L usages: notebook webcams.
>> 
>> Actually, the sq905 cameras are not "notebook webcams." They are cheap, 
>> consumer entry level dual-mode cameras. They can be used as hand-held still 
>> cameras, to shoot still photos, and they can also be used as webcams. When 
>> sold, they usually came with some kind of mounting device that could hold 
>> them rigidly for webcam use. There are lots of similar cameras. Mercifully, 
>> not all of those others have the problems of the sq905, which have led to 
>> the present impasse.
>> 
>> 
>> If
>>>  you think about non-notebook webcams [1] or about security cams, you'll
>>>  see that the only way for you to have the flipping information is inside
>>>  some userspace file.
>> 
>> However, there are obvious differences. For those cameras the question 
>> might well come up about how to control the movement of the camera, or, at 
>> least, to be aware of which way the camera is pointing. For these, the 
>> topic is an inherent property of the particular model of the camera -- or a 
>> defect, if someone wants to say so. Since the property is not determined by 
>> USB number, it is inherently impossible outside of the module to create a 
>> table which contains the needed information.
>> 
>> My intention here was to re-focus attention on the original problem which 
>> brought up the current discussion, and to that end the problem must be 
>> clearly understood. To have the flipping information inside some userspace 
>> file might solve some other problem and may be a generally very good idea. 
>> But it will not, can not, and never will be able to solve this problem.
>> 
>>>
>>>  For example, I have here one video input bttv board with 16 cameras,
>>>  connected on 4 bttv chips. I have also a Pelco camera, that has just one
>>>  support socket. Depending on the place you mount it, the camera has to be
>>>  rotated by 180 degrees. It can also be useful to mount it rotated by 90
>>>  degrees.
>> 
>> Good. So one needs external controls and userspace tools. Did I ever say 
>> that such things should never be done? No. All I said was that there is a 
>> problem, presently on the table, and those kinds of things are not, can not 
>> be, never were, and never will be solutions for _this_ problem.
>> 
>>>
>>>  After mounting the cameras, no matter what apps you are using (a
>>>  streaming video app to broadcast it via internet, a security app, etc),
>>>  the rotation
>>>  information for that input, on that particular PCI, bus won't change.
>>>
>>>  As we've standardized the VIDIOC_QUERYCAP bus_info, now, the information
>>>  for the camera at input 3 of bttv hardware at PCI addres 03:00.3 is
>>>  unique. It is also unique the same information for a notebook mounted
>>>  webcam (since the USB bus ID won't change for that devices).
>> 
>> Errrm... Again, the cameras in question here are not notebook mounted 
>> webcams.
>> 
>>>
>>>  So, if we standardize where this information is stored, for example, at
>>>  /etc/libv4l and/or at ~/.libv4lrc, and let libv4l handle such info, this
>>>  will be something consistent for all applications using libv4l. Other
>>>  apps that might not want to trust on libv4l can also read the info at the
>>>  same file.
>> 
>> Sorry, this will not work here. It may solve some other problem, but not 
>> this problem. Or, if one wants to "store" the information there, I don't 
>> care, really, but then there needs to be a way to get the information from 
>> the module, where it is, and get written into said table, which is where 
>> you want it, and this needs to happen every time an sq905 camera gets 
>> plugged in -- without pestering the user about the matter every time that 
>> such a camera gets hooked up.
>> 
>> Comparison: I have tossed a coin. Is it going to come up heads? Or tails? 
>> It is possible to know which, because the coin has been tossed. It would 
>> not even be cheating to look at it, or allow someone who did look at it, to 
>> pass to us the information. But we are not going to look, and if someone 
>> tells us we will not listen because we have not agreed on what language to 
>> use for communication. Instead, we will put a guess about the outcome into 
>> a table. We will make it a nice table, which can be revised using nice GUI 
>> tools, so it is easy for the user. So if our guess is wrong let the user 
>> fix it. Then next time we toss the coin the table entry will be right 
>> because either it was right before, or now someone fixed it???
>> 
>>>
>>>  So, I really think that this should be part of the approach.
>> 
>> I was not even addressing what should or should not be part of the approach 
>> to some other problems. My point was that such discussion is not germane to 
>> the problem of how to pass on the correct orientation of the sensor, for 
>> the sq905 cameras. There are lots of other problems out there to solve. No 
>> denying that.
>> 
>> Also an overview is often very helpful. Also trying to visualize what might 
>> be needed in the future is helpful. All of this can be extremely helpful. 
>> But not everyone can see or imagine every possible thing. For example, it 
>> seems that some of the best minds in the business are stunned when 
>> confronted with the fact that some manufacturer of cheap electronics in 
>> Taiwan has produced a lot of mass-market cameras with the sensors turned 
>> upside down, along with some other cameras having the same USB ID with 
>> different sensors, which act a bit differently. Clearly, if such a thing 
>> happened once it can happen again. So how to deal with such a
>> problem? Something similar or worse will surely come up again.
>> 
>>>
>>>  I agree that we should have a way to get a hint about a camera rotation,
>>>  if this is somehow provided by reading at the hardware.
>>>
>>>  [1] Normal webcams can be mounted on some hardware and have some
>>>  orientations that would be different than expected. I've seen this before
>>>  on some PC-based harware where the camera is enclosed inside the clause,
>>>  like on Automatic Transfer Machines. Also, the user may want to use the
>>>  camera on an inverted position, to make easy to fix it somewhere.
>> 
>> Again, these are in fact separate issues. The fundamental issue required 
>> for supporting the sq905 cameras is that an agreed-upon method must exist 
>> by which precisely two pieces of information can be passed along by the 
>> module. Which of the two pieces of information is relevant and needs to be 
>> sent along is, alas, only known from within the module. That is the 
>> backdrop for the entire discussion. As far as I know, this thread would not 
>> exist if that need had not come up. These two pieces of information are, 
>> precisely, "frame data requires flipping across a horizontal axis unless 
>> camera is upside down" and "frame data requires flipping across a vertical 
>> axis unless camera is held up to a mirror"
>> 
>> So, what do these two deep questions, which confound the assembled wisdom 
>> of an entire list of Linux video developers, have to do with tables in 
>> userspace? None that I can see, unless someone wants to provide a mechanism 
>> for the information, having been collected in the module, to be available 
>> to the table in userspace.
>
> I'm not saying that userspace tables would solve all problems. I'm just 
> saying that this should be part of the solution.

I meant what I said. I do not see how that tables in userspace carry any 
relevance at all to this particular problem. Perhaps I am dense. Might 
tables in userspace help with the solution of some other problems? Well, 
yes. But then we are discussing those problems, not this one.

>
> For sure we need to have a way for retrieving this information for devices 
> like the sq905 cameras, where the information can't be currently be 
> determined by userspace.

Yes, indeed. Except for just one word, "currently." It does not fit here. 
The matter is not one of present inability, just because we are not clever 
enough. The logic of the situation is inexorable. Well, in part I take 
that back. If the operating system were fundamentally redesigned, making 
it possible to run a video device completely in userspace, then and only 
then the needed information could be determined by userspace. But I do not 
expect that will happen very soon.

>
> In the case of sq905, this information is static, right?

Yes. Exactly. You get the data the way it came out. That's it. No choices 
about that are possible. Precisely how the data came out depends only on 
the camera. An inquiry to the camera before streaming is started can 
provide the relevant information.

If so, IMO, the 
> better approach is to use a flag at the v4l2_input, as already discussed in 
> this thread.

OK.

Theodore Kilgore

Re: [RFC] How to pass camera Orientation to userspace

[Thomas Kaiser]

>> Also an overview is often very helpful. Also trying to visualize what 
>> might be needed in the future is helpful. All of this can be extremely 
>> helpful. But not everyone can see or imagine every possible thing. For 
>> example, it seems that some of the best minds in the business are 
>> stunned when confronted with the fact that some manufacturer of cheap 
>> electronics in Taiwan has produced a lot of mass-market cameras with 
>> the sensors turned upside down, along with some other cameras having 
>> the same USB ID with different sensors, which act a bit differently. 
>> Clearly, if such a thing happened once it can happen again. So how to 
>> deal with such a
>> problem?

Actually, this happens and is happening!

Just step back a get an other view.

These consumer products are manly produced for the Windoz audience.

After introduction of Win XP the consumer where told that USB device 
will run out of the box in Win XP, which is sometimes true, but .....

But on all (Windowz) Webcams (are Linux Webcams available?) I buy, I 
find a sticker which tells me to first insert the driver CD before 
connecting the cam to the PC. When you do, like instructed, your cam 
works like you expected!

Evan the USB ID is the same like the other webcam from the other vendor, 
you are (more or less) forced to install the driver from this particular 
vendor, you get a new driver! Doesn't matter if the sensor is mounted 
upside down, the "new" driver takes care about this. So, it looks like 
the cam in the Windowz World just works because you were forced to 
install the driver from the CD.

So I guess the Windoz diver just knows more then the USB ID.

In the Linux World most of the drive are re-engineered, we don't know 
how to detect how the sensor is mounted, do we?

Actually, what I try to say, is that only the cam can know how the 
sensor is mounted. Thus, the kernel module has to provide this 
information to user space (by query the hardware).

The "pivot" is an other thing.

Thomas

Re: [RFC] How to pass camera Orientation to userspace

[Mauro Carvalho Chehab]

On Tue, 24 Feb 2009 20:12:00 -0600 (CST)
kilgota@banach.math.auburn.edu wrote:

> > For sure we need to have a way for retrieving this information for devices 
> > like the sq905 cameras, where the information can't be currently be 
> > determined by userspace.  
> 
> Yes, indeed. Except for just one word, "currently." It does not fit here. 

Yes, it fits, since, after implementing the API, the userspace will have this
information by using the agreed API.

> > If so, IMO, the 
> > better approach is to use a flag at the v4l2_input, as already discussed in 
> > this thread.  
> 
> OK.

As it seems to be a consensus that the better is to use a flag inside
v4l2_input, could you please provide us an RFC patch to implement it and update
the V4L2 spec?

Cheers,
Mauro

Re: [RFC] How to pass camera Orientation to userspace

[kilgota]

On Wed, 25 Feb 2009, Thomas Kaiser wrote:

>>> Also an overview is often very helpful. Also trying to visualize what 
>>> might be needed in the future is helpful. All of this can be extremely 
>>> helpful. But not everyone can see or imagine every possible thing. For 
>>> example, it seems that some of the best minds in the business are stunned 
>>> when confronted with the fact that some manufacturer of cheap electronics 
>>> in Taiwan has produced a lot of mass-market cameras with the sensors 
>>> turned upside down, along with some other cameras having the same USB ID 
>>> with different sensors, which act a bit differently. Clearly, if such a 
>>> thing happened once it can happen again. So how to deal with such a
>>> problem?
>
> Actually, this happens and is happening!
>
> Just step back a get an other view.

I had plenty of other views. SQ905, SQ905C, MR97310, SN9C2028, and one or 
two more. And, oh yes, the GT8290 chip on which Grandtech went broke, 
which was intended for mass storage still cameras but had a 32-bit CBW 
instead of a 31-bit CBW (off by one error and nobody caught it).

>
> These consumer products are manly produced for the Windoz audience.


Very true. Usually, it even says on the package that it will not work on 
Mac. But since Linux is unmentionable they did not say anything about it. 
So we make it work anyway.

>
> After introduction of Win XP the consumer where told that USB device will run 
> out of the box in Win XP,

Not all devices. Perhaps what they really meant is now you don't need to 
install two drivers if you have two Mass Storage Transparent Scsi Bulk 
Transport flash drives. To that extent, it seems that they were truthful. 
There are lots of proprietary devices out there for which AFAICT the 
drivers are still not included in Windows. Some of these are unfinished 
projects of mine, too, such as the JL2005C cameras. The big bugbear with 
those is the compression algorithm. It is a horror.

> which is sometimes true, but .....
>
> But on all (Windowz) Webcams (are Linux Webcams available?) I buy, I find a 
> sticker which tells me to first insert the driver CD before connecting the 
> cam to the PC. When you do, like instructed, your cam works like you 
> expected!

Of course. Hardware will not work without a driver.

>
> Evan the USB ID is the same like the other webcam from the other vendor, you 
> are (more or less) forced to install the driver from this particular vendor, 
> you get a new driver!

Not true. Windows, even back in the days of Windows 98, searched for the 
Vendor:Product number to look for the driver. There were several places to 
search for the number. For example, the INF files. And I have a box full 
of SQ905 cameras for which I will personally guarantee that every one of 
them will work on Windows 98 with the Windows driver from the CD that came 
with any other one. Now, if the picture is always facing in the right 
direction, that is another question, naturally.

Doesn't matter if the sensor is mounted upside down, 
> the "new" driver takes care about this. So, it looks like the cam in the 
> Windowz World just works because you were forced to install the driver from 
> the CD.

And while we are on that topic, I could definitely assert that not always 
did the "manufacturer" get this right, either. I, recall, developed the 
original SQ905 stillcam driver for Linux. I got letters from users and 
sample photos, too, where the photos came out of the camera and lettering 
on a sign visible in the photo was bass ackwards. They were cheap cameras, 
and the people who sold some of them were just a little bit sloppy. Their 
virtue, as cheap cameras go, was comparatively good optics in some of 
them. I have seen lots of cameras about the same price, which had much 
more features and much worse pictures.

>
> So I guess the Windoz diver just knows more then the USB ID.

No, not really. See above.

>
> In the Linux World most of the drive are re-engineered, we don't know how to 
> detect how the sensor is mounted, do we?


Well, yes, we do. And that is what this discussion was about. How to use 
that knowledge constructively while writing a kernel driver.

>
> Actually, what I try to say, is that only the cam can know how the sensor is 
> mounted. Thus, the kernel module has to provide this information to user 
> space (by query the hardware).

Well, that is more like it. Yes, one does have to ask the camera. But the 
camera will tell its answer truthfully. And this is before any streaming 
and image processing has started, too. What a deal.

>
> The "pivot" is an other thing.

Very true. And worth paying attention to. But it is not the same issue.

>
> Thomas
>

Theodore Kilgore

Re: [RFC] How to pass camera Orientation to userspace

[kilgota]

On Wed, 25 Feb 2009, Mauro Carvalho Chehab wrote:

> On Tue, 24 Feb 2009 20:12:00 -0600 (CST)
> kilgota@banach.math.auburn.edu wrote:
>
>>> For sure we need to have a way for retrieving this information for devices
>>> like the sq905 cameras, where the information can't be currently be
>>> determined by userspace.
>>
>> Yes, indeed. Except for just one word, "currently." It does not fit here.
>
> Yes, it fits, since, after implementing the API, the userspace will have this
> information by using the agreed API.
>
>>> If so, IMO, the
>>> better approach is to use a flag at the v4l2_input, as already discussed in
>>> this thread.
>>
>> OK.
>
> As it seems to be a consensus that the better is to use a flag inside
> v4l2_input, could you please provide us an RFC patch to implement it and update
> the V4L2 spec?

With respect, I think that it would be better to consult with some of the 
other people who are most closely involved in the issue. This would 
include Jean-Francois (one obviously does not want to come up with some 
scheme which would interfere with other camera drivers, and he has a good 
overview on that) and Hans de Goede (v4l will use the new method) and Adam 
Baker. I will be happy to participate in discussions with them and work 
something up.

Thank you very much for helping to come to an apparent consensus about 
what kind of characteristics that the solution has to have. I appreciate 
your attention to the issue, and at this point I hope for a good and a 
fairly quick resolution.

Theodore Kilgore

Re: [RFC] How to pass camera Orientation to userspace

[Hans de Goede]

<really big snip>

>> So, what do these two deep questions, which confound the assembled 
>> wisdom of an entire list of Linux video developers, have to do with 
>> tables in userspace? None that I can see, unless someone wants to 
>> provide a mechanism for the information, having been collected in the 
>> module, to be available to the table in userspace.
> 
> I'm not saying that userspace tables would solve all problems. I'm just 
> saying that this should be part of the solution.
> 
> For sure we need to have a way for retrieving this information for 
> devices like the sq905 cameras, where the information can't be currently 
> be determined by userspace.
> 
> In the case of sq905, this information is static, right? If so, IMO, the 
> better approach is to use a flag at the v4l2_input, as already discussed 
> in this thread.
> 

Yes we all seem to agree on this. Adam, since you started this thread can you 
write a small RFC with that solution worked out with proposed videodev2.h changes?

Note that I'm only talking about input flags for the orientation problem and 
not the pivotting problem. I think the pivotting problem may need some more 
discussion. But since we all seem to be in agreement wrt to orientation problem 
and specifically the sq905 problem, lets do one more RFC, then everyone does a 
+1 to that and we move forward with this as a solution for the orientation problem.

Regards,

Hans


p.s.

For the pivotting problem I'm tending towards a special control class which 
contains read-only controls which are really camera properties. This will allow 
us to cope with any granularity of pivoting sensors. This could then also be 
used for in example aperture. But lets start a new thread for that.

Re: [RFC] How to pass camera Orientation to userspace

[Thomas Kaiser]

 >> Actually, this happens and is happening!

I got OT. I just needed to vent!

Sorry for the spam.

For the sensor mounting, I think the cam knows how the sensor is mounted 
and therefor the driver knows (driver -> first abstraction layer between 
software and hardware). Therefor the drive has to report the sensor 
orientation to user space (read only).

Thomas

PS: May be, I am OT all ready.

[PATCH] libv4lconvert support for SQ905C decompression

[kilgota]

Hans,

Below is a patch for libv4lconvert, to support the decompression used by 
the SQ905C cameras (0x2770:0x905C) and some other related cameras. There 
is at the moment no support module for these cameras in streaming mode, 
but I intend to submit one.

This contribution was created in whole by me, based upon code in 
libgphoto2 which was created in whole by me, and which was licensed for
libgphoto2 under the LGPL license.

Signed-off-by: Theodore Kilgore <kilgota@auburn.edu>

The content of the file sq905c.patch follows 
-----------------------------------------------------------------------
diff -uprN libv4lconvert-old/Makefile libv4lconvert-new/Makefile
--- libv4lconvert-old/Makefile	2009-03-01 15:37:38.000000000 -0600
+++ libv4lconvert-new/Makefile	2009-03-01 14:19:14.000000000 -0600
@@ -12,7 +12,7 @@ endif

  CONVERT_OBJS  = libv4lconvert.o tinyjpeg.o sn9c10x.o sn9c20x.o pac207.o \
  		mr97310a.o flip.o crop.o jidctflt.o spca561-decompress.o \
-		rgbyuv.o spca501.o bayer.o
+		rgbyuv.o spca501.o sq905c.o bayer.o
  TARGETS       = $(CONVERT_LIB) libv4lconvert.pc
  INCLUDES      = ../include/libv4lconvert.h

diff -uprN libv4lconvert-old/libv4lconvert-priv.h libv4lconvert-new/libv4lconvert-priv.h
--- libv4lconvert-old/libv4lconvert-priv.h	2009-03-01 15:37:38.000000000 -0600
+++ libv4lconvert-new/libv4lconvert-priv.h	2009-03-01 17:12:02.000000000 -0600
@@ -47,6 +47,10 @@
  #define V4L2_PIX_FMT_MR97310A v4l2_fourcc('M','3','1','0')
  #endif

+#ifndef V4L2_PIX_FMT_SQ905C
+#define V4L2_PIX_FMT_SQ905C v4l2_fourcc('9','0','5','C')
+#endif
+
  #ifndef V4L2_PIX_FMT_PJPG
  #define V4L2_PIX_FMT_PJPG v4l2_fourcc('P', 'J', 'P', 'G')
  #endif
@@ -180,6 +184,9 @@ void v4lconvert_decode_pac207(const unsi
  void v4lconvert_decode_mr97310a(const unsigned char *src, unsigned char *dst,
    int width, int height);

+void v4lconvert_decode_sq905c(const unsigned char *src, unsigned char *dst,
+  int width, int height);
+
  void v4lconvert_bayer_to_rgb24(const unsigned char *bayer,
    unsigned char *rgb, int width, int height, unsigned int pixfmt);

diff -uprN libv4lconvert-old/libv4lconvert.c libv4lconvert-new/libv4lconvert.c
--- libv4lconvert-old/libv4lconvert.c	2009-03-01 15:37:38.000000000 -0600
+++ libv4lconvert-new/libv4lconvert.c	2009-03-01 17:13:42.000000000 -0600
@@ -61,6 +61,7 @@ static const struct v4lconvert_pixfmt su
    { V4L2_PIX_FMT_SN9C10X,      V4LCONVERT_COMPRESSED },
    { V4L2_PIX_FMT_PAC207,       V4LCONVERT_COMPRESSED },
    { V4L2_PIX_FMT_MR97310A,     V4LCONVERT_COMPRESSED },
+  { V4L2_PIX_FMT_SQ905C,       V4LCONVERT_COMPRESSED },
    { V4L2_PIX_FMT_PJPG,         V4LCONVERT_COMPRESSED },
  };

@@ -608,6 +609,7 @@ static int v4lconvert_convert_pixfmt(str
      case V4L2_PIX_FMT_SN9C10X:
      case V4L2_PIX_FMT_PAC207:
      case V4L2_PIX_FMT_MR97310A:
+    case V4L2_PIX_FMT_SQ905C:
      {
        unsigned char *tmpbuf;

@@ -633,6 +635,10 @@ static int v4lconvert_convert_pixfmt(str
  	  v4lconvert_decode_mr97310a(src, tmpbuf, width, height);
  	  src_pix_fmt = V4L2_PIX_FMT_SBGGR8;
  	  break;
+	case V4L2_PIX_FMT_SQ905C:
+	  v4lconvert_decode_sq905c(src, tmpbuf, width, height);
+	  src_pix_fmt = V4L2_PIX_FMT_SRGGB8;
+	  break;
        }
        src = tmpbuf;
        /* Deliberate fall through to raw bayer fmt code! */
diff -uprN libv4lconvert-old/sq905c.c libv4lconvert-new/sq905c.c
--- libv4lconvert-old/sq905c.c	1969-12-31 18:00:00.000000000 -0600
+++ libv4lconvert-new/sq905c.c	2009-03-01 17:08:12.000000000 -0600
@@ -0,0 +1,222 @@
+/*
+ * sq905c.c
+ *
+ * Here is the decompression function for the SQ905C cameras. The functions
+ * used are adapted from the libgphoto2 functions for the same cameras, 
+ * which was 
+ * Copyright (c) 2005 and 2007 Theodore Kilgore <kilgota@auburn.edu>
+ * This version for libv4lconvert is 
+ * Copyright (c) 2009 Theodore Kilgore <kilgota@auburn.edu>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation; either version 2.1 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ */ 
+
+#include <stdlib.h>
+
+#include "libv4lconvert-priv.h"
+
+
+#define CLIP(x) ((x)<0?0:((x)>0xff)?0xff:(x))
+
+
+static int
+sq905c_first_decompress (unsigned char *output, unsigned char *input,
+					    unsigned int outputsize)
+{
+	unsigned char parity = 0;
+	unsigned char nibble_to_keep[2];
+	unsigned char temp1 = 0, temp2 = 0;
+	unsigned char input_byte;
+	unsigned char lookup = 0;
+	unsigned int i = 0;
+	unsigned int bytes_used = 0;
+	unsigned int bytes_done = 0;
+	unsigned int bit_counter = 8;
+	unsigned int cycles = 0;
+	int table[9] = { -1, 0, 2, 6, 0x0e, 0x0e, 0x0e, 0x0e, 0xfb};
+	unsigned char lookup_table[16]
+		     ={0, 2, 6, 0x0e, 0xf0, 0xf1, 0xf2, 0xf3, 0xf4,
+		           0xf5, 0xf6, 0xf7, 0xf8, 0xf9, 0xfa, 0xfb};
+	unsigned char translator[16] = {8,7,9,6,10,11,12,13,14,15,5,4,3,2,1,0};
+
+	nibble_to_keep[0] = 0;
+	nibble_to_keep[1] = 0;
+
+        while (bytes_done < outputsize) {
+        	while (parity < 2 ) {
+			while ( lookup > table[cycles]) {
+				if (bit_counter == 8) {
+        				input_byte = input[bytes_used];
+        				bytes_used ++;
+					temp1 = input_byte;
+        				bit_counter = 0;
+        			}
+				input_byte = temp1;
+				temp2 = (temp2 << 1) & 0xFF;
+				input_byte = input_byte >> 7;
+				temp2 = temp2 | input_byte;
+				temp1 = (temp1 <<1)&0xFF;
+				bit_counter ++ ;
+    				cycles ++ ;
+        			if (cycles > 9) {
+					return -1; 
+    				}
+        			lookup = temp2 & 0xff; 
+			}
+			temp2 = 0;
+			for (i=0; i < 17; i++ ) {
+				if (lookup == lookup_table[i] ) {
+					nibble_to_keep[parity] = translator[i];
+					break; 
+				}
+				if (i == 16) {
+					return -1;
+				} 
+			} 
+        		cycles = 0;
+        		parity ++ ;
+        	} 
+            	output[bytes_done] = (nibble_to_keep[0]<<4)|nibble_to_keep[1];
+            	bytes_done++;
+        	parity = 0; 
+        }
+        return 0;
+}
+
+static int
+sq905c_second_decompress (unsigned char *uncomp, unsigned char *in, 
+						    int width, int height)
+{
+	int diff = 0;
+	int tempval = 0;
+	int i, m, parity;
+	unsigned char delta_left = 0;
+	unsigned char delta_right = 0;
+	int input_counter = 0;
+	int delta_table[] = {-144, -110, -77, -53, -35, -21, -11, -3,
+				2, 10, 20, 34, 52, 76, 110, 144};
+	unsigned char *templine_red;
+	unsigned char *templine_green;
+	unsigned char *templine_blue;
+	templine_red = malloc(width);
+	if (!templine_red) {
+		free(templine_red);
+		return -1;
+	} 
+	for(i=0; i < width; i++){
+	    templine_red[i] = 0x80;
+	}
+	templine_green = malloc(width); 
+	if (!templine_green) {
+		free(templine_green);
+		return -1;
+	} 
+	for(i=0; i < width; i++){
+	    templine_green[i] = 0x80;
+	}
+	templine_blue = malloc(width); 
+	if (!templine_blue) {
+		free(templine_blue);
+		return -1;
+	} 
+	for(i=0; i < width; i++){
+	    templine_blue[i] = 0x80;
+	}
+	for (m=0; m < height/2; m++) {
+		/* First we do an even-numbered line */
+		for (i=0; i< width/2; i++) {
+			parity = i&1;
+	    		delta_right = in[input_counter] &0x0f;
+			delta_left = (in[input_counter]>>4)&0xff;
+			input_counter ++;
+			/* left pixel (red) */
+			diff = delta_table[delta_left];
+			if (!i) 
+				tempval = templine_red[0] + diff;
+			else 
+				tempval = (templine_red[i]
+				        + uncomp[2*m*width+2*i-2])/2 + diff;
+			tempval = CLIP(tempval);
+			uncomp[2*m*width+2*i] = tempval;
+			templine_red[i] = tempval;
+			/* right pixel (green) */
+			diff = delta_table[delta_right];
+			if (!i) 
+				tempval = templine_green[1] + diff;
+			else if (2*i == width - 2 ) 
+				tempval = (templine_green[i]
+						+ uncomp[2*m*width+2*i-1])/2 
+							+ diff;
+			else
+				tempval = (templine_green[i+1]
+						+ uncomp[2*m*width+2*i-1])/2 
+							+ diff;
+			tempval = CLIP(tempval);
+			uncomp[2*m*width+2*i+1] = tempval;
+			templine_green[i] = tempval;
+		}
+		/* then an odd-numbered line */
+		for (i=0; i< width/2; i++) {
+			delta_right = in[input_counter] &0x0f;
+			delta_left = (in[input_counter]>>4)&0xff;
+			input_counter ++;
+			/* left pixel (green) */
+			diff = delta_table[delta_left];
+			if (!i) 
+				tempval = templine_green[0] + diff;
+			else 
+				tempval = (templine_green[i]
+				    	    + uncomp[(2*m+1)*width+2*i-2])/2 
+						+ diff;
+			tempval = CLIP(tempval);
+			uncomp[(2*m+1)*width+2*i] = tempval;
+			templine_green[i] = tempval;
+			/* right pixel (blue) */
+			diff = delta_table[delta_right];
+			if (!i) 
+				tempval = templine_blue[0] + diff;
+			else 
+				tempval = (templine_blue[i]
+					    + uncomp[(2*m+1)*width+2*i-1])/2 
+						+ diff;
+			tempval = CLIP(tempval);
+			uncomp[(2*m+1)*width+2*i+1] = tempval;
+			templine_blue[i] = tempval;
+		}
+	}
+	free(templine_green);
+	free(templine_red);
+	free(templine_blue);
+	return 0;
+}
+
+void v4lconvert_decode_sq905c(const unsigned char *src, unsigned char *dst,
+  int width, int height)
+{
+	int size;
+	unsigned char *temp_data;
+	unsigned char *raw;
+	/* here we get rid of the 0x50 bytes of header in src. */
+	raw = src + 0x50;
+	size = width*height/2;
+	temp_data = malloc(size);
+	if (!temp_data) 
+		goto out; 
+	sq905c_first_decompress (temp_data, raw, size);
+	sq905c_second_decompress (dst, temp_data, width, height);
+out:
+	free(temp_data);
+}
+