USB mini-summit at LinuxCon Vancouver

USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

I'm pleased to announce a USB mini-summit at LinuxCon Vancouver.

What:	USB mini-summit
When:	Tuesday, August 16th, all day
Where:	At the conference venue, room TBD pending confirmation from
	Angela Brown.

Proposed topics include USB virtualization, and improved bandwidth APIs
between the USB core and drivers (especially webcam drivers).  See the
detailed topic list below.  Anyone is also welcome to propose or show up
with a USB related topic.  MUSB?  USB 3.0 gadget drivers?  USB-IP?

The USB mini-summit does overlap with the virtualization mini-summit by
a day, but I'm hoping we can schedule talks so some of the
virtualization folks can make it to the USB mini-summit.  The other
option was on Friday during the conference which was not ideal.

Proposed topics:

Topic 1
-------

The KVM folks suggested that it would be good to get USB and
virtualization developers together to talk about how to virtualize the
xHCI host controller.  The xHCI spec architect worked closely with
VMWare to get some extra goodies in the spec to help virtualization, and
I'd like to see the other virtualization developers take advantage of
that.  I'd also like us to hash out any issues they have been finding in
the USB core or xHCI driver during the virtualization effort.


Topic 2
-------

I'd also like to get the V4L and audio developers who work with USB
devices together with the core USB folks to talk about bandwidth
management under xHCI.

One of the issues is that since the xHCI hardware does bandwidth
management, not the xHCI driver, a schedule that will take too much
bandwidth will get rejected much sooner than any USB driver currently
expects (during a call to usb_set_interface).  This poses issues, since
most USB video drivers negotiate the video size and frame rate after
they call usb_set_interface, so they don't know whether they can fall
back to a less bandwidth-intensive setting.  Currently, they just submit
URBs with less and less bandwidth until one interval setting gets
accepted that won't work under xHCI.

A second issue is that that some drivers need less bandwidth than the
device advertises, and the xHCI driver currently uses whatever periodic
interval the device advertises in its descriptors.  This is not what the
video/audio driver wants, especially in the case of buggy high speed
devices that advertise the interval in frames, not microframes.  There
needs to be some way for the drivers to communicate their bandwidth
needs to the USB core.  We've known about this issue for a while, and I
think it's time to get everyone in the same room and hash out an API.

(I will send out an API proposal later this month.)

Hope to see you there!

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
> Topic 1
> -------
> 
> The KVM folks suggested that it would be good to get USB and
> virtualization developers together to talk about how to virtualize the
> xHCI host controller.  The xHCI spec architect worked closely with
> VMWare to get some extra goodies in the spec to help virtualization, and
> I'd like to see the other virtualization developers take advantage of
> that.  I'd also like us to hash out any issues they have been finding in
> the USB core or xHCI driver during the virtualization effort.

Do people really want to virtualize the whole xHCI controller, or just
specific ports or devices to the guest operating system?

If just specific ports, would something like usbip be better for virtual
machines, with the USB traffic going over the network connection between
the guest/host?

> Hope to see you there!

Thanks for putting this together.

Do we need to sign up somewhere to give the organizers a sense of how
many people will be attending?

thanks,

greg k-h

Re: USB mini-summit at LinuxCon Vancouver

[Gerd Hoffmann]

   Hi,

>> The KVM folks suggested that it would be good to get USB and
>> virtualization developers together to talk about how to virtualize the
>> xHCI host controller.  The xHCI spec architect worked closely with
>> VMWare to get some extra goodies in the spec to help virtualization, and
>> I'd like to see the other virtualization developers take advantage of
>> that.  I'd also like us to hash out any issues they have been finding in
>> the USB core or xHCI driver during the virtualization effort.
>
> Do people really want to virtualize the whole xHCI controller, or just
> specific ports or devices to the guest operating system?

SR/IOV support is an optional xHCI feature.  As I understand it you can 
create a VF which looks like a real xHCI controller.  This is partly 
done in hardware and partly by software.  Then you can assign it some 
ressources (specific ports) and pass it to the guest.

> If just specific ports, would something like usbip be better for virtual
> machines, with the USB traffic going over the network connection between
> the guest/host?

There are several ways depending on the use case.  Usually the guest 
sees a (fully software emulated) host adapter with usb devices 
connected, where the usb devices can be (a) emulated too or (b) real usb 
devices passed through to the guest.  The later is done by passing the 
guests requests to the real device via usbfs.

One problem with emulating usb fully in software is the polling design 
of the hardware which makes the emulation quite cpu intensive.  Using a 
xHCI VF should help here alot, but works for the pass through use case 
only of course.

cheers,
   Gerd

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 06/10/2011 02:21 AM, Sarah Sharp wrote:
> I'm pleased to announce a USB mini-summit at LinuxCon Vancouver.
>
> What:	USB mini-summit
> When:	Tuesday, August 16th, all day
> Where:	At the conference venue, room TBD pending confirmation from
> 	Angela Brown.
>
> Proposed topics include USB virtualization, and improved bandwidth APIs
> between the USB core and drivers (especially webcam drivers).  See the
> detailed topic list below.  Anyone is also welcome to propose or show up
> with a USB related topic.  MUSB?  USB 3.0 gadget drivers?  USB-IP?
>

I would like to give a short presentation on / demo off the usb redirection
over the network (USB-IP replacement with a less dumb protocol) I've been
working on, followed by a questions and answers session.

For those who want to give this a try now, see:
http://hansdegoede.livejournal.com/9682.html

Sheets from the presentation I gave on this at FOSDEM:
http://people.fedoraproject.org/~jwrdegoede/usb-redir.pdf

> The USB mini-summit does overlap with the virtualization mini-summit by
> a day, but I'm hoping we can schedule talks so some of the
> virtualization folks can make it to the USB mini-summit.  The other
> option was on Friday during the conference which was not ideal.
>
> Proposed topics:
>
> Topic 1
> -------
>
> The KVM folks suggested that it would be good to get USB and
> virtualization developers together to talk about how to virtualize the
> xHCI host controller.  The xHCI spec architect worked closely with
> VMWare to get some extra goodies in the spec to help virtualization, and
> I'd like to see the other virtualization developers take advantage of
> that.  I'd also like us to hash out any issues they have been finding in
> the USB core or xHCI driver during the virtualization effort.

I'm not really happy with how the management / hand over of userspace
to kernel space drivers and vice versa works. This is a problem when
doing usb-redirection to a virtual machine. I think this is best discussed
in a separate thread, and then if needed further discussed during the
mini summit. I'll send a mail on this shortly. I'll use the same address
list as this mail, except that I'm going to cut linux-kernel from the CC list.


> Topic 2
> -------
>
> I'd also like to get the V4L and audio developers who work with USB
> devices together with the core USB folks to talk about bandwidth
> management under xHCI.
>
> One of the issues is that since the xHCI hardware does bandwidth
> management, not the xHCI driver, a schedule that will take too much
> bandwidth will get rejected much sooner than any USB driver currently
> expects (during a call to usb_set_interface).  This poses issues, since
> most USB video drivers negotiate the video size and frame rate after
> they call usb_set_interface, so they don't know whether they can fall
> back to a less bandwidth-intensive setting.  Currently, they just submit
> URBs with less and less bandwidth until one interval setting gets
> accepted that won't work under xHCI.
>
> A second issue is that that some drivers need less bandwidth than the
> device advertises, and the xHCI driver currently uses whatever periodic
> interval the device advertises in its descriptors.  This is not what the
> video/audio driver wants, especially in the case of buggy high speed
> devices that advertise the interval in frames, not microframes.  There
> needs to be some way for the drivers to communicate their bandwidth
> needs to the USB core.  We've known about this issue for a while, and I
> think it's time to get everyone in the same room and hash out an API.

Interesting, being able to tell the core how much bandwidth a device
will actually use (versus what it claims as maxpacketsize in its
descriptors) is something which we really need. I know of at least
2 usb1 webcam chipsets (and drivers) which have only 1 altsetting which
claim 1023 bytes maxpacketsize. But they also have a register which
allows the driver to configure how large the largest (iso) packet it
sends will actually be.

Currently these drivers have this "beauty" to be able to tell the
linux usb core that they won't be using 1023 as maxpacketsize but
something else, and thus function without needing full usb1 bandwidth:

struct usb_host_interface *alt;
alt = &gspca_dev->dev->config->intf_cache[0]->altsetting[1];
alt->endpoint[0].desc.wMaxPacketSize = cpu_to_le16(packet_size);
ret = usb_set_interface(gspca_dev->dev, gspca_dev->iface, 1);

Yes they are overwriting the kernels cached descriptors ...

###

Something which I would also like to bring to everyone's attention
is that we really need to fix the ehci schedule code wrt scheduling
usb1 transfers over usb2 hubs.

The current code is very broken when it comes to periodic transfers,
it basically disallows using the last microframe (let alone the
crossing of the frame boundary and using the first microframe of
the next frame).

This means that trying to submit isoc transfers with a size of 1023
will just plain fail, even if this is the only device on the entire
bus. This is one of the reasons I ended up doing the hack above, so
that these devices will at least work (be it at a decreased framerate)
through a usb2 hub.

Things become even messier when the device has a build in usb audio
microphone, often this will just not work.

Finally fixing this has become more important now then ever since
sandybridge machines (and maybe generation one core i# machines too,
I don't know) no longer have a companion controller, but instead
have an integrated usb2 hub.

Regards,

Hans

Improving kernel -> userspace (usbfs) usb device hand off

[Hans de Goede]

Hi all,

The current API for managing kernel -> userspace is a bit
rough around the edges, so I would like to discuss extending
the API.

First of all an example use case scenarios where the current API
falls short.

1) Redirection of USB devices to a virtual machine, qemu, vbox, etc.
all have the ability to redirect a USB device to the virtual machine,
and they all use usbfs for this. The first thing which will happen
here when the user selects a device to redirect is a
IOCTL_USBFS_DISCONNECT. This causes the kernel driver to see a
device unplug, with no chances for the kernel driver to do anything
against this.

Now lets say the user does the following:
-write a file to a usb flash disk
-redirect the flash disk to a vm

Currently this will cause the usb mass storage driver to see a
disconnect, and any possible still pending writes are lost ...

This is IMHO unacceptable, but currently there is nothing we can
do to avoid this.

2) So called dual mode cameras are (cheap) stillcams often even
without an lcdscreen viewfinder, and battery backed sram instead
of flash, which double as a webcam. We have drivers for both the
stillcam function of these (in libgphoto2, so using usbfs) as
well as for the webcam function (v4l2 kernel drivers).

These drivers work well, and are mature. Yet the user experience
is rather poor. Under gnome the still-cam contents will be
automatically be made available as a "drive" using a gvfs-gphoto2 fuse
mount. This however involves sending a disconnect to the v4l2 kernel
driver, and thus the /dev/video# node disappearing. So if a user
wants to use the device as a webcam he/she needs to first go to
nautilus and unmount the gvfs mount. Until that is done the user will
simply get a message from an app like cheese that he has no webcam,
not even an ebusy error, just that he has no such device.

Again not good.

###

So what do we need to make this situation better:
1) A usb_driver callback alternative to the disconnect callback,
    I propose to call this soft_disconnect. This serves 2 purposes
    a) It will allow the driver to tell the caller that that is not
       a good idea by returning an error code (think usb mass storage
       driver and mounted filesystem
    b) It will allow for example a v4l2 driver to keep its /dev/video
       node around
    Note that b) means that the normal disconnect handler should still
    be called after a soft reconnect on a real disconnect.
2) A usb_driver soft_reconnect callback to match the soft_disconnect
3) A mechanism for a usb_driver to signal a usbfs fd owner of the device
    it would like the device back. So for example the gvfs mount can be
    automatically unmounted (if not busy).

4) A IOCTL_USBFS_SOFT_DISCONNECT ioctl which will call the drivers
    soft_disconnect if it has one, and otherwise fall back to the
    regular disconnect.
5) A method for a usbfs fd owning app to know the device driver would
    like the device back. I suggest using poll with POLLIN to signal this.

Regards,

Hans

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]

Hi,

On Fri, Jun 10, 2011 at 09:55:13AM +0200, Hans de Goede wrote:
> Currently this will cause the usb mass storage driver to see a
> disconnect, and any possible still pending writes are lost ...
> 
> This is IMHO unacceptable, but currently there is nothing we can
> do to avoid this.
> 
> 2) So called dual mode cameras are (cheap) stillcams often even
> without an lcdscreen viewfinder, and battery backed sram instead
> of flash, which double as a webcam. We have drivers for both the
> stillcam function of these (in libgphoto2, so using usbfs) as
> well as for the webcam function (v4l2 kernel drivers).
> 
> These drivers work well, and are mature. Yet the user experience
> is rather poor. Under gnome the still-cam contents will be
> automatically be made available as a "drive" using a gvfs-gphoto2 fuse
> mount. This however involves sending a disconnect to the v4l2 kernel
> driver, and thus the /dev/video# node disappearing. So if a user
> wants to use the device as a webcam he/she needs to first go to
> nautilus and unmount the gvfs mount. Until that is done the user will
> simply get a message from an app like cheese that he has no webcam,
> not even an ebusy error, just that he has no such device.

that sounds quite weird. Should only happen if still image and video
functions are on different configurations or different alt-settings of
the same interface. But if they are on same configurations and separate
interfaces, you should be able to bind gphoto to the still image
interface and v4l2 to the camera interface.

How's the device setup ?

> So what do we need to make this situation better:
> 1) A usb_driver callback alternative to the disconnect callback,
>    I propose to call this soft_disconnect. This serves 2 purposes
>    a) It will allow the driver to tell the caller that that is not
>       a good idea by returning an error code (think usb mass storage
>       driver and mounted filesystem

I'm not sure you even need a driver callback for that. Should we leave
that to Desktop manager ?

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Hans de Goede]

Hi,

On 06/10/2011 10:22 AM, Felipe Balbi wrote:
> Hi,
>
> On Fri, Jun 10, 2011 at 09:55:13AM +0200, Hans de Goede wrote:
>> Currently this will cause the usb mass storage driver to see a
>> disconnect, and any possible still pending writes are lost ...
>>
>> This is IMHO unacceptable, but currently there is nothing we can
>> do to avoid this.
>>
>> 2) So called dual mode cameras are (cheap) stillcams often even
>> without an lcdscreen viewfinder, and battery backed sram instead
>> of flash, which double as a webcam. We have drivers for both the
>> stillcam function of these (in libgphoto2, so using usbfs) as
>> well as for the webcam function (v4l2 kernel drivers).
>>
>> These drivers work well, and are mature. Yet the user experience
>> is rather poor. Under gnome the still-cam contents will be
>> automatically be made available as a "drive" using a gvfs-gphoto2 fuse
>> mount. This however involves sending a disconnect to the v4l2 kernel
>> driver, and thus the /dev/video# node disappearing. So if a user
>> wants to use the device as a webcam he/she needs to first go to
>> nautilus and unmount the gvfs mount. Until that is done the user will
>> simply get a message from an app like cheese that he has no webcam,
>> not even an ebusy error, just that he has no such device.
>
> that sounds quite weird. Should only happen if still image and video
> functions are on different configurations or different alt-settings of
> the same interface. But if they are on same configurations and separate
> interfaces, you should be able to bind gphoto to the still image
> interface and v4l2 to the camera interface.
>
> How's the device setup ?
>

These are very cheap devices, and as such poorly designed. There still
and webcam functionality is on the same interface. This is likely done
this way because the devices cannot handle both functions at the same
time.

>> So what do we need to make this situation better:
>> 1) A usb_driver callback alternative to the disconnect callback,
>>     I propose to call this soft_disconnect. This serves 2 purposes
>>     a) It will allow the driver to tell the caller that that is not
>>        a good idea by returning an error code (think usb mass storage
>>        driver and mounted filesystem
>
> I'm not sure you even need a driver callback for that. Should we leave
> that to Desktop manager ?

Not sure what you mean here, but we need for a way for drivers to say
no to a software caused disconnection. See my usb mass storage device
which is still mounted getting redirected to a vm example. This cannot
be reliably done from userspace. Where as it is trivial to do this
from kernel space. One could advocate to make the existing disconnect
ioctl use the new soft_disconnect usb_driver callback instead of
adding a new usbfs ioctl for this, but that means that a driver
can block any and all userspace triggered disconnects. Where as
having a new ioctl, means that apps which want to play nice can play
nice, while keeping the possibility of a hard userspace initiated
disconnect.

One could also argue that making the existing disconnect ioctl return
-EBUSY in some cases now is an ABI change.

Regards,

Hans

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 3867 bytes --]

Hi,

On Fri, Jun 10, 2011 at 10:36:47AM +0200, Hans de Goede wrote:
> >On Fri, Jun 10, 2011 at 09:55:13AM +0200, Hans de Goede wrote:
> >>Currently this will cause the usb mass storage driver to see a
> >>disconnect, and any possible still pending writes are lost ...
> >>
> >>This is IMHO unacceptable, but currently there is nothing we can
> >>do to avoid this.
> >>
> >>2) So called dual mode cameras are (cheap) stillcams often even
> >>without an lcdscreen viewfinder, and battery backed sram instead
> >>of flash, which double as a webcam. We have drivers for both the
> >>stillcam function of these (in libgphoto2, so using usbfs) as
> >>well as for the webcam function (v4l2 kernel drivers).
> >>
> >>These drivers work well, and are mature. Yet the user experience
> >>is rather poor. Under gnome the still-cam contents will be
> >>automatically be made available as a "drive" using a gvfs-gphoto2 fuse
> >>mount. This however involves sending a disconnect to the v4l2 kernel
> >>driver, and thus the /dev/video# node disappearing. So if a user
> >>wants to use the device as a webcam he/she needs to first go to
> >>nautilus and unmount the gvfs mount. Until that is done the user will
> >>simply get a message from an app like cheese that he has no webcam,
> >>not even an ebusy error, just that he has no such device.
> >
> >that sounds quite weird. Should only happen if still image and video
> >functions are on different configurations or different alt-settings of
> >the same interface. But if they are on same configurations and separate
> >interfaces, you should be able to bind gphoto to the still image
> >interface and v4l2 to the camera interface.
> >
> >How's the device setup ?
> >
> 
> These are very cheap devices, and as such poorly designed. There still
> and webcam functionality is on the same interface. This is likely done
> this way because the devices cannot handle both functions at the same
> time.

ok got it.

> >>So what do we need to make this situation better:
> >>1) A usb_driver callback alternative to the disconnect callback,
> >>    I propose to call this soft_disconnect. This serves 2 purposes
> >>    a) It will allow the driver to tell the caller that that is not
> >>       a good idea by returning an error code (think usb mass storage
> >>       driver and mounted filesystem
> >
> >I'm not sure you even need a driver callback for that. Should we leave
> >that to Desktop manager ?
> 
> Not sure what you mean here, but we need for a way for drivers to say
> no to a software caused disconnection. See my usb mass storage device
> which is still mounted getting redirected to a vm example. This cannot

in that case, why don't you just flush all data and continue ? Also,
desktop manager knows that a particular device mounted, so it could also
ask the user if s/he wants to continue.

I'm not sure preventing a disconnect is a good thing.

> be reliably done from userspace. Where as it is trivial to do this
> from kernel space. One could advocate to make the existing disconnect
> ioctl use the new soft_disconnect usb_driver callback instead of
> adding a new usbfs ioctl for this, but that means that a driver
> can block any and all userspace triggered disconnects. Where as
> having a new ioctl, means that apps which want to play nice can play
> nice, while keeping the possibility of a hard userspace initiated
> disconnect.
> 
> One could also argue that making the existing disconnect ioctl return
> -EBUSY in some cases now is an ABI change.

OTOH, if the user really wants to move the usb device to the guest OS,
he has just requested for that, so should we prevent it ? what we need
is for the applications to be notified to exit cleanly and release the
device because the user has requested to do so. No ?

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Hans de Goede]

Hi,

On 06/10/2011 10:42 AM, Felipe Balbi wrote:
> Hi,
>
> On Fri, Jun 10, 2011 at 10:36:47AM +0200, Hans de Goede wrote:
>>> On Fri, Jun 10, 2011 at 09:55:13AM +0200, Hans de Goede wrote:

<snip>

>>>> So what do we need to make this situation better:
>>>> 1) A usb_driver callback alternative to the disconnect callback,
>>>>     I propose to call this soft_disconnect. This serves 2 purposes
>>>>     a) It will allow the driver to tell the caller that that is not
>>>>        a good idea by returning an error code (think usb mass storage
>>>>        driver and mounted filesystem
>>>
>>> I'm not sure you even need a driver callback for that. Should we leave
>>> that to Desktop manager ?
>>
>> Not sure what you mean here, but we need for a way for drivers to say
>> no to a software caused disconnection. See my usb mass storage device
>> which is still mounted getting redirected to a vm example. This cannot
>
> in that case, why don't you just flush all data and continue ? Also,
> desktop manager knows that a particular device mounted, so it could also
> ask the user if s/he wants to continue.
>
> I'm not sure preventing a disconnect is a good thing.
>

I assume you are sure preventing data loss is a good thing? Because in
this example the 2 are the same.

Also note I'm not suggestion at always preventing the disconnect, I'm
suggesting to add a new try_disconnect ioctl, which apps which want
to behave nicely can use instead of the regular disconnect ioctl, and
then drivers can prevent the disconnect. Apps using the old ioctl will
still get an unconditional disconnect as before.

>> be reliably done from userspace. Where as it is trivial to do this
>> from kernel space. One could advocate to make the existing disconnect
>> ioctl use the new soft_disconnect usb_driver callback instead of
>> adding a new usbfs ioctl for this, but that means that a driver
>> can block any and all userspace triggered disconnects. Where as
>> having a new ioctl, means that apps which want to play nice can play
>> nice, while keeping the possibility of a hard userspace initiated
>> disconnect.
>>
>> One could also argue that making the existing disconnect ioctl return
>> -EBUSY in some cases now is an ABI change.
>
> OTOH, if the user really wants to move the usb device to the guest OS,
> he has just requested for that, so should we prevent it ? what we need
> is for the applications to be notified to exit cleanly and release the
> device because the user has requested to do so. No ?

We are talking about a device with a mounted file system on it here,
any process could be holding files open on there, and there currently
exists no mechanism to notify all apps to exit cleanly and release
the files. Even if there were some method for a desktop environment
like gnome to ask apps to release those files, and all gnome apps
where to be modified to support that mechanism, then there are still
1000-s of non gnome (or kde, xfce, whatever) apps which will not
support that.

We already have a mechanism to cleanly close down a filesystem, it
is called unmount. And it will fail if apps have files open. All I'm
suggesting is forwarding this ebusy failure to the application
trying to disconnect the driver from the usb mass storage interface.

Simply removing the filesystem from under apps holding files open will
lead to io errors, and very unhappy apps.

Regards,

Hans

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 3971 bytes --]

Hi,

On Fri, Jun 10, 2011 at 02:19:20PM +0200, Hans de Goede wrote:
> >>>>So what do we need to make this situation better:
> >>>>1) A usb_driver callback alternative to the disconnect callback,
> >>>>    I propose to call this soft_disconnect. This serves 2 purposes
> >>>>    a) It will allow the driver to tell the caller that that is not
> >>>>       a good idea by returning an error code (think usb mass storage
> >>>>       driver and mounted filesystem
> >>>
> >>>I'm not sure you even need a driver callback for that. Should we leave
> >>>that to Desktop manager ?
> >>
> >>Not sure what you mean here, but we need for a way for drivers to say
> >>no to a software caused disconnection. See my usb mass storage device
> >>which is still mounted getting redirected to a vm example. This cannot
> >
> >in that case, why don't you just flush all data and continue ? Also,
> >desktop manager knows that a particular device mounted, so it could also
> >ask the user if s/he wants to continue.
> >
> >I'm not sure preventing a disconnect is a good thing.
> 
> I assume you are sure preventing data loss is a good thing? Because in
> this example the 2 are the same.
> 
> Also note I'm not suggestion at always preventing the disconnect, I'm
> suggesting to add a new try_disconnect ioctl, which apps which want
> to behave nicely can use instead of the regular disconnect ioctl, and
> then drivers can prevent the disconnect. Apps using the old ioctl will
> still get an unconditional disconnect as before.

I'm just not sure if this should sit in the kernel. You can easily have
some sort of device manager (much like udisks for storage) to track this
down for ya. So instead of calling usbfs straight, you need to go via
the "device manager". This device manager should know that before
calling 'disconnect' it must be sure there's nothing pending.

> >>One could also argue that making the existing disconnect ioctl return
> >>-EBUSY in some cases now is an ABI change.
> >
> >OTOH, if the user really wants to move the usb device to the guest OS,
> >he has just requested for that, so should we prevent it ? what we need
> >is for the applications to be notified to exit cleanly and release the
> >device because the user has requested to do so. No ?
> 
> We are talking about a device with a mounted file system on it here,
> any process could be holding files open on there, and there currently
> exists no mechanism to notify all apps to exit cleanly and release
> the files. Even if there were some method for a desktop environment
> like gnome to ask apps to release those files, and all gnome apps
> where to be modified to support that mechanism, then there are still
> 1000-s of non gnome (or kde, xfce, whatever) apps which will not
> support that.

Still, I'm not sure this is argument good enough to add another sort of
conflicting ioctl to the kernel. It won't prevent apps from calling the
other disconnect anyway. And besides, if an app can be modified to use
the try_disconnect ioctl, they can be modified for the system
notification (on userland) too, can't they ?

> We already have a mechanism to cleanly close down a filesystem, it
> is called unmount. And it will fail if apps have files open. All I'm
> suggesting is forwarding this ebusy failure to the application
> trying to disconnect the driver from the usb mass storage interface.
> 
> Simply removing the filesystem from under apps holding files open will
> lead to io errors, and very unhappy apps.

Correct, but does the solution _have_ to sit in kernel ? I'm not sure.
If you add the try_disconnect ioctl and all applications one your
desktop, except one, are playing nice, you will still have problems. To
me, it sounds much nicer to add such things on userland alone, in a way
such that it doesn't matter if it's a usb storage, mmc card, or eSATA
drive. That should be hidden from apps IMHO.

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Alan Stern]

On Fri, 10 Jun 2011, Hans de Goede wrote:

> Hi all,
> 
> The current API for managing kernel -> userspace is a bit
> rough around the edges, so I would like to discuss extending
> the API.
> 
> First of all an example use case scenarios where the current API
> falls short.
> 
> 1) Redirection of USB devices to a virtual machine, qemu, vbox, etc.
> all have the ability to redirect a USB device to the virtual machine,
> and they all use usbfs for this. The first thing which will happen
> here when the user selects a device to redirect is a
> IOCTL_USBFS_DISCONNECT. This causes the kernel driver to see a
> device unplug, with no chances for the kernel driver to do anything
> against this.
> 
> Now lets say the user does the following:
> -write a file to a usb flash disk
> -redirect the flash disk to a vm
> 
> Currently this will cause the usb mass storage driver to see a
> disconnect, and any possible still pending writes are lost ...
> 
> This is IMHO unacceptable, but currently there is nothing we can
> do to avoid this.

You haven't given a proper description of the problem.  See below.

> 2) So called dual mode cameras are (cheap) stillcams often even
> without an lcdscreen viewfinder, and battery backed sram instead
> of flash, which double as a webcam. We have drivers for both the
> stillcam function of these (in libgphoto2, so using usbfs) as
> well as for the webcam function (v4l2 kernel drivers).
> 
> These drivers work well, and are mature. Yet the user experience
> is rather poor. Under gnome the still-cam contents will be
> automatically be made available as a "drive" using a gvfs-gphoto2 fuse
> mount. This however involves sending a disconnect to the v4l2 kernel
> driver, and thus the /dev/video# node disappearing. So if a user
> wants to use the device as a webcam he/she needs to first go to
> nautilus and unmount the gvfs mount. Until that is done the user will
> simply get a message from an app like cheese that he has no webcam,
> not even an ebusy error, just that he has no such device.
> 
> Again not good.

As Felipe has mentioned, this sounds like the sort of problem that 
can better be solved in userspace.  A dual-mode device like the one 
you describe really is either a still-cam or a webcam, never both at 
the same time.  Hence what users need is a utility program to switch 
modes (by loading/unloading the appropriate programs or drivers).  Or 
maybe a desktop daemon that could accomplish the same result 
automatically, based on requests from user programs.

> ###
> 
> So what do we need to make this situation better:
> 1) A usb_driver callback alternative to the disconnect callback,
>     I propose to call this soft_disconnect. This serves 2 purposes
>     a) It will allow the driver to tell the caller that that is not
>        a good idea by returning an error code (think usb mass storage
>        driver and mounted filesystem

Not feasible.  usb-storage has no idea whether or not a device it
controls has a mounted filesystem.  (All it does is send SCSI commands
to a device and get back the results.)  Since that's the main use
case you're interested in, this part of the proposal seems destined to
fail.

But userspace _does_ know where the mounted filesystems are.  
Therefore userspace should be responsible for avoiding programs that
want to take control of devices holding these filesystems.  That's the
reason why usbfs device nodes are owned by root and have 0644 mode;
there're can be written to only by programs with superuser privileges
-- and such programs are supposed to be careful about what they do.

>     b) It will allow for example a v4l2 driver to keep its /dev/video
>        node around
>     Note that b) means that the normal disconnect handler should still
>     be called after a soft reconnect on a real disconnect.

In some sense the disconnect callback for usb-storage already _is_
"soft".  Although the driver cannot refuse the disconnect, it _can_
continue to communicate with the device until the callback returns.

(As it happens, usb-storage _doesn't_ do any further communication with
the device.  This is mostly for historical reasons, to compensate for
shortcomings in the SCSI stack in earlier kernel versions.  On the
other hand, there really isn't much that you would want to send to a
mass-storage device during a soft disconnect.  Perhaps tell it to flush
its cache out to the storage medium -- but if there are no open file 
handles for the device and no mounted filesystems then the cache 
will already be flushed.)

> 2) A usb_driver soft_reconnect callback to match the soft_disconnect
> 3) A mechanism for a usb_driver to signal a usbfs fd owner of the device
>     it would like the device back. So for example the gvfs mount can be
>     automatically unmounted (if not busy).

This also should be handled in userspace.  USB drivers never "want
back" a device they are no longer bound to -- in fact, the device model
used throughout the kernel makes this whole idea meaningless.  Device
drivers don't want devices back.  Rather, _users_ want to turn control
of devices over to specific drivers.  That's why this problem needs to
be handled in userspace.

> 4) A IOCTL_USBFS_SOFT_DISCONNECT ioctl which will call the drivers
>     soft_disconnect if it has one, and otherwise fall back to the
>     regular disconnect.
> 5) A method for a usbfs fd owning app to know the device driver would
>     like the device back. I suggest using poll with POLLIN to signal this.

It seems as if you're trying to implement some notion of allowing a 
device to have more than one driver at the same time.  This is so far 
out from the way the kernel behaves now, adopting it would be very 
difficult if not impossible.  Certainly the USB stack isn't the place 
to start.

Alan Stern

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Mauro Carvalho Chehab]

Em 10-06-2011 11:48, Alan Stern escreveu:
> On Fri, 10 Jun 2011, Hans de Goede wrote:
> 
> 
> As Felipe has mentioned, this sounds like the sort of problem that 
> can better be solved in userspace.  A dual-mode device like the one 
> you describe really is either a still-cam or a webcam, never both at 
> the same time.  Hence what users need is a utility program to switch 
> modes (by loading/unloading the appropriate programs or drivers). 

Unloading a driver in order to access the hardware via userspace?
This sounds a very bad idea do me. What happens if another hardware 
is using the same driver?

Cheers,
Mauro.

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Alan Stern]

On Fri, 10 Jun 2011, Mauro Carvalho Chehab wrote:

> Em 10-06-2011 11:48, Alan Stern escreveu:
> > On Fri, 10 Jun 2011, Hans de Goede wrote:
> > 
> > 
> > As Felipe has mentioned, this sounds like the sort of problem that 
> > can better be solved in userspace.  A dual-mode device like the one 
> > you describe really is either a still-cam or a webcam, never both at 
> > the same time.  Hence what users need is a utility program to switch 
> > modes (by loading/unloading the appropriate programs or drivers). 
> 
> Unloading a driver in order to access the hardware via userspace?
> This sounds a very bad idea do me. What happens if another hardware 
> is using the same driver?

A kernel driver wouldn't have to be unloaded.  It could simply be 
unbound from the device via sysfs or usbfs.

Alan STern

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Theodore Kilgore]

On Fri, 10 Jun 2011, Hans de Goede wrote:

> Hi all,
> 
> The current API for managing kernel -> userspace is a bit
> rough around the edges, so I would like to discuss extending
> the API.

[...]

> 2) So called dual mode cameras are (cheap) stillcams often even
> without an lcdscreen viewfinder, and battery backed sram instead
> of flash, which double as a webcam. We have drivers for both the
> stillcam function of these (in libgphoto2, so using usbfs) as
> well as for the webcam function (v4l2 kernel drivers).
> 
> These drivers work well, and are mature. Yet the user experience
> is rather poor. Under gnome the still-cam contents will be
> automatically be made available as a "drive" using a gvfs-gphoto2 fuse
> mount. This however involves sending a disconnect to the v4l2 kernel
> driver, and thus the /dev/video# node disappearing. So if a user
> wants to use the device as a webcam he/she needs to first go to
> nautilus and unmount the gvfs mount. Until that is done the user will
> simply get a message from an app like cheese that he has no webcam,
> not even an ebusy error, just that he has no such device.
> 
> Again not good.

[...]

As I have been involved in writing the drivers (both the kernel and the 
libgphoto2 drivers) for many of the affected cameras, perhaps I should 
expand on this problem. There are lots of responses to this original 
message of Hans. I will try to take some of their comments into account, 
below. First, some background.

1. All of the cameras in question have only one USB Vendor:Product number. 
In this sense, they are not "good citizens" which have different Product 
numbers for the two distinct modes of functioning. Thus, the problems are 
from that standpoint unavoidable.

2. Until recently in the history of Linux, there was an irreconcilable 
conflict. If a kernel driver for the video streaming mode was present and 
installed, it was not possible to use the camera in stillcam mode at all. 
Thus the only solution to the problem was to blacklist the kernel module 
so that it would not get loaded automatically and only to install said 
module by hand if the camera were to be used in video streaming mode, then 
to rmmod it immediately afterwards. Very cumbersome, obviously. 

3. The current state of affairs is an advance on (2) but it is still 
inelegant. What happens now is, libusb has been revised in such a way that 
the kernel module is disabled (though still present) if a userspace driver 
(libgphoto2) wants to access the device (the camera). If it is desired to 
do video streaming after that, the camera needs to be re-plugged, which 
then causes the module to be automatically re-loaded.

4. Hans is absolutely correct about the problem with certain Gnome apps 
(and certain distros which blindly use them), which load up the libgphoto2 
driver for the camera as soon as it is detected. The consequence (cf. item 
3) is that the camera can never be used as a webcam. The only solution to 
that problem is to disable the automatic loading of the libgphoto2 driver.

5. It could be said that those who came up with the "user-friendly" 
"solution" described in (4) were not very clever, and perhaps they ought 
to fix their own mess. I would strongly agree that they ought to have 
thought before coding, as the result is not user-friendly in the least. 

6. The question has been asked whether the cameras are always using the 
same interface. Typically, yes. The same altsetting? That depends on the 
camera. Some of them use isochronous transport for streaming, and some of 
them rely exclusively upon bulk transport. Those which use bulk transport 
only are confined to altsetting 0.

Some possible solutions?

Well, first of all it needs to be understood that the problem originates 
as a bad feature of something good, namely the rigid separation of 
kernelspace and userspace functionality which is an integral part of the 
Linux security model. Some other operating systems are not so careful, and 
thus they do not have a problem about supporting dual-mode hardware.

Second, it appears to me that the problem is most appropriately addressed 
from the userspace side and perhaps also from the kernelspace side. 

In userspace, it would be a really good idea if those who are attempting 
to write user-friendly apps and to create user-friendly distros would 
actually learn some of the basics of Linux, such as the rudiments of the 
security model. Since dual-mode cameras are known to exist, it would have 
been appropriate, when one is detected, to pop up a dialog window which 
asks the user to choose a webcam app or a stillcam app. Even this minor 
innovation would stop a lot of user grief. Frankly, I am mystified that 
this was not done.

This still leaves the problem (see item 3, above) that a dual-mode camera 
needs to be re-plugged in order to re-enable the access to /dev/video* if 
it has been used in stillcam mode. If it is possible to do a re-plug in 
software, this would help a lot. It does not matter if the re-plug is done 
in userspace or in kernelspace, so long as it can be done, somehow, after 
libusb relinquishes the camera. Or, fix things up somehow so that the 
kernel module automatically re-activates itself when the userspace app 
(using libgphoto2) is closed down.

Finally, another possible alternative would be to figure out how to do 
something in the kernel module which permits the camera to be accessed by 
libusb.

If there is going to be discussion about this problem in Vancouver, I am 
really sorry that I will probably be able to attend. Classes for the Fall 
semester begin at Auburn University on August 17. Nevertheless, this is a 
problem in which I have been interested for a long time.

Theodore Kilgore

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 6682 bytes --]

Hi,

On Fri, Jun 10, 2011 at 01:16:47PM -0500, Theodore Kilgore wrote:
> As I have been involved in writing the drivers (both the kernel and the 
> libgphoto2 drivers) for many of the affected cameras, perhaps I should 
> expand on this problem. There are lots of responses to this original 
> message of Hans. I will try to take some of their comments into account, 
> below. First, some background.
> 
> 1. All of the cameras in question have only one USB Vendor:Product number. 
> In this sense, they are not "good citizens" which have different Product 

there's nothing in the USB spec that says you need different product IDs
for different modes of operation. No matter if it's still or webcam
configuration, the underlying function is the same: capture images using
a set of lenses and image sensor.

> numbers for the two distinct modes of functioning. Thus, the problems are 
> from that standpoint unavoidable.

I don't see any problems in this situation. If, for that particular
product, webcam and still image functionality are mutually exclusive,
then that's how the product (and their drivers) have to work.

If the linux community decided to put webcam functionality in kernel and
still image functionality on a completely separate driver, that's
entirely our problem.

> 2. Until recently in the history of Linux, there was an irreconcilable 
> conflict. If a kernel driver for the video streaming mode was present and 
> installed, it was not possible to use the camera in stillcam mode at all. 
> Thus the only solution to the problem was to blacklist the kernel module 
> so that it would not get loaded automatically and only to install said 
> module by hand if the camera were to be used in video streaming mode, then 
> to rmmod it immediately afterwards. Very cumbersome, obviously. 

true... but why couldn't we combine both in kernel or in userspace
altogether ? Why do we have this split ? (words from a newbie in V4L2,
go easy :-p)

> 3. The current state of affairs is an advance on (2) but it is still 
> inelegant. What happens now is, libusb has been revised in such a way that 
> the kernel module is disabled (though still present) if a userspace driver 
> (libgphoto2) wants to access the device (the camera). If it is desired to 
> do video streaming after that, the camera needs to be re-plugged, which 
> then causes the module to be automatically re-loaded.

It's still wrong. This should be just another
USB_REQ_SET_CONFIGURATION(). If this is was just one single driver, you
could easily do that.

> 4. Hans is absolutely correct about the problem with certain Gnome apps 
> (and certain distros which blindly use them), which load up the libgphoto2 
> driver for the camera as soon as it is detected. The consequence (cf. item 
> 3) is that the camera can never be used as a webcam. The only solution to 
> that problem is to disable the automatic loading of the libgphoto2 driver.

Or, to move the libgphoto2 driver to kernel, combine it in the same
driver that handles streaming. No ?

> 5. It could be said that those who came up with the "user-friendly" 
> "solution" described in (4) were not very clever, and perhaps they ought 
> to fix their own mess. I would strongly agree that they ought to have 
> thought before coding, as the result is not user-friendly in the least. 

I agree here

> 6. The question has been asked whether the cameras are always using the 
> same interface. Typically, yes. The same altsetting? That depends on the 
> camera. Some of them use isochronous transport for streaming, and some of 
> them rely exclusively upon bulk transport. Those which use bulk transport 
> only are confined to altsetting 0.

the transfer type or the way the configurations are setup shouldn't
matter much for the end functionality.

> Some possible solutions?
> 
> Well, first of all it needs to be understood that the problem originates 
> as a bad feature of something good, namely the rigid separation of 
> kernelspace and userspace functionality which is an integral part of the 
> Linux security model. Some other operating systems are not so careful, and 
> thus they do not have a problem about supporting dual-mode hardware.

You'd need to describe this fully. What's the problem in having one
driver handle all modes of operation of that particular camera ? Sounds
like a decision from V4L2 folks not to take still image cameras in
kernel. Am I wrong ?

> Second, it appears to me that the problem is most appropriately addressed 
> from the userspace side and perhaps also from the kernelspace side. 
> 
> In userspace, it would be a really good idea if those who are attempting 
> to write user-friendly apps and to create user-friendly distros would 
> actually learn some of the basics of Linux, such as the rudiments of the 
> security model. Since dual-mode cameras are known to exist, it would have 
> been appropriate, when one is detected, to pop up a dialog window which 
> asks the user to choose a webcam app or a stillcam app. Even this minor 
> innovation would stop a lot of user grief. Frankly, I am mystified that 
> this was not done.

I think this is still not good. It should be more "fluid". If the camera
has separate alternate settings or different configurations, all it
takes (from a USB standpoint) is to send the correct SetConfiguration or
SetInterface command and the camera should happily change its mode of
operation.

Which means that a simple ioctl() to change the mode of operation would
suffice should the driver be combined.

> This still leaves the problem (see item 3, above) that a dual-mode camera 
> needs to be re-plugged in order to re-enable the access to /dev/video* if 
> it has been used in stillcam mode. If it is possible to do a re-plug in 
> software, this would help a lot. It does not matter if the re-plug is done 
> in userspace or in kernelspace, so long as it can be done, somehow, after 
> libusb relinquishes the camera. Or, fix things up somehow so that the 
> kernel module automatically re-activates itself when the userspace app 
> (using libgphoto2) is closed down.
> 
> Finally, another possible alternative would be to figure out how to do 
> something in the kernel module which permits the camera to be accessed by 
> libusb.

I don't see the possibility of combining both drivers and use
/dev/videoX for streaming and something like /dev/camX for still image.
Now, the only thing you have to be sure in kernel space is that you
don't allow both to be accessed together, but there are ways to handle
it.

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
> > Topic 1
> > -------
> > 
> > The KVM folks suggested that it would be good to get USB and
> > virtualization developers together to talk about how to virtualize the
> > xHCI host controller.  The xHCI spec architect worked closely with
> > VMWare to get some extra goodies in the spec to help virtualization, and
> > I'd like to see the other virtualization developers take advantage of
> > that.  I'd also like us to hash out any issues they have been finding in
> > the USB core or xHCI driver during the virtualization effort.
> 
> Do people really want to virtualize the whole xHCI controller, or just
> specific ports or devices to the guest operating system?

A host OS could chose to virtualize the whole xHCI controller if it
wanted to.  That's part of the reason why xHCI does all the bandwidth
checking in hardware, not in software.

> If just specific ports, would something like usbip be better for virtual
> machines, with the USB traffic going over the network connection between
> the guest/host?

It could be done that way too.  But that doesn't help if you're trying
to run Windows under Linux, right?  Only if all the guest OSes use the
same USB IP protocol then it would work.

> > Hope to see you there!
> 
> Thanks for putting this together.
> 
> Do we need to sign up somewhere to give the organizers a sense of how
> many people will be attending?

I private ack by email would be great.  Or you can ack by facebook:
https://www.facebook.com/event.php?eid=229532200405657  I could add the
event in upcoming.yahoo.com if anyone uses that.

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
> > On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
> > > Topic 1
> > > -------
> > > 
> > > The KVM folks suggested that it would be good to get USB and
> > > virtualization developers together to talk about how to virtualize the
> > > xHCI host controller.  The xHCI spec architect worked closely with
> > > VMWare to get some extra goodies in the spec to help virtualization, and
> > > I'd like to see the other virtualization developers take advantage of
> > > that.  I'd also like us to hash out any issues they have been finding in
> > > the USB core or xHCI driver during the virtualization effort.
> > 
> > Do people really want to virtualize the whole xHCI controller, or just
> > specific ports or devices to the guest operating system?
> 
> A host OS could chose to virtualize the whole xHCI controller if it
> wanted to.  That's part of the reason why xHCI does all the bandwidth
> checking in hardware, not in software.

And here I thought it did that so it would be "correct" :)

> 
> > If just specific ports, would something like usbip be better for virtual
> > machines, with the USB traffic going over the network connection between
> > the guest/host?
> 
> It could be done that way too.  But that doesn't help if you're trying
> to run Windows under Linux, right?  Only if all the guest OSes use the
> same USB IP protocol then it would work.

usbip works on Windows as well as Linux.

But how could you run Windows with a xHCI controller in a guest, as
Windows has no xHCI driver?  What would it expect to see?

> > > Hope to see you there!
> > 
> > Thanks for putting this together.
> > 
> > Do we need to sign up somewhere to give the organizers a sense of how
> > many people will be attending?
> 
> I private ack by email would be great.  Or you can ack by facebook:
> https://www.facebook.com/event.php?eid=229532200405657  I could add the
> event in upcoming.yahoo.com if anyone uses that.

Nice try, but I'm still not going to join facebook...

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Alan Stern]

On Fri, 10 Jun 2011, Felipe Balbi wrote:

> I don't see any problems in this situation. If, for that particular
> product, webcam and still image functionality are mutually exclusive,
> then that's how the product (and their drivers) have to work.
> 
> If the linux community decided to put webcam functionality in kernel and
> still image functionality on a completely separate driver, that's
> entirely our problem.

And the problem is how to coordinate the two of them.

> > 2. Until recently in the history of Linux, there was an irreconcilable 
> > conflict. If a kernel driver for the video streaming mode was present and 
> > installed, it was not possible to use the camera in stillcam mode at all. 
> > Thus the only solution to the problem was to blacklist the kernel module 
> > so that it would not get loaded automatically and only to install said 
> > module by hand if the camera were to be used in video streaming mode, then 
> > to rmmod it immediately afterwards. Very cumbersome, obviously. 
> 
> true... but why couldn't we combine both in kernel or in userspace
> altogether ? Why do we have this split ? (words from a newbie in V4L2,
> go easy :-p)

I think the problem may be that the PTP protocol used in the still-cam
mode isn't suitable for a kernel driver.  Or if it is suitable, it
would have to be something like a shared-filesystem driver -- nothing
like a video driver.  You certainly wouldn't want to put it in V4L2.

> Or, to move the libgphoto2 driver to kernel, combine it in the same
> driver that handles streaming. No ?

No.  Something else is needed.

Alan Stern

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

Hi,

On Fri, Jun 10, 2011 at 05:18:39PM -0400, Alan Stern wrote:
> > > 2. Until recently in the history of Linux, there was an irreconcilable 
> > > conflict. If a kernel driver for the video streaming mode was present and 
> > > installed, it was not possible to use the camera in stillcam mode at all. 
> > > Thus the only solution to the problem was to blacklist the kernel module 
> > > so that it would not get loaded automatically and only to install said 
> > > module by hand if the camera were to be used in video streaming mode, then 
> > > to rmmod it immediately afterwards. Very cumbersome, obviously. 
> > 
> > true... but why couldn't we combine both in kernel or in userspace
> > altogether ? Why do we have this split ? (words from a newbie in V4L2,
> > go easy :-p)
> 
> I think the problem may be that the PTP protocol used in the still-cam
> mode isn't suitable for a kernel driver.  Or if it is suitable, it
> would have to be something like a shared-filesystem driver -- nothing
> like a video driver.  You certainly wouldn't want to put it in V4L2.

ach true.. Had forgotten that detail. Needs more thinking.

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Theodore Kilgore]

On Fri, 10 Jun 2011, Felipe Balbi wrote:

> Hi,
> 
> On Fri, Jun 10, 2011 at 01:16:47PM -0500, Theodore Kilgore wrote:
> > As I have been involved in writing the drivers (both the kernel and the 
> > libgphoto2 drivers) for many of the affected cameras, perhaps I should 
> > expand on this problem. There are lots of responses to this original 
> > message of Hans. I will try to take some of their comments into account, 
> > below. First, some background.
> > 
> > 1. All of the cameras in question have only one USB Vendor:Product number. 
> > In this sense, they are not "good citizens" which have different Product 
> 
> there's nothing in the USB spec that says you need different product IDs
> for different modes of operation. No matter if it's still or webcam
> configuration, the underlying function is the same: capture images using
> a set of lenses and image sensor.

True, true. But I will add that most of these cameras are Class 255, 
Subclass 255, Protocol 255 (Proprietary, Proprietary, Proprietary).

> 
> > numbers for the two distinct modes of functioning. Thus, the problems are 
> > from that standpoint unavoidable.
> 
> I don't see any problems in this situation. If, for that particular
> product, webcam and still image functionality are mutually exclusive,
> then that's how the product (and their drivers) have to work.

Yes.

> 
> If the linux community decided to put webcam functionality in kernel and
> still image functionality on a completely separate driver, that's
> entirely our problem.

As I understand, the basic reason why webcam functionality needs to be in 
the kernel is speed. Quick reaction time, and faster data transmission. 
Most but not all webcams use isochronous data transport. The ones which do 
not, cannot get as high a rate of frames per second. Isochronous data 
transport has historically seemed to need kernel support in order to work 
properly. Related to this, libusb has not supported isochronous data 
transport. But still cameras use bulk transport. This is natural because 
one needs to move data which is stored on the camera to a place on the 
computer where it can be processed (if needed) and stored. There is no 
impediment to userspace talking to a USB device, so why not go ahead and 
do the job that way if the kernel is not needed?

The second thing to mention is that libgphoto2 at this point is supporting 
well over 1,000 cameras. True, there are not 1,000 driver libraries, but 
there are quite a few. My impression is that nobody wants to put stuff 
like that in the kernel unless it is absolutely necessary, just because 
some of those cameras are dual-mode cameras. Don't people complain on a 
fairly regular basis about kernel code bloat?


> 
> > 2. Until recently in the history of Linux, there was an irreconcilable 
> > conflict. If a kernel driver for the video streaming mode was present and 
> > installed, it was not possible to use the camera in stillcam mode at all. 
> > Thus the only solution to the problem was to blacklist the kernel module 
> > so that it would not get loaded automatically and only to install said 
> > module by hand if the camera were to be used in video streaming mode, then 
> > to rmmod it immediately afterwards. Very cumbersome, obviously. 
> 
> true... but why couldn't we combine both in kernel or in userspace
> altogether ? Why do we have this split ? (words from a newbie in V4L2,
> go easy :-p)

See above.
 
> > 3. The current state of affairs is an advance on (2) but it is still 
> > inelegant. What happens now is, libusb has been revised in such a way that 
> > the kernel module is disabled (though still present) if a userspace driver 
> > (libgphoto2) wants to access the device (the camera). If it is desired to 
> > do video streaming after that, the camera needs to be re-plugged, which 
> > then causes the module to be automatically re-loaded.
> 
> It's still wrong. This should be just another
> USB_REQ_SET_CONFIGURATION(). If this is was just one single driver, you
> could easily do that.

Well, I think it ought to be possible anyway. Up to that point, I agree. 
Though I confess I do not exactly know how.

 
> > 4. Hans is absolutely correct about the problem with certain Gnome apps 
> > (and certain distros which blindly use them), which load up the libgphoto2 
> > driver for the camera as soon as it is detected. The consequence (cf. item 
> > 3) is that the camera can never be used as a webcam. The only solution to 
> > that problem is to disable the automatic loading of the libgphoto2 driver.

I would only add to this, that it is not pleasant to have to deal with 
frustrated users. At this point, my usual advice is that this is a distro 
problem. Ask your distro for help.

> 
> Or, to move the libgphoto2 driver to kernel, combine it in the same
> driver that handles streaming. No ?

See above. 

> 
> > 5. It could be said that those who came up with the "user-friendly" 
> > "solution" described in (4) were not very clever, and perhaps they ought 
> > to fix their own mess. I would strongly agree that they ought to have 
> > thought before coding, as the result is not user-friendly in the least. 
> 
> I agree here

Excellent. I really did have to restrain myself when writing that, too. 

> 
> > 6. The question has been asked whether the cameras are always using the 
> > same interface. Typically, yes. The same altsetting? That depends on the 
> > camera. Some of them use isochronous transport for streaming, and some of 
> > them rely exclusively upon bulk transport. Those which use bulk transport 
> > only are confined to altsetting 0.
> 
> the transfer type or the way the configurations are setup shouldn't
> matter much for the end functionality.
> 
> > Some possible solutions?
> > 
> > Well, first of all it needs to be understood that the problem originates 
> > as a bad feature of something good, namely the rigid separation of 
> > kernelspace and userspace functionality which is an integral part of the 
> > Linux security model. Some other operating systems are not so careful, and 
> > thus they do not have a problem about supporting dual-mode hardware.
> 
> You'd need to describe this fully. 

Certain other operating systems are not fussy at all about their device 
drivers. Those operating systems are also notorious for bad security and 
for crashing, and one of the things which gets blamed for that is all 
those hardware drivers from the hardware manufacturers. Linux tries to do 
things right, which clearly can also cause problems.

What's the problem in having one
> driver handle all modes of operation of that particular camera ? Sounds
> like a decision from V4L2 folks not to take still image cameras in
> kernel. Am I wrong ?

Still image cameras can be handled perfectly well in userspace. There is 
no need for kernel support, at all. Moreover, many still cameras will not 
stream at all; they simply were not designed, built, or sold to do that.  
As far as V4L2 was concerned, it was developed in order to meet the need 
for supporting webcams. Clearly, that was a conscious decision. But you 
ask whether it is possible to support a dual-mode camera entirely in V4L2. 
I would say that of course it ought to be possible, but that would require 
a large portion of the functionality of Gphoto to be duplicated in V4L2, 
just to support those dual-mode cameras. Having some experience with both 
Gphoto and V4L2, I have to say that I am not very taken with the idea. 
There has to be a way which is not so arduous, one would hope.

> 
> > Second, it appears to me that the problem is most appropriately addressed 
> > from the userspace side and perhaps also from the kernelspace side. 
> > 
> > In userspace, it would be a really good idea if those who are attempting 
> > to write user-friendly apps and to create user-friendly distros would 
> > actually learn some of the basics of Linux, such as the rudiments of the 
> > security model. Since dual-mode cameras are known to exist, it would have 
> > been appropriate, when one is detected, to pop up a dialog window which 
> > asks the user to choose a webcam app or a stillcam app. Even this minor 
> > innovation would stop a lot of user grief. Frankly, I am mystified that 
> > this was not done.
> 
> I think this is still not good. It should be more "fluid". If the camera
> has separate alternate settings or different configurations, all it
> takes (from a USB standpoint) is to send the correct SetConfiguration or
> SetInterface command and the camera should happily change its mode of
> operation.

Often true, but not necessarily. 

> 
> Which means that a simple ioctl() to change the mode of operation would
> suffice should the driver be combined.
> 
> > This still leaves the problem (see item 3, above) that a dual-mode camera 
> > needs to be re-plugged in order to re-enable the access to /dev/video* if 
> > it has been used in stillcam mode. If it is possible to do a re-plug in 
> > software, this would help a lot. It does not matter if the re-plug is done 
> > in userspace or in kernelspace, so long as it can be done, somehow, after 
> > libusb relinquishes the camera. Or, fix things up somehow so that the 
> > kernel module automatically re-activates itself when the userspace app 
> > (using libgphoto2) is closed down.
> > 
> > Finally, another possible alternative would be to figure out how to do 
> > something in the kernel module which permits the camera to be accessed by 
> > libusb.
> 
> I don't see the possibility of combining both drivers and use
> /dev/videoX for streaming and something like /dev/camX for still image.
> Now, the only thing you have to be sure in kernel space is that you
> don't allow both to be accessed together, but there are ways to handle
> it.

Which leaves a problem for still cameras. Most of them, in fact, do not 
need any such /dev/camX as you are proposing, at all. The only cameras 
which need the kernel are those which need /dev/videoX. For pure still 
cameras, /dev/videoX is entirely superfluous, at best. Dual-mode cameras 
only need /dev/videoX when streaming and otherwise need for /dev/videoX to 
go away.

I do not believe that we have found the optimal solution, yet. The ideal 
thing would be some kind of hack which allows the kernel to be used when 
it is needed, and when it is not needed it does not interfere.

Theodore Kilgore

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Theodore Kilgore]

On Fri, 10 Jun 2011, Alan Stern wrote:

> On Fri, 10 Jun 2011, Felipe Balbi wrote:
> 
> > I don't see any problems in this situation. If, for that particular
> > product, webcam and still image functionality are mutually exclusive,
> > then that's how the product (and their drivers) have to work.
> > 
> > If the linux community decided to put webcam functionality in kernel and
> > still image functionality on a completely separate driver, that's
> > entirely our problem.
> 
> And the problem is how to coordinate the two of them.
> 
> > > 2. Until recently in the history of Linux, there was an irreconcilable 
> > > conflict. If a kernel driver for the video streaming mode was present and 
> > > installed, it was not possible to use the camera in stillcam mode at all. 
> > > Thus the only solution to the problem was to blacklist the kernel module 
> > > so that it would not get loaded automatically and only to install said 
> > > module by hand if the camera were to be used in video streaming mode, then 
> > > to rmmod it immediately afterwards. Very cumbersome, obviously. 
> > 
> > true... but why couldn't we combine both in kernel or in userspace
> > altogether ? Why do we have this split ? (words from a newbie in V4L2,
> > go easy :-p)
> 
> I think the problem may be that the PTP protocol used in the still-cam
> mode isn't suitable for a kernel driver.  Or if it is suitable, it
> would have to be something like a shared-filesystem driver -- nothing
> like a video driver.  You certainly wouldn't want to put it in V4L2.
> 
> > Or, to move the libgphoto2 driver to kernel, combine it in the same
> > driver that handles streaming. No ?
> 
> No.  Something else is needed.
> 
> Alan Stern
> 

Agreed. Something else is needed. But what? Also, very good point about 
PTP.

Theodore Kilgore

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Xiaofan Chen]

On Sat, Jun 11, 2011 at 6:43 AM, Theodore Kilgore
<kilgota@banach.math.auburn.edu> wrote:
> I do not believe that we have found the optimal solution, yet. The ideal
> thing would be some kind of hack which allows the kernel to be used when
> it is needed, and when it is not needed it does not interfere.

Just wondering if you can use libusb-1.0 for the user space still image
functionality.

libusb-1.0 offers the following functions to do that for you under Linux.

int 	libusb_kernel_driver_active (libusb_device_handle *dev, int interface)
 	Determine if a kernel driver is active on an interface.
int 	libusb_detach_kernel_driver (libusb_device_handle *dev, int interface)
 	Detach a kernel driver from an interface.
int 	libusb_attach_kernel_driver (libusb_device_handle *dev, int interface)
 	Re-attach an interface's kernel driver, which was previously
detached using libusb_detach_kernel_driver().

So you can detach the kernel v4l2 driver at the beginning and later
re-attach it when you finish.

-- 
Xiaofan

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Theodore Kilgore]

On Sat, 11 Jun 2011, Xiaofan Chen wrote:

> On Sat, Jun 11, 2011 at 6:43 AM, Theodore Kilgore
> <kilgota@banach.math.auburn.edu> wrote:
> > I do not believe that we have found the optimal solution, yet. The ideal
> > thing would be some kind of hack which allows the kernel to be used when
> > it is needed, and when it is not needed it does not interfere.
> 
> Just wondering if you can use libusb-1.0 for the user space still image
> functionality.
> 
> libusb-1.0 offers the following functions to do that for you under Linux.
> 
> int 	libusb_kernel_driver_active (libusb_device_handle *dev, int interface)
>  	Determine if a kernel driver is active on an interface.
> int 	libusb_detach_kernel_driver (libusb_device_handle *dev, int interface)
>  	Detach a kernel driver from an interface.
> int 	libusb_attach_kernel_driver (libusb_device_handle *dev, int interface)
>  	Re-attach an interface's kernel driver, which was previously
> detached using libusb_detach_kernel_driver().
> 
> So you can detach the kernel v4l2 driver at the beginning and later
> re-attach it when you finish.

Well, then, this solves the problem, doesn't it? Of course, those who deal 
with creating those "simple" and "user-friendly" GUI environments would 
probably still do well if they would open a dialog box for dual-mode 
hardware.

Theodore Kilgore

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Hans de Goede]

Hi,

Given the many comments in this thread, I'm just
going reply to this one, and try to also answer any
other ones in this mail.

As far as the dual mode camera is involved, I agree
that that should be fixed in the existing v4l2
drivers + libgphoto. I think that Felipe's solution
to also handle the stillcam part in kernel space for
dual mode cameras (and add a libgphoto cam driver which
knows how to talk the new kernel API for this), is
the best solution. Unfortunately this will involve
quite a bit of work, but so be it.


On 06/10/2011 04:48 PM, Alan Stern wrote:
> On Fri, 10 Jun 2011, Hans de Goede wrote:
>
>> Hi all,
>>
>> The current API for managing kernel ->  userspace is a bit
>> rough around the edges, so I would like to discuss extending
>> the API.
>>
>> First of all an example use case scenarios where the current API
>> falls short.
>>
>> 1) Redirection of USB devices to a virtual machine, qemu, vbox, etc.
>> all have the ability to redirect a USB device to the virtual machine,
>> and they all use usbfs for this. The first thing which will happen
>> here when the user selects a device to redirect is a
>> IOCTL_USBFS_DISCONNECT. This causes the kernel driver to see a
>> device unplug, with no chances for the kernel driver to do anything
>> against this.
>>
>> Now lets say the user does the following:
>> -write a file to a usb flash disk
>> -redirect the flash disk to a vm
>>
>> Currently this will cause the usb mass storage driver to see a
>> disconnect, and any possible still pending writes are lost ...
>>
>> This is IMHO unacceptable, but currently there is nothing we can
>> do to avoid this.
>
> You haven't given a proper description of the problem.  See below.

I think I've given an excellent description of the problem, drivers
can be unbound from devices, and there is no way for drivers to block
this. All I'm asking for is for a new usbfs try_disconnect ioctl which
the currently bound driver has a chance blocking nothing more nothing
less.

<snip dual mode camera stuff>

>> So what do we need to make this situation better:
>> 1) A usb_driver callback alternative to the disconnect callback,
>>      I propose to call this soft_disconnect. This serves 2 purposes
>>      a) It will allow the driver to tell the caller that that is not
>>         a good idea by returning an error code (think usb mass storage
>>         driver and mounted filesystem
>
> Not feasible.  usb-storage has no idea whether or not a device it
> controls has a mounted filesystem.  (All it does is send SCSI commands
> to a device and get back the results.)  Since that's the main use
> case you're interested in, this part of the proposal seems destined to
> fail.
>

This is not completely true, I cannot rmmod usb-storage as long as
disks using it are mounted. I know this is done through the global
module usage count, so this is not per usb-storage device. But extending
the ref counting to be per usb-storage device should not be hard.

All the accounting is already done for this.

> But userspace _does_ know where the mounted filesystems are.
> Therefore userspace should be responsible for avoiding programs that
> want to take control of devices holding these filesystems.  That's the
> reason why usbfs device nodes are owned by root and have 0644 mode;
> there're can be written to only by programs with superuser privileges
> -- and such programs are supposed to be careful about what they do.
>

Yes, and what I'm asking for is for an easy way for these programs to
be careful. A way for them to ask the kernel, which in general is
responsible for things like this and traditionally does resource
management and things which come with that like refcounting: "unbind
the driver from this device unless the device is currently in use".

Why can't this be done in userspace, simply put:
1) Process a checks if the usb-storage device is not used
2) Process b mounts it after the check
3) Process a unbinds the driver

Yes this can be avoided if all binding/unbinding and all mounting
happens under control of a single instance. However this requires
completely re-inventing userspace...

Also note that the notion of this can be extended to other devices,
want to ubs-redirect a usb printer to a vm better not do it halfway
through a printjob being spooled. Want to usb-redirect a usb webcam
to a vm, better not do it while something is streaming video from
the webcam, etc. etc.

And please don't come with the inevitable "if it hurts do not do
that" argument. We want to provide this kind of functionality to
non tech savy computer users, and things should just work, including
telling the user that the device is currently in use rather then
wrecking his printjob or filesystem.

Regards,

Hans

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Theodore Kilgore]

On Sat, 11 Jun 2011, Hans de Goede wrote:

> Hi,
> 
> Given the many comments in this thread, I'm just
> going reply to this one, and try to also answer any
> other ones in this mail.
> 
> As far as the dual mode camera is involved, I agree
> that that should be fixed in the existing v4l2
> drivers + libgphoto. I think that Felipe's solution
> to also handle the stillcam part in kernel space for
> dual mode cameras (and add a libgphoto cam driver which
> knows how to talk the new kernel API for this), is
> the best solution. Unfortunately this will involve
> quite a bit of work, but so be it.

Hans,

It appears to me that the solution ought to be at hand, actually.

I was not aware of the recent changes in libusb, which I understand are 
supposed to allow a kernel driver to be hooked up again.

To review the situation:

1. As of approximately 2 years ago, libusb already was so configured as to 
suspend the kernel module for a dual-mode device if a userspace-based 
program tried to claim the device.

2. At this point with the more recent versions of libusb (see the last 
message from yesterday, from Xiaofan Chen), we are supposed to be able to 
re-activate the kernel module for the device when it is relinquished by 
userspace.

This ought to take care of the problems completely, provided that the new 
capabilities of libusb are actually used and called upon in libgphoto2.

I have checked on what is happening, just now, on my own machine. I have 
libusb version 1.08 which ought to be recent enough. The advertised 
abilities did not work, however. Presumably, what is missing is on the 
other end of the problem, most likely in the functions in libgphoto2 which 
hook up a camera. That code would presumably need to call upon the new 
functionality of libusb. My currently installed version of libgphoto2 
(from svn, but several months old) clearly does not contain the needed 
functionality. But it might have been put in recently and I did not 
notice. I guess that the first thing to do is to update my gphoto tree and 
then to see what happens. If things still don't work, then something needs 
to be updated and then things ought to work.

I will try to see that something gets done about this. Thank you for 
raising the old issue of dual-mode devices yet again, and thanks to 
Xiaofan Chen for pointing out that the needed missing half of the 
functionality is supposed to exist now in libusb. That had escaped my 
attention. 

Theodore Kilgore

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Alan Stern]

On Sat, 11 Jun 2011, Hans de Goede wrote:

> >> So what do we need to make this situation better:
> >> 1) A usb_driver callback alternative to the disconnect callback,
> >>      I propose to call this soft_disconnect. This serves 2 purposes
> >>      a) It will allow the driver to tell the caller that that is not
> >>         a good idea by returning an error code (think usb mass storage
> >>         driver and mounted filesystem
> >
> > Not feasible.  usb-storage has no idea whether or not a device it
> > controls has a mounted filesystem.  (All it does is send SCSI commands
> > to a device and get back the results.)  Since that's the main use
> > case you're interested in, this part of the proposal seems destined to
> > fail.
> >
> 
> This is not completely true, I cannot rmmod usb-storage as long as
> disks using it are mounted. I know this is done through the global
> module usage count, so this is not per usb-storage device. But extending
> the ref counting to be per usb-storage device should not be hard.
> 
> All the accounting is already done for this.

It would be harder than you think.  All the accounting is _not_ already
being done.  What you're talking about would amount to a significant
change in the driver model core and the SCSI core.  It isn't just a USB
thing.

> > But userspace _does_ know where the mounted filesystems are.
> > Therefore userspace should be responsible for avoiding programs that
> > want to take control of devices holding these filesystems.  That's the
> > reason why usbfs device nodes are owned by root and have 0644 mode;
> > there're can be written to only by programs with superuser privileges
> > -- and such programs are supposed to be careful about what they do.
> >
> 
> Yes, and what I'm asking for is for an easy way for these programs to
> be careful. A way for them to ask the kernel, which in general is
> responsible for things like this and traditionally does resource
> management and things which come with that like refcounting: "unbind
> the driver from this device unless the device is currently in use".

Sure.  At the moment the kernel does not keep track of whether a device 
is currently in use -- at least, not in the way you mean.

I'm not saying this can't be done.  But it would be a bigger job than 
you think, and this isn't the appropriate thread to discuss it.

Alan Stern

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Hans de Goede]

Hi,

On 06/11/2011 06:19 PM, Theodore Kilgore wrote:
>
>
> On Sat, 11 Jun 2011, Hans de Goede wrote:
>
>> Hi,
>>
>> Given the many comments in this thread, I'm just
>> going reply to this one, and try to also answer any
>> other ones in this mail.
>>
>> As far as the dual mode camera is involved, I agree
>> that that should be fixed in the existing v4l2
>> drivers + libgphoto. I think that Felipe's solution
>> to also handle the stillcam part in kernel space for
>> dual mode cameras (and add a libgphoto cam driver which
>> knows how to talk the new kernel API for this), is
>> the best solution. Unfortunately this will involve
>> quite a bit of work, but so be it.
>
> Hans,
>
> It appears to me that the solution ought to be at hand, actually.
>
> I was not aware of the recent changes in libusb, which I understand are
> supposed to allow a kernel driver to be hooked up again.
>
> To review the situation:
>
> 1. As of approximately 2 years ago, libusb already was so configured as to
> suspend the kernel module for a dual-mode device if a userspace-based
> program tried to claim the device.
>
> 2. At this point with the more recent versions of libusb (see the last
> message from yesterday, from Xiaofan Chen), we are supposed to be able to
> re-activate the kernel module for the device when it is relinquished by
> userspace.
>
> This ought to take care of the problems completely, provided that the new
> capabilities of libusb are actually used and called upon in libgphoto2.
>
> I have checked on what is happening, just now, on my own machine. I have
> libusb version 1.08 which ought to be recent enough. The advertised
> abilities did not work, however. Presumably, what is missing is on the
> other end of the problem, most likely in the functions in libgphoto2 which
> hook up a camera. That code would presumably need to call upon the new
> functionality of libusb. My currently installed version of libgphoto2
> (from svn, but several months old) clearly does not contain the needed
> functionality. But it might have been put in recently and I did not
> notice. I guess that the first thing to do is to update my gphoto tree and
> then to see what happens. If things still don't work, then something needs
> to be updated and then things ought to work.
>
> I will try to see that something gets done about this. Thank you for
> raising the old issue of dual-mode devices yet again, and thanks to
> Xiaofan Chen for pointing out that the needed missing half of the
> functionality is supposed to exist now in libusb. That had escaped my
> attention.

Actually libusb and libgphoto have been using the rebind orginal driver
functionality of the code for quite a while now, unfortunately this
does not solve the problem, unless we somehow move to 1 central
coordinator for the device the user experience will stay subpar.

Example, user downloads pictures from the camera using shotwell,
gthumb, fspot or whatever, keeps the app in question open and the app
in question keeps the gphoto2 device handle open.

User wants to do some skyping with video chat, skype complains it
cannot find the device, since the kernel driver currently is unbound.

-> Poor user experience.

With having both functions in the kernel, the kernel could actually
allow skype to use the dual mode cameras as video source, and if
the user then were to switch to f-spot and try to import more photo's
then he will get an -ebusy in f-spot. If he finishes skyping and
then returns to f-spot everything will just continue working.

This is the kind of "seamless" user experience I'm aiming for here.

Regards,

Hans

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Theodore Kilgore]

On Sun, 12 Jun 2011, Hans de Goede wrote:

> Hi,
> 
> On 06/11/2011 06:19 PM, Theodore Kilgore wrote:
> > 
> > 
> > On Sat, 11 Jun 2011, Hans de Goede wrote:
> > 
> > > Hi,
> > > 
> > > Given the many comments in this thread, I'm just
> > > going reply to this one, and try to also answer any
> > > other ones in this mail.
> > > 
> > > As far as the dual mode camera is involved, I agree
> > > that that should be fixed in the existing v4l2
> > > drivers + libgphoto. I think that Felipe's solution
> > > to also handle the stillcam part in kernel space for
> > > dual mode cameras (and add a libgphoto cam driver which
> > > knows how to talk the new kernel API for this), is
> > > the best solution. Unfortunately this will involve
> > > quite a bit of work, but so be it.
> > 
> > Hans,
> > 
> > It appears to me that the solution ought to be at hand, actually.
> > 
> > I was not aware of the recent changes in libusb, which I understand are
> > supposed to allow a kernel driver to be hooked up again.
> > 
> > To review the situation:
> > 
> > 1. As of approximately 2 years ago, libusb already was so configured as to
> > suspend the kernel module for a dual-mode device if a userspace-based
> > program tried to claim the device.
> > 
> > 2. At this point with the more recent versions of libusb (see the last
> > message from yesterday, from Xiaofan Chen), we are supposed to be able to
> > re-activate the kernel module for the device when it is relinquished by
> > userspace.
> > 
> > This ought to take care of the problems completely, provided that the new
> > capabilities of libusb are actually used and called upon in libgphoto2.
> > 
> > I have checked on what is happening, just now, on my own machine. I have
> > libusb version 1.08 which ought to be recent enough. The advertised
> > abilities did not work, however. Presumably, what is missing is on the
> > other end of the problem, most likely in the functions in libgphoto2 which
> > hook up a camera. That code would presumably need to call upon the new
> > functionality of libusb. My currently installed version of libgphoto2
> > (from svn, but several months old) clearly does not contain the needed
> > functionality. But it might have been put in recently and I did not
> > notice. I guess that the first thing to do is to update my gphoto tree and
> > then to see what happens. If things still don't work, then something needs
> > to be updated and then things ought to work.
> > 
> > I will try to see that something gets done about this. Thank you for
> > raising the old issue of dual-mode devices yet again, and thanks to
> > Xiaofan Chen for pointing out that the needed missing half of the
> > functionality is supposed to exist now in libusb. That had escaped my
> > attention.
> 
> Actually libusb and libgphoto have been using the rebind orginal driver
> functionality of the code for quite a while now, 

Oh? I can see that libusb is doing that, and I can also see that there is 
a "public" function for _unbinding_ a kernel driver, namely 

int usb_detach_kernel_driver_np()

found in usb.h

and it is used in libgphoto, as well.

I am not sure that there is any corresponding rebind function which is 
public. Is it perhaps

int usb_get_driver_np()

???

By context (looking at libgphoto2-port/usb/libusb.c) I would think that 
this function is not the rebind function, but is only checking whether or 
not there is any potential conflict with a kernel driver. If I am right, 
then where is the publicly exported rebind function, and where does it 
currently get used in libgphoto2? 

So frankly after my eagerness yesterday I do not see how it can easily be 
made to work, after all.

unfortunately this
> does not solve the problem, unless we somehow move to 1 central
> coordinator for the device the user experience will stay subpar.
> 
> Example, user downloads pictures from the camera using shotwell,
> gthumb, fspot or whatever, keeps the app in question open and the app
> in question keeps the gphoto2 device handle open.
> 
> User wants to do some skyping with video chat, skype complains it
> cannot find the device, since the kernel driver currently is unbound.
> 
> -> Poor user experience.

Poor user experience, or merely poor user? The user ought to know better. 
Of course, I do agree that there are lots of such people, and it is a good 
idea to try to put up warning signs. 


> 
> With having both functions in the kernel, the kernel could actually
> allow skype to use the dual mode cameras as video source, and if
> the user then were to switch to f-spot and try to import more photo's
> then he will get an -ebusy in f-spot. If he finishes skyping and
> then returns to f-spot everything will just continue working.
> 
> This is the kind of "seamless" user experience I'm aiming for here.
> 
> Regards,
> 
> Hans

Yes, I can see where you are coming from. But if the camera really will 
not let you run skype and fspot at the same time, which I do not believe 
it would allow on _any_ operating system, then each app should give an 
error message which says it cannot be run unless and until the other app 
has been closed. If that has to happen at the kernel level, then OK.

Theodore Kilgore

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Xiaofan Chen]

On Mon, Jun 13, 2011 at 5:20 AM, Theodore Kilgore
<kilgota@banach.math.auburn.edu> wrote:
> On Sun, 12 Jun 2011, Hans de Goede wrote:
>> Actually libusb and libgphoto have been using the rebind orginal driver
>> functionality of the code for quite a while now,
>
> Oh? I can see that libusb is doing that, and I can also see that there is
> a "public" function for _unbinding_ a kernel driver, namely
>
> int usb_detach_kernel_driver_np()
>
> found in usb.h
>
> and it is used in libgphoto, as well.
>
> I am not sure that there is any corresponding rebind function which is
> public. Is it perhaps
>
> int usb_get_driver_np()
>
> ???
>
> By context (looking at libgphoto2-port/usb/libusb.c) I would think that
> this function is not the rebind function, but is only checking whether or
> not there is any potential conflict with a kernel driver. If I am right,
> then where is the publicly exported rebind function, and where does it
> currently get used in libgphoto2?

http://gphoto.svn.sourceforge.net/viewvc/gphoto/trunk/libgphoto2/libgphoto2_port/usb/libusb.c?revision=13652&view=markup

The rebind happened under the function "static int gp_port_usb_close
(GPPort *port)".
Since libgphoto2 is still using libusb-0.1, the unbind is using usbfs IOCTL
directly (USBDEVFS_CONNECT).

> So frankly after my eagerness yesterday I do not see how it can easily be
> made to work, after all.
>
>> unfortunately this
>> does not solve the problem, unless we somehow move to 1 central
>> coordinator for the device the user experience will stay subpar.

Now I understand what Hans is saying. It will be a lot of work trying
to sort out this issue in userspace. What can be the single central
coordinator? A device manager applet listing the program or service
which hold the device?

>> Example, user downloads pictures from the camera using shotwell,
>> gthumb, fspot or whatever, keeps the app in question open and the app
>> in question keeps the gphoto2 device handle open.
>>
>> User wants to do some skyping with video chat, skype complains it
>> cannot find the device, since the kernel driver currently is unbound.
>>
>> -> Poor user experience.
>
> Poor user experience, or merely poor user? The user ought to know better.
> Of course, I do agree that there are lots of such people, and it is a good
> idea to try to put up warning signs.

It is difficult to call the users "poor users" in this case. Since they may
not know that the other open program is holding the device. Some
warning message may help, not "I can not find the device" though. It
would be better to pinpoint which program is holding the device
and then ask the user to close that program. I understand this is
easily said than done...

Similar experiences for Windows about the serial port, sometimes
it is difficult for the user to know that some program or service
are holding the serial port so that the other program or will fail or
Windows complain that it is still open when you want to undock
the computer.

>>
>> With having both functions in the kernel, the kernel could actually
>> allow skype to use the dual mode cameras as video source, and if
>> the user then were to switch to f-spot and try to import more photo's
>> then he will get an -ebusy in f-spot. If he finishes skyping and
>> then returns to f-spot everything will just continue working.
>>
>> This is the kind of "seamless" user experience I'm aiming for here.
>>
> Yes, I can see where you are coming from. But if the camera really will
> not let you run skype and fspot at the same time, which I do not believe
> it would allow on _any_ operating system, then each app should give an
> error message which says it cannot be run unless and until the other app
> has been closed. If that has to happen at the kernel level, then OK.
>

Yes. From what I read, to solve it in kernel or to solve it in user space
are both a lot of work.

Personally I tend to think to solve it in user space is more feasible.

-- 
Xiaofan

Re: [Libusb-devel] Improving kernel -> userspace (usbfs) usb device hand off

[Michael Bender]

On Jun 12, 2011, at 7:03 PM, Xiaofan Chen wrote:

> On Mon, Jun 13, 2011 at 5:20 AM, Theodore Kilgore
> <kilgota@banach.math.auburn.edu> wrote:
>> On Sun, 12 Jun 2011, Hans de Goede wrote:
>>> Actually libusb and libgphoto have been using the rebind orginal  
>>> driver
>>> functionality of the code for quite a while now,
>>
>> Oh? I can see that libusb is doing that, and I can also see that  
>> there is
>> a "public" function for _unbinding_ a kernel driver, namely
>>
>> int usb_detach_kernel_driver_np()
>>
>> found in usb.h
>>
>> and it is used in libgphoto, as well.
>>
>> I am not sure that there is any corresponding rebind function which  
>> is
>> public. Is it perhaps
>>
>> int usb_get_driver_np()
>>
>> ???
>>
>> By context (looking at libgphoto2-port/usb/libusb.c) I would think  
>> that
>> this function is not the rebind function, but is only checking  
>> whether or
>> not there is any potential conflict with a kernel driver. If I am  
>> right,
>> then where is the publicly exported rebind function, and where does  
>> it
>> currently get used in libgphoto2?
>
> http://gphoto.svn.sourceforge.net/viewvc/gphoto/trunk/libgphoto2/libgphoto2_port/usb/libusb.c?revision=13652&view=markup
>
> The rebind happened under the function "static int gp_port_usb_close
> (GPPort *port)".
> Since libgphoto2 is still using libusb-0.1, the unbind is using  
> usbfs IOCTL
> directly (USBDEVFS_CONNECT).
>
>> So frankly after my eagerness yesterday I do not see how it can  
>> easily be
>> made to work, after all.
>>
>>> unfortunately this
>>> does not solve the problem, unless we somehow move to 1 central
>>> coordinator for the device the user experience will stay subpar.
>
> Now I understand what Hans is saying. It will be a lot of work trying
> to sort out this issue in userspace. What can be the single central
> coordinator? A device manager applet listing the program or service
> which hold the device?

Something like that. Have a user-space device allocation mechanism so
that apps are not constrained by a particular interface semantic (i.e.  
not
required to open /dev/video and have a kernel driver deliver pixels to  
the
apps).

Or hid that behind a library API and let instances of the library  
coordinate
with each other; as long as an app uses the documented public library  
API
to access a webcam, then everyone will play fair with each other.

Look at the mess that this userspace/kernel driver issue has for code in
an application like gphoto - that's insane that a photo manipulation app
needs to know anything about userspace/kernel switching!

>>> Example, user downloads pictures from the camera using shotwell,
>>> gthumb, fspot or whatever, keeps the app in question open and the  
>>> app
>>> in question keeps the gphoto2 device handle open.
>>>
>>> User wants to do some skyping with video chat, skype complains it
>>> cannot find the device, since the kernel driver currently is  
>>> unbound.
>>>
>>> -> Poor user experience.
>>
>> Poor user experience, or merely poor user? The user ought to know  
>> better.
>> Of course, I do agree that there are lots of such people, and it is  
>> a good
>> idea to try to put up warning signs.
>
> It is difficult to call the users "poor users" in this case. Since  
> they may
> not know that the other open program is holding the device. Some
> warning message may help, not "I can not find the device" though. It
> would be better to pinpoint which program is holding the device
> and then ask the user to close that program. I understand this is
> easily said than done...
>
> Similar experiences for Windows about the serial port, sometimes
> it is difficult for the user to know that some program or service
> are holding the serial port so that the other program or will fail or
> Windows complain that it is still open when you want to undock
> the computer.
>
>>>
>>> With having both functions in the kernel, the kernel could actually
>>> allow skype to use the dual mode cameras as video source, and if
>>> the user then were to switch to f-spot and try to import more  
>>> photo's
>>> then he will get an -ebusy in f-spot. If he finishes skyping and
>>> then returns to f-spot everything will just continue working.
>>>
>>> This is the kind of "seamless" user experience I'm aiming for here.
>>>
>> Yes, I can see where you are coming from. But if the camera really  
>> will
>> not let you run skype and fspot at the same time, which I do not  
>> believe
>> it would allow on _any_ operating system, then each app should give  
>> an
>> error message which says it cannot be run unless and until the  
>> other app
>> has been closed. If that has to happen at the kernel level, then OK.
>>
>
> Yes. From what I read, to solve it in kernel or to solve it in user  
> space
> are both a lot of work.

Yes and a kernel-based solution locks you into a kernel-based webcam
driver paradigm, or an even uglier loopback driver.

> Personally I tend to think to solve it in user space is more feasible.

I agree.

mike

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

Hi,

On Fri, Jun 10, 2011 at 05:43:06PM -0500, Theodore Kilgore wrote:
> > there's nothing in the USB spec that says you need different product IDs
> > for different modes of operation. No matter if it's still or webcam
> > configuration, the underlying function is the same: capture images using
> > a set of lenses and image sensor.
> 
> True, true. But I will add that most of these cameras are Class 255, 
> Subclass 255, Protocol 255 (Proprietary, Proprietary, Proprietary).

well, if the manufacturer doesn't want to implement UVC for whatever
reason, it's his call ;-)

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: USB mini-summit at LinuxCon Vancouver

[Alexander Graf]

Am 10.06.2011 um 22:50 schrieb Greg KH <greg@kroah.com>:

> On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
>> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
>>> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
>>>> Topic 1
>>>> -------
>>>> 
>>>> The KVM folks suggested that it would be good to get USB and
>>>> virtualization developers together to talk about how to virtualize the
>>>> xHCI host controller.  The xHCI spec architect worked closely with
>>>> VMWare to get some extra goodies in the spec to help virtualization, and
>>>> I'd like to see the other virtualization developers take advantage of
>>>> that.  I'd also like us to hash out any issues they have been finding in
>>>> the USB core or xHCI driver during the virtualization effort.
>>> 
>>> Do people really want to virtualize the whole xHCI controller, or just
>>> specific ports or devices to the guest operating system?
>> 
>> A host OS could chose to virtualize the whole xHCI controller if it
>> wanted to.  That's part of the reason why xHCI does all the bandwidth
>> checking in hardware, not in software.
> 
> And here I thought it did that so it would be "correct" :)
> 
>> 
>>> If just specific ports, would something like usbip be better for virtual
>>> machines, with the USB traffic going over the network connection between
>>> the guest/host?
>> 
>> It could be done that way too.  But that doesn't help if you're trying
>> to run Windows under Linux, right?  Only if all the guest OSes use the
>> same USB IP protocol then it would work.
> 
> usbip works on Windows as well as Linux.

Do you have a reliable, working usbip solution at hand that work on Windows and Linux and doesn't require real network access, which can be a no-go for some scenarios?

> 
> But how could you run Windows with a xHCI controller in a guest, as
> Windows has no xHCI driver?  What would it expect to see?

There are drivers for xhci adapters on Windows. Also, this whole discussion is pretty much future oriented - which most likely means built-in xhci drivers anywhere.

No, in all seriousity, the main reason to go for FV vs PV is that so far, the best tested and working drivers are built for real hardware. I'm still unsatisfied with the PV driver situation for virtio on Windows. It's just incredibly hard to get Windows drivers right - and open source developers sure are not good at it :)

But sure, let's talk about it during LinuxCon as well - there's a good chance you know more there than me :)

> 

Alex

Re: USB mini-summit at LinuxCon Vancouver

[Alexander Graf]

Am 10.06.2011 um 22:50 schrieb Greg KH <greg@kroah.com>:

> On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
>> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
>>> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
>>>> Topic 1
>>>> -------
>>>> 
>>>> The KVM folks suggested that it would be good to get USB and
>>>> virtualization developers together to talk about how to virtualize the
>>>> xHCI host controller.  The xHCI spec architect worked closely with
>>>> VMWare to get some extra goodies in the spec to help virtualization, and
>>>> I'd like to see the other virtualization developers take advantage of
>>>> that.  I'd also like us to hash out any issues they have been finding in
>>>> the USB core or xHCI driver during the virtualization effort.
>>> 
>>> Do people really want to virtualize the whole xHCI controller, or just
>>> specific ports or devices to the guest operating system?
>> 
>> A host OS could chose to virtualize the whole xHCI controller if it
>> wanted to.  That's part of the reason why xHCI does all the bandwidth
>> checking in hardware, not in software.
> 
> And here I thought it did that so it would be "correct" :)
> 
>> 
>>> If just specific ports, would something like usbip be better for virtual
>>> machines, with the USB traffic going over the network connection between
>>> the guest/host?
>> 
>> It could be done that way too.  But that doesn't help if you're trying
>> to run Windows under Linux, right?  Only if all the guest OSes use the
>> same USB IP protocol then it would work.
> 
> usbip works on Windows as well as Linux.

Do you have a reliable, working usbip solution at hand that work on Windows and Linux and doesn't require real network access, which can be a no-go for some scenarios?

> 
> But how could you run Windows with a xHCI controller in a guest, as
> Windows has no xHCI driver?  What would it expect to see?

There are drivers for xhci adapters on Windows. Also, this whole discussion is pretty much future oriented - which most likely means built-in xhci drivers anywhere.

No, in all seriousity, the main reason to go for FV vs PV is that so far, the best tested and working drivers are built for real hardware. I'm still unsatisfied with the PV driver situation for virtio on Windows. It's just incredibly hard to get Windows drivers right - and open source developers sure are not good at it :)

But sure, let's talk about it during LinuxCon as well - there's a good chance you know more there than me :)

> 

Alex

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Mauro Carvalho Chehab]

Em 13-06-2011 06:05, Felipe Balbi escreveu:
> Hi,
> 
> On Fri, Jun 10, 2011 at 05:43:06PM -0500, Theodore Kilgore wrote:
>>> there's nothing in the USB spec that says you need different product IDs
>>> for different modes of operation. No matter if it's still or webcam
>>> configuration, the underlying function is the same: capture images using
>>> a set of lenses and image sensor.
>>
>> True, true. But I will add that most of these cameras are Class 255, 
>> Subclass 255, Protocol 255 (Proprietary, Proprietary, Proprietary).
> 
> well, if the manufacturer doesn't want to implement UVC for whatever
> reason, it's his call ;-)

This argument is bogus.

UVC were implemented too late. There are lots of chipsets that are not UVC-compliant,
simply because there were no UVC at the time those chipsets were designed.

Still today, newer devices using those chipsets are still at the market.

This is the same as saying that we should not support USB 1.1 or USB 2.0
because they're not fully USB 3.0 compliant.

Mauro.

Re: Improving kernel -> userspace (usbfs) usb device hand off

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

Hi,

On Mon, Jun 13, 2011 at 10:06:58AM -0300, Mauro Carvalho Chehab wrote:
> Em 13-06-2011 06:05, Felipe Balbi escreveu:
> > Hi,
> > 
> > On Fri, Jun 10, 2011 at 05:43:06PM -0500, Theodore Kilgore wrote:
> >>> there's nothing in the USB spec that says you need different product IDs
> >>> for different modes of operation. No matter if it's still or webcam
> >>> configuration, the underlying function is the same: capture images using
> >>> a set of lenses and image sensor.
> >>
> >> True, true. But I will add that most of these cameras are Class 255, 
> >> Subclass 255, Protocol 255 (Proprietary, Proprietary, Proprietary).
> > 
> > well, if the manufacturer doesn't want to implement UVC for whatever
> > reason, it's his call ;-)
> 
> This argument is bogus.
> 
> UVC were implemented too late. There are lots of chipsets that are not UVC-compliant,
> simply because there were no UVC at the time those chipsets were designed.
> 
> Still today, newer devices using those chipsets are still at the market.
> 
> This is the same as saying that we should not support USB 1.1 or USB 2.0
> because they're not fully USB 3.0 compliant.

I would think the small wink at the end was enough to label the reply as
a joke. Apparently not :-)

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Mon, Jun 13, 2011 at 12:44:57PM +0200, Alexander Graf wrote:
> 
> Am 10.06.2011 um 22:50 schrieb Greg KH <greg@kroah.com>:
> 
> > On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
> >> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
> >>> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
> >>>> Topic 1
> >>>> -------
> >>>> 
> >>>> The KVM folks suggested that it would be good to get USB and
> >>>> virtualization developers together to talk about how to virtualize the
> >>>> xHCI host controller.  The xHCI spec architect worked closely with
> >>>> VMWare to get some extra goodies in the spec to help virtualization, and
> >>>> I'd like to see the other virtualization developers take advantage of
> >>>> that.  I'd also like us to hash out any issues they have been finding in
> >>>> the USB core or xHCI driver during the virtualization effort.
> >>> 
> >>> Do people really want to virtualize the whole xHCI controller, or just
> >>> specific ports or devices to the guest operating system?
> >> 
> >> A host OS could chose to virtualize the whole xHCI controller if it
> >> wanted to.  That's part of the reason why xHCI does all the bandwidth
> >> checking in hardware, not in software.
> > 
> > And here I thought it did that so it would be "correct" :)
> > 
> >> 
> >>> If just specific ports, would something like usbip be better for virtual
> >>> machines, with the USB traffic going over the network connection between
> >>> the guest/host?
> >> 
> >> It could be done that way too.  But that doesn't help if you're trying
> >> to run Windows under Linux, right?  Only if all the guest OSes use the
> >> same USB IP protocol then it would work.
> > 
> > usbip works on Windows as well as Linux.
> 
> Do you have a reliable, working usbip solution at hand that work on
> Windows and Linux and doesn't require real network access, which can
> be a no-go for some scenarios?

What do you mean by "real network access"?  A virtual network connection
to the guest and host should be sufficient, right?

> > But how could you run Windows with a xHCI controller in a guest, as
> > Windows has no xHCI driver?  What would it expect to see?
> 
> There are drivers for xhci adapters on Windows.

Not well-working ones, see the windows driver mailing list as well as
the libusb mailing list for problems that people are having due to there
not being any "official" Microsoft xhci drivers shipping yet.

> Also, this whole discussion is pretty much future oriented - which
> most likely means built-in xhci drivers anywhere.

Agreed.

thanks,

greg k-h

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Mon, Jun 13, 2011 at 12:44:57PM +0200, Alexander Graf wrote:
> 
> Am 10.06.2011 um 22:50 schrieb Greg KH <greg@kroah.com>:
> 
> > On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
> >> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
> >>> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
> >>>> Topic 1
> >>>> -------
> >>>> 
> >>>> The KVM folks suggested that it would be good to get USB and
> >>>> virtualization developers together to talk about how to virtualize the
> >>>> xHCI host controller.  The xHCI spec architect worked closely with
> >>>> VMWare to get some extra goodies in the spec to help virtualization, and
> >>>> I'd like to see the other virtualization developers take advantage of
> >>>> that.  I'd also like us to hash out any issues they have been finding in
> >>>> the USB core or xHCI driver during the virtualization effort.
> >>> 
> >>> Do people really want to virtualize the whole xHCI controller, or just
> >>> specific ports or devices to the guest operating system?
> >> 
> >> A host OS could chose to virtualize the whole xHCI controller if it
> >> wanted to.  That's part of the reason why xHCI does all the bandwidth
> >> checking in hardware, not in software.
> > 
> > And here I thought it did that so it would be "correct" :)
> > 
> >> 
> >>> If just specific ports, would something like usbip be better for virtual
> >>> machines, with the USB traffic going over the network connection between
> >>> the guest/host?
> >> 
> >> It could be done that way too.  But that doesn't help if you're trying
> >> to run Windows under Linux, right?  Only if all the guest OSes use the
> >> same USB IP protocol then it would work.
> > 
> > usbip works on Windows as well as Linux.
> 
> Do you have a reliable, working usbip solution at hand that work on
> Windows and Linux and doesn't require real network access, which can
> be a no-go for some scenarios?

What do you mean by "real network access"?  A virtual network connection
to the guest and host should be sufficient, right?

> > But how could you run Windows with a xHCI controller in a guest, as
> > Windows has no xHCI driver?  What would it expect to see?
> 
> There are drivers for xhci adapters on Windows.

Not well-working ones, see the windows driver mailing list as well as
the libusb mailing list for problems that people are having due to there
not being any "official" Microsoft xhci drivers shipping yet.

> Also, this whole discussion is pretty much future oriented - which
> most likely means built-in xhci drivers anywhere.

Agreed.

thanks,

greg k-h

Re: USB mini-summit at LinuxCon Vancouver

[Alexander Graf]

On 13.06.2011, at 18:29, Greg KH wrote:

> On Mon, Jun 13, 2011 at 12:44:57PM +0200, Alexander Graf wrote:
>> 
>> Am 10.06.2011 um 22:50 schrieb Greg KH <greg@kroah.com>:
>> 
>>> On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
>>>> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
>>>>> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
>>>>>> Topic 1
>>>>>> -------
>>>>>> 
>>>>>> The KVM folks suggested that it would be good to get USB and
>>>>>> virtualization developers together to talk about how to virtualize the
>>>>>> xHCI host controller.  The xHCI spec architect worked closely with
>>>>>> VMWare to get some extra goodies in the spec to help virtualization, and
>>>>>> I'd like to see the other virtualization developers take advantage of
>>>>>> that.  I'd also like us to hash out any issues they have been finding in
>>>>>> the USB core or xHCI driver during the virtualization effort.
>>>>> 
>>>>> Do people really want to virtualize the whole xHCI controller, or just
>>>>> specific ports or devices to the guest operating system?
>>>> 
>>>> A host OS could chose to virtualize the whole xHCI controller if it
>>>> wanted to.  That's part of the reason why xHCI does all the bandwidth
>>>> checking in hardware, not in software.
>>> 
>>> And here I thought it did that so it would be "correct" :)
>>> 
>>>> 
>>>>> If just specific ports, would something like usbip be better for virtual
>>>>> machines, with the USB traffic going over the network connection between
>>>>> the guest/host?
>>>> 
>>>> It could be done that way too.  But that doesn't help if you're trying
>>>> to run Windows under Linux, right?  Only if all the guest OSes use the
>>>> same USB IP protocol then it would work.
>>> 
>>> usbip works on Windows as well as Linux.
>> 
>> Do you have a reliable, working usbip solution at hand that work on
>> Windows and Linux and doesn't require real network access, which can
>> be a no-go for some scenarios?
> 
> What do you mean by "real network access"?  A virtual network connection
> to the guest and host should be sufficient, right?

Yes, which would either mean we inject a second network adapter into the guest which needs to be configured to an IP address, hence possibly colliding with some other IP range that's in use outside, or reuse some real network connection, which is neither always given, nor always desired. We have a full new PV serial protocol (virtio-serial) just for exactly that reason: To be able to run a daemon inside the guest that does not require IP access to talk to the host.

USB/IP really has its own huge set of issues. Sure, it might be good for certain use cases, but it's definitely not a one-size-fits-all solution. But so far, doing FV emulation right has worked out a lot better for most use cases than inventing PV solutions. Real hardware simply seems to be supported by others, while we can't afford to write drivers for all OSs :). So IMHO, PV should be the last resort, when emulating real hardware utterly fails. In this case, I believe the hardware designers finally got the spec usable for us, so it should be good! :)

> 
>>> But how could you run Windows with a xHCI controller in a guest, as
>>> Windows has no xHCI driver?  What would it expect to see?
>> 
>> There are drivers for xhci adapters on Windows.
> 
> Not well-working ones, see the windows driver mailing list as well as
> the libusb mailing list for problems that people are having due to there
> not being any "official" Microsoft xhci drivers shipping yet.

Yes, which is an issue that's hopefully gone when we're finally having xHCI actually properly emulated and all the infrastructure for it laid out. We still have UHCI and OHCI emulation today already to fall back to if really necessary, but installing an xHCI driver inside the guest really isn't more work than installing USB/IP inside the guest.


Alex

Re: USB mini-summit at LinuxCon Vancouver

[Alexander Graf]

On 13.06.2011, at 18:29, Greg KH wrote:

> On Mon, Jun 13, 2011 at 12:44:57PM +0200, Alexander Graf wrote:
>> 
>> Am 10.06.2011 um 22:50 schrieb Greg KH <greg@kroah.com>:
>> 
>>> On Fri, Jun 10, 2011 at 12:48:15PM -0700, Sarah Sharp wrote:
>>>> On Thu, Jun 09, 2011 at 08:18:05PM -0700, Greg KH wrote:
>>>>> On Thu, Jun 09, 2011 at 05:21:03PM -0700, Sarah Sharp wrote:
>>>>>> Topic 1
>>>>>> -------
>>>>>> 
>>>>>> The KVM folks suggested that it would be good to get USB and
>>>>>> virtualization developers together to talk about how to virtualize the
>>>>>> xHCI host controller.  The xHCI spec architect worked closely with
>>>>>> VMWare to get some extra goodies in the spec to help virtualization, and
>>>>>> I'd like to see the other virtualization developers take advantage of
>>>>>> that.  I'd also like us to hash out any issues they have been finding in
>>>>>> the USB core or xHCI driver during the virtualization effort.
>>>>> 
>>>>> Do people really want to virtualize the whole xHCI controller, or just
>>>>> specific ports or devices to the guest operating system?
>>>> 
>>>> A host OS could chose to virtualize the whole xHCI controller if it
>>>> wanted to.  That's part of the reason why xHCI does all the bandwidth
>>>> checking in hardware, not in software.
>>> 
>>> And here I thought it did that so it would be "correct" :)
>>> 
>>>> 
>>>>> If just specific ports, would something like usbip be better for virtual
>>>>> machines, with the USB traffic going over the network connection between
>>>>> the guest/host?
>>>> 
>>>> It could be done that way too.  But that doesn't help if you're trying
>>>> to run Windows under Linux, right?  Only if all the guest OSes use the
>>>> same USB IP protocol then it would work.
>>> 
>>> usbip works on Windows as well as Linux.
>> 
>> Do you have a reliable, working usbip solution at hand that work on
>> Windows and Linux and doesn't require real network access, which can
>> be a no-go for some scenarios?
> 
> What do you mean by "real network access"?  A virtual network connection
> to the guest and host should be sufficient, right?

Yes, which would either mean we inject a second network adapter into the guest which needs to be configured to an IP address, hence possibly colliding with some other IP range that's in use outside, or reuse some real network connection, which is neither always given, nor always desired. We have a full new PV serial protocol (virtio-serial) just for exactly that reason: To be able to run a daemon inside the guest that does not require IP access to talk to the host.

USB/IP really has its own huge set of issues. Sure, it might be good for certain use cases, but it's definitely not a one-size-fits-all solution. But so far, doing FV emulation right has worked out a lot better for most use cases than inventing PV solutions. Real hardware simply seems to be supported by others, while we can't afford to write drivers for all OSs :). So IMHO, PV should be the last resort, when emulating real hardware utterly fails. In this case, I believe the hardware designers finally got the spec usable for us, so it should be good! :)

> 
>>> But how could you run Windows with a xHCI controller in a guest, as
>>> Windows has no xHCI driver?  What would it expect to see?
>> 
>> There are drivers for xhci adapters on Windows.
> 
> Not well-working ones, see the windows driver mailing list as well as
> the libusb mailing list for problems that people are having due to there
> not being any "official" Microsoft xhci drivers shipping yet.

Yes, which is an issue that's hopefully gone when we're finally having xHCI actually properly emulated and all the infrastructure for it laid out. We still have UHCI and OHCI emulation today already to fall back to if really necessary, but installing an xHCI driver inside the guest really isn't more work than installing USB/IP inside the guest.


Alex

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Hi Sarah/Greg,

Em 09-06-2011 21:21, Sarah Sharp escreveu:
> I'm pleased to announce a USB mini-summit at LinuxCon Vancouver.
> 
> What:	USB mini-summit
> When:	Tuesday, August 16th, all day
> Where:	At the conference venue, room TBD pending confirmation from
> 	Angela Brown.
> 
> Proposed topics include USB virtualization, and improved bandwidth APIs
> between the USB core and drivers (especially webcam drivers).  See the
> detailed topic list below.  Anyone is also welcome to propose or show up
> with a USB related topic.  MUSB?  USB 3.0 gadget drivers?  USB-IP?
> 
> The USB mini-summit does overlap with the virtualization mini-summit by
> a day, but I'm hoping we can schedule talks so some of the
> virtualization folks can make it to the USB mini-summit.  The other
> option was on Friday during the conference which was not ideal.
> 
> Proposed topics:
> 
> Topic 1
> -------
> 
> The KVM folks suggested that it would be good to get USB and
> virtualization developers together to talk about how to virtualize the
> xHCI host controller.  The xHCI spec architect worked closely with
> VMWare to get some extra goodies in the spec to help virtualization, and
> I'd like to see the other virtualization developers take advantage of
> that.  I'd also like us to hash out any issues they have been finding in
> the USB core or xHCI driver during the virtualization effort.
> 
> 
> Topic 2
> -------
> 
> I'd also like to get the V4L and audio developers who work with USB
> devices together with the core USB folks to talk about bandwidth
> management under xHCI.
> 
> One of the issues is that since the xHCI hardware does bandwidth
> management, not the xHCI driver, a schedule that will take too much
> bandwidth will get rejected much sooner than any USB driver currently
> expects (during a call to usb_set_interface).  This poses issues, since
> most USB video drivers negotiate the video size and frame rate after
> they call usb_set_interface, so they don't know whether they can fall
> back to a less bandwidth-intensive setting.  Currently, they just submit
> URBs with less and less bandwidth until one interval setting gets
> accepted that won't work under xHCI.
> 
> A second issue is that that some drivers need less bandwidth than the
> device advertises, and the xHCI driver currently uses whatever periodic
> interval the device advertises in its descriptors.  This is not what the
> video/audio driver wants, especially in the case of buggy high speed
> devices that advertise the interval in frames, not microframes.  There
> needs to be some way for the drivers to communicate their bandwidth
> needs to the USB core.  We've known about this issue for a while, and I
> think it's time to get everyone in the same room and hash out an API.
> 
> (I will send out an API proposal later this month.)

While discussing the topics for the media workshop that will happen together
with this year's KS/2011, one issue related to the USB stack came on our
discussions: it is related to multi-function USB devices, as you can
see on this thread:
	http://www.spinics.net/lists/linux-media/msg36195.html

We have several cases of multi-function devices at the media subsystem.
For example, most TV grabber devices provides Remote Controller, Video,
audio and MPEG streaming capabilities, either implementing the standard
USB API's, or the vendor class API's. There are even some devices that
support USB storage, 3G modem and Digital TV.

Some of those devices have some resources that are mutually exclusive.

For example, some Digital Cameras can provide either access to the stored
images, or can be used as webcams. Using them as webcams automatically
deletes all stored images from it. So, a properly implemented driver
should be returning -EBUSY (or -EPERM?) if someone tries to stream for
such devices if is there any pictures stored on it. Symmetrically,
accessing the stored pictures there should return -EBUSY if the device
is streaming.

Currently, the data access for Digital Cameras is implemented via libusb,
libgphoto and gvfs, while the streaming interface is implemented via
a gspca Kernel driver.

I know that this problem were somewhat solved for 3G modems, with the usage
of the userspace problem usb_modeswitch, and with some quirks for the USB
storage driver, but I'm not sure if such tricks will scale forever, as more
functions are seen on some USB devices.

So, if we have some time, maybe we could start some discussions about that
during the USB mini-summit.

It should be noticed that the media subsystem has currently similar problems 
on devices that provide both analog and digital TV, as some resources, like 
the tuner can't be used simultaneously by the two API's. This happens even
with PCI devices. So, maybe, in the end, we'll be writing some kernel library
for resource locking, but it would be good if we can have preliminary
discussions there, and let the final discussions to happen during KS/2011. 

What do you think?

Thanks!
Mauro

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> I know that this problem were somewhat solved for 3G modems, with the usage
> of the userspace problem usb_modeswitch, and with some quirks for the USB
> storage driver, but I'm not sure if such tricks will scale forever, as more
> functions are seen on some USB devices.

Well, no matter how it "scales" it needs to be done in userspace, like
usb_modeswitch does.  We made that decision a while ago, and it is
working out very well.  I see no reason why you can't do it in userspace
as well as that is the easiest place to control this type of thing.

I thought we had a long discussion about this topic a while ago and came
to this very conclusion.  Or am I mistaken?

thanks,

greg k-h

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 04-08-2011 20:22, Matthew Dharm escreveu:
> On Thu, Aug 4, 2011 at 3:56 PM, Greg KH <greg@kroah.com <mailto:greg@kroah.com>> wrote:
> 
>     On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>     > I know that this problem were somewhat solved for 3G modems, with the usage
>     > of the userspace problem usb_modeswitch, and with some quirks for the USB
>     > storage driver, but I'm not sure if such tricks will scale forever, as more
>     > functions are seen on some USB devices.
> 
>     Well, no matter how it "scales" it needs to be done in userspace, like
>     usb_modeswitch does.  We made that decision a while ago, and it is
>     working out very well.  I see no reason why you can't do it in userspace
>     as well as that is the easiest place to control this type of thing.
> 
>     I thought we had a long discussion about this topic a while ago and came
>     to this very conclusion.  Or am I mistaken?
> 
>  
> We keep having the discussion over and over again.  But, you are correct: the conclusion was that this all needs to live in userspace.

In the case of 3G modem x USB storage only, it is possible to handle it on userspace.

However, when there are more functions added, an they're not (completely) mutually exclusive,
then I don't see an easy way (if is there a way) for doing it at userspace.

Several devices offer more than one function at the same time, but some
resources are mutually exclusive. A TV stick with just one tuner, and
both analog and digital demods offer both analog and digital TV at the
same time. So, both analog and digital parts of the driver will offer
the device to userspace. However, unpredictable results will happen if
userspace tries to use both at the same time.

The Digital camera devices that offer PTP transfers and V4L support also
fall at the same type of trouble. Some of those devices just delete
all pictures from the memory, if streaming is started. So, receiving
an automatic Skype video call may delete all pictures you took.

Worse than that, currently, the PTP protocol is handled via libusb, while
streaming is done via V4L2 API.

The best technical approach, IMO, is to implement the PTP protocol in
kernelspace, and do some sort of inter-subsystem locking to prevent such
troubles.

Regards,
Mauro

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Thu, 4 Aug 2011, Greg KH wrote:

> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > I know that this problem were somewhat solved for 3G modems, with the usage
> > of the userspace problem usb_modeswitch, and with some quirks for the USB
> > storage driver, but I'm not sure if such tricks will scale forever, as more
> > functions are seen on some USB devices.
> 
> Well, no matter how it "scales" it needs to be done in userspace, like
> usb_modeswitch does.  We made that decision a while ago, and it is
> working out very well.  I see no reason why you can't do it in userspace
> as well as that is the easiest place to control this type of thing.
> 
> I thought we had a long discussion about this topic a while ago and came
> to this very conclusion.  Or am I mistaken?
> 
> thanks,
> 
> greg k-h
> 

Greg,

A little bit more of precision would help me a lot, here. Precisely what 
is one supposed to do in userspace? I would naturally assume that the user 
will control the device by announcing what the user wants to do with the 
device. Such control would therefore have to start in userspace. But this 
is such an obvious truism that you must mean something else. What, then?

To say to do things "like usb_modeswitch" is rather vague. For one thing, 
usb_modeswitch, being something out there in userspace, does _not_ affect 
what is in the kernel. However, what is (or is not) in the kernel might 
make it possible (or, in the alternative, impossible) to facilitate the 
action of some kind of userspace function-switching program similar to 
usb_modeswitch, not so? 

As to the "long discussion about this topic a while ago" that may have 
been a discussion in which I was also involved. For, I do remember a 
discussion about this topic a few months ago. In one respect my memory 
differs from yours, however: It was not my impression that any definite 
conclusions were reached, certainly not a consensus. BTW, I did try to lay 
out what I remembered as three alternatives that came up, along with 
arguments for and against each of them, in my message which kicked off 
this thread today. 

It is of course a real possibility that you have seen the perfect solution 
in your mind to these problems, which have vexed a lot of people for 
years, and others have not. If that is the way it is, then perhaps you 
just wish that we would all shut up, implement what is obvious, and we 
could all be happy. This could certainly be the case. You have written a 
lot more code than I have, and you obviously must have started doing that 
when you were at least 30 years younger, perhaps even 40 years younger, 
than I was when I started. So I would be the first to say that you are 
much better at it than I am. But there are others here, too, who seem 
actively to have been sucked into this discussion, and they are similarly 
younger and more clever than I am, too. 

Giving due consideration to this, it would probably get the job done a lot 
faster if you just take the time to describe what grand vision you 
yourself have in mind for the solution of the problem, with a sufficient 
accounting of the details that people can all see what it is and exactly 
how and why it would work absolutely perfectly, and then we can just get 
busy and do it.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Oliver Neukum]

Am Freitag, 5. August 2011, 00:56:03 schrieb Greg KH:
> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > I know that this problem were somewhat solved for 3G modems, with the usage
> > of the userspace problem usb_modeswitch, and with some quirks for the USB
> > storage driver, but I'm not sure if such tricks will scale forever, as more
> > functions are seen on some USB devices.
> 
> Well, no matter how it "scales" it needs to be done in userspace, like
> usb_modeswitch does.  We made that decision a while ago, and it is
> working out very well.  I see no reason why you can't do it in userspace
> as well as that is the easiest place to control this type of thing.
> 
> I thought we had a long discussion about this topic a while ago and came
> to this very conclusion.  Or am I mistaken?

Circumstances change. We want to keep the stuff in user space as much and
as long as we can. However user space has limitations:

- it has by necessity a race between resumption and access by others
- it cannot resume anything we run a (rw) filesystem over.

Furthermore, today PM actions that lead to a loss of mode are initiated
by user space. If we ever want to oportunistically suspend a system
we also need to restore mode from inside the kernel.

We could avoid all that trouble, if we persuaded vendors to use plain
USB configurations for those purposes.

	Regards
		Oliver

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/05/2011 12:56 AM, Greg KH wrote:
> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>> I know that this problem were somewhat solved for 3G modems, with the usage
>> of the userspace problem usb_modeswitch, and with some quirks for the USB
>> storage driver, but I'm not sure if such tricks will scale forever, as more
>> functions are seen on some USB devices.
>
> Well, no matter how it "scales" it needs to be done in userspace, like
> usb_modeswitch does.  We made that decision a while ago, and it is
> working out very well.  I see no reason why you can't do it in userspace
> as well as that is the easiest place to control this type of thing.
>
> I thought we had a long discussion about this topic a while ago and came
> to this very conclusion.  Or am I mistaken?

I think we've had multiple discussions about this, surrounding various
topics / use cases. I would not call the do it in userspace a conclusion.

I rather more have a feeling of getting stonewalled on this by various people
surrounding the usb system. Me shutting up on this is basically a case of:
"Discussion not getting anywhere and I don't have the time to do a kernel
  proof of concept myself right now".

To be clear about the stonewalling I'm not talking about the dual cam issue
here, nor about the morphing devices thing which usb_modeswitch fixes.

I think it is important to separate oranges from apples here, there are
at least 3 different problem classes which all seem to have gotten thrown
onto a pile here:

1) The reason Mauro suggested having some discussion on this at the
USB summit is because of a discussion about dual mode cameras on the
linux media list. Dual-mode cameras are (usually very cheap) digital
photo cameras which can take still pictures in stand alone mode and
store them in onboard memory, like regular digital photo cameras, this
is one mode. The other mode is they can operate as a regular webcam,
the hardware actually is more regular webcam hardware with some memory
and a battery added (usually no viewfinder screen).

These devices typically use 1 usb interface (and thus 1 driver) for
both modes. We currently have drivers for both modes, but they are
separate drivers, which leads to fighting for device ownership,
with the stillmode driver always winning as that is userspace and
it simply kicks of the kernel webcam /streaming mode driver making
it see a device unplug, even if an app was streaming from the device
at the time. The solution here is simple, move both functions to
1 driver, which can then properly return -EBUSY when the device
is active in one mode and an app tries to use it in the other mode.

This 1 driver will likely end up being a kernel driver, since doing
streaming drivers in userspace is problematic, as the v4l2 API assumes
a kernel device node, and we don't want to invent a new API, we want
something which works with existing applications.

2) So called morphing devices. For example a USB 3G modem which
initially shows up as a cdrom drive with windows software, only
to reveal its real identity (with completely different usb
descriptors) after some magic command. This is solved by the
usb_modeswitch command, nothing to see here move along.

3) Re-direction of usb devices to virtual machines. This works by using
the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
The basics of this work fine, but it lacks proper locking / safeguards
for when a vm takes over a usb device from the in kernel driver.

Example of the problem: usb mass storage devices is mounted, filesystem
is in use and dirty, with writebacks pending, user redirects to vm,
usb mass storage driver sees the cable beeing yanked out of the device.

Now we've lost the pending writebacks and have a dirty, if not journaling
possibly corrupt, fs on the device.

This is the issue on which I feel a bit stonewalled. Simple putting your
fingers in your ears and singing la la la do it in userspace is not going
to cut it here. There is no way to do this race free in userspace, unless
all possible callers of mount get modified. Moreover 99% of the necessary
accounting for this is already done in the kernel. We already have the notion
of a block device being in use. We simply need to add some code to pass
this notion to the usb mass storage driver, and add a new try_disconnect
callback for usb drivers. I'm not saying this is going to be completely
straight forward, but it ain't rocket science either. And it so very
obviously is the *right* thing to do, that I'm getting very tired of
the do it in userspace song I keep hearing.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouveroliver

[Oliver Neukum]

Am Freitag, 5. August 2011, 09:45:56 schrieb Hans de Goede:
> This is the issue on which I feel a bit stonewalled. Simple putting your
> fingers in your ears and singing la la la do it in userspace is not going
> to cut it here. There is no way to do this race free in userspace, unless
> all possible callers of mount get modified. Moreover 99% of the necessary
> accounting for this is already done in the kernel. We already have the notion
> of a block device being in use. We simply need to add some code to pass
> this notion to the usb mass storage driver, and add a new try_disconnect
> callback for usb drivers. I'm not saying this is going to be completely
> straight forward, but it ain't rocket science either. And it so very
> obviously is the right thing to do, that I'm getting very tired of
> the do it in userspace song I keep hearing.

Doing a try_disconnect() would also solve the dual camera issue.
But it doesn't really interfere with the user space vs. kernel space
issue. You simply have to expand the ioctl interface to have
an ioctl that triggers this API call in the kernel.

A V4L2 device would return an error if its device node is opened,
otherwise disconnect.

	Regards
		Oliver

		

Re: USB mini-summit at LinuxCon Vancouveroliver

[Hans de Goede]

Hi,

On 08/05/2011 09:59 AM, Oliver Neukum wrote:
> Am Freitag, 5. August 2011, 09:45:56 schrieb Hans de Goede:
>> This is the issue on which I feel a bit stonewalled. Simple putting your
>> fingers in your ears and singing la la la do it in userspace is not going
>> to cut it here. There is no way to do this race free in userspace, unless
>> all possible callers of mount get modified. Moreover 99% of the necessary
>> accounting for this is already done in the kernel. We already have the notion
>> of a block device being in use. We simply need to add some code to pass
>> this notion to the usb mass storage driver, and add a new try_disconnect
>> callback for usb drivers. I'm not saying this is going to be completely
>> straight forward, but it ain't rocket science either. And it so very
>> obviously is the right thing to do, that I'm getting very tired of
>> the do it in userspace song I keep hearing.
>
> Doing a try_disconnect() would also solve the dual camera issue.
> But it doesn't really interfere with the user space vs. kernel space
> issue. You simply have to expand the ioctl interface to have
> an ioctl that triggers this API call in the kernel.
>
> A V4L2 device would return an error if its device node is opened,
> otherwise disconnect.

Getting a bit offtopic here, but no a try_disconnect will fix the
userspace stillcam mode driver being able to disconnect the device
while the webcam function is active. If the webcam is not active
userspace will still "win", and possibly never return the device
back to the kernel driver (this already happens today with
gvfs-gphoto creating a fuse mount and keeping the device open
indefinitely, locking out the webcam function).

Likewise a v4l2 control panel like app (think alsamixer for
videodevs to set brightness / contrast, etc.) can keep the /dev/video
node open indefinitely. Unless we rewrite most of userspace, we need
to allow the device to be open in bode modes *at the same time* and
only fail with -EBUSY when something really exclusive is requested
(so not just having the device open, or setting contrast, but
trying to stream and read/delete pictures from the stillcam
memory at the same time).

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 05-08-2011 04:45, Hans de Goede escreveu:
> Hi,
> 
> On 08/05/2011 12:56 AM, Greg KH wrote:
>> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>>> I know that this problem were somewhat solved for 3G modems, with the usage
>>> of the userspace problem usb_modeswitch, and with some quirks for the USB
>>> storage driver, but I'm not sure if such tricks will scale forever, as more
>>> functions are seen on some USB devices.
>>
>> Well, no matter how it "scales" it needs to be done in userspace, like
>> usb_modeswitch does.  We made that decision a while ago, and it is
>> working out very well.  I see no reason why you can't do it in userspace
>> as well as that is the easiest place to control this type of thing.
>>
>> I thought we had a long discussion about this topic a while ago and came
>> to this very conclusion.  Or am I mistaken?
> 
> I think we've had multiple discussions about this, surrounding various
> topics / use cases. I would not call the do it in userspace a conclusion.
> 
> I rather more have a feeling of getting stonewalled on this by various people
> surrounding the usb system. Me shutting up on this is basically a case of:
> "Discussion not getting anywhere and I don't have the time to do a kernel
>  proof of concept myself right now".
> 
> To be clear about the stonewalling I'm not talking about the dual cam issue
> here, nor about the morphing devices thing which usb_modeswitch fixes.
> 
> I think it is important to separate oranges from apples here, there are
> at least 3 different problem classes which all seem to have gotten thrown
> onto a pile here:
> 
> 1) The reason Mauro suggested having some discussion on this at the
> USB summit is because of a discussion about dual mode cameras on the
> linux media list. Dual-mode cameras are (usually very cheap) digital
> photo cameras which can take still pictures in stand alone mode and
> store them in onboard memory, like regular digital photo cameras, this
> is one mode. The other mode is they can operate as a regular webcam,
> the hardware actually is more regular webcam hardware with some memory
> and a battery added (usually no viewfinder screen).

Yes, that's my concern.

> These devices typically use 1 usb interface (and thus 1 driver) for
> both modes. 

I also want to cover the case where there are two drivers involved, as
this is a typical trouble we need to do with all devices that have both
analog and digital TV's.

A typical media device exposes several different API's to userspace,
each with its own character device, and sometimes implemented by a
different driver.

For example, Conexant cx23102 USB chipsets provide 2 USB interfaces,
but require 3 different drivers, and exposes 4 different userspace
API's to userspace:

USB interface 0: Remote Controller
	event/input API - Handled by drivers/media/rc/mceusb.c

USB interface 1: Audio/Video
	ALSA API - Handled by drivers/media/video/cx231xx/cx231xx-audio.c
		It is a separate driver, called cx231xx-alsa
	V4L2 API - Handled by drivers/media/video/cx231xx/cx231xx-video.c
	DVB API - Handled by drivers/media/video/cx231xx/cx231xx-dvb.c
		Both DVB and V4L2 are currently part of the same driver
(cx231xx).

The mceusb is completely independent of the other drivers. The cx231xx-alsa
also is almost independent. It only shares a lock with the main cx231xx
driver, and uses the I2C functions implemented by the main driver.

Now, we're starting to see devices like ZTE MF845 that have TV, 3G and 
USB storage, exposing 2 more userspace API's to userspace (vfs and tty).

I don't think that userspace usb_modeswitch-like type of locking "scales"
such complex devices.

Besides that, on a large amount of USB sticks, if you enable both V4L2
and DVB, the power consumption may exceed the 500mA power limit of USB,
and can overheat the device. There are several reports of burnt devices
due to that. Also, there's no way for both to work, as they share the
same tuner, that can either be tuning an analog or a digital channel.

To me, the right solution seems to add some sort of locking schema that
will protect the common resources on complex hardware. Something like:

int get_hw_resource(struct device, enum hw_resource_type);
void put_hw_resource(struct device, enum hw_resource_type);

enum hw_resource_type {
	HW_RES_USB,
	HW_RES_I2C,
	HW_RES_TUNER,
	HW_RES_USB,
...
};

So that, when a device need, for example, an exclusive lock for the USB
device, for example, it could be doing something like:

rc = get_hw_resource(dev, HW_RES_USB);
if (rc < 0)
	return -EBUSY;

and, after finishing, doing a put_hw_resource().

> We currently have drivers for both modes, but they are
> separate drivers, which leads to fighting for device ownership,
> with the stillmode driver always winning as that is userspace and
> it simply kicks of the kernel webcam /streaming mode driver making
> it see a device unplug, even if an app was streaming from the device
> at the time. The solution here is simple, move both functions to
> 1 driver, which can then properly return -EBUSY when the device
> is active in one mode and an app tries to use it in the other mode.
> 
> This 1 driver will likely end up being a kernel driver, since doing
> streaming drivers in userspace is problematic, as the v4l2 API assumes
> a kernel device node, and we don't want to invent a new API, we want
> something which works with existing applications.
> 
> 2) So called morphing devices. For example a USB 3G modem which
> initially shows up as a cdrom drive with windows software, only
> to reveal its real identity (with completely different usb
> descriptors) after some magic command. This is solved by the
> usb_modeswitch command, nothing to see here move along.
> 
> 3) Re-direction of usb devices to virtual machines. This works by using
> the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> The basics of this work fine, but it lacks proper locking / safeguards
> for when a vm takes over a usb device from the in kernel driver.
> 
> Example of the problem: usb mass storage devices is mounted, filesystem
> is in use and dirty, with writebacks pending, user redirects to vm,
> usb mass storage driver sees the cable beeing yanked out of the device.
> 
> Now we've lost the pending writebacks and have a dirty, if not journaling
> possibly corrupt, fs on the device.
> 
> This is the issue on which I feel a bit stonewalled. Simple putting your
> fingers in your ears and singing la la la do it in userspace is not going
> to cut it here. There is no way to do this race free in userspace, unless
> all possible callers of mount get modified. Moreover 99% of the necessary
> accounting for this is already done in the kernel. We already have the notion
> of a block device being in use. We simply need to add some code to pass
> this notion to the usb mass storage driver, and add a new try_disconnect
> callback for usb drivers. I'm not saying this is going to be completely
> straight forward, but it ain't rocket science either. And it so very
> obviously is the *right* thing to do, that I'm getting very tired of
> the do it in userspace song I keep hearing.
> 
> Regards,
> 
> Hans

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Fri, 5 Aug 2011, Oliver Neukum wrote:

> Am Freitag, 5. August 2011, 00:56:03 schrieb Greg KH:
> > On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > > I know that this problem were somewhat solved for 3G modems, with the usage
> > > of the userspace problem usb_modeswitch, and with some quirks for the USB
> > > storage driver, but I'm not sure if such tricks will scale forever, as more
> > > functions are seen on some USB devices.
> > 
> > Well, no matter how it "scales" it needs to be done in userspace, like
> > usb_modeswitch does.  We made that decision a while ago, and it is
> > working out very well.  I see no reason why you can't do it in userspace
> > as well as that is the easiest place to control this type of thing.
> > 
> > I thought we had a long discussion about this topic a while ago and came
> > to this very conclusion.  Or am I mistaken?
> 
> Circumstances change. We want to keep the stuff in user space as much and
> as long as we can. However user space has limitations:
> 
> - it has by necessity a race between resumption and access by others
> - it cannot resume anything we run a (rw) filesystem over.
> 
> Furthermore, today PM actions that lead to a loss of mode are initiated
> by user space. If we ever want to oportunistically suspend a system
> we also need to restore mode from inside the kernel.
> 
> We could avoid all that trouble, if we persuaded vendors to use plain
> USB configurations for those purposes.

But that would happen, I suspect, in an alternate universe. Better to 
anticipate the trouble, I suspect. :-{

Moreover, the spark for the current discussion was the problem of 
dual-mode cameras, which can work both as webcams and stillcams, not the 
3G modems that you mention. The problems are analogous but not identical. 

	-- dual-mode cameras are, typically, Class Proprietary devices in 
all of their functions. None of them that I know of are Mass Storage 
devices. Therefore, usb_modeswitch would have to be rewritten completely 
in order to be used for such hardware. As things stand right now, it has 
nothing at all to do with the problem. Not to say, of course, that the 
experience gained with usb_modeswitch is totally irrelevant.

	-- I don't have one of those modems, but I have the impression 
that the need to access the "disk" partition on one of them is basically a 
one-shot deal. Namely, one needs to load some firmware or so on the "disk" 
before the gadget can be used. The problem with a dual-mode camera is that 
the user ought to be able to switch at will between a 
download-the-pictures stillcam application and a stream application. 
While, of course, not being able to start one of these activities while 
the other is going on, because that would cause obvious trouble. 

The similarity of the modem-and-mass-storage device and the web-and-still 
camera is, of course, that both partake of being devices that will do more 
than one kind of thing and they need to be supported in that respect. 
Other hardware exists with similar characteristics, but sometimes the 
functionaliy is not dual but even triple, and one can reasonably suspect 
that more of this kind of thing is going to come at us in the future. I 
think it is a good occasion to sit back and think things over a bit.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
> Hi,
> 
> On 08/05/2011 12:56 AM, Greg KH wrote:
> >On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> I think it is important to separate oranges from apples here, there are
> at least 3 different problem classes which all seem to have gotten thrown
> onto a pile here:
> 
> 1) The reason Mauro suggested having some discussion on this at the
> USB summit is because of a discussion about dual mode cameras on the
> linux media list.
...
> 3) Re-direction of usb devices to virtual machines. This works by using
> the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> The basics of this work fine, but it lacks proper locking / safeguards
> for when a vm takes over a usb device from the in kernel driver.

Hi Hans and Mauro,

We have do room in the schedule for the USB mini-summit for this
discussion, since the schedule is still pretty flexible.  The
preliminary schedule is up here:

http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html

I think it's best to discuss the VM redirection in the afternoon when
some of the KVM folks join us after Hans' talk on USB redirection over
the network.

It sounds like we need a separate topic for the dual mode cameras and TV
tuners.  Mauro, do you want to lead that discussion in the early morning
(in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
slot)?  I want to be sure we have all the video/media developers who are
interested in this topic present, and I don't know if they will be going
to the KVM forum.

Link PM was originally slated for the 10am slot, but since we're missing
several people for that discussion (Alan Stern, Andiry Xu, and any of
the Intel folks who did the Moorestown USB 2.0 LPM), I think it will be
more useful to have that discussion on-list or at another conference.

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Mon, 8 Aug 2011, Sarah Sharp wrote:

> On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
> > Hi,
> > 
> > On 08/05/2011 12:56 AM, Greg KH wrote:
> > >On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > I think it is important to separate oranges from apples here, there are
> > at least 3 different problem classes which all seem to have gotten thrown
> > onto a pile here:
> > 
> > 1) The reason Mauro suggested having some discussion on this at the
> > USB summit is because of a discussion about dual mode cameras on the
> > linux media list.
> ...
> > 3) Re-direction of usb devices to virtual machines. This works by using
> > the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> > The basics of this work fine, but it lacks proper locking / safeguards
> > for when a vm takes over a usb device from the in kernel driver.
> 
> Hi Hans and Mauro,
> 
> We have do room in the schedule for the USB mini-summit for this
> discussion, since the schedule is still pretty flexible.  The
> preliminary schedule is up here:
> 
> http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
> 
> I think it's best to discuss the VM redirection in the afternoon when
> some of the KVM folks join us after Hans' talk on USB redirection over
> the network.
> 
> It sounds like we need a separate topic for the dual mode cameras and TV
> tuners.  Mauro, do you want to lead that discussion in the early morning
> (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> slot)?  I want to be sure we have all the video/media developers who are
> interested in this topic present, and I don't know if they will be going
> to the KVM forum.

Sarah,

Alas. I would suspect that I am one of the people most interested in the 
topic of dual-mode cameras, since I have worked on supporting them both in 
libgphoto2 and in the kernel. But I teach in a university for a living, 
and the first classes of Fall Semester 2011 start on August 17 in Auburn, 
Alabama. Knowing this, I decided, months ago, that I simply could not 
attend a conference which starts on August 16 in Vancouver.

So, after starting all of the current mailing-list discussion on the topic 
I will not be at the conference. I can only hope that those who do attend 
will keep me current about what gets discussed.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

On Mon, Aug 08, 2011 at 01:23:56PM -0500, Theodore Kilgore wrote:
> 
> 
> On Mon, 8 Aug 2011, Sarah Sharp wrote:
> 
> > On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
> > > Hi,
> > > 
> > > On 08/05/2011 12:56 AM, Greg KH wrote:
> > > >On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> > > I think it is important to separate oranges from apples here, there are
> > > at least 3 different problem classes which all seem to have gotten thrown
> > > onto a pile here:
> > > 
> > > 1) The reason Mauro suggested having some discussion on this at the
> > > USB summit is because of a discussion about dual mode cameras on the
> > > linux media list.
> > ...
> > > 3) Re-direction of usb devices to virtual machines. This works by using
> > > the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> > > The basics of this work fine, but it lacks proper locking / safeguards
> > > for when a vm takes over a usb device from the in kernel driver.
> > 
> > Hi Hans and Mauro,
> > 
> > We have do room in the schedule for the USB mini-summit for this
> > discussion, since the schedule is still pretty flexible.  The
> > preliminary schedule is up here:
> > 
> > http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
> > 
> > I think it's best to discuss the VM redirection in the afternoon when
> > some of the KVM folks join us after Hans' talk on USB redirection over
> > the network.
> > 
> > It sounds like we need a separate topic for the dual mode cameras and TV
> > tuners.  Mauro, do you want to lead that discussion in the early morning
> > (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> > slot)?  I want to be sure we have all the video/media developers who are
> > interested in this topic present, and I don't know if they will be going
> > to the KVM forum.
> 
> Sarah,
> 
> Alas. I would suspect that I am one of the people most interested in the 
> topic of dual-mode cameras, since I have worked on supporting them both in 
> libgphoto2 and in the kernel. But I teach in a university for a living, 
> and the first classes of Fall Semester 2011 start on August 17 in Auburn, 
> Alabama. Knowing this, I decided, months ago, that I simply could not 
> attend a conference which starts on August 16 in Vancouver.
> 
> So, after starting all of the current mailing-list discussion on the topic 
> I will not be at the conference. I can only hope that those who do attend 
> will keep me current about what gets discussed.

No worries, I'll be taking notes and post them to the Linux USB and
Linux media lists.

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Hi Sarah,

Em 08-08-2011 14:58, Sarah Sharp escreveu:
> On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
>> Hi,
>>
>> On 08/05/2011 12:56 AM, Greg KH wrote:
>>> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>> I think it is important to separate oranges from apples here, there are
>> at least 3 different problem classes which all seem to have gotten thrown
>> onto a pile here:
>>
>> 1) The reason Mauro suggested having some discussion on this at the
>> USB summit is because of a discussion about dual mode cameras on the
>> linux media list.
> ...
>> 3) Re-direction of usb devices to virtual machines. This works by using
>> the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
>> The basics of this work fine, but it lacks proper locking / safeguards
>> for when a vm takes over a usb device from the in kernel driver.
> 
> Hi Hans and Mauro,
> 
> We have do room in the schedule for the USB mini-summit for this
> discussion, since the schedule is still pretty flexible.  The
> preliminary schedule is up here:
> 
> http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
> 
> I think it's best to discuss the VM redirection in the afternoon when
> some of the KVM folks join us after Hans' talk on USB redirection over
> the network.
> 
> It sounds like we need a separate topic for the dual mode cameras and TV
> tuners.  

Yes, I suspect so. This subject is pending for quite some time, and it would
be good to have people think together around a solution for that.

> Mauro, do you want to lead that discussion in the early morning
> (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> slot)?  I want to be sure we have all the video/media developers who are
> interested in this topic present, and I don't know if they will be going
> to the KVM forum.

>From my side, both slots work ok. Hans, what would fit better for you?

> Link PM was originally slated for the 10am slot, but since we're missing
> several people for that discussion (Alan Stern, Andiry Xu, and any of
> the Intel folks who did the Moorestown USB 2.0 LPM), I think it will be
> more useful to have that discussion on-list or at another conference.
> 
> Sarah Sharp

Thanks!
Mauro

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/08/2011 07:58 PM, Sarah Sharp wrote:
> On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
>> Hi,
>>
>> On 08/05/2011 12:56 AM, Greg KH wrote:
>>> On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
>> I think it is important to separate oranges from apples here, there are
>> at least 3 different problem classes which all seem to have gotten thrown
>> onto a pile here:
>>
>> 1) The reason Mauro suggested having some discussion on this at the
>> USB summit is because of a discussion about dual mode cameras on the
>> linux media list.
> ...
>> 3) Re-direction of usb devices to virtual machines. This works by using
>> the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
>> The basics of this work fine, but it lacks proper locking / safeguards
>> for when a vm takes over a usb device from the in kernel driver.
>
> Hi Hans and Mauro,
>
> We have do room in the schedule for the USB mini-summit for this
> discussion, since the schedule is still pretty flexible.  The
> preliminary schedule is up here:
>
> http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
>
> I think it's best to discuss the VM redirection in the afternoon when
> some of the KVM folks join us after Hans' talk on USB redirection over
> the network.
>

That seems best to me too. I'm available at both proposed time slots,
and I would like to join both discussions.

> It sounds like we need a separate topic for the dual mode cameras and TV
> tuners.  Mauro, do you want to lead that discussion in the early morning
> (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> slot)?  I want to be sure we have all the video/media developers who are
> interested in this topic present, and I don't know if they will be going
> to the KVM forum.

I would really like to see the dual mode camera and TV tuner discussion
separated. They are 2 different issues AFAIK.

1) Dual mode cameras:

In the case of the dual mode camera we have 1 single device (both at
the hardware level and at the logical block level), which can do 2 things,
but not at the same time. It can stream live video data from a sensor,
or it can retrieve earlier taken pictures from some picture memory.

Unfortunately even though these 2 functions live in a single logical block,
historically we've developed 2 drivers for them. This leads to fighting
over device ownership (surprise surprise), and to me the solution is
very clear, 1 logical block == 1 driver.

2) Tv tuners:

Here we have a bunch of logical blocks, each with their own driver
(and not 2 drivers for one block), which together form not 1 but 2
video pipelines, typically one pipeline for analog TV, and one
for DVB. The problem here is some blocks are shared between the
2 pipelines.

The solution here, at least to me, is clear too, we need some way
of claiming/locking the pipeline blocks. If pipe1 starts streaming
video data it locks all the blocks it uses. If some app then tries
to use pipe2 (and thus for example tune the tuner to a different
frequency), it will fail with -EBUSY when pipe2 tries to take the
tuner lock and gets told it is in use.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Tue, 9 Aug 2011, Hans de Goede wrote:

> I would really like to see the dual mode camera and TV tuner discussion
> separated. They are 2 different issues AFAIK.
> 
> 1) Dual mode cameras:
> 
> In the case of the dual mode camera we have 1 single device (both at
> the hardware level and at the logical block level), which can do 2 things,
> but not at the same time. It can stream live video data from a sensor,
> or it can retrieve earlier taken pictures from some picture memory.
> 
> Unfortunately even though these 2 functions live in a single logical block,
> historically we've developed 2 drivers for them. This leads to fighting
> over device ownership (surprise surprise), and to me the solution is
> very clear, 1 logical block == 1 driver.

According to Theodore, we have developed 5 drivers for them because the
stillcam modes in different devices use four different vendor-specific
drivers.  Does it really make sense to combine 5 drivers into one?  I 
think some sort of sharing protocol would work out better.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Marko Ristola]

Hi

I've been thinking about the Kernel driver side.
Mauro and others emailed requirements on Jun or July.

I'm sorry for this spam: maybe you have thought this already.

A linked list of read/write locks as the solution for these protections could be
a base for the general solution. Locks could be accessed either by name "string name"
or by an integer identifier.

The bridge driver would be the container of the lock list.
Lock take / release handles would be changeable by Bridge driver at driver init.
This way bridge could tune the lock taking actions. Sub devices would not have
to know the details of the bridge device.

When implemented as a library ".ko" module, this could be
used by all related kernel drivers. The locking code would be very general.
Maybe adding a lock list into each PCI bus device would solve the device export problem to KVM too.

If some module wouldn't handle the proper locking yet,
it would not deliver the protection, but it would work as before: no regressions.

The library could also be called so that a driver would ask three locks at the same time.
If the driver would get all three locks, it would return success.
If the driver would not get all three locks, it would not lock any of them (with _trylock case).

I don't have time to implement this feature.

Happy meeting for all of you,
Marko Ristola

Re: USB mini-summit at LinuxCon Vancouver

[Sarah Sharp]

On Tue, Aug 09, 2011 at 09:52:44AM +0200, Hans de Goede wrote:
> Hi,
> 
> On 08/08/2011 07:58 PM, Sarah Sharp wrote:
> >On Fri, Aug 05, 2011 at 09:45:56AM +0200, Hans de Goede wrote:
> >>Hi,
> >>
> >>On 08/05/2011 12:56 AM, Greg KH wrote:
> >>>On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> >>I think it is important to separate oranges from apples here, there are
> >>at least 3 different problem classes which all seem to have gotten thrown
> >>onto a pile here:
> >>
> >>1) The reason Mauro suggested having some discussion on this at the
> >>USB summit is because of a discussion about dual mode cameras on the
> >>linux media list.
> >...
> >>3) Re-direction of usb devices to virtual machines. This works by using
> >>the userspace usbfs interface from qemu / vmware / virtualbox / whatever.
> >>The basics of this work fine, but it lacks proper locking / safeguards
> >>for when a vm takes over a usb device from the in kernel driver.
> >
> >Hi Hans and Mauro,
> >
> >We have do room in the schedule for the USB mini-summit for this
> >discussion, since the schedule is still pretty flexible.  The
> >preliminary schedule is up here:
> >
> >http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
> >
> >I think it's best to discuss the VM redirection in the afternoon when
> >some of the KVM folks join us after Hans' talk on USB redirection over
> >the network.
> >
> 
> That seems best to me too. I'm available at both proposed time slots,
> and I would like to join both discussions.
> 
> >It sounds like we need a separate topic for the dual mode cameras and TV
> >tuners.  Mauro, do you want to lead that discussion in the early morning
> >(in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30
> >slot)?  I want to be sure we have all the video/media developers who are
> >interested in this topic present, and I don't know if they will be going
> >to the KVM forum.
> 
> I would really like to see the dual mode camera and TV tuner discussion
> separated. They are 2 different issues AFAIK.

Ok, great, I've put both topics in the morning, in separate slots.

Sarah Sharp

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/09/2011 04:19 PM, Alan Stern wrote:
> On Tue, 9 Aug 2011, Hans de Goede wrote:
>
>> I would really like to see the dual mode camera and TV tuner discussion
>> separated. They are 2 different issues AFAIK.
>>
>> 1) Dual mode cameras:
>>
>> In the case of the dual mode camera we have 1 single device (both at
>> the hardware level and at the logical block level), which can do 2 things,
>> but not at the same time. It can stream live video data from a sensor,
>> or it can retrieve earlier taken pictures from some picture memory.
>>
>> Unfortunately even though these 2 functions live in a single logical block,
>> historically we've developed 2 drivers for them. This leads to fighting
>> over device ownership (surprise surprise), and to me the solution is
>> very clear, 1 logical block == 1 driver.
>
> According to Theodore, we have developed 5 drivers for them because the
> stillcam modes in different devices use four different vendor-specific
> drivers.

Yes, but so the the webcam modes of the different devices, so for
the 5 (not sure if that is the right number) dual-cam mode chipsets
we support there will be 5 drivers, each supporting both the
webcam and the access to pictures stored in memory of the chipset
they support. So 5 chipsets -> 5 drivers each supporting 1 chipset,
and both functions of the single logical device that chipset
represents.

>  Does it really make sense to combine 5 drivers into one?

Right, that is not the plan. The plan is to simply stop having 2 drivers
for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
+ 5 webcam, to just 5 drivers. We will also likely be able to share
code between the code for the 2 functionalities for things like generic
set / get register functions, initialization, etc.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Adam Baker]

On Tuesday 09 August 2011, Hans de Goede wrote:
> Hi,
> 
> On 08/09/2011 04:19 PM, Alan Stern wrote:

> >  Does it really make sense to combine 5 drivers into one?
> 
> Right, that is not the plan. The plan is to simply stop having 2 drivers
> for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
> + 5 webcam, to just 5 drivers. We will also likely be able to share
> code between the code for the 2 functionalities for things like generic
> set / get register functions, initialization, etc.
> 

Unfortunately as Theodore recently pointed out you don't go from 10 to 5, you 
go from 10 to 10 where 5 of the new 10 are only used on Win32, FreeBSD and OSX 
(but they aren't any simpler because they still rely on libusb) and the other 
5 that are only used on Linux become significantly more complicated than they 
currently are.

It has also just occured to me that it might be possible to solve the issues 
we are facing just in the kernel. At the moment when the kernel performs a 
USBDEVFS_DISCONNECT it keeps the kernel driver locked out until userspace 
performs a USBDEVFS_CONNECT. If the kernel reattached the kernel driver when 
the device file was closed then, as gvfs doesn't keep the file open the 
biggest current issue would be solved instantly. If a mechanism could be found 
to prevent USBDEVFS_DISCONNECT from succeeding when the corresponding 
/dev/videox file was open then that would seem to be a reasonable solution.

Hans had expressed the opinion that merely having the device open to control 
the camera not to stream shouldn't  prevent stillcam operation - I disagree 
because if you are setting up the controls you are probably already streaming 
so you can see what you are doing and if not you are probably about to.

Of course changing the behaviour of USBDEVFS_DISCONNECT is not something to be 
done lightly. I don't know how many other users there are for it and if the 
current behaviour is actually correct for any of them. Cleaning up on file 
close does have the useful side effect though that applications no longer need 
to worry about the fact that even if they clean up properly on a normal exit, 
if they crash they leave the kernel driver permanently disabled

Regards

Adam

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/09/2011 10:31 PM, Adam Baker wrote:
> On Tuesday 09 August 2011, Hans de Goede wrote:

<snip>

> It has also just occured to me that it might be possible to solve the issues
> we are facing just in the kernel. At the moment when the kernel performs a
> USBDEVFS_DISCONNECT it keeps the kernel driver locked out until userspace
> performs a USBDEVFS_CONNECT. If the kernel reattached the kernel driver when
> the device file was closed then, as gvfs doesn't keep the file open the
> biggest current issue would be solved instantly. If a mechanism could be found
> to prevent USBDEVFS_DISCONNECT from succeeding when the corresponding
> /dev/videox file was open then that would seem to be a reasonable solution.

<sigh>

This has been discussed over and over and over again, playing clever
tricks with USBDEVFS_[DIS]CONNECT like adding a new USBDEVFS_TRYDISCONNECT
which the v4l2 driver could intercept won't cut it. We need some central
manager of the device doing multiplexing between the 2 functions, and you
can *not* assume that either side will be nice wrt closing file descriptors.

Examples:
1) You are wrong wrt gvfs, it does keep the libgphoto2 context open all the
time, and through that the usbfs device nodes.

2) Lets say a user starts a photo managing app like f-spot, and that opens
the device through libgphoto2 on startup, then the user switches to another
virtual desktop and forgets all about having f-spot open. Notice that if
the user now tries to stream he will not get a busy error, but the app trying
to do the streaming will simply not see the camera at all (kernel driver
unbound /dev/video# node is gone).

3) Notice that little speaker icon in your panel on your average Linux
desktop, that keeps the mixer of the audio device open *all the time* it
is quite easy to imagine a similar applet for v4l2 device controls (see
for example gtk-v4l) doing the same. Or a user could simply start up a
v4l2 control panel app like gtk-v4l, qv4l2 or v4l2ucp, and leave it running
minimized ...

4) Some laptops have a Fn + F## key which enables / disables the builtin
webcam by turning its power on / off. Effectively plugging it into / out
of a usb port. We would like to have an on screen notification of this one
day like we have now for brightness and volume controls, based on udev
events. But the current dual mode cam stuff causes udev events for
a *new* video device being added / an existing one being removed
each time libgphoto2 releases / takes control of the camera.

5) More in general, more and more software is dynamically monitoring the
addition / removal of (usb) devices using udev, our current solution
suggests to this software the /dev/video device is being unplugged /
re-plugged all the time, not pretty.


All in all what we've today is a kludge, and if we want to provide
a "seamless" user experience we need to fix it.

Don't get me wrong usbfs is a really nice solution for driving
usb devices from userspace, like scanners and all other sorts of
devices. But what all these devices have in common is that they
have no kernel driver. Having a userspace based driver and a kernel
driver "fight it out" just does not work well.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Tue, 9 Aug 2011, Hans de Goede wrote:

> Hi,
> 
> On 08/09/2011 04:19 PM, Alan Stern wrote:
> > On Tue, 9 Aug 2011, Hans de Goede wrote:
> > 
> > > I would really like to see the dual mode camera and TV tuner discussion
> > > separated. They are 2 different issues AFAIK.
> > > 
> > > 1) Dual mode cameras:
> > > 
> > > In the case of the dual mode camera we have 1 single device (both at
> > > the hardware level and at the logical block level), which can do 2 things,
> > > but not at the same time. It can stream live video data from a sensor,
> > > or it can retrieve earlier taken pictures from some picture memory.
> > > 
> > > Unfortunately even though these 2 functions live in a single logical
> > > block,
> > > historically we've developed 2 drivers for them. This leads to fighting
> > > over device ownership (surprise surprise), and to me the solution is
> > > very clear, 1 logical block == 1 driver.
> > 
> > According to Theodore, we have developed 5 drivers for them because the
> > stillcam modes in different devices use four different vendor-specific
> > drivers.
> 
> Yes, but so the the webcam modes of the different devices, so for
> the 5 (not sure if that is the right number) dual-cam mode chipsets
> we support there will be 5 drivers, each supporting both the
> webcam and the access to pictures stored in memory of the chipset
> they support. So 5 chipsets -> 5 drivers each supporting 1 chipset,
> and both functions of the single logical device that chipset
> represents.
> 
> >  Does it really make sense to combine 5 drivers into one?
> 
> Right, that is not the plan. The plan is to simply stop having 2 drivers
> for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
> + 5 webcam, to just 5 drivers. We will also likely be able to share
> code between the code for the 2 functionalities for things like generic
> set / get register functions, initialization, etc.

Probably this is a good time for some inventory to be done.

 sure of the exact number of drivers. The jeilinj driver does not count 
here. It is a standard mass storage device as a still camera, and if it is 
going to be hooked up as a webcam then it has to have some buttons pushed 
just so, prior to hookup -- after which it comes up with a different USB 
Product number as a proprietary webcam device, supported by the jeilinj 
driver. Thus, this is not one of the droids we are looking for.

Specifically, the currently affected drivers are sq905, sq905c, mr97310a, 
and sn9c2028. 

Fuji Finepix seems to produce both still cameras (PTP protocol) and 
webcams. If there are currently any dual-mode cameras, it is not obvious. 
Perhaps any of these which can be dual mode are like the jeilinj cameras 
and come up under different Product numbers in the different modes, which 
is not a problem. 

There do seem to exist stv06xx dual-mode cameras:

stv0680/stv0680.c:      { "STM:USB Dual-mode camera",   0x0553, 0x0202, 1 
},

is found in camlibs/stv0680, for example, along with a whole bunch of 
other cameras with the same Vendor:Product number. It seems, too, that the 
same Vendor:Product number is listed in gspca/stv0680.c

So I guess we found another affected driver.


The intersection seems to be void between the lists of the spca50x cameras 
and webcams, though one can not be certain about the future. I do know of 
another dualmode camera, powered by the JL2005A chip. I wrote a kernel 
module for it some months ago. Said module has not been sent upward 
because of mysterious issues with one of the three cameras used for 
testing it.

We also find among the gspca driver files this

* GSPCA sub driver for W996[78]CF JPEG USB Dual Mode Camera Chip.
 *
 * Copyright (C) 2009 Hans de Goede <hdegoede@redhat.com>

but I am not sure about which cameras are using this.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Xiaofan Chen]

On Wed, Aug 10, 2011 at 4:57 AM, Hans de Goede <hdegoede@redhat.com> wrote:
> This has been discussed over and over and over again, playing clever
> tricks with USBDEVFS_[DIS]CONNECT like adding a new
> USBDEVFS_TRYDISCONNECT
> which the v4l2 driver could intercept won't cut it. We need some central
> manager of the device doing multiplexing between the 2 functions, and you
> can *not* assume that either side will be nice wrt closing file descriptors.

So it seems to be both good and bad for Linux's ability to do
multiplexing between the two drivers (one purpose-built kernel driver
and one generic usbfs driver to be used libusb). The good thing
is that it is quite useful in many use cases. The ability to detach
the existing kernel driver and use usbfs is the base for many libusb
based program under Linux and it beats Windows and Mac OS X
handsomely in this aspect. On the other hand, based on the
discussions, it seems to adds quite some issues/complexities
in some other use cases.

Windows typically does not have such an issue. You can use
only one driver. In order to use libusb-win32 or libusb-1.0 Windows
backend, you have to replace the existing kernel driver with
libusb-win32 kernel driver (libusb0.sys) or libusb-1.0 Windows
backend driver (currently winusb.sys).

The exception is the libusb0.sys filter driver. In that case, you
use libusb0.sys as the upper filter driver on top of the existing
kernel driver. On the other hand, libusb-win32 filter driver is not
really widely used due to issues with older versions.

> All in all what we've today is a kludge, and if we want to provide
> a "seamless" user experience we need to fix it.
>
> Don't get me wrong usbfs is a really nice solution for driving
> usb devices from userspace, like scanners and all other sorts of
> devices. But what all these devices have in common is that they
> have no kernel driver. Having a userspace based driver and a kernel
> driver "fight it out" just does not work well.
>

All in all, I think this is a good summary. There are cases where
a kernel driver and a usbfs are both used, an example is FTDI
device where ftdi-sio driver and the user space libftdi (based
on libusb) are used, but there are problems with that as well
with regard to switching between the two drivers.

There are a few discussions here.
http://libusb.6.n5.nabble.com/Patch-libusb-os-linux-usbfs-c-Distingush-between-usbfs-and-other-kernel-mode-drivers-td3199947.html
http://libusb.6.n5.nabble.com/open-device-exclusively-td4524397.html
And there are no good solutions to a "simple" issue as the
above mentioned.

Not so sure if this is a good suggestion or not: just wondering how
the other side (eg Windows) deal with these dual mode cameras?
Does Windows use single driver or does Windows use two drivers
for these dual mode cameras? Since they are customized device,
then there must be a vendor driver for them. And since they will
function as a video cam and a still cam device, I assume there
will be standard Windows drivers on top of the vendor drivers. So
there will be more than one drivers associated with them.


-- 
Xiaofan

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Tue, 9 Aug 2011, Hans de Goede wrote:

> Hi,
> 
> On 08/09/2011 04:19 PM, Alan Stern wrote:
> > On Tue, 9 Aug 2011, Hans de Goede wrote:
> > According to Theodore, we have developed 5 drivers for them because the
> > stillcam modes in different devices use four different vendor-specific
> > drivers.
> 
> Yes, but so the the webcam modes of the different devices, so for
> the 5 (not sure if that is the right number) dual-cam mode chipsets
> we support there will be 5 drivers, each supporting both the
> webcam and the access to pictures stored in memory of the chipset
> they support. So 5 chipsets -> 5 drivers each supporting 1 chipset,
> and both functions of the single logical device that chipset
> represents.
> 
> >  Does it really make sense to combine 5 drivers into one?
> 
> Right, that is not the plan. The plan is to simply stop having 2 drivers
> for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
> + 5 webcam, to just 5 drivers. We will also likely be able to share
> code between the code for the 2 functionalities for things like generic
> set / get register functions, initialization, etc.

Okay, I didn't realize that the different cameras used different webcam 
drivers as well as different stillcam drivers.

As far as I can see, there's nothing to stop anybody from adding the 
stillcam functionality into the webcam drivers right now.  If some 
common code can be abstracted out into a shared source file, so much 
the better.

That would solve the problem, right?

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Wed, 10 Aug 2011, Alan Stern wrote:

> On Tue, 9 Aug 2011, Hans de Goede wrote:
> 
> > Hi,
> > 
> > On 08/09/2011 04:19 PM, Alan Stern wrote:
> > > On Tue, 9 Aug 2011, Hans de Goede wrote:
> > > According to Theodore, we have developed 5 drivers for them because the
> > > stillcam modes in different devices use four different vendor-specific
> > > drivers.
> > 
> > Yes, but so the the webcam modes of the different devices, so for
> > the 5 (not sure if that is the right number) dual-cam mode chipsets
> > we support there will be 5 drivers, each supporting both the
> > webcam and the access to pictures stored in memory of the chipset
> > they support. So 5 chipsets -> 5 drivers each supporting 1 chipset,
> > and both functions of the single logical device that chipset
> > represents.
> > 
> > >  Does it really make sense to combine 5 drivers into one?
> > 
> > Right, that is not the plan. The plan is to simply stop having 2 drivers
> > for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
> > + 5 webcam, to just 5 drivers. We will also likely be able to share
> > code between the code for the 2 functionalities for things like generic
> > set / get register functions, initialization, etc.
> 
> Okay, I didn't realize that the different cameras used different webcam 
> drivers as well as different stillcam drivers.

Oh, yes. They are Proprietary devices. And that means what it says. :-)
And all different from each other, too.
 
> As far as I can see, there's nothing to stop anybody from adding the 
> stillcam functionality into the webcam drivers right now.  If some 
> common code can be abstracted out into a shared source file, so much 
> the better.
> 
> That would solve the problem, right?

I think everyone involved believes that it would solve the problem. 

The question has been all along whether or not there is any other way 
which would work. Also the question of what, exactly, "belongs" in the 
kernel and what does not. For, if something has been historically 
supported in userspace (stillcam support, in this case) and has worked 
well there, I would think it is kind of too bad to have to move said 
support into the kernel just because the same hardware requires kernel 
support for another functionality and the two sides clash. I mean, the 
kernel is already big enough, no? But the logic that Hans has set forth 
seems rather compelling. 

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Wed, 10 Aug 2011, Theodore Kilgore wrote:

> > Okay, I didn't realize that the different cameras used different webcam 
> > drivers as well as different stillcam drivers.
> 
> Oh, yes. They are Proprietary devices. And that means what it says. :-)
> And all different from each other, too.
>  
> > As far as I can see, there's nothing to stop anybody from adding the 
> > stillcam functionality into the webcam drivers right now.  If some 
> > common code can be abstracted out into a shared source file, so much 
> > the better.
> > 
> > That would solve the problem, right?
> 
> I think everyone involved believes that it would solve the problem. 
> 
> The question has been all along whether or not there is any other way 
> which would work. Also the question of what, exactly, "belongs" in the 
> kernel and what does not. For, if something has been historically 
> supported in userspace (stillcam support, in this case) and has worked 
> well there, I would think it is kind of too bad to have to move said 
> support into the kernel just because the same hardware requires kernel 
> support for another functionality and the two sides clash. I mean, the 
> kernel is already big enough, no? But the logic that Hans has set forth 
> seems rather compelling. 

The alternative seems to be to define a device-sharing protocol for USB
drivers.  Kernel drivers would implement a new callback (asking them to
give up control of the device), and usbfs would implement new ioctls by
which a program could ask for and relinquish control of a device.  The
amount of rewriting needed would be relatively small.

A few loose ends would remain, such as how to handle suspends, resumes,
resets, and disconnects.  Assuming usbfs is the only driver that will
want to share a device in this way, we could handle them.

Hans, what do you think?

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Wed, 10 Aug 2011, Alan Stern wrote:

> On Wed, 10 Aug 2011, Theodore Kilgore wrote:
> 
> > > Okay, I didn't realize that the different cameras used different webcam 
> > > drivers as well as different stillcam drivers.
> > 
> > Oh, yes. They are Proprietary devices. And that means what it says. :-)
> > And all different from each other, too.
> >  
> > > As far as I can see, there's nothing to stop anybody from adding the 
> > > stillcam functionality into the webcam drivers right now.  If some 
> > > common code can be abstracted out into a shared source file, so much 
> > > the better.
> > > 
> > > That would solve the problem, right?
> > 
> > I think everyone involved believes that it would solve the problem. 
> > 
> > The question has been all along whether or not there is any other way 
> > which would work. Also the question of what, exactly, "belongs" in the 
> > kernel and what does not. For, if something has been historically 
> > supported in userspace (stillcam support, in this case) and has worked 
> > well there, I would think it is kind of too bad to have to move said 
> > support into the kernel just because the same hardware requires kernel 
> > support for another functionality and the two sides clash. I mean, the 
> > kernel is already big enough, no? But the logic that Hans has set forth 
> > seems rather compelling. 
> 
> The alternative seems to be to define a device-sharing protocol for USB
> drivers.  Kernel drivers would implement a new callback (asking them to
> give up control of the device), and usbfs would implement new ioctls by
> which a program could ask for and relinquish control of a device.  The
> amount of rewriting needed would be relatively small.
> 
> A few loose ends would remain, such as how to handle suspends, resumes,
> resets, and disconnects.  Assuming usbfs is the only driver that will
> want to share a device in this way, we could handle them.
> 
> Hans, what do you think?
> 
> Alan Stern

Alan,

Hans seems to have argued cogently for doing all of this in the kernel and 
for abandoning the usbfs-based drivers for these particular drivers for 
dual-mode cameras and, I would conjecture, for drivers for dual-mode 
hardware in general. Therefore, I anticipate that he won't like that very 
much.

My position:

I do not have preconceptions about how the problem gets handled, and at 
this point I remain agnostic and believe that all approaches ought to be 
carefully analysed. I can imagine, abstractly, that things like this 
could be handled by

-- moving all basic functionality to the kernel, and fixing the 
relevant libgphoto2 drivers to look to the kernel instead of to libusb. 
(What Hans argues for, and I am not opposed if his arguments convince 
other concerned parties)

-- doing some kind of patch job to make current arrangement somehow to 
work better (this seems to be the position of Adam Baker; I do share
the skepticism Hans has expressed about how well this could all be 
pasted together)

-- doing something like the previous, but also figuring out how to bring 
udev rules into play, which would make it all work better (just tossing 
this one in, for laughs, but who knows someone might like it)

-- moving the kernel webcam drivers out of the kernel and doing with these 
cameras _everything_ including webcam function through libusb. I myself do 
not have the imagination to be able to figure out how this could be done 
without a rather humongous amount of work (for example, which streaming 
apps that are currently available would be able to live with this?) but 
unless I misunderstand what he was saying, Greg K-H seems to think that 
this would be the best thing to do.

Which one of these possibile approaches gets adopted is a policy issue 
which I would consider is ultimately way above my pay grade.

My main motivation for bringing up the issue was to get it to the front 
burner so that _something_ gets done. It is a matter which has been left 
alone for too long. Therefore, I am very glad that the matter is being 
addressed.

Let me add to this that I have gotten permission for time off and for a 
expense money which might possibly cover my air fare. I hope to arrive in 
Vancouver by sometime on Monday and intend to attend the mini-summit. I 
suggest that we get all intersted parties together and figure out what is 
the best way to go.

I hope everyone who is actively concerned can meet in Vancouver, and if 
all goes well then on Monday as well as Tuesday. I can hang around for 
another day or two after Tuesday, but I do not expect to register for 
LinuxCon or be involved in it. When I leave Vancouver I will probably go 
to Seattle and spend a couple of days with my oldest son, the musician, 
before coming home on the next weekend.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Hans Verkuil]

On Wednesday, August 10, 2011 20:33:25 Theodore Kilgore wrote:
> 
> On Wed, 10 Aug 2011, Alan Stern wrote:
> 
> > On Wed, 10 Aug 2011, Theodore Kilgore wrote:
> > 
> > > > Okay, I didn't realize that the different cameras used different webcam 
> > > > drivers as well as different stillcam drivers.
> > > 
> > > Oh, yes. They are Proprietary devices. And that means what it says. :-)
> > > And all different from each other, too.
> > >  
> > > > As far as I can see, there's nothing to stop anybody from adding the 
> > > > stillcam functionality into the webcam drivers right now.  If some 
> > > > common code can be abstracted out into a shared source file, so much 
> > > > the better.
> > > > 
> > > > That would solve the problem, right?
> > > 
> > > I think everyone involved believes that it would solve the problem. 
> > > 
> > > The question has been all along whether or not there is any other way 
> > > which would work. Also the question of what, exactly, "belongs" in the 
> > > kernel and what does not. For, if something has been historically 
> > > supported in userspace (stillcam support, in this case) and has worked 
> > > well there, I would think it is kind of too bad to have to move said 
> > > support into the kernel just because the same hardware requires kernel 
> > > support for another functionality and the two sides clash. I mean, the 
> > > kernel is already big enough, no? But the logic that Hans has set forth 
> > > seems rather compelling. 
> > 
> > The alternative seems to be to define a device-sharing protocol for USB
> > drivers.  Kernel drivers would implement a new callback (asking them to
> > give up control of the device), and usbfs would implement new ioctls by
> > which a program could ask for and relinquish control of a device.  The
> > amount of rewriting needed would be relatively small.
> > 
> > A few loose ends would remain, such as how to handle suspends, resumes,
> > resets, and disconnects.  Assuming usbfs is the only driver that will
> > want to share a device in this way, we could handle them.
> > 
> > Hans, what do you think?
> > 
> > Alan Stern
> 
> Alan,
> 
> Hans seems to have argued cogently for doing all of this in the kernel and 
> for abandoning the usbfs-based drivers for these particular drivers for 
> dual-mode cameras and, I would conjecture, for drivers for dual-mode 
> hardware in general. Therefore, I anticipate that he won't like that very 
> much.
> 
> My position:
> 
> I do not have preconceptions about how the problem gets handled, and at 
> this point I remain agnostic and believe that all approaches ought to be 
> carefully analysed. I can imagine, abstractly, that things like this 
> could be handled by
> 
> -- moving all basic functionality to the kernel, and fixing the 
> relevant libgphoto2 drivers to look to the kernel instead of to libusb. 
> (What Hans argues for, and I am not opposed if his arguments convince 
> other concerned parties)
> 
> -- doing some kind of patch job to make current arrangement somehow to 
> work better (this seems to be the position of Adam Baker; I do share
> the skepticism Hans has expressed about how well this could all be 
> pasted together)
> 
> -- doing something like the previous, but also figuring out how to bring 
> udev rules into play, which would make it all work better (just tossing 
> this one in, for laughs, but who knows someone might like it)
> 
> -- moving the kernel webcam drivers out of the kernel and doing with these 
> cameras _everything_ including webcam function through libusb. I myself do 
> not have the imagination to be able to figure out how this could be done 
> without a rather humongous amount of work (for example, which streaming 
> apps that are currently available would be able to live with this?) but 
> unless I misunderstand what he was saying, Greg K-H seems to think that 
> this would be the best thing to do.

Webcam drivers can rely on i2c sensor drivers to handle the sensor. These
sensor drivers can be used in other non-USB devices as well (particularly
embedded SoC devices). Moving webcam drivers out of the kernel would likely
mean that you end up with duplicate kernel and userspace sensor drivers.
Not a good idea.

In all fairness I have to say that most webcam drivers program the
sensor directly instead of relying on an i2c driver. Often because they
were reverse engineered. But I know of several that should really be split
into a webcam driver and a separate i2c sensor driver.

Video hardware tends to rely heavily on i2c devices hooked up to them. The
big advantage of keeping everything in the kernel is that that makes i2c
driver sharing between USB, PCI, SoC, etc. much easier. Video hardware
also often has to toggle between different modes or functionalities (e.g.
analog TV vs DVB, MPEG encoded video vs raw). Having all components in the
kernel makes this a much easier job.

I think the approach favored by Hans de Goede is the right one based on what
I've read.

Whatever solution is chosen, in no way should that ever force us to duplicate
i2c sensor control in userspace. This would completely defeat all our efforts
to make sensor drivers reusable between webcams and SoCs.

Regards,

	Hans

> 
> Which one of these possibile approaches gets adopted is a policy issue 
> which I would consider is ultimately way above my pay grade.
> 
> My main motivation for bringing up the issue was to get it to the front 
> burner so that _something_ gets done. It is a matter which has been left 
> alone for too long. Therefore, I am very glad that the matter is being 
> addressed.
> 
> Let me add to this that I have gotten permission for time off and for a 
> expense money which might possibly cover my air fare. I hope to arrive in 
> Vancouver by sometime on Monday and intend to attend the mini-summit. I 
> suggest that we get all intersted parties together and figure out what is 
> the best way to go.
> 
> I hope everyone who is actively concerned can meet in Vancouver, and if 
> all goes well then on Monday as well as Tuesday. I can hang around for 
> another day or two after Tuesday, but I do not expect to register for 
> LinuxCon or be involved in it. When I leave Vancouver I will probably go 
> to Seattle and spend a couple of days with my oldest son, the musician, 
> before coming home on the next weekend.
> 
> Theodore Kilgore
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: USB mini-summit at LinuxCon Vancouver

[Greg KH]

On Wed, Aug 10, 2011 at 01:33:25PM -0500, Theodore Kilgore wrote:
> -- moving the kernel webcam drivers out of the kernel and doing with these 
> cameras _everything_ including webcam function through libusb. I myself do 
> not have the imagination to be able to figure out how this could be done 
> without a rather humongous amount of work (for example, which streaming 
> apps that are currently available would be able to live with this?) but 
> unless I misunderstand what he was saying, Greg K-H seems to think that 
> this would be the best thing to do.

No, I never said that.  Or if I accidentally did, I do not think that is
the best thing to do at all, sorry for any confusion.

> Which one of these possibile approaches gets adopted is a policy issue 
> which I would consider is ultimately way above my pay grade.

As no one is being paid here, there are no "pay grades", so don't worry
about that :)

greg k-h

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 10-08-2011 15:33, Theodore Kilgore escreveu:

> Hans seems to have argued cogently for doing all of this in the kernel and 
> for abandoning the usbfs-based drivers for these particular drivers for 
> dual-mode cameras and, I would conjecture, for drivers for dual-mode 
> hardware in general. Therefore, I anticipate that he won't like that very 
> much.
> 
> My position:
> 
> I do not have preconceptions about how the problem gets handled, and at 
> this point I remain agnostic and believe that all approaches ought to be 
> carefully analysed. I can imagine, abstractly, that things like this 
> could be handled by
> 
> -- moving all basic functionality to the kernel, and fixing the 
> relevant libgphoto2 drivers to look to the kernel instead of to libusb. 
> (What Hans argues for, and I am not opposed if his arguments convince 
> other concerned parties)

Not looking on the amount of work to be done, I think that this would
give better results, IMO.

> -- doing some kind of patch job to make current arrangement somehow to 
> work better (this seems to be the position of Adam Baker; I do share
> the skepticism Hans has expressed about how well this could all be 
> pasted together)

Adam Baker's proposal of a locking between usbfs and the kernel driver seems
to be interesting, but, as he pointed, there are some side effects to consider,
like suspend/resume, PM, etc.

> -- doing something like the previous, but also figuring out how to bring 
> udev rules into play, which would make it all work better (just tossing 
> this one in, for laughs, but who knows someone might like it)

I don't think this is a good alternative.

> -- moving the kernel webcam drivers out of the kernel and doing with these 
> cameras _everything_ including webcam function through libusb. I myself do 
> not have the imagination to be able to figure out how this could be done 
> without a rather humongous amount of work (for example, which streaming 
> apps that are currently available would be able to live with this?) but 
> unless I misunderstand what he was saying, Greg K-H seems to think that 
> this would be the best thing to do.

I also don't think that this a good alternative. As Hans V. pointed, one of
our long term targets is to create per-sensor I2C drivers that are independent
from the bridges. Also, moving it to userspace would require lots of work
with the duplication of V4L and gspca core into userspace for the devices
that would be moved, and may have some performance impacts.

> Which one of these possibile approaches gets adopted is a policy issue 
> which I would consider is ultimately way above my pay grade.
> 
> My main motivation for bringing up the issue was to get it to the front 
> burner so that _something_ gets done. It is a matter which has been left 
> alone for too long. Therefore, I am very glad that the matter is being 
> addressed.
> 
> Let me add to this that I have gotten permission for time off and for a 
> expense money which might possibly cover my air fare. I hope to arrive in 
> Vancouver by sometime on Monday and intend to attend the mini-summit. I 
> suggest that we get all intersted parties together and figure out what is 
> the best way to go.
> 
> I hope everyone who is actively concerned can meet in Vancouver, and if 
> all goes well then on Monday as well as Tuesday. I can hang around for 
> another day or two after Tuesday, but I do not expect to register for 
> LinuxCon or be involved in it.

It will be great to have you there for those discussions.

> When I leave Vancouver I will probably go 
> to Seattle and spend a couple of days with my oldest son, the musician, 
> before coming home on the next weekend.
> 
> Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Wed, 10 Aug 2011, Greg KH wrote:

> On Wed, Aug 10, 2011 at 01:33:25PM -0500, Theodore Kilgore wrote:
> > -- moving the kernel webcam drivers out of the kernel and doing with these 
> > cameras _everything_ including webcam function through libusb. I myself do 
> > not have the imagination to be able to figure out how this could be done 
> > without a rather humongous amount of work (for example, which streaming 
> > apps that are currently available would be able to live with this?) but 
> > unless I misunderstand what he was saying, Greg K-H seems to think that 
> > this would be the best thing to do.
> 
> No, I never said that.  Or if I accidentally did, I do not think that is
> the best thing to do at all, sorry for any confusion.

OK.
 
> > Which one of these possibile approaches gets adopted is a policy issue 
> > which I would consider is ultimately way above my pay grade.
> 
> As no one is being paid here, there are no "pay grades", so don't worry
> about that :)

I meant this, of course, in a metaphorical sense.

Cheers,
Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Wed, 10 Aug 2011, Mauro Carvalho Chehab wrote:

> Em 10-08-2011 15:33, Theodore Kilgore escreveu:
> 
> > Hans seems to have argued cogently for doing all of this in the kernel and 
> > for abandoning the usbfs-based drivers for these particular drivers for 
> > dual-mode cameras and, I would conjecture, for drivers for dual-mode 
> > hardware in general. Therefore, I anticipate that he won't like that very 
> > much.
> > 
> > My position:
> > 
> > I do not have preconceptions about how the problem gets handled, and at 
> > this point I remain agnostic and believe that all approaches ought to be 
> > carefully analysed. I can imagine, abstractly, that things like this 
> > could be handled by
> > 
> > -- moving all basic functionality to the kernel, and fixing the 
> > relevant libgphoto2 drivers to look to the kernel instead of to libusb. 
> > (What Hans argues for, and I am not opposed if his arguments convince 
> > other concerned parties)
> 
> Not looking on the amount of work to be done, I think that this would
> give better results, IMO.

Okay. I would guess that I am one of the guys who gets to do the work, 
though.

> 
> > -- doing some kind of patch job to make current arrangement somehow to 
> > work better (this seems to be the position of Adam Baker; I do share
> > the skepticism Hans has expressed about how well this could all be 
> > pasted together)
> 
> Adam Baker's proposal of a locking between usbfs and the kernel driver seems
> to be interesting, but, as he pointed, there are some side effects to consider,
> like suspend/resume, PM, etc.
> 
> > -- doing something like the previous, but also figuring out how to bring 
> > udev rules into play, which would make it all work better (just tossing 
> > this one in, for laughs, but who knows someone might like it)
> 
> I don't think this is a good alternative.

I was trying to mention all alternatives. I should have also mentioned 
"leave things the way they are" but that is certainly out the window.

> 
> > -- moving the kernel webcam drivers out of the kernel and doing with these 
> > cameras _everything_ including webcam function through libusb. I myself do 
> > not have the imagination to be able to figure out how this could be done 
> > without a rather humongous amount of work (for example, which streaming 
> > apps that are currently available would be able to live with this?) but 
> > unless I misunderstand what he was saying, Greg K-H seems to think that 
> > this would be the best thing to do.
> 
> I also don't think that this a good alternative. As Hans V. pointed, one of
> our long term targets is to create per-sensor I2C drivers that are independent
> from the bridges. Also, moving it to userspace would require lots of work
> with the duplication of V4L and gspca core into userspace for the devices
> that would be moved, and may have some performance impacts.

A good argument, though it probably does not affect the devices on my list 
one way or the other. 

> 
> > Which one of these possibile approaches gets adopted is a policy issue 
> > which I would consider is ultimately way above my pay grade.
> > 
> > My main motivation for bringing up the issue was to get it to the front 
> > burner so that _something_ gets done. It is a matter which has been left 
> > alone for too long. Therefore, I am very glad that the matter is being 
> > addressed.
> > 
> > Let me add to this that I have gotten permission for time off and for a 
> > expense money which might possibly cover my air fare. I hope to arrive in 
> > Vancouver by sometime on Monday and intend to attend the mini-summit. I 
> > suggest that we get all intersted parties together and figure out what is 
> > the best way to go.
> > 
> > I hope everyone who is actively concerned can meet in Vancouver, and if 
> > all goes well then on Monday as well as Tuesday. I can hang around for 
> > another day or two after Tuesday, but I do not expect to register for 
> > LinuxCon or be involved in it.
> 
> It will be great to have you there for those discussions.

My take on this was that it seems to have become important for me to 
attend, which, frankly, I was not expecting. So thanks for the nice words.

> 
> > When I leave Vancouver I will probably go 
> > to Seattle and spend a couple of days with my oldest son, the musician, 
> > before coming home on the next weekend.
> > 
> > Theodore Kilgore
> 

Re: USB mini-summit at LinuxCon Vancouver

[Adam Baker]

On Tuesday 09 August 2011, Hans de Goede wrote:
> Hi,
> 
> On 08/09/2011 10:31 PM, Adam Baker wrote:
> > On Tuesday 09 August 2011, Hans de Goede wrote:
> <snip>
> 
> > It has also just occured to me that it might be possible to solve the
> > issues we are facing just in the kernel. At the moment when the kernel
> > performs a USBDEVFS_DISCONNECT it keeps the kernel driver locked out
> > until userspace performs a USBDEVFS_CONNECT. If the kernel reattached
> > the kernel driver when the device file was closed then, as gvfs doesn't
> > keep the file open the biggest current issue would be solved instantly.
> > If a mechanism could be found to prevent USBDEVFS_DISCONNECT from
> > succeeding when the corresponding /dev/videox file was open then that
> > would seem to be a reasonable solution.
> 
> <sigh>
> 
> This has been discussed over and over and over again, playing clever
> tricks with USBDEVFS_[DIS]CONNECT like adding a new USBDEVFS_TRYDISCONNECT
> which the v4l2 driver could intercept won't cut it. We need some central
> manager of the device doing multiplexing between the 2 functions, and you
> can *not* assume that either side will be nice wrt closing file
> descriptors.
> 
> Examples:
> 1) You are wrong wrt gvfs, it does keep the libgphoto2 context open all the
> time, and through that the usbfs device nodes.

It seems that that depends, on my system gvfs isn't actually automounting the 
camera after it detects it and the file is only open (according to lsof) when 
the device is actually mounted. As soon as you unmount it the device gets 
closed again. Because it does do a brief open,  USBDEVFS_DISCONNECT then close 
at connection time it does still disable the kernel driver.

> 
> 2) Lets say a user starts a photo managing app like f-spot, and that opens
> the device through libgphoto2 on startup, then the user switches to another
> virtual desktop and forgets all about having f-spot open. Notice that if
> the user now tries to stream he will not get a busy error, but the app
> trying to do the streaming will simply not see the camera at all (kernel
> driver unbound /dev/video# node is gone).

This does seem like a situation where your approach could potentially give a 
better user experience. I'm wondering slightly how you define busy though. For 
webcams the streamon and streamoff ioctls tell you if you are using mmap or 
userptr transfers but you don't know if when the user has finished if they 
just use read. For stillcam mode it is again hard to determine a busy 
condition other than being in the middle of transfering an individual picture.
> 
> 3) Notice that little speaker icon in your panel on your average Linux
> desktop, that keeps the mixer of the audio device open *all the time* it
> is quite easy to imagine a similar applet for v4l2 device controls (see
> for example gtk-v4l) doing the same. Or a user could simply start up a
> v4l2 control panel app like gtk-v4l, qv4l2 or v4l2ucp, and leave it running
> minimized ...
> 

This again needs a usable concept of busy

> 4) Some laptops have a Fn + F## key which enables / disables the builtin
> webcam by turning its power on / off. Effectively plugging it into / out
> of a usb port. We would like to have an on screen notification of this one
> day like we have now for brightness and volume controls, based on udev
> events. But the current dual mode cam stuff causes udev events for
> a *new* video device being added / an existing one being removed
> each time libgphoto2 releases / takes control of the camera.
> 

Would such a system know what camera is supposed to be the internal one so it 
doesn't show the camera as turned on just because you plug in an external 
camera. If so then it won't turn on and off as an external camera changes 
modes. If not then showing on when any camera is usable and off when it isn't 
seems like sensible behaviour.

> 5) More in general, more and more software is dynamically monitoring the
> addition / removal of (usb) devices using udev, our current solution
> suggests to this software the /dev/video device is being unplugged /
> re-plugged all the time, not pretty.
> 
> 
> All in all what we've today is a kludge, and if we want to provide
> a "seamless" user experience we need to fix it.

I think in summary I'm concerned about the possibility of perfect being the 
enemy of good enough. At the moment we've got a significant usability problem 
(a web search for gvfs-gphoto2-volume-monitor turns up mostly instructions on 
how to disable it). If we come up with a solution that whilst it would be 
perfect there isn't enough effort available to implement then that is worse 
than a solution that fixes most of the problem. This is an even greater 
concern when the technically superior solution has a higher long term 
maintenance overhead (as we no longer get Win32 and OSX users helping to 
maintain the stillcam drivers).

I'm not sure if there is anything in this discussion that is relevant to the 
cameras in phones or tablets. These appear to the user as if they are dual 
mode devices but they don't have any independent storage - taking a photo is 
more like capturing a single higher than usual res frame so I suspect they 
aren't going to be an issue.

Regards

Adam

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/10/2011 06:09 PM, Alan Stern wrote:
> On Wed, 10 Aug 2011, Theodore Kilgore wrote:
>
>>> Okay, I didn't realize that the different cameras used different webcam
>>> drivers as well as different stillcam drivers.
>>
>> Oh, yes. They are Proprietary devices. And that means what it says. :-)
>> And all different from each other, too.
>>
>>> As far as I can see, there's nothing to stop anybody from adding the
>>> stillcam functionality into the webcam drivers right now.  If some
>>> common code can be abstracted out into a shared source file, so much
>>> the better.
>>>
>>> That would solve the problem, right?
>>
>> I think everyone involved believes that it would solve the problem.
>>
>> The question has been all along whether or not there is any other way
>> which would work. Also the question of what, exactly, "belongs" in the
>> kernel and what does not. For, if something has been historically
>> supported in userspace (stillcam support, in this case) and has worked
>> well there, I would think it is kind of too bad to have to move said
>> support into the kernel just because the same hardware requires kernel
>> support for another functionality and the two sides clash. I mean, the
>> kernel is already big enough, no? But the logic that Hans has set forth
>> seems rather compelling.
>
> The alternative seems to be to define a device-sharing protocol for USB
> drivers.  Kernel drivers would implement a new callback (asking them to
> give up control of the device), and usbfs would implement new ioctls by
> which a program could ask for and relinquish control of a device.  The
> amount of rewriting needed would be relatively small.
>
> A few loose ends would remain, such as how to handle suspends, resumes,
> resets, and disconnects.  Assuming usbfs is the only driver that will
> want to share a device in this way, we could handle them.
>
> Hans, what do you think?
>

First of all thanks for the constructive input!

When you say: "device-sharing protocol", do you mean 2 drivers been
attached, but only 1 being active. Or just some additional glue to make
hand-over between them work better?

I've 2 concerns with this approach:
1) Assuming we are going for the just make hand over work better solution
we will still have the whole disappear / reappear act of the /dev/video#
node, which I don't like at all.

If for example skype gets started it will say the user has no camera. If it
were to say the device is busy, the user just might make a link to some
application using the device in stillcam mode still being open.

2) The whole notion of the device being in use is rather vague when it comes
to the userspace parts of this. Simply leaving say F-Spot running, or having
a gvfs libgphoto share mounted, should not lead to not being able to use the
device in webcam mode. But currently it will.

Fixing all users of libgphoto2 wrt this is unlikely to happen, and even if
we do that now, more broken ones will likely come along later. I estimate
98% of all cameras are not dual mode cameras, so the average stillcam
application developer will not test for this.

That leaves us with fixing the busy notion inside libgphoto2, iow, release
the device as soon as an operation has completed. This will be quite slow,
since both drivers don't know anything about each other, they will just
know there is some $random_other_driver. So they need to assume the
device state is unclean and re-init the device from scratch each time.

Where as if we have both functions in one driver, that can remember the
actual device state and only make changes if needed, so downloading +
deleting 10 photos will lead to setting it to stillcam mode once, rather
then 20 times.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/11/2011 01:04 AM, Adam Baker wrote:
> On Tuesday 09 August 2011, Hans de Goede wrote:
>> Hi,
>>
>> On 08/09/2011 10:31 PM, Adam Baker wrote:
>>> On Tuesday 09 August 2011, Hans de Goede wrote:
>> <snip>
>>
>>> It has also just occured to me that it might be possible to solve the
>>> issues we are facing just in the kernel. At the moment when the kernel
>>> performs a USBDEVFS_DISCONNECT it keeps the kernel driver locked out
>>> until userspace performs a USBDEVFS_CONNECT. If the kernel reattached
>>> the kernel driver when the device file was closed then, as gvfs doesn't
>>> keep the file open the biggest current issue would be solved instantly.
>>> If a mechanism could be found to prevent USBDEVFS_DISCONNECT from
>>> succeeding when the corresponding /dev/videox file was open then that
>>> would seem to be a reasonable solution.
>>
>> <sigh>
>>
>> This has been discussed over and over and over again, playing clever
>> tricks with USBDEVFS_[DIS]CONNECT like adding a new USBDEVFS_TRYDISCONNECT
>> which the v4l2 driver could intercept won't cut it. We need some central
>> manager of the device doing multiplexing between the 2 functions, and you
>> can *not* assume that either side will be nice wrt closing file
>> descriptors.
>>
>> Examples:
>> 1) You are wrong wrt gvfs, it does keep the libgphoto2 context open all the
>> time, and through that the usbfs device nodes.
>
> It seems that that depends, on my system gvfs isn't actually automounting the
> camera after it detects it and the file is only open (according to lsof) when
> the device is actually mounted.

Right, but on some systems it does auto-mount, and even if it does not
auto-mount and the user then just clicks on its icon in the filemanager
to look at the photo's it will mount, and it will be non obvious to the
user to umount it.

Once mounted, the /dev/video# node will be gone. So if for example skype
gets started it will say the user has no camera. If it were to say the device
is busy, the user just might make a link to the mounted gvfs share, but even
then that is a big might.

The whole disappear / reappear act of the /dev/video# node is one of my
concerns with the current solution.

> As soon as you unmount it the device gets
> closed again. Because it does do a brief open,  USBDEVFS_DISCONNECT then close
> at connection time it does still disable the kernel driver.

Then you've a bug in your libgphoto2, Fedora had a similar bug which I fixed
(and send the fix upstream). libgphoto2 now a days does make the call to
re-attach the kernel driver.

>
>>
>> 2) Lets say a user starts a photo managing app like f-spot, and that opens
>> the device through libgphoto2 on startup, then the user switches to another
>> virtual desktop and forgets all about having f-spot open. Notice that if
>> the user now tries to stream he will not get a busy error, but the app
>> trying to do the streaming will simply not see the camera at all (kernel
>> driver unbound /dev/video# node is gone).
>
> This does seem like a situation where your approach could potentially give a
> better user experience. I'm wondering slightly how you define busy though. For
> webcams the streamon and streamoff ioctls tell you if you are using mmap or
> userptr transfers but you don't know if when the user has finished if they
> just use read. For stillcam mode it is again hard to determine a busy
> condition other than being in the middle of transfering an individual picture.

Right, so my plan is to define busy for stillcam mode as not being in
the middle of some operation (be it download or delete).

>> 3) Notice that little speaker icon in your panel on your average Linux
>> desktop, that keeps the mixer of the audio device open *all the time* it
>> is quite easy to imagine a similar applet for v4l2 device controls (see
>> for example gtk-v4l) doing the same. Or a user could simply start up a
>> v4l2 control panel app like gtk-v4l, qv4l2 or v4l2ucp, and leave it running
>> minimized ...
>>
>
> This again needs a usable concept of busy

Actually this one is easy. Most streaming drivers already handle controls
while not streaming by remembering the set values, and these cached values
usually then get set on stream start. There are multiple reasons for
this:
- usb transfers are expensive no need to send a value to the device
   if it may change before it gets actually used
- in many cases parts of the device are powered down when not streaming,
   so sending values while not streaming just drops them in a black hole


<snip>

> If we come up with a solution that whilst it would be
> perfect there isn't enough effort available to implement then that is worse
> than a solution that fixes most of the problem.

Agreed.

> This is an even greater
> concern when the technically superior solution has a higher long term
> maintenance overhead (as we no longer get Win32 and OSX users helping to
> maintain the stillcam drivers).

I've been subscribed to the libgphoto2-devel list for quite some time now,
I believe I still have to see the first win32 / OS-X user input. We do get
occasional FreeBSD input, usually in the form of "it no longer compiles".

I'm not saying that portability is not important, but contributions
from users on other platforms should not be overrated either.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Jean-Francois Moine]

On Wed, 10 Aug 2011 17:14:05 -0300
Mauro Carvalho Chehab <mchehab@infradead.org> wrote:

> > -- moving the kernel webcam drivers out of the kernel and doing with these 
> > cameras _everything_ including webcam function through libusb. I myself do 
> > not have the imagination to be able to figure out how this could be done 
> > without a rather humongous amount of work (for example, which streaming 
> > apps that are currently available would be able to live with this?) but 
> > unless I misunderstand what he was saying, Greg K-H seems to think that 
> > this would be the best thing to do.  
> 
> I also don't think that this a good alternative. As Hans V. pointed, one of
> our long term targets is to create per-sensor I2C drivers that are independent
> from the bridges. Also, moving it to userspace would require lots of work
> with the duplication of V4L and gspca core into userspace for the devices
> that would be moved, and may have some performance impacts.

If only gspca subdrivers are concerned, moving them to usespace should
be easy: only the gspca main has to be rewritten, and its best place is
in the v4l library.

-- 
Ken ar c'hentañ	|	      ** Breizh ha Linux atav! **
Jef		|		http://moinejf.free.fr/

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 11-08-2011 05:57, Jean-Francois Moine escreveu:
> On Wed, 10 Aug 2011 17:14:05 -0300
> Mauro Carvalho Chehab <mchehab@infradead.org> wrote:
> 
>>> -- moving the kernel webcam drivers out of the kernel and doing with these 
>>> cameras _everything_ including webcam function through libusb. I myself do 
>>> not have the imagination to be able to figure out how this could be done 
>>> without a rather humongous amount of work (for example, which streaming 
>>> apps that are currently available would be able to live with this?) but 
>>> unless I misunderstand what he was saying, Greg K-H seems to think that 
>>> this would be the best thing to do.  
>>
>> I also don't think that this a good alternative. As Hans V. pointed, one of
>> our long term targets is to create per-sensor I2C drivers that are independent
>> from the bridges. Also, moving it to userspace would require lots of work
>> with the duplication of V4L and gspca core into userspace for the devices
>> that would be moved, and may have some performance impacts.
> 
> If only gspca subdrivers are concerned, moving them to usespace should
> be easy: only the gspca main has to be rewritten, and its best place is
> in the v4l library.

It would end by having two userspace drivers, as the data access driver is 
at libgphoto2. So, the problem will just be moved to userspace. You'll
additionally need to deal with conflicts between kernelspace ALSA driver
and userspace video driver.

Regards,
Mauro.

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Thu, 11 Aug 2011, Hans de Goede wrote:

> > The alternative seems to be to define a device-sharing protocol for USB
> > drivers.  Kernel drivers would implement a new callback (asking them to
> > give up control of the device), and usbfs would implement new ioctls by
> > which a program could ask for and relinquish control of a device.  The
> > amount of rewriting needed would be relatively small.
> >
> > A few loose ends would remain, such as how to handle suspends, resumes,
> > resets, and disconnects.  Assuming usbfs is the only driver that will
> > want to share a device in this way, we could handle them.
> >
> > Hans, what do you think?
> >
> 
> First of all thanks for the constructive input!
> 
> When you say: "device-sharing protocol", do you mean 2 drivers been
> attached, but only 1 being active. Or just some additional glue to make
> hand-over between them work better?

I was thinking that the webcam driver would always be attached, but 
from time to time usbfs would ask to use the device.  When the webcam 
driver gives away control, it remains bound to the device but does not 
send any URBs.  If it needs to send an URB, first it has to ask usbfs 
to give control back.

> I've 2 concerns with this approach:
> 1) Assuming we are going for the just make hand over work better solution
> we will still have the whole disappear / reappear act of the /dev/video#
> node, which I don't like at all.

That will not happen any more, because the webcam driver will always be 
bound to the device.

> If for example skype gets started it will say the user has no camera. If it
> were to say the device is busy, the user just might make a link to some
> application using the device in stillcam mode still being open.
> 
> 2) The whole notion of the device being in use is rather vague when it comes
> to the userspace parts of this. Simply leaving say F-Spot running, or having
> a gvfs libgphoto share mounted, should not lead to not being able to use the
> device in webcam mode. But currently it will.

That's true -- but it's true no matter what solution we adopt.  The
various drivers (whether in the kernel or in userspace) will have to
decide for themselves when they can give up control.

> Fixing all users of libgphoto2 wrt this is unlikely to happen, and even if
> we do that now, more broken ones will likely come along later. I estimate
> 98% of all cameras are not dual mode cameras, so the average stillcam
> application developer will not test for this.

Not all users of libgphoto2 have to be changed; only those which manage
dual-mode cameras.  Adding a few ioctls to ask for and give up control
at the appropriate times must be a lot easier than porting the entire
driver into the kernel.

> That leaves us with fixing the busy notion inside libgphoto2, iow, release
> the device as soon as an operation has completed. This will be quite slow,
> since both drivers don't know anything about each other, they will just
> know there is some $random_other_driver. So they need to assume the
> device state is unclean and re-init the device from scratch each time.

Well, a user program can assume that the kernel driver left the device
in a clean state.  The reverse isn't always true, however -- it's one
of the drawbacks of using a userspace driver.

Besides, even though drivers don't always have to re-init the device
from scratch every time, they do send all the current settings each
time they use the device.  So maybe the extra overhead is tolerable.

> Where as if we have both functions in one driver, that can remember the
> actual device state and only make changes if needed, so downloading +
> deleting 10 photos will lead to setting it to stillcam mode once, rather
> then 20 times.

Depends how the ask-for-control ioctl is implemented.  It might return
a value indicating whether or not the webcam driver took control during
the interval when the user program wasn't using the device.  If usbfs
retained control the entire time, the program should be able to assume
the device's state hasn't changed.

I'm not claiming that this is a better solution than putting everything
in the kernel.  Just that it is a workable alternative which would
involve a lot less coding.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 11-08-2011 11:56, Alan Stern escreveu:
> On Thu, 11 Aug 2011, Hans de Goede wrote:
> 
>>> The alternative seems to be to define a device-sharing protocol for USB
>>> drivers.  Kernel drivers would implement a new callback (asking them to
>>> give up control of the device), and usbfs would implement new ioctls by
>>> which a program could ask for and relinquish control of a device.  The
>>> amount of rewriting needed would be relatively small.
>>>
>>> A few loose ends would remain, such as how to handle suspends, resumes,
>>> resets, and disconnects.  Assuming usbfs is the only driver that will
>>> want to share a device in this way, we could handle them.
>>>
>>> Hans, what do you think?
>>>
>>
>> First of all thanks for the constructive input!
>>
>> When you say: "device-sharing protocol", do you mean 2 drivers been
>> attached, but only 1 being active. Or just some additional glue to make
>> hand-over between them work better?
> 
> I was thinking that the webcam driver would always be attached, but 
> from time to time usbfs would ask to use the device.  When the webcam 
> driver gives away control, it remains bound to the device but does not 
> send any URBs.  If it needs to send an URB, first it has to ask usbfs 
> to give control back.
> 
>> I've 2 concerns with this approach:
>> 1) Assuming we are going for the just make hand over work better solution
>> we will still have the whole disappear / reappear act of the /dev/video#
>> node, which I don't like at all.
> 
> That will not happen any more, because the webcam driver will always be 
> bound to the device.
> 
>> If for example skype gets started it will say the user has no camera. If it
>> were to say the device is busy, the user just might make a link to some
>> application using the device in stillcam mode still being open.
>>
>> 2) The whole notion of the device being in use is rather vague when it comes
>> to the userspace parts of this. Simply leaving say F-Spot running, or having
>> a gvfs libgphoto share mounted, should not lead to not being able to use the
>> device in webcam mode. But currently it will.
> 
> That's true -- but it's true no matter what solution we adopt.  The
> various drivers (whether in the kernel or in userspace) will have to
> decide for themselves when they can give up control.

Between two or more kernel drivers, a resource locking mechanism like the one 
you've proposed works fine, but, when the driver is on userspace, there's one
additional issue that needs to be addressed: What happens if, for example,
if a camera application using libgphoto2 crashes? The lock will be enabled, and
the V4L driver will be locked forever. Also, if the lock is made generic enough
to protect between two different userspace applications, re-starting the
camera application won't get the control back.

To avoid such risk, kernel might need to implement some ugly hacks to detect
when the application was killed, and put the device into a sane state, if this
happens.

>> Fixing all users of libgphoto2 wrt this is unlikely to happen, and even if
>> we do that now, more broken ones will likely come along later. I estimate
>> 98% of all cameras are not dual mode cameras, so the average stillcam
>> application developer will not test for this.
> 
> Not all users of libgphoto2 have to be changed; only those which manage
> dual-mode cameras.  Adding a few ioctls to ask for and give up control
> at the appropriate times must be a lot easier than porting the entire
> driver into the kernel.

Again, applications that won't implement this control will take the lock forever.

>> That leaves us with fixing the busy notion inside libgphoto2, iow, release
>> the device as soon as an operation has completed. This will be quite slow,
>> since both drivers don't know anything about each other, they will just
>> know there is some $random_other_driver. So they need to assume the
>> device state is unclean and re-init the device from scratch each time.
> 
> Well, a user program can assume that the kernel driver left the device
> in a clean state.  The reverse isn't always true, however -- it's one
> of the drawbacks of using a userspace driver.
> 
> Besides, even though drivers don't always have to re-init the device
> from scratch every time, they do send all the current settings each
> time they use the device.  So maybe the extra overhead is tolerable.
> 
>> Where as if we have both functions in one driver, that can remember the
>> actual device state and only make changes if needed, so downloading +
>> deleting 10 photos will lead to setting it to stillcam mode once, rather
>> then 20 times.
> 
> Depends how the ask-for-control ioctl is implemented.  It might return
> a value indicating whether or not the webcam driver took control during
> the interval when the user program wasn't using the device.  If usbfs
> retained control the entire time, the program should be able to assume
> the device's state hasn't changed.
> 
> I'm not claiming that this is a better solution than putting everything
> in the kernel.  Just that it is a workable alternative which would
> involve a lot less coding.
> 
> Alan Stern
> 

Re: USB mini-summit at LinuxCon Vancouver

[Alan Cox]

> Between two or more kernel drivers, a resource locking mechanism like the one 
> you've proposed works fine, but, when the driver is on userspace, there's one
> additional issue that needs to be addressed: What happens if, for example,
> if a camera application using libgphoto2 crashes? The lock will be enabled, and
> the V4L driver will be locked forever. Also, if the lock is made generic enough
> to protect between two different userspace applications, re-starting the
> camera application won't get the control back.

Actually there are more issues than that - you've also got to worry about
a security/permission model, and that is hard to get right, especially if
you are not very careful that anything that can be retrieved which might
violate the security model (eg the last frame on the capture) has been
blanked before handover etc.

> To avoid such risk, kernel might need to implement some ugly hacks to detect
> when the application was killed, and put the device into a sane state, if this
> happens.

At which point history says it's easier to do the job right, once, in the
kernel.

> Again, applications that won't implement this control will take the lock forever.

And applications that are touching both video (even indirectly) and still
camera may get surprise deadlocks if they accidentally reference both the
still and video device even via some library or service.

> > Well, a user program can assume that the kernel driver left the device
> > in a clean state.  The reverse isn't always true, however -- it's one

Not it cannot - the user program doesn't know

a) if the kernel driver has ever been loaded
b) the values the kernel driver leaves set (and those will change no
doubt at times)
c) if the camera has been plugged and unplugged and not yet had the
kernel driver loaded

To me it sounds like a recipe for disaster. For those tiny number of
devices involved just use V4L and if need be some small V4L tweaks to
handle still mode. In most cases the interface is basically identical and
I'd bet much of the code is identical too.

Alan

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

Alan Cox raised a bunch of good points.  I'm not going to respond to 
them; they pretty much speak for themselves.

On Thu, 11 Aug 2011, Mauro Carvalho Chehab wrote:

> Between two or more kernel drivers, a resource locking mechanism like the one 
> you've proposed works fine,

It's not a locking mechanism.  More like cooperative multi-access.

>  but, when the driver is on userspace, there's one
> additional issue that needs to be addressed: What happens if, for example,
> if a camera application using libgphoto2 crashes? The lock will be enabled, and
> the V4L driver will be locked forever. Also, if the lock is made generic enough
> to protect between two different userspace applications, re-starting the
> camera application won't get the control back.

You're wrong, because what I proposed isn't a lock.  If the user 
program dies, the usbfs device file will automatically be closed and 
then usbfs will freely give control back to the webcam driver.

If the program hangs at the wrong time, then the webcam driver won't be 
able to regain control.  At least the user will have the option of 
killing the program when this happens; a similar hang in a kernel 
driver tends to be much uglier.

Two different user programs trying to drive the device at the same time 
doesn't necessarily have to cause a problem.  At any particular moment, 
only one of them would be in control of the device.

> To avoid such risk, kernel might need to implement some ugly hacks to detect
> when the application was killed, and put the device into a sane state, if this
> happens.

No ugly hack is needed.  usbfs can directly inform the webcam driver 
whether or not the device file was closed while the user program 
retained control.  If that didn't happen, it's safe to assume that the 
program gave up control voluntarily.

> > Not all users of libgphoto2 have to be changed; only those which manage
> > dual-mode cameras.  Adding a few ioctls to ask for and give up control
> > at the appropriate times must be a lot easier than porting the entire
> > driver into the kernel.
> 
> Again, applications that won't implement this control will take the lock forever.

No, because there is no lock.  Programs that haven't been changed will 
continue to behave as they do today -- they will unbind the webcam 
driver and assume full control of the device.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Thu, 11 Aug 2011, Alan Cox wrote:

> Actually there are more issues than that - you've also got to worry about
> a security/permission model, and that is hard to get right, especially if
> you are not very careful that anything that can be retrieved which might
> violate the security model (eg the last frame on the capture) has been
> blanked before handover etc.

As far as I can tell, these same security issues exist today.  I don't 
see them getting any worse than they are now.

> And applications that are touching both video (even indirectly) and still
> camera may get surprise deadlocks if they accidentally reference both the
> still and video device even via some library or service.

No, not deadlocks.  Just -EBUSY errors.

> > > Well, a user program can assume that the kernel driver left the device
> > > in a clean state.  The reverse isn't always true, however -- it's one
> 
> Not it cannot - the user program doesn't know
> 
> a) if the kernel driver has ever been loaded
> b) the values the kernel driver leaves set (and those will change no
> doubt at times)
> c) if the camera has been plugged and unplugged and not yet had the
> kernel driver loaded

That's true.  The program can't assume that a kernel driver was ever 
bound to the device; all it can assume is that _if_ a kernel driver was 
bound then it left the device in a sane state -- whatever "sane" might 
mean in this context.

> To me it sounds like a recipe for disaster. For those tiny number of
> devices involved just use V4L and if need be some small V4L tweaks to
> handle still mode. In most cases the interface is basically identical and
> I'd bet much of the code is identical too.

I'm not against moving the whole thing into the kernel.  I'm just
pointing out that an easier-to-code-up solution will accomplish much
the same result.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Thu, 11 Aug 2011, Alan Stern wrote:

> On Thu, 11 Aug 2011, Alan Cox wrote:
> 
> > Actually there are more issues than that - you've also got to worry about
> > a security/permission model, and that is hard to get right, especially if
> > you are not very careful that anything that can be retrieved which might
> > violate the security model (eg the last frame on the capture) has been
> > blanked before handover etc.
> 
> As far as I can tell, these same security issues exist today.  I don't 
> see them getting any worse than they are now.
> 
> > And applications that are touching both video (even indirectly) and still
> > camera may get surprise deadlocks if they accidentally reference both the
> > still and video device even via some library or service.
> 
> No, not deadlocks.  Just -EBUSY errors.
> 
> > > > Well, a user program can assume that the kernel driver left the device
> > > > in a clean state.  The reverse isn't always true, however -- it's one
> > 
> > Not it cannot - the user program doesn't know
> > 
> > a) if the kernel driver has ever been loaded
> > b) the values the kernel driver leaves set (and those will change no
> > doubt at times)
> > c) if the camera has been plugged and unplugged and not yet had the
> > kernel driver loaded
> 
> That's true.  The program can't assume that a kernel driver was ever 
> bound to the device; all it can assume is that _if_ a kernel driver was 
> bound then it left the device in a sane state -- whatever "sane" might 
> mean in this context.
> 
> > To me it sounds like a recipe for disaster. For those tiny number of
> > devices involved just use V4L and if need be some small V4L tweaks to
> > handle still mode. In most cases the interface is basically identical and
> > I'd bet much of the code is identical too.
> 
> I'm not against moving the whole thing into the kernel.  I'm just
> pointing out that an easier-to-code-up solution will accomplish much
> the same result.
> 
> Alan Stern
> 

Alan,

As I said, I am agnostic, though leaning in the direction that Hans de 
Goede is pointing. What he says about a single control mechanism seems to 
make a lot of sense. If you can come up with an outline of the "easier to 
code" solution, that would be interesting, though.

I assume you are also going to be in Vancouver? If you will be there on 
Monday, then Hans and I are already planning to meet and discuss. 

BTW, as to using V4L with "tweaks" to handle still mode, it would probably 
be more difficult than is imagined. For, though the operations required to 
process still images and webcam frames are in principle similar, the 
priorities and constraints are too different. Therefore, my understanding 
is that the libgphoto2 image processing routines, not the libv4l image 
processing routines, would still be used for still images.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Mauro Carvalho Chehab]

Em 11-08-2011 17:01, Theodore Kilgore escreveu:

> As I said, I am agnostic, though leaning in the direction that Hans de 
> Goede is pointing. What he says about a single control mechanism seems to 
> make a lot of sense. If you can come up with an outline of the "easier to 
> code" solution, that would be interesting, though.
> 
> I assume you are also going to be in Vancouver? If you will be there on 
> Monday, then Hans and I are already planning to meet and discuss. 
> 
> BTW, as to using V4L with "tweaks" to handle still mode, it would probably 
> be more difficult than is imagined. For, though the operations required to 
> process still images and webcam frames are in principle similar, the 
> priorities and constraints are too different. Therefore, my understanding 
> is that the libgphoto2 image processing routines, not the libv4l image 
> processing routines, would still be used for still images.

I agree with Alan Cox: most of the code that the driver needs is already there: 
register read/write routines, bulk transfer support, etc. The amount of extra 
code for adding still cam functionality is probably not big.

>From the kernel driver's perspective, it doesn't matter if the access will come
via libv4l, libgphoto2 or whatever. The driver should be able to allow simultaneous
open, while protecting the data access when userspace requests data stream or
still image retrieve.

instead of using the V4L2 device node to access the stored images, it probably makes 
more sense to use a separate device for that, that will handle a separate set of 
ioctl's, and just use read() to retrieve the image data, after selecting the desired
image number, via ioctl().

It probably makes sense to add a new set of callbacks at the gspca core in order
to handle the new device node, and letting it to avoid start streaming while the
store access is happening, and vice-versa. Alternatively, we may create a separate
"still cam" core library to handle the new device node,.

If all agree around such solution, I suggest to take the most complex case and try
to map it into the driver and core, and see how it behaves, testing with some simple
command line applications, only changing the libgphoto2 code after those initial
tests. Writing a simple code for reading still images should be easy, and we have 
already some testing tools for V4L2.

After coding the core changes that are common to all drives, I suspect that adding 
the remaining 4 drivers will be quick.

With regards to libgphoto2, all it needs to do is to test if the new device nodes
exist. If they exist, then the new code will be used. Otherwise, it will fallback
to the libusb. This way, we can incrementally add the Dual mode drivers into the
kernel.

There is one advantage on using this strategy: if, in the future, new Dual Cams
arise, one can write first a still cam kernel driver, adding V4L support later.

Cheers,
Mauro

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Thu, 11 Aug 2011, Mauro Carvalho Chehab wrote:

> Em 11-08-2011 17:01, Theodore Kilgore escreveu:
> 
> > As I said, I am agnostic, though leaning in the direction that Hans de 
> > Goede is pointing. What he says about a single control mechanism seems to 
> > make a lot of sense. If you can come up with an outline of the "easier to 
> > code" solution, that would be interesting, though.
> > 
> > I assume you are also going to be in Vancouver? If you will be there on 
> > Monday, then Hans and I are already planning to meet and discuss. 
> > 
> > BTW, as to using V4L with "tweaks" to handle still mode, it would probably 
> > be more difficult than is imagined. For, though the operations required to 
> > process still images and webcam frames are in principle similar, the 
> > priorities and constraints are too different. Therefore, my understanding 
> > is that the libgphoto2 image processing routines, not the libv4l image 
> > processing routines, would still be used for still images.
> 
> I agree with Alan Cox: most of the code that the driver needs is already there: 
> register read/write routines, bulk transfer support, etc. The amount of extra 
> code for adding still cam functionality is probably not big.
> 
> >From the kernel driver's perspective, it doesn't matter if the access will come
> via libv4l, libgphoto2 or whatever. The driver should be able to allow simultaneous
> open, while protecting the data access when userspace requests data stream or
> still image retrieve.
> 
> instead of using the V4L2 device node to access the stored images, it probably makes 
> more sense to use a separate device for that, that will handle a separate set of 
> ioctl's, and just use read() to retrieve the image data, after selecting the desired
> image number, via ioctl().
> 
> It probably makes sense to add a new set of callbacks at the gspca core in order
> to handle the new device node, and letting it to avoid start streaming while the
> store access is happening, and vice-versa. Alternatively, we may create a separate
> "still cam" core library to handle the new device node,.
> 
> If all agree around such solution, I suggest to take the most complex case and try
> to map it into the driver and core, and see how it behaves, testing with some simple
> command line applications, only changing the libgphoto2 code after those initial
> tests. Writing a simple code for reading still images should be easy, and we have 
> already some testing tools for V4L2.
> 
> After coding the core changes that are common to all drives, I suspect that adding 
> the remaining 4 drivers will be quick.
> 
> With regards to libgphoto2, all it needs to do is to test if the new device nodes
> exist. If they exist, then the new code will be used. Otherwise, it will fallback
> to the libusb. This way, we can incrementally add the Dual mode drivers into the
> kernel.
> 
> There is one advantage on using this strategy: if, in the future, new Dual Cams
> arise, one can write first a still cam kernel driver, adding V4L support later.
> 
> Cheers,
> Mauro
> 

Try it out with one camera seems to me to be a reasonable way 
forward. 

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Thu, 11 Aug 2011, Theodore Kilgore wrote:

> Alan,
> 
> As I said, I am agnostic, though leaning in the direction that Hans de 
> Goede is pointing. What he says about a single control mechanism seems to 
> make a lot of sense. If you can come up with an outline of the "easier to 
> code" solution, that would be interesting, though.

That approach has been outlined in the other emails I have posted 
recently.

> I assume you are also going to be in Vancouver? If you will be there on 
> Monday, then Hans and I are already planning to meet and discuss. 

No, I'm not going to Vancouver.  However I will attend the Linux 
Plumbers conference in Santa Rosa.

Alan Stern

Re: USB mini-summit at LinuxCon Vancouver

[Theodore Kilgore]

On Thu, 11 Aug 2011, Alan Stern wrote:

<snip>

> No, I'm not going to Vancouver.  

Well, someday, somewhere, we will be together in the same place at the 
same time and able to enjoy a good seafood restaurant.

However I will attend the Linux 
> Plumbers conference in Santa Rosa.

Which I expect I will not. 

Until next time we meet, cheers.

Theodore Kilgore

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/11/2011 10:32 PM, Mauro Carvalho Chehab wrote:

<snip stuff I agree with>

> instead of using the V4L2 device node to access the stored images, it probably makes
> more sense to use a separate device for that, that will handle a separate set of
> ioctl's, and just use read() to retrieve the image data, after selecting the desired
> image number, via ioctl().

I don't see a lot of added value in doing things this way. We can simply
have a set of new ioctls on the /dev/video# node for this and a new
V4L2_CAP_STILL_IMAGE to indicate the availability of these ioctls. This will keep
the driver a lot simpler then doing 2 separate device nodes for 1 device.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Hans de Goede]

Hi,

On 08/11/2011 04:56 PM, Alan Stern wrote:
> On Thu, 11 Aug 2011, Hans de Goede wrote:
>
>>> The alternative seems to be to define a device-sharing protocol for USB
>>> drivers.  Kernel drivers would implement a new callback (asking them to
>>> give up control of the device), and usbfs would implement new ioctls by
>>> which a program could ask for and relinquish control of a device.  The
>>> amount of rewriting needed would be relatively small.
>>>
>>> A few loose ends would remain, such as how to handle suspends, resumes,
>>> resets, and disconnects.  Assuming usbfs is the only driver that will
>>> want to share a device in this way, we could handle them.
>>>
>>> Hans, what do you think?
>>>
>>
>> First of all thanks for the constructive input!
>>
>> When you say: "device-sharing protocol", do you mean 2 drivers been
>> attached, but only 1 being active. Or just some additional glue to make
>> hand-over between them work better?
>
> I was thinking that the webcam driver would always be attached, but
> from time to time usbfs would ask to use the device.  When the webcam
> driver gives away control, it remains bound to the device but does not
> send any URBs.  If it needs to send an URB, first it has to ask usbfs
> to give control back.
>

Oh, interesting...

<snip lots of good stuff>

> I'm not claiming that this is a better solution than putting everything
> in the kernel.  Just that it is a workable alternative which would
> involve a lot less coding.

This is definitely an interesting proposal, something to think about ...

I have 2 concerns wrt this approach:

1) It feels less clean then just having a single driver; and
2) I agree it will be less coding, but I doubt it will really be that much
less work. It will likely need less new code (but a lot can be more or
less copy pasted), but it will need changes across a wider array of
subsystems / userspace components, requiring a lot of coordinating,
getting patches merged in different projects, etc. So in the end I
think it too will be quite a bit of work.

I guess that what I'm trying to say here is, that if we are going to
spend a significant amount of time on this, we might just as well
go for the best solution we can come up with even if that is some
more work.

Regards,

Hans

Re: USB mini-summit at LinuxCon Vancouver

[Alan Cox]

> instead of using the V4L2 device node to access the stored images, it probably makes 
> more sense to use a separate device for that, that will handle a separate set of 
> ioctl's, and just use read() to retrieve the image data, after selecting the desired
> image number, via ioctl().

How will you handle the permission model, what about devices where you
need the V4L2 controls to take the picture in the first place ?

It seems it should really be the same device and just a minor tweak to
the API. V4L2 already provides frame reading and mapping functionality
and the media layer provides post processing services which I could
conceive some setups needing to use.

gphoto would just open the v4l device ask for still image stuff and
discover it wasn't available.

Re: USB mini-summit at LinuxCon Vancouver

[Alan Stern]

On Fri, 12 Aug 2011, Hans de Goede wrote:

> > I'm not claiming that this is a better solution than putting everything
> > in the kernel.  Just that it is a workable alternative which would
> > involve a lot less coding.
> 
> This is definitely an interesting proposal, something to think about ...
> 
> I have 2 concerns wrt this approach:
> 
> 1) It feels less clean then just having a single driver; and

Agreed.

> 2) I agree it will be less coding, but I doubt it will really be that much
> less work. It will likely need less new code (but a lot can be more or
> less copy pasted), but it will need changes across a wider array of
> subsystems / userspace components, requiring a lot of coordinating,
> getting patches merged in different projects, etc. So in the end I
> think it too will be quite a bit of work.
> 
> I guess that what I'm trying to say here is, that if we are going to
> spend a significant amount of time on this, we might just as well
> go for the best solution we can come up with even if that is some
> more work.

Okay, go ahead.  I have no objection.

Alan Stern